[Koha] Open source ballot design

Wed Oct 14 09:25:39 NZDT 2009

[Section 5.1 identifies an important voting preference problem which may
be very likely to occur in our final poll.]

Reply inline:

On Tue, October 13, 2009 09:21, MJ Ray wrote:

[...]

1.  INFORMED VOTING.

>  Should we turn the wiki pages into
> some sort of consultation information brochure?

The wiki pages could be much easier to read in general but that is a
different problem.  The wiki pages will do if we use them.  Before people
initiate the ballot, they should be requested to examine the wiki pages
and the mailing list archive to become an informed voter.

[...]

2,  TAKING COMMENTS INTO ACCUNT.

> I suspect statistics is less obvious to most of the community than
> making a great Library Management System, but I think it's worth
> posting the ballot design and taking comments into account.

I am pleased that you are not opposed to some more input.  I trust that
you advise Nicole Engard well about which comments are worth incorporating
and which would cause problems.

>
> I beg everyone to check that any suggested changes have some grounding
> in statistical theory before posting them, else the review process
> probably won't work because we'll get overloaded.  I wrote
> http://people.debian.org/~mjr/surveys.html a few years ago which might
> interest some people.

I really do not think that there will be very many comments overall.  I
doubt that many comments would even reach the level of skewing statistics.

3.  IRC MEETING.

>
> [...]
>> At Wednesday's IRC meeting it was announced, if I recall correctly, that
>> two weeks would be taken to analyse the previous poll before a final
>> ballot would be presented.  That period should give sufficient time for
>> most if not all of my suggestions to have an opportunity to have some
>> effect on the drafting of a final ballot.
>
> I feel that recollection is incorrect.  No particular time was
> suggested, although I'd prefer it not to drag forever. Please check:
> http://stats.workbuffer.org/irclog/koha/2009-10-07#i_316162

Checking the log it is not clear to me what the two week period referred
to in that meeting had been intended to cover.  Certainly there was a
statement of giving time for analysis of the previous results, a two week
period, and statements about informing the work for the next poll.  What
people are commenting about precisely can be difficult to determine.  From
the meeting I had expected that you would report some analysis of the
previous poll to everyone which could be used to inform the writing of the
final ballot.  Had I thought that there would be no open opportunity to
inform the process I would complained loudly at the meeting.

This is the trouble with ambiguous references and assuming that others
understand the same intent which you mean to convey.  This is why the
wording of the questions and options needs more eyes to find areas of
misunderstanding.

4.  INFORMATION SET.

>
> So:
>
> 1. do we need information brochure(s) about the options?
>    - who will produce it/them?
>    - who will referee it/them to make sure it's fair?
>    - how should it be distributed?

We could include a simple neutral one sentence descriptive statements next
to each option on the ballot.  Remember that koha.org has not had a
prominent statement of Koha's origins with HLT for a few years.  Few
librarians would know that SPI was established to provide a legal entity
for free software projects starting with Debian, the largest GNU/Linux
distribution.

Otherwise, the wiki and mailing list record will have to do for anything
more unless someone is volunteering to work extra hard.

5.  STATISTICAL PROBLEMS.

>
> 2. are there other trained statisicians in our community?

You may be the only one but that does not mean that you have anticipated
every problem and the best ways to resolve them.  You should inform Nicole
from a broader base of input.

I have read extensively on voting theory and maximising preferences years
ago.  That does not make me a trained statistician but no one in our
community with valuable input should be ignored.  We should strive to not
let the patches fall on the floor.

5.1.  VOTING PROBLEM.

The vote on first choices for forming a foundation in the previous survey
had an equal division between the "independent foundation now" option and
what had been presented as alternatives in the previous survey.

Have you considered how we should address the possibility that an absolute
majority may be in favour of a range of interim foundation choices but the
largest first rank vote count may be for "wait to agree on an independent
foundation"?

Which foundation forming option should we choose?

Hypothetical first rank vote.

Horowhenua Library Trust (HLT) for an interim period:  35%
Software Freedom Conservancy (Conservancy) for an interim period:  0%
Software in the Public Interest (SPI) for an interim period:  20%
Wait to agree on an independent foundation:  45%
No opinion:  0%

Hypothetical second rank vote.

Horowhenua Library Trust (HLT) for interim an period:  60%
Software Freedom Conservancy (Conservancy) for an interim period:  0%
Software in the Public Interest (SPI) for an interim period:  35%
Wait to agree on an independent foundation:  5%
No opinion:  0%

Hypothetical third rank vote.

Horowhenua Library Trust (HLT) for an interim period:  5%
Software Freedom Conservancy (Conservancy) for an interim period:  50%
Software in the Public Interest (SPI) for an interim period:  40%
Wait to agree on an independent foundation:  5%
No opinion:  0%
Hypothetical fourth rank vote.

Horowhenua Library Trust (HLT) for an interim period:  5%
Software Freedom Conservancy (Conservancy) for an interim period: 35%
Software in the Public Interest (SPI) for an interim period:  5%
Wait to agree on an independent foundation:  55%
No opinion:  0%

Comparing each option to the other as if many two option ballots had been
cast could show a stronger weight for one preference in what would become
the critical comparison.  Do you prefer HLT or "wait to agree on an
independent foundation"?

5.1.1.  RUN-OFF ELECTION REMEDY.

An easy remedy in the absence of an absolute majority is holding a run-off
poll with a single two option choice.  Such a remedy would probably be the
simplest to understand but it unfairly excludes other options which people
may prefer without having a ballot of many options comparing each to the
other.

A ballot with many comparisons to the other can be difficult for people to
understand.

Any run-off election is an unnecessary extra poll to be conducted. 
However, some form of run-off election would be preferable to accepting a
minority choice which merely happened to have more votes in a multi-option
ballot.

5.1.2.  PURE RANKED PREFERENCE REMEDY.

A ranked preference ballot could elicit that information but people should
know that is how it would be used.  The Schulze method is a ranked
preference analysis which might be used in comparing each choice to the
other as if many ballots would be held with only two options,
http://en.wikipedia.org/wiki/Schulze_method .

This Schulze method is generally considered the fairest system for
mathematically maximising people's preferences in voting.

A modified Schulze method is used by the Debian project in voting but some
of their modifications can distort the statistics to disfavour a majority
vote in some circumstances.

5.1.3.  PARTIAL RANKED PREFERENCE REMEDY.

Another way of treating this particular issue as it can be anticipated is
to break the question down by type of choice.

What type of foundation forming option should we use to start a foundation?
A.  Form interim foundation while discussing an independent foundation
following majority choice of interim home.  (Exclude option B.)  [  ]
B.  Only wait to agree on an independent foundation. (Exclude option A.) [  ]
C.  No opinion.  [  ]

All the options including the "wait to agree on an independent foundation"
option could be listed on the next page for ranking.  If an absolute
majority would select A, then the largest preference for interim options
would be selected.  The Schulze method could still be used for comparing
the preferences.

5.1.4.  SIMPLEST REMEDY.

A simple ranked ballot using the pure Schulze method would be easiest to
understand and convey all preferences but it would need to be explained in
advance how the rankings could work to reject an option which had no
absolute majority but had the largest number of first rank preferences.

5.2.  OTHER PROBLEMS.

There are other problems but I really did not even have time to present
this one today.

6.  ANALYTICAL DISCUSSION.

>
> 3. how do we gather opinions on the survey design nicely?

This should be a thoroughly dispassionate analytical discussion. 
Obviously something went wrong in a recent discussion which had started
reasonably but over an issue where we have no direct control.  Having no
direct control raises the sense of frustration.  We will all be on guard
to avoid harshness in our discourse.

Criticism should never be taken personally.  We have a great community
which should not be afraid of discussion.  Contention and disagreement
conducted in a civil tone helps solve problems.  Lack of discussion merely
ignores problems without attempting to solve them.

[...]

Thomas Dukleth
Agogme
109 E 9th Street, 3D
New York, NY  10003
USA
http://www.agogme.com
+1 212-674-3783