Instant Runoff Voting Has Not Increased Turnout in San Francisco

Proponents of Instant Runoff Voting commonly claim that IRV has the benefit of increasing voter turnout. Their logic is that with ordinary delayed (not “instant”) runoffs, fewer voters tend to show up to cast votes in the decisive two-candidate runoff election.

However, analysis of the entire history of IRV elections in San Francisco reveals no increase in turnout. Actually it showed a decrease in turnout, albeit a statistically insignificant one. One possible hyptothesis advanced in the media about why that is:

Quote from Almost Nobody You Know Will Vote For the Next Mayor of San Francisco by Chris Roberts, SF Weekly, 5 Oct 2011:Voter turnout was high in 1999, when the mayoral race broke down into easy, left vs. right lines. With ranked choice voting, a dozen candidates are asking for second and third place votes, and – as [USF professor Corey] Cook says – “there are a dozen shades of gray.” Too much ambiguity reduces turnout, as does local media’s handling of the top candidates. “The more that papers print that Ed Lee is a shoo-in,” said a consultant working for one of Lee’s chief rivals, “the less people are going to vote.”

Warren D. Smith, a Princeton math Ph.D. who has been studying elections for over a decade, explains as follows. [This analysis was excised and compiled by W.D. Smith and Clay Shentrup from a thread at the CES discussion forum on Google Groups with several contributors. Later David Cary pointed out a problem which caused us to do some rechecking, correcting, and updating with new datapoint from year 2011.]

About Election Turnout in San Francisco pre- and post-IRV

San Francisco’s first Instant Runoff (IRV) election occurred in November 2004, see http://www.usatoday.com/tech/news/techpolicy/evoting/2004-11-05-ranked-evote-snafu_x.htm and other news sources. The turnout percentages of election in SF are tabulated on the sfgov web sitehttp://www.sfgov2.org/index.aspx?page=1670. Here are the numbers. Warning: The calculations below were using 40% for the 2011 turnout based on press accounts, but the correct value was later claimed to be 42.47% on the sfgov site.

I. SUMMARY:

quantity IRV era (Nov.2004 thru 2011) pre-IRV old era (Nov.1960 thru March 2004)
#ELECTIONS 14 70
HARMONIC_MEAN_TURNOUT% 40.43 45.81
GEOMETRIC_MEAN_TURNOUT 43.91 48.53
ARITHMETIC_MEAN_TURNOUT 47.06 50.91
STD_DEV(TURNOUT) 18.66 14.81
TURNOUT±1σ 47.06±4.98 50.91±1.77
DIFFERENCE 3.85±5.29
Conclusion based on this: although the IRV era had lower turnouts, the difference’s sign is not statistically significant. (There certainly is not evidence IRV increases turnout, at least not using this crude analysis.)

II. THE RAW DATA PLUS SEVERAL ANALYSES:

  1. TURNOUTS IN THE IRV ERA (Nov.2004 thru 2011, data in reverse chronological order):[42.47, 61.02, 34.72, 22.58, 28.07, 81.25, 40.22,24.88, 64.75, 35.62, 60.66, 37.11, 53.61, 74.31]
  2. TURNOUTS IN THE OLD (PRE-IRV) ERA (Nov.1960 thru March 2004, reverse chronological order):[42.51, 54.46, 45.67, 59.16, 38.38, 50.08, 34.15, 16.58, 29.62, 32.55, 66.59, 44.90, 48.84, 44.95, 55.85, 46.23, 30.41, 43.52, 61.97, 40.47, 46.56, 51.86, 55.40, 32.05, 37.21, 20.11, 69.01, 42.18, 50.73, 47.61, 57.78, 41.58, 44.84, 67.07, 48.33, 40.36, 51.18, 25.60, 39.71, 61.39, 39.65, 27.02, 70.39, 48.10, 45.99, 47.24, 63.38, 51.20, 25.22, 65.71, 34.79, 51.16, 51.91, 55.06, 57.25, 58.12, 51.75, 53.08, 73.11, 67.11, 66.42, 72.70, 61.95, 56.66, 60.00, 74.31, 63.75, 82.02, 85.03, 86.08]

More sophisticated analyses: The above has been the simplest possible analysis. However, reasons then came to our attention for considering various more complicated corrections to it.

Jan Kok pointed out that many of the mid-year elections before 1992 had very high turnout (we do not know why). November elections are usual, mid-year elections unusual.

Here is the data with mid-year elections removed. Again, the IRV era has lower turnout than the pre-IRV era, but this is now even less statistically significant.

quantity IRV era (2004-2011) pre-IRV era (1960-2004) nonRunoffs pre-IRV Runoffs
#ELECTIONS 8 35 10
HARMONIC_MEAN_TURNOUT% 45.81 51.64 39.03
GEOMETRIC_MEAN_TURNOUT 49.92 43.62 42.25
ARITHMETIC_MEAN_TURNOUT 53.61 56.73 44.68
STD_DEV(TURNOUT) 18.60 15.40 12.93
TURNOUT±1σ 53.61±6.58 56.73±2.60
DIFFERENCE 3.12±7.07

Raw data:

Mid-year {elections and runoff elections} removed:

IRVera := [42.47, 61.02, 22.58, 81.25, 35.62, 60.66, 53.61, 74.31];

OldEraNonRunoff := [45.67, 50.08, 29.62, 66.59, 44.95, 55.85, 30.41, 61.97, 51.86, 55.4, 37.21, 69.01, 47.61, 57.78, 44.84, 67.07, 51.18, 61.39, 27.02, 70.39, 45.99, 63.38, 25.22, 65.71, 55.06, 57.25, 51.75, 73.11, 72.7, 61.95, 60, 74.31, 82.02, 85.03, 86.08];

OldEraRunoffs := [54.46, 38.38, 16.58, 32.55, 48.84, 46.56, 50.73, 40.36, 51.91, 66.42];

Also note that in the above dataset the runoffs tend to have lower average turnout than both nonrunoff and IRV elections. But Jan Kok then observed that odd years were quite different from even years. The runoffs in the odd years were in some cases higher than the main election! (Turnout was 2.8% lower on average.) And the average of all the odd-year runoff turnouts is greater than the average of the odd-year IRV turnouts (46.98 vs 37.27%).

Here’s the odd-year-only dataset:

Year Main Runoff
2009 22.58
2007 35.62
2005 53.61
2003 45.67 54.46
2001 29.62 16.58
1999 44.95 48.84
1997 30.41
1995 51.86 46.56
1993 37.21
1991 47.61 50.73
1989 44.84
1987 51.18 40.36
1985 27.02
1983 45.99
1981 25.22
1979 55.06 51.91
1977 51.75
1975 72.70 66.42
1973 60

Warren Smith then noticed that the pre-IRV turnouts seem to show a declining trend with time. Just listing the 70 pre-IRV turnouts in reverse chronological order:y := [42.51, 54.46, 45.67, 59.16, 38.38, 50.08, 34.15, 16.58, 29.62, 32.55, 66.59, 44.90, 48.84, 44.95, 55.85, 46.23, 30.41, 43.52, 61.97, 40.47, 46.56, 51.86, 55.40, 32.05, 37.21, 20.11, 69.01, 42.18, 50.73, 47.61, 57.78, 41.58, 44.84, 67.07, 48.33, 40.36, 51.18, 25.60, 39.71, 61.39, 39.65, 27.02, 70.39, 48.10, 45.99, 47.24, 63.38, 51.20, 25.22, 65.71, 34.79, 51.16, 51.91, 55.06, 57.25, 58.12, 51.75, 53.08, 73.11, 67.11, 66.42, 72.70, 61.95, 56.66, 60.00, 74.31, 63.75, 82.02, 85.03, 86.08]

and least-square fitting y=Ax+B where x=[1, 2, 3,..., 70] we find y=37.25+0.385x approximately which means turnout% was declining 0.385 per election, i.e. roughly 0.60 per year, over the 45-year-long pre-IRV era, for a total turnout% decrease of 26.6%.

Going back to our original comparison

quantity IRV era (Nov.2004 thru 2011) Pre-IRV old era (Nov.1960 thru March 2004)
TURNOUT±1σ 47.06±4.98 50.91±1.77
DIFFERENCE 3.85±5.29
we could argue that to compensate for the time-declining trend (so that we can compare apples with apples, i.e. same-time to same-time), we should subtract half of the decline, i.e. 26.56/2=13.28 from the pre-IRV average number. (And call that 13.28±1 to crudely estimate the error in the estimate of the decline.) This would yield
quantity IRV era (Nov.2004 thru 2011) Pre-IRV era (corrected for time-decline)
TURNOUT±1σ 47.06±4.98 37.63±2.03
DIFFERENCE 9.43±5.38 (IRV turnout greater)

This would now turn the tables and support the view IRV increases turnout – except that again this is too small an effect to be statistically significant. One could go further: one could argue that 0.60-per-year decline rate also ought to be reckoned as continuing during the IRV-years also, which would upon adding 3.5 years of further decline, further boost IRV’s (decline-compensated) turnout advantage from 9.43±5.38 to 11.53±5.38 which would then be starting to become statistically significant (98% confidence that the decline-corrected IRV turnout exceeds the decline-corrected non-IRV turnout). Of course, to buy that you’d need to believe in this continuing decline and that it is occurring for reasons wholy unrelated to IRV versus non-IRV so that it is valid to compensate for it when assessing the effect of IRV/nonIRV. Obviously it is mathematically impossible for the decline to continue forever (in either time direction) and also the decline apparently actually accelerated (more than doubled?!) after the switch to IRV… so there is some reason to question both those assumptions… so perhaps therefore we should only use half the decline-compensation, not the whole ball of wax. If so, the statistical significance vanishes.

So ok, I think we now have done enough analysis to reach a conclusion.

Conclusion

The evidence from San Francisco up to 2011 does not yield a statistically significant conclusion at the 99% level that IRV increases turnout, and also does not yield a statistically significant conclusion that IRV decreases turnout. The effect, if any, is small and can be regarded using different analysis techniques as having moved in either the positive or negative direction.

We thank David Cary for pointing out an arithmetic error in our decline-trend calculation (now corrected).
email