2020 presidential political polls: a preliminary postmortem
2020 PRESIDENTIAL POLLS:
A PRELIMINARY POSTMORTEM
by CRAIG HELMSTETTER | Nov. 6, 2020
Another close election, another round of polling criticism. Perhaps none (yet) more damning than The Atlantic’s David Graham, whose November 4th article opens: “Even with the results of the presidential contest still out, there’s a clear loser in this election: polling.”
So, how far off were the polls this year? The picture is not yet clear since (as of this writing) we are still awaiting final vote tallies in many states. However, they were off by more than any of my fellow American Association for Public Opinion Research members would like to see. (Even though many of us would quibble with the idea of using polling data to predict vote outcomes instead of simply providing a snapshot of opinions at a point in time.)
Looking at the 15 battleground states the APM Research Lab had identified going into the election—presumably some of the most difficult to accurately poll, and the subject of most current angst—we see that:
Most polling accurately reflected the race outcome, even if the margin between the two leading candidates was often over-estimated by the polls.
Polling in some states was better than in others.
The polls generally did a better job estimating support for Biden than support for Trump.
Let’s take a look.
For convenience, I have relied on two polling aggregators: Real Clear Politics and FiveThirtyEight. Both outlets track the hundreds of national and state level polls that came out this election season. FiveThirtyEight’s polling average is weighted to reflect their quality grading.
In addition, I have posted the results of the most recent polls conducted by a pollster graded as A-, A, or A+ by FiveThirtyEight (not all polls are conducted with the same rigor and integrity). [1]
The table speaks to points 1 and 2 above: The Biden–Trump vote margin from both of the aggregators and the A-rated pollsters is larger than my chosen cut-off point of 3 percentage points in nearly half of the comparisons in the table. On the national level, for example, Real Clear Politics’ polling average had Biden up by just over 7 percentage points, while he currently appears to have won the popular vote by a much slimmer 2 percentage point margin.
Listen to Craig’s discussion of polling
on Minnesota Public Radio News
The public is more forgiving of polling errors if they are in the direction of the outcome, so I have also noted where the winner suggested by the polls differs from those produced at the voting booth (except in Georgia, Nevada, North Carolina, and Pennsylvania, where the winner had not been declared at the time of this analysis).
As you can see by the “b” notations in the table, the big problem shows up in electoral heavyweight Florida (29 votes).
In addition, Emerson College’s poll of Ohio voters just days before the election suggested Biden would narrowly win. Both of the polling averages out of Ohio favored Trump (although he eventually won by a much bigger margin than was suggested by the polling averages).
So, the polling was particularly bad in places like Florida, Ohio and Wisconsin. Or, more positively, polling appears to have been more reflective of the eventual margins in Arizona, Colorado, Georgia, and Minnesota.
I am, however, surprised that the national polling, which performed much better than state-level polling in 2016, did not produce a Biden–Trump margin closer to the eventual outcome.
How did this year’s polls do in estimating preference for Biden and Trump?
It appears that many polls were off not because they overestimated public support for Biden, but because they underestimated support for Trump. I came to this conclusion by comparing the results of recent A-rated polls to the actual vote in this year’s battleground states.
For example: Sienna College’s survey of Florida voters, conducted just before election day, found that 47% of them preferred Biden. The actual vote result was 48%. Does that mean the poll was wrong? No.
All poll results are best thought of as ranges within which the public is very likely to fall if we were to ask the same question to every person instead of the relatively small sample captured in any given survey. To continue with our example from Florida, given the survey’s margin of error, the actual percentage of voters who preferred Biden at the time of the survey should have been somewhere between 43.8% and 50.2%. Florida’s 48% actual vote for Biden falls squarely within the poll’s range.
That was not the case for Colorado, Iowa, or Minnesota, where separate polls each under-estimated support for Biden, if only slightly. Looking at support for Trump, we see underestimates in five states as well as a recent national poll. While pre-election surveys should underestimate support to a small degree due to late decisions by some voters, the polls I examined in Florida and Wisconsin underestimated support for Trump by a notably large margin.
Some basic takeaways from this preliminary postmortem
It may be that the “shy Trump voter” who supposedly gave pollsters trouble in 2016 were not emboldened to be more candid with survey interviewers by the President’s four years in office. Or, more likely, perhaps more the extensive post-election evaluations to come will find some methodological shortcomings, such as the widespread under-accounting for voters without a college degree that led many state-level polls astray in 2016.
Despite the polling misses in the analysis above, it is important to remember that this year’s election was held in the midst of the Coronavirus pandemic. One of the trickiest parts of political polling is estimating the number of voters who will turn out. That estimate serves as the denominator for all of the other results in a political poll. Pollsters had no pre-existing “pandemic adjustment model” to rely on to help them predict who would turn out in these truly unique circumstances.
By the way, surveys designed to get at American’s policy preferences outside of voting do not have this same built-in difficulty: typical public opinion surveys do not require pollsters to estimate who will vote, or any other elusive denominator. Thus, while I understand David Graham’s concerns, I believe that public opinion polling, when done right, can still be a valuable way “…to understand what other Americans actually believe.”
-Craig (On Twitter: @c_helmstetter)
p.s. While not polling per se, the election forecasting done by likes of FiveThirtyEight, Cook Political Report, and The Economist relies heavily on polling. While their predictions, when off, cast a long shadow over the polling industry, it is important to note that (1) their predictions are typically probabilistic (even when Trump has just a 1 in 10 chance of winning, that is still a chance), and (2) of the 71 Minnesota-specific predictions we tracked across 10 races this year, only 2 were off; in 2018 we tracked 75 Minnesota-specific forecasts across 11 races with no clear misses.
[1] Most recent polls conducted by pollsters rated A-, A, or A+ by FiveThirtyEight:
National: SurveyUSA, conducted October 29-31; credibility interval= 3.2%.
Arizona: Marist College/NBC News, conducted October 29-NOV 1; margin of error= 3.9%.
Colorado: SurveyUSA, conducted October 1-6; credibility interval= 3.9%.
Florida: Siena College/The New York Times Upshot, conducted October 27-31; margin of error= 3.2%.
Georgia: Emerson College, conducted October 29-31; credibility interval= 3.5%.
Iowa: Selzer & Co/Des Moines Register, conducted October 26-29; margin of error= 3.4%.
Maine: Emerson College, conducted October 29-31; credibility interval= 3.9%.
Michigan: Emerson College, conducted October 29-31; credibility interval= 3.4%.
Minnesota: SurveyUSA, conducted October 23-27; credibility interval= 4.6%.
Nevada: Emerson College, conducted October 29-31; credibility interval= 3.6%.
New Hampshire: Suffolk University, conducted October 8-12; margin of error= 4.4%.
North Carolina: Emerson College, conducted October 29-31; credibility interval= 3.3%.
Ohio: Emerson College, conducted October 29-31; credibility interval= 3.8%.
Pennsylvania: Marist College, October 29-November 1; margin of error= 3.8%.
Virginia: The Washington Post/George Mason University, conducted October 13-19; margin of error= 4.0%.
Wisconsin: Siena College/The New York Times Upshot, conducted October 26-30; margin of error= 3.0%.