Why Were the Presidential Polls Wrong?
Not since 1948, with the non-election of Thomas Dewey, have the polls been so wrong. Just as in 1948, however, the problem does not rest in the field of statistics, but rather in the field of research methodology.
RealClear Politics identified 16 battleground states and provided data for the latest polls in each of those states. They had combined those polls and posted an overall average for each state. We all understand that these polls include a margin of error that is mathematically linked to the sample size. All else being equal, larger samples have less error; smaller samples have more error. Although RealClear does not provide an overall margin of error for their average of polls, that margin can be easily calculated by taking into account the size of the various samples (and making one or two reasonable assumptions).
Based on these numbers, the RealClear difference between Trump and Clinton was indeed within the margin of error for seven of the 16 battleground states, including Arizona, Colorado, Florida, Georgia, Maine, New Hampshire, and Virginia. In one state, the polls underestimated Clinton’s performance. The RealClear summary hinted at a small Trump lead in Nevada, a state that Clinton would win. The other eight errors, however, were all cases in which Trump’s performance was underestimated, including Iowa, Michigan, Minnesota, Missouri, Ohio, Pennsylvania, North Carolina, and Wisconsin. Some of these states had been correctly predicted to go to Trump (like Missouri and Ohio), but the differences were larger than anticipated. Some of these states eventually went to Clinton (like Minnesota), but the margins were much smaller than expected. Ultimately, however, the unexpected differences in states like Pennsylvania andMichigan (which, for the record, still has not been called) moved the celebration from the Javits Center to Trump Tower.
Republican political consultant Mike Murphy suggested, “Tonight, data died.” He pointed to a very real failure of election forecasting, but his quote suggests a problem with statistics. In fact, however, the problem was in the failure to obtain representative random samples.
The error in 1948 was because many voters did not have telephones. A random sample restricted to homes with telephones might totally misrepresent the political tendencies of homes without telephones. The homes without phones generally had fewer economic resources and different voting priorities. The failure to use random representative polls of all potential voters resulted in the faulty prediction.
Although some will suggest that the polls were intentionally rigged, the fact that the error in Nevada was in the opposite direction suggests otherwise. Pollsters were trying to get an accurate read. In fact, the polling has been extremely accurate in recent elections. Based on the polling data in 2004, for example, I successfully predicted the winner in every single state.
As a consequence, it is progressively more difficult for pollsters to complete a representative random sample of voter behavior. Our polls are more likely to predict how people who do not use caller ID will behave. It is possible that people who seek more control over their telephones might tend to be the same sort of people who seek more control over their government and other areas of their private lives.
The 2016 poll numbers might have been rigged, but not through any intentional work by the media. Rather, they were unintentionally rigged by the realistic hurdles faced by behavioral scientists, as they seek to explain and understand human behavior.
Even if we are able to fix the research methodology, it is important to remember that statistics always include a margin of error. The data only provide insights into the world. Like the weather forecast, they are not intended to provide any guarantees. Stephen King’s comment about statistical analysis is right on target: in his novel, “11/22/63,” King writes: “There’s always a window of uncertainty.