Op-ed: Rethinking the Polls. Counting chickens that will never hatch and the unlikely voter

This is the third piece in a series of op-eds about polling in the election. Read the first here and the previous piece here. John E. Newhagen is an associate professor emeritus at the University of Maryland’s Philip Merrill College of Journalism. 

The key to predicting who is going to win an election is understanding who will vote – and that spells trouble for pollsters. The best predictor of voting intention is to ask the respondents if they intend to vote. The problem, however, is that real turnout has been around 55 percent for the last 55 years, while, historically, over 70 percent of survey respondents routinely say they intended to vote.

Yikes! So are 20 percent of respondents selected in random probability samples lying?  Yes.

presidential-turnout

Social science has a more polite term for the phenomenon—they call it “social desirability biasing.” The idea is that people know what social norms are—in this case the idea is that in a democracy it is a citizen’s responsibility to vote—and some of them will lie to save face. Pressure comes in the form of not-so-subtle prompts on Election Day where voters sport “I Voted” stickers on their lapels and at school, where teachers laud importance of participation from the earliest grades. Show me a civics teacher who says voting isn’t worth the effort and I will show you a soon-to-be-out-of-work educator.

i-voted

Pollsters are now faced with the task of sorting out truth tellers from the liars in order to make valid predictions. Voting trends from years past is a very good predictor because it is a habitual behavior and there are ways to check to see if someone voted in the last election. But pollsters, who are short on time and money, rarely check. Thus, most horserace survey results we see reported in mainstream media and on political sites, such as realclearpolitics.com (RCP), are based on double digit overestimates of ephemeral and elusive estimates of who really is a “likely voter.”

The polling industry tries to deal with the problem, but there are about as many different ways to define a likely voter as there are pollsters. The process can be grouped into two broad categories: one is to use a set of filter questions and the other is to weigh results. Both approaches have problems.

The Pew Charitable Trust (Pew) has done some ambitious work examining the bank of question many pollsters use. That seven-question bank was invented by Gallup researchers about 50 years ago and still widely used today.

The most direct way of predicting voter turnout is to simply ask:

  • How likely are you to vote in the general election this November? Definitely will vote, probably will vote, etc.?

Pew acknowledges that many major pollsters use this question, and this question alone, to determine whom they will include in their sample as a “likely voter” in their survey. Further, PEW reports that many surveys included respondents who say that they “definitely will vote,” and those who say they “probably will vote,” because the two categories add up to about 90 percent of the responses in most samples.

A Pew project, studying the topic, found that 70 percent of the respondents contacted said they “definitely will vote.” But when they checked a database—called a “national voter file,” that records who really did vote— they found that only 53.9 percent actually did vote, leaving 16 percent who lied. Those numbers are interesting for two reasons. First, the actual turnout in the 2012 election was 55 percent, which is very close to Pew’s verified number. Second, a rate of 16 percent liars is in line with studies in social desirability biasing carried out during the last half century.

But the verification rate plummets to 7.2 percent among the other 20 percent who say they “probably will vote.”

Adding the two groups, the overall rate of verified voters sinks 41.3 percent and the number of liars increases to a whopping 48.7 percent. Yikes. What if the liars are different than the truth tellers in some way that could throw the results? Do real voters feel more strongly about certain key issues than nonvoters? Do the two groups embrace different issues?

Here are the other questions in the old Gallup battery:

  • Please rate your chance of voting in November. (This is just another way of asking the voting likelihood question but is usually scored on a 10-point scale.)
  • How much thought have you given to the coming November election? (This question is not even about a behavior and presents ample opportunity to fudge.)
  • Have you ever voted in your precinct or election district? (This can be used in a verification study, but that rarely happens)
  • Would you say you follow what’s going on in government and public affairs most of the time, some of the time, only now and then, hardly at all? (Of course following the news is more socially desirable than ignoring it and is also subject to over reporting.)
  • How often would you say you vote? (Same social desirability problem as all of the above.)
  • In the 2012 presidential election. Did you vote for Barack Obama or Mitt Romney, or did things come up that kept you from voting, or did you happen to vote? (It turns out this question actually has problems because people over report voting for the winner and a surprising number actually don’t remember who they voted for).

Notice that the first five questions all have socially desirable outcomes. The final question is an experiment and the phrase in bold is worded in such a way to give the respondent a graceful way out. Analysis, however, shows it didn’t work very well. So it is not clear that an index or weighted score of the entire bank does much better than just asking about voting intention in the first place and moving along.

My favorite is:

  • Name your polling place.

This question was asked by The Washington Post a number of years ago and I do not know if they still use it. It seems to me that this question is a bit hard to fake.

Weighting and Filtering Data, a Tricky Process at Best

Most pollsters begin with a list of registered voters’ telephone numbers, which is a public record and fairly easy to obtain. People on that list are obviously more likely to vote, but not in numbers large enough to make a meaningful prediction about the upcoming election. So the filtering process begins in a quest for the Holy Grail: real voters. The individual questions or an index of weighted score for the entire battery all correlate fairly highly, at least in terms of social science research, to actual verified voting. But correlations of even 60 percent or 70 percent, which are considered to be very high by social science standards, come nowhere close to the precision needed to predict an election outcome correctly.

Another approach is to include other kinds of variables in likely voter models, including demographic characteristics, partisanship, and ideology. For instance, party affiliation historically shows Republicans vote at higher rates than Democrats; thus, pollsters might weigh Republicans’ scores higher than for Democrats. This has the same problem as the voting battery, the variables do correlate to voting, but not nearly as strongly as a good horserace poll requires.

All this leads to the conclusion that pollsters are counting chickens that will never hatch.

Now, except for the critical issue of turnout, this entire exercise might not even be worth thinking about if – and this is a huge if – the liars are not different in their beliefs and political dispositions than true voters. But it does not take much imagination to think of ways they might be different on key issues such as abortion or immigration.

Is there reason to worry? You bet.

The New York Times recently gave the very same dataset taken from a recent poll to four of the most respected pollsters for analysis. What they got back were four different outcomes that varied as much as 4 percent, which is usually equal to or greater than actual margin of victory in real elections. Three predicted a Clinton victory, but one predicted a Trump victory. Those differences can only be accounted for by their different analytical strategies, among which defining a likely voter is central.

clinton-trump-margin

Another source of concern can be seen by simply eyeballing a few weeks’ results on the RCP polling results page, which lists virtually all the major polls and is a lot of fun if you are a polling junkie. Two important trends emerge. The first is that there is consistently at least a 10 percent difference between the polls, with some predicating a Clinton win and some predicting a Trump victory. The second thing to note is that the individual polls seem to have personalities, that is, they are consistently different from other polls by the same amount over time. If a poll is 4 percent above the average for the group, it tends to stay that way week after week. If all the polls are based on random samples those difference are again an artifact of analytical strategy which can inevitably be traced to the likely voter decision process.

What an informed journalist should know and what the pollsters publish

It is critical for journalists trying to evaluate this mess to understand what calculus is being used to determine the likely voter and the problem is that very few actually report what they are doing. They generally withhold this information from the methodology sections of their reports despite the fact the most respected professional organization, the American Association for Public Opinion Research (APPOR), says they should. One notable exception is the USC-LA Times poll, which has consistently been predicting about a 5 percent Trump victory (that number is down lately). The folks doing that poll are very forthright in explaining how they are weighting their results and why it generates a Trump win when others do not – one reason is that their system favors Republicans because they are more predictable and vote at higher rates.

Further, taking an average across all the polls is also scary. One analyst for The Washington Post, who reports the RCP average, emailed me that confidence intervals tend to decrease in such a meta-analysis. He would like to overlook the fact that individual polls showing small differences between candidates are meaningless if the confidence interval is large, as discussed in an earlier article. But if the aggregated results are using different models to define likely voters the math behind his assumption to ignore confidence intervals goes in the dumpster. And remember that the range among the aggregated polls tends to be 10 percent or more.

Speaking of confidence intervals, the boiler plate accompanying most polls claiming to focus on likely voters begins with a sample of about 1,000 registered voters with a CI of about 2-3 percent. However, most polls report that about 40 percent, or 350 to 400 respondents, are eliminated in the culling process described above, which inflates the CI to about plus or minus 5 percent, thus further diluting the polls sensitivity and ability to predict the outcome of the election.

Academic Hubris or a Real Problem?

So, does this discussion point to a problem journalists should think about when reporting polls or is it simply academic hubris?

Well, Mitt Romney went to bed on Election Night thinking he was going to win based on the polls he saw. RCP admits their average was off in 2012. During the closing month of the campaign, they showed a slight Romney lead. The RAND poll, by contrast, showed a 3.8 point Obama lead – which was almost exactly correct.

 

rcp-2012

The folks who supported Great Britain staying in the European Union thought they would squeak out a victory based on a bump in the polls in their favor the last few days before the referendum.

But both Romney and BREXIT supporters woke up to learn they had lost. In the end the polls predicting a Romney win underestimated Barrack Obama’s ability to turn out young voters, and the polls predicting a pro-European Union outcome overestimated turnout among young supporters. The polls’ likely voter calculations were far enough off to skew their predictions.

A more recent example came in the dramatic Oct. 2 election in Colombia to ratify a peace deal with leftist rebels to end a bloody 50-year-old war. The referendum lost by less than 1 percent, a stunning defeat for President Juan Manuel Santos, who had predicted the referendum would win by a 3-to-1 margin. But only 37 percent of the electorate actually voted. It turned out that hard-core members of a right wing party, made up of only 23.4 percent of the electorate, turned out in droves and defeated the deal, while many Colombians with sympathies toward the peace accord stayed home on Election Day. (Santos can take solace in the fact he was awarded the Nobel Peace Prize for his efforts.)

This should be a cautionary tale for both Hillary Clinton and Donald Trump, and turns our attention to the issue turnout.

Turnout, the big unknown

The conventional wisdom is that a lot of the 40 to 45 percent of people who say they support Donald Trump are made up of older white males with no more than a high school education. It appears that this group is rock solid and will turn out in very high numbers. It does not seem to matter what Trump does or says—they are firm.

On the other hand, Hillary Clinton seems to have a core slightly larger than Trump’s, say something less than 50 percent. Those folks tend to be young, African American, female and white college graduates.

Now, the thing that has the Clinton campaign terrified is that key segments in her coalition are not going to vote. This includes so-called millennials, Hispanics and even some women. There have even been suggestions that African Americans, the group that was singly most responsible for her nomination, may not turn out at hoped-for rates.

Meanwhile, the solidity of Trump support came into question when the vulgar tape demeaning women surfaced. As of this writing, it turns out that a lot of Republican support, including evangelical Christians, may not be as solid as was assumed. Politicians and other opinion leaders who had pledged to support Trump were abandoning him like fleas jumping off a dead dog after the tape was aired. Some called for his resignation. And the problematic part for the pollsters is that they continued to report that they found Hillary Clinton to be repugnant, which means they have no real option but to sit out the election and thereby pulling them out of a group of habitual voters that would have normally been assumed to be Republican “likely voters.”

Add to this the uncertainty regarding the fact that about 15 percent of survey respondents say they intend to vote for third party candidates, which is an unusually high number which causes the uncertainty to grow even larger.

Figuring out who will vote is like “landing a plane in a hurricane”

The Washington Post reports that the chairman of the National Republican Congressional Committee, Greg Walden, told House members on a private conference call with Paul Ryan that navigating this election is now like “landing an airplane in a hurricane.”

In the wake of the offensive tapes, Walden said the ground is shifting quickly and urged fellow House members to keep conducting internal polls so that they have an accurate read of how vulnerable they might now be as the bottom falls out from under Donald Trump’s campaign, the Post reported.

It would seem that at this point even the politicians are grasping at straws in this turbulence. The important point I am trying to make is that the uncertainty is coming not so much from where folks’ loyalties are as it is from figuring out who will actually show up at the polls. It is also interesting that Walden has more faith in the proprietary polls, which neither you nor I see as their only hope. Let me say it just one more time:

Yikes!

Leave a Reply

Your email address will not be published. Required fields are marked *