This is the sixth piece in a series of op-eds about polling in the 2016 presidential election. Read the previous piece here. John E. Newhagen is an associate professor emeritus at the University of Maryland’s Philip Merrill College of Journalism.
President Trump has cited post-election polls showing low approval numbers as “fake” in his notorious Twitter posts and elsewhere. On the other hand, he claimed his approval ratings are, in fact, really positive when citing a Rasmussen Reports survey in his marathon February 16 news conference.
This presents what may be a defining moment for contemporary journalism. Should journalists report facts they know to be false? Worse yet, should they report lies? That is, false facts used to intentionally mislead the public. Journalists are taught to tell both sides of the story, but what if one side is intentionally or recklessly presenting false information?
A close examination of Trump’s relationship with public opinion polls provides and interesting opportunity to apply this question. Numbers are not contentious; they may be right or wrong, but they are what they are. So, let’s look at post-electoral approval ratings and evaluate the claims Trump makes about them:
Trump has said repeatedly that mainstream polls showing him with negative approval rating are “fake.” However, on February 17, he referred to a Rasmussen poll which showed him with a 55 percent approval rating, and a 45 percent disapprove score, or a net of +10, according to RealClearPolitics.com. He praised that poll, contrasting it to the “fake” polls. But, on the same day, Gallup reported Trump with only 38 percent approval and 56 percent disapproval, or a net of -18 percent.
That’s a whopping 28 percent gap between polls. How are journalists covering polls, not to mention the media’s consumers, supposed make sense of those numbers? Trump’s claims about public support and his rants about the press as “the enemy of the American people” make this an intimidating task. The table below shows a RCP list of 11 prominent polls on the date in question.
Here are some common sense guidelines:
- Look for trends: Among the 11 polls RealClearPolitics reported between January 31 and February 17, eight showed Trump with net negative approval ratings, with an average 51.5 percent registering disapproval. The Fox News and Emerson College polls both showed Trump with a net +1 percent approval, but were inside confidence intervals and therefore are not statistically significant. The other eight polls, all of which were outside confidence intervals, showed an average net -9 percent approval. The Rasmussen poll cited by Trump is what statisticians call an outlier because it deviates from other polls by such a large margin. It is not impossible that an outlier is correct, but it is highly unlikely. However, most statisticians will discard outlying data in practice because it distorts broader, underlying trends.
When Trump focuses on one poll’s results as favorable, even though other polls do not support the conclusion he values, he is cherry picking. Journalists are warned against that practice, and generally taught that looking for overarching trends is good reporting technique. There is no reason polls should be any different.
Here is the strategy I would suggest to take a closer look at outliers such as the Rasmussen poll:
- Figure out what the outlier did differently: Are there significant differences in the way the outlier sampled or analyzed data? In the case of Rasmussen, the answer is clearly “yes.” Boring down into the Rasmussen website to their methodology page, a task that took a half hour, reveals two troubling factors:
Rasmussen uses a technique called “robo-sampling,” where a computer dials the phone and even asks the questions, to collect data. Respondents press numbers on their keypads to respond. Robo-polls are fast and cheap, which accounts for Rasmussen’s claim that it collects data for a new poll every night. Rasmussen tries to put a good face on the practice, and states their “automated polling systems use a single, digitally-recorded, voice to conduct the interview while traditional firms rely on phone banks, boiler rooms, and operator-assisted technology.”
But the industry generally disagrees; the fact is the highest quality data comes from live, trained interviewers. One big problem with robo-polling is self-selection; robocalls are annoying and easy to hang up on, and only the highly motivated will take the time to respond. Rasmussen does not report a completion rate, which can be below 5 percent. Ask yourself the question: would Trump’s highly motivated supporters be more likely to take part?
Second, Rasmussen is still filtering its data through a set of “likely voter” screening questions. This is what got pollsters in trouble before the election, and it is difficult to understand why Rasmussen is still doing it. Rasmussen further manipulates data by “partisan weighting… through a dynamic weighting system that takes into account the state’s voting history, national trends, and recent polling in a particular state or geographic area.” Fooling around with data like this is scary because it wipes out any potential claim that the sample is a valid estimate of the general population. FiveThirtyEight.com, a highly-respected source for detailed critical analysis of public opinion polls, says Rasmussen consistently reported numbers with at least a 4 percent bias toward Republicans.
- Does the pollster have an ax to grind? This is a common sense question that journalists think about when evaluating any source. Here, the answer is “yes,” based on studies summarized by Wikipedia. The poll’s founder and namesake has a reputation for conservative bias, and has consulted for Republican candidates in the past. Two studies ranked the poll 20th out of 23 polls, or lower, in 2012 post-election quality evaluations. One the other hand, there is no clear pattern of bias in the eight other polls RCP reports. On the contrary, Gallup, which shows the largest negative approval rating of any polling group, has been attacked by the liberal advocacy group MoveOn.org for holding a conservative bias in past elections.
Rasmussen is a dramatic outlier among its peers; its sampling technique is not statistically valid, its administration places speed and cost above accuracy, its analytical techniques are questionable and the organization has a reputation for favoring conservative Republicans.
Does this mean Rasmussen is a “fake” poll? Yes, there are shades of grey between the black and white; but at some point, the evidence becomes overwhelming. The Rasmussen poll, at best, lacks face validity. Trump’s claim that the Rasmussen poll, and that poll alone, is a valid indicator of public opinion is reckless and misleading.
Does Trump have evidence the other polls are “fake?” If he does, he is not sharing it. The RCP review of all polls taken just before the election shows mainstream polls generally gave a 2 percent to 4 percent edge to Clinton, but those margins fell well within sampling confidence intervals. And, as pointed out here before the election, in those cases the poll should be described as “not showing a significant difference between candidates.” Journalists that tried to hedge that statistical reality and predicted a win for Clinton made the mistake, not the polls. Further, Clinton did win the national popular vote. If the public opinion industry let down journalism, it was in their omission of statewide polls that would have betrayed Trump’s impending Electoral College victory. But the polls, neither pre-, nor post-election, are “fake.”
Bottom line: Trump’s claim that the vast majority of polls are fake, and that he really has a substantial post-election positive approval rating is a lie. Our next step requires a profound reexamination of a journalistic canon: should Trump’s claim be published? Two Supreme Court decisions I read as a journalism student many years ago come to mind.
Falsely shouting fire in a crowded theater: The first case is Schenck v. United States. In 1919, Justice Oliver Wendell Holmes, Jr. used the phrase to explain why he thought a defendant’s speech in opposition to the draft during World War I was not protected free speech under the First Amendment of the United States Constitution. The paraphrasing does not generally include (but does usually imply) the word falsely, i.e., “falsely shouting fire in a crowded theater,” which was the original wording used in Holmes’ opinion. This highlights the idea that speech deemed dangerous and false is not protected, as opposed to speech that is dangerous but also true. It is critical to the context of Trump’s contemporary claims. “Fake,” or “false” claims imply the speaker to be knowingly misleading the audience with risk of harm to them.
Reckless disregard and malicious intent: Every serious journalist should know what New York Times v. Sullivan (1964) had to say about libel. The Reporters Committee for Freedom of the Press summarizes the two key elements necessary to show libel: reckless disregard for the truth and malicious intent. In today’s context is it important to consider that recklessness and maliciousness both imply intent. Sources publishing “fake news” know it is false and has the potential to do harm.
While not obvious at first glance, it seems to me these time-tested concepts, both born in the United States Supreme Court, offer clear guidance to politicians and journalist about the limits of speech in the public sphere. The model I have in my mind for the president of the United States, journalists and others with access to media, is that they work to control crisis. Some may do a better job than others, but we all hope that their intent is to bring the truth to readers and viewers. However, the current information ecology seems to be one where some right-wing web pages, radio shows, and the president are the crisis – they are knowingly propagating false information as if it were real news. And this includes reporting about public opinion polls. Journalists are now challenged to understand that our new administration is not “business as usual,” and deal with it accordingly.
Correction: This post has been updated to correct a previous claim that Rasmussen’s final pre-election poll showed Trump winning the popular vote by 2 percentage points. We regret this error.