If you’re like me and you regularly – and, okay, maybe a little obsessively – seek out public opinion polls for insight into what Americans think about the current political climate, then you might find yourself wondering from time to time why polls sometimes disagree with one another.
For instance, how many Americans so far approve of the job Donald Trump is doing in the White House? According to a Quinnipiac University poll released on January 25, 2017, Trump’s approval rating stands at a mere 36 percent. However, according to a poll from Rasmussen, which was released a mere four days later on January 29, the percentage of Americans who approve of Trump’s job performance is considerably higher at 53 percent. What gives?
Perhaps not surprisingly, a major reason for disagreement among polls has to do with the sample of people who take part in a given survey.
As discussed in a recent post by the Pew Research Center, pollsters conduct public opinion surveys using one of three different types of survey samples, with the choice about which type of sample to use dependent on the specific goals of the survey – whether to estimate true public opinion or to help win an upcoming election, for instance. If a polling firm wishes to gauge true public opinion, then they will likely survey a broad and representative sample of all adults in the population. However, not all adults head to the polls and vote in elections. So, if a pollster wishes to use the results of a survey to help gauge opinions only among voters, then they will probably narrow things down a bit and instead survey only those who are registered to vote. Moreover, pollsters may decide to narrow things down even further and use screening questions to filter out individuals who appear unlikely to vote. In this case, pollsters can choose to survey only those deemed to be likely to vote.
This matters because the decision about which type of survey sample to use doesn’t just introduce random noise and variation into poll results. It actually introduces systematic variation, which can yield quite differing results among otherwise similar polls.
Courtney Kennedy, director of survey research at Pew Research Center, explains:
In non-election years like this one, most pollsters survey all adults, but not all follow this convention. A number of pollsters continue to do surveys of registered or even likely voters. Why does this matter for Trump’s approval ratings? It’s about demographics. Voters as a group skew older and whiter than the general public. And older Americans, as well as white Americans, tilt more Republican than other groups. So, voter-only polls tend to get somewhat more favorable views of a Republican president or candidate and less favorable views of Democrats [emphasis added]. This pattern was evident during Barack Obama’s presidency, with his overall ratings tending to be somewhat higher among the general public than among registered or likely voters.
So, because voters tend to be older and whiter than the general population and older and white Americans tend to lean Republican, polls based only on registered and likely voters tend to yield more favorable numbers for Republicans.
Kennedy goes on to cite further evidence from recent polls asking Americans whether they so far approve of the job President Trump is doing in the White House. As she states, “Pew Research Center’s general population poll conducted Feb. 7-12 recorded Trump’s presidential approval rating at 39%. Among registered voters in that survey, his rating was 42%.”
Now what if we step back and take a wider view of things to consider a greater number and variety of polls? Can we spot a similar pattern in polls asking about whether respondents view Mr. Trump favorably or unfavorably? After all, polls have been asking Americans this question since at least as far back as May 2015, shortly before he announced his candidacy for President.
In short, the answer is yes – but it’s a bit messy.
Trump’s Net Favorability Ratings By Survey Sample Type
To determine whether Trump’s favorability ratings are generally higher among polls based on registered and likely voters compared to polls based on all adults, I compiled a set of 403 polls from HuffPost Pollster. Each poll asked a sample of Americans whether they view Donald Trump favorably or unfavorably. Among the entire set of polls, 64 surveyed likely voters, 163 surveyed registered voters, and 176 surveyed all adults. Ending dates for the polls ranged from between May 26, 2015 and February 14, 2017.
However, polls varied greatly in terms of recency. For polls based on likely voters, the median survey ending date was 166 days ago, compared to 233 days ago for polls based on registered voters and 212 days ago for polls based on all adults. I therefore tried to control for differences in recency using a method called nearest neighbor propensity score matching. This winnowed the data set down to 192 polls, with an equal number (64) based on likely voters, registered voters, or all adults. Moreover, propensity score matching largely erased differences in poll recency. Among the restricted set of 192 polls, the median survey ending date for polls based on likely voters was 166 days ago, which was equivalent to the median survey ending date for polls based on registered voters. The median survey ending date for polls based on all adults was just slightly higher at 170 days ago.
The graphic below shows President Trump’s net favorability ratings broken down by sample type – likely voters vs. registered voters vs. all adults – after controlling for differences in recency between sample types. Note that because net favorability ratings were calculated as the difference between the percentage of people who view Mr. Trump favorably and the percentage who view him unfavorably, positive numbers represent a greater share of people with favorable views of the President whereas negative scores represent a greater share of people with unfavorable views. Each line represents a moving average based on the five preceding poll numbers.
As you can see, there’s indeed some evidence of systematic differences in poll results based on whether surveys were conducted using likely voters, registered voters, or all adults.
In general, President Trump receives lower favorability ratings from polls based on all adults, as we would expect based on the analysis from Pew Research. Among polls conducted between July 12, 2015 and February 8, 2017, Trump’s average favorability score among all adults, weighted by recency, is 36 percent. His weighted average net favoriability score is -24 percentage points.
Meanwhile, Trump’s favorability ratings tend to be two or three points higher among polls that use likely voters or registered voters. For polls conducted with likely voters, Trump’s average favorability score, weighted by recency, is 38 percent, and his average net favorability score is -20 percentage points. For polls based on registered voters, his average net favorabiilty rating, weighted by recency, is 39 percent, and his average net favorability rating is -18 percentage points.
But, as you might have noticed, the whole thing is a bit messy. And whether Trump performs better among likely or registered voters, compared with all adults, varies with time. For instance, Trump performed considerably well among registered voters during the period of time between September 2015 and March 2016. He also performed rather well among both registered and likely voters from September 2016 through to election day.
Presently, based on polls that ended either February 7 or 8, Trump is performing slightly best among likely voters, with a current average net favorability rating of -2.4 percentage points. For polls based on registered voters, Trump’s current average net favorability rating stands at -5.8 percentage points, and for polls based on all adults it is -11.4 percentage points.
So, all of this is to say that, when you’re reading about a poll in the news, you should stop to consider a basic question about how the poll was conducted. Specifically, was it conducted using likely voters, registered voters, or all adults?
And if a poll based on likely or registered voters suggests 50 percent of Americans view President Trump favorably, you should probably subtract two or three points from this number if you want to get a better sense for how the broader public feels.
Brian Kurilla is a psychological scientist with a Ph.D. in cognitive psychology. You can follow Brian on Twitter @briankurilla