Big test for pollsters

The polling industry faces a big test this election as we will see whether or not they are reliable in this age of cell-phones and the public’s growing unwillingness to answer their questions.  From Robert J. Samuelson:

Among pollsters, there’s fear that changing technology (mainly cellphones) and growing public unwillingness to do interviews are undermining telephone surveys — and that there’s no accurate replacement in sight. A recent study by the Pew Research Center reported its response rate at 9 percent, down from 36 percent in 1997. Put differently: in 1997, Pew made about three residential calls to get one response; now it makes 10.

Beginning with answering machines and caller-ID in the ’70s and ’80s, suspicious Americans have become more selective in screening calls. Robo-calls — automated messages for products, politicians, charities and polls — have deepened the hostility. “The mass of communications coming into people’s homes ends up being a blur,” says Pew pollster Scott Keeter.

Cellphones pose problems because people who use them exclusively — people who don’t have landline phones — are younger, poorer and more Democratic than the general population. By late 2011, 32 percent of Americans 18 and over had only a cellphone, up from 16 percent in early 2008. Among those 25 to 29, the share was 60 percent. Under-surveying these people could distort polls. Many pollsters, though not all, now canvass cellphones. But this is increasingly expensive. By present trends, half of Americans could be exclusive cellphone users by the 2016 election. . .

Less reassuring is telephone polling’s steep and rising costs, which could cause cash-strapped media organizations to balk. Contacting cellphones is expensive, because numbers must be dialed by hand. By contrast, computers can automatically dial landline numbers, making it easier to reach live people. (Congress prohibited this for cellphones to protect people from paying for unsolicited incoming calls.) A typical survey costs Pew from $60,000 to $100,000, says Keeter. That would cover renting tens of thousands of landline and cellphone numbers to produce 1,500 interviews of about 20 minutes each.

The solution seems obvious: switch to the Internet. But technically, that’s hard. Internet users may not be a representative sample of the U.S. population. Does the person behind that e-mail live in the United States? Permanent panels of respondents may act differently from randomly contacted people. Experiments are under way. Meanwhile, pollsters are stretched between a past that’s growing untenable and a future that doesn’t yet exist.

via Robert J. Samuelson: Pollsters’ moment of truth – The Washington Post.

To pick up on some of our earlier conversation, it may well be true that pollsters are undercounting Republicans.  But they are also undercounting those who exclusively use cell phones; that is, younger voters who tend to vote Democrat.  But we shall see what happens on November 6.

About Gene Veith

Professor of Literature at Patrick Henry College, the Director of the Cranach Institute at Concordia Theological Seminary, a columnist for World Magazine and TableTalk, and the author of 18 books on different facets of Christianity & Culture.

  • Rose

    I wonder if people are getting calls on their Obamaphones to remind them to vote.

  • Rose

    I wonder if people are getting calls on their Obamaphones to remind them to vote.

  • larry

    I think this is healthy for democracy. “Predicting” elections in order to “push” them way before the vote and hence “just going through the motions” has far too long been occurring. The more candidates are kept “guessing” the more they might be on their toes and actually come up with substantive things rather than just verbally “pushing” political punch lines to gather up their group.

  • larry

    I think this is healthy for democracy. “Predicting” elections in order to “push” them way before the vote and hence “just going through the motions” has far too long been occurring. The more candidates are kept “guessing” the more they might be on their toes and actually come up with substantive things rather than just verbally “pushing” political punch lines to gather up their group.

  • Tom Hering

    But they are also undercounting those who exclusively use cell phones; that is, younger voters who tend to vote Democrat. (Dr. Veith)

    But younger voters also tend to not vote. So the undercounting may not matter much.

  • Tom Hering

    But they are also undercounting those who exclusively use cell phones; that is, younger voters who tend to vote Democrat. (Dr. Veith)

    But younger voters also tend to not vote. So the undercounting may not matter much.

  • Kirk

    No. They’re under counting Republicans. That’s the narrative.

  • Kirk

    No. They’re under counting Republicans. That’s the narrative.

  • Joe

    Interesting; I am the 26.6 percent,:

    – More than one-in-four U.S. homes, or 26.6 percent, had only a wireless phone as of June 2010, up from 13.6 percent in 2007.

    – The percentage of wireless-only homes increased in every state, ranging from 35.2 percent in Arkansas to 12.8 percent in Rhode Island and New Jersey.

    – Rhode Island and New Jersey were the lowest at 12.8 percent of adults and children in cellular-only households. Next was Connecticut, at 13.6. New York was at 17 percent, and California at 18 percent.

    – By contrast, Arkansas had the highest concentration of people in cell-only households, at 35.2 percent. Next were Mississippi, at 35.1 percent, Texas at 32.5 percent, North Dakota, at 32.3 percent, Idaho at 31.7 percent, and Kentucky at 31.5 percent.

    http://www.cbsnews.com/2100-3480_162-20056730.html

  • Joe

    Interesting; I am the 26.6 percent,:

    – More than one-in-four U.S. homes, or 26.6 percent, had only a wireless phone as of June 2010, up from 13.6 percent in 2007.

    – The percentage of wireless-only homes increased in every state, ranging from 35.2 percent in Arkansas to 12.8 percent in Rhode Island and New Jersey.

    – Rhode Island and New Jersey were the lowest at 12.8 percent of adults and children in cellular-only households. Next was Connecticut, at 13.6. New York was at 17 percent, and California at 18 percent.

    – By contrast, Arkansas had the highest concentration of people in cell-only households, at 35.2 percent. Next were Mississippi, at 35.1 percent, Texas at 32.5 percent, North Dakota, at 32.3 percent, Idaho at 31.7 percent, and Kentucky at 31.5 percent.

    http://www.cbsnews.com/2100-3480_162-20056730.html

  • Steve Billingsley

    In commercial market research (survey research done for companies such as Wal-Mart, Apple, etc.) over half of the quantitative research is now fielded online via the use of online access panels or targeted customer databases. Years of ROR (research-on-research) has confirmed that if sampled competently, the research is every bit as reliable (in many cases more so) than telephone research. Polling has not made that jump yet because it can’t crack the nut of projection and representativeness. But frankly, they haven’t tried particularly hard. From what I have seen, I think this is the year that political polling jumps the shark and will have to have a major re-think. Rasmussen and Gallup have tried to solve this with weighting schemes (their secret sauce so to speak) and have had some degree of success. Pollsters like to present themselves as scientific, but it is a bit more like alchemy than true science.

    I do think that Republican voters (not necessarily registered Republicans) have been fairly consistently underrepresented this cycle but that isn’t a new problem and I don’t know that it is any sort of conspiracy. I think it is just because polling is pretty hard.

  • Steve Billingsley

    In commercial market research (survey research done for companies such as Wal-Mart, Apple, etc.) over half of the quantitative research is now fielded online via the use of online access panels or targeted customer databases. Years of ROR (research-on-research) has confirmed that if sampled competently, the research is every bit as reliable (in many cases more so) than telephone research. Polling has not made that jump yet because it can’t crack the nut of projection and representativeness. But frankly, they haven’t tried particularly hard. From what I have seen, I think this is the year that political polling jumps the shark and will have to have a major re-think. Rasmussen and Gallup have tried to solve this with weighting schemes (their secret sauce so to speak) and have had some degree of success. Pollsters like to present themselves as scientific, but it is a bit more like alchemy than true science.

    I do think that Republican voters (not necessarily registered Republicans) have been fairly consistently underrepresented this cycle but that isn’t a new problem and I don’t know that it is any sort of conspiracy. I think it is just because polling is pretty hard.

  • Jon

    I just have a hunch that things are going to be a lot redder this time around than even the polls can capture.

  • Jon

    I just have a hunch that things are going to be a lot redder this time around than even the polls can capture.

  • Tom Hering

    Maybe it’s time for pollsters to get off their duffs and go back to the early days of door-to-door, face-to-face interviews. And maybe find themselves a place like this.

  • Tom Hering

    Maybe it’s time for pollsters to get off their duffs and go back to the early days of door-to-door, face-to-face interviews. And maybe find themselves a place like this.

  • MichaelZ

    So wait…Joe @5 says that the states with the highest “cell-only” households are dyed in the wool red-states, and yet the article claims that cell-only households are more liberal/democrat. How does that work? Are the democrats in those red-states all dumping their landlines in and democrats in blue-states keeping them for some reason? This sounds fishy.

  • MichaelZ

    So wait…Joe @5 says that the states with the highest “cell-only” households are dyed in the wool red-states, and yet the article claims that cell-only households are more liberal/democrat. How does that work? Are the democrats in those red-states all dumping their landlines in and democrats in blue-states keeping them for some reason? This sounds fishy.

  • DonS

    We did have this discussion, on this blog, in detail back in September.

    Steve @ 6 is right. Most pollsters, since the 2008 election cycle, have adopted cell phone polling in one form or another to try to capture cellphone-only voters. Robopollers, like Rasmussen and PPP, use proprietary weighting schemes, involving Internet databases and other “secret sauce” factors, to account for those households, since they are not allowed to autodial cellphones. So these households are, indeed, being accounted for in the current election cycle. In fact, I suspect that they are being over-accounted for, which in turn explains some of the weird polling disparities we are seeing, particularly polling samples which are clearly overweighted to Democrats.

    The other factor, of course, is the low response rate, which has dropped precipitously since 1997, as noted in the posted article, from 36% to 9%, and even since 2008, when it was about 15%. As we previously discussed, when the response rate is this low, is a self-selection bias occurring? In other words, are those 9% who agree to spend 20 minutes with a pollster representative of the population as a whole? I doubt it.

    Additionally, weighting for demographic characteristics creates problems. It is certainly proper to weight your initial all-adults sample for census-identified demographics, to ensure that the sample is representative. But some pollsters weight it again later in the process, after identifying registered voters, or even after identifying likely voters. Doing that can lead to overweighting non-white voters, since minority citizens tend not to be registered or to actually vote in the same proportions as white citizens. This can skew your results to the Democratic side, which is why actual election results usually skew Republican compared to pre-election polls.

    One more factor creating problems for pollsters is early voting. If a respondent says they have already voted, they are immediately passed through the likely voter screen. However, the polls are showing that about twice as many voters claim to have already voted as election registrars are reporting. When respondents lie about their early votes, the polls suffer.

  • DonS

    We did have this discussion, on this blog, in detail back in September.

    Steve @ 6 is right. Most pollsters, since the 2008 election cycle, have adopted cell phone polling in one form or another to try to capture cellphone-only voters. Robopollers, like Rasmussen and PPP, use proprietary weighting schemes, involving Internet databases and other “secret sauce” factors, to account for those households, since they are not allowed to autodial cellphones. So these households are, indeed, being accounted for in the current election cycle. In fact, I suspect that they are being over-accounted for, which in turn explains some of the weird polling disparities we are seeing, particularly polling samples which are clearly overweighted to Democrats.

    The other factor, of course, is the low response rate, which has dropped precipitously since 1997, as noted in the posted article, from 36% to 9%, and even since 2008, when it was about 15%. As we previously discussed, when the response rate is this low, is a self-selection bias occurring? In other words, are those 9% who agree to spend 20 minutes with a pollster representative of the population as a whole? I doubt it.

    Additionally, weighting for demographic characteristics creates problems. It is certainly proper to weight your initial all-adults sample for census-identified demographics, to ensure that the sample is representative. But some pollsters weight it again later in the process, after identifying registered voters, or even after identifying likely voters. Doing that can lead to overweighting non-white voters, since minority citizens tend not to be registered or to actually vote in the same proportions as white citizens. This can skew your results to the Democratic side, which is why actual election results usually skew Republican compared to pre-election polls.

    One more factor creating problems for pollsters is early voting. If a respondent says they have already voted, they are immediately passed through the likely voter screen. However, the polls are showing that about twice as many voters claim to have already voted as election registrars are reporting. When respondents lie about their early votes, the polls suffer.

  • http://facebook.com/mesamike Mike Westfall

    My wife has been doing some phone banking for a particular conservative advocacy group, and maybe one out of ten don’t hang up on her. She’s got a script to read before asking the one and only question, but the script is obviously trying to push the callee, culminating in, “I don’t think Obama’s policies are working.” then the question: “What do you think? Have Obama’s policies helped or hurt?”

    I’d hang up on that too.

  • http://facebook.com/mesamike Mike Westfall

    My wife has been doing some phone banking for a particular conservative advocacy group, and maybe one out of ten don’t hang up on her. She’s got a script to read before asking the one and only question, but the script is obviously trying to push the callee, culminating in, “I don’t think Obama’s policies are working.” then the question: “What do you think? Have Obama’s policies helped or hurt?”

    I’d hang up on that too.

  • http://www.toddstadler.com/ tODD

    MichaelZ (@9), you’re conflating an awful lot of things, there.

    First of all, the article says that “people who don’t have landline phones are … more Democratic than the general population”. It doesn’t say they’re all Democrats.

    Also, please note the relatively small range of cell-only houses across the states — from 13% to 35% of households.

    Then note that, for states in which the statistics are broken out regionally, the urban areas are consistently more cell-only than the rest of the state, as should surprise no one. Nor should it be controversial to claim that urban areas are more liberal, no matter how conservative the state.

    Finally, the states with the highest cell-only households are hardly “dyed in the wool red-states”. Arkansas, with the greatest percentage of such households, is about as “purple” as a state can get — at least if you look at the average margin of victory for presidential elections from 1992-2008. And Oregon, clocking in at the seventh most cell-only state, is fairly blue. And, in general, there’s not a lot of correlation between the two numbers.

    In fact, because I’m like that, I ran a scatter plot comparing the percentage of cell-only households to the average margin of victory for each state, and I got a pretty weak R-squared value: 25%.

    In short, not actually fishy, no.

  • http://www.toddstadler.com/ tODD

    MichaelZ (@9), you’re conflating an awful lot of things, there.

    First of all, the article says that “people who don’t have landline phones are … more Democratic than the general population”. It doesn’t say they’re all Democrats.

    Also, please note the relatively small range of cell-only houses across the states — from 13% to 35% of households.

    Then note that, for states in which the statistics are broken out regionally, the urban areas are consistently more cell-only than the rest of the state, as should surprise no one. Nor should it be controversial to claim that urban areas are more liberal, no matter how conservative the state.

    Finally, the states with the highest cell-only households are hardly “dyed in the wool red-states”. Arkansas, with the greatest percentage of such households, is about as “purple” as a state can get — at least if you look at the average margin of victory for presidential elections from 1992-2008. And Oregon, clocking in at the seventh most cell-only state, is fairly blue. And, in general, there’s not a lot of correlation between the two numbers.

    In fact, because I’m like that, I ran a scatter plot comparing the percentage of cell-only households to the average margin of victory for each state, and I got a pretty weak R-squared value: 25%.

    In short, not actually fishy, no.

  • http://www.biblegateway.com/versions/Contemporary-English-Version-CEV-Bible/ sg

    Both our pastors use cell only.

  • http://www.biblegateway.com/versions/Contemporary-English-Version-CEV-Bible/ sg

    Both our pastors use cell only.

  • http://steadfastlutherans.org/ SAL

    However the election goes decent numbers of pollsters will lose some credibility. That’s because the most reputable poll has Romney winning handily (+5) and other reputable pollsters have Obama winning the swing states handily.

    If Romney were ahead by 5% then he wouldn’t be losing most of the swing states. Somebody is wrong here.

  • http://steadfastlutherans.org/ SAL

    However the election goes decent numbers of pollsters will lose some credibility. That’s because the most reputable poll has Romney winning handily (+5) and other reputable pollsters have Obama winning the swing states handily.

    If Romney were ahead by 5% then he wouldn’t be losing most of the swing states. Somebody is wrong here.

  • http://www.toddstadler.com/ tODD

    SAL (@14), it’s actually clearer than that. Rasmussen — presumably the poll you refer to as “the most reputable”*, which typically means “favoring My Guy” — has state-level polls, as well.

    Let’s assume the swing states are Colorado, Florida, Iowa, Michigan, Nevada, New Hampshire, North Carolina, Ohio, Pennsylvania, Virginia, and Wisconsin (those would be RealClearPolitics’ choices). When you compare Rasmussen’s polls to the RCP average, Rasmussen only significantly differs in the following states:

    * Colorado: Rasmussen has Romney +4, while the RCP average is a dead tie.
    * Iowa: Rasmussen has it as a tie, and the RCP average is Obama +1.
    * New Hampshire: Rasmussen has Romney +2, RCP average is Obama +1.
    * Ohio: Rasmussen has Romney +2, the RCP average is Obama +2.1.
    * Virginia: Rasmussen is at Romney +2, while the RCP average is a dead tie.
    * Wisconsin: Rasmussen has it tied, while the RCP average is Obama +2.3

    Colorado and Ohio have the biggest differences of around 4 percentage points. New Hampshire is next closest. Just a little while longer to see whether Rasmussen’s divergent tack is correct or not.

    *FWIW, Rasmussen has the smallest sample size, and therefore the largest margin of error, of any national pollster that I could find.

  • http://www.toddstadler.com/ tODD

    SAL (@14), it’s actually clearer than that. Rasmussen — presumably the poll you refer to as “the most reputable”*, which typically means “favoring My Guy” — has state-level polls, as well.

    Let’s assume the swing states are Colorado, Florida, Iowa, Michigan, Nevada, New Hampshire, North Carolina, Ohio, Pennsylvania, Virginia, and Wisconsin (those would be RealClearPolitics’ choices). When you compare Rasmussen’s polls to the RCP average, Rasmussen only significantly differs in the following states:

    * Colorado: Rasmussen has Romney +4, while the RCP average is a dead tie.
    * Iowa: Rasmussen has it as a tie, and the RCP average is Obama +1.
    * New Hampshire: Rasmussen has Romney +2, RCP average is Obama +1.
    * Ohio: Rasmussen has Romney +2, the RCP average is Obama +2.1.
    * Virginia: Rasmussen is at Romney +2, while the RCP average is a dead tie.
    * Wisconsin: Rasmussen has it tied, while the RCP average is Obama +2.3

    Colorado and Ohio have the biggest differences of around 4 percentage points. New Hampshire is next closest. Just a little while longer to see whether Rasmussen’s divergent tack is correct or not.

    *FWIW, Rasmussen has the smallest sample size, and therefore the largest margin of error, of any national pollster that I could find.

  • DonS

    tODD @ 15 and SAL @ 14: I believe SAL is referring to Gallup, not Rasmussen. The Gallup national tracking poll currently shows a 5 point Romney lead. Rasmussen is at 2 points, and Gallup certainly qualifies as the “most reputable poll”. Note that Gallup has suspended their polling the last two nights because of Sandy.

    SAL, Gallup is seeing a much more Republican-leaning electorate than are most of the other pollsters, including Rasmussen. If you look at their website you will see that their party identification polling shows more Republicans than Democrats currently and they are also measuring much higher voter intensity among Republicans than Democrats. The amazing thing is that most of the other national polls are showing a small Romney lead (tie to +3), even though their electorates are demographically modeled to be D+4 or more. Generally, Romney is winning independents big, which allows him to lead a D+4 poll.

    The state polls, generally, have been even more Democratic-leaning, explaining the discrepancy we are seeing. For example, even Rasmussen’s last poll was D+5 (it showed Romney up 2), and the ones showing a 3-5 point Obama lead are at D+9 and up. This is despite the fact that the electorate in the Democratic wave year of 2008 in Ohio was only D+5. How these pollsters are getting D+9 samples, consistently, is the mystery. The same types of wacko samples are occurring in the polling in VA, NC, FL, and CO (to a somewhat lesser extent). Either these state pollsters are measuring a Democratic voter intensity that is not otherwise apparent anecdotally, or whether they are seriously erring in their turnout models is what we will find out on November 6.

    The one thing we can know for sure is that Romney is NOT going to win the PV and lose the EV, barring some kind of crazy anomaly never before seen in modern U.S. elections. Demographically, since Republican strength is in relatively lower population red states, and Republicans do not run up the kind of vote margins that Democrats do in their populous blue states, such an outcome would be almost unimaginable. It is worth noting, as well, that no modern candidate (since the Civil War — I didn’t check before that) has ever won the PV by more than 1.5% and lost the EV.

    This means that either the national polls showing Romney ahead, or the state polls showing Obama with an electoral vote lead, are wrong. Given the attitudes and actions of the national campaigns, and the general mood of the electorate, my money is on the national polls. And I think the Romney win will be substantial.

    But, as tODD says, we will know very shortly.

  • DonS

    tODD @ 15 and SAL @ 14: I believe SAL is referring to Gallup, not Rasmussen. The Gallup national tracking poll currently shows a 5 point Romney lead. Rasmussen is at 2 points, and Gallup certainly qualifies as the “most reputable poll”. Note that Gallup has suspended their polling the last two nights because of Sandy.

    SAL, Gallup is seeing a much more Republican-leaning electorate than are most of the other pollsters, including Rasmussen. If you look at their website you will see that their party identification polling shows more Republicans than Democrats currently and they are also measuring much higher voter intensity among Republicans than Democrats. The amazing thing is that most of the other national polls are showing a small Romney lead (tie to +3), even though their electorates are demographically modeled to be D+4 or more. Generally, Romney is winning independents big, which allows him to lead a D+4 poll.

    The state polls, generally, have been even more Democratic-leaning, explaining the discrepancy we are seeing. For example, even Rasmussen’s last poll was D+5 (it showed Romney up 2), and the ones showing a 3-5 point Obama lead are at D+9 and up. This is despite the fact that the electorate in the Democratic wave year of 2008 in Ohio was only D+5. How these pollsters are getting D+9 samples, consistently, is the mystery. The same types of wacko samples are occurring in the polling in VA, NC, FL, and CO (to a somewhat lesser extent). Either these state pollsters are measuring a Democratic voter intensity that is not otherwise apparent anecdotally, or whether they are seriously erring in their turnout models is what we will find out on November 6.

    The one thing we can know for sure is that Romney is NOT going to win the PV and lose the EV, barring some kind of crazy anomaly never before seen in modern U.S. elections. Demographically, since Republican strength is in relatively lower population red states, and Republicans do not run up the kind of vote margins that Democrats do in their populous blue states, such an outcome would be almost unimaginable. It is worth noting, as well, that no modern candidate (since the Civil War — I didn’t check before that) has ever won the PV by more than 1.5% and lost the EV.

    This means that either the national polls showing Romney ahead, or the state polls showing Obama with an electoral vote lead, are wrong. Given the attitudes and actions of the national campaigns, and the general mood of the electorate, my money is on the national polls. And I think the Romney win will be substantial.

    But, as tODD says, we will know very shortly.

  • Trey

    Todd, the Rasmussen poll is credible because it was most accurate last ring around. Moreover, it looks at likely voter not registers and does not slant the sample based on 2008. RCP is an average of polls and those are either likely or registered voters. It’s like apples to bananas (to oranges is cliche). What about Gallup? Your contention regarding the Rasmussen poll is that he likes because it supports his guy is spoiling the well fallacy. At least you follow it with some evidence, e.g., sampling size. Where did you find this?

  • Trey

    Todd, the Rasmussen poll is credible because it was most accurate last ring around. Moreover, it looks at likely voter not registers and does not slant the sample based on 2008. RCP is an average of polls and those are either likely or registered voters. It’s like apples to bananas (to oranges is cliche). What about Gallup? Your contention regarding the Rasmussen poll is that he likes because it supports his guy is spoiling the well fallacy. At least you follow it with some evidence, e.g., sampling size. Where did you find this?

  • SAL

    #15 Todd, Gallup is typically viewed as the most reputable poll, not Rasmussen.

    #16 Don, I also lean towards the national polls being right but I’d not bet on it.

  • SAL

    #15 Todd, Gallup is typically viewed as the most reputable poll, not Rasmussen.

    #16 Don, I also lean towards the national polls being right but I’d not bet on it.


CLOSE | X

HIDE | X