Nate Silver: The New Pollster Genius?

By Daniel Engber:

It’s well after midnight on the East Coast, and the results are in: Nate Silver has won the 2012 presidential election by a landslide. His magic formula for predictions, much maligned in some corners in recent weeks, appears to have hit the mark in every state—a perfect 50 green M&Ms for accuracy. Now my Twitter feed is blowing up with announcements of his coronation as the Emperor of Math and the ruler of the punditocracy. Wait—it was even more than that, they say: a victory for blogging, and also one for rational thought. He proved the haters wrong! He proved science right! Is this guy getting lucky tonight or what?

But all these stats triumphalists have it wrong. Nate Silver didn’t nail it; the pollsters did. The vaunted Silver “picks”—the ones that scored a perfect record on Election Day—were derived from averaged state-wide data. According to the final tallies from FiveThirtyEight, Obama led by 1.3 points in Virginia, 3.6 in Ohio, 3.6 in Nevada, and 1.9 in Colorado. He won all those states, just like he won every other state in which he’d led in averaged, state-wide polls. That doesn’t mean that Silver’s magic model works. It means that polling works, assuming that its methodology is sound, and that it’s done repeatedly.

About Scot McKnight

Scot McKnight is a recognized authority on the New Testament, early Christianity, and the historical Jesus. McKnight, author of more than forty books, is the Professor of New Testament at Northern Seminary in Lombard, IL.

  • http://bookwi.se Adam Shields

    That seems like a petty response. Silver has never claimed that he is working everything by himself (at least not that I have seen.) He is working with data and just claims that he is weighting it so that it is more accurate as a whole than any individual poll could be.

  • http://bobcornwall.com Bob Cornwall

    I have to agree with Adam above. Silver has simply figured out a fairly sound way of reading the polls, which tend to be all over the place. He understands which polls should be valued above others, and has a formula that allows this. Others have tried and failed to duplicate his success. So I agree this is a petty response. And to Scot — what’s the point of posting it. Are you suggesting something is wrong with Silver’s analysis or just want to get a discussion going?

  • Jag

    There are a lot of sour grapes toward Silver because it was common knowledge in the “Fox world” that he was, had to be, wrong with his analysis. There is a growing tendency to dismiss science/math as simply an opinion when it produces a result that is inconsistent with our own opinion.

  • Jag
  • Kyle J

    Dual lessons really:

    Polls still work, despite low response rates.

    A really smart guy can figure out how to make them work even better by aggregating and weighting them.

    Quick rundown of which polls were closest here:

    http://www.washingtonpost.com/blogs/ezra-klein/wp/2012/11/07/which-pollsters-to-trust-in-one-chart/

  • scotmcknight

    Bob, for no reason other than the discussion. I have relied on Pew and Rasmussen because they were judged most accurate in 2008, but my son pushed me to see Silver’s stuff. So I got interested in the discussion; this article suggests the pollsters already had that info and that Silver has used them well.

  • Jared

    Silver’s information is helpful; however, it is overhyped and its sophistication is praised to much. There are much simpler ways to arrive at the information that Silver has. Any model that bases the overwhelming amount of their projection on the state polls is going to be just as accurate as those polls. The polls were accurate, Silver was accurate.

    Just a simple averaging of state polls would yield a very similar result. the RCP average of the state polls correctly predicted 49 out of 50 states correctly, more than almost every pundit and just one short of Silver’s prediction. And the one state difference in the two models was Florida which the 538 model’s confidence in the state was a whopping 50.3% chance of an Obama win. It was pretty much blind luck that 538 correctly predicted that state correctly and best a simple average of the polls by one state.

    Further, his predictions are only so impressive when you realize that he make his final projections 24 hours before the actual results are in, and further illustrates that he is a slave to the current polling data. If he was somehow able to use data points to correctly predict elections 3 months out, he would be doing something that no one else was able to do.

    For me, Silver is most useful not for the prediction that he makes but the analysis that he and his team provide in the actual content of the blog.

  • Tony Springer

    Nate:stats = Scot: the gospel

  • http://www.timgombis.com Tim Gombis

    I tell you what — it’s been a great few days to be Nate Silver!

  • http://everythingnew.org Jeff Cook

    — “Results ask Nate Silver if they’re significant.” — @collision
    — “Nate Silver’s samples have only a median and a mode. Because no number would be mean to Nate Silver.” — @dilefante
    — “Nate Silver can recite pi. Backwards.” -@smedette
    — “When Nate Silver goes out to eat, his tips alway accurately correspond to the quality of the service.” — @benhuh
    — “When Nate Silver asks you, ‘Wanna make a bet?’ the correct answer is no.” — @perko
    — “There are no imaginary numbers, only ones Nate Silver hasn’t acknowledged yet.” — @tbq_
    — “Nate Silver can divide by zero.” — @jkfecke

  • Kenny Johnson

    @Jared,

    Nate is getting a a lot more credit than maybe is due — in large part because he was the one that was singled out and attacked by conservative pundits, but Nate does do something a little more than just average the polls. He weights them and applies some economic and other data to the model. He also tries to use that polling data to forecast the election day by day (something RCP doesn’t do). So, for example, when people were claiming that Romney had momentum, Nate (and a few other polling analysts) correctly pointed out that it wasn’t the case and Obama was still the 75/25 favorite to win. Nate also correctly predicted a Obama win since the general election started. He showed Obama with a popular vote lead even when the national polls seemed to suggest Romney was in the lead. RCP, for example totally missed the Pop Vote.

    But you’re right — pretty much all the polling aggregators had similar predictions (Votamatic, Pollster, RCP, PEC, etc.) about which states were going where. I was checking all of them. I believe Nate and Votamatic were the only ones who called all the correct states. Nate missed the Senate races — which I think PEC did much better at. PEC also nailed a lot of the predictions (popular vote, states (he missed Florida), senate races, etc.)

    Either way… all the polling aggregators did much better than the pundits.

  • Kenny Johnson

    Sam Wang at PEC shows that he actually was closer overall to the results even though he missed FL. http://election.princeton.edu/2012/11/09/karl-rove-eats-a-bug/

  • http://krusekronicle.com Michael W. Kruse

    Adam #1 Bob #2

    The critique of Silver is not a Fox World sour grapes response. I’d invite people to check out the math behind Silver’s estimations:

    http://www.physicscentral.com/buzz/blog/index.cfm?postid=9140341535810311671

    He is making the case that Silver underestimated the likelihood of the prevailing candidate prevailing in the swing states. If the values were all as low as he predicted (say 75% instead of 98%) then his chances of having all the states come out as predicted was something like 7%. You would do just as well by taking an aggregate of the polls and come up with the exact same electoral college breakdown, which is exactly what half a dozen other pundits did.

  • unapologetic catholic

    To begin with:

    “If he was somehow able to use data points to correctly predict elections 3 months out, he would be doing something that no one else was able to do.”

    That is correct. And, that’s exactly what he did. Since the inception of the campaign he had forecast that the chances of an Obama victory hovered around 75%, dipping to 60% after the first debate. Although the chance of an Obama victory changed in a relatively narrow range until increasing gradually to a 90% chance the days before the election, he had repeatedly and consistently been forecasting an Obama victory. He shows all his work on his website for all to see and critique.

    The point is that nobody believed him. As he repeatedly said, if Romney wins, then all of the polls are wrong. Nobody took him seriously. Generally, the commentators claimed many polls were biased, and other favored Romney and the election would be “too close to call.”

    Actually, no, it wasn’t too close to call at all. If what he did was so easy and elementary, why did everybody think the election was too close to call? Silver was willing to bet Joe Scarborough $2000 on the election outcome. Intrade and European bookies all had Obama as a heavy favorite weeks prior to the election. How did everyone else, especially the political experts, not see what was going to happen?

    Silver has an answer for that, too. In his book, he quotes both Democratic and Republican political analysts before the 2008 election. Those analysts also had no idea of the obvious event that would be occurring. His conclusion is that political pundits do not know what they are talking about and are as accurate as often as a coin flip.

    Here’s National Review Online’s 2012 election predictions taken from 21 political analysts and commentators—people like Hugh Hewitt who claim to understand politics.

    http://www.nationalreview.com/articles/332645/election-crystal-ball-nro-symposium?pg=1

    The result: 20 out of 21 called it wrong, either predicting a Romney landslide or the race was too close to call. One NRO person forecast an Obama win only few days before the election when Silver had a 90% chance of an Obama win.

    How could all of these people be so wrong? Silver convincingly says that they were wrong because they don’t know what they are talking about.

    NRO is conservative, but I am positive that the results would be the same from liberal commentators as well.

    These “pundits” are upset that Silver has shown that they wear no clothes.

  • Morbert

    I don’t really understand this entry. The reason Silver is getting so much attention is there is a myth going around the political world that scientific approaches like Silver’s are just an alternative worldview, no better or worse than other political pundits’. Therefore, when he tenders correct probabilities, everyone is shocked.

    Silver deserves his fame by virtue of the fact that political pundits have an abysmal philosophy.

    http://xkcd.com/1131/

  • Kenny Johnson

    @Michael,

    The link is broken. It looks like they removed the post for some reason. And I couldn’t seem to find a cached version.

    But the bulk of the criticism came from right-wing pundits, which is odd, because as has been mentioned, there were others making similar predictions based on the polling data. So I’m not sure why he was singled out.

    But I did some some criticism from others like Sam Wang — but it was more about methodology specifics.

  • http://browardemergent.blogspot.com/ Steve

    I am in Florida, and was a canvassing captain for the Obama campaign. I have followed RCP all year and was recently turned on to Nate Silver by a friend. It’s not just simple averaging, like RCP does, but the adjustments that seem to matter. That’s where the skill comes in. Nate’s call for Florida on Election Day morning was amazing IMO.

  • SteveSherwood

    Jared, Silver gave percentage odds for victory from the start. They spiked over 90% right at the end, but even after the Denver debate, Silver always had Obama with at least a 68% chance of winning. He is getting credit, as others have mentioned for two reasons. 1. He was so resoundingly derided by right wing pundits and 2. The percentages, adjusted daily, really have a sense that he had stuck his neck out in a way that other “I think this is going to break for Obama based upon the polls” folks didn’t have. It was those percentages, in the 80s the week before the election, that I think really infuriated conservative “pundits” like Dick Morris and Karl Rove and fascinated folks. Maybe they were just a marketing gimmick thought up by his bosses at NYT, but they got everyone’s attention.

    Is he getting too much fame and credit? Maybe, but let’s picture what massive ridicule he’d have received if he’d been wrong. He likely would have been fired. With great risk came the possibility of great reward. I’ve seen him interviewed. He comes off as the perfect stereotype of the math/stat obsessed “geek.” Bravo for him for having a moment in the sun!

  • Eric E

    For what its worth, the Physics Central article was not very good. I read it when it came out but it seems like it has since been removed. Nate Silver is a Bayesian. When he says “I’m 90% sure that this will happen” he doesn’t mean that there is a 10% chance he will be wrong. That’s a frequentist interpretation of probability and it works well for things like rolling dice many times but it doesn’t work well for what Silver is doing. When Silver says “90%,” he is talking about his confidence. It represents the certainty of his state of knowledge. Multiplying the confidences doesn’t give us a correct result in this case.

    Additionally, even assuming a frequentist framework, it is really sloppy to assume that the statistics are independent. She uses the example of rain on different days: If there is a .846 probability of rain on Monday and a .794 probability of rain on Tuesday, then there is a .672 probability of rain on both days. Fine. But what if we are talking about different cities in the path of a storm? Say city 1 has a .86 probability of rain Monday at 6:00 pm, city 2 has a .94 probability of rain at 6:10 pm, city 3 has a probability of .81 of rain Monday at 6:20 pm, then the chances of them all getting rain is not .65 (which is what you get if you simplisticly multiply them together like the author of that article did). It will be much higher than that because they are not independent events. Rolling a die multiple times are independent events, national elections in different states are not.


CLOSE | X

HIDE | X