Montana: Where Nate Silver got it wrong in Election 2012

Montana senate forecast NY Times Nate Silver

Nate Silver’s Montana Senate forecast from the NY Times

I almost minored in Political Science in my university days. I was president of my Philosophy Society for a while, and my local Golden Key International Honour Society. But other than that, I am, like most of you, a novice in the strange world of politics.

Yet, again like many of you, I watched this year’s election coverage with a certain level of neurosis and/or addiction. Early on, thankfully, I discovered Nate Silver’s prophetic blog fivethirtyeight.blogs.nytimes.com. Soon I was checking it twice a day. Simply put, it’s brilliant. If you haven’t discovered it yet, I’m sorry. It’s a bit late now.

But you can still buy his book.

The blog focused on the Presidential election, the title coming from the Electoral College’s 538 members. But it did also look at state-by-state Senate elections. His Presidential forecast is uncanny. Scary even. So scary that a website called http://isnatesilverawitch.com/ has been devoted to understanding his (supernatural?) powers.

The wonderful thing is that Nate didn’t hold much back. He openly discussed his feelings about various polls, gave greater “weight” to the ones that had better methodologies, and overall demystified much of the electoral process. In fact, for the last 2 weeks at least I have been utterly mystified that most of the media kept talking about “Romney momentum” and “too close to call” when Nate Silver had clearly shown that Romney’s “momentum” ended about a week after it started (beginning just after the first debate, Oct 5, and ending Oct 12), and that after Oct 20, Romney never had a better than 1 in 3 chance of winning. By November 5, Romneys chances were less than 1 in 10.

Apparently this caused a lot of ruckus amongst the pundits.

With my current cold and the fact that I’m in England, I couldn’t stay up to see the late night state-by-state numbers roll in as I might have back in Montana. But, with Nate’s careful number-crunching, I went to sleep confident that we would have 4 more years of Obama.

What I wasn’t so sure about was my home state’s Senate race, between incumbent Democrat Jon Tester and Republican Denny Rehberg (a long-time Representative). My parents, both relatively newly retired, have been stumping for Tester for weeks now, going door-to-door, cooking meals, and making phone calls. And I made sure to get an absentee ballot in to support him, knowing well that it wouldn’t mean much in the Presidential election.

Based on my parents’ data, and the polls (in the image), Rehberg had been ahead early on, but Tester was coming back and had, in the last week or so, overtaken the challenger in 4 out of 5 polls (with the 5th being a tie). In fact, Rehberg hadn’t won a poll since the Mason-Dixon poll on September 19. And Mason-Dixon, if you serf around Nate Silver’s site, seems to have a habit of giving strong advantages to Republicans (their last poll in Florida gave Romney a 6 point advantage, for instance).

So why hadn’t Nate suggested that Tester was more likely to win? 

This is the Nate Silver blog post I want to see. I’m not sure if he was off in any of the other Senate races. But I would love a frank discussion of why he was wrong here.

Montana Senate results NY Times

Montana Senate results – NY Times

As you can see, Romney won handily, as Nate predicted. And Montana, if you look at the results, might be an understandably strange place. The rural-urban divide exists, sort of. Some of Obama’s strongest counties were Big Horn and Deer Lodge counties, each with under 5000 votes and, obviously, no great number of college students. On the other side we do have counties called Petroleum County and Carbon County (so much for a the liberal ‘green’ vote). Gallatin County, home of one of our two major universities, went to Romney, 51% to 46%. Obama only won two “urban” areas: ultra-hippy-liberal (my college town) Missoula and the Irish-Catholic-drunk-mining town (the town that most people in Helena, even the drunk Irish Catholics, refer to as “Butt”) Butte (Silver Bow County).

So why would Montanans, who voted overwhelmingly for Romney, support a Democrat for Senate?  Patricia Cohen of the NY Times puts it fairly well:

When he won a Senate seat in 2006, Jon Tester, a Democrat and organic farmer, squeaked past his Republican competitor in this mostly Republican state in one of the closest races in the country.

He learned Wednesday morning that he had done it again, defeating Denny Rehberg, a Republican who currently holds the state’s only House seat.

Mr. Tester is a distinctly Montana brand of Democrat, and as such is often at odds with the mainstream of the national party. He is against same-sex marriage — though he stops short of calling for a constitutional ban — and favors strengthening gun rights.

Mr. Rehberg, who aligned himself with Tea Party values after his sixth re-election, in 2010, highlighted aisle-crossing moments this time. Thus, in a twist, the Montana Republican Party recently ran televised ads touting Mr. Rehberg’s votes against the budget plan proposed by Representative Paul D. Ryan, the Republican vice-presidential nominee, presumably to attract independent — and even Democratic — voters.

Simply put: Montana is weird. An anti-Gay marriage Democrat against an anti-Paul Ryan Republican. And that goes for all of our elected officials. Our last Governor (a Democrat) had a Republican running mate. That same Governor, if I remember correctly, admitted to enjoying a beer or two, while driving, around his farm – yet still finally, in 2005, made drinking while driving illegal in Montana.

Yes, you may remember Montana as the state that “didn’t have a speed limit” (the limit was in fact “reasonable and prudent,” whatever that means) and at the same time it was legal to have a beer while driving. Of course to drive at an unreasonable speed was still illegal, as was driving while drunk, but you can see where people might see Montana (even recently) as the “Wild West.”

So perhaps all of this weirdness made Nate’s head spin a bit too much. But our history of electing eccentric Democrats should have tipped him off – that and the polling…

But, as I said, I’m a novice here and Nate Silver is the undisputed heavyweight. I’m happy I discovered him well before the election and slept soundly as a result. But I’m very curious about where and why he went wrong, specifically in Montana. It would seem to be somewhere in his formula for the “State Fundamentals” (which have Rehberg up 8.5%) but just where, and how, is a mystery.

  • Matthew Harris

    It could be argued that Nate Silver didn’t “get it wrong” at all.

    One of the problems is that people see “30%” and think “0%” and see “70%” and see “100%”. If I took a fair dice out, and said that there was a 1-in-3 chance that it would come up a 1 or a 2, that would be a mathematically true statement. If I roll the dice and it comes up 1 or 2, that doesn’t mean that my statement wasn’t true. Similarly, if you look at Tester’s situation (incumbent Senator in a state who is a liberal compared to the politics of that state), every three elections with those circumstances, the incumbent liberal Senator is going to win against his challenger. Tester was just the 1-in-3 that did.
    That is the mathematical answer.
    The non-mathematical answer is that Silver’s model only works when it has lots of good polling. In the month before election day, there had been, I believe, 5 polls in Montana? And even if they all showed Tester ahead by small margins, that wasn’t enough evidence of a Tester lead to overturn the basic political lean of Montana. So maybe the model needs to be adjusted for that, but it seems that Silver was busy enough this year with the Presidential race that he didn’t spend much time working on his Senatorial model.

    • Justin Whitaker

      Thanks for the input, Matthew. The mathematical part is simple enough for me. It is obviously the non-mathematical side that I’d like to hear more about. As I said, Montana is a bit weird in politics, so it’s difficult to talk about any ‘political lean’ (at the state-level at least).

      In the last 40 years, MT has gone to only one Democrat in the Presidential elections and that was Clinton in 1992 (I’m sure you’ll find that Perot was of particular influence in MT then). But at the state level we’ve had Max Baucus (D) for over 30 years, Brian Schweitzer (D) as Governor for the last 8 years and a fairly even mix of D/R over the last 40 years, and the already discussed Jon Tester (D).

      So, going back to the math – our polls, especially the Public Policy Polling, which Mr Silver holds in high regard, show a consistent lean toward Tester. Why didn’t that math lead to a 66-34% in *favor* of Tester? Again, I’m a novice at this – but just looking at the Massachusetts polling, which was much more mixed in the final week, I’m amazed that Silver was able to project at 94% probability for Warren (who did go on to win). Granted, as you suggest, Massachusetts did have many more polls than MT over the course of the year, so perhaps that helped; but my sense is that polls going back over a month are of much less consequence than those in the final days.

  • Jay

    Enjoyed the post, very enlightening on Montana.

  • Interested

    Ah, it seems that people still don’t understand what Nate does. He assesses probability, which is not the same as a prediction. And in the Montana senate race, he did give Tester a one in 3 chance of winning based upon his assessment of the polls. If I held out a cloth bag, containing one blue ball and two red balls, told you to put your hand in and pull out one at random, there is a one in three probability that you would pull out the blue one. And if you did pull out the blue one, would you ask why that happened?

    • Justin Whitaker

      What I’m asking is why, given the math (polls, and all that) that suggest that there are in fact two blue ball and one red ball (or something close to that) in the bag – did Silver suggest the opposite?

  • Aiden

    First of he have Tester a one-third chance of winning, which is actually a reasonably good odds. Basically you bet me a dollar I will give you three if he win, but bet me a dollar Rehberg wins I will only give you $1.50. His final forecast for Romney was $0.09 would win you a $1.

    Montana was also sparsely polled, 15 polls in the last year. Strangely though he might have put too much weight on his “state fundamentals” and not enough on the polls given that last poll to show Rehberg with a lead was back in September.

    Interestingly enough the only senate seat he called worse was North Dakota which only had 7 polls, all showing strongly Republican. Now we get to where else I think he might have gone wrong. Nate’s model includes correlation between state clusters. I assume that Montana, given it is right next door to North Dakota, have similar demographics and also being physically adjacent means that they are highly correlated in the model. So the strong Republican polling in North Dakota may have leaned his prediction to Montana towards the Republicans more than it would have if he had left them uncorrelated. There was no PPP polls done in North Dakota, only Rasmussen and Mason-Dixon. Rasmussen is well known to have a Republican lean.

    In short I think Nate was trying to read Montana from North Dakota polls, when he should have been doing it the other way around.

    • Justin Whitaker

      Aiden, great point re: North Dakota. But looking at that data, I’m much more sympathetic with Silver. Crunching the ND polling numbers, which leaned strongly for the Republican, I too would have given strong odds in favor of the Republican. In ND, the polls didn’t reflect the actual voter turnout. Given the paucity of data and the Republican lean of at least 2 of those polls, I might have called it a toos-up or only slightly in favor of the Republican.

      But in MT, the polls (the recent ones at least) consistently suggested a Tester win, and as such – and with my novice understanding of Montana’s weird politics and “State Fundamentals” built on living there for 30+ years – I would have given Tester at least a better-than-50% chance of winning.

      But I think you’re right to draw ND into the picture – as it might be that Silver’s “State Fundamentals” in both were seriously flawed.

  • http://www.endlesslycreatingmyself.com Emily

    I like Tester. I voted for him. He sure beats the competition.

    But being against gay marriage, especially when your son is gay, marks you as a far less visionary politician than I’d like in office.

  • Pingback: The Election's Biggest Winner: Arithmetic - Forbes

  • Bryan

    Enjoyed the post and your perspective as someone born in the ‘hippie’ town (Missoula) and raised in the drunk-irish-catholic town (Butte).

  • Steph

    Mr. Silver was even more wrong about North Dakota’s Senate race, where Heitkamp was given about a 10% chance of winning.

  • CharleyCarp

    Late entrant, just wanted to remind you of the demographics in Big Horn, Blaine, Glacier, and Roosevelt counties. I know you know, but doubt your out-of-state readers see it as immediately as we do when they look at the map.

  • Sarah R

    You called it right on the head, Nate Silver got Montana wrong because he didn’t trust the polls because of his bias in thinking that Montana is a typically “red” state and hence the 8.5 state fundamental points. But he neglected to account for the fact that we have voted 2 democrat senators and a democrat governor. Montana is weird, and it is more subtle than just awarding these 8.5 republican points. He would have called it correctly if he didn’t have this bias. I also trusted Nate with full faith and was rather upset about him predicting a Tester demise. I wrote a editorial in the Kalispell Daily Interlake about how awesome Tester was in responding to me by his office personally calling me when I wrote a letter. I also volunteered in VA for the Obama campaign on election day and it was there that a friend of mine, when I mentioned that I was upset at the chances against Tester told me about these arbitrary 8.5 points and that he thought Nate should have just gone with the polls this time.

  • colleen browne

    Why would anyone listen to a person so obviously infantile that he would use a silly and inaccurate comment about the most Democratic city in the state? The only people who refer to Butte as butt are imbeciles like him. I cannot recall taking anyone seriously who would engage in such nonsense.


CLOSE | X

HIDE | X