Florida. Comparing Economist and Fivethirtyeight forecasts.

Here’s our current forecast for Florida:

We’re forecasting 52.6% of the two-party vote for Biden, with a 95% predictive interval of approx [47.0%, 58.2%], thus an approx standard error of 2.8 percentage points. The 50% interval from the normal distribution is mean +/- 2/3 s.e., thus approx [50.7%, 54.5%]. Yes, I know these predictive distributions are not exactly normal, but it’s close enough for our purposes here.

What about the forecast at fivethirtyeight.com, which we discussed a couple weeks ago? Here’s where they are today:

They’re forecasting 51% of the two-party vote for Biden, with an 80% interval of approx [46%, 56%]. The 80% interval for the normal distribution is mean +/- 1.28 s.e., so the standard error from the fivethirtyeight Florida forecast is about 3.9 percentage points. This gives them a 50% interval of approx [48.4%, 53.6%] and a 95% interval of approx [43.2%, 58.8%].

Alternatively, we can go with fivethirtyeight’s uncertainty visualization:

The 50% prediction interval for Florida is roughly (+4% Trump, +8% Biden), i.e. (0.48%, 0.54%) for Biden’s two-party vote share. Taking the middle 95% of these dots gives something like [+16% Trump, +20% Biden], which corresponds to an approximate 95% interval of [42%, 60%] for Biden.

Comparing the two forecasts

The Economist and Fivethirtheight forecasts differ in their point predictions for Florida. I assume that the big difference here is that we’re using different fundamentals-based predictions, so we’re partially pooling toward a prior of Biden getting 54% of the national vote, while their prior is something more like 52%.

The biggest difference, though, is in the uncertainties. Our forecast standard error is 2.8 percentage points; theirs is 3.9 percentage points. That’s 40% higher!

I was going to say that this increased uncertainty comes from the Fivethirtyeight model assuming that 2020 is a particularly unpredictable election, but I guess not, because here’s Nate saying, “this year’s uncertainty is about average, which means that the historical accuracy of polls in past campaigns is a reasonably good guide to how accurate they are this year.” So now I’m really not sure where that big uncertainty in the Florida prediction is coming from. Maybe their between-state correlation in forecast errors is too low, so they need artificially high uncertainties in the individual states to get the right national error?

The difference between the uncertainties in the Economist and Fivethirtyeight really shows up in the 95% intervals: ours is [47%, 58%]; theirs is [42%, 60%]. Their interval seems way too wide to me: do they really think Biden might only get 42% of the two-party vote share in Florida?

It can be hard to visualize 19-1 odds, though, so we could also compare 50% intervals. Ours is [50.7%, 54.5%]; theirs is [48.4%, 53.6%]. Again, these are all approximate, based on reading numbers off some graphs.

I find our intervals more plausible. That is, I think it reasonable to apply 25% probability to the event “Biden gets less than 50.7% of the two-party vote in Florida this year,” 25% probability to “Biden gets more than 54.5%,” and 50% to the in-between. In contrast, I’d assign quite a bit less than 50% probability to the event “Biden gets less than 48.4% or more than 53.6% of the two-party vote in Florida this year.”

You might disagree. That’s fine. I guess you could bet on some of these numbers; as I’ve discussed earlier (also here), I don’t take the prediction markets so seriously.

Making predictions, seeing where they look implausible, and using this to improve our modeling

My point here is not to come up with bets. Rather, my point is to use these probabilistic predictions to get a sense of how the models work. This is central to our workflow, to make all sorts of inferences and predictions from a fitted model and focus in on what looks wrong. This was the sort of reasoning that motivated us to fix our model last month. Open code is great, but it also helps to look at some specific predictions.

Looking at specifics has worked for me before. After the 2008 election I made some maps (posted on the Fivethirtyeight site, actually) using poll data to estimate how different people would vote. The next day, I got slammed by blogger and political activist Kos for some implausible estimates from my maps. And Kos was right! It was good for me that I plotted a bunch of maps—not just, say, some regression coefficients—because that gave enough information for an outsider to catch problems with what I was doing. After a couple months of retooling, I posted some improved maps based on a better model. (Since then we’ve done more; see for example here.)

One of the advantages of the Bayesian approach is that it allows granular predictions of . . . everything in your model. These predictions can be a great way to understand how your model works and diagnose problems with it. This is a strong point for the Fivethirtyeight forecast as well as our Economist model: we’re all using predictions as tools for model understanding and model criticism.

Bounds of applicability

One challenge when interpreting these forecasts is that they don’t represent all possible outcomes. Everything we’re doing is in terms of the share of the two-party vote, so, if there’s any serious third-party challenge, all bets are off—or, maybe I should say, our bets are off. Also, both forecasts are framed as Biden vs. Trump, so if either candidate dies or is incapacitated or is otherwise removed from the ballot before the election, it’s not quite clear how to interpret the models’ probabilities. I mean, sure, we could just take them to represent Dem vs. Rep, and that probably wouldn’t be so far off, but this possibility does not seem to be included in either model.

None of this is such a big deal when considering 50% intervals, but when we start talking about 95% intervals, we need to be careful about what’s being conditioned on.

And what if the votes never get counted because of a mega-version of the 2000 Florida dispute over vote counting? Again, it’s not clear how this would be handled in a hypothetical bet, but I guess that some adjustment would be necessary. Elliott Morris estimates that we should reduce Biden’s win probability (that’s win probability, not vote share) by about 7% if we want to include higher-than-usual ballot rejection rates in the model. I’d say that the existing models do implicitly account for the possibility of small disputes (such as those 30,000 votes that were never counted for Gore in Florida) but nothing huge such as 20% of the ballots in the state getting lost or ruled invalid.

Other unlikely outcomes, though, are implicitly in the model. Outbreaks of wars, stock market crashes, a new pandemic, hurricanes, a cure for cancer, shark attacks, shocking revelations about lax email security, etc., they seem to belong more in the “error terms” of the models.

As is often the case in statistics, there’s no sharp line on what is being conditioned on in these forecasts. That’s just the way things are, and we just have to be open about it.