Probabilistic forecasts cause general misunderstanding. What to do about this?

The above image, taken from a site at the University of Virginia, illustrates a problem with political punditry: There’s a demand for predictions, and there’s no shortage of outlets promising a “crystal ball” or some other sort of certainty.

Along these lines, Elliott Morris points us to this very reasonable post, “Poll-Based Election Forecasts Will Always Struggle With Uncertainty,” by Natalie Jackson, who writes:

Humans generally do not like uncertainty. We like to think we can predict the future. That is why it is tempting to boil elections down to a simple set of numbers: The probability that Donald Trump or Joe Biden will win the election. Polls are a readily available, plentiful data source, and because we know that poll numbers correlate strongly with election outcomes as the election nears, it is enticing to use polls to create a model that estimates those probabilities.

Jackson concludes that “marketing probabilistic poll-based forecasts to the general public is at best a disservice to the audience, and at worst could impact voter turnout and outcomes.”

This is a concern to Elliott, Merlin, and me, given that we have a probabilistic poll-based forecast of the election! We’ve been concerned about election forecast uncertainty, but that hasn’t led us to take our forecast down.

Jackson continues:

[W]e do not really know how to measure all the different sources of uncertainty in any given poll. That’s particularly true of election polls that are trying to survey a population — the voters in a future election — that does not yet exist. Moreover, the sources of uncertainty shift with changes in polling methods. . . . In short, polling error is generally larger than the reported margin of error. . . . Perhaps the biggest source of unmeasurable error in election polls is identifying “likely voters,” the process by which pollsters try to figure out who will vote. The population of voters in the future election simply does not yet exist to be sampled, which means any approximation will come with unknown, unmeasurable (until after the election) errors.

We do account for nonsampling error in our model, so I’m not so worried about us understating polling uncertainty in general terms. But I do agree that ultimately we’re relying on pollster’s decisions.

Jackson also discusses one of my favorite topics, the challenge of communicating uncertainty:

Most people don’t have a solid understanding of how probability works, and the models are thoroughly inaccessible for those not trained in statistics, no matter how hard writers try to explain it. . . . It is little wonder that research shows people are more likely to overestimate the certainty of an election outcome when given a probability than when shown poll results.

That last link is to a paper by Westwood, Messing, and Lelkes that got some pushback a few months ago when it appeared on the internet, with one well-known pundit saying (mistakenly, in my opinion) “none of the evidence in the paper supports their claims. It shouldn’t have been published.”

I looked into all this in March and wrote a long post on the paper and the criticisms of it. Westwood et al. made two claims:

1. If you give people a probabilistic forecast of the election, they will, on average, forecast a vote margin that is much more extreme than is reasonable.

2. Reporting probabilistic forecasts can depress voter turnout.

The evidence for point 1 seemed very strong. The evidence for point 2 was not so clear. But point 1 is important enough on its own.

Here’s what I wrote in March:

Consider a hypothetical forecast of 52% +/- 2%, which is the way they were reporting the polls back when I was young. This would’ve been reported as 52% with a margin of error of 4 percentage points (the margin of error is 2 standard errors), thus a “statistical dead heat” or something like that. But convert this to a normal distribution, you’ll get an 84% probability of a (popular vote) win.

You see the issue? It’s simple mathematics. A forecast that’s 1 standard error away from a tie, thus not “statistically distinguishable” under usual rules, corresponds to a very high 84% probability. I think the problem is not merely one of perception; it’s more fundamental than that. Even someone with a perfect understanding of probability has to wrestle with this uncertainty.

As is often the case, communication problems are real problems; they’re not just cosmetic.

Even given all this, Elliott, Merlin, and I are keeping our forecast up. Why? Simplest answer is that news orgs are going to be making probabilistic forecasts anyway, so we want to do a good job by accounting for all those sources of polling error that Jackson discusses.

Just one thing

Jackson’s article is on a site with the url, “centerforpolitics.org/crystalball/”. The site is called “Sabato’s Crystal Ball.”

Don’t get me wrong: I think Jackson’s article is excellent. We publish our pieces where we can, and to publish an article does not imply an endorsement of the outlet where it appears. I just think it’s funny that a site called “Crystal Ball” decided to publish an article all about the problems with pundits overstating their certainty. In all seriousness, I suggest they take Jackson’s points to heart and rename their site.