MRP Carmelo Anthony update . . . Trash-talking’s fine. But you gotta give details, or links, or something!

Before getting to the main post, let me just say that I’m a big fan of Nate Silver. Just for one example: I’m on record as saying that primary elections are hard to predict. So I don’t even try. But there’s lots of information out there: poll data, fundraising numbers, expert opinion, delegate selection rules, etc etc. I admire that Nate’s out there working on this. In academia we often work over and over on the problems we already know how to solve, squeezing out marginal improvements. That’s fine. But I admire someone like Nate who attacks all these problems, using whatever information and tools are available. That’s important.

OK, now on to the main post:

I really can’t figure out what’s going on here. Nate Silver is an admirable statistical analyst, full of good ideas, and then . . . .

OK, here’s the time line:

2009: Nate and I publishe an article with me in the New York Times using multilevel regression and poststratification (MRP):

2011: Nate says Carmelo Anthony is underrated:

2012-2016: Nate does poll aggregation (using published numbers from public polls), and my colleagues and I analyze raw data from opinion polls to get national and state-level averages. Nate’s using reported toplines so he doesn’t need to worry about MRP, survey weighting, etc.; we’re working with raw data so we use MRP to adjust for known differences between sample and population, and to get stable estimates for geographic and demographic subgroups.

Again, I can see why Nate doesn’t need MRP—he’s using the weighted estimates that survey organizations supply in their public summaries, and I guess that’s usually a reasonable choice. He’s a poll aggregator, and that’s what poll aggregators do: they aggregate polls.

And, again, here are some reasons why we do need MRP (or something like it):

– Estimating public opinion in slices of the population

– Improved analysis using the voter file

– Polling using the Xbox that outperformed conventional poll aggregates

– Changing our understanding of the role of nonresponse in polling swings

– Post-election analysis that’s a lot more trustworthy than exit polls

2018: Nate characterizes MRP as “the Carmelo Anthony of election forecasting methods.” At first that sounds cool—MRP is awesome, and Nate’s on record as saying that Carmelo is underrated—but, no, it’s not so cool because, apparently, sometime between 2011 and 2018, Nate’s decided that Carmelo ain’t so great, and sometime between 2009 and 2018, Nate’s decided that MRP ain’t so great.

In 2018, Nate described MRP as “not quite ‘hard’ data.” This is really weird. Nate knows a lot about data—and so do I! And I don’t know what the hell Nate is talking about. The data used in MRP are as “hard” as the data used in all the polls that Nate is reporting. The only difference is, he’s buying the weighted-estimation sausage pre-wrapped in the supermarket, and we’re back in the butcher shop preparing it all. (I’m just riffing off the line about sausage and legislation.) What does he think, that these these survey weightings just come out of nowhere? Nope! They’re full of subjective decisions. And see here for what happens if these decisions aren’t well thought out.

I think the old, pre-2018 Nate Silver would’ve thought it was really cool how we adjusted the survey data using party ID to get a more accurate result. What happened to the old Nate Silver, who wanted to use the most up-to-date statistical methods to get the best possible answers? He seems to have been replaced by a new, trash-talking Nate Silver who throws around terms like “hard data,” as if whatever summary report put out by survey organization XYZ using whatever adjustment they happen to use, is somehow on the “hard data” pedestal.

Look. Survey organizations are great. Surveying is hard work. But these published toplines? They’re made by humans. MRP estimates made by Yougov, or Langer Research Associates, or Microsoft Research, even Columbia University—these are hard data too, just as hard data as anything produced by exit polls, or the New York Times, or even ESPN.

2020: A colleague who wishes to remain anonymous points me to this new blast from Nate:

My objections to MRP are almost entirely *practical* not theoretical. There’s nothing wrong with smoothing or blending polling data with some other inputs in a clever way to produce more accurate forecasts. That’s what 538 has been doing for 13 years!

But you label it “MRP” and all of the sudden people who ought to know better think it’s a technique with God-like powers. Worse, they use it as an excuse to not be very transparent, in terms of not publishing raw inputs or detailed outputs or much in the way of methodology.

Nor have many of the MRP models I’ve seen spent much time working on estimating uncertainty, including the possibility of correlated errors, which is where often *most* of the work is in modeling if you want your models to remotely resemble real-world conditions.

In the very first MRP paper from 22 years ago we had correlated errors (that is, our model had variation at the regional as well as state levels).

I disagree that most of the work in survey adjustment is in correlated errors. Consider the notorious state polls in 2016 in Michigan, Wisconsin, and Minnesota. It’s our impression that the big problem there was not correlated errors but rather that these surveys did not adjust for education. And the other problem was poll aggregators—including me!—focusing on the aggregation and not looking carefully enough at where those numbers were coming from.

Back in 2016, both Nate and I should’ve done a bit more MRP and a bit less “smoothing or blending polling data with some other inputs in a clever way.”

Moving forward: I think it’s pretty clear that when Nate is criticizing MRP in polling, he’s not criticizing what I do, or what David Rothschild at Microsoft Research does, or what YouGov does, or what Gary Langer does, or what other respected pollsters do. And I don’t think he’s criticizing the research of Jeff Lax, Justin Phillips, or the other political scientists studying state and local opinion using MRP.

Specifically, Nate is criticizing “people who ought to know better” who “think it’s a technique with God-like powers.” Who are these people? I’d like to at least see some references of people who think MRP is “a technique with God-like powers.” What are the MRP models that Nate has seen that have not considered the possibility of correlated errors?

This is the goddamn internet, for chrissake. If you have a claim that you can support, then link to it already!!!!

If there are people out there “who ought to know better” who think MRP is “a technique with God-like powers,” then tell us who they are. If there people who use MRP “as an excuse to not be very transparent, in terms of not publishing raw inputs or detailed outputs or much in the way of methodology,” then tell us who they area. Give us some quotes, give us some links. And I’ll be first in line to explain how MRP can go wrong. Indeed, I’ve already done it!

I really don’t understand what’s going on here. Nate does good work. I’d be glad to join him in criticizing bad work, but it’s hard to do so when he never gives any examples.

Trash-talking’s fine. But you gotta give details. Otherwise you’re not trash-talking. You’re just trash.

P.S. Please again read the first paragraph of this post. I think Nate’s great. I don’t understand what’s his problem with MRP—a 22-year-old statistical method that’s still going strong—so it would be good to get to the bottom of this one.