A colleague pointed me to this post from political analyst Nate Silver:
At the risk of restarting the MRP [multilevel regression and poststratification] wars: For the last 3 models I’ve designed (midterms, primaries, now revisiting stuff for the general) trying to impute how a state will vote based on its demographics & polls of voters in other states is only a mediocrely accurate method.
It’s a decent stand-in when you have few polls and weak priors. Our models do use it a little bit of it. But generally speaking, looking directly at the polls in a state is quite a bit more accurate. And often, so are simpler “fundamentals” methods (e.g. national polls + PVI).
Part of the issue is that seemingly demographically similar voters in different states may not actually vote all that similarly. e.g. a 46-year-old Hispanic man in California probably has different views on average than a 46-year-old Hispanic man in Idaho or Florida…
That’s partly because there are likely some unobserved characteristics (maybe the voter in Florida is Cuban-American and the one in California is Mexican-American) but also because states have different political cultures and are subject to different levels of campaign activity.
MRP does partial pooling. If you are interested in estimating public opinion in the 50 states, you’ll want to use all available information, including state polls and also national polls. To do this you’ll need a model of time trends in national and state opinions. That’s what Elliott, Merlin, and I do here. We don’t use MRP (except to the extent that the individual polls use MRP to get their topline estimates), but if we had the raw data from all the polls, and we had the time to devote to the project, we would do MRP on the raw data.
Because MRP does partial pooling of different sources of information, it should not do any worse than any one piece of this information.
So when Nate writes that “looking directly at the polls in a state is quite a bit more accurate” than MRP, I’m pretty sure that he’s just doing a crappy version of MRP.
A crappy version of any method can do really bad, I’ll grant you that.
The most useful step at this point would be for Nate to share his MRP code and then maybe someone could take a look and see what he’s doing wrong. Statistics is hard. I’ve made lots of mistakes myself (for example, here’s the story an embarrassing example in polling analysis from a few years back).
There indeed are examples where MRP won’t help much, if the separate estimates from each state have low bias and low uncertainty. But in that case MRP will just do very little partial pooling; it should not perform worse than the separate estimates.
Similarly for the fundamentals-based models that Nate mentions. Our forecast partially pools toward the fundamentals-based models. More generally, if the fundamentals-based models predict well, that should just help MRP. But you do have to include that information in the model: MRP, like any statistical method, is only as good as the information that goes into it.
Cooperation, not competition
I feel like the problem is that that Nate is seeing different methods as competing rather than cooperating. From his perspective, it’s MRP or state polls or fundamentals. But it doesn’t have to be one or the other. We can use all the information!
To put it another way: choosing a statistical method or a data source is not like an election where you have to pick the Democrat or the Republican. You can combine information and partially pool. The details of how to do this can get complicated, and it’s easy enough to get it wrong—that’s been my experience. But if you’re careful, you can put the information together and get more out of it than you’d learn from any single source. That’s why people go to the trouble of doing MRP in the first place.
I’m not saying that Nate should go use MRP
There are many roads to Rome. What’s important about a statistical method is not what it does with the data, so much as what data it uses. Nate’s done lots of good work over the years, and if he can manage to use all the relevant information in a reasonable way, then he can get good answers, whether or not he’s using a formal Bayesian adjustment procedure such as MRP.
Indeed, I’d rather that Nate use the method he’s comfortable with and do it well, than use some crappy version of MRP that gives bad answers. I do think there are advantages of using MRP—at some point, doing multiple adjustments by hand is like juggling plates, and you start to have issues with impedance matching—but it also can be hard to start from scratch. So I can accept Nate’s argument that MRP, as he has tried to implement it, has problems. The point of this post is just to clear up misunderstandings that might arise. If you do MRP right, you should do better than any of the individual data sources.
People are different. Your model should account for that.
Nate correctly writes that, “a 46-year-old Hispanic man in California probably has different views on average than a 46-year-old Hispanic man in Idaho or Florida.” Yup. Part of this is that the average political positions in California, Idaho, and Florida differ. In a basic MRP model with no interactions, the assumption would not be that middle-aged Hispanics vote the same on average in each state. Rather, the assumption would be that the average difference (on the logistic scale) between Hispanics and non-Hispanic whites would be the same in each state, after adjusting for other demographics such as age, education, and sex that might be included in the model. That said, in real life the average difference in attitudes, comparing Hispanics to non-Hispanic whites with the same demographics in the same state, will vary by state, and if you think this is important you can (indeed should) include it in your MRP model.
The relevant point here is that you should be able to directly feed your substantive understanding of voters into the construction of your MRP model. And that’s how it should be.
P.S. Zad sends in the above photo showing us what’s really inside the black box.