In the scenario [Broockman and Kalla’s paper] depicts, “youth turnout” appears to jump implausibly by 11 percentage points — from 43% in 2016 to 54% in 2020 — beating its all-time modern record of 48% (in 2008). . . .
The authors get this idea from the fact that a certain category of young people — mainly Democrats and independents in the 18-34 age bracket — currently boost Sanders’ relative standing in the polls by saying they would turn out to vote for Sanders but would not turn out for a moderate Democrat.
According to the authors, these Bernie-or-Bust respondents represent about 11 percent of all non-Republican 18-34-year-olds, which means that if they all turned out to vote for Sanders, the turnout rate for the “non-Republican 18-34-year-old” demographic would be 11 percentage points higher than it would be if Joe Biden were the nominee and they all stayed home.
At some point in the authors’ thought process they appear to have confused this counterfactual difference in turnout — the difference between a world where Sanders wins the nomination and a world where Biden does — with a sequential increase in turnout that occurs over some historical time period — say, from 2016 to 2020. . . .
A Sanders nomination in 2020 could result in a youth turnout rate 11 points higher than what Biden, counterfactually, would have gotten, without any need for Sanders’ turnout to be 11 points higher than the 2016 level. For example, hypothetical Sanders could get a 49% turnout rate, with hypothetical Biden getting 38%. Or he could get 47% with Biden getting 36%.
In no sense would Sanders have to attain any particular level of turnout — let alone the 54.4% rate drawn in the chart — for his turnout rate to be 11 points higher than what Biden would have gotten had he been the nominee. . . .
In addition, about one-fifth of the pro-Sanders respondents the authors counted in the numerator are spuriously included, because they did not actually indicate in the authors’ survey that they would stay home if a moderate Democrat were nominated. (They said they were undecided or would vote third party.) The authors themselves explain this problem in the caption of their Figure 5, but inexplicably use the 11% figure anyway. . . .
Even with zero increase in youth turnout, Sanders still beats Biden. . . . When they re-weight their matchup polls to simulate 2016 turnout patterns — thus ruling out any youth turnout differential between the candidates, by assumption — they still find Sanders outperforming Biden: Sanders’ margin over Trump is +0.8%; Biden’s is +0.2% (Fig. 6, p.12). Despite the authors’ rhetoric about Sanders being dependent on an implausible youth turnout surge, their own analysis finds that he isn’t. . . .
Broockman and Kalla wrote a response:
The critique misrepresents the analyses our paper conducted and seems to misunderstand how our paper draws its conclusions. . . . At the same time, given that three of the five candidates we examined in our paper have dropped out since we wrote it, we also agree it is reasonable for observers of the Democratic primary to pay renewed attention to a finding in our paper about Joe Biden specifically, although we think the right interpretation of our data is that it is statistically ambiguous with respect to Biden . . .
In our paper (p. 16), the following text leads up to and describes what Figure 8 shows: “It is beyond the scope of this short paper—and indeed likely impossible—to determine whether nominating Sanders would in fact stimulate an 11pp or larger increase in turnout (or decrease in third party voting) among Democrats and Independents under 35 than would otherwise occur if another Democrat were nominated. It may well be that nominating Sanders would increase turnout among this group some. However, other results help contextualize how large a turnout increase of a full 11pp among under 35 Democrats and Independents is.” . . . that 11pp estimate represents the counterfactual increase the survey toplines implicitly credit Sanders with relative to what would otherwise occur if a different Democrat were nominated. Second, that the purpose of Figure 8 is not to calculate that “overall youth turnout must be 54.4% for Sanders to win” but instead, quoting our paper, “[to] help contextualize how large a turnout increase of a full 11pp among under 35 Democrats and Independents is”. . . .
We can see why, without reading the text of the paper describing what the graph is intended to show and how we constructed it, one might interpret this graph as indicating something else. . . . This makes the same mistake of treating Figure 8 as if it is what we use to reach our estimate instead of what we use to visualize the estimate we already reached. . . . It is true that an 11pp turnout surge among non-Republican youth is a smaller absolute number of votes than an 11pp turnout surge among all youth; but nowhere do we make the mistake of equating the two. Again, an 11pp surge among non-Republican youth is what Sanders’ topline poll numbers appear to be banking on; and, as we note in the paper and above, the purpose of this graph is to help readers contextualize what an 11pp turnout increase (among a given demographic) looks like by placing it in the context of historical variation in overall youth turnout. . . .
The 9.2pp estimate [suggested by Ackerman] is not obviously “correct” as it involves another assumption that is also plain to see in Figure 5. Blank ballots (what a literal interpretation of “undecided” would mean given how these responses are counted in polls) are relatively rare in Presidential elections, so we think it is reasonable to discount the 4.7pp decrease in these responses among left-leaning young people with Sanders present, and think of them instead as reflecting a need for increased participation. . . . A central message in our paper is that Sanders’ electability numbers are dependent on whether “Bernie or bust” respondents actually behave this way on election day. . . .
Our paper did not discuss specific comparisons of Sanders to Biden; it compared Sanders to an average of the three moderate candidates in our data: Bloomberg, Buttigieg, and Biden (Klobuchar was not included). . . . There is a lot more statistical uncertainty when throwing out the data on Buttigieg and Bloomberg (who have since dropped out) — and so we think the main take-away should be that our data is somewhat ambiguous about Biden specifically. . . . We think it is reasonable to expect Biden to be fairly similar to the other moderates and for differences between them to be largely due to noise (sampling variability), a suspicion that appears to be confirmed in other public polls as we discuss below. On the other hand, our data also can’t reject that he is equal to Sanders. In other words, both Biden and Sanders supporters could look at our data and reasonably say “it doesn’t rule out my view of things” . . . we think the main takeaway of our paper for discussions of electability at this point in the primary should not be our somewhat noisy results for Sanders vs. Biden. Rather, it should be that analysts of other polls, which appear to find an edge for Biden on average, should check whether assumptions about changes in youth participation underpin their conclusions about Sanders’ performance. . . .
There’s a lot more detail, both in Ackerman’s post and in Broockman and Kalla’s response. As I wrote in my earlier post, the data are generally consistent with the baseline assumption that the more moderate candidate would do better in the general election. But, as Ackerman and Broockman/Kalla each point out, there’s enough variation from election to election that we can’t conclude too much from any general pattern. I expect that in 2016, Martin O’Malley or Amy Klobuchar or various other people would’ve won the election, had they been the Democratic nominee. I’m guessing that Sanders would’ve received fewer votes than
Overall, I’m in agreement with Broockman and Kalla in the above debate. I don’t think Broockman and Kalla were confused about counterfactuals, and I think they were working carefully with the data. The bigger picture is that there’s only so much we can learn from head-to-head general election polls at this point. Even in this era of polarization, you can’t completely trust what voters say, this early in the election. Broockman and Kalla were not saying that the polls showed Sanders was less electable than Biden; what they were doing was pointing out weaknesses in the argument claiming that polls showed Sanders being more electable than Biden. There are reasons to think that Sanders is less electable than Biden, and there are reasons to think that Biden is less electable than Sanders. Broockman and Kalla are pointing out difficulties in naive interpretation of the head-to-head polls.
In Ackerman’s defense, I can understand his annoyance with headlines such as “Sanders . . . would need a youth turnout miracle” or “New Survey Suggests Bernie Can Win Only With Enormous Youth Turnout.” Broockman and Kalla’s study never suggested that Bernie could win only with enormous youth turnout. Ackerman’s mistake, perhaps, was to assume that their study made that claim, and then go backward looking to see what went wrong. Once we see the result as a mapping from data and assumptions to predictions, the whole thing makes more sense. That said, I have not looked at these claims with great care. As of now, what I’m saying is that Broockman and Kalla’s reasoning made sense to me. You can feel free to read all the above-linked documents and fill in the steps yourself.