It all started for me on 2 Jan when I received this email from Keith Donohue in Fargo, North Dakota:
I am a longtime reader, and I am curious about your reaction to an (in press) journal article that I recently came across. . . .
The paper is “Declines in Religiosity Predicted Increases in Violent Crime—But Not Among Countries with Relatively High Average IQ”, by Clark and colleagues, which is available on research gate and is in press at Psychological Science. Some of the authors have also written about this work in the Boston Globe. . . .
Psychology has a long and ignoble history of using dubious measures for intellectual ability to make general claims about differences in average intelligence between groups of people – racial/ethnic groups, immigrant groups, national groups, etc. Often, these claims have aligned with prevailing prejudices or supported frankly racist social policies. This history disturbs me, and seeing echoes of it a flagship journal for my field disturbs me more. I guess what I am trying to say is that my concerns about this paper go beyond its methodological issues, and I need to be honest about that.
With that in mind, I have tried to organize and highlight the methodological issues that might interest you or might be worth commenting on. . . .
I posted something on it that very day:
Someone pointed me to an over-the-top social science paper that is scheduled to be published soon. I then wasted 2 hours writing some combination of statistical commentary and rant.
I expect that, once the paper is published, there will be major controversy, as its empirical findings, such as they are, are yoked to political opinions which seem pretty much targeted to offend lots of people in its academic field.
Fortunately, by the time my post appears, the furor should have quieted. . . .
So, what happened? It wasn’t a matter of the furor quieting. There was no furor at all! Psychological Science just published the paper, and everyone was like, OK, sure, no prob. Just another This Week in Psychological Science.
Then on 31 May my scheduled post appeared:
The Association for Psychological Science (a “highly educated and intelligent group,” for sure) has decided that they should be humble about promoting the unique and relatively novel values that thrive among them. And these unique and relatively novel values include . . . “criminal justice reform and unrestricted sociosexuality.” If members of the Association for Psychological Science want criminal justice reform and unrestricted sociosexuality for themselves, that’s fine. Members of the APS should be able to steal a loaf of bread without fear of their hands getting cut off, and they should be able to fool around without fear of any other body parts getting cut off . . . but for groups with lower self-control and cognitive ability—I don’t know which groups these might be, you’ll just have to imagine—anyway, for those lesser breeds, no criminal justice reform or unrestricted sociosexuality for you. Gotta bring down the hammer—it’s for your own good! . . .
I also discussed the methodological problems of the paper:
Set aside for a moment problems with the data and statistical analysis, and suppose that the data show exactly what the authors claimed, that time trends in religious attendance correlate with time trends in homicide rates in low-IQ countries but not in high-IQ countries. Suppose that’s all as they say. How can you, from that pattern, draw the conclusion that “The prescriptive values of highly educated groups (such as secularism, but also libertarianism, criminal justice reform, and unrestricted sociosexuality, among others) may work for groups that are highly cognitively sophisticated and self-controlled, but they may be injurious to groups with lower self-control and cognitive ability”? You can’t. To make such a claim is not a gap in logic, it’s a chasm. Aristotle is spinning in his goddam grave, and Lewis Carroll, Georg Cantor, and Kurt Godel ain’t so happy either. This is story time run amok.
All this is beyond the serious data problems with the article.
I also wrote that I was surprised that Psychological Science would publish this paper, given its political content and given that academic psychology is pretty left-wing (Roy Baumeister aside) and consciously anti-racist.
It’s some combination of: (a) for the APS editors, support of the in-group is more important than political ideology, and Baumeister’s in the in-group, (b) nobody from the journal ever went to the trouble of reading the article from beginning to end (I know I didn’t enjoy the task!), (c) if they did read the paper, they’re too clueless to have understood its political implications.
A few days later, the cognitive dissonance of the Association for Psychological Science stepped up a notch when they issued this statement:
From its inception, the field of psychological science has studied the causes and harmful impacts of stereotypes, prejudice, and disparity. . . . our profession now has a sizable body of research that can be brought to bear to understand the persistent racism and subsequent confrontations we are witnessing in cities and communities across our nation and throughout the world. . . .
We stand ready to add our voice and expertise to bring about positive change and to stand against injustice and racism in all forms. . . .
As a diverse, international organization of more than 30,000 scientists and researchers around the world working to understand all aspects of mind and behavior, APS profoundly respects the essential worth of all people and cultures, regardless of race, gender, and ethnicity.
This seemed kinda wack, given that they’d just published a paper with a racial-essentialist perspective that slammed “criminal justice reform and unrestricted sociosexuality.” So much for standing against injustice and respecting the essential worth of all people!
Here was my take on how the Association for Psychological Science could help to bring about positive change:
As the statistics indicate, there are serious disparities in American society. But I’m not convinced that the profession of psychological science “has a sizable body of research that can be brought to bear to understand the persistent racism and subsequent confrontations we are witnessing in cities and communities across our nation and throughout the world.” I’m not convinced that the implicit association test and things like it have anything useful to add, and I’m not convinced that traditional racial explanations of social and economic disparities have anything to add.
I understand that, as individuals and as an organization, the members and leaders of the Association for Psychological Science would like to be helpful in these difficult times. As individuals, they can help in various ways: they can peacefully protest, they can join neighborhood watch groups to protect their communities, they can be good parents, friends, and neighbors, they can vote and write their congressmembers etc. They can be productive members of society and coach Little League and support local businesses, they can minimize their ecological footprint and bring a smile to work every day. They can give Ted talks, they can blog, they can even go on twitter. Lots of ways to make yourself useful. But, to paraphrase John Tukey, the combination of an archive of Psychological Science articles and an aching desire to be helpful does not ensure that reasonable help can be extracted from a given archive of articles.
I get that we all want to feel useful, but maybe promoting old journal articles isn’t actually useful at all.
I still haven’t heard from the Association for Psychological Science on this—maybe they feel bad for publishing lies about me a few years ago—but in the meantime the authors of the disputed article retracted it, on 17 June.
Here’s the story, and here’s what the authors wrote:
Over the past weeks, we have heard considerable concern and scrutiny over our recent article on religion, violence, and IQ in Psychological Science. In particular, papers by Karis et al. (2015; pointed out by Fearon and Eisner) and by Dickins et al. (2017; pointed out by Sear) have prompted us to spend much of the last week digging deeper into the research behind our measures—a level of vetting we should have done before submitting the paper. As a result, we no longer have confidence in our findings. Because of imputation for many countries, the homicide data have limitations that call our conclusions into question. The IQ data, however, have much more serious issues. The persistence of these highly questionable data sources in the psychological literature has convinced us that research with certain kinds of flaws should be pulled from the record as its existence steers us further from the truth rather than closer to it. We now believe our paper falls into that category and are retracting it immediately.
It’s good when people can admit their mistakes. In this case there was no need for them to refer to Karis, Fearon, Eisner, Dickins, and Sear, as these problems were pointed out by Keith Donohue in his email:
This data seems to be collections of IQ estimates made from proxies of intelligence, such as school achievement or scores on other tests of mental abilities. Based on my training and experience with intelligence testing, this research decision seems problematic. Also problematic is the decision to input missing values for some nations based on data from neighboring nations. . . .
My point here is not that Donohue is a better reference than these other sources but rather that the problems were well known to seemingly everyone who’d looked at these data—except for the four authors of this paper and the editorial board of Psychological Science.
I guess it’s tough if you’re editing a journal that’s published so much junk science. It kinda lowers your standards, and you’ll publish pretty much anything that’s headline-worthy and has a bunch of p-values in it.
The other message is that open post-publication review has once again outperformed secret pre-publication review. Remember, the problem with peer review is . . . the peers.
I wouldn’t ever want a paper of mine to be reviewed by a peer of Roy Baumeister, Robert Sternberg, or various other dukes and earls of the academic psychology establishment.
I don’t know what will happen next. The above-linked Retraction Watch article quotes the former editor of the journal as saying that the current editor “is working on how best to respond to criticisms” of that paper. I’m not sure why they need to respond, now that the paper has been retracted.
It’s pretty clear that all this fuss is about the article’s political content, not its data problems.
Here’s an interesting question: what if the journal article had the same inflammatory political content and the same breathtaking leaps of logic, but no data problems? What would Psychological Science do then? I’m not sure.
For example, suppose the four authors of that paper had put together some statistics on racial differences in standardized tests in the U.S. and then run some state-level analysis interacting with religiosity and ethnic composition? With U.S. data there aren’t such data problems as with the cross-national IQ data.
They could’ve supplemented this with some experiments on undergraduates and on Mechanical Turk participants, finding some differences—you can always find some differences—and then spun some story about how religion is needed to keep the masses under control. They could’ve still taken that bold stand against criminal justice reform and unrestricted sociosexuality.
I guess that such an article could’ve been accepted by the journal, for the same reasons they accepted the original paper. If so, then any retraction would not have been able to use the low quality of the international data as an excuse: they would’ve had to face up to the problems of story time. And that would be tough, given that the APS has gone all-in on story time (“We stand ready to add our voice and expertise to bring about positive change”) when coming from the left, if not the right.
One more thing
I’m glad that the authors stand by the principle of data quality, but there’s something about their retraction that bothered me.
In their note, they didn’t say anything about “criminal justice reform and unrestricted sociosexuality.” But, as I wrote in my earlier post, their conclusions about society do not follow from their statistical analysis, even setting aside any data problems. So, sure, their data were a mess, but even if their data were perfect, I don’t think they could draw all those conclusions in their paper.
But if Psychological Science were to retract all the papers they’ve published where the substantive conclusions don’t follow from the data analysis, they’d have to retract a lot of papers.
What to do?
I guess that, when see a published article, we have to judge it on its own merits rather than relying on information from the journal.
I mean, sure, sometimes you’ll get a cue such as “written by X and appearing in Psychological Science” or “published in PNAS and edited by Y” and you’ll know not to take it seriously, because it’s just pushing some political or professional agenda, but in general that’s not enough information, and you’ll have to actually take a look at the article or borrow the judgment of a trusted colleague or arbiter, or follow the hivemind of open post-publication review.
Even papers from Harvard published in Lancet get things right sometimes.
P.S. Zad sends the above image, titled “How to properly clean your data.” If only Team Baumeister were as competent as this cat, think of how much trouble would’ve been avoided!