You don’t want a criminal journal… you want a criminal journal

“You don’t want a criminal lawyer… you want a criminal lawyer.” — Jesse Pinkman.

In what sense is it a “blood sport” to ask someone for their data?

That’s our question for the day. But it’ll take us a few paragraphs to get there.

1. A case of missing data

Jordan Anaya points us to this news article by Tom Bartlett, “The Criminologist Accused of Cooking the Books,” which wrote of “statistical irregularities in five published papers”:

These were more than mere rounding errors or minor oversights. These were giant red flags: percentages that couldn’t be true, standard deviations that didn’t add up, and a host of other issues that defy easy explanation. The numbers didn’t make sense.

The obvious next step in determining what went awry would be to reanalyze the data. That data is in the possession of Eric Stewart, a professor of criminology at Florida State University and the only co-author who appears on all five papers. It’s Stewart who could, presumably, clear up what happened. But so far he’s been less than forthcoming, refusing to share key data even with his co-authors. Meanwhile the information he has offered has only deepened the confusion. For instance, a recently published correction for one of the five papers is itself shot through with numerical oddities, leading to suspicions that the numbers don’t make sense because the numbers might be, at least in part, made-up. . . .

In emails and text messages sent to colleagues, Stewart has portrayed himself as the target of “data thugs” who are attempting to ruin his career. . . .

“Data thugs,” huh? I guess that’s like “methodological terrorists,” i.e., people who tell uncomfortable truths.

Bartlett continues with more details on disappearing data and researchers who don’t want to talk about it.

So far, so bad, it’s a Wansink-like story about some people getting caught with their data down. Maybe somewhere there were some data that told some story, but some messy things happened between data collection, analysis, and publication. And, now that the publications are out there, the authors don’t want to give anything up. No big deal, just another day in the world of Retraction Watch.

The story then continued in the way that one might imagine. Here’s Bartlett again:

[Justin] Pickett [a coauthor of one of the papers, who was concerned about the data problems and wanted to get to the bottom of things] had lost patience with Stewart. In June, he sent a letter to the editors of Criminology, the journal that had published their paper . . .

In short, the findings in the paper — the one that Stewart had so generously invited Pickett to be a co-author on at a crucial moment in his career — were not only meaningless but might well be fabricated.

The lead author of that paper is Brian Johnson, a professor of criminology and criminal justice at the University of Maryland and also a co-editor of Criminology. When Pickett first emailed to tell him that Stewart had seemingly included hundreds of duplicates in the data, Johnson realized this was bad news. “That’s a huge fucking mistake. Man, what a shitshow,” Johnson wrote in an email to Pickett. “Obviously we are going to have to retract the article or at a minimum update it with new data and findings.”

Cursing aside, it sounded like that the journal Criminology was gonna do the right thing.

2. Things go sour

But then came the backpedaling:

[Johnson] pushed back at suggestions that the paper might be partially based on fictional results: “I have absolutely no reason to doubt that the surveys are real and I do not think other people making accusations have any evidence of that either.”

Au contraire. As Bartlett discusses in his article, there was no evidence that one of these surveys was ever conducted, and the data from another one seemed to be in Wansink-level disarray.

And then:

[T]he editor in chief of Criminology, David McDowall, seemed less than eager about getting to the bottom of what might be wrong with this particular paper. He confirmed that he had seen the letter Pickett sent but says that he “didn’t read it in great depth.” As for the possibility of retraction, which Pickett has requested, McDowall was dubious about the concept. “I don’t even quite know what retraction is,” McDowall says. “I imagine that it could occur. I would think there would be legalistic implications.”

Ahhhh, now we’re getting somewhere . . . the kind of people who would make up data, they’re also the kind of people who would attack via frivolous lawsuits. This is the kind of selection bias whereby honest open people get pushed aside, while scammers get every benefit of the doubt. Everyone’s afraid the scammers will sue.

But it gets worse:

What seemed to disturb McDowall more than possible errors or fabrication was Pickett’s motivation in questioning the data in the first place. . . .

“It seems to me that it’s pretty hostile for Justin to start making these claims,” McDowall says. Pickett says “negative things” about Stewart, according to McDowall, although he couldn’t recall specific statements. . . .

The Javert paradox rears its ugly head! Call out misconduct and you get slammed for being a malcontent.

3. And now to our question

Bartlett continues:

“I think criminology is maybe a little behind other disciplines because we haven’t adopted the blood sport of ruining other people’s careers,” [McDowall] says.

Jeez, who are those picky people who want to ruin someone’s career, just cos they make up data. What next, are we going to retroactively fire A. J. Liebling from the New Yorker? Mark Twain made stuff up all the time, and nobody criticized him! J. K. Rowling too. Not to mention J. R. R. Tolkien. Did you think trolls and orcs were real??? So, sure, it’s ok to make up hobbits, quidditch, jumping frogs, all sorts of things . . . but you fabricate one little dataset, and all of a sudden people are out get you! Jeez.

In all seriousness, all these people ever asked were to see the damn data. In what sense is it a “blood sport” to ask someone for their data?

I am so so sick of this hyperbole. Terrorism, Stasi, bullying, blood sport, blah blah blah. For chrissake. They’re just asking this guy for his data.

4. And then it gets worse

Bartlett continues:

When the journal has run into similar issues, the editors have dealt with it in what McDowall believes is a more humane manner. “This is not the first time that papers were published in the journal that were complete gibberish,” he says.

Wha????

Did I hear that right???

I mean, sure, “This is not the first time that papers were published in the journal that were complete gibberish.” That’s true of lots of statistics journals too, I’m sure. In some ways, I admire the journal editor for this admission. Still, it seems to miss the point.

To draw a New York analogy . . . suppose you’re eating dinner at an expensive restaurant and you see a big fat cockroach crawl across your plate. You call the manager and he tells you to shut up: “Yeah, our kitchen is infested with roaches. So what? What’s your problem???”

I don’t think I’d go back to that restaurant again.

Sure, all restaurants have roaches. I accept that. But the manager’s supposed to feel a little bad about it, right?

After Bartlett’s story came out, the editors of Criminology issued a statement, which begins:

Recent events have brought our private deliberations as the Co-Editors of Criminology, the flagship journal of the American Society of Criminology, under public scrutiny.

“Private deliberations,” huh? You’re the ones who talked about “blood sports” and how you publish “gibberish.” If you wanted your deliberations to stay private, you didn’t have to go publicly attack your critics, dude.

They continue:

We believe the preferred approach is to employ the classical comment-and-reply model. Critics submit a comment to the journal, presenting their objections. The journal editors review the objections for importance and relevance, focusing on the potential contribution to scholarship. If the editors have doubts about the contents of the comment, they seek the advice of anonymous reviewers. If the editors accept the comment, they offer the original authors an opportunity to provide an equally detailed reply. Depending on the circumstances of the case and the nature of the reply, the editors may offer the author of the comment a final opportunity to respond.

This sounds like B.S. to me. “The classical comment-and-reply model,” indeed. That’s the “classical” model where the editors keep the gatekeeper role, which is what kept so many of these problems in the first place.

For some journals, maybe the comment-and-reply model works, but social science journals rarely run comments or corrections or letters to the editor with corrections.

See for example this story. In this case I noticed a major problem with a published article in the American Sociological Review. I sent a letter to the editor, who sent it to three reviewers, none of whom disagreed on the merits of my criticisms, but the editors declined to publish the letter because they judged it to not be in the top 10% of submissions to the journal. In this case there was no suggestion of research misconduct. The authors just made a mistake. So much for the classical comment-and-reply model.

Amusingly, the above-linked statement cites the American Sociological Review as an exemplar of the comment-and-reply model, even though in my own experience they couldn’t handle a comment—and it was a simple one, a pure statistical issue with no suggestion whatsoever of impropriety.

Here’s more on this story from James Heathers.

5. Criminal research

I’ve had some weird experiences in this field, associated with an article that Torbjørn Skardhamar, Mikko Aaltonen, and I wrote for the Journal of Quantitative Criminology which included the following passage:

We wanted to reanalyze the dataset of Nelson et al. However, when we asked them for the data, they said they would only share the data if we were willing to include them as coauthors. We did not want to do so, and so we found the papers ourselves.

A blog commenter asked what happened, and Skardhamar replied:

We did not include such a comment at first, but but then again: not using the original data needed an explanation, and that was how it was. (We could have asked yet again, of course, but I got the feeling it would not have changed the outcome).

So I guess I knew that there could be data issues. But I had no idea things could get as bad as described in Bartlett’s article.

6. The big picture

What really gets me about these editors is that they don’t act as if they care about the actual scientific claims being made. After all, if you really thought these claims were potentially important, you’d want their flaws revealed right away. Or if you thought the criticisms were themselves flawed, you’d like those flawed criticisms exposed. You’d be a hurry to get to the bottom of the story. Wanting to bury the criticisms or never publish them at all: that’s not the action of someone who gives a damn. Which is kinda sad for the editor of Criminology, given that criminology is actually an important applied subject. It’s not like he’s editing a journal in pure math, or art history, or something else of primarily intellectual interest.

It reminds me of something that someone, maybe Jordan Anaya, said about Brian Wansink, along the lines of: it’s sad, the critics know more about Wansink’s studies than Wansink himself does.

Kinda like the way that various big-shot social psychologists defend bad work: if the really cared about these claims, they’d be concerned about the quality of the theory and evidence. But if all they care about are (a) their friends getting good press and living stress-free lives, (b) a general feeling that academic psychology is a good thing, and (c) a belief that evolutionary psychology is true, then they’re much more motivated to shoot the messenger.

P.S. Just to be clear: I don’t have any direct information on whether anyone involved in the above story committed any crimes. What I think is that criminology is important, and it’s a scandal that the editors of the purportedly leading journal in the field don’t seem to care if they publish fake data, gibberish, etc. It was embarrassing for the field of statistics that one of our leading journals published that Bible Code paper, but that was back in 1994! And we’ve tried to clean up our act since then. The journal Criminology should clean up its act too, or it’s gonna get a reputation like Lancet or Psychological Science or PNAS, of being willing to publish just about any crap as long as it tells a good story.