BMJ FAIL: The system is broken. (Some reflections on bad research, scientism, the importance of description, and the challenge of negativity)


It’s not the British Medical Journal’s “fault” that they published a bad paper. I mean, sure, yeah, it’s 100% their fault, but you can’t fault a journal for publishing the occasional dud. And there’s not really a mechanism for retracting a paper that’s just seriously flawed, if no fraud is suspected.

So the system is broken. Not in allowing bad papers to be published—there’s no way to avoid that—but in not having a way to say, Whoops! We messed up there.

For that matter, I don’t think the Journal of Personality and Social Psychology ever published a “this was a screw-up” notice regarding that ESP paper from 2011.

Four themes

This post ties together four recent themes:

– Bad research

– Scientism or the fallacy of measurement

– The importance of descriptive science

– The challenge of negativity

The bad research is this article from the British Medical Journal making strong claims based on models like this:

Scientism or the fallacy of measurement arises when researchers take a particular measurement or statistical procedure and simply assert it as their goal, as in this recent article published by the U.S. National Academy of Sciences:

Evidence for bisexual orientation requires that Minimum Arousal have an inverted U-shaped distribution and that Absolute Arousal Difference be U-shaped.

The importance of descriptive science is that it can be valuable to look at raw data and perform statistical analyses without feeling the need to jump to big generic statements.

Finally, the challenge of negativity is that critical work is often held to a higher standard than original claims. This might make sense in terms of the process of science—we should be open to speculation, and new ideas are often wrong but this is often how we can move forward—but what it means in practice is that if you’re cheerleading, you can be all kinds of sloppy and this is accepted by the science establishment, but if you’re critical you have to be extra careful. It’s the research incumbency rule.

The story is that the British Medical Journal published a bad article. That happens: journals publish bad articles all the time. I’m sure I’ve published some bad articles myself on occasion! I posted a comment, and the authors posted a content-free reply. This is standard operating procedure in all of science—indeed, we’ve been trained to give minimal, unthinking responses to criticism.

But it makes me sad. These researchers—their careers aren’t over! They’ll have lots more chances to do more work in their jobs. Why not take this opportunity to learn something from their mistakes? The problem they’re working on is important, so it’s important to try to get things right.

That’s bad research, but that example also featured scientism, in that the research paper used statistical methods (in this case, meta-analysis) without careful thought of the quality of the data or the details of the model they were fitting.

It would be as if you tried to build a bus by throwing together an engine, some gears, wheels, some seats, a steering wheel, etc., but without a real plan of how they fit together.

This relates to the importance of descriptive science: don’t go making strong claims from analysis of data that you haven’t fully described or understood.

And then, the challenge of negativity. Suppose I were to try to publish a paper in the British Medical Journal saying that their recently published paper was worthless. That would take a lot of work! Not really worth the effort, probably. It’s great that the British Medical Journal has a place for online comment, but it’s too bad that, for those readers who don’t look really carefully, the published article will stand on its own.

There’s no easy answer here. It’s not like the journal should retract their article just because a couple people say it’s wrong, and it’s not like the journal editors could go in and reread the article and decide that it’s a hot mess and decide to pull it from their journal. That way lies madness. So I have no great solution, beyond more open and prominent post-publication review.

The aforementioned BMJ paper publishes its prepublication reviews, but in this case, as in so many others, the peers missed some important problems. I guess it’s possible that the article was first sent to JAMA, NEJM, and Lancet and the reviewers at those journals saw through the problems with the article, so it ended up at BMJ. But I guess it doesn’t matter.

You know that joke:

Q: What do you call the guy who finished last in his class at medical school?

A: Doctor.

Here’s another:

Q: What do you call a paper that has serious flaws but finally made its way through the peer review process, somewhere?

A: Published.

It’s a problem.

Was the system always broken?

But none of this is new. Scientific journals have always published papers of varying quality, including some that were flat-out wrong in the sense of not offering the claimed evidence (recall evidence vs. truth). And it’s not like they were ever removing bad papers from their records. Actually, now that everything’s online, it’s easier than ever to remove or fix mistakes. We’re not seeing this now, but we weren’t seeing it before either. So maybe all that’s going on is we’re noticing it more?

There’s a temptation to say the system isn’t broken. Even if you don’t want to go full “Harvard” and say “the replication rate in psychology is quite high—indeed, it is statistically indistinguishable from 100%” (that quote’s like an outtake from a low-budget remake of The Social Network), you might say that science is self-correcting, and mistakes get corrected. And maybe that’s true—but it seems like a serious flaw in the system that there’s a mistake out there, and we all know it’s a mistake, but it’s gonna be sitting there in the literature, wasting people’s time, forever.

Again . . .

I have no animus toward this journal, its editors, or the authors of the article under discussion. We all make mistakes, and there are hundreds of examples I could’ve picked to make this point. I just had to pick one example, that’s all.