A trolley problem for our times
OK, I couldn’t quite frame this one as a trolley problem—maybe those of you who are more philosophically adept than I am can do this—so I set it up as a cookie problem?
Here it is:
Suppose someone was to knock on your office door and use some mix of persuasion, fomo, and Harvard credentials to convince you to let them in so they can deliver some plates of handmade cookies. You tell everyone about this treat and you share it with your friends in the news media. Then a few days some people from an adjoining office smell something funny . . . and it seems that what that those charming visitors left on your desk was not delicious cookies, but actually was a steaming pile of poop!
What would you do?
If you’re the management of Lancet, the celebrated medical journal, then you might well just tell the world that nothing was wrong, sure, there was a minor smell, maybe a nugget or two of poop was in the mix, but really what was on your desk were scrumptious cookies that we should all continue to eat.
People have been sending me lots of emails regarding this recent Lancet study of chloro-hydro-oxy-spirograph or whatever it’s called.
For example, see the blog by Peter Ellis (“A peer-reviewed study that probably used fabricated data”):
The data in that study, and in at least one preprint on a second treatment, were provided by an Illinois firm called Surgisphere. Allegedly the data represents the treatment and health outcomes of 96,032 patients from 671 hospitals in six continents. However, there is simply no plausible way I [Ellis] can think of that the data are real. . . .
If Surgisphere can name the 671 participating hospitals or otherwise prove that the data is real I [Ellis] will retract that statement, delete this post or write whatever humbling apology they desire. . . .
Surgisphere has five employees with LinkedIn accounts. Other than the CE and co-author of the Lancet paper, these are a VP of Business Development and Strategy, a Vice President of Sales and Marketing, and two Science Editors (actually, one Science Editor and one Scoence Editor, which does not inspire confidence in their attention to detail while editing). LinkedIn also records one employee of QuartzClinical – a director of sales marketing. . . .
Surgisphere should be a billion dollar company if it’s done this 670 times, but it clearly is not. In fact, Dun and Bradstreet estimate its revenue at $45,245. You couldn’t even do the discovery stage of an EHR integration project at a single hospital for that, never mind deploy anything. . . .
And the news article by Catherine Offord (“Disputed Hydroxychloroquine Study Brings Scrutiny to Surgisphere”):
Surgisphere is currently headquartered in Palatine, Illinois, and run by Desai, who trained in vascular surgery, a subject on which he has published many scientific articles and books. Until February 10 of this year, Desai was employed by Northwest Community Hospital (NCH) in suburban Arlington Heights. He tells The Scientist that he resigned for family reasons.
Court records in Cook County, Illinois, show that Desai is named in three medical malpractice lawsuits filed in the second half of 2019. . . .
He also sent a comment purporting to be from Alan Loren, the executive vice president and chief medical officer of NCH: “Dr. Desai was employed at NCH and resigned in February 2020. We did not have any problems with him while he was here.”
Asked by The Scientist if he made this statement, Loren says, “What I can tell you is that he was employed here and he did resign. I can’t speak to whether or not there were any problems.” . . .
Reviews of the company’s products on Amazon are polarized, and a handful of positive reviews that appeared to impersonate actual physicians were removed when those doctors complained to Amazon. Kimberli S. Cox, a breast surgical oncologist based in Arizona, tells The Scientist that she was one of several practicing physicians who in 2008 discovered five-star reviews next to names that were identical or very similar to their own, that they had not written. She and her colleagues successfully persuaded Amazon to take the reviews down. . . .
This is all in addition to the statistical concerns from James Watson that we discussed last week. But it makes sense that the paper would have statistics problems, given that there’s no evidence that anyone on this team has any statistical expertise—and these particular adjustments would present a challenge even to an expert.
The most damning criticism
But the most general and comprehensive takedown (I want to call it “trenchant,” but I’ve never used the terms “trenchant,” “tendentious,” or “disingenuous” in my life, so I don’t want to start now) comes from from this guy who edits a medical journal in England. He took a look at this Surgisphere controversy and drew some important general conclusions:
The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”.
And he doesn’t just target the individual doctors involved in the studies. He also slams the medical journals:
Journal editors deserve their fair share of criticism too. We aid and abet the worst behaviours. . . . unhealthy competition . . . Our love of “significance” pollutes the literature with many a statistical fairy-tale.
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivised to be right. Instead, scientists are incentivised to be productive and innovative. . . . Instead of changing incentives, perhaps one could remove incentives altogether. . . . Or reward better pre and post publication peer review. . . .
Lots of ideas. The author concludes:
Those who have the power to act seem to think somebody else should act first. . . . The good news is that science is beginning to take some of its worst failings very seriously. The bad news is that nobody is ready to take the first step to clean up the system.
It’s too bad it took this latest Surgisphere scandal to get all these ideas out there.
If only . . .
If prominent medical journal editors had been thinking about all this five years ago, just think of how much good could’ve been done. Between, say, 11 Apr 2015 and today, there would’ve been time to open up the main medical journals and really improve scientific communication, in advance of this pandemic we’ve been dealing with.
OK, not all the journals, not all at once. But, if back in early 2015, the editor of at least one major medical journal had been on board, that would’ve been amazing.
I hope . . .
I hope the author of the above-linked article can get in touch with the editor of the Lancet as soon as possible so that his ideas can be implemented right away. It’s not too late! Sure, some of these ideas are radical, but I’ve heard that the government in England is looking for “weirdos and misfits with odd skills,” so no problem.
I hope that radical journal editor can persuade the Lancet editor to do something already!
It would be so easy for the Lancet editorial board to just publicly ask the authors of the Surgisphere paper to respond directly to their critics on pubpeer and other forums.
The Lancet editorial board could also require the authors release their data and code. If necessary, de-identified data. Or data summaries. Or a paper trail of the data. And the code could be released in any case.
If the authors refuse to respond seriously to criticism and supply code and some data, the editors could follow previous Lancet practice, as in the Wakefield vaccine episode, and just retract the paper 12 years from now. Actually, they don’t have to wait 12 years! Or, if they don’t want to retract, they could keep the paper up on their website and watermark it with some note like this:
Serious questions have been raised about the data and analysis in this paper. The authors have refused to share data and code or to respond to the post-publication review.
Instead, we get this (from the above-linked news article by Offord):
On May 29, Jessica Kleyn, a press officer at The Lancet journals, informed The Scientist in an emailed statement that the authors had corrected the Australian data in their paper and redone one of the tables in the supplementary information with raw data rather than the adjusted data Desai said had been shown before.
“The results and conclusions reported in the study remain unchanged,” Kleyn adds in the email. “The original full-text article will be updated on our website. The Lancet encourages scientific debate and will publish responses to the study, along with a response from the authors, in the journal in due course.”
The sad thing is, the authors of that paper don’t even work for Lancet! The Surgisphere team published a paper with questionable statistics and and questionable data in their journal. You’d think that, instead of coming to defense of the authors, they’d be annoyed! See the
trolley cookie problem at the top of this post.
The real scandal . . .
The real scandal of that Lancet/Surgisphere paper is not that the authors may have done a poor statistical analysis, thus making their causal claims empty at best and misleading at worst. Bad statistical analyses get done all the time, and they get published in top journals all the time. Statistics is hard, there aren’t enough competent statisticians to go around (and certainly not if you restrict your author list to M.D.’s), and otherwise-competent statisticians make mistakes too.
Nor is the real scandal that the data were misreported and may (or may not) be faked in some way. Cheats, fakers, and plain old sloppy researchers abound, and the fact that they’ve reached whatever status they have at top universities suggests that they’ll be around for awhile. Errors, sloppiness, and out-and-out fraud will continue to get in print. Recall Clarke’s law.
No, the real scandal is that the respected medical journal Lancet aids and abets in poor research practices by serving as a kind of shield for the authors of a questionable paper, by acting as if secret pre-publication review has more validity than open post-publication review.
This was a paper whose conclusions absolutely relied on complicated statistical adjustment, which is exactly the sort of problem where open data, open code, and post-publication
Check out this recent online exchange:
The author of the controversial Surgisphere paper applauds Lancet and its editor for “incisive reviews and bold decisions in publishing.”
But what did the “incisive reviews” say? Under what grounds were those questionable results published? We don’t know, and we’ll probably never now, because peer reviews are secret!
To put it another way, if it’s true that they “intensively review all COVID-19 research papers,” but we see that what gets published has serious problems . . . then this tells us something about the limitations of the intensive review process. And the behavior of the journal after the criticisms have been raised tells us something about the limitations of the system. Again, the author of this article should get in contact with the Lancet editorial board, toot sweet.
P.S. Zad shares the above photo, captioned “When someone finally shares their raw data and code.”
P.P.S. See here for a reminder of why I feel the need to waste time on this, rather than just spending my work time “doing science.”