How to think about “medical reversals”?

Bill Harris points to this press release, “Almost 400 medical practices found ineffective in analysis of 3,000 studies,” and asks:

The intent seems good; does the process seem good, too? For one thing, there is patient variation, and RCTs seem focused on medians or means. Right tails can be significant.

This seems related to the last email I sent you (“What if your side wins?”).

From the abstract of the research article, by Diana Herrera-Perez, Alyson Haslam, Tyler Crain, Jennifer Gill, Catherine Livingston, Victoria Kaestner, Michael Hayes, Dan Morgan, and Adam Cifu:

Through an analysis of more than 3000 randomized controlled trials (RCTs) published in three leading medical journals (the Journal of the American Medical Association, the Lancet, and the New England Journal of Medicine), we have identified 396 medical reversals.

I’m not sure what to think about this! I’m sympathetic to the aims and conclusions of this article, but I can see there can be problems with the details.

In particular, what qualifies as a “medical reversal”? From the linked article:

Low-value medical practices are medical practices that are either ineffective or that cost more than other options but only offer similar effectiveness . . . Medical reversals are a subset of low-value medical practices and are defined as practices that have been found, through randomized controlled trials, to be no better than a prior or lesser standard of care. . . .

The challenge comes in when making this judgment from data. I fear that pulling out conclusions from the published literature will lead to the judgment being made based on statistical significance, and that doesn’t seem quite right. On the other hand, you have to start somewhere, and there’s a big medical literature to look through: we wouldn’t want to abandon all that and start from scratch. So I’m not quite sure what to think.