“What if your side wins?”

Bill Harris writes:

Thanks for posting my question the other day.

Here’s another, somewhat related question.

What if “your side” wins? What if, starting today, every analysis is done properly? Null hypothesis significance testing is something you read about only in history of statistics books. When binary decisions are made, they are supported with real decision analyses, not NHST.

There’s a lot of current research based on NHST. What would you do with that?

Researchers, I presume, would feel free to redo any NHST work they thought was both important and realistic for them to work on.

But what about practitioners? For example, in medicine and healthcare, what do you propose doctors should do with drugs and procedures justified with NHST? What should patients who read about this transition in statistical practice do?

I suspect current research results are not all bad (I certainly hope that’s the case), but it seems that some certainly is. How do practitioners and patients discern the difference? Are there useful rules of thumb?

My response: Whatever or not is done by medical research in the future, the problem you describe is already here. The quick answer is that what’s important from each experiment are the design, measurement protocols, and raw data, not the published conclusions. To the extent that all we have are the conclusions and some incomplete descriptions of measurement and design, we kind of need to reconstruct what the data could have looked like. From a purely practical perspective, my answer to your question is that we’ll do what we always do, which is to trust the best judgment of doctors and authors of medical textbooks and review articles.