This came up in discussion the other day. In statistics and medicine, we’re generally told to rely when possible on statistically significant results from randomized trials. But, as we know, statistical significance has all sorts of problems, most notably that it ignores questions of cost and benefit, and it doesn’t play well with uncertainty. Hence my post, evidence-based medicine eats itself.

In comments, Nick wrote:

Ok, so it is not easy, but small incremental gains can get you a long way.

The amelioration of symptoms and prognosis of almost every common disease has improved since I [Nick] started clinical medicine in 1987; progress built on very many RCTs, none of them perfect but together forming a tapestry of overlapping evidential strands that can be read.

This made me wonder: Would this benefit have occurred without randomized clinical trials (RCTs), just by clinicians and researchers trying different things and publishing their qualitative findings? I have no idea (by which I really mean I have no idea, not that I’m saying that RCTs have no value).

There are famous examples of mistakes being made from poorly-adjusted observational studies (see for example here), and where this bias disappears in a randomized controlled trial. But my question is not, Can randomized clinical trials work well in particular examples? or, Are there examples where nonrandomized comparisons can be misleading? or, Can randomized trials be analyzed better? My question is, If randomized clinical trials were never done at all, would we be worse off than we are now, in terms of medical outcomes?

I think it’s possible that the answer to this question is No, that the damage done by statistical-significance thinking is greater than the benefits of controlled studies.

I have no idea how to estimate where we would be under that counterfactual, but perhaps it’s a useful thought experiment.