Open forensic science, and some general comments on the problems of legalistic thinking when discussing open science

Jason Chin, Gianni Ribeiro, and Alicia Rairden write:

The mainstream sciences are experiencing a revolution of methodology. This revolution was inspired, in part, by the realization that a surprising number of findings in the bioscientific literature could not be replicated or reproduced by independent laboratories. In response, scientific norms and practices are rapidly moving towards openness. . . .

In this article, the authors suggest that open science reforms are distinctively suited to addressing the problems faced by forensic science. Openness comports with legal and criminal justice values, helping ensure expert forensic evidence is more reliable and susceptible to rational evaluation by the trier of fact. In short, open forensic science allows parties in legal proceedings to understand and assess the strength of the case against them, resulting in fairer outcomes. Moreover, several emerging open science initiatives allow for speedier and more collaborative research.

This all sounds good, but I imagine there will be a lot of resistance. The adversary nature of the legal system seems so much different from the collaborative spirit of open science. Indeed, one of the many problems I’ve seen in trying to promote open science is that it sometimes seems to have a kind of legalistic frameworks, focusing for example on arguments about p-values rather than on the substance of what is being studied. For example, when those ovulation-and-clothing researchers criticized my criticisms of their work, they didn’t address questions of measurement and effect size which, in my opinion, were central to understanding what went wrong with their study. One problem with a legalistic attitude toward science is that it can encourage the attitude that, if you just follow the protocols, you’ll do good science. Actually, though, honesty and transparency are not enough.

Related: The purported CSI effect and the retroactive precision fallacy.