Resolving the cathedral/bazaar problem in coronavirus research (and science more generally): Could we follow the model of genetics research (as suggested by some psychology researchers)?

The other day I wrote about the challenge in addressing the pandemic—a worldwide science/engineering problem—using our existing science and engineering infrastructure, which is some mix of government labs and regulatory agencies, private mega-companies, smaller companies, university researchers, and media entities and rich people who can direct attention and resources.

The current system might be the best option available, but one problem is that we see lots of small, messy studies—experiments with small samples and no control groups, surveys with questionable sampling plans and incomplete writeups, a need to balance evidence across similar but not quite comparable studies, and a general unavailability of data.

This all sounds a lot like various subfields of cargo-cult science: the world of ESP studies, embodied cognition, anything-and-sex-ratio, air rage, himmicanes, pizzagate, etc., the PNAS or Psychological Science world in which everything gets publicity, careful and sloppy work alike, where everything’s a conceptual replication, where failures never are acknowledged and battles are fought at land, at sea, in the air, and in the news media.

I’m not saying coronavirus research is junk science. Not at all. Some of it may be bad science, but it all seems to be addressing real questions.

What I’m saying is that coronavirus research, in all its varying levels of quality, is being developed and transmitted in the same sociological matrix that also handles junk science. A system that thrived on the PNAS-NPR-Ted-Gladwell circuit is not being adapted to a much more serious purpose. And, again, even if the current decentralized system driven by career ambitions is the best we can do, it’s clearly flawed, as we already knew.

In my earlier post I referred to the classic dichotomy of “the cathedral”—centralized, closed-source development—versus “the bazaar,” which is decentralized and open source. But, as we discussed in comments, the analogy is imperfect, as one notable flaw of the current “bazaar” is that it’s not open source. People keep coming out with papers and not releasing their data or even their code.

Relevant to all this is a new article, “Psychological Science is Not Yet a Crisis-Ready Discipline,” by Hans IJzerman, Neil Lewis, Netta Weinstein, Lisa DeBruine, Stuart Ritchie, Simine Vazire, Patrick Forscher, Richard Morey, James Ivory, Farid Anvari, Andrew Przybylski, who write:

The threat presented by the new coronavirus disease (henceforth COVID-19) has mobilised public health practitioners, epidemiologists, and policymakers. It has also motivated scientists across a range of disciplines, including psychology, to ask whether their expertise might facilitate effective pandemic responses . . . That said, if we are to empirically understand and intervene in global pandemics, and steer policy in decisions where human lives hang in the balance, humility, caution, and realism are warranted. . . .

Before application can occur, we must first establish systematic guidelines for flagging trustworthy and actionable research findings. . . .

Much of their article is about psychology research and is not so relevant to work in biomedicine or epidemiology, two fields which, whatever their flaws, have questions and research methods that are more settled than those in much of psychology. For example, there has been lots of discussion recently about false-positive antibody tests and variation in the infection fatality ratio, but nobody doubts that antibodies, infections, and fatalities exist (in comparison to, say, implicit bias testing, embodied cognition, or terror management theory, three subfields of social psychology where there is doubt not just on certain empirical findings but of the entire theoretical structure). This is not a slam on psychology: it just means that psychology is a very difficult field to study. Biomedicine and epidemiology are easier.

Anyway, right now I’m not so interested in the particular problems of psychology addressed by IJzerman et al., but I did notice this quote in their paper:

As a start, we should look towards allied fields that have grappled with scaling their evidence to learn from their expertise and experience.

The field of genetics started from a similar position with small, independently collected samples that produced unreliable findings. Attempts to identify candidate genes for many constructs of interest kept stalling . . . In one prominent example, 52 patients provided genetic material for an analysis of the relationship between the 5-HTT gene and major depression, a finding that spurred enormous interest in the biological mechanisms underlying depression. Unfortunately, as with the current situation in psychology, these early results were contradicted by failed replication studies.

Technological advances in genotyping unlocked different approaches for geneticists. Instead of working in isolated teams, geneticists pooled resources via consortium studies and thereby accelerated scientific progress and quality. Their recent studies (with, e.g., N>1,000,000) dwarf previous candidate gene studies in terms of sample size. To accomplish this, geneticists devoted considerable time to developing research workflows, data harmonization systems, and processes that increased the accuracy of their measurements.

This seems very relevant to our problems right now with coronavirus research: lots of small studies, no coordination, no data sharing, scientific papers being unleashed on the world and then fighting each other in social media and the news media like some sort of killer robot drones going at each other . . .

But if the genetics researchers could get together and solve this problem, maybe the biomedical researchers and epidemiologists (and all the non-epidemiologists who are doing bad epidemiology research which then gets unfairly blamed on epidemiologists) could get together too in some way to develop research workflows, data harmonization systems, and processes that increase the accuracy of their measurements.

I don’t think it’s about to happen, but maybe it’s a model for the future?

I don’t think genetics research is perfect—in particular, I have the impression that they may have “harmonized” on some problematic statistical methods, and any standardization has tradeoffs—still, maybe what they do in genetics could represent a way forward for public health research, which is now such a mess.