Different challenges in replication in biomedical vs. social sciences

Summary

– In biological sciences, it might be reasonable to expect real effects to replicate, but carrying out the measurement required to study this replication is difficult for technical reasons.

– In social sciences, it might be straightforward to replicate the data collection, but effects of interest could vary so much by context that replication could be difficult.

This is all interesting because it has nothing to do with p-values, forking paths, or statistical analysis. It’s all about the difficulty of measurement and variation in underlying effects: two topics that are typically ignored entirely in statistics textbooks and courses.

Background

And here’s how this came up:

Kleber Neves, who works with the Brazilian Reproducibility Initiative, writes:

We briefly discussed how the technical expertise required to perform the experiments is an aspect that differentiates biomedical and social sciences.

On that note, I thought you might be interested in this paper I just stumbled upon by David Peterson. It is a compared ethnography of a molecular biology lab with a psych lab, whose main point is about the very distinction we discussed.

I replied that I wonder whether things will change now that biologists can demonstrate their procedures on Youtube. On the other hand, there must be some biologists who view their lab techniques as trade secrets and don’t want their work easily replicated…

And Neves responded:

Yes, there’s actually a journal for recorded experiments, which works as a mild incentive for sharing your trade secrets. I’d say Youtube is common when the technique is new and nobody is physically available with the expertise. That said, I think most of this motor learning still happens through an apprenticeship of sorts (someone more experienced goes to the bench with you to teach, closely following you until you’re good at it).

What I find curious is that it ends up adding a new forking path. It goes something like this: “well, this experiment did not give a significant result, but maybe you don’t have a good ‘hand’ for the technique, it’s probably why your controls were not as they should be. Do it again until you get it right”.