“Repeating the experiment” as general advice on data collection


Izzy Kates points to the above excerpt from Introductory Statistics, by Neil Weiss, 9th edition, and points out:

Nowhere is repeating the experiment mentioned. This isn’t the only time this mistake is made.

Good point! We don’t mention replication as a statistical method in our books either! Even when we talk about the replication crisis, and the concern that certain inferences won’t replicate on new data, we don’t really present replication as a data-collection strategy. Part of this is that in social sciences such as economics and political science, it’s rarely possible to do a direct replication—the closest example would be when we have a time series of polls, but in that case we’re typically interested in changes over time, so these polls are replications of methods but with possible changes in the underlying thing being measured.

I agree with Kates that if you’re going to give advice in a statistics book about data collection, random sampling, random assignment of treatments, etc., you should also talk about repeating the entire experiment. The problem is nothing special to Weiss’s book. I don’t know that I’ve ever seen a statistics textbook recommend repeating the experiment as a general method, in the same way they recommend random sampling, random assignment, etc.

Remember the 50 Shades of Gray story, in which a team of researchers had a seemingly strong experimental finding but then decided to perform a replication, which gave a null result, making them realize how much they were able to fake themselves out with forking paths.

Or, for a cautionary tale, the 64 Shades of Gray story, in which a different research team didn’t check their experimental work with a replication, thus resulting in the publication of some pretty ridiculous claims.

So, my advice to researchers is: If you can replicate your study, do so. Better to find the mistakes yourself than to waste everybody else’s time.

P.S. We’re still making final edits on Regression and Other Stories. So I guess I should add something there.