I just posted this a few hours ago, but it’s such an important message that I’d like to post it again.
Actually, maybe we should just post nothing but the above graph every day, over and over again, for the next 20 years.
This is hugely important, one of the most important things we need to understand about statistics.
The top graph is what got published, the bottom graph is the preregistered replication from Joe Simmons and Leif Nelson.
Remember: Just cos you have statistical significance and a graph that shows a clear and consistent pattern, it doesn’t mean this pattern is real, in the sense of generalizing beyond your sample.
But the top graph looked like such strong evidence!
It’s so easy to get fooled. Think of all the studies with results that look like that top graph.