It happens all the time

Under the subject line, “Here is another one for your archive,” someone points me to a news article and writes:

What would have happened had the guy not discovered his coding error? Or what if he had, but the results were essentially unchanged? My guess if that nothing would happen until someone got the data and tested the robustness of the statistical procedures used to produce the published results.

My reply: Oh, yes, I’ve blogged this one. What with the lag, I don’t think it has appeared yet. When I wrote the post, there had been no retraction but I still expressed skepticism for the usual reasons. Then I added a P.S. to discuss the retraction.

I purposely am not giving the details here because I want to make the larger point that pre-publication review doesn’t mean much. Even post-publication review doesn’t mean much. Review is an ongoing process. Yes, sometimes we can find the smoking gun that says, Don’t trust this guy!, but usually it’s not so simple—and we shouldn’t expect it to be so simple. To put it another way, don’t trust stuff just cos it’s been published and promoted and no obvious problems have been found—yet.