How scientists perceive advancement of knowledge from conflicting review reports

Kevin Lewis pointed me to this article. It seemed kinda familiar, I took a look at the abstract, and I realized . . . I reviewed this article for the journal! Here was my referee report:

The paper seems fine to me. I have only two minor comments, both relating to the abstract.

1. I disagree with the “normative principles of statistical and scientific inference positing that new study findings can never reduce our level of knowledge.” I agree that knowledge should increase in expectation, but in any given case, can’t our level of knowledge decrease, just by bad luck? For example, suppose we’re comparing two treatments, and in fact A is slightly better than B. But we do an experiment with small sample, and it just happens that B outperforms A on the data at hand. Then our useful knowledge of the world has decreased.

2. The abstract says, “the majority of participants do not feel that ‘we know more’ as the result of the two new studies when the second study fails to replicate the first.” Looking at Figure 1, the percent who says “we know more” under the “opposite effect” condition is very close to 50% for all cases except for experiment 2. So I feel that the framing as “the majority of participants”, while literally correct, is misleading. Maybe replace “the majority of” with “half or more of”.

Based on the abstract to the published paper, it seems that they completely ignored my first suggestion and only minimally implemented my second suggestion.

On one hand, this is annoying: I went to the effort to write a review report, I gave two minimal comments, both of which were reasonable and easily implemented—but the authors still couldn’t be bothered to follow through.

On the other hand, maybe this is good news. Peer review is kinda horrible, so it’s heartening that the authors felt free to ignore reviewer comments and just write what they really felt, even though in this case it was kinda wrong and misleading. Gotta respect that sense of confidence.

In all seriousness, I think the paper is fine, setting aside these two minor issues.