This study could be just fine, or not. Maybe I’ll believe it if there’s an independent preregistered replication.

David Allison sent along this article, Sexually arousing ads induce sex-specific financial decisions in hungry individuals, by
Tobias Otterbringa and Yael Sela, and asked whether I buy it.

I replied that maybe I’ll believe it if there’s an independent preregistered replication. I’ve just seen too many of these sort of things to ever believe them at first sight.

Allison responded:

My intuition agrees with yours. The thing that initially caught my eye was a study with high human interest appeal of an evolutionary psychology finding of the type that some have described as “just so stories.” I too have published such ‘just so stories’ and many people (including me) are drawn to them, but lately questions about their robustness and replicability have been raised. For a recent example, see here.

The second thing that caught my eye was that the key findings involve a high-order interaction. Higher order interactions of course can be real, and even prespecified, but when a finding is, by definition, so “conditionally dependent” it raises the question of whether this is perhaps a chance finding due in part, to the often inherent multiple testing involved in subgrouping analyses and higher order interactions. In addition, interaction tests often have low power which tends to increase the false positive rate under reasonable assumptions.

In a further look at the paper, I see no statement that the study was pre-registered. I also note that there seem to be post hoc data analytic decisions made which, as you have described in your paper on “a garden of forking paths” may also lead to non-replicable findings. There is no statement that the assignment of subjects to conditions was random. An ad hoc measure of hunger was used when there are standard pre-existing measures available (e.g., here) and better still, it would have been relatively easy to randomize subjects to simply skip breakfast and lunch for a day or not and one would have been a more valid approach for assessing the causal effects of hunger.

When all of these factors are put together, it raises skepticism. None of these factors mean that the finding is wrong or that the study is not cool, interesting, and well executed and honestly reported, but it does support the intuition that the result has a low subjective probability of replication.