Kevin Lewis points us to this article by Paige Shaffer et al., “Gambling Research and Funding Biases,” which reports, “Gambling industry funded studies were no more likely than studies not funded by the gambling industry to report either confirmed, partially confirmed, or rejected hypotheses.”
The paradox is that this particular study was itself funded by the gambling industry! So we’re in a kind of infinite regress here.
Anyway, for the purpose of this post I’m not interested in gambling or conflicts of interest. Rather, my interest was triggered by the phrase, “confirmed, partially confirmed, or rejected hypotheses.”
In social science, I think we should move away from the idea that we have hypotheses that are confirmed or not. My concern here is similar to my issues with null hypothesis statistics testing:
1. Hypotheses in social science are vague enough that it’s just about never clear what it would mean for a hypothesis to be “confirmed” or “rejected.” There are no true zeroes, and even when it comes to estimation, the size and direction of comparisons can vary across people and scenarios.
2. Conflation of rejection in a null hypothesis significance test with rejection of a substantive hypothesis.
3. Rejection of null hypothesis A taken as support for, or confirmation of, favored alternative hypothesis B.
4. Dichotomization—or, one might say, premature dichotomization—throwing away information at all stages of a study, from design and data collection through coding and data analysis.
I think there are a few more issues here that I’m forgetting. The key point here is that the problems of null hypothesis significance testing arise not just with original studies but also with replications, meta-analyses, literature reviews, and the like.