Judith Rich Harris on the garden of forking paths

Ethan Ludwin-Peery writes:

I finally got around to reading The Nurture Assumption and I was surprised to find Judith Rich Harris quite lucidly describing the garden of forking paths / p-hacking on pages 17 and 18 of the book. The edition I have is from 2009, so it predates most of the discussion of these topics, and for all I know this section was in the first edition as well. I’ve never heard this mentioned about JRH before, and I thought you might be interested.

Here’s the passage from Harris’s book:

It is unusual for a socialization study to have as many as 374 subjects. On the other hand, most socialization studies gather a good deal more data from their subjects than we did in our IQ-and-books study: there are usually several measurements of the home environment and several measurements of each child. It’s a bit more work but well worth the trouble. If we collect, say, five different measurements of each home and five different measurements of the child’s intelligence, we can pair them up in twenty-five ways, yielding twenty-five possible correlations. Just by chance alone, it is likely that one or two of them will be statistically significant. What, none of them are? Never fear, all is not lost: we can split up the data and look again, just as we did in our broccoli study. Looking separately at girls and boys immediately doubles the number of correlations, giving us fifty possibilities for success instead of just twenty-five. Looking separately at fathers and mothers is also worth a try. “Divide and conquer” is my name for this method. It works like buying lottery tickets: buy twice as many and you have twice as many chances to win.

And that’s not even the whole story, as she hasn’t even brought up choices in data coding and exclusion, and choices in how to analyze the data.

I replied that we’ve been aware forever of the problem of multiple comparisons but we didn’t realize how huge a problem it was in practice, and Ludwin-Peery replied:

Indeed! The most surprising thing was that she seems to have been aware of how widespread it was (at least in socialization research).