Causal inference, adjusting for 300 pre-treatment predictors

Linda Seebach points to this post by Scott Alexander and writes:

A recent paper on increased risk of death from all causes (huge sample size) found none; it controlled for some 300 cofounders. Much previous research, also with large (though much smaller) sample sizes found very large increased risk, but used under 20 confounders.

This somehow reminds me of what happens with polygenic scores. Each additional confounder reduces the effect, and by the time you get to 300+ it’s all gone.

Anyway, it looked like something you might find worth commenting on.

From Alexander’s post:

For years, we’ve been warning patients that their sleeping pills could kill them. How? In every way possible. People taking sleeping pills not only have higher all-cause mortality. They have higher mortality from every individual cause studied. . . . Even if you take sleeping pills only a few nights per year, your chance of dying double or triple. . . .

When these studies first came out, doctors were understandably skeptical. . . . The natural explanation was that the studies were confounded. People who have lots of problems in their lives are more stressed. Stress makes it harder to sleep at night. People who can’t sleep at night get sleeping pills. Therefore, sleeping pill users have more problems, for every kind of problem you can think of. When problems get bad enough, they kill you. This is why sleeping pill users are more likely to die of everything.

He continues:

This is a reasonable and reassuring explanation. But people tried to do studies to test it, and the studies kept finding that sleeping pills increased mortality even when adjusted for confounders. . . .

And he goes through a few large studies from 2012 onward that estimate that sleeping pills are associated with a large increased risks of illness and death, after adjusting for age, sex, and various physical and mental health risk factors (which I’ll assume were measured before the study began, hence the “pre-treatment” in the title above).

And then Alexander comes to a relatively new study, from 2017:

They do the same kind of analysis as the other studies, using a New Jersey Medicare database to follow 4,182,305 benzodiazepine users and 35,626,849 non-users for nine years. But unlike the other studies, they find minimal to zero difference in mortality risk between users and non-users. Why the difference? . . .

They adjusted for three hundred confounders.

This [says Alexander] is a totally unreasonable number of confounders to adjust for. I’ve never seen any other study do anything even close. Most other papers in this area have adjusted for ten or twenty confounders. Kripke’s study adjusted for age, sex, ethnicity, marital status, BMI, alcohol use, smoking, and twelve diseases. Adjusting for nineteen things is impressive. It’s the sort of thing you do when you really want to cover your bases. Adjusting for 300 different confounders is totally above and beyond what anyone would normally consider.

It’s funny that Alexander says this. I’ve never adjusted for hundreds of confounders myself, nor have I done the equivalent in sample surveys and adjusted for hundreds of poststratification variables.

But I think Jennifer has adjusted for hundreds of confounders in some real problems, so maybe she’d have some comments on this.

You certainly should be able to adjust for 300 pre-treatment variables in a causal analysis. After all, if you only adjust for the first 10 variables on your list, you’re also adjusting for the other 290, just in a super-regularized way setting all those adjustments to zero. More generally, we can used regularized regression / machine learning to adjust for lots and lots of predictors and their interactions. I think Jennifer is going to put this in Advanced Regression and Multilevel Models (the successor to Regression and Other Stories).

The results of any particular analysis will be sensitive to model specification so I’ll make no general claim on what to believe in one of these studies. Maybe someone who works in this area can take a look.

P.S. I really like that Alexander said “adjust” rather than “control.” I think he understands a lot more about statistics than I do about psychiatry.