NPR’s gonna NPR (special coronavirus junk science edition)

1. The news!

Zad’s cat, pictured above, is not impressed by this bit of cargo-cult science that two people sent to me:

No vaccine or effective treatment has yet been found for people suffering from COVID-19. Under the circumstances, a physician in Kansas City wonders whether prayer might make a difference, and he has launched a scientific study to find out. . . . “It has to be a true supernatural intervention,” says Dr. Dhanunjaya Lakkireddy.

Lakkireddy is the principal investigator in a clinical trial involving 1000 patients . . . The four-month study, launched on May 1, will investigate “the role of remote intercessory multi-denominational prayer on clinical outcomes in COVID-19 patients” . . . Lakkireddy has assembled a steering committee of medical professionals to oversee the study.

That makes sense. It’s not like medical professionals would be busy with anything else right now.

The NPR story continues:

The investigators will assess how long the patients remain on ventilators, how many suffer from organ failure, how quickly they are released from intensive care and how many die. . . .

Lakkireddy says he can not explain how people praying remotely for someone they don’t know (or a group of people,) could actually make a difference in their health outcomes . . . Lakkireddy says he has no idea what he will find. “But it’s not like we’re putting anyone at risk,” he says. “A miracle could happen. There’s always hope, right?”

OK, NPR got their wish. It’s all about clicks, right? The attention economy. And they got my attention. Instead of writing, say, on recommendations for experimental design for coronavirus studies, I’m posting on this. So they’ve already done some damage by running this story. Sure, you could blame me for this—shouldn’t I take responsibility for what I post on this blog? Not really, I reply. I’m a creature of endorphins just like the rest of you. NPR pushed my buttons, and here I go.

Actually, I’ve been showing a lot of self-control this week. Just the other day someone pointed me to a horrible new paper by some dude I’ve written a lot about in the past, and I was like, no, I’m not touching that one at all!

2. Intercessory prayer

Intercessory prayer, though, that’s an old one. Deb Nolan and I wrote about it in the first edition of our book, way back in 2002:

We were reading the Web-based magazine Salon one day and saw an article reporting on a study of the effectiveness of prayer on the health outcomes of 990 patients at a critical-care unit. The article continues:

But does it do any good? Everybody’s got an opinion but nobody knows for sure, because the faith-health dichotomy has never received much in the way of serious scientific scrutiny.

Until now. A massive study published in the Oct. 25 issue of the Archives of Internal Medicine (a journal of the American Medical Association) showed that heart patients who had someone praying for them suffered fewer complications than other patients.

. . .

Dr. Harold G. Koenig, director of Duke University’s Center for the Study of Religion/Spirituality and Health, has spent his entire professional life looking at how spirituality affects a person’s physical well-being. This particular study is significant, he says, “because it’s published in an AMA journal, it has a huge sample, and it shows significant results.”

The prayed-for patients had better results, on average, than the control patients, and the difference was significant at the 4% level. On the other hand, three tests were performed on the data, and one might be skeptical about the findings, given that neither the patients in the study nor their doctors knew about the prayers being conducted on their behalf. . . .

Back then, we thought of studies of implausible phenomena reporting “p less than 0.05” from multiple hypothesis tests were just an amusing oddity. In those days before Gregory Francis, Uri Simonsohn, etc., we had no idea how prevalent they were in much of the published scientific literature.

We’ve discussed intercessory prayer on the blog before. As I wrote a few years ago,

What amazed me about this study—beyond the philosophically untenable (to me) idea that God is unable to interfere with the randomization but will go to the trouble of improving the health of the prayed-for people by some small amount, just enough to assure publication—was the effort the researchers put in to diminish any possible treatment effect.

It’s reasonable to think that prayer could help people in many ways, for example it is comforting to know that your friends and family care enough about your health to pray for it. But in this study they chose people to pray who had no connection to the people prayed for—and the latter group were not even told of the intervention. The experiment was explicitly designed to remove all but supernatural effects, somewhat in the manner that a magician elaborately demonstrates that there are no hidden wires, nothing hidden in the sleeves, etc. Similarly with that famous embodied cognition study: the elderly words were slipped into the study so unobtrusively as to almost remove any chance they could have an effect.

See here for further discussion of this point.

3. NPR

So how does this appear on the national news? Just as they run fake-science stories about football the week before the Super Bowl, and fake-science stories about basketball during March Madness and the NBA playoffs, now’s the time to run fake-science stories about the coronavirus now. And it’s a human-interest story.

I guess NPR justifies it by some mixture of (1) lots of their listeners are religious so this sort of story shows them respect, (2) even if it doesn’t work, little harm is done, and (3) hey, maybe it really will work and this is a big story! It’s hard for me to argue against #1 and #2—after all, there are lots of news items on sports, Hollywood, etc., and we can think of junk science as just another form of entertainment or human-interest story. #3 bothers me, though. To think, “Hey, this might work,” is like thinking that that lottery ticket you just bought might win the hundred million dollar jackpot. As we’ve discussed elsewhere in the context of Pascal’s wager, once you accept the possibility that intercessory prayer might “work” and cure people, you have to accept the possibility that it could have a negative effect on survival.

For example, maybe you and your friends all pray really hard for Grandma, and God’s listening, sees all your love and decides to do her a favor, put her out of her misery, and send her to heaven right away! Your prayer worked. But it looks like a negative outcome in your study.

4. God’s mighty thumb on the scale

And that doesn’t even get into the possibility that God is screwing around with the randomization.

I’m not joking here. Conditional on a belief on a directly-intervening God, one of the most striking things is the lack of direct evidence. No photos of the guy, no videos of a burning bush, no fun and games with the stars in the night sky. Dude’s so boring, he just operates through the laws of physics. Shroud of Turin? C’mon—that’s about as convincing as those photos of the Loch Ness monster. This is not to say that God doesn’t exist—I have no idea—but just that, if he does exist, and he does have the habit of making supernatural interventions, he seems pretty careful about covering his tracks and never doing it in a way that leaves unambiguous evidence of supernatural activity. There’s always some alternative explanation.

So, now to the prayer study. If it’s direct prayer—get all the cousins to call Grandma on Skype and everyone prays together—then it’s no problem, God can intervene, and there’s an alternative explanation involving her mood, hormones, etc. But intercessory prayer—that’s another story. Suppose God really does want to help these people, but he wants to continue covering his tracks and keeping all evidence ambiguous. How can he do this? One way is to manipulate the random assignment! Too much unbalance on pre-treatment predictors—that can be detected (an all-knowing God will be aware of modern econometrics, balance tests, etc.)—but God can induce imbalance on the potential outcomes, and that’s fine, it’s undetectable in principle. So I anticipate that this new study will be a failure, just like all the others (although they might be able to p-hack their way into a publication, more publicity, hundreds of citations, a Ted talk or two, a book contract, an endorsement from Columbia University’s very own Dr. Oz, and maybe some funding from NIH’s office of alternative medicine)—but maybe it will only look like a failure. Maybe it will be a stunning success, manipulated into looking like no effect through the workings of a self-effacing God.

I guess this is why they never put statisticians on NPR. We’re such party poopers.

P.S. For more on this topic, Paul Alper points us to this 2001 parody article by Leonard Leibovici and these online comments.

P.P.S. Am I being mean to mock? I don’t think so. If we want to take this study seriously, I guess we could think of it as a form of performance art. Generating random numbers and telling stories about them can be fun. Randomness can inspire creative thinking. Just look at Jackson Pollock. Or Philip K. Dick. Or Rorschach. Kinda too bad that medical professionals are wasting time on this, but, hey, they need to blow off steam too. They’re not on call 24 hours a day. If I’m allowed to goof off and blog like this, they should be allowed to goof off and play with random numbers.

Let me emphasize that I have no objection to people studying the empirical implications of religious models. My problem is not with the idea that God can intervene in people’s lives. Who am I to question other people’s religious experiences? I only know what I’ve seen; I can’t judge the visions of others. I recommend they study the effects of prayer directly, without the randomization and all the other trappings of cargo cult science.

It’s hard to even write about topics such as religion out of the concern that people will say you’re disrespecting them. All I can say is . . . I’m not disrespecting your religion. I’m disrespecting the idea of studying a supernatural phenomenon using a randomized clinical trial—but, hey, do it if you want. It’s not that much worse than studying embodied cognition, beauty and sex ratio, ESP, subliminal smiley faces, and all sorts of other things that might as well be supernatural given their noise-to-signal ratio.

In all seriousness, I think these discussions do have statistical value, in the same way that we can deepen our understanding of inference by thinking about problems such as Newcomb’s paradox, as it can force us to think more carefully about what do we mean by treatment effects, randomization, etc. If you work it through carefully, you can figure out exactly what assumptions of the conventional causal model don’t work in the God-can-screw-with-the-randomization design.

Also, I’m legitimately annoyed at this doctor, and his enablers in the science establishment and the news media, who on one hand claim this is serious science and could improve people’s lives, and on the other hand say, “it’s not like we’re putting anyone at risk.” You can’t have it both ways. For one thing, there’s the opportunity cost of what the hospital staff could be doing if they weren’t wasting their time doing paperwork on this silly experiment. But, besides that, once you do consider the possibility that this push-button prayer really does something, then of course you could be putting people at risk. How can you be so sure the effects would be positive? It’s all just such a mess.