Barry Dehlin writes:
See this blog post by “Scott Alexander” about an RCT on the use of cloth and surgical masks. As he points out, this issue is momumentally important RIGHT NOW and a real statistical expert should be evaluating this study and the conclusions the authors draw. Here, I think the value of your expertise would be huge and topical, and would encourage you to review and post on this issue.
I don’t know how relevant my statistical expertise is here, actually. I’ll get back to that at the end of this post.
Now to the details. First I took a look at the research paper in question, “A cluster randomised trial of cloth masks compared with medical masks in healthcare workers,” by C. Raina MacIntyre et al., and was published in the British Medical Journal in 2015. Here’s what they report:
The aim of this study was to compare the efficacy of cloth masks to medical masks in hospital healthcare workers (HCWs). The null hypothesis is that there is no difference between medical masks and cloth masks.
Setting: 14 secondary-level/tertiary-level hospitals in Hanoi, Vietnam. . . . 1607 hospital HCWs . . .
Hospital wards were randomised to: medical masks, cloth masks or a control group (usual practice, which included mask wearing). . . .
Results: The rates of all infection outcomes were highest in the cloth mask arm, with the rate of ILI statistically significantly higher . . . Penetration of cloth masks by particles was almost 97% and medical masks 44%. . . .
OK, the conclusions seem pretty clear. In this small study, medical masks worked better than cloth masks. That makes sense! To learn more, we’d want more data in other settings.
Next I went to Scott Alexander’s post, which begins:
Huh? Did MacIntyre et al. really say not to make your own face mask??? I missed that! Let me take a closer look . . . I guess Alexander is picking up on the following bits from the linked paper:
The results caution against the use of cloth masks . . . as a precautionary measure, cloth masks should not be recommended for HCWs . . . HCWs should not use cloth masks as protection against respiratory infection. . . it is important to consider the potential risk of using cloth masks. . . .
I guess it’s all how you read it. As a statistician, I read the MacIntyre et al. paper and immediately interpreted the result as a comparison: cloth masks vs. medical masks. Medical masks are better than cloth masks. As a practitioner, Alexander read the paper as making an absolute statement that cloth masks are no good for the general population. That’s not how I read the paper—after all, “compared with medical masks in healthcare workers” is right there in the title!—but I guess this is a warning to all of us who write research papers that, if we’re not super-careful, people can draw conclusions from our work that are not in our data. And, to be fair to Alexander, there are those quotes from the paper: “The results caution against the use of cloth masks,” etc.
Alexander then looks at the article more carefully and reports, “the authors themselves lean towards the hypothesis that that cloth masks are actively bad.” I guess I didn’t take that so seriously. At this point, I pretty much make it a point to focus on the details of the study (what’s actually being measured and compared) and to look carefully at the title and abstract (as this is the main message sent by the paper) and not worry so much about unsupported speculation. What I’m saying is, if the paper had some speculation that Alexander is skeptical of, that’s fine, it’s good to know. But I don’t think some speculation deep in the paper is the main message of the paper. The main message is first in the title, and second in the abstract, and in both it’s clear that the paper is comparing cloth to medical masks, and that it’s for health care workers, not the general population. Alexander is arguing against some claims made in the discussion section of the paper. I’m not disagreeing with Alexander’s arguments on that point; I just think it’s a mistake for him to read that paper as making any claims about the efficacy of cloth masks compared to no masks for the general population.
The background is that a lot of people are mad at public health authorities for discouraging mask-wearing. The evidence on homemade masks still seems unclear, but I’m sympathetic to the argument that there’s nothing to lose by wearing masks in crowded places. That research article from 2015 seems fine to me: it clearly separates its empirical conclusions from its speculations, and I’m sure that there have been some studies since then on the effectiveness of masks in other settings.
OK, so what about the business of my statistical expertise? It didn’t really come up here, except in the negative sense that I could see what they were doing in that paper so I didn’t have to look at the details too carefully. I had a good GRE score and I’m paid well, so I guess that gives me credibility in some quarters. Perhaps more relevant here was my expertise as a consumer of statistical results, as this led me to focus on the title, abstract, and main conclusions and not get distracted by the speculations in that paper. Alexander’s post is reasonable too and makes use of his expertise as a clinician.
When it comes to the important question of whether we should all be wearing masks in crowded places, my statistical expertise isn’t so relevant! I was able to read the above-linked research article and focus on its data-based conclusions, and I’m here to remind you that the question isn’t so much, Should we wear masks? but How should we do it?, but that doesn’t take a lot of statistical expertise. Subject-matter expertise and relevant data are much more important here. If we had some data, then statistical analysis could become more relevant, but I’m guessing you can get most of the way there by following the usual principle of not trying to collapse uncertainty into certainty. Statistical expertise is also relevant for data collection, but again the basic idea seems pretty clear: get some people in the lab wearing no masks or different masks in real-world conditions and see what you find, then do something similar in the wild.