The checklist manifesto and beyond

A few years ago, I was motivated to write about the intervention and the checklist: two paradigms for improvement, after reading Atul Gawande’s classic checklist manifesto. My angle was that in statistics we are trained to think about interventions (and studying their causal effects), but intervention is not the only paradigm.

Gaurav Sood sends along the following contextualization of the checklist issue:

We fail because we don’t know or because we don’t execute on what we know (Gorovitz and MacIntyre). Of the things that we don’t know are things that no else knows either—they are beyond humanity’s reach for now. Ignore those for now. This leaves us with things that “we” know but the practitioner doesn’t.

Practitioners do not know because the education system has failed them, because they don’t care to learn, or because the production of new knowledge outpaces their capacity to learn. Given that, you can reduce ignorance by (a) increasing the length of training, (b) improving the quality of training, (c) setting up continued education, (d) incentivizing knowledge acquisition, and (e) reducing the burden of how much to know by creating specializations, etc. On creating specialties, Gawande has a great example: “there are pediatric anesthesiologists, cardiac anesthesiologists, obstetric anesthesiologists, neurosurgical anesthesiologists, …”

Ignorance, however, ought not to damn the practitioner to error. If you know that you don’t know, you can learn. Ignorance, thus, is not a sufficient condition for failure. But ignorance of ignorance is. To fix overconfidence, leading people through provocative, personalized examples may prove useful.

Ignorance and ignorance about ignorance, however, are not the only reason we fail. We also fail because we don’t execute on what we know. Practitioners fail to apply what they know because they are distracted, lazy, have limited attention and memory, etc. To solve these issues, we can (a) reduce distractions, (b) provide memory aids, (c) automate tasks, (d) train people on the importance of thoroughness, (e) incentivize thoroughness, etc.

Checklists are one way to work toward two inter-related aims: educating people about the necessary steps needed to make a decision and aiding memory. But awareness of steps is not enough. To incentivize people to follow the steps, you need to develop processes to hold people accountable. Audits are one way to do that. Meetings set up at appropriate times during which people go through the list is another way.

Sood also shares some interesting notes he prepared on Gawande’s classic checklist manifesto. Here are some excerpts from Sood’s notes:

Portions of the book suggest that this is less about checklists and about engineering processes that reduce errors. Any process can be called a checklist—you do X followed by Y followed by Z—but that is stretching it.

. . .

Problem Statement: How do you make sure that people know and are following the process correctly?
To improve outcomes—study routine failures and how you would amend the process to improve outcomes.

. . .

Americans today undergo an average of seven operations in their lifetime, with surgeons performing more than fifty million operations annually—the amount of harm remains substantial.

. . .

“On average, the study reported, it took doctors seventeen years to adopt the new treatments for at least half of American patients.”

. . .

Concern w/ Some Checklist Implementations And Some Solutions

– Incentives for following checklists may be weak. How do you get people to follow?
– social pressure — checklist publicly marked as in a surgery
communicate clearly the issues and evidence on the efficacy of checklists
– get people to own the checklists—put their name, get their ideas on it — induce accountability
– checks of whether the stuff was followed and incentives and rewards based on that.
– People may stop using their brain and just follow the checklist
-how to put in checklists that clarify that brain cells are imp. and incentivize that.

– Train people to use checklists