Do Checklists Make a Difference? A Natural Experiment from Food Safety Enforcement

• Author: Daniel E. Ho,Phil Wyman,Sam Sherman
• DOI: http://doi.org/10.1111/jels.12178
• Date: 01 June 2018
• Published date: 01 June 2018

Do Checklists Make a Difference? A

Natural Experiment from Food

Safety Enforcement

Daniel E. Ho, Sam Sherman, and Phil Wyman*

Inspired by Atul Gawande’s bestselling Checklist Manifesto, many commentators have called

for checklists to solve complex problems in law and public policy. We study a unique

natural experiment to provide the first systematic evidence of checklists in law. In 2005, the

Public Health Department of Seattle and King County revised its health code, subjecting

half of inspection items to a checklist, with others remaining on a free-form recall basis.

Through in-depth qualitative analysis, we identify the subset of code items that remained

substantively identical across revisions, and then apply difference-in-differences to isolate

the checklist effect in more than 95,000 inspections from 2001--2009. Contrary to scholarly

and popular claims that checklists can improve the administration of law, the checklist has

no detectable effect on inspector behavior. Making a violation more salient by elevating it

from “noncritical” to “critical” status, however, has a pronounced effect. The benefits of

checklists alone are considerably overstated.

I. Introduction

Atul Gawande’s bestselling Checklist Manifesto argued that dramatic improvements in

complex decisions can be achieved by cheap and easy-to-use checklists (Gawande

2009). Generalizing from the evident success in surgery, Gawande claimed that check-

lists—simple, enumerated lists of information or steps—could improve complex deci-

sions in “virtually any endeavor” (2009:158). While Gawande acknowledged potential

limitations, others quickly latched on: “Checklists are hot [and have] captured the

imagination of the media and have inspired publication of a manifesto on their

*Address correspondence to Daniel E. Ho, William Benjamin Scott and Luna M. Scott Professor of Law, Stanford

Law School; Professor (by courtesy) of Political Science; Senior Fellow, Stanford Institute for Economic and Pol-

icy Research; 559 Nathan Abbott Way, Stanford, CA 94305; email: dho@law.stanford.edu. Sherman is J.D.

Candidate, Stanford Law School; Wyman was Health and Environmental Investigator, Public Health—Seattle and

King County and is Environmental Health Specialist, Snohomish Health District.

We thank Mark Bossart for help securing data, Janet Anderberg for insights into the health code revision,

Becky Elias, Ki Straughn, and Dalila Zelkanovic for help in developing and carrying out a simplified inspection

model, Claire Greenberg and Aubrey Jones for research assistance, and Becky Elias, Atul Gawande, Sandy

Handan-Nader, Tim Lytton, Oluchi Mbonu, Bill Simon, David Studdert, and two anonymous referees for

comments.

242

Journal of Empirical Legal Studies

Volume 15, Issue 2, 242–277, June 2018

power in managing complexity” (Davidoff 2010:206).

1

Or as the Times put it, check-

lists are “a classic magic bullet” (Aaronovitch 2010).

Fueled in part by Gawande’s manifesto, scholars have, in turn, advocated for

greater adoption of checklists in law and public policy. Richard Thaler and Cass Sun-

stein consider the checklist as part of choice architecture (Thaler et al. 2012:432). Bill

Simon (2012:394) urged “[p]rotocols of the sort that Gawande developed,” suggesting

provocatively that checklists might have averted major financial and accounting scandals,

such as options backdating and Enron’s collapse. In criminal law, scholars have pro-

posed checklists to help police and prosecutors comply with Brady obligations (Findley

2010; Griffin 2011), to improve defense lawyering (Bibas 2011), and to help judges

weigh testimony (Levine 2016). A Federal Courts Study Committee recommended a leg-

islative drafting checklist (Maggs 1992) and checklists have been proposed to improve

election administration (Douglas 2015), safety regulation (Hale & Borys 2013), and edu-

cation (Institute of Education Sciences & National Center for Education Statistics,

Department of Education 2014). In the regulatory context, Gawande himself argued

that checklists would enable building inspectors and industry alike to make “the reliable

management of complexity a routine” (Gawande 2009:79). If so, checklists might

address longstanding challenges with decentralized law enforcement and administration

(Ho 2017:5–12; Mashaw 1985), where few proven techniques exist to address dramatic

disparities in complex legal decisions (Ho & Sherman 2017).

Yet the legal reception to checklists has not been uniform. Checklists, rather than

solving the problem of bureaucracy, may create it (Ho 2017:57, 82–83; Simon

2012:395). Ford (2005) associates checklists with the pathologies of rigid command-and-

control regulation, and McAllister (2012) warns that checklists may lead to mechanistic

application failing to detect real risk. Warned Stephen Cutler, Director of the Division

of Enforcement at the Securities and Exchange Commission: “don’t fall victim to a

checklist mentality” (Cutler 2004). Opening a congressional hearing on emergency pre-

paredness, entitled “Beyond the Checklist,” Representative Langevin admonished: “Our

nation’s leaders are not seeing the big picture. Instead, they are driving our depart-

ments and agencies to focus so much effort on checking boxes that there is barely time

left to actually combat a potential pandemic” (Langevin 2007). Across areas, checklist

skepticism abounds. Koski (2009:831) critiqued the checklist mentality in school reform,

noting that parties in structural litigation “develop[ed] more paperwork, checklists, and

bureaucratic oversight, essentially ‘teacher-proofing’ the reform process.” In family law,

Ross (2003:206) concluded that “[t]here is ... no easy checklist that agencies and courts

can rely upon to predict whether a child can be safe with his or her parents.” In securi-

ties regulation, Ford (2008:29) opined that “[c]reating ever-longer lists of prohibited

behavior or checklists of compliance-related best practices will not be effective if the

basic culture of the firm does not foster law-abiding behavior.” In food safety, Stuart

1

Gawande’s book treatment rightly acknowledges conditions under which checklists may not work, such as a

command-and-control ethos (2009:75--76), the role of institutional culture (2009:160), and difficulties in imple-

mentation (2009:170), yet many commentators simply focus on the claims that checklists are “simple, cheap,

effective” (2009:97) and “quick and simple” (2009:128).

243Do Checklists Make a Difference?

and Worosz (2012:294) documented one “veteran food safety auditor” who claimed that

“about 70% of the items on food safety checklists are irrelevant to food safety.”

A core empirical premise of proponents is that checklists, by reminding individuals,

should improve the detection of errors, violations, or mistakes for complex tasks

(Gawande 2007, 2009; Hales et al. 2008; Thaler et al. 2012). Despite the litany of reform

proposals and sharp disagreement over their effectiveness, as we articulate below, this

empirical premise has never been subject to rigorous empirical scrutiny in law and pub-

lic policy.

We address this lacuna by providing the first evidence in a legal context of the

effectiveness of checklists. We study a unique and inadvertent natural experiment in

food safety enforcement conducted by the Public Health Department in Seattle and

King County. In 2005, the department revised its health code, subjecting roughly half of

violations (classified as critical) to a checklist to be employed by inspectors, but leaving

other (noncritical) violations to a free-form recall basis, as was the case before for all vio-

lations. Most importantly, while the revision added and revised a range of items,

through in-depth qualitative research, including engagement with staff responsible for

implementing the code revision, we classify the subset of violations that remained identi-

cal before and after the revision.

This approach offers several advantages to understanding the causal effect of

checklists on code citation. First, a common critique of checklist studies is that the intro-

duction of the checklist is accompanied by greater training, teamwork, and a reorienta-

tion of tasks, hence confounding the intervention (Bosk et al. 2009). By focusing on

identical code items—for which there was no additional training and change in instruc-

tions—we are able to isolate the effect of a checklist independent of other factors. Sec-

ond, because the checklist applied only to half the violations, a difference-in-differences

design accounts for temporal changes in sanitation and enforcement practices, as well

as selection bias in who adopts checklists. Our design therefore addresses two common

limitations to extant checklist studies: before-after design studies cannot rule out tempo-

ral changes (e.g., increased managerial commitment to quality improvement) and cross-

sectional comparisons cannot rule out preexisting quality differences between adopting

and nonadopting institutions (see De Jager et al. 2016).

Third, our design uses internal administrative data to account for assignments of

establishments to inspectors. Our treatment effect is hence identified by changes in how

the same establishments are inspected by the same inspectors before and after the code

revision, for the subset of code items that remained identical except for the checklist

format. This has a considerable advantage in the inspection context, where there are

well-documented differences in inspection stringency by inspector (Ho 2012, 2017).

Last, because the checklist format was merely incidental to the substantive changes in

the health code, our design rules out Hawthorne effects, whereby research subjects may

improve performance due to the knowledge of being observed (Haynes et al. 2009:497).

We study data from 95,087 inspections, and 15 identical violations (i.e., 1,426,305

potential violations) scored in each inspection by 37 inspectors serving before and after

the intervention for 2001–2009. Applying logistic difference-in-differences regressions

with inspector and establishment fixed effects, we find that the checklist has no

244 Ho et al.