Rigorous Policy Pilots: Experimentation in the Administration of the Law

• Author: Colleen V. Chien
• Position: Justin D'Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013?2015 White House Senior Advisor, Innovation and Intellectual Property
• Pages: 2313-2350

2313

Rigorous Policy Pilots: Experimentation in

the Administration of the Law

Colleen V. Chien*

ABSTRACT: Rigorous tests are being used every day to develop effective

medical treatments, drive consumer engagement, and, more generally,

discover what works. But so far, rigorous policy piloting—temporarily

introducing a change in law or policy in order to learn from it using well-

designed and well-implemented methods—has not been used w idely because

of the perception that policy experimentation is unfair and possibly illegal,

difficult, and rare. This Essay draws upon case law and agency practice to

show that, to the contrary, rigorous policy pilots are presumptively legal,

feasible, and increasingly common, proceeding in several steps. First, it finds

that many kinds of pilots, including those that vary internal agency processes,

or which are opt-in are unlikely to be controversial. But a review of relevant

cases suggests that courts are likely to uphold even pilots that treat like

members of a population differently, including through randomization, when

they advance learning. Further, it finds experimentation, by itself, to be

unlikely to create special procedural or substantive hurdles. Second, it finds

that agencies are engaging in a range of rigorous piloting activities to fill

informational gaps in policy- and law-making, some of which simulate and

others which effect policy variation on a temporary basis, and that

*

Justin D’Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa

Clara University School of Law; 2013–2015 White House Senior Advisor, Innovation and

Intellectual Property. A companion online appendix to this Essay with policy experiments the

USPTO could try can be found at Colleen V. Chien, Rigorous Policy Pilots the USPTO Could Try, 104

IOWA L. REV. ONLINE 1 (forthcoming 2019). This Essay was the subject of a companion workshop,

Rigorous Policy Pilots, co-sponsored by the Administrative Conference of the U nited States, Penn

Program on Regulation, Columbia Center for Constitutional Governance, Partnership for Public

Service, and Santa Clara High Tech Law Institute held in 2019. Materials from that workshop, a

list of federal agency experiments and evaluation components, and relevant readings and

resources are posted to which are available at: https://www.law.upenn.edu/institutes/ppr/policypilots. I

thank Jenna Clark, Jonathan Liu, Lauryn Younge, Jiun-Ying Wu, Arti Rai, Bernard Chao, David

Schwartz, Cary Cognialese, Todd Rubin, Reeve Bull, Dan Correa, Lindsay Laferriere, Michael

Abramowicz, Heidi Johnson, Jonah Probell, Chris Walke r, Be rt H uang , Ol atun de J ohns on, Jess ica

Bulman-Pozen, Richard Briffault, Eric Talley, Stuart Graham, Rose Cuillon-Villasor, Suzanne

Kim, David Noll, and audiences at the Iowa Law Review Symposium on Administering Patent Law,

MIT Media Lab, and Unified Patents Conference for their helpful input to this project.

2314 IOWA LAW REVIEW [Vol. 104:2313

developments such as the growth of open data are making such forms of

information gathering easier. It draws from agency experience to develop a

framework for proposing a policy pilot and identify steps that would further

support the use of rigorous pilots. A companion online appendix applies this

framework to propose several rigorous pilots that the United States Patent and

Trademark Office (“USPTO”), building on its already strong tradition of

piloting, to evolve its own policies and practices with respect to patent quality

(through the robust vetting of applications in view of non-patent literature

and team/time examination on demand) and inclusion in innovation

(through automated error correction and addressing gender bias in

examination).

I.INTRODUCTION ........................................................................... 2315

II. RIGOROUS POLICY PILOTS ARE PRESUMPTIVELY LEGAL ............. 2320

A.CONSTITUTIONAL CHALLENGES TO RIGOROUS POLICY

PILOTS .................................................................................. 2326

1.The Interests Rationally Furthered by Policy

Pilots ............................................................................. 2326

2.Unequal or Equal Treatment? ................................... 2328

B.CHALLENGES TO EXPERIMENTAL RULEMAKING....................... 2329

1.Administering Policy Pilots ......................................... 2330

2.Judicial Review of Experimental Agency Action ....... 2331

3.Judicial Deference or Indifference to

Experimental Evidence? ............................................. 2332

C.CONCLUSION ......................................................................... 2334

III. (A FRAMEWORK FOR PROPOSING) RIGOROUS POLICY PILOTS

(THAT) ARE FEASIBLE AND WORTHWHILE ................................. 2335

A.STEP 1 (“M”): SELECT A QUESTION THAT MATTERS ............... 2339

B.STEP 2 (“A”): CONSIDER EXISTING AUTHORITY AND

AGENCY RESOURCES .............................................................. 2342

C.STEP 3 (“T”): IDENTIFY A TREATMENT AND THEORY OF

CHANGE ............................................................................... 2343

D.STEP 4 (“T”): SPECIFYING THE TEST STRATEGY ...................... 2344

E.STEP 5 (“E”): EVIDENCE OR THERE’S NO EASY WAY TO

MEASURE X........................................................................... 2346

F.STEP 6 (“R”): RESOURCES ...................................................... 2347

IV.CONCLUSION .............................................................................. 2348

2019] RIGOROUS POLICY PILOTS 2315

I. INTRODUCTION

“The country needs and, unless I mistake its temper, the country

demands bold, persistent experimentation. It is common sense to

take a method and try it: If it fails, admit it frankly and try another.

But above all, try something.”1

- Franklin Delano Roosevelt, Address at Oglethorpe University, May

22, 1932

In the fall of 1960, the Food and Drug Administration (“FDA”) received

an application for a drug for treating morning sickness. Already approved and

sold in dozens of countries,2 “Kevadon” was well-positioned for approval and

entry into the United States. But its reviewer, a Canadian-born physician

named Frances Kelsey, wasn’t convinced. Over several months and rounds of

document exchanges, she continued to find evidence of the drug’s safety and

effectiveness lacking. The manufacturer mounted a high-pressure campaign

and complained about Kelsey, whom they called a “petty bureaucrat” to her

boss.3

The drug—better known by its generic name, thalidomide, and never

approved for morning sickness in the United States—was later revealed to be

the source of numerous horrific birth defects.4 Kelsey was lionized and

bestowed with honors.5 But while Kelsey’s fame is well-deserved, the story has

a darker legacy. As Vincent DeVita, former head of the National Cancer

Institute, describes in The Death of Cancer, this episode “sent the message to

those who worked at the FDA that the way to do right by people was to say

no.”6 The subsequently enacted Drug Regulation Act put drug discovery out

of the reach of all but the largest companies7 and bred the risk-aversion for

which the agency is known,8 according to critics. One way the agency stays out

of trouble is to keep risky medicines off the market, a good thing. But there

1. Franklin Delano Roosevelt, Address at Oglethorpe University (May 22, 1932), available

at http://www.fdrlibrary.marist.edu/_resources/images/msf/msf00486.

2. Frances Oldham Kelsey: Medical Reviewer Famous for Averting a Public Health Tragedy, U.S.

FOOD & DRUG ADMIN., https://www.fda.gov/aboutfda/history/virtualhistory/historyexhibits/

ucm345094.htm (content current as of Feb. 1, 2018).

3. Robert D. McFadden, Frances Oldham Kelsey, Who Saved U.S. Babies from Thalidomide, Dies

at 101, N.Y. TIMES (Aug. 7, 2015), https://www.nytimes.com/2015/08/08/science/frances-

oldham-kelsey-fda-doctor-who-exposed-danger-of-thalidomide-dies-at-101.html.

4. VINCENT DEVITA & ELIZABETH DEVITA-RAEBURN, THE DEATH OF CANCER 195 (2015).

5. Including as the second female recipient of the President’s Award for Distinguishe d

Federal Civilian Service. Dr. Frances Kathleen Oldham Kelsey, NAT’L INST. HEALTH, https://

cfmedicine.nlm.nih.gov/physicians/biography_182.html (last updated June 3, 2015).

6. DEVITA & DEVITA-RAEBURN, supra note 4, at 196.

7. See generally Hans-Georg Eichler et al., The Risks of Risk Aversion in Drug Regulation, 12

NATURE REVIEWS DRUG DISCOVERY 907 (2013) (including among the adverse effects of excessive

regulatory risk aversion: slow approval times, the denial of useful medications to patients, and the

inability of small companies to compete in the marketplace).

8. See id. at 908–11.