Randomizing law.

• Author: Abramowicz, Michael

Governments should embrace randomized trials to estimate the efficacy of different laws and regulations. Just as random assignment of treatments is the most powerful method of testing for the causal impact of pharmaceuticals, randomly assigning individuals or firms to different legal rules can help resolve uncertainty about the consequential impacts of law. In this Article, we explain why randomized testing is likely to produce better information than nonrandom evaluation of legal policies. We then offer guidelines for conducting legal experimentation successfully, considering a variety of obstacles, including ethical ones. Randomization will not be useful for all policies, but once government gains better experience with randomization, administrative agencies should presumptively issue randomization impact statements justifying decisions to implement particular policies. Making the content of law partially contingent on the results of randomized trials will promote ex ante bipartisan agreements, as politicians with different empirical predictions will tend to think that the experiments will support their position.

INTRODUCTION I. THE POWER OF RANDOMIZED CONTROLS II. THE PROBLEMS OF NONRANDOM EVALUATION A. Conventional Regression Analysis 1. Omitted Variable Bias 2. Publication Bias and Misspecification B. The Laboratory of the States Reconsidered III. CAVEATS: LIMITS OF RANDOMIZATION STUDIES A. Interpretive Problems 1. Non-Double-Blind Randomization 2. Generalizability a. Self-Selection b. Experimenter Selection 3. Imperfect Randomization a. Attrition b. Crossover c. Spillovers B. Other Issues 1. Costs 2. Ethical Concerns 3. Equality Concerns IV. GUIDELINES AND APPLICATIONS A. General Guidelines B. Institution-Specific Guidelines 1. Administrative Agencies: The Case for a Randomization Impact Statement 2. Legislatures: The Case for Self-Execution C. Applications 1. Securities Law a. A Short-Sale Experiment b. Experimental Sarbanes-Oxley Repeal 2. Tax Law 3. Civil Rights CONCLUSION INTRODUCTION

Legal scholars have debated the impacts of government policy for millenia. In 81 B.C., Chinese scholars argued about the desirability of monopolies in the salt and iron industries in a succession of essays and public debates. (1) These debates were theoretical--with scholars predicting the positive and negative effects of monopolies as compared to a competitive market. Over two thousand years later, theoretical debates over policies remain the norm. But theory alone cannot resolve many policy issues because different theories point in different directions. Scholars attempt to inform these debates by parsing historical data, but regression analysis of policy is fraught with complications. There is little policy variation on many topics of national importance, and the variation that does exist is correlated with many other factors. Empirical policy evaluation often resembles a drug study in which the experimental population does not receive an assigned treatment and instead gets to choose whether to take the medicine or the placebo.

Policymakers and commentators frequently refer loosely to new laws and legal institutions as experiments, but in contrast to medical experimentation, (3) these innovations rarely randomly designate treatment and control groups. There have been a handful of exceptions since 1968, including randomized "social experiments" that were performed to assess the impact of government policies. (4) But the legal literature has virtually ignored them. Legal scholars have discussed the results of particular social experiments, (5) and they have commented occasionally that additional social experiments could provide useful information in one field or another. (6) But these legal scholars have not addressed the normative question of whether the legal system should generally seek to incorporate experimental methods, and if so, what approaches the legal system should take to maximize the chance that experiments will improve policy.

Perhaps as a partial result of this scholarly neglect, past social experiments have clustered in specific policy areas. As the label "social experimentation" suggests, most of the experiments have been in the area of social services, testing whether expenditures on entitlements succeed in achieving social goals, such as reducing poverty. (7) For example, a recent experiment executed under a Medicare statute requiring randomized testing of programs (8) assessed whether telephone contact by nurses to at-risk Medicare patients would reduce program costs." Another class of randomized studies evaluated criminal justice policies. (10) A rare exception to these two areas has been a set of experiments on electricity pricing. (11) Experiments have almost never varied the legal rights and obligations of ordinary citizens or entities in areas such as securities law or taxation. (12) Instead, experiments have focused on the possible provision of new services or on those who might be thought of as forfeiting rights by committing crimes.

This Article advances the case for randomizing law, including the legal rights and obligations expressed in statutes and regulations. (13) Randomized experiments have the potential not only to be governmentally funded academic exercises, but also to serve as integral components of the legal process. In this Article, we argue that government should embrace randomized trials of statutes and regulations as a tool for testing the effectiveness of those laws. Just as random assignment of treatments is the most powerful method of testing for the causal impact of pharmaceuticals, random assignment of individuals, firms, or jurisdictions to different legal rules can help resolve uncertainty about the consequences of laws and regulations.

Beyond endorsing randomized legal experimentation in areas where such experiments have not generally been contemplated, this Article considers how the policy process should change to accommodate randomized experimentation. Administrative law, we argue, should accept decisions by agencies to randomize policies and perhaps even be more deferential to policy decisions made after a process of experimentation. Ultimately, the executive branch could make formalized consideration of randomized control trials as central to the regulatory process as formalized consideration of the costs and benefits of regulations. If experimentation begins to occur with sufficient frequency in agencies, Congress or other legislatures might themselves initiate experiments more frequently. The possibility of experimentation may reduce legislative disagreement. Where disagreements are truly empirical, partisans on both sides of an issue may believe that they would benefit from experimentation. A self-executing experiment can, in effect, serve to resolve a bet among competing legislative factions, with the experimental outcome automatically affecting the content of the legislation. Meanwhile, if a legal culture of randomization developed sufficiently, a legislator's refusal to endorse an experiment might be interpreted as evidence that the legislator's empirical claims about a policy mask some other agenda.

The Article proceeds as follows. Part I lays out the affirmative case for randomized control trials and describes our central proposal. Part II describes the problems of nonrandom evaluation of legal policies. Conventional regression analysis is subject to problems, including omitted variable bias, publication bias, and misspecification. Part III discusses potential problems and pitfalls of randomized policy experiments, as well as responses to these complications. Some of these problems involve challenges of interpreting even randomized legal experiments, though, in general, randomization should make interpretation somewhat easier. The more challenging problems from the perspective of policy implementation are that randomized legal policy may be costly or raise ethical concerns. Finally, Part IV offers some guidelines for legal experimentation, including specific recommendations for legislatures and administrative agencies, and it then describes specific applications in which randomization seems especially likely to be fruitful.

1. THE POWER OF RANDOMIZED CONTROLS

   The idea that randomization could be used to create a quality control group has existed since 1925, when Ronald Fisher, the father of modern statistics, suggested using random assignments in research involving agricultural trials that arose out of his work at the Rothamsted Experimental Station. (14) In his 1935 book, The Design of Experiments, Fisher explained the power of the technique with the arresting example of a "[l]ady [who] declares that by tasting a cup of tea made with milk she can discriminate whether the milk or the tea infusion was first added to the cup." (15) Fisher proposed mixing eight cups of tea--four with milk first and four with milk last--and "presenting them to the subject for judgment in a random order." (16)

   Intentionally interjecting uncertainty into the experimental design could have the perverse effect of enhancing the ability of a researcher to control the experiment. As David Harrington has noted,

   [i]n one of the delightful ironies of modern science, the randomized trial "adjusts" for both observed and unobserved heterogeneity in a controlled experiment by introducing chance variation into the study design. If interventions for patients are chosen by chance, then the law of large numbers implies that the average values of patient characteristics should be roughly equal in the intervention groups. (17) Randomization itself produces the controlled environment in which a similar group may provide a source of comparison. Of course, randomization does not mean that the control and treatment groups will be identical. If we looked at, for example, the heights of people in each group, we would see the normally distributed bell curve. But the point is that we would...