Randomness Pre‐Considered: Recognizing and Accounting for “De‐Randomizing” Events When Utilizing Random Judicial Assignment

• Date: 01 June 2020
• Author: Dane Thorley
• DOI: http://doi.org/10.1111/jels.12248
• Published date: 01 June 2020

Journal of Empirical Legal Studies

Volume 17, Issue 2, 342–382, June 2020

Randomness Pre-Considered: Recognizing

and Accounting for “De-Randomizing”

Events When Utilizing Random Judicial

Assignment

Dane Thorley*

This article contributes to the growing literature challenging the general assumption of

and reliance on random judicial assignment by identifying common court procedures

and practices that threaten unbiased causal inference. These “de-randomizing” events,

which include differing probabilities of assignment, post-assignment judicial changes,

nonrandom missingness, and nonrandom assignment itself, should be accounted for

when making causal claims but are commonly either ignored or not even recognized by

researchers utilizing random judicial assignment. The article explores how these de-

randomizing events violate the key empirical assumptions underlying randomized studies

and offers methodological solutions. It also presents original data from a survey of the

30 largest U.S. state-level criminal courts, outlining their assignment protocols and iden-

tifying the extent to which they feature the de-randomizing events described in the

article.

I. Introduction

In his 2010 article titled “Randomness Reconsidered: Modeling Random Judicial Assignment

in the U.S. Courts of Appeals,”

1

Matthew Hall challenged the long-standing assumption th at

judges in the U.S. Federal Courts of Appeals were randomly assigned to their cases.

2

Specifi-

cally, Hall argued that earlier research claimingtoidentifyastrongcausalrelationship

*Address correspondence to Dane Thorley, Associate Professor of Law, J. Reuben Clark Law School, Brigham

Young University, 341 E. Campus Dr, Provo, UT 84602; email: thorleyd@law.byu.edu.

Special thanks to Ian Ayres, Adam Chilton, Donald Green, Shigeo Hirano, Daniel Ho, Jeffrey Lax, Gregory Wawro,

and anonymous reviewers for helpful guidance and feedback.

1

Matthew Hall, Randomness Reconsidered: Modeling Random Judicial Assignment in the U.S. Courts of Appeals,

7 J. Emp. Legal Stud. 574 (2010).

2

The assumption of random assignment is so embedded in legal research that the “random assignment of federal

appellate judges to panels has become a ‘hallmark’ of the system.” Emerson H. Tiller & Frank B. Cross, A Modest

Proposal for Improving American Justice, 99 Colum. L. Rev. 215, 216 (1999).

342

between the ideologies of judges and the tenor of their decisions

3

was systematically flawed

because the investigators had assumed that all cases were assigned to circuit panels on a ran-

dom basis. After demonstrating this assumption to be erroneous and accounting for the

courts’ actual assignment procedures (some of which were, if fact, random), Hall found that

the researchers had identified ideological effects where none existed and failed to identify

effects for circumstances in which ideology did indeed appear to make a difference.

The mistakes highlighted by Hall are not uncommon in the courts and judicial behav-

ior literature. An increasingly large number of empirical articles have utilized random assign-

ment of judges to make causal claims regarding an array of legal, political, and economic

outcomes. These studies, many of which are reviewed below, rely on the fact (or often the

assumption) that the judges in their samples are assigned to cases on a truly random basis.

This is because random assignment allows researchers to overcome the most important and

difficult hurdle associated with causal identification—controlling for unobserved heterogene-

ity. Despite the essential role that random case assignment plays in such studies, however,

researchers often fail to delve into the specific mechanics of a court system’s assignment pro-

cess, choosing instead to rely on the assumptionsofpreviousstudiesorthesimpleassurances

of court officials.

4

In reality, however, verifying random assignment and using the resulting

data to make causal claims is often a complicated process and should mirror the multi-step

analysis conducted in researcher-driven randomized studies such as field experiments.

This article adds to the growing body of methodological literature exploring the use

of random judicial assignment

5

by identifying a set of common assignment procedures that

I call “de-randomizing” events. These events, which include nonrandom assignment itself,

3

Cass R. Sunstein, Lisa Michelle Ellman & David Schkade, Ideological Voting on Federal Courts of Appeals: A Pre-

liminary Investigation, 90 Va. L. Rev. 301 (2004).

4

This is not to suggest that courts are being misleading (although some are, see infra note 99 and accompanying

text). Rather, the colloquial use of “random” is not equivalent to the statistical definition. See Section IV.A.1 for a

more in-depth discussion of this confusion.

5

In addition to the Hall 2010 piece, Hall, supra note 1, see Theodore Eisenberg, Talia Fisher & Issi Rosen-Zvi,

Does the Judge Matter? Exploiting Random Assignment on a Court of Last Resort to Assess Judge and Case Selec-

tion Effects, 9 J. Emp. Legal Stud. 246 (2012); J. Robert Brown, Jr. & Allison Herren Lee, Neutral Assignment of

Judges at the Court of Appeals, 78 Tex. L. Rev. 1037 (2000); and Burton M. Atkins & William Zavoina, Judicial

Leadership on the Court of Appeals: A Probability Analysis of Panel Assignment in Race Relations Cases on the

Fifth Circuit, 18 Am. J. Pol. Sci. 701 (1974).

Two recent articles also explore the panel composition procedures (as opposed to the case assignment proce-

dures) in U.S. Circuit Courts in detail. See Adam S. Chilton & Marin K. Levy, Challenging the Randomness of

Panel Assignment in the Federal Courts of Appeal, 101 Cornell L. Rev. 1 (2015); Marin K. Levy, Panel Assignment

in the Federal Courts of Appeal, 103 Cornell L. Rev. 65 (2017).

It is also worth noting that the popular press has expressed concerns regarding the non-randomness of panel

composition and case assignment in the federal courts, although for non-methodological reasons. See Joe

Palazzolo, The Problem with Not-So-Random Case Assignment, Wall St. J. (Nov. 4, 2014. 4:44 PM), https://bl ogs.

wsj.com/law/2013/11/04/the-problem-with-not-so-random-case-assignment/; Benjamin Weiser & Joseph Gold-

stein, Federal Court Alters Rules on Judge Assignments, N.Y. Times (Dec. 23, 2013), https://www.nytimes.com/

2013/12/24/nyregion/federal-court-alters-rules-on-judge-assignments.html; Alison Frankel, Chief Judge: Rakoff

Assignment to Citi Case Was ‘Totally Random,’ Reuters (Nov. 30, 2011), http://blogs.reuters.com/alison-frankel/

2011/11/30/chief-judge-rakoff-assignment-to-citi-case-was-totally-random/.

Randomness Pre-Considered 343

must be accounted for in order to make unbiased causal claims but are commonly either

ignored or not even recognized by researchers relying on random judicial assignment. In

classifying and illustratingthese events, I also highlight the increasing need for legal empiri-

cists to develop field-specific methods that integrate robust statistical techniques with the

specialized procedural anddoctrinal knowledge required to adequately study the law.

6

Additionally, I attempt to fill in what others have noted

7

to be a dearth of informa-

tion on the assignment protocols of courts other than the U.S. Courts of Appeals by pre-

senting original data from a survey of the 30 largest state-level criminal courts in the

United States, outlining their assignment protocols, and identifying the extent to which

they feature the “de-randomizing” events highlighted in this article.

This article proceeds in five parts. In Section II, I review the basic criteria required

for unbiased causal inference and explain the empirical advantages of random assign-

ment. In Section III, I provide a short review of the literature that has utilized random

assignment of judges. In Section IV, I discuss the various “de-randomizing” events that

researchers should account for in order to make unbiased causal inferences and provide

solutions—where they exist—for those events. In Section V, I present original data from

a survey of state-level criminal justice systems on the judicial assignment procedures used

in their courts. I briefly conclude in Section VI.

II. Random Assignment and Causal Inference

Over the last decade, the volume of legal scholarship utilizing random assignment

has grown rapidly.

8

Researchers have randomly made offers of legal representation,

9

6

I agree with Epstein and King’s proclamation that “the law is important enough to have a subfield devoted to

methodological concerns, as does almost every other discipline that conducts empirical studies.” Lee Epstein &

Gary King, The Rules of Inference, 69 U. Chi. L. Rev. 1, 11 (2002). They go on to say that “[u]nfortunately, the

complete list of all law review articles devoted to improving understanding, explicating, or adapting the rules of

inference is as follows: none.” Id. While this is certainly not quite as true anymore, the underlying sentiment is still

reflective of the field.

7

See Adam M. Samaha, Randomization in Adjudication, 51 Wm. & Mary L. Rev. 1, 47 (2009).

8

While still not predominantly featured in law reviews—due in part to their quantitative nature—the number of

articles featuring randomized experiments or naturally occurring randomizations that have appeared in top law

journals have dramatically increased in recent years. See Donald P. Green & Dane R. Thorley, Field Experimenta-

tion and the Study of Law and Policy, 10 Ann. Rev. L. & Soc. Sci. 53, 56 fig.1 (2014).

9

See Carroll Seron, Greg Van Ryzin, Martin Frankel & Jean Kovath, The Impact of Legal Counsel on Outcomes for

Poor Tenants in New York City’s Housing Court: Results of a Randomized Experiment, 35 L. & Soc’y Rev.

419 (2001) (in which the authors randomly offered legal assistance to individuals involved in property disputes);

D. James Greiner & Cassandra Wolos Pattanayak, Randomized Evaluation in Legal Assistance: What Difference

Does Representation (Offer and Actual Use) Make? 121 Yale L.J. 2118 (2012) (in which the authors randomly

offered legal assistance to individuals seeking unemployment benefits); D. James Greiner, Cassandra Wolos

Pattanayak & Jonathan Hennessy, The Limits of Unbundled Legal Assistance: A Randomized Study in a Massachu-

setts District Court and Prospects for the Future, 126 Harv. L. Rev. 901 (2013) (in which the authors randomly

offered legal assistance to individuals involved in landlord-tenant disputes).

344 Thorley