Case Selection and the Study of Judicial Politics

• Date: 01 September 2008
• Author: Jonathan P. Kastellec,Jeffrey R. Lax
• Published date: 01 September 2008
• DOI: http://doi.org/10.1111/j.1740-1461.2008.00129.x

Case Selection and the Study of

Judicial Politics

Jonathan P. Kastellec and Jeffrey R. Lax*

One complication in studying the Supreme Court and the judicial hierarchy

is that the Court’s docket is now nearly completely discretionary. Although

the Justices’ strategies in picking cases affect the observations we make and

the inferences we draw, this is rarely taken into account in studies of judicial

politics. In this article, we study how case selection can affect our inferences

within judicial politics, including those about decision making in the

Supreme Court itself (such as whether law constrains the Justices) and

throughout the judicial hierarchy (such as whether lower courts comply

with Supreme Court doctrine). We use simulation analysis to show that

the inferential problems raised by the Court’s case selection range from

moderate to severe. At stake are substantive conclusions within some of the

most important and controversial debates in judicial politics.

I. Introduction

One complication in studying the Supreme Court and the judicial hierarchy

is that the Court’s docket is now nearly completely discretionary. The Justices

themselves choose which cases they will hear, and it is universally recognized

that they eschew less weighty cases, choosing instead to take hard, important,

*Address correspondence to Jonathan P. Kastellec, Department of Political Science, Columbia

University, 420 W. 118th St., New York, NY 10027; email: JPK2004@columbia.edu. Kastellec

is Ph.D. candidate; Lax is Assistant Professor, Department of Political Science, Columbia

University.

Earlier versions of this article were presented at the Annual Meeting of the American Political

Science Association, August 31–September 3, 2006, and the Annual Meeting of the Conference

on Empirical Legal Studies, November 9–10, 2007. We would like to thank Michael Bailey,

Cheryl Boudreau, Theodore Eisenberg, Andrew Gelman, Shigeo Hirano, David Klein, Dan

Klerman, Eduardo Leoni, Justin Phillips, Kirk Randazzo, and Rebecca Weitz-Shapiro for helpful

comments and suggestions, and Jeffrey Segal, Donald Songer, Charles Cameron, David Klein,

and Robert Hume for generously sharing their data with us. Replication data sets and statistical

code are available at 〈http://www.columbia.edu/~jpk2004/selection_replication.html〉.

Journal of Empirical Legal Studies

Volume 5, Issue 3, 407–446, September 2008

© 2008, Copyright the Authors

Journal compilation © 2008, Cornell Law School and Wiley Periodicals, Inc.

407

or controversial cases (Easterbrook 1982; Perry 1991; Kritzer & Richards

2002).1They can select cases in order to develop particular legal doctrines

or to rectify noncompliance in the lower courts, to name just two further

possibilities.

The precise selection strategy employed by the Justices will affect the

set of Supreme Court cases we observe in a given area of the law in a given

time period. Legal scholars and others have long been concerned about

making generalizations about law, legal practice, and judicial behavior from

the Supreme Court given that the cases the Justices hear are not represen-

tative (Edwards 1985; Cross 1997; Friedman 2006). Regardless of the par-

ticulars of case selection, it hardly seems likely that the Court chooses cases

randomly. Accordingly, the Court’s selection process raises the potential for

selection bias in the inferences we draw from its cases.

Since Heckman’s (1979) seminal article, social scientists have been

aware that selection bias may materially affect model estimation and substan-

tive inferences (see, e.g., Grier et al. 1994; Poe & Meernik 1995; Timpone

1998; Hart 2001; Ashworth et al. 2008).2However, this possibility has not

been incorporated into standard research designs in judicial politics.

Indeed, the problem of selection bias is usually noted only in passing, with

conclusions drawn as though case selection were random.3

Consider a simple example of why selection matters for studying

judicial behavior. Over the 20th century, the reversal rate on the U.S. Courts

of Appeals in published decisions was roughly 30 percent (Songer et al.

2000:105). In contrast, the Supreme Court reversed around 60 percent of

cases it heard in the same period (Segal & Spaeth 2002:263). One possibility

is that Supreme Court Justices simply have a higher propensity to reverse

lower courts than courts of appeals judges, but the more likely explanation

is that the Justices use their discretionary docket to select cases to reverse,

while appellate court judges frequently hear routine cases. Although this

1Indeed, the Judiciary Act of 1925 was realized directly as a result of the Justices’ professed desire

to not have to decide hundreds of “run-of-the-mill” cases each term (Hartnett 2000).

2Even if sample selection does not substantially affect estimation, selection models can help

establish the robustness of traditional estimation techniques (Allee & Huth 2006).

3By definition, the one area of research where selection has been incorporated is that of certiorari

(cert)—the Court’s decision whether to hear a particular case or not. Caldeira et al. (1999) and

Cameron et al. (2000), for instance, account for the cases the Court chooses not to hear in their

analyses. This is almost never done in more general studies of judicial decision making.

408 Kastellec and Lax

discrepancy in well-known, it helps illustrate that selection bias is relevant for

evaluating even simple patterns of judicial behavior.

Other examples turn up when we think about litigants or the courts of

appeals directly. Studying only cases that go to trial and ignoring settled cases

may produce a biased sample of the underlying population of cases (Priest &

Klein 1984; Eisenberg 1990). Similarly, taking into account litigants’ deci-

sion to appeal can affect our understanding of decision making on the U.S.

Courts of Appeals (Clermont & Eisenberg 2001). Finally, studying only

published cases to the exclusion of unpublished decisions may lead to either

an incomplete or misleading assessment of a particular court’s output

(Siegelman & Donohue 1990; Merritt & Brudney 2001; Law 2005).

To obtain a full measure of the possible selection bias at the Supreme

Court level, consider the typical path a case must take to be decided by the

Supreme Court. The parties must be engaged in a dispute that is not settled

before a trial court’s decision. One party must appeal following that decision

(by no means a costless action), again without a settlement before the

appellate court decision. One party must then bear the costs of a cert

petition, which must compete against thousands of other petitions for the

Supreme Court to actually grant cert. Each step in this process creates

different selection biases, which may be compounded as a case moves

through the judicial process.

It is diffcult to think of a major debate within judicial politics that is not

potentially touched by this problem. Scholars rely on observed Supreme

Court cases to study the preferences of Supreme Court Justices (Grofman &

Brazill 2002; Martin & Quinn 2002; Segal & Spaeth 2002), external con-

straints on their decision making (Spiller & Gely 1992; Segal 1997; Bergara

et al. 2003), the treatment of case facts (Segal 1984; McGuire 1990; Hagle

1991; Ignagni 1994), the role of law (George & Epstein 1992; Richards &

Kritzer 2002), and ideological change on the Court (Epstein et al. 1998,

2007b; Martin & Quinn 2007). At stake, therefore, are many of the most

important and controversial substantive debates in judicial politics.

In this article, we present the first investigation (to the best of our

knowledge) of how selection bias resulting from the Supreme Court’s case

selection might affect the inferences we draw using common research

designs in judicial politics. We ask with what confidence we can trust infer-

ences that do not take selection bias into account. We explore studies of the

Supreme Court’s decision making itself, as well as research that extrapolates

from the cases the Court hears to those it does not in order to make

inferences about the behavior of other legal actors. The former include such

Case Selection and the Study of Judicial Politics 409