The Negative Effect Fallacy: A Case Study of Incorrect Statistical Reasoning by Federal Courts

• Author: Anthony Fowler,Christopher S. Havasy,Ryan D. Enos
• Date: 01 September 2017
• DOI: http://doi.org/10.1111/jels.12158
• Published date: 01 September 2017

The Negative Effect Fallacy: A Case Study

of Incorrect Statistical Reasoning by

Federal Courts

Ryan D. Enos, Anthony Fowler,* and Christopher S. Havasy

This article examines the ne gative effect fallacy, a flawed statistica l argument first utilized

by the Warren Court in Elkins v. United States. The Court argued that empir ical evidence

could not determine wh ether the exclusiona ry rule prevents futur e illegal searches and

seizures because “it is ne ver easy to prove a negati ve,” inappropriately c onflating the

philosophical and arit hmetic definitions of the word negative. Subsequent ly, the Court has

repeated this mistake i n other domains, includ ing free speech, voting r ights, and

campaign finance. The falla cy has also proliferated into the fed eral circuit and district

court levels. Narrowly , our investigation aims t o eradicate the use of the nega tive effect

fallacy in federal cour ts. More broadly, we highlight several cha llenges and concerns with

the increasing use of sta tistical reasoning i n court decisions. As c ourts continue to

evaluate statistica l and empirical questio ns, we recommend that they e valuate the

evidence on its own merit rather than relying on co nvenient arguments e mbedded in

precedent.

I. Introduction

The law has an uneasy relationship with statistical and scientific evidence.

1

Legal history

is ripe with examples of lawyers and judges relying on controversial or incorrect inter-

pretations of empirical evidence. Perhaps most famously, the Supreme Court in Brown

v. Board of Education

2

relied on dubious psychological evidence in order to justify its

holding that racial segregation violates the Equal Protection Clause of the Fourteenth

*Address correspondence to Anthony Fowler, 1155 E. 60th St., 165, Chicago, IL 60637; anthony.fowler@ uchi-

cago.edu. Enos is Associate Professor, Department of Government, Harvard University; Fowler is Associate

Professor, Harris School of Public Policy, University of Chicago; Havasy is Ph.D. Candidate, the Department of

Government, Harvard University.

We thank Scott Ashworth, Richard Fallon, Jon Gould, Cody Gray, Pablo Montagnes, Nick Stephanopoulos,

Matthew Stephenson, and Susannah Barton Tobin for insightful comments and helpful guidance.

1

See David L. Faigman, To Have and to Have Not: Assessing the Value of Social Science to the Law as Science

and Policy, 38 Emory L.J. 1005, 1008 (1989). Cf. David Reisman, Some Observations on Law and Psychology, 19

U. Chi. L. Rev. 30, 32 (1951).

2

347 U.S. 483 (1954).

618

Journal of Empirical Legal Studies

Volume 14, Issue 3, 618–647, September 2017

Amendment.

3

Although some have argued that this psychological evidence was largely

inconsequential to the Court’s decision in Brown,

4

the Court’s apparent reliance on it

has been widely criticized.

5

Perhaps this state of affairs between law and statistics is not surprising; a standard

legal education does not include rigorous training in statistics or the evaluation of scien-

tific evidence. Federal judges and their clerks bring their limited statistical experience

with them to the bench, and they have little incentive to develop these skills. As a result,

statistical errors in one case can propagate to others through precedent. Allison Orr

Larsen finds that legal reasoning based on empirical information can form “factual

precedents” through “the tendency of lower courts to over-rely on Supreme Court opin-

ions and to apply generalized statements of fact from old cases to new ones.”

6

In the case of the flawed psychological evidence used in Brown, the wider legal

profession eventually corrected itself—similar research is no longer used as factual

precedent by federal courts. But other mistakes in scientific reasoning have received less

scrutiny by the legal profession, despite their widespread consequences. In this article,

we examine an incorrect statistical argument from an exclusionary rule case in 1960,

7

which then propagated over time and across various different legal domains. In subse-

quent decades the same incorrect reasoning was applied repeatedly by the Supreme

Court to cases regarding the exclusionary rule, voting rights, and free speech.

8

Most

recently, the Court used the argument to justify its ruling in Arizona Free Enterprise,in

which the Court struck down the matching funds provision of Arizona campaign finance

laws.

9

Proliferating further, lower federal circuit courts have widely adopted this fallacy

across different legal domains.

10

The specific error in question involves the empirical task of “proving a negative”

or, more appropriately, determining whether a law in question decreases a particular

3

Id. at 494 n.11.

4

See James E. Ryan, The Limited Influence of Social Science Evidence in Modern Desegregation Cases, 81 N.C.

L. Rev. 1659, 1665 (2003); Jack M. Balkin, Rewriting Brown, in Jack M. Balkin and Bruce A. Ackerman, eds., What

Brown v. Board of Education Should Have Said 51 (2002).

5

For early criticisms of the footnote, see Charles L. Black, Jr., The Lawfulness of the Segregation Decisions, 69

Yale L.J. 421, 427 (1960); Edmund Cahn, Jurisprudence, 30 NYU L. Rev. 150, 167 (1955). For recent discussions

of the controversy surrounding the footnote, see Michael Heise, Brown v. Board of Education, Footnote 11, and

Multidisciplinarity, 90 Cornell L. Rev. 279, 292--95 (2005); Sanjay Mody, Note, Brown Footnote Eleven in Histori-

cal Context: Social Science and the Supreme Court’s Quest for Legitimacy, 54 Stan. L. Rev. 793, 803--09 (2002).

6

Allison Orr Larsen, Factual Precedents, 162 U. Pa. L. Rev. 59, 62 (2013).

7

Elkins v. United States, 364 U.S. 206, 218 (1960).

8

See Section III for a list of these cases.

9

Arizona Free Enter. v. Bennett, 131 S. Ct. 2806, 2823 (2011).

10

See Section III for a list of these cases.

619The Negative Effect Fallacy