This Article radically rethinks the treatment of statistical estimation evidence in civil litigation, focusing for convenience on the federal courts. It proposes an approach that harmonizes legal standards and statistical concepts, replacing the arbitrary and elevated standards of conventional hypothesis testing with an approach that fits what we otherwise think the preponderance standard means.
Through careful argumentation using Bayesian hypothesis testing, the Article offers a fundamental default rule for statistical estimation evidence: if the evidence comes from a credibly designed and implemented study, then it is presumptively admissible and enough to withstand motions for judgment under either Rule 50 or Rule 56, whenever it points in the direction of the party offering it. In many situations, the fundamental default rule may be cast as the policy that evidence is legally sufficient and presumptively admissible whenever it yields a finding of statistical significance at the 50 percent level--equivalently, whenever its p-value is less than 0.5. Thus, the Article indicates that many courts' current practice is far too strict with respect to statistical estimation evidence.
The Article also discusses the appropriate gatekeeping role for federal district courts under Federal Rules of Evidence 403 and 702, and it engages the question of when courts might legitimately move away from the fundamental default rule for policy reasons.
INTRODUCTION I. BAYESIAN HYPOTHESIS TESTING APPLIED TO LEGAL SUFFICIENCY AND THE PREPONDERANCE STANDARD YIELDS A FUNDAMENTAL DEFAULT RULE A. A Brief Description of Conventional Hypothesis Testing B. The Bayesian Hypothesis Testing Approach C. Why Bayesian Hypothesis Testing is the Right Framework For Legal Sufficiency D. The Fundamental Default Rule Results from Viewing the Evidence in the Light Most Favorable to the Non-Movant E. Conventional Hypothesis Testing Leads to an Inappropriate Rule For Determining Legal Sufficiency of Statistical Estimation Evidence, Unless the Significance Level of 30% is Used F. Summary of Legal Sufficiency and the Preponderance Standard II. HOW TO ANALYZE ADMISSIBILITY OF STATISTICAL ESTIMATION EVIDENCE UNDER THE FEDERAL RULES AND ASSOCIATED CASE LAW A. Reliability Under Rule 702 and the Daubert Trilogy B. Understanding the Concepts of Strength and Credibility of Estimation Evidence 1. Technical Implementation 2. Credibility for the Proffered Purpose 3. Analogy to Non-Statistical Evidence C. The Proper Application of Rule 702 to Estimation Evidence Is Friendly to the Fundamental Default Rule 1. Rule 702 and the Daubert Trilogy Do Not Allow Gatekeeping Based on Conventional Hypothesis Testing at Conventional Significance Levels a. Daubert b. Joiner c. Kumho Tire 2. The Proper Focus in Gatekeeping is on Technical Implementation and Credibility for the Proper Purpose a. Credibility for the Proffered Purpose: Mayor of Philadelphia v. Educational Equality League b. Credibility for the Proffered Purpose: Merck v. Garza c. Cherry Picking d. Summary III. Policy Considerations Related to the Fundamental Default Rule A. The Fundamental Default Rule Would Improve Administrability and Litigation Practice B. The Quantity of Litigation Activity Might or Might Not Increase with the Fundamental Default Rule, but the Rule Likely Would Shift Bargaining Power to Plaintiffs IV. FEDERAL COURTS COULD USE COMMON LAW POWERS TO ADJUST THE EVIDENTIARY STANDARD IN CASES BASED ON FEDERAL LAW A. In Cases Based on State Law Claims, the State's Law as to Statistical Estimation Evidence Should Control in Federal Court B. The Supreme Court Has Common Law Powers to Alter the Standard of Evidence for Federal Law Claims, but It Should Use Them Transparently Rather Than Characterizing These Powers in Terms of the Federal Rules of Evidence CONCLUSION APPENDIX: MULTIPLE STUDIES INTRODUCTION
This Article sets forth a theory of how federal courts should handle statistical estimation evidence--quantitative estimates with hypothesis testing used to quantify its strength--in civil litigation under certain important conditions. (1) An example of statistical estimation evidence, discussed in detail below, is the use of data from a randomized controlled trial of the effects of the drug Lipitor, which thousands of plaintiffs alleged caused them to develop Type 2 diabetes. (2) In the litigation that ensued, this statistical estimation evidence went to the question of general causation--whether Lipitor does, in general, cause diabetes. The disposition of many cases turned on this question.
In addition to mass torts such as the Lipitor case just noted, statistical evidence plays an important role in federal litigation involving other important areas of the law, including employment discrimination cases involving race-or sex-based differences in promotion rates; securities fraud litigation involving changes in stock prices on dates of alleged corrective disclosures; and antitrust cases involving effects on prices in concentrated industries.
My central claim in this Article is that litigants and courts are applying the wrong standard when--as has often been the case--they use statistical standards conventionally used by scholars in scholarly work. I develop this argument in several pieces. First, in Part I, I address the requirements imposed on statistical estimation evidence by the dominant preponderance standard for proof in civil litigation. In this Part, I summarize the conventional hypothesis testing approach, which is based on statistical significance testing, and contrast it to the Bayesian hypothesis testing alternative. I then argue that the Bayesian hypothesis testing approach fits the preponderance standard much better than does the conventional hypothesis testing approach. And I develop what I call the "fundamental default rule of statistical estimation evidence." (3) Under this rule, when the preponderance standard applies, statistical estimation evidence should be considered legally sufficient and presumptively admissible whenever it points in the direction of the party proffering it. As I discuss in Section I.E, this result may be understood in terms of conventional hypothesis testing methodology, but with the unusually high significance level of 50 percent. (4)
In Part II, I turn to issues related to admissibility of expert testimony related to statistical estimation evidence. If estimation evidence makes it into the trial record at all, it is virtually always through that channel. Litigants battle to get their experts' testimony admitted and to exclude the other side's. With respect to evidence deemed admissible, the parties struggle just as vigorously over summary judgment. Parties dispute the methods each other's experts use, they argue that claimed statistical significance is illusory, and sometimes they even accuse each other's experts of doing "junk science" in violation of the standards set forth by Federal Rule of Evidence 702 and the Daubert trilogy. (5) Some judges display a sophisticated grasp of statistical concepts in determining which expert testimony to admit, and which will be deemed legally sufficient. (6) Others struggle to referee the battle of the experts.
Litigants and courts frequently focus on conventional hypothesis testing, by which I mean null hypothesis significance testing, typically at the significance level of 5%, because that is the approach many statistics-using scholars take in their scholarly activities. But as Professor Stephen Burbank once wrote: "Courtrooms are not laboratories." (7) Professor Frederick Schauer has elaborated on this point, explaining that sometimes "bad science makes good law." (8) As I shall argue, acting otherwise leads to statistical standards that map poorly onto the legal standards that courts otherwise say they use for civil litigation. (9)
In Part III, I turn to policy considerations that adopting my analysis and standard would implicate. These relate to administrability, which would be greatly improved, and to the social costs and benefits associated with what would likely be a shift in bargaining power toward various types of plaintiffs in complex litigation.
In Part IV, I ask whether courts might prefer to apply more demanding standards than the preponderance standard, given my claims about that standard in Parts I and II. This is a classic question about the legitimate powers of courts to make substantive law. Whereas state courts generally have such powers, the federal courts on which I concentrate in this Article have such powers only in limited circumstances. I address these circumstances, differentiating between claims rooted in state law--which raise interesting but manageable Erie-related questions--from those involving only federal law. Federal courts in some instances have the power to impose the standards that I argue, in Parts I and II, fail to match the preponderance standard. But because adopting such standards involves substantive lawmaking, federal courts owe litigants and the public more transparency about that activity than they have provided. Rule of law values demand more in a system of government founded on predictable laws and observable lawmaking. Throughout this Article I assume for simplicity that the party offering the evidence is the plaintiff, as they have the burden of proof, and I consider how courts should handle either a defendant's challenge to the legal sufficiency of estimation evidence (via Rule 50 or Rule 56 of the Federal Rules of Civil Procedure), or a defendant's motion to exclude the plaintiff's expert testimony about that evidence (via Rule 702 of the Federal Rules of Evidence). I explain why common practice by litigants and courts fails to satisfy the preponderance standard. And I use a combination of black-letter doctrine and mathematical statistics to justify a simple alternative--what I call the fundamental default rule of estimation evidence. According to this...