Sample Data as Evidence: Meeting the Requirements of Daubert and the Recently Amended Federal Rules of Evidence

• Publication year: 2010

Sample Data as Evidence: Meeting the Requirements of Daubert and the Recently Amended Federal Rules of Evidence

Gregory Todd Jones and Reidar Hagtvedt

Introduction

"You don't have to eat the whole ox to know the hide is tough."[1] As the complexity of business litigation grows, it has become increasingly clear that accounting for all of the persons or things considered relevant in a particular dispute, or "eating the whole ox," if you will, is an unnecessarily onerous, nearly impossible task. Instead, litigants and the courts have often turned to statistical sampling, which involves collecting samples from a given population—a small subset of the relevant persons or things—deriving statistics from this sample data, and arriving at conclusions regarding the population based upon these sample statistics. This has not always been the case. In James S. Kirk & Co. v. Federal Trade Commission,[2] a case involving a dispute over a manufacturer's claim that its soap was made from olive oil, an administrative law judge refused to admit sampling evidence and instead heard testimony from seven hundred women regarding their understanding of the manufacturer's claim.[3]

However, with the development and expanding deployment of scientific survey and sampling methods, courts have gradually become more willing to admit evidence based upon sample data.[4] The Federal Rules of Evidence now specifically provide a basis for ruling favorably on the admissibility of sampling data in the form of public opinion poll evidence.[5] Further, the Federal Judicial Center's Manual for Complex Litigation supports the use of sampling, noting:

Statistical methods may often be useful to estimate, to specified levels of accuracy, the characteristics of a "population" or "universe" of events, transactions, attitudes, or opinions by observing those characteristics in a relatively small segment or "sample" of the population. The use of acceptable sampling techniques, in lieu of discovery and presentation of voluminous data from the entire population, may produce substantial savings in time and expense. In some cases, sampling techniques may provide the only practicable means to collect and present relevant data.[6]

Indeed, statistical sample analysis has been undertaken in many different types of litigation, including motions for change in venue,[7] trademark and advertising cases,[8] pornography prosecutions,[9] drug trafficking prosecutions,[10] copyright infringement and software piracy litigation,[11] the auditing of waste, fraud, and abuse of welfare programs,[12] labor and employment litigation,[13] and a wide variety of other contexts.[14] In fact, some courts have conducted sampling studies sua sponte.[15] In 1993, the Supreme Court articulated new standards for judicial admissibility of scientific expert testimony.[16] In Daubert v. Merrell Dow Pharmaceuticals,[17] the Court ruled that the Federal Rules of Evidence superceded the seventy-year-old standard[18] for the admissibility of scientific evidence originally articulated in Frye v. United States.[19] The Court explicitly rejected Frye's general acceptance requirement, relegating general acceptance to merely one of many factors, including examination of error rates and the testing of expert theories, to be weighed in determining admissibility.[20] Under Daubert, the courts are "gatekeepers" that function to insure expert evidence is "good science."[21]

In December 2000, an amendment to Federal Rule of Evidence 702[22] (FRE 702) took effect. Many observers believe the new rule does little more than codify Daubert and its progeny.[23]

Prior to the amendment, FRE 702 read in its entirety:

If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education, may testify thereto in the form of an opinion or otherwise.[24]

The December 2000 amendment added language that qualifies the nature of the data upon which experts may base their testimony:

If scientific, technical, or other specialized knowledge will assist the trier of fact to understand the evidence or to determine a fact in issue, a witness qualified as an expert by knowledge, skill, experience, training, or education, may testify thereto in the form of an opinion or otherwise, if (1) the testimony is based upon sufficient facts or data, (2) the testimony is the product of reliable principles and methods, and (3) the witness has applied the principles and methods reliably to the facts of the case.[25]

This new rule creates a mishmash of sometimes overlapping guidelines that focus on the expert's qualifications, the sufficiency of the expert's facts or data, the reliability of the principles and methods employed, and the reliability of applying these principles and methods to the specific facts of a given case.[26]

Most often, courts allow statistical sampling evidence as the basis for expert opinion testimony subject to FRE 703.[27] And like any other expert testimony, this testimony must satisfy the restrictions of Daubert and FRE 702. This Article explores the relationship between these restrictions and testimony related to statistical sampling evidence. Part I briefly describes the development of statistical sampling as an acceptable type of evidence. Part II explores the history of the Daubert rule and the subsequent amendment to FRE 702, examining the manner in which various courts have applied the new rule. This examination is organized according to the primary requirements of Daubert and FRE 702. First, Part II considers expert qualification, which forms a threshold requirement for the admissibility of expert evidence preceding any analysis based upon the factors delineated in Daubert and FRE 702. Second, Part II examines the three prongs of the Daubert test, arguably codified in the newly amended version of FRE 702, namely: 1) the requirement of sufficient facts or data; 2) the requirement of reliable principles and methods; and 3) the requirement that these principles and methods be reliably applied. Finally, Part II contemplates two procedural issues brought to bear under a Daubert analysis—the mandated analysis of the burden of proof with regard to reliability and the question concerning when to employ the new FRE 702. Part III outlines statistical sampling principles that speak to the sufficiency and reliability questions raised by Daubert and FRE 702. Part III also offers practical guidance to assist in shoring up statistical sampling evidence to prevent its exclusion, providing a checklist that may be employed to assail similar evidence offered by adversaries.

I. The Waxing of Sampling Evidence

Despite a long tradition illustrating practical and reliable use of statistical samples in other social science disciplines, courts have not historically warmed up to the idea that such samples are appropriately admissible evidence.[28] Ironically, a case where the court refused to admit sample evidence became a watershed in the changing attitudes toward such evidence. In Sears, Roebuck & Co. v. City of Inglewood,[29] Sears sued the city of Inglewood, California, seeking a sales tax refund Sears mistakenly paid.[30] To substantiate its $27,000 claim, Sears hired experts to audit a probability sample of sales slips from 33 out of the 286 days in question.[31] The experts estimated the overpaid taxes to be $28,250 with a standard error of $1150, but this was unacceptable to the judge who demanded they audit the entire inventory of 950,000 sales slips.[32] This arduous tabulation resulted in a figure of $26,750.22.[33]

As early as 1953, however, the court in United States v. United Shoe Machinery Corp.[34] directly endorsed sampling. United Shoe was an antitrust case in which the United States government charged the defendant with violations of the Sherman Act.[35] In its defense, United Shoe intended to call a large number of their customers as witnesses. The presiding judge, Charles Wyzanski, chose instead to utilize a sample of customers selected at "random" by means of a scheme of his own design.[36] In his opinion, Judge Wyzanski wrote: "If antitrust trials are to be kept manageable, samples must be used."[37] In later writings, he offered more detailed viewpoints:

[T]he judge can perform a useful function if he, through pre-trial conferences or at a later stage of the litigation when he is more aware of its dimensions, provides for appropriate samplings of the conduct and the effects. If the judge is fortunate, the parties may agree on the sampling. But where they do not, it seems to me to be the judge's responsibility first to elicit from witnesses on the stand the criteria necessary to determine what are fair samples and then to direct the parties to prepare such samples for examination and cross-examination. Sampling will make the record not merely more informative but shorter.[38]

Litigants often use sampling evidence in today's courtrooms with the explicit support of the Federal Rules of Evidence and the Federal Judicial Center's Manual for Complex Litigation.[39] "The burden of proof rests upon the [proponent] to show, through testimony of an appropriate expert, that the sample was selected in accordance with these principles so that its results can be projected to the

population."[40] However, "‘[o]nce this is established, there remain only questions of relevancy, materiality, and weight.'"[41]

II. Daubert and FRE 702

The admissibility of expert testimony, including that of experts seeking to testify regarding conclusions reached through the application of statistical sampling methods, can be outcome determinative in modern day litigation.[42] Further, the difficulties faced by litigators are not confined to the federal courts, which are bound by Daubert and FRE 702. In fact, many states have adopted Daubert either by direct reference or by...