THE PROFICIENCY OF EXPERTS.

• Author: Garrett, Brandon L.

INTRODUCTION 903 I. TRUE EXPERTISE = PROFICIENCY 910 A. Judicial Qualification of Experts Using Education and 910 Experience B. Identifying Experts Through Performance 914 1. Results: False Positive, False Negative, and Inconclusive 919 2. Nature of the Proficiency Study 921 C. A Case Study: Fingerprint Examiner Proficiency 924 II. JUDICIAL ATTITUDES TOWARD PROFICIENCY DATA 931 A. Disregarding Proficiency 935 B. Admissibility and Proficiency 936 1. Use of Proficiency Data to Exclude Evidence Entirely 937 2. Concerns with General Inadequacy of Proficiency Testing 939 3. Judicial Acceptance of Proficiency 940 a. General Proficiency 940 b. Individual and Lab Proficiency 942 c. Weight and Proficiency 943 d. Discover of Proficiency Data 944 e. Rethinking Proficiency and Judicial Gatekeeping 947 i. Proficiency, Qualification, and Admissibility 948 ii. Rethinking Proficiency in the Courtroom 948 III. REGULATING EXPERT EVIDENCE 950 A. Federal Regulation of Proficiency 950 B. Regulating the Quality of Proficiency Testing 954 C. International Approaches 957 CONCLUSION 958 INTRODUCTION

Expert witnesses appear in a vast number of cases every year. (1) In civil cases, experts often address questions central to liability and damages, and in criminal cases, they address questions touching on both guilt and punishment. Following the Supreme Court's ruling in Dauben v. Merrell Dow Pharmaceuticals, (2) and subsequent revisions to Federal Rule of Evidence 702 dealing with expert evidence, (3) judges now have a much greater authority and responsibility to inquire into the reliability and validity of expert's methods. The threshold question, however, is whether a person is "qualified" to be an expert based on "knowledge, skill, experience, training, or education." (4) To answer this question, courts routinely accept a witness's own self-serving statements of expertise buoyed by educational credentials, professional training, or experience, rarely spending much time on this threshold question before moving on to examine the methods used and conclusions reached by the putative expert. (5) In this Article, we seek to revitalize the expert qualification inquiry and encourage greater reflection on what should be required of expert witnesses.

What does it mean to label someone an "expert"? From a social scientific perspective, the label "expert" means something different than simply having specialized education or experience. It is not a matter of credentials or work history but rather a question of performance: "Expertise is defined as a sequence of mastered challenges with increasing levels of difficulty in specific areas of functioning." 6 Experts are those who are particularly proficient on a task or who are especially knowledgeable about a subject. The expert interpreter can translate one language into another with a high degree of accuracy across language samples that would remain inscrutable to non-experts. The expert cytologist can differentiate cancerous cells from non-cancerous cells with a high degree of accuracy and with a high degree of reliability when those samples are submitted for retesting, while most non-experts examining the samples would perform at levels no better than chance. An expert on American history can answer questions on arcane historical topics with a degree of accuracy and reliability far beyond that exhibited by the average person. Expertise may be acquired in many different ways, but anyone who claims to be an expert should be able to prove that expertise empirically through superior performance within the domain of purported expertise. (7) Ideally, assessments of expertise make use of a performance measure that indisputably separates good performance from bad. Where such a "gold standard" for good performance exists, (8) an expert can be subjected to what is commonly called proficiency testing. (9) In proficiency testing, the putative expert's response on a test can be objectively scored as correct or incorrect. Thus, a candidate to become a court interpreter can be presented with a number of foreign phrases whose English meanings are known and her interpretations can be evaluated for their accuracy. Or a cytologist can be presented with cell samples known to include cancerous and non-cancerous cells to assess how accurately the cytologist distinguishes the two types of cells. Proficiency testing also permits an assessment of an individual's reliability, or consistency, in performance: to what extent does the person give the same answers across like items and different answers across different items? (10) Increasingly difficult tests can be used to identify those most expert at a task.

Unlike judges who focus on the experience or credentials of an expert, social scientists who study experts emphasize performance over self-serving statements and credentials because the latter are often unreliable guides to true expertise. "Experts have often been identified by self-proclamation or acclamation by other experts as well as by experience, titles, and degrees. However, these methods can be misleading when searching for an expert." (11) In some domains, governments require that persons and organizations claiming to be experts on some task demonstrate that expertise empirically, through performance on proficiency tests tailored to that task. Thus, persons seeking to serve as court interpreters must pass proficiency tests designed to ensure expertise in the languages to be interpreted, (12) and clinical labs that screen human samples for diagnostic testing must participate in regular proficiency testing that produces results the public can examine and compare. (13)

In one large and important domain, however, neither federal nor state governments require performance-based evidence of expertise: in order to qualify as an expert witness at trial, a person need only proclaim that she has specialized knowledge or ability, obtained through education or experience, that would enable her to supply relevant information that a non-expert witness could not supply. (14) The testimony that the expert hopes to give is supposed to be the product of reliable methods applied to sufficient data, but there is no requirement that the expert demonstrate her proficiency at giving correct answers to the kinds of questions counsel will pose to her at trial. (15)

Take the example of fingerprint experts, who have testified in criminal cases for over a hundred years. (16) The latent fingerprint examiner uses some objective criteria initially, when categorizing a print as having a "whorl" or a "loop" pattern. (17) However, the analysis that follows is "purely subjective," requiring an evaluation of details in a print and comparing it to details in another print. (18) Such a subjective method can be valid and reliable. (19) But based on what we currently know, fingerprint analysis has a "substantial" error rate, making it especially important to assess how "expert" an individual examiner is at fingerprint identifications. (20) Yet, judges do not ask fingerprint examiners to come forward with evidence that they can correctly match latent prints to known prints with a high (or any) degree of proficiency. The examiner seeking to testify need only describe training and familiarity with the methods in the field to determine whether latent fingerprints from a crime scene match the fingerprints taken from known individuals. Judges do not inquire further even if she boasts that she has a zero or near-zero error rate in fingerprint identifications. Under present practice, as we will describe, both federal and state courts regularly accept proxies for expert performance in lieu of actual performance data. Judges also deny opposing parties access to proficiency testing data when it exists, despite unsupported claims by experts regarding their supposed proficiency. (21)

This odd state of affairs--in which faith is placed in self-proclamations of expertise and judges ignore the most probative evidence on expertise--cannot be attributed to the lack of gold-standard measures of performance in the areas in which experts seek to testify in court. An expert testifying in any area for which there are better and worse ways of doing a task or correct and incorrect answers to questions can be subjected to proficiency testing. These conditions apply, for instance, to medical experts in civil cases and forensic experts in criminal cases. (22) In fact, some fields of expertise already engage in proficiency testing, but only disappointingly few do so, and even fewer make their proficiency data public or willingly share it in discovery. These fields usually adopted proficiency testing when mandated to do so or as a part of an effort to regulate quality within a field to dispel credibility concerns, as with the move to proficiency testing by fingerprint examiner associations. Not only is proficiency testing commonly not done in many fields, but what little is done is often not sufficiently challenging or realistic in its design. A leading commercial provider for forensic proficiency tests candidly explained: "Easy tests are favored by the community." (23) Where an expert's primary, or perhaps only, market is the courtroom, as is the case with most forensic experts and a variety of experts in civil litigation, the expert has little to gain and much to lose from engaging voluntarily in proficiency testing. Absent a mandate from courts or regulators, widespread proficiency testing is unlikely to occur.

Only recently has this proficiency problem begun to receive real attention. Most notably, the White House Presidential Council for Advisers on Science and Technology (PCAST) issued a report in September 2016 underscoring the "essential" need for proficiency testing of forensic experts to assess "an examiner's capability and performance in making accurate judgments," in a manner that is realistic, routine, and under the supervision of a disinterested...