Do Jurors Give Appropriate Weight to Forensic Identification Evidence?

• Author: Suzanne O. Kaasa,William C. Thompson,Tiamoyo Peterson
• Published date: 01 June 2013
• DOI: http://doi.org/10.1111/jels.12013
• Date: 01 June 2013

Do Jurors Give Appropriate Weight to

Forensic Identification Evidence?

William C. Thompson, Suzanne O. Kaasa, and Tiamoyo Peterson*

Do jurors give appropriate weight to forensic identification evidence? When judging the

value of forensic evidence, are they sensitive to the probability of a false match? To answer

these questions, we conducted two jury simulation experiments—the first with undergradu-

ate participants, the second with members of a county jury pool. The experiments examined

the weight that participants gave to forensic DNA evidence relative to Bayesian norms when

evaluating a hypothetical criminal case. We found that aggregate judgments were generally

consistent with Bayesian expectations, although people overvalued the DNA evidence when

the probability of a false report of a match was high relative to the random match probability.

Judgments of the chances the defendant was guilty varied appropriately in response to the

variation in the probability of a false report of a match, as did verdicts. Our findings refute

claims that jurors are always conservative Bayesians when evaluating forensic evidence and

suggest, instead, that they use a variety of judgmental strategies and sometimes engage in

fallacious statistical reasoning. In light of these findings, we identify circumstances in which

forensic evidence may be overutilized, discuss implications for legal policy, and suggest

additional lines of research.

I. Introduction

In 2009, a National Research Council report on forensic science identified two sources of

error that must be considered when evaluating forensic identification evidence, such as

evidence of a DNA match. First, there is the possibility of a coincidental match: “The two

samples might actually come from different individuals whose DNA appears to be the same

within the discriminatory capability of the tests . . .” (National Research Council 2009:121).

Second, there is the possibility of a false report of a match: “two different DNA profiles

could be mistakenly determined to be matching” (2009:121). The NRC report declared

that “[b]oth sources of error need to be explored and quantified in order to arrive at

reliable error rate estimates for DNA analysis” (2009:121). It called for the development of

*Address correspondence to William C. Thompson, Department of Criminology, Law & Society, University of

California, Irvine, CA 92617; email: William.Thompson@uci.edu. Thompson is Professor of Criminology, Law &

Society and Professor of Law at UCI; Kaasa and Peterson recently completed doctoral studies in Psychology and Social

Behavior at UCI.

The research reported here was supported by National Science Foundation Grant No. SES-0617672. The authors

thank Mr. Alan Carlson, Chief Executive Officer and Jury Commissioner of the Superior Court of Orange County,

California, and his staff, for their assistance in completing this research.

bs_bs_banner

Journal of Empirical Legal Studies

Volume 10, Issue 2, 359–397, June 2013

359

“quantifiable measures of the reliability and accuracy of forensic analyses” not just for DNA

evidence but for all the forensic identification sciences (2009:23).

If this call is heeded, we are likely to see further development and use in the

courtroom of two types of statistics to characterize the value of forensic evidence. The first

statistic is the frequency in a reference population (or populations) of matching charac-

teristics or features. These frequencies are sometimes described as random match prob-

abilities (RMPs) because they are thought to reflect the probability that a randomly chosen

person or item from the reference population will have a characteristic or feature that links

them to the evidence—in other words, the probability of a coincidental match. At present,

RMPs are routinely presented in connection with forensic DNA evidence (Kaye &

Sensabaugh 2011), but are rarely presented in connection with other types of forensic

identification evidence. The second statistic, which we will call the false report probability

(FRP), is the probability that a match will mistakenly be reported between items that do not

share matching characteristics. Although FRPs have rarely been presented in court

(Thompson et al. 2003), empirical data have been collected on rates of falsely reported

matches in DNA testing (Koehler 1997; Roeder 1994), latent fingerprint analysis

(Thompson et al. in press; Tangen et al. 2011; Koehler 2008; Cole 2005), voice identifica-

tion (Gonzales-Rodriguez et al. 2006), and bullet lead analysis (National Research Council

2004; Thompson 2005). If the NRC’s recommendations for validation of forensic methods

are followed, additional data on the rate of false match reports will be collected and may

well be presented in connection with forensic evidence in jury trials.

These developments raise anew a longstanding debate over jurors’ ability to under-

stand and make appropriate use of such statistical data when evaluating forensic science

evidence. At stake in this debate are important policy and evidentiary questions about the

circumstances under which statistical evidence should be admissible in connection with

forensic evidence and how it should be presented. The debate began over 40 years ago

when Finkelstein and Fairley (1970) argued that jurors are likely to give too little weight to

forensic evidence unless they receive explicit instructions on how to use Bayes’s rule. They

cited evidence that people are “conservative” in Bayesian updating tasks involving proba-

bilistic evidence—that is, people tend to revise their opinions less than Bayes’s rule indi-

cates that they should in light of new evidence (Edwards 1968; Slovic & Lichtenstein 1971).1

Finkelstein and Fairley’s suggestion that jurors be instructed on Bayesian updating

drew a famously eloquent response from Laurence Tribe (1971), who objected to what he

saw as an unwarranted intrusion of mathematics in the trial process. Tribe acknowledged

the usefulness of Bayesian analysis as a way for experts to think about evidence, but he

objected on several grounds to the introduction of Bayesian decision aids into jury trials,

1Bayes’s rule describes how a judgment about the probability of an event should be revised in light of new evidence

(see generally Lempert 1977; Robertson & Vignaux 1995). In studies of probabilistic inference (e.g., Gigerenzer &

Hoffrage 1995; Tversky & Kahneman 1982), Bayes’s rule is often treated as a norm for assessing the weight people

assign to a piece of evidence. People are said to overvalue a piece of evidence, or to give it more weight than it

deserves, when they adjust a judgment about the probability of an event more than Bayes’s rule says they should in

light of new evidence; they are said to undervalue the evidence, or give it too little weight, when they adjust a

probability judgment less than Bayes’s rule says they should after receiving the evidence (Edwards 1968).

360 Thompson et al.

and he worried that jurors would misuse random match probabilities by mistakenly equat-

ing them with the probability of the defendant’s innocence. Tribe’s arguments led the

Minnesota Supreme Court to reject the use of RMPs entirely in connection with forensic

evidence, at least for a time,2although courts in other jurisdictions have generally admitted

RMPs (Kaye 2010:Ch. 2). Subsequent commentators (e.g., Saks & Kidd 1980) have largely

shared Finkelstein and Fairley’s concern that jurors will underutilize forensic evidence.

A number of empirical studies have compared the judgments of simulated jurors in

cases involving forensic evidence with Bayesian norms (Thompson & Schumann 1987;

Faigman & Baglioni 1988; Goodman 1992; Smith et al. 1996; Schklar & Diamond 1999;

Nance & Morris 2002, 2005; for reviews of the early studies, see Koehler 2001; Kaye &

Koehler 1991; Thompson 1989). Participants in these studies were sensitive to variations in

the random match probability—they gave more weight to the forensic evidence when the

RMP was low than when it was higher. In general, however, the judgments of simulated

jurors were “conservative” relative to Bayesian norms. They appeared to give less weight to

the forensic match than Bayesian models said they should, a finding that seemed to support

Finkelstein and Fairley’s concerns about undervaluation for forensic evidence, while allay-

ing Tribe’s concern that such statistics might be misused in a manner that would be

prejudicial.

But comparing human judgments to Bayesian norms is a tricky process involving a

number of potentially problematic assumptions (Slovic & Lichtenstein 1971; DuCharm

1970). The early studies presented no data on the false report probability (FRP). When

computing the Bayesian posterior probabilities to which jurors’ probabilistic judgments

were compared, the researchers implicitly assumed that the forensic evidence was error-

free. This is an assumption that study participants may not have shared (Navon 1981;

Schklar & Diamond 1999), hence participants’ apparent conservatism relative to Bayesian

norms might have been due, at least in part, to legitimate skepticism about the reliability

of the evidence rather than to a failure to use the evidence appropriately. Three more

recent studies (Nance & Morris 2005, 2002; Schklar and Diamond 1999) employed

better normative models that incorporated both the RMP and the FRP and found that

participants’ judgments were still, on average, more conservative than Bayesian norms.

However, all three studies had another methodological limitation: they were designed in

a manner that made it difficult for participants to give judgments that exceeded Bayesian

norms.

These studies asked mock jurors to estimate the probability of a defendant’s guilt in

a hypothetical case on a scale of 0–100 percent (or probability of 0–1.0). Some of the

estimates were made before jurors learned about incriminating forensic evidence and were

called “prior probabilities”; other estimates were made after jurors learned about the

forensic evidence and were called “posterior probabilities.” To determine whether jurors

were giving appropriate weight to the forensic evidence, the researchers compared the

2The court found error in the admission of frequency statistics in State v. Carlson, 267 N.W.2d 170 (Minn. 1978) (hair

evidence); State v. Boyd, 331 N.W.2d 480 (Minn. 1983) (serological evidence); and State v. Kim, 398 N.W.2d 544 (Minn.

1987) (serological evidence); but see State v. Bloom, 516 N.W.2d 159 (Minn. 1994) (allowing frequency statistics in

connection with DNA evidence).

Do Jurors Appropriately Weigh Forensic Identification Evidence? 361