Whose fault? - Daubert, the NAS report, and the notion of error in forensic science.

• Author: Risinger, D. Michael
• Position: National Academy of Sciences

"Handwriting is even more precise than DNA evidence for identification purposes."(1)

The notion of "error" and "error rates" is central both to the Daubert opinion (2) and to the recent NAS Report on the strengths and weaknesses of forensic science in the United States. (3) As might be expected, the NAS Report does a better job of explaining the kinds of error it is concerned with than did the opinion in Daubert. However, both to a greater or lesser degree fall short of a full consideration of the concept of error, and so doing, they invite confusion about how inaccurate results in criminal adjudication may occur, and who if anyone is to blame.

When I set out to write on this state of affairs, I was not particularly surprised by it. Courts at all levels are at times notoriously imprecise about important concepts, and the NAS Committee that generated the report was operating in a setting and in a tradition where the notion of error seemed to them, perhaps wrongly, to be intuitively obvious. What I was surprised about when I looked into the matter was that this unexamined approach to the concept of error prevailed in the very discipline where one would expect it to have been carefully taxonomized and theorized to a fare-thee-well in at least a dozen different ways, that is, philosophy in general and epistemology in particular. Nor is this condition any great secret within the philosophical literature. (4) It has been repeatedly noted as a glaring lacuna for well over a century. (5) However, very little appears to have been done about it. (6)

Needless to say, I will not be attempting a full-scale examination of the concept of error in this paper. However, I believe there are some observations that can be made that may be helpful in domesticating in helpful ways the notion of error as it might apply to forensic science expertise. Error in relation to forensic science presents fewer difficulties than a fully generalized treatment, because there are certain problems necessarily taken on by a full scale philosophic treatment that can be safely put aside. The issues of radical skepticism and the very possibilities of knowledge and error can be properly assumed away, for instance, because the givens of the law as a practical enterprise resolve those for the purposes of the law. (7) And the difficult issue of normative error can also be sidestepped, (8) because forensic science explicitly deals only with conclusions about empirical facts.

However, limiting ourselves to the notion of factual error still leaves plenty to do. To begin at the beginning, such error can only exist as a function of the human mind. There can be no error except in regard to a belief or action that is judged at a later time by some human agent to have been wrong in some respect, by some invoked criterion. A mouse's stillbirth is not an error, merely an event (although it may be referred to in anthropomorphic metaphor, and somewhat misleadingly, as an "error of nature"). Independent of sentient belief, purpose or action, there is no error in the natural world. Whatever happens simply is. For our purposes, only humans can make errors, or be in error. (9)

There are in fact two different fundamental approaches to the concept of error which are important to consider, and which are significantly in tension with each other. (10) We may label them the normative idea of error and, for want of a better term, the objective idea of error. Failing to separate the two can lead to various confusions and troubles.

Although in the end it is not really important which is taken as primary, I regard the normative notion as foundational both in normal usage and in underlying psychology. In this most fundamental sense, an assertion of error entails a claim or charge of both mistake and fault, that is, a proposition that asserts that a belief, claim, or action is or was wrong in some respect, coupled with an argument invoking a ground to assert its wrongness which is made in such a way as also to assert that the person whose belief, claim, or action is under consideration was at fault in taking the action, or was at fault at the time in holding the belief or asserting the claim, or at least would now be at fault in continuing to do so. This in turn entails the proposition that there can be legitimate grounds to assert the wrongness of a belief or action, and to criticize another for so believing or acting. At base, an assertion of error in this sense is an attack on both the truth of and the justification for an explicit or implied (through action) knowledge claim, thus connecting the concept of error intimately with the most common traditional definition of knowledge as "justified true belief." (11)

On a general level, the territory of belief, claim, or action subject to a potential charge of error is coterminous with the extent of potential beliefs, claims, or actions. The grounds for charging error of belief or claim are at least equally extensive, while the grounds for charging error of action include attacks on any belief or claim that motivated the action, plus attacks on the propriety or rationality of the action even given the accuracy of those beliefs or claims.

Again on a general level, the range of grounds for charging error include such things as divine revelation, logical inconsistency, esthetic preference, and a multitude more. For our purposes we must very quickly narrow both the kinds of belief, claim, or action under consideration, and the grounds that will be considered in-bounds for making a charge of error. After all, our main focus is expertise proffered in courts of law, and forensic science as it is currently practiced, and the kinds of error and the kinds of argument properly under consideration in this context are considerably (and blessedly) much narrower than the entire possible field of the erroneous.

Within this more constrained field, the notion of error and the acceptable grounds for asserting error become much more limited. First, as previously noted, we are dealing with beliefs or claims concerning empirical facts in the world. Both the grounds for justifying such beliefs and the grounds for attacking them are limited, in general, by notions of empirical evidence, either of the critical common sense variety, or of the formal variety which is the domain of science in the modern sense of the word. (12) So a charge of error in the normative sense is a charge that the person or group has beliefs that are unwarranted by empirical evidence, or undertakes various practices the results of which do not mean or deliver what is claimed for them. (13) In this sense, a claim of error may be a profoundly serious moral claim, especially if the alleged errors are taken as truth by various social actors in a way that injures a human or a group of humans.

However, there is another way the term "error" is commonly used, especially in the setting of science, (14) and most especially in the science of testing (the most relevant scientific discipline for our purposes, as we shall see), where there is no normative charge at all, at least in the primary setting in which the word "error" is invoked. This approach to "error" only applies to results, and it is a purely post hoc judgment. In this context, a decision based upon a belief that something is the case, or even that is it most likely to be the case, is an error if it turns out wrong, no matter how strong the warrant for the belief. On the other hand, a decision that turns out right because the hoped-for result obtains, is generally not an error. (15) A set of four hypotheticals will illustrate the contrast between the two notions of error.

Assume that, in a basic five-card draw poker game, a player draws to an inside straight believing this to be a good strategy. (16) He does not fill the straight. Was the decision to draw an error? Clearly from the objective point of view the answer must be yes. The result was not the one intended or desired or hoped for, and it redounded to the player's detriment in the player's own terms. But what about the normative perspective? Here the answer is likely to be yes also. One of the first general rules one learns in regard to playing poker is that it is a bad gamble to draw to an inside straight. The chances of filling it are remote, and the hand without the fill is nearly worthless. So by the standards of intelligent poker play, assuming the object is to win, the belief that drawing to an inside straight is an intelligent strategy, whether based on ignorance or mistaken belief that could have been avoided by reflection or education, or on a belief in the special instinctive hunch powers of the player in spite of knowing the odds, is an error, and the act based on that belief, is an error. And this would be so whether the player filled the straight or not (which will happen a little more than once every twelve attempts). (17) A belief that this is a good strategy is still an error, and a lucky shot even knowing the odds doesn't change the arguments concerning the nature of the action (decision), since this is always analyzed ex ante. The "no harm, no foul" principle does not apply to the determination of normative error.

But in the objective framework, the filled straights would not count as "errors" at all, since this is always analyzed ex post merely by reference to desired results.

Now consider this further homely hypothetical. A person notices that the (donated) prize at a church lottery is worth more than the total cost of the one thousand lottery tickets to be sold. He has arrived after only one ticket has been sold. He buys the other 999 tickets. The only ticket he has not bought is the winner. Was it an error to buy the 999 tickets?

From the objective perspective, the ticket purchase would be an error, since the decision came out wrong, albeit against long odds. Again, the classification as error rests on comparing actual outcomes to desired...