CHAPTER 4 THE SCIENTIFIC METHOD AND "JUNK SCIENCE"

• Jurisdiction: United States

Chapter 4 The Scientific Method and "Junk Science"

Overview

Following the U.S. Supreme Court case in 1993 called Daubert v. Merrell Dow Pharmaceuticals, Inc.¹ all areas of forensic expert testimony were required to establish that they were scientifically "reliable." The first inquiry in establishing reliability is whether the area of forensic evidence meets the "scientific method." Daubert set down several non-exclusive tests to determine scientific reliability, including (1) whether the underlying method can be or has been tested; (2) whether the method has been subject to peer review and publication; (3) the method's known or potential error rate; and (4) the level of the method's acceptance within the relevant discipline.

In NRC Report, published in 2009, made various recommendations for more testing in forensic disciplines and validation of conclusions in almost all areas of forensics except DNA. We will examine this report and its conclusions in this chapter.

The term "junk science" was coined to refer to theories that do not meet the scientific method. This does mean that such a theory might not be later proved scientifically valid. It may mean simply that the theory has not yet been tested. For example, there is much brain research currently being done that has shown incapacity to form criminal intent and that may ultimately prove lack of "guilty knowledge" or propensity to commit certain crimes. This area, called neuroimaging, has not yet achieved general acceptance in the scientific community and despite arguments that it meets the Daubert tests for scientific reliability, it has been rejected by many courts:

... [D]effense lawyers have rushed to bring brain scans into courtrooms. Some of what they propose is out-and-out chicanery; some may hold real value; whatever the case, the job of piloting the public through the complex neuroscientific maze — in order that potential jurors may better judge whether a violent offender should be condemned to death, to a long or life sentence in America's barbaric present-day prison system, or should have their sentences reduced or changed because of a brain irregularity or insult—is vital to society.²

We will discuss neuroimaging as one of the areas of supposed "junk science."

The area of junk science that is most difficult to prove by the scientific method are hypotheses that exposure to certain chemicals or forces causes a specific disease. The reason is because in order to prove causation, the scientist must rule out all other possible causes. Isolating one particular agent to the exclusion of all others is a daunting scientific task. Daubert involved just such an hypothesis, namely, that ingesting the drug Bendectin caused birth defects.

For our purposes, we will concentrate on the forensic disciplines used in criminal cases. The science underlying these disciplines is generally directed either toward identifying an individual based on evidence such as fingerprints, DNA, handwriting, or the like, or demonstrating what likely occurred at a crime scene, for example, through examining blood spatter patterns or the trajectory of ballistics.

The NRC Report highlighted a number of problems that affect the scientific reliability of most forms of forensic evidence today:

With more and better educational programs, accredited laboratories, certified forensic practitioners, sound operational principles and procedures, and serious research to establish the limits and measures of performance in each discipline, forensic science experts will be better able to analyze evidence and coherently report their findings in the courts. The current situation, however, is seriously wanting, both because of the limitations of the judicial system and because of the many problems faced by the forensic science community.³

The scientific method requires that two different examiners must arrive at the same conclusion, assuming all variables are constant. This requirement of "replicability" is key to science. However, there are a number of areas of forensics that require subjective examination and conclusion, which precludes replicability.

In the area of forensics, there are a number of techniques that have not been proven under the scientific method to be reliable as a form of individual identification. These include bite-mark identification, lip print identification, palm print identification, bullet identification through lead composition, and neuroimaging, as well as more creative theories such as whether one can be identified by smell.

Chapter Objectives

Based on this chapter, students will be able to:

1. Explain the scientific method.
2. Identify the scientific hypotheses underlying fingerprints, DNA testing, eyewitness identification, firearm and tool mark identification, lip print and palm print identification, and neuroimaging.
3. Understand the concept of "error rate" in science.
4. Describe the concept of junk science.
5. Identify the reason various types of junk science fail to qualify under the scientific method.
6. Appreciate the difference in proving causation in a personal injury civil case and proving identity in a criminal case.
7. Articulate the arguments for and against admission of neuroimaging technology to prove mental state.

What Is the Scientific Method?

The dictionary definition of "science" is "the observation, identification, description, experimental investigation, and theoretical explanation of phenomena."⁴ In essence, a scientific "theory" results from the observation of phenomena, the creation of a hypothesis, and the testing of that hypothesis to explain or disprove the phenomena in question. A scientist and his or her colleagues create tests or experiments to try to test the validity of the hypothesis.

The National Research Council defined the scientific method as follows:

Scientists continually observe, test, and modify the body of knowledge. Rather than claiming absolute truth, science approaches truth either through breakthrough discoveries or incrementally, by testing theories repeatedly. Evidence is obtained through observations and measurements conducted in the natural setting or in the laboratory. In the laboratory, scientists can control and vary the conditions in order to isolate exclusive effects and thus better understand the factors that influence certain outcomes. Typically, experiments or observations must be conducted over a broad range of conditions before the roles of specific factors, patterns, or variables can be understood. Methods to reduce errors are part of the study design, so that, for example, the size of the study is chosen to provide sufficient statistical power to draw conclusions with a high level of confidence or to understand factors that might confound results. Throughout scientific investigations, the investigator must be as free from bias as possible, and practices are put in place to detect biases (such as those from measurements, human interpretation) and to minimize their effects on conclusions.⁵

Following the Daubert decision, in an effort to help federal judges assess scientific evidence, the Federal Judicial Center published an extensive Reference Manual on Scientific Evidence.⁶ David Goodstein, in his article "What Is Science," which appears in the manual, says:

If one asks a scientist the question, What is science?, the answer will almost surely be that science is a process, a way of examining the natural world and discovering important truths about it. In short, the essence of science is the scientific method.

Another author of the manual defined "good science" as follows:

Good science is usually described as dependent upon qualities such as falsifiable hypotheses, replication, verification, peer-review and publication, general acceptance, consensus, communalism, universalism, organized skepticism, neutrality, experiment/empiricism, objectivity, dispassionate observation, naturalistic explanation, and use of the scientific method.⁷

The scientific method requires these steps:

1. Hypothesis.
2. Observation and/or experimentation to prove the hypothesis false.
3. Sufficient sample size to draw valid conclusions.
4. Elimination of any alternative hypothesis for the results.
5. Explanation for any contradictions of the evidence.

The National Research Council prepared a detailed analysis of many forms of forensic evidence in 2009, and recommended more uniformity of training, common standards for forensic laboratories, and in particular, more scientific testing to establish scientific reliability for every area of forensic evidence it examined, except DNA. These include:

• fingerprints
• firearms examination
• tool marks
• bite marks
• impressions (tires, footwear)
• bloodstain pattern analysis
• handwriting
• hair
• DNA
• coatings (e.g., paint)
• chemicals (including drugs)
• materials (including fibers)
• fluids
• serology
• fire and explosive analysis
• digital evidence

Defining the Relevant Scientific Community

One important issue is who has done the scientific testing, whether they have any bias, and whether their results have been peer-viewed and accepted by other objective scientists. The Daubert case included as one test for reliability whether the science was "generally accepted in the scientific community." Yet, with the exception of DNA, law enforcement has generally been involved in developing forensic evidence testing without submitting their results to universities or outside laboratories for verification. This creates the question of whether the science has been tested and proven to the "Relevant Scientific Community." For example, in evaluating polygraph reliability, should a court look at the studies conducted by the American Polygraph Association? If they did, the reliability statistics would approach 80%.⁸ On the other hand, if the court looked at the myriad of websites that claim they can teach someone how to fool a polygraph in ten hours,⁹ it might come to the opposite conclusion.

Determining...