Evaluating Empirical Research Methods: Using Empirical Research in Law and Policy

• Publication year: 2021
• Citation: Vol. 81

81 Nebraska L. Rev. 777. Evaluating Empirical Research Methods: Using Empirical Research in Law and Policy

777

Evaluating Empirical Research Methods: Using Empirical Research in Law and Policy

Jennifer K. Robbennolt*

TABLE OF CONTENTS

I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 777
II. Social Science Methods and Legal Questions . . . . . . . . . . . 779
A. Archival Research - The Nebraska Death Penalty
Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 780
B. Experimental Research - Death Qualification and
Penalty Phase Instructions. . . . . . . . . . . . . . . . . . 785
III. Obstacles to Evaluating Empirical Methodology. . . . . . . . . . 789
A. Biased Assimilation . . . . . . . . . . . . . . . . . . . . . 790
B. Methodological Background . . . . . . . . . . . . . . . . . . 795
IV. Strategies for Evaluating Empirical Research . . . . . . . . . . 797
A. Explore the Tensions. . . . . . . . . . . . . . . . . . . . . 798
B. Research in Context . . . . . . . . . . . . . . . . . . . . . 799
C. Methodological Training . . . . . . . . . . . . . . . . . . . 801
V. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 804

I. INTRODUCTION

David Baldus and his colleagues have conducted an extensive and useful study of capital punishment decisionmaking in the state of Nebraska. (fn1) Their research is a fine example of how empirical research can be relevant to questions of law and policy. In determining whether to have a system of capital punishment and, if so, how to structure such a system, it is extremely useful to have data about various aspects of how the existing system operates. The same can be said for innumerable legal and policy issues ranging from various aspects

778

of jury decisionmaking in both criminal and civil cases,(fn2) to eyewitness identifications,(fn3) to inheritance policy,(fn4) to affirmative action,(fn5) and to contract law,(fn6) to mention just a few. Legislators routinely consider issues that are informed by empirical research and judges are called upon to act as gatekeepers who regulate the admissibility of scientific evidence in court.(fn7)

In any discussion of the relevance and usefulness of empirical research in the law, there is, however, persistent tension between the methods of social science and the theory, goals, and settings of law and policy. As David Faigman notes,

A more vexing aspect of data collection for the law comes from the basic incongruity between what scientists study and what the law is interested in knowing. . . . The law invariably relies on applied science, which typically involves extrapolating from controlled laboratory tests that are highly artificial or generalizing from noncontrolled field tests that contain many confounding variables.(fn8)

Using the study of capital punishment in Nebraska as a starting point, the purpose of this paper is to explore more broadly the tensions and tradeoffs at the intersection of social science methodology and the law from the perspective of legal and policy decisionmakers who are called upon to utilize empirical research. Those who use social scientific research to inform the law must do more than distinguish "good" research from "bad," they must also face the inevitable question of how to appropriately use well-done, but inherently imperfect research, for legal and policy purposes.

779

Toward this end, Part II uses empirical research on capital punishment decisionmaking to illustrate the tension between methodological concerns and legal theory or settings. The first example is drawn from the recent study of capital punishment in Nebraska conducted by Baldus and his colleagues and demonstrates a tension between methodological concerns and legal theory. The second example draws on experimental research that examines the effects of death qualifying jurors and the comprehensibility of capital punishment penalty phase jury instructions. This example highlights the tradeoff between realism and experimental control, a tension inherent in social science research that has implications for how empirical research is used in the law. Part III identifies several obstacles to careful consideration of these tensions and tradeoffs, and suggests that cognitive bias and a lack of familiarity with the scientific method can interfere with evaluations of the methodology of empirical research. Part IV suggests ways of overcoming these obstacles in order to better achieve careful consideration of the tensions and tradeoffs that exist at the intersection of law and methodology.

II. SOCIAL SCIENCE METHODS AND LEGAL QUESTIONS

There are any number of methodological approaches to addressing research questions that have relevance for law and policy, each with its own advantages and limitations. In considering the relevance of empirical research for answering legal and policy questions, decisionmakers ought to be concerned about several different aspects of the research. For example, those evaluating empirical research should be concerned that the measures used to quantify legal concepts are sufficient to capture the construct at issue (construct validity),(fn9) the degree to which the research design allows inferences to be drawn about causal relationships between variables (internal validity),(fn10) and the degree to which the research findings can be generalized to persons, times, and settings beyond those in which the research was conducted (external validity).(fn11) The following sections illustrate some of the tensions between these methodological concerns and legal concepts or settings.

780

A. Archival Research - The Nebraska Death Penalty Study

The study of capital punishment in Nebraska recently conducted by Baldus and his colleagues is an example of archival research in which the researchers measure variables present in actual cases and attempt to identify relationships between those variables. Archival studies, such as this one, have the important benefit of high external validity. That is, this study was able to examine decisions from real cases in which judges and prosecutors were faced with rich case facts and subtleties of evidence, and were required to make decisions about sentences for real defendants. Archival research in which case characteristics and decisions are carefully coded is extremely useful for identifying patterns and associations between aspects of the cases and the decisions that are made.(fn12)

Specifically, the study was largely concerned with the relationship between culpability and sentencing outcomes. Thus, the study examined the disposition of death-eligible cases over a twenty-six year period, measuring culpability primarily by looking at the record in each case and coding "the strength of evidence of each of the statutory aggravating and mitigating circumstances" in order to examine the ways in which dispositions varied with culpability.(fn13)

781

In order to quantify different levels of the legally relevant concept of case "culpability," some degree of abstraction is necessary; that is, the concept of culpability must be operationalized so that it may be measured. In measuring culpability, the study relied heavily on counting the number of statutory aggravating and mitigating factors in each case.(fn14) There are several good justifications for this approach. While a wide variety of case characteristics could be and, in fact, were coded, the small sample of capital cases in Nebraska precludes the use of a large number of variables in the statistical analyses.(fn15) Accordingly, counting statutory aggravators and mitigators is a relatively straightforward way in which to measure defendant culpability. Such an approach is quantifiable, it is based on factors that the legislature has deemed to be notably relevant case characteristics, and it likely captures many of the case factors that are important to decisionmak-

781

ing. Indeed, these measures appear to have had a high degree of predictive power.(fn16)

Nonetheless, these measures do not completely correspond to the legal theory of capital punishment, which emphasizes individualized assessment of each case. The Supreme Court has emphasized that the decisionmaker is to make an "individualized decision" about whether a particular defendant should be sentenced to death.(fn17) In Woodson v. North Carolina, the Court found that:

A process that accords no significance to relevant facets of the character and record of the individual offender or the circumstances of the particular offense excludes from consideration in fixing the ultimate punishment of death the possibility of compassionate or mitigating factors stemming from the diverse frailties of humankind. It treats all persons convicted of a designated offense not as uniquely individual human beings, but as members of a faceless, undifferentiated mass to be subjected to the blind infliction of the penalty of death.(fn18)

The Court concluded that, "in capital cases the fundamental respect for humanity underlying the Eighth Amendment requires consideration of the character and record of the individual offender and the circumstances of the particular offense as a constitutionally indispensable part of the process of inflicting the penalty of death."(fn19) Consistent with this principle, the Court has held that the sentencer must be able to consider, "as a mitigating factor, any aspect of a defendant's character or record and any of the circumstances of the offense that the defendant proffers as a basis for a sentence less than death."(fn20)

782

Nebraska law regarding capital sentencing is consistent with these dictates. In particular, Nebraska Revised Statute section 29-2522 requires that sentencing judges consider "[w]hether sufficient aggravating circumstances exist to justify imposition of a sentence of death" and "[w]hether...