Empirical doctrine.

• Author: Allen, Jessie
• Position: Using empirical studies to measure legal doctrines determinacy

Abstract

We can observe and measure how legal decision makers use formal legal authorities, but there is no way to empirically test the determinative capacity of legal doctrine itself. Yet discussions of empirical studies of judicial behavior sometimes conflate judges' attention to legal rules with legal rules determining outcomes. Doctrinal determinacy is not the same thing as legal predictability. The extent to which legal outcomes are predictable in given contexts is surely testable empirically. But the idea that doctrine's capacity to produce or limit those outcomes can be measured empirically is fundamentally misguided. The problem is that to measure doctrinal determinacy, we would have to adopt standards of legal correctness that violate fundamental conceptual and normative aspects of the legal institution we wish to study. In practice the promise of empirical data on doctrinal determinacy makes it seem less urgent to investigate other contributions doctrinal reasoning makes to law. Doctrinal reasoning might affect decision makers in ways that contribute importantly to the legal process without determining outcomes. Trying to understand those effects is a research project to which the empirical methods of social science have much to contribute.

Contents Introduction I. Seeing Doctrinal Indeterminacy as an Empirical Question. II. Doctrinal Determinacy Is Not Measurable Empirically A. Existing Empirical Studies Do Not Measure Doctrinal Determinacy. 1. Looking for Correlations between Judges' Characteristics and Legal Outcomes in Large Databases of Judicial Opinions 2. Studies of Legal Authority's Influence on Judges 3. Testing "Motivated Reasoning" in Legal Decision-Making B. No Imaginable Empirical Method Could Measure Doctrinal Determinacy 1. Imagining a Test 2. The Objection from Scientific Method 3. Very, Very Hard Cases III. Why We Should Care About the Impossibility of Measuring Legal Determinacy Empirically Conclusion Introduction

"adjective: empirical based on, concerned with, or verifiable by observation or experience rather than theory or pure logic."

--Concise Oxford English Dictionary (1)

For most of the last century, legal theorists ignored the work of political scientists who studied judicial decision-making. But lately legal scholars look to empiricists to validate or reevaluate conceptual theories about how law works. Empirical observations can certainly reveal whether legal decision makers pay attention to formal legal doctrine. And empirical methods can measure the predictability of legal outcomes in various contexts. But sometimes claims about the capacity of empirical legal studies appear to go further. Sometimes legal theorists suggest that, in a given jurisdiction, we could empirically test whether "there are cases, statutes, maxims, principles, canons, authorities, or statements in learned legal treatises available to justify decisions in favor of both parties in all or at least most litigated cases," (2) or, if the available legal authorities justify uniquely correct legal outcomes in most, or at least some, cases. (3) This is a claim that legal scholars could empirically measure "doctrinal determinacy." This article explains why that claim is false.

It matters whether doctrinal determinacy can be measured empirically. Judges, lawyers, and the general public all act as if legal doctrine is substantively determinate and at the same time profess to know that it is not. As Keith Bybee puts it:

On the one hand, judicial decision making is portrayed ... as a matter of legal principle and impartial reason ... guided by a shared set of publicly stated rules and norms. On the other hand, judicial decision making is portrayed ... as a matter of preference and politics, an activity largely driven by the personal beliefs and policy commitments that judges bring to the bench. (4) If we could measure doctrinal determinacy, we could reconcile these two perennially opposing views in a compromise position that recognized some limited form of determinacy. Such a compromise is desirable because faith in some degree of doctrinal determinacy underwrites the enforcement of judicial decisions, which, after all, continue to be written in doctrinal language.

This article aims to show why it is not possible to empirically measure the sort of doctrinal determinacy that could legitimize law enforcement. That does not mean rejecting empirical social science research as a method for understanding legal decision-making. In particular, the degree to which legal outcomes are predictable in given contexts is surely observable empirically and is an important piece of information. (5) But doctrinal determinacy is not the same thing as legal predictability. (6) Legal outcomes often are predictable because of factors that are not traceable to doctrine. (7) Predicting how a lawsuit will come out is not the same thing as demonstrating that the case should be decided only one way based on the available legal authorities, and that is the determinacy question. The bottom line is that empirical methods can be used to observe how judges engage with doctrine in reaching legal decisions, but there is no way to empirically measure doctrine's capacity to determine those decisions. (8)

The aim here is not to devalue empirical legal studies. Nor is the goal to criticize legal empiricists, who for the most part avoid conflating data on how judges go about using legal materials with evidence about the extent to which those materials produce determinate legal answers. Rather, this article aims to sweep away misguided interpretations of empirical work.

The promise of empirically observing the extent of substantive doctrinal determinacy tends to divert interest away from more promising empirical questions. So long as we remain stuck in a miasmic sense that doctrine must be somewhat determinate in a way that might be shown empirically, it is hard to motivate the search for other less familiar doctrinal effects. My next article will draw on some recent work in psychology to propose a way that formal legal doctrine might produce psychological and social effects that could contribute to a distinctively legal process, apart from producing determinate answers. This research is discussed briefly below. (9)

Much of my argument in this article is framed in opposition to assertions of Frederick Schauer, probably the foremost living proponent of law as a formal rule-based system. I take issue with Schauer's repeated suggestions that the extent of doctrinal determinacy, or indeterminacy, is an "empirical" question. (10) But the larger project is broadly congruent with Schauer's understanding of legal reasoning as a distinctive rule-focused social practice and has been greatly influenced by Schauer's work. Schauer has long argued that the distinctive quality of legal reasoning is related to its rule-based nature but does not depend entirely on whether legal rules determine outcomes. (11) Investigating what formal doctrinal reasoning contributes to legal reasoning, even if rules never substantively determine answers, is in some ways an extension of Schauer's emphasis on the phenomenology of rule-based reasoning in law.

On a methodological level, this article aims to contribute to New Legal Realism, an approach to legal scholarship that seeks to build "a bridge between formal law and the social sciences." (12) The original Realists of the early twentieth century tended to dismiss formal legal methods either as trivial window dressing or a pernicious obfuscation of the political forces that really drove legal outcomes. (13) But self-identified New Legal Realists consider that legal forms have meaning. (14) Clarifying what empirical studies can and cannot show about the role of legal doctrine should help advance the New Legal Realist inquiry into "when and how formal law matters." (15)

There is an urgent pragmatic reason for gaining a better understanding of what formal doctrinal reasoning contributes to legal decision-making. At least in the United States, there is a pronounced shift away from adjudication based on formal doctrinal analysis toward informal settlement and alternative dispute resolution. (16) Identified legal conflicts are increasingly resolved through mediation, arbitration, plea-bargaining, and negotiation. Part of the reason the shift toward less formal treatment of legal conflicts is deemed acceptable and thought not to destabilize the rule of law is the belief that these informal resolutions take place "in the shadow of the law." (17) The idea is that the results of these nondoctrinal resolutions are constrained, or at least shaped, by a background of substantive rules, principles, and values established by applicable legal authorities. But if legality resides more in the formal practice than the substance of doctrinal reasoning, by shifting to informal, all-things-considered negotiation we are giving up the aspect of our legal institutions that constitutes legality.

Something paradoxical is happening. In the last half of the twentieth century, court dockets were surging while, at the same time, social scientists who studied judicial decision-making were bent on showing that the legal doctrines courts ostensibly applied did not actually drive legal outcomes. Today, empirical legal studies sometimes hypothesize that doctrine is involved significantly in legal decisionmaking. At the same time, legal claims are more likely to be resolved nondoctrinally. Ironically, at the moment of increasing consensus that formal legal authorities "matter," we are busy dismantling the formal practices that have been the traditional mode of applying those authorities to conflicts. Once it seemed that skeptical critiques and evidence of judges' "attitudinal" decision-making threatened formal doctrinal practice. But now it is as though increased confidence in the substantive role of legal doctrine has made formal doctrinal reasoning...