Academic integrity and legal scholarship in the wake of Exxon Shipping, footnote 17.

• Author: Epstein, Lee

INTRODUCTION

Midway through his majority opinion in Exxon Shipping Co. v. Baker, (1) Justice Souter inserted a short, three-sentence footnote:

The Court is aware of a body of literature running parallel to anecdotal reports, examining the predictability of punitive awards by conducting numerous "mock juries," where different "jurors" are confronted with the same hypothetical case. See, e.g., C. Sunstein, R. Hastie, J. Payne, D. Schkade, W. Viscusi, Punitive Damages: How Juries Decide (2002); Schkade, Sunstein, & Kahneman, Deliberating About Dollars: The Severity Shift, 100 Colum. L. Rev. 1139 (2000); Hastie, Schkade, & Payne, Juror Judgments in Civil Cases: Effects of PlaintifFs Requests and Plaintiff's Identity on Punitive Damage Awards, 23 Law & Hum. Behav. 445 (1999); Sunstein, Kahneman, & Schkade, Assessing Punitive Damages (with Notes on Cognition and Valuation in Law), 107 Yale L.J. 2071 (1998). Because this research was funded in part by Exxon, we decline to rely on it. (2) To the casual reader, Footnote 17 seems innocuous enough. But to members of the scholarly and legal communities it is anything but bland. Perhaps not since Footnote 4 in United States v. Carolene Products (3) or Footnote 11 in Brown v. Board of Education (4) has a footnote in a Supreme Court opinion generated so much debate. (5)

Why has this little footnote created such a big stir? Part of the answer is that Footnote 17 is akin to the beast in the tale of the blind men and the elephant. In an effort to "see" the elephant, each man touches a different (but only one) part: the elephant's leg, its tail, and its trunk. As a result, the men can't agree on whether the elephant looks like--a pillar, a rope, or a tree.

And so it goes with Footnote 17. When experimentalists read it, they tend to focus on the first sentence and express resentment at Justice Souter's use of a second set of quotes around the word "jurors." To them, the scare quotes mock experiments for their failure to capture the experience of serving on a real jury. (6)

To legal academics working in the area of punitive damages, the second sentence of Footnote 17--the string citation--moves to the fore. In their corner of the world, the footnote reignites fierce debates over the various studies Justice Souter cites. (7) Representing one side, Jeffrey Rachlinksi argues that "[t]he Justices say they did not rely on the [studies] Exxon funded, even though it is actually very good work AND they seem to have been affected by it." (8) On the other side, David Hoffman--among other important academics (9)--has expressed agreement with the Court's "skepticism" about the cited works. (10)

The third and final sentence of Footnote 17 ("Because this research was funded in part by Exxon, we decline to rely on it.") also has been the subject of much discussion, especially among those who conduct empirical research. But while the first two sentences of Footnote 17 have generated debate and even resentment, the reaction to the third sentence has been almost uniformly positive. Even those who disagree over the value of the cited studies praise sentence three. As Rachlinkski wrote, the Justices "disparage funded studies in Footnote 17. Now, I agree funded studies can be suspect ... so maybe that is a good move." (11) Hoffman put it this way: "I was ... fairly shocked to see the Court acknowledge the problem of deep capture in such an open way." (12) Hoffman and Rachlinski are hardly alone. When we looked over the existing commentary, we came across only one scholar who questioned the Court's refusal to rely on the cited studies; (13) the other commentaries all expressed enthusiasm for the Court's approach. (14)

For the reasons we lay out in Parts I and II, the exuberance over Footnote 17 may be understandable, but at bottom we think it is misguided: courts and attorneys should not assess the integrity of academic work solely by whether an interested party funded it. Rather, the legal community should adopt the standards already in place in the scientific community. Part III explains the most prominent of these standards--reliability, validity, and transparency--and demonstrates how, with the help of empirically minded law scholars, appellate court judges and litigators could apply them.

1. THE OVERWHELMING RESPONSE TO FOOTNOTE 17, SENTENCE 3: "GOOD FOR THE COURT"

   When it comes to the last sentence of Footnote 17, the response we have heard (and read) time and time again is: "Good for the Court!" (15) And indeed this was our first response. Coincidentally enough, when Exxon Shipping came down, one of us (Epstein) was leading a workshop on conducting empirical legal scholarship, and had just finished a lecture on how hypothesis testing differs from representing a client. While a lawyer helps make the best case for his client by dismantling the opposition's arguments, a scientist strengthens her hypothesis by making the strongest possible case against it. (16)

   To the students in the workshop, Footnote 17 embraced this very idea--that research is, or at least should be, different from lawyering. The footnote reflects the intuition that money exerts a pull on perspective: when people pay you, your work product tends to reflect their interests.

   And, in fact, this is not mere intuition; substantial evidence exists to support the claim that conclusions follow from the money. (17) As Nature, the science journal, summarizes it,

   [E]vidence [on commercial sponsorship of research] is consistent with the truism that although, in principle, science may be objective and its findings independent of other interests, scientists can be imperfect and subjective. There are circumstances where selection of evidence, interpretation of results or emphasis of presentation might be inadvertently or even deliberately biased by a researcher's other interests. (18) Doubtless this is true. As all data analysts know, every empirical project requires the researcher to make scores of decisions--from how to draw a random sample to the particular regression models, of the hundreds estimated, to present--all of which could go either way. Even seemingly straightforward tasks confront the researcher with a nearly uncountable number of choices.

   By way of illustration, suppose we wanted to study the amount of sponsored research undertaken by legal scholars. We might start with a stack of law review articles and then, for each article in which the author thanks an industry, government agency, or organization for funding support, we would enter a "1" on our spreadsheet. Articles that do not explicitly acknowledge such support would receive a "0." (This is known as "coding data," which is the process of translating properties or attributes of the world (i.e., variables) into a form that researchers can systematically analyze and is a part of nearly every empirical project.) (19)

   Sounds simple, right? Wrong. For this article, we attempted to implement this very protocol and immediately ran into problems. What, for example, do we do with an author who works for the National Bureau of Economic Research (NBER)? While the author's acknowledgment may not mention funding for the particular article in question (and so would not count as a funding disclosure under our coding system), isn't remuneration support? Maybe yes, maybe no. Then there is the author employed by the National Center for State Courts. If he does not acknowledge support from the National Center, we confront the same problem as we did for the NBER, plus yet another complication: the National Center receives contributions from numerous corporations, including 3M, Dow Chemical, Ford, and so on. (20) This raises the question of what to do about indirect funding--even if the author acknowledges support from, say, the National Center for State Courts, ought we dig deeper and check sponsorship records or even the affiliations of board members?

   The point is that many of the decisions we made in this small-scale study--as in all research--could have gone either way. Did we try to make them without regard to the conclusions we would ultimately reach? Sure, but humans tend to be biased in small, imperceptible ways--and, just as surely, they may allow their funder to affect their choices. If our benefactor were concerned with exposing the role of industry in legal research, perhaps we would have scoured the records of the various organizations. If, on the other hand, our funder was an industry interested in minimizing the role of corporate support, we may have adopted more restrictive coding rules. Either way, even this seemingly small decision--just one of many we would make over the course of the research--could have a big impact our conclusions. As Nature explains, "[i]t would be naive to think that such incentives have no effect on what gets published, and indeed several studies of the clinical literature have demonstrated associations between the conclusions of a study and the source of its funding." (21)

2. ANOTHER RESPONSE: "A MISGUIDED APPROACH TO ACADEMIC RESEARCH"

   All of this goes to a simple point: treating funded research with some degree of skepticism is not without merit. For this reason, it easy to imagine why many would agree with Barday's sentiment: the type of "hired-gun" research condemned in Exxon Shipping is "problematic even if the results [are] accurate because ... it creates an appearance of bias." (22)

   On reflection, though, Barday's reaction and, more importantly, Footnote 17 itself push the point way too far, evincing a misunderstanding of empirical research. With regard to Footnote 17, this problem manifests in two ways: a failure on the part of the Justices first, to consider the troubling implications of their approach to sponsored; and second, to contemplate (much less apply) existing norms within the scientific community. (23) Both claims deserve some elaboration.

   1. The (Troubling) Logic of Footnote 17

      Of the troubling aspects of the...