Law and Psychology Grows Up, Goes Online, and Replicates

• DOI: http://doi.org/10.1111/jels.12180
• Date: 01 June 2018
• Author: Krin Irvine,David A. Hoffman,Tess Wilkinson‐Ryan
• Published date: 01 June 2018

Law and Psychology Grows Up, Goes

Online, and Replicates

Krin Irvine, David A. Hoffman, and Tess Wilkinson-Ryan*

Over the last 30 years, legal scholars have increasingly deployed experimental studies,

particularly hypothetical scenarios, to test intuitions about legal reasoning and behavior.

That movement has accelerated in the last decade, facilitated in large part by cheap and

convenient Internet participant recruiting platforms like Amazon Mechanical Turk. The

widespread use of online subjects, a practice that dramatically lowers the barriers to entry

for experimental research, has been controversial. At the same time, the field of

experimental psychology is experiencing a public crisis of confidence widely discussed in

terms of the “replication crisis.” At present, law and psychology research is arguably in a

new era, in which it is both an accepted feature of the legal landscape and also a target of

fresh skepticism. The moment is ripe for taking stock. In this article, we bring an empirical

approach to these problems. Using three canonical law and psychology findings, we

document the challenges and the feasibility of reproducing results across platforms. We

evaluate the extent to which we are able to reproduce the original findings with

contemporary subject pools (Amazon Mechanical Turk, other national online platforms,

and in-person labs). We partially replicate all three results, and show marked similarities in

subject responses across platforms. In the context of the experiments here, we conclude

that meaningful replication requires active intervention in order to keep the materials

relevant and sensible. The second aim is to compare Amazon Mechanical Turk subjects to

the original samples and to the replication samples. We find, consistent with the weight of

recent evidence, that the Amazon Mechanical Turk samples are reasonably appropriate for

these kinds of scenario studies. Subjects are highly similar to subjects on other online

platforms and in-person samples, though they differ in their high level of attentiveness.

Finally, we review the growing replication literature across disciplines, as well as our

firsthand experience, to propose a set of standard practices for the publication of results in

law and psychology.

I. Introduction

In the last decade, the costs of doing law and psychology research have falle n dramat-

ically. Unsurprisingly, the supply of experimental papers—especially hypothetical

vignette studies—is up, both in peer-reviewed journals and in student-run law

*Address correspondence to David A. Hoffman; email: dhoma1@law.upenn.edu. Irvine is graduate student at the

University of Chicago’s Booth School of Business; Hoffman is Professor of Law at the University of Pennsylvania

School of Law; Wilkinson-Ryan is Professor of Law and Psychology at the University of Pennsylvania School

of Law.

320

Journal of Empirical Legal Studies

Volume 15, Issue 2, 320–355, June 2018

reviews.

1

Scenario studies now inform a wide range of subjects, including interna-

tional law,

2

torts,

3

criminal procedure,

4

contracts,

5

and securities.

6

It has never been

easier to run, and publish, a quickie experiment.

7

Commentators have identified two concerns about this state of the scholarship.

First, they worry that contemporary law and psychology studies are too dependent on a

narrow, unusual subject pool: Amazon Mechanical Turk (MTurk). MTurk is extraordi-

narily inexpensive—some researchers pay as little as 10 cents per subject—and does not

require any sort of institutional affiliation. Many scholars have conjectured that the

MTurk subject pool is unrepresentative of the broader population in both observable

and unobservable ways.

8

MTurk participants are clearly younger and more liberal than

the U.S. population. They are also, by definition, willing to do online survey research

for very small payments, an odd enough trait that it raises questions about other hidden

idiosyncrasies. Ease of use also raises suspicions of false positives—if it is very easy to test

a hypothesis repeatedly, the odds of at least one statistically significant result are, of

course, higher. Skepticism about MTurk is now widespread among consumers and arbit-

ers of empirical legal studies, from student editors to peer referees.

The second source of skepticism of law and psychology arises from new debates

about the value of psychological research more generally. The field of psychology is in

1

See Adam J. Berinsky, Gregory A. Huber & Gabriel S. Lenz, Evaluating Online Labor Markets for Experimental

Research: Amazon.com’s Mechanical Turk, 20 Pol. Anal. 351, 351 (2012) (“Interest in experimental research has

increased substantially in political science.”); See generally Shari Seidman Diamond & Pam Mueller, Emp irical

Legal Scholarship in Law Reviews, 6 Ann. Rev. L. Soc. Sci. 581, 589 (2010) (discussing the rise of experimental

research in law).

2

See, e.g., Adam Chilton & Dustin Tingley, Why the Study of International Law Needs Experiments, 52 Colum. J.

Transnat’l L. 173 (2013) (discussing how experimental research methods are increasingly utilized in international

law).

3

See generally Jennifer K. Robbennolt & Valerie P. Hans, The Psychology of Tort Law (2016).

4

See, e.g., Avani Mehta Sood, Cognitive Cleansing: Experimental Psychology and the Exclusionary Rule, 103 Geo.

L.J. 1543 (2015) (using scenario study to conduct experiments on exclusionary rule); see also Holger Spamann

& Lars Kl€

ohn, Justice is Less Blind, and Less Legalistic, Than We Thought: Evidence from an Experiment with

Real Judges, 45 J. Legal Stud. 255 (2016) (investigating judicial decision making through scenario simulation).

5

See, e.g., George S. Geis, An Experiment in the Optimal Precision of Contract Default Rules, 80 Tul. L. Rev.

1109 (2006) (conducting experimental study to determine optimal precision of contract default rules); see also

George S. Geis, Empirically Assessing Hadley v. Baxendale, 32 Fla. St. U. L. Rev. 897 (2005) (using empirical assess-

ment to analyze key case in contract law); Yuval Feldman & Doron Teichman, Are All Contractual Obligations

Created Equal? 100 Geo. L.J. 5 (2011) (using experimental surveys to study responses to a breach of contract).

6

See, e.g., Jill Fisch & Tess Wilkinson-Ryan, Why Do Investors Make Costly Mistakes?: An Experiment on Mutual

Fund Choice, 162 U. Pa. L. Rev. 605 (2014).

7

Others have made similar points. See Kathryn Zeiler, The Future of Empirical Legal Scholarship: Where Might

We Go from Here? 66 J. Legal Educ. 78, 84--87 (2016) (discussing lack of interest in methods of intros pection

within legal academy).

8

See text accompanying notes 34 through 35.

321Law and Psychology Grows Up

the midst of a mid-life crisis of sorts, stemming from recent claims that a surprising frac-

tion of experimental psychological findings are not replicable—or at least that they are

not reliably replicable in their original form. The crowd-sourced “Replication Project”

and its offshoots represent a profound challenge to many subdisciplines inside of psy-

chology, as they seem to call into question foundational findings—not to mention the

careers they have launched.

9

Indeed, though psychology has come under particularly

public scrutiny, the failure to reproduce scientific results is under active discussion

across the academy, in economics, in medicine, and even in physics.

In this article, we bring together these two sources of skepticism about law and

psychology, taking seriously both the critiques and the justifications of the field and its

methods. The article offers a case study of a replication project, as well as a set of sys-

tematic comparisons of subject characteristics across survey platforms. Specifically, we

replicate three canonical law and psychology papers, using MTurk, the commercial sur-

vey firm Survey Software International (SSI), Prolific Academic, a new online service

designed for researchers, and an in-person lab run by a university. Using thousands of

respondents, including in-person and nationally representative pools, and with different

scenarios, we draw some lessons for experimental law and psychology.

Researchers in the social sciences have offered ample evidence that MTurk sub-

jects respond to classic psychology experiments in predictable and reliable ways—at least

as compared to other common sample populations. We find the same here. We also

find that they are significantly more attentive than subjects in other subject pools.

In our view, allaying some of these more arithmetic concerns about MTurk opens

up the deeper conversation about experimental psychology in legal studies. Resistance

to MTurk and like online platforms arises in part from a worry that as the costs of doing

research fall, the quality will suffer. That reasonable fear ought to be addressed on its

own terms. How should consumers of this field distinguish between good work and bad,

signal and noise? We do not think the answer is to sort based on subject pool, but agree

that the demands of quality control are higher in a world of universal access to random-

izing survey software and cheap online subjects.

This article also makes a more theoretical contribution, drawing on our research

experience. We argue that context, specifically temporal context, is an unappreciated treat-

ment for experiments about law. It is a truism that the present moment influences peo-

ple’s views about legal institutions and rules—and yet it is also easily overlooked. Social,

political, and legal change over time make rote replication of law and psychology experi-

ments not just challenging but almost irrelevant. Though we did in fact reproduce many

findings of the canonical papers we studied by using their original materials, we came to

view that approach as counterproductively rigid, and even unfair to the original authors.

“Replication” of law and psychology demands that researchers focus on the underlying

mechanism of action and work to translate those insights to a modern vernacular.

We conclude with some proposals for best practices in the field. Unlike others, we

do not recommend precommitment to particular research designs. We do, however,

9

See text accompanying notes 10 through 18.

322 Irvine et al.