Measuring law's normative force

• Published date: 01 December 2023
• Author: Kevin L. Cope
• Date: 01 December 2023
• DOI: http://doi.org/10.1111/jels.12364

ORIGINAL ARTICLE

Measuring law’s normative force

Kevin L. Cope

School of Law, University of Virginia, Charlottesville, Virginia, USA

Correspondence

Kevin L. Cope, School of Law, University of Virginia, 580 Massie Rd., Charlottesville, VA 22903, USA.

Email: kcope@law.virginia.edu

Abstract

An important question in legal theory and policy is when people are willing to put

aside their policy preferences to uphold higher-order legal values. That is, when does

constitutional or international law, for instance, have “normative force”?Around

two-dozen experimental studies have attempted to measure this question empirically,

but their designs contain an inherent limitation. While they are interested in gauging

the effect of internalizing a norm, they measure only the effect of exposure to that

norm. This is significant because subjects in the treatment group whose priors are

strongly contrary to the treatment message on the legality of a policy may effectively

be “treatment resistant”: it is difficult to successfully treat them because their prior

beliefs on the issue are entrenched; as a result, they simply do not believe the treat-

ment message. This treatment failure attenuates any effects, and where a significant

portion of the treatment and/or control group is not successfully treated, the results

will be biased toward small, null, or even backfire findings. This article first formally

models the mechanism underlying experiments on law’s normative force. I then dem-

onstrate a methodological solution to the problem of treatment resistance. By using

the experimental treatment as an instrumental variable and employing a post-

treatment treatment-uptake test, the researcher can estimate the causal effect of the

real explanatory variable of interest: sincerely holding a belief about a policy’s

higher-order lawfulness. Using new data from a 2022 survey experiment conducted

on US residents, I illustrate this method for three constitutional or international law

issues. The theoretical and empirical results together suggest that backfire effects

documented by some studies do not reflect a tepid or negative response against the

legal source per se, but rather reflect treatment resistance. These findings suggest that

we should re-evaluate the existing body of experimental studies on law’s normative

force, and they should prompt researchers to reconsider how we conduct future

research in this domain.

DOI: 10.1111/jels.12364

This is an open access article under the terms of the Creative Commons Attribution License, which permits use,

distribution and reproduction in any medium, provided the original work is properly cited.

©2023 The Authors. Journal of Empirical Legal Studies published by Cornell Law School and Wiley Periodi-

cals LLC.

J Empir Leg Stud. 2023;20:1005–1044. wileyonlinelibrary.com/journal/jels 1005

INTRODUCTION

Under what circumstances do people want their government to follow the law?

Presumably, most people want government officials to abide by higher-order

legal norms, like constitutional and international law, most of the time. After

all, the main purposes of these institutions are to promote social stability and

predictability, and to protect codified rights, each of which, all things equal,

the populace largely values. Sometimes, though, people might support their

governments’violating those rules. In the face of a terror threat, for instance,

many people support torturing a suspect, even though they know doing so

would violate international law and the constitution (Chilton et al., 2024;A.

S. Chilton & Versteeg, 2016; Wallace, 2013). In such cases, people may genu-

inely respect international law in principle, but their preference for safety leads

them to prefer a policy that they view as security-enhancing over state compli-

ance with the law.

When higher-order legal rules and popular opinion conflict, an important

question for legal theorists and policymakers is: under what conditions are people

willing to put aside their policy preferences to uphold those higher-order legal

values? How much independent value do they attach to these rules themselves? If

higher-order legal norms can sway popular opinion in this way, then we can say

that law has “normative force”or that the legal norms are valued “as an institution”

(Llewellyn, 1934). This compliance mechanism is especially critical for state compli-

ance with law, especially institutions that lack external enforcement mechanisms

(A. Chilton & Versteeg, 2020; Goldsmith & Levinson, 2008).

It should be possible to measure empirically when and how much a particu-

lar higher-order legal rule sways public opinion. Indeed, at least two-dozen

studies have attempted to explore this question (e.g., Chaudoin, 2014;

A. S. Chilton, 2014; A. S. Chilton & Versteeg, 2016; Cope & Crabtree, 2020;

Kreps & Wallace, 2016; Linos, 2011,2013; Lupu & Wallace, 2019; Putnam &

Shapiro, 2017; Tingley & Tomz, 2014; Wallace, 2013). These studies are mostly

designed as survey experiments. In these studies, subjects are typically asked

how much they support some real or hypothetical state policy or action that is

arguably illegal. A random subset of respondents is also told that the proposed

policy violates international or constitutional law, while the control set is not

given this treatment. The researcher then observes the difference in expressed

policy support between these groups. If there is a substantial difference, we can

say that the law appears to have normative force. Many of these studies have

found that international law does indeed have some such sway over public opin-

ion. Yet, others have found null results or even a backfire effect (in which the

treatment makes respondents more supportive of illegal policies).

Yet, there exists a methodological limitation in these study designs which

has scarcely been recognized (Cope & Crabtree, 2022), let alone analyzed, for

which this article proposes a solution. Many of the experimental studies focus

1006 MEASURING LAW’S NORMATIVE FORCE

on real-world issues such as torture of terrorists, the migrant family-separation

policies of the United States, or refugee policy in Turkey. However, respondents

may already possess existing beliefs on the legality of such policies. As a result,

these respondents may simply not believe or otherwise internalize the treatment

message. The “real”treatment of interest in studying law’s normative force is

not mere exposure to a message, but rather, the mediator of truly believing that

which the message communicates; the experimental exposure is essentially

merely an intent to treat. Viewing the treatment this way, many respondents in

the treatment group are effectively “treatment resistant”: it is difficult or impos-

sible to successfully treat them (e.g., Barabas & Jerit, 2010; Gaines et al., 2007)

by changing their views. Conversely, many respondents in the control group are

effectively “control resistant.”And where a significant portion of the treatment

and/or control group is not successfully treated, treatment is unrandomized, and

the results will be biased toward small, null, or even backfire findings

(cf. Druckman & Leeper, 2012). It appears that this problem may plague a large

fraction of the research in this domain.

The simplest, most-intuitive explanation for a small, null, or backfire effect

is that respondents hold a tepid or even negative view of the cited higher-order

authority. In the case of backfire, they substitute that authority’s endorsement

for their own judgment, but negatively. Indeed, this phenomenon is likely the

driver of small or backfire effects in some limited contexts, such as societies with

well-documented hostility to certain international institutions, such as Turkey

(Cope & Crabtree, 2022) and Israel (Lupu & Wallace, 2019). The difficulty is

that small, null, or backfire findings driven by an unpopular norm are often

observationally indistinguishable from such findings driven by treatment resis-

tance. The researcher may therefore erroneously assume that the norm is de-

valued.

This article proposes a methodological solution to this problem, which

allows the researcher to measure the effect of a person’s true belief about a

policy’s higher-order lawfulness. Because of treatment resistance, the treatment

assignment is not fully random. But because the experimental treatment affects

the outcome (policy support) only through the mediator of true belief, the exper-

imental treatment is a plausible instrument for true belief about lawfulness. We

can gauge the extent of this true belief simply by asking respondents, post-

treatment, what they believe about the higher-order lawfulness of the proposed

policy—what I call a treatment-uptake test. The researcher can then use two-

stage least squares analysis to estimate the causal effect of true belief about

higher-order lawfulness on policy support: that is, the normative force of law.

This article proceeds, first, by formally modeling the mechanism underlying

experiments on law’s normative force. I then propose the methodological solu-

tion: using the experimental treatment as an instrumental variable, and a

treatment-uptake test. I then illustrate the problem and proposed solution using

new data from a survey experiment conducted in the United States in spring

MEASURING LAW’S NORMATIVE FORCE 1007