Inputs and Outputs on Appeal: An Empirical Study of Briefs, Big Law, and Case Complexity

• DOI: http://doi.org/10.1111/jels.12263
• Author: Gregory C. Sisk,Adam M. Samaha,Michael Heise
• Date: 01 September 2020
• Published date: 01 September 2020

Journal of Empirical Legal Studies

Volume 17, Issue 3, 519–555, September 2020

Inputs and Outputs on Appeal:

An Empirical Study of Briefs, Big Law,

and Case Complexity

Adam M. Samaha,*Michael Heise, and Gregory C. Sisk

The relationship between judicial inputs and outputs remains opaque. Conventional

wisdom is plentiful but useful evidence is not. This study examines civil appeals in three

federal appellate courts, concentrating on the briefing, the attorneys, and the complexity

of the case. We find no evidence that short briefs are more persuasive, while the evidence

in favor of long briefs on the appellant’s side is provocative. We also find suggestive evi-

dence that the experience of the lead lawyer on the appellee’s side matters. However, “Big

Law” firms and large teams of lawyers do not seem to perform better on appeal, all else

equal. Finally, different kinds of case complexity point in different directions. The pres-

ence of a cross-appeal is associated with judge votes to reverse—but not necessarily in favor

of the cross-appellant. At the same time, complex trial-level proceedings may be associated

with judge votes to affirm.

I. Introduction

Judicial decision making remains, in important respects, enigmatic. Judges organize

information flows and staff activity within their chambers without fully revealing those

decision architectures to outsiders. Each judge’s interactions with colleagues are largely

opaque as well. Even the law clerks and other staff who work closely with judges do not

witness the complete decision-making process, from whenever it may begin in earnest

until gavels come down, orders are announced, and opinions go to print.

*Address correspondence to Adam M. Samaha, New York University School of Law, 40 Washington Sq. S.,

New York, NY 10012; email: adam.samaha@nyu.edu. Samaha is the Inez Milholland Professor of Civil Liberties at

NYU School of Law; Heise is the William G. McRoberts Professor in the Empirical Study of Law at Cornell Law

School; Sisk is the Pio Cardinal Laghi Distinguished Chair in Law at the University of St. Thomas–Minnesota

School of Law.

This project relied on a team of research assistants for the data-collection effort. We thank Alexandra Ferrara

(NYU School of Law, Class of 2019) in particular for research on options and strategies, as well as Naz Akyol, Alexis

Alvarez, Conor Gaffney, Sofia Fernandez Gold, Madelyn Fife, Chloe Lewis, Alexandra Liebl, Eric Loverro, Jonathan

Stahl, Lucian Wang, Ryan Woods, Nicole Zeman, and Molly Zhu. We thank officials at the clerks’ offices in the Sec-

ond, Eighth, and Ninth Circuits for their guidance on court practices. We also thank participants at NYU’s Law

and Economics Colloquium and Furman Fellows Workshop for their helpful comments. Mistakes remain our own.

519

Outsiders can see case results, however, and we can improve our understanding of

upstream inputs. Some litigation inputs have been studied repeatedly, it is true, such as

party capability and the traits of the judges assigned to cases (see Section II.B). Other

inputs have received attention infrequently or not at all. With increasing availability of

primary sources and automation, a new range of study is becoming feasible. Over time,

we can develop better pictures of the informational environment in which judicial deci-

sion making occurs, and make progress on a series of research questions about the con-

nections among inputs and outputs.

More specifically, we can improve our view of information loads and types that are

fixed upstream from judgment. For instance, we can inquire whether concise advocacy is

rewarded such that “[l]ess is more” (Garner 2014:617) and it pays off to “[a]cquire a rep-

utation as a lawyer who often comes in short of the limits” (Scalia & Garner 2008:24), or

whether more words conveying more information are better for the advocate, if not court

personnel. Likewise, we can investigate the influence of relatively focused arguments in

contrast with multi-barreled issue attacks and extensive lists of legal authorities. At the

same time, we can ask whether courts respond to the types of attorneys and parties who

they hear and see. More broadly, we can try to identify which of the influential inputs are

controlled—and perhaps obscured—by the strategic choices of lawyers and parties, and

which are hardwired into the proceedings by other forces. These questions are tightly

linked to everyday choices facing attorneys, and to an informed evaluation of how our

judicial system operates.

First steps were taken in earlier work. Sisk and Heise (2015) and Samaha (2017)

investigated how judges on a federal court of appeals responded to brief length, issues

presented, and sources cited, along with attorney experience and party type, in civil cases

that were terminated on the merits. A lengthier brief on the appellant’s side was associ-

ated with judge votes to reverse, not affirm, for instance—yet a larger number of issues

presented was associated with votes to affirm. The experience level of the appellee’s attor-

ney was significant, but not the that of the appellant’s. Identifying the causal force of

these inputs without an objective measure of case quality or an experimental design is

highly challenging, of course, and these earlier studies were exploratory.

For this project we expand the data, include additional controls, and update our

models. We study the Second, Eighth, and Ninth Circuits using the most complex dataset

on appeals of which we are aware. Although the sample of appeals is not exceptionally

large, we are able to examine each appeal along many dimensions. The full set encom-

passes more than 250,000 data points. Importantly, we have added proxies for case com-

plexity at the trial level and litigation resources on appeal, including the participation of

“Big Law” attorneys. Alongside controls for party type, issue type, procedural stage, and

certain judge traits, these models allow us to offer more-refined suggestions about causal

pathways.

1

1

In addition, we have assembled appellate briefs and district court opinions for content analysis and other auto-

mated work in later research.

520 Samaha et al.

Our principal findings recommend several revisions to the preexisting picture.

While we still see no evidence that shorter briefs are more persuasive, the evidence favor-

ing longer briefs is provocative. If there is a meaningful statistical association between

judge votes and brief length, it might well lie on the appellant’s side and toward the high

end of the spectrum. We also find suggestive evidence that the experience of the appel-

lee’s lead lawyer correlates with votes to protect the result below. But Big Law and large

teams of lawyers do not seem to perform better on appeal, all else equal. Furthermore,

different kinds of case complexity point in different directions. The presence of a cross-

appeal, such that both sides are attacking the trial court, is associated with judge votes to

reverse—but not necessarily in favor of the cross-appellant. At the same time, complex

trial-level proceedings may be associated with judge votes to affirm.

Caution is warranted for each of these findings. Beyond notable variation across cir-

cuits and the sensitivity of some variables to error clustering choices, not every theoreti-

cally relevant case-level variable can be accounted for in our data. For example, longer

appellant briefs might not have an independent causal effect but instead proxy for

unobserved vulnerability of district court decisions, despite our trial-level variables and

other controls. The appellate process is complex and remains partially opaque; much is

left to learn about causal forces. At minimum, the research presented here offers a

broader and deeper descriptive account of the civil appellate system and the practical

choices that lawyers make every day—along with a stronger platform for future theoreti-

cal and empirical work, both observational and experimental.

Section II reviews appellate litigation basics, decision theory, and prior research.

Section III describes our data, including the two approaches that we took to sample and

code appeals. The data for the original studies in the Ninth Circuit were not collected in

the same way as the new data from the Second and Eighth Circuits, which tend to be

more precise. Section IV presents our models. Section V reports results. Section VI offers

consolidated lessons and potential ways forward.

II. Practice, Theory, and Prior Research

II.A. Appellate Practice and Rules

The selection of civil disputes for litigation and appeal occurs in several stages, often

in familiar patterns. A small fraction of complaints are brought to the attention of a

court instead of compromised or let pass without remedy (Miller & Sarat 1981:536–43).

Some litigated complaints are taken up by lawyers, while an impressively large number

are filed pro se. Many—indeed most—filed disputes are settled, while others are

addressed on the merits by a trial court. Parties and attorneys then decide whether to

appeal and cross-appeal adverse outcomes. As with trial court proceedings, some frac-

tion of appeals will be settled or dismissed for jurisdictional defects, failure to comply

with court deadlines, and so forth. In fact, the role of staff attorneys in screening

appeals for summary termination has increased alongside appellate docket loads

(Songer & Haire 2017:158–59).

Inputs and Outputs on Appeal 521