Rankings without U.S. News: A revealed preference approach to evaluating law schools

• Published date: 01 June 2024
• Author: Jesse Rothstein,Albert Yoon
• Date: 01 June 2024
• DOI: http://doi.org/10.1111/jels.12380

ORIGINAL ARTICLE

Rankings without U.S. News: A revealed preference

approach to evaluating law schools

Jesse Rothstein

1

| Albert Yoon

2

1

University of California, Berkeley, Berkeley, California, USA

2

University of Toronto, Toronto, Canada

Correspondence

Albert Yoon, University of Toronto, Toronto, Canada.

Email: albert.yoon@utoronto.ca

Abstract

Since their inception in 1989, the U.S. News & World Report law school rankings have

influenced how schools, students, and the legal profession itself think about legal education. In

the Fall of 2022, however, several of the most selective law schools formally withdrew from the

annual rankings. In so doing, these schools laid bare longstanding criticisms of the rankings’

questionable criteria and opaque methodology. While the long-term effect of this boycott

remains to be seen, school rankings are likely here to stay. In this Article we design a more infor-

mative approach to rankings, based on actual decisions students make. Using individual-level

data provided by the Law School Admissions Council (LSAC), we analyze the universe of

applicants to U.S. law schools for the period 1988 through 2017. In so doing, we are the first to

create a revealed preference ranking based solely on where applicants matriculate given offers

of admission. Our approach relies neither on potentially faulty data collection from schools nor

arbitrary decisions about which factors to emphasize in rankings, thereby minimizing the scope

for manipulation. It also allows us to quantify the magnitude of differences in preferences

among schools and to test their statistical significance. Matriculants reveal a strong preference

for a handful of the most selective schools; outside of the top tier, however, matriculants do not

appear to draw meaningful distinctions between schools ranked adjacently or even near to each

other. While existing school rankings sow more confusion than clarity, our analysis provides a

rigorous and transparent alternative, and a blueprint for redesigning school rankings.

INTRODUCTION

Students who pursue higher education face a bewildering set of choices. Many

adopt a portfolio approach, applying to both “safety”schools (where admission

DOI: 10.1111/jels.12380

This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs

License, which permits use and distribution in any medium, provided the original work is properly cited, the use is

non-commercial and no modifications or adaptations are made.

©2024 The Authors. Journal of Empirical Legal Studies published by Cornell Law School and Wiley Periodi-

cals LLC.

J Empir Leg Stud. 2024;21:279–336. wileyonlinelibrary.com/journal/jels 279

is likely) and “reach”schools (schools they prefer but where admission is less

likely). Does it matter where one goes to school? For scholars who have closely

examined the question, the answer appears to be, not that much. Controlling for

student ability, attending more selective schools have little effect on longer-term

outcomes for most students (Dale & Krueger, 2002,2014), though there is some

evidence of larger effects on very high incomes (Chetty et al., 2023). It appears to

matter most for students who are the firstin their families to attend college or uni-

versity, many of whom are under-represented minorities (Bowen & Bok, 2000).

The admissions process is understandably stressful (Barnard, 2018). Some

students fall short of their expectations (Strauss, 2016) while others exceed them.

In deciding where to matriculate, students draw on what they know about

schools, for example, curricula, student profile, school resources, and so forth.

Academic counselors can help calibrate students’expectations to admissions

realities, though many students lack access to this resource (Steinberg, 2009).

Students also draw on rankings of schools, such as that published each year

by U.S. News and World Report (U.S. News). Educational rankings provide a

cheap, accessible tool to inform students’decisions about complex, hard to

observe aspects of schools. Publications generate their rankings using their own

criteria but share a common feature of ordinally ranking the schools. In so

doing, they provide students a heuristic of evaluating schools. Every bit of infor-

mation about each school reduces to its easily digestible ordinal rank that allows

students, if they so choose, to directly compare schools.

Educational rankings themselves have generated considerable debate. Critics

point to flaws in the methodologies used to generate these rankings, often based

on factors that are inherently subjective or difficult to measure (Altbach, 2006;

Fauzi, 2020). Others point to the rankings’emphasis on measures tied to quality

of research rather than teaching (Altbach, 2012). Others contend it is futile to

try to reduce something as multi-faceted and complex as educational institutions

to a single dimension—ordinal rank—by which students directly compare

schools with one another (Gladwell, 2011).

Efficacy aside, the influence of educational rankings depends on a few fac-

tors, notably prestige of the publication issuing the ranking. Another is the

availability of competing rankings, which can diminish the influence of any sin-

gle publication. For undergraduate institutions, numerous rankings exist.

1

For

graduate programs, more specialized by definition, fewer exist. For law schools,

one publication dominates: U.S. News (Lewis, 1995). U.S. News constructs its

law school rankings, first published in 1990, based on measured attributes of

schools (yield rates, median LSAT score, etc.) and subjective, non-public fac-

tors, with weights chosen by the publication. Over the years, U.S. News has

tweaked its methodology while expanding the number of schools it ordinally

ranks.

1

The list includes Barron’s, Forbes, Niche, Princeton Review, Quacaquarelli Symons, U.S. News & World Report.

280 RANKINGS WITHOUT U.S. NEWS

The U.S. News law rankings have long generated criticism. The American

Bar Association issued a statement that it neither endorses nor cooperates with

“any law ranking system”(ABA, 2023a,2023b). Critics decry the U.S. News

methodology, contending its criteria are manipulatable, subjective, or in some

instances, both (Seto, 2007). The Indiana Law Journal dedicated an entire issue

to deconstructions of the U.S. News rankings by leading jurists and scholars, in

mostly negative terms. Other critics point to the rankings becoming the prover-

bial tail that wags the dog, as rankings drive law deans’governance decisions

(Emens, 2009). On occasion, law schools have been exposed for fudging their

reported numbers in an effort to prop up their rankings (Cohen, 2012). Notwith-

standing these detractions, each February the legal community—students, pro-

fessors, practitioners—anxiously await the latest U.S. News rankings.

In the Fall of 2022, law schools experienced a moment of awakening.

Shortly before Thanksgiving, Yale Law School announced it would no longer

contribute data to the rankings, criticizing the rankings for its “misguided for-

mula that discourages law schools from doing what is best for legal education”

(Yale Law School, 2022) Harvard Law School followed a day later, stating it

had “become impossible to reconcile our principles and commitments with the

methodology and incentives the U.S. News rankings reflect”(Doan-Nguyen &

Peña, 2022). Other schools followed (Watanabe, 2022). In response to this exo-

dus, U.S. News in January 2023 announced changes to its methodology (U.S.

News, 2023). No school reversed its initial boycott decision, and by the end of

January 2023, 40 law schools had joined the boycott (Caron, 2023). More

recently, schools in other disciplines announced their boycott the U.S. News

rankings (Korn, 2023), leading to speculation as to the future of school rankings

writ large (Strathern, 2000).

The longer-term effect of the U.S. News boycott remains to be seen, but two

things are clear. One, existing school rankings are deeply flawed, sowing more

confusion than clarity. If the beneficiaries are meant to be students themselves,

the rankings fall short in their stated goal (Morse, 2022) of helping students

make informed choices in choosing a school. Two, rankings are likely here to

stay. Students seem to need the simplicity that rankings provide, and there is

clearly a market for them (Shore & Wright, 2015). Rankings also provide a

means by which people of different levels of engagement can discuss the same

topic (Strathern, 2000). For these reasons, we believe the discussion should

move past existing rankings toward a compelling alternative.

This article develops a set of rankings that avoids the pathologies of existing

ones and embraces rigor and transparency. Our rankings are based on actual

decisions that students make, not on others’beliefs as to what aspects of law

schools matter. Using a unique dataset provided by the Law School Admissions

Council (LSAC), we observe every law school applicant to U.S. law schools

from the period 1989 to 2017. For each applicant, we know the universe of law

schools applied, schools (if any) accepted, and school (if any) matriculated.

RANKINGS WITHOUT U.S. NEWS 281