Distinguishing judges: an empirical ranking of judicial quality in the United States Courts of Appeals.

• Author: Anderson, Robert, IV
• Position: Introduction through IV Results and Interpretation, p. 315-349

1. INTRODUCTION

   How can one evaluate the performance of federal appellate judges? This question implicitly arises every time a federal appellate judge is nominated to the United States Supreme Court. And because the federal appellate bench is the most common source of Supreme Court nominees in recent decades, (1) this question is relevant to most modern Supreme Court nominations. But the question of judicial performance is at least as important outside the context of Supreme Court appointments, as the courts of appeals are the final arbiters of most disputes in the federal courts. Thus, the outcome of virtually every litigated matter in the federal system hinges on the quality of federal appellate decision-making, and therefore the performance of these judges implicates fundamental questions about the rule of law.

   However, the importance of evaluating the performance of federal judges has not motivated systematic assessment of individual judges' work product in legal scholarship. Indeed, aside from anecdotal information, little is known about the performance of individual federal appellate judges. of course there is no dearth of scholarly critique of federal courts' products--the opinions in individual cases--but these critiques are not systematically organized into evaluation of the producers of the opinions--the judges themselves. Thus, in spite of the fact that the performance of individual judges has important implications for the functioning of the judicial system and rule of law, scholars still do not have a good idea of which judges are performing well and which judges are performing poorly. Few academic studies have even attempted to evaluate federal judges using quantitative data, and, when they have, they have generally received harsh criticism from scholarly commentators. (2)

   The recent nomination of then-Judge Sonia Sotomayor to the Supreme Court illustrates the de facto alternative to systematic approaches to judicial quality. The Sotomayor nomination, like nominations of federal appellate judges in the past, tended to focus on detailed scrutiny of a small number of high-profile opinions, distracting from the broader, systematic examination of the nominee's body of work as a whole. (3) In the absence of reliable information about judicial performance, center stage in the debate is yielded to anecdotal accounts of anonymous sources, (4) isolated remarks from the judge's public appearances, and short passages in opinions culled from the tens of thousands of pages the nominee has written. Although we have the benefit of a more thorough evaluation from the American Bar Association, its approach has been called biased (5) and may be no more objective than the confirmation hearings. The result is that the evaluation of judicial performance is biased, subjective, and based on a narrow slice of information rather than on the judge's record as a whole.

   The frustration with the prevailing approaches to assessing judicial quality, both in the context of Supreme Court appointments and otherwise, has led scholars and legal commentators to develop quantitative techniques to measure judicial performance. (6) The most prominent approaches in recent years use large databases of citations to evaluate the "influence," "prestige," or "quality" of judges. One of the first such papers, by Professors Landes, Lessig, and Solimine (hereinafter "Landes et al."), used citation counts to opinions to measure "judicial influence" in the federal courts of appeals. (7) More recently, Professors Choi and Gulati have expanded on the Landes et al. study, using citation counts to measure "productivity," "quality," and "independence" on the federal courts of appeals. (8) In contrast to the typical evaluation of judges' opinions that legal scholars perform in law reviews, the citation literature abstracts away from the details of the cases to systematically evaluate the whole body of the judges' work product.

   The citation studies have revealed information about judicial performance that was not previously well known outside the ranks of experienced appellate advocates and federal judges themselves--if it was known at all. Perhaps for this reason, the studies have attracted considerable attention in legal scholarship, including multiple responses to Choi and Gulati's first judge ranking paper in the Southern California Law Review, (9) a symposium published in the Florida State University Law Review, (10) and conference proceedings published in the Duke Law Journal. (11) As might be expected, however, the responding scholars and judges have not enthusiastically welcomed this quantitative intrusion into the traditional purview of qualitative legal commentary. Although some of the commentators' responses have focused on very fair criticisms of the methodology employed by Choi and Gulati, several scholarly responses seem to have a broader point--one that rejects the very idea of quantitative assessment of judging. (12)

   Those who reject the notion that judicial performance is quantifiable will, of course, be disappointed by any variation of the Choi and Gulati techniques. But for those who see quantitative methods as a valuable tool in systematic assessment of the judiciary, there is an opportunity to build on existing citation studies to reveal another perspective on judicial performance. The opportunity arises because of two key limitations of the citation studies that limit the effectiveness of the existing work. First, the citation studies are "count-based," meaning that the number of citations is the key variable of interest in evaluating the judges. Among other problems, this approach treats negative citations the same as positive citations, even though negative citations might reflect negatively on judicial quality. Second, the citation studies rate judges based only on opinions they have authored, rather than all cases in which they have participated. opinion authorship, although closely tied to individual judges, raises a host of problems, not the least of which is selection bias in opinion assignment. Constructing a judicial evaluation technique that responds to these two problems offers a clearer, more comprehensive view of judicial performance.

   This Article attempts to construct such a technique for ranking federal appellate judges--one that does not have the same drawbacks of the existing citation-count studies. The problem of treating positive and negative citations alike is addressed by using "treatment" data from Shepard's Citations (provided by LexisNexis) to rank judges according to the positive or negative citations of their peers: other federal appellate judges. Using treatment data allows judges whose decisions are cited more positively to receive higher rankings and judges who are cited more negatively to receive lower rankings. The problem of selection bias is addressed by using panel membership, rather than opinion authorship, as the link between judges and citations. This means that judges who contribute to producing a decision that is positively cited when they did not author the opinion are ranked higher than those who only contribute to producing a decision that is positively cited when they author the opinion. These two innovations are designed to produce a measure of the quality of the average opinion produced jointly by an appellate panel, rather than the visibility or notoriety of opinions authored individually by a judge, as in the citation-count measures.

   Although this project was conceived as a means of building on the existing citation-count studies, the results are so strikingly different from those of the citation-count models that this study is more properly viewed as a break with the existing literature. Indeed, the picture of judicial performance that emerges from this study poses a challenge to widely-held conceptions about the identities of the "top" judges in the federal appellate courts. Some of the most prominent judges in the citation-count models and law review literature appear only average when ranked by the mix of positive and negative citations to their opinions. Similarly, some relatively low-profile judges who rarely make the pages of law review scholarship emerge as some of the nation's most highly rated judges in this ranking. The reason is relatively clear: while citation counts tend to reward the most provocative judicial entrepreneurs, this study rewards the careful judicial craftsperson. Thus, the results of this study provide a means of assessing the quality of the typical decision rendered by an appellate judge, rather than the notoriety of his or her high-profile decisions.

   This Article outlines a judicial evaluation tool that is as transparent and objective as in the method used by Choi and Gulati, but one that more directly measures the characteristics most people--especially litigants in the federal courts--are likely to think of and care about as judicial "quality." The justification for this alternative measure is that the average litigant likely cares much more about the quality of federal appellate judges than about the judges' passing of an ideological litmus test, or the likelihood the litigant's dispute will be immortalized in casebooks and law review articles. Moreover, this approach provides a means of assessing judicial performance for the purpose of judicial administration that is complementary to, rather than duplicative of, the productivity measures already used in judicial assessment, such as caseloads and backlogs. Finally, by using a much larger dataset and more detailed information than the existing studies, the quality measures in this study offer a preliminary but revealing look into the interplay of ideology and precedent in the federal appellate courts.

   This Article proceeds as follows: Part II surveys the burgeoning literature evaluating judicial performance and explains the contribution of this Article in extending that literature. It...