Distinguishing judges: an empirical ranking of judicial quality in the United States Courts of Appeals.

• Author: Anderson, Robert, IV
• Position: IV Results and Interpretation continued through Conclusion, p. 349-384

The results of this analysis differ dramatically from those of prior judge ranking studies. (111) The top ten percent of the rankings (roughly ranks 1-40) contains some well-known judges, such as now-Justices Stephen Breyer, Anthony Kennedy, and Sonia Sotomayor, as well as Judges Henry Friendly and Kenneth Starr, among others. In addition, the top of the rankings contains some names that also stood out in Choi and Gulati's measure, such as Judge Bruce Selya of the First Circuit. But most of the names at the top are not very familiar, such as that of first-place Judge William Riley of the Eighth Circuit. Similarly, the bottom of the ranking contains some notoriously ideological judges but also contains some judges who are surprising to see at the bottom of a quality ranking. Overall, most of the judges fall into a rather large group in the middle and are not statistically distinguishable from one another.

To compare the results with those of Choi and Gulati in the context of more recent judges, consider Table III, which presents the "composite" ranking from Choi and Gulati's "tournament" (112) side-by-side with the rankings from Table II. Because Choi and Gulati ranked a much smaller number of judges, only 95 of the 383 are presented, with their ranks renumbered accordingly. Comparing the two tables suggests that there is no visible relationship between the Choi and Gulati rankings and the quality rankings as computed in this study. Some of the lowest judges in Choi and Gulati's ranking, such as Judge Stanley Marcus of the Eleventh Circuit and Judge Karen Henderson of the D.C. Circuit, come out near the top of this Article's ranking.

The most remarkable surprises, however, are probably the judges who ended up in the middle or bottom of this ranking but at the top of the Choi and Gulati ranking. In particular, the conspicuous absence of the influential judge par excellence in the Choi and Gulati ranking--Judge Richard Posner--from the top of this list is striking. In most rankings in the citation-count studies, Judge Posner ranks first among federal appellate judges, in many cases far ahead of the second place judge. (113) In Table III, however, we find that Judge Posner, although ranked first in the Choi and Gulati study, is well below the median in terms of the mix of positive and negative citations. Similarly, the judges who placed second (Frank Easterbrook), third (J. Harvie Wilkinson), fourth (Paul Niemeyer), and fifth (Jerry Smith) in Choi and Gulati's composite measure are similarly undistinguished in this ranking, all hidden in the middle of Table III. Only Judge David Ebel of the Tenth Circuit and Judge Diane Wood of the Seventh Circuit make the top ten judges on both lists in Table III. Why the striking differences between the results in Tables II and III, on the one hand, and the citation-count studies, on the other hand?

To answer this question, we must first determine why the rankings in column 4 of Tables II and III are so different from those of the citation-count studies. Is it the use of inside-circuit as well as outside-circuit decisions? Or is it the use of panels on which judges served rather than the opinions they wrote? Or both? Consider column 5 of Table III, which presents the results of the same (panel-based) model based only on outside-circuit citations. The rankings have a strong relationship to those in column 4, which is not surprising because they include many of the same data points but still bear no relationship to the Choi and Gulati rankings. It does not appear that the inside-circuit versus outside-circuit citations are driving the different results between the two approaches.

What about the use of panel membership in this study rather than only authored opinions as in citation-count studies? Column 6 conducts the same analysis as column 4 but ranks judges based on positive and negative citations to their authored opinions, rather than on the panels in which they participated. Again, the results resemble those in column 4 but differ sharply from those of Choi and Gulati. Finally, column 7 of Table III presents the opinion-based model utilizing only outside-circuit citations, making the dataset most similar to Choi and Gulati. The results still are not any closer to resembling those of the citation-count studies. Thus, the differences in rankings between this study and the count studies result not only from the use of inside-circuit citations or panels rather than authored opinions but also from the use of positive and negative citations.

One might still argue, however, that the results in this Article do not meaningfully measure whether decisions are cited positively or negatively. For example, perhaps the Shepard's Citation treatments are not as valid and reliable as the studies have suggested and perhaps the objectivity of citation counts makes a better measure of quality. To determine whether this explanation accounts for the different results in this Article, one can examine the coefficients on variables for which we know the effect--the control variables--to learn how well the positive and negative treatments from Shepard's Citations correspond to actual judicial citations. For example, it would be very surprising if judges did not cite precedents in their own circuits more favorably than those outside their circuits, or if judges did not cite other judges of their political party more favorably than judges of the opposite party. So if the Shepard's Citations measures did not reflect these empirical regularities, we might suspect a problem with the data. But if the Shepard's Citations measures do reflect these empirical regularities, we know that they are, in fact, measuring something meaningful.

Table IV presents the results for these control variables, one column for each column of Table II. As expected, the coefficients are positive and large for inside-circuit citations (including the Eleventh Circuit citing the Fifth Circuit), indicating that inside-circuit citations are treated more positively than outside-circuit citations. Similarly, the ideological variables are also significant and in the directions that one would expect: (114) judges appointed by Republicans cite judges appointed by Democrats more negatively than they cite those appointed by Republicans, and the opposite is true as well. Most of the ideological variables are also highly statistically significant, although nowhere near the circuit variables either in magnitude or in statistical significance. These control variables, all in the expected directions and virtually all highly statistically significant, suggest that the positive and negative Shepard's treatment codes, on average, measure exactly what they purport to measure.

2. Sitting Judges and the Sotomayor Nomination

   The analysis performed above attempts (perhaps overly ambitiously) to place federal appellate judges who served at different times over a fifty-year period on a common scale of quality. This requires introducing control variables for different time periods such that, perhaps counterintuitively, some judges who are higher in the ranking can have a lower proportion of positive votes than some judges who are lower in the ranking, depending on when the judges served. To the extent that these controls are incorrectly constructed, they may distort the analysis and present biased results. Nevertheless, some control variables are necessary to compare judges intertemporally in a dataset with such strong patterns over time. Because some might object to the choice of control variables or find the methodology less than intuitive, this section develops an alternative ranking for current judges based on simple percentages.

   By comparing positive and negative citations to judges who served over the same time period, one arguably can get a rough sense of relative judicial quality without using any control variables. (115) This section uses the opportunity presented by the nomination of then-Judge Sonia Sotomayor to the Supreme Court to present such a ranking for the roughly ten-year period she served on the Second Circuit. The following two tables therefore present a ranking of the ninety-five active (116) judges who served during the same period that then-Judge Sotomayor was a circuit judge, beginning on October 7, 1998 (the date of her commission) (117) and ending in mid-2008 (the start of data collection). The judges are ranked by the percentage of positive citations (among all positive and negative citations) to the published opinions each judge wrote during the period. (118) Separate tables are presented for outside-circuit citations, a panel in another circuit citing a judge's opinion, and inside-circuit citations, a panel in the same circuit citing the judge's opinion.

   Table V presents the simplified ranking based on outside-circuit citations and Table VI presents the simplified ranking based on inside-circuit citations. The results are rather surprising. Then-Judge Sotomayor ranks third out of ninety-five active judges in outside-circuit citations, with 81.6% of her citations positive. However, she ranks eightieth out of ninety-five in inside-circuit citations, with 76% of her citations positive. Indeed, then Judge Sotomayor's inside-circuit citations are more negative than her outside circuit citations, which is true of only five other judges in Table V. Moreover, the gap between her inside-circuit and outside-circuit ratings is larger than any of these other five judges. Even more surprising, the next largest difference of this kind is that of Judge Merrick Garland of the D.C. Circuit, who was widely reported to be one of the three top courts of appeals candidates for the nomination that ultimately went to then-Judge Sotomayor. (119) What could explain this remarkable pattern?

   One possible explanation for the discrepancy could be the widely held perception that both the Second and D.C. Circuits are highly influential and...