INTRODUCTION I. DATA DESCRIPTION II. RESULTS III. DISCUSSION CONCLUSION INTRODUCTION
Evaluating the legal profession is a daunting task. The profession is highly decentralized, and lawyers work in myriad practice settings. Most litigation ends in settlement, (1) creating little or no public record. Lawyers increasingly specialize, (2) which complicates comparison across areas of law. Because lawyers and clients typically choose one another, it is difficult to separate lawyer ability from ease characteristics. (3) For these reasons, much of our understanding of legal representation comes from careful examination of discrete segments of the profession, (4) practice settings, (5) or geographic regions. (6)
What is missing is a comprehensive evaluation of legal representation. Lawyers--like most workers--are heterogeneous in ability, (7) but we have only a limited understanding of how lawyers of different quality are distributed within and across the profession. Human capital theory posits that higher wages attract higher-skilled workers, (8) but we have little empirical evidence to support or rebut this theory as applied to lawyers. A related point is that we lack a good understanding of how lawyers influence ease outcomes.
Given the paucity of existing data, we decided to survey members of the profession. We decided against surveying lawyers, however, given their limited perspective and likely biases. (9) Instead, we decided to survey judges. Of course, judges are not without their own biases, (10) and their perception of lawyers is limited to written documents and in-court observations. But most judges preside over courts that have a general jurisdiction, and so they encounter lawyers in diverse areas of law and practice settings, which allows them to make comparisons across the population of lawyers.
Other scholars have surveyed judges to better understand institutional aspects of the legal profession, such as jury verdicts, (11) oral argument, (12) court-appointed experts, (13) clerkship hiring, (14) gender bias, (15) and judicial retirement, (16) to name a few. Our survey differs in focusing on how the adversarial system influences legal outcomes.
Our survey was of 666 federal and state judges--both appellate and trial--and was conducted in the spring and summer of 2008. The survey asked judges to answer questions relating to their perceptions of the quality of legal representation, and how that quality--and significant disparities in quality between opposing counsel--influences how they and juries decide cases. We also asked judges for their recommendations for improving law schools, the practicing bar, and the judiciary.
We found that judges perceive significant disparities in the quality of legal representation in criminal cases, and that these disparities occur in 20% to 40% of the cases they hear. Federal judges generally rate prosecutors as comparable in quality to public defenders and significantly better than court-appointed counsel or retained counsel. State judges agree with respect to the high quality of prosecutors but hold retained counsel in higher regard than public defenders or court-appointed counsel. In civil cases, judges gave their highest ratings to lawyers handling commercial litigation and intellectual property and their lowest ratings to immigration and family lawyers. Federal judges reported that the lawyers on one side of immigration and civil rights cases are consistently abler; in contrast, state judges found sharp quality differences in family law but did not find that the differences systematically favored one side. Both federal and state judges reported greater disparities in the quality of representation in civil cases than in criminal cases.
Judges see themselves as responding differently from juries to significant disparities in the quality of legal representation. The majority of judges responded that they engage in additional research to compensate for these disparities when they arise. In contrast, most judges thought that jurors are inclined, other things being equal, to favor the litigant with the higher-quality lawyer.
When asked to propose reforms aimed at improving legal representation, most judges suggested curricular changes, both doctrinal and clinical, in law schools. They also recommended reducing disparities in resources for legal services, either by increasing wages for lawyers in the public sector or by increasing public financing for indigent litigants. They cited a need to help judges handle increased caseloads by increasing the number of judges, and a high percentage of judges called for higher judicial salaries.
The Article proceeds as follows. Part I describes our design and methodology of the survey. Part II presents the results. Part III discusses implications of these results, and the final Part concludes.
We surveyed federal and state judges separately. We now describe the process by which we administered the survey, the questions we asked, and basic summary statistics.
Federal Survey: We mailed the federal survey to 456 active Article III district and appellate judges, randomly selected from the list of 834 such judges provided by the clerkship office at Northwestern University School of Law in the fall of 2007. (17) The randomization was conducted within each federal circuit, excluding the Federal Circuit; forty judges were selected from each circuit. (18) They were not asked to provide their names or any other unique identifiers. A few did identify themselves, however, offering to provide additional comments in person or by phone, and we contacted them.
State Survey: We administered the state survey with the generous help of the National Center for State Courts (NCSC), which sent the survey to judicial groups that had an affiliation with the NCSC. These included the American Judges Association (both trial and appellate judges), the Conference of Chief Judges (appellate only), and the justices of the state supreme courts. The NCSC sent the invitation via e-mail. Those willing to participate could click on a link that took them to the secure, encrypted online instrument. (19) As with the federal survey, the state respondents were assured anonymity. Because some state judges have a limited jurisdiction (for example, they preside over only criminal or only civil cases), we asked additional questions concerning the docket of the participating judge.
Content of the Survey: We describe the substantive questions and the judges' responses in greater detail in Part II. Most questions were in multiple-choice format, asking the judge either to provide his or her response on a five-point scale (for example, ranking the quality of legal representation from poor (1) to excellent (5)), or to choose a response among a nonordinal set of choices (for example, changes to the practicing bar that the judge believed would most benefit the judiciary). We also invited judges to provide open-ended comments at the end of the survey, which approximately one-quarter of the judges did. Where relevant, we integrate these comments into the Article.
In each table we report the number of judges who responded to each question. With some of the questions, the number of responses varies slightly because some judges did not answer all the questions. This variation occurs primarily among state judges when answering questions relating to criminal and some areas of civil law. (20)
Summary Statistics: We are particularly interested in two sets of comparisons: between federal and state judges and between appellate and trial judges. Table 1 breaks down the survey statistics by these categories, as do subsequent tables. (21)
A general note about the tables: most responses report a mean for each judge group. To evaluate statistical significance across judges on a single question, or within the same judge for a repeated set of questions, we ran an ANOVA (analysis of variance) test using a Bonferroni correction to evaluate the differences in means between multiple groups. (22) Unless otherwise stated, we report statistical significance levels at the p
Table 1 shows that the overall response rate for federal judges was 52%. The district judge response rate was 52%, while the circuit judge response rate was 49%, a difference that was not statistically significant. For the state judges, the overall response rate was 37%--still significantly higher than the typical response rate for unsolicited e-mail surveys. (23) Because one of the state judicial organizations participating in the survey--the American Judges Association--consists of both state trial and appellate judges, we were unable to determine the precise number of surveys sent to trial and appellate judges, respectively, within this group.
Table 2 reports summary statistics for the judges. (24) In this and the subsequent tables, the numbers preceding the question correspond to the number of the question in the survey.
The vast majority of judges have had experience in private practice--typically in a firm environment. A sizable percentage have also had experience practicing criminal law, the difference being that federal judges were more likely to have been criminal defense lawyers while state judges were more likely to have been prosecutors.
The number of responses varies by region, reflecting differences in the number of judges, but differences in response rates across regions were small and not statistically significant. Responses at the state trial level are disproportionately high from the region corresponding to the Sixth Circuit; the reason is doubtless the high level of membership of Kentucky judges in the American Judges Association. (25) With few exceptions, (26) their responses were not statistically distinguishable from other state trial judges.
We now report the judges' survey responses in categories described below. We reserve our interpretation of these results until Part III.