Randomized evaluation in legal assistance: what difference does representation (offer and actual use) make?

AuthorGreiner, D. James

INTRODUCTION A. The Research Program B. Offers Versus Actual Use I. OUR STUDY A. The Unemployment System and First-Level Appeals in Massachusetts B. The Status of Research on Representation in First-Level Appeals Prior to Our Study C. The Harvard Legal Aid Bureau D. Study Methodology E. Results II. THE CURRENT STATE OF OUR KNOWLEDGE A. Three Methodological Problems 1. Failure to Specify the Intervention: When Is Representation Assigned? 2. Selection Effects: How Is Representation Assigned? a. Judge-Induced Selection Effects: Outcome-Driven Counsel Appointments? b. Client-Induced Selection Effects, Part I: Only the Strong Want To (and Do) Survive? c. Client-Induced Selection Effects, Part II: Different Circumstances? d. Lawyer-Induced Selection Effects: Different Ways of Culling Clients? e. Selection Effects: Are Plausible Guesses Possible? 3. Accounting for Uncertainty B. How Well-Run Randomized Experiments Solve These Problems III. WHERE DO WE GO FROM HERE? A. The Limits of Randomized Studies 1. Systemic Change 2. Fielding Studies 3. Provider Objections B. Maximizing Information 1. Nonpecuniary Interests 2. Outreach, Intake, and Client Choice 3. System Accessibility and Accuracy CONCLUSION APPENDIX: "POWER CALCULATIONS" INTRODUCTION

  1. The Research Program

    Particularly with respect to low-income clients in civil cases, how much of a difference does legal representation make? (1) The question is fundamental to the legal profession and to legal pedagogy, and it implicates overlapping areas of inquiry: from access to justice and the civil Gideon movement, (2) to the design of adjudicatory systems; from legal ethics and the unauthorized practice of law, to efficiency in the delivery of legal services (a subject addressed in the organic statute of the Legal Services Corporation). (3) In 2007, we initiated discussions with various legal services providers to generate interest in a series of randomized trials (4) that we hoped would provide gold-standard answers to the question of how much of a difference both an offer of and actual use of legal representation make. We sought practice areas in which demand for legal services outstripped a provider's capacity, and in those areas, we suggested that the provider allow us to randomize which of several eligible potential clients would receive an offer of representation. (5) The randomization would create "treated" (offered representation from this service provider) and "control" (no such offer) groups identical (up to random variation) in all ways except for the offer of representation. Examination of official records for the treated and control groups would allow rigorous measurement of the difference the offer of representation made with respect to the set of outcomes recorded in those records (an important but not exhaustive set of possible outcomes of interest). Fancier statistical techniques might allow inferences about the effect of actual use of representation for the set of persons who requested help from the particular provider.

    As we explain in Part III, this provider-centered framework is not the only kind of randomized trial one can use to measure representation effects. Thus, one of our hopes was to use these early efforts as a springboard for additional randomized studies that were court-centered as well as spread across various other dimensions of interest, such as legal area (e.g., housing, family law, immigration, bankruptcy), level of representation (e.g., full attorney-client relationships, lawyer-for-the-day, advice), type of adjudicator (e.g., judge, administrative law judge (ALJ)), and style of adjudication (e.g., inquisitorial, adversarial).

    Our primary hope was, and still is, that gold-standard research would produce information useful to the legal services community. We foresee several possible uses. (6) First, our understanding is that both nationally and in every state in which the issue has been analyzed, demand (both nascent and expressed) (7) for legal services outstrips supply, (8) posing an acute problem of resource allocation (9) that has become the stuff of newspaper editorials. (10) Rigorous assessments of how much of a difference an offer or actual use of representation made in different service areas would allow funders and providers themselves to allocate scarce resources more effectively. Second, rigorous evaluation of particular programs would help determine whether programs could be altered so as to make them more effective. (11) Of particular interest to us were provider outreach and intake systems, because depending on how these were designed, it struck us that some providers might not be reaching the populations of potential clients that most needed, and would most benefit from, help. Third, if we were to discover that an offer or actual use of representation made less of a difference in certain tribunals, perhaps those tribunals could be examined to see whether the relative lack of a representation effect might be due to tribunal characteristics or practices that made these systems accessible to pro se litigants. That, in turn, might provide direction for efforts from the legal services community (to the extent allowed) (12) to advocate for reform of other tribunals so as to increase their accessibility. Fourth, we believe that a gold-standard evaluation demonstrating that a particular legal services program was effective would be useful in cost-benefit analyses (13) that might lead to arguments for increased funding, (14) which we fervently endorse.

    We pause to note that an evaluation of whether an offer or actual use of representation affects client outcomes is not the same thing as measuring the quality of a services provider's lawyering. There could be a variety of reasons why representation (offer or actual use) might not affect measured outcomes despite world-class advocacy, including, for example, (i) that the provider's outreach and intake system is producing only an unusually competent and motivated client base that does not actually need assistance, or needs it less, or (ii) that the adjudicatory system in which clients are operating is friendly toward pro se litigants, leaving less room for representation to make a difference, or (iii) that the issues to be litigated in a particular area are relatively straightforward (at least for the set of potential clients who seek representation), again leaving less room for representation to make a difference.

    Returning to our research program, not all legal services providers embraced our suggestions to engage in gold-standard evaluation, (15) but some did. In doing so, these providers demonstrated the courage necessary to subject their programs to evaluation that did not have a preordained result. Among the most courageous were the students of the Harvard Legal Aid Bureau (HLAB), a student-run, faculty-overseen legal services office that is part of the clinical educational program at Harvard Law School. The first study in our research program consisted of a randomized evaluation of HLAB's representation of claimants seeldng unemployment benefits. As we explain, the quality of HLAB representation in the unemployment context is both high and well-respected.

    This first study led to unexpected results. With respect to the claimants reached by the outreach and intake systems of HLAB's unemployment practice, and with respect to outcomes measurable from official records (which concerned a claimant's pecuniary interests), the randomized evaluation determined that an offer of HLAB representation had no statistically significant effect on the probability that a claimant would prevail, but that the offer did delay the adjudicatory process. This finding does not mean that we know that the HLAB offer had no positive effect on a claimant's probability of success. We can say, however, that any such effect is unlikely to have been large (16) (or else the data probably would have shown it), and that we do have a high degree of confidence in the delay finding.

    Meanwhile, because roughly one-third of HLAB's client base consisted of claimants who were erroneously denied benefits as an initial matter and who would eventually have that erroneous denial reversed, the statistically significant delay probably (17) meant that many of these claimants who were offered HLAB assistance suffered the harm of having to wait longer for their benefits to begin. We note that courts, legal services providers, commentators, and the U.S. Department of Labor have repeatedly and emphatically stated that the unemployment benefits system depends on speed, in a way other benefits programs do not, for its effectiveness. (18) As we explain, there were also potential consequences for the financing of the unemployment system associated with cases in which there was an erroneous grant of benefits that would eventually be reversed. With respect to the actual use of, as opposed to an offer of, representation, we were able to determine that a delay effect was again present; we could come to no firm conclusion regarding a use-of-representation effect on the win rate. In sum then, in a purely pecuniary sense, the set of claimants HLAB's intake system allowed it to reach might have been better off not receiving the HLAB offer of assistance.

    The unexpected results of the HLAB study prompted us to reexamine the literature purporting to assess quantitatively how much of a difference legal representation makes in civil cases. We found that despite the fact that the question has been studied dozens of times, (19) very few of these studies (only two, really, both of which were randomized) (20) are worthy of credence. Almost all of these studies follow the same basic design, which is a comparison of the outcomes of cases with representation versus cases without representation, sometimes with regressions using predictors available in official case files. Partly as a result of this design, almost all such studies suffer...

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT