Why affirmative action does not cause black students to fail the bar.

• Author: Ho, Daniel E.
• Position: Response to Richard H. Sander, Stanford Law Review, vol. 57, p. 367, November 2004

Richard H. Sander, A Systemic Analysis of Affirmative Action in American Law Schools, 57 STAN. L. REV. 367 (2004).

In a widely discussed empirical study, Richard Sander concludes that affirmative action at U.S. law schools causes blacks to fail the bar. (1) If correct, this conclusion would turn the jurisprudence, policy, and law of affirmative action on its head. (2) But the article misapplies basic principles of causal inference, which enjoy virtually universal acceptance in the scientific community. (3) As a result, the study draws internally inconsistent and empirically invalid conclusions about the effects of affirmative action. Correcting the assumptions and testing the hypothesis directly shows that for similarly qualified black students, attending a higher-tier law school has no detectable effect on bar passage rates.

Part I clarifies the assumptions implicit in Sander's study and explains the inconsistent and indefensible premises on which it rests. Part II presents results from a reanalysis of the data, using alternative methods that correct and reduce the role of these unjustifiable assumptions. The reanalysis suggests that Sander's conclusions are untenable on their own terms. (4) Part

III concludes.

At the outset, it is important to note that because all the schools in the LSAC Bar Passage Study on which Sander's analysis relies employ some system of affirmative action, no direct conclusion about the effects of affirmative action can be sustained. (5) While researchers in other areas have capitalized on variation in affirmative action rules to identify the effects of affirmative action, such variation does not exist here. (6)

Because there is no information in the data set with which to examine the direct causal effect of affirmative action, Sander is relegated to investigating a different quantity of interest: the causal effect of attending a higher-tier law school. While this is not a causal effect of affirmative action per se, it may be informative in assessing affirmative action's policy impact. For instance, if Sander is correct in claiming that similarly qualified students who go to higher-tier schools (1) are "mismatched" in terms of academic credentials, (2) learn less, and (3) are thus more likely to fail the bar, (7) then affirmative action might appear to hurt those it aims to help.

So how do we investigate the causal effect of attending a higher-tier school? Here, using nontechnical terms, I introduce the assumptions required to interpret Sander's findings as causal effects (8) and show that the study's assumptions are implausible and internally inconsistent.

Two basic tenets underlie any causal inference. The first is that causal inference is inherently counterfactual. (9) If we are interested in how Student A's bar performance would be affected by attending a first-tier versus a second-tier law school, we would ideally observe A attending both schools. Yet if A attends a first-tier school, we cannot observe her in the counterfactual world where she attends a second-tier school. This "fundamental problem of causal inference" (10) plagues even controlled, randomized laboratory experiments: If a unit is exposed to the treatment, we do not observe it under control.

The second tenet of causal inference is that we must at least be able to imagine conducting an experiment that manipulates a "treatment," or causal factor of interest. Laboratory scientists assess causal effects by actually conducting such experiments. For Sander's study, this would require randomly assigning a subset of students to tiers (the treatment) and observing differences in bar passage rates (the outcome). Randomization and a sufficiently large sample ensure that the students we are comparing across tiers are similar, such that different rates of bar passage can be attributed to the treatment. To estimate the average causal effect we can then simply calculate the difference in bar passage rates across tiers.

The problem for legal scholars and social scientists is that laboratory experiments are often infeasible, expensive, or unethical. Instead, to investigate causal effects researchers must resort to analyzing data in which there is no treatment randomization (so-called "observational data"). The hypothetical experiment discussed above nonetheless elucidates the key assumptions in standard methods (e.g., regression) used to infer causal effects from observational data. The goal of such methods is simply to get as close as possible to the hypothetical experiment by holding constant all other factors that affect the outcome but are present prior to the treatment.

I focus here on the key result in Sander's study of the causal effect of law school tier on bar passage. (11) The study attempts to explain the outcome of bar passage with a regression...