William T. Bielby & Pamela Coukos, "statistical Dueling" With Unconventional Weapons: What Courts Should Know About Experts in Employment Discrimination Class Actions

• Publication year: 2007

"STATISTICAL DUELING" WITH UNCONVENTIONAL WEAPONS: WHAT COURTS SHOULD KNOW ABOUT EXPERTS IN EMPLOYMENT DISCRIMINATION CLASS ACTIONS

William T. Bielby^*

Pamela Coukos^**

ABSTRACT

When statistical evidence is offered in a litigation context, the result can be bad law and bad statistics. In recent high-profile, high-stakes employment discrimination class actions against large multinationals like UPS, Wal-Mart, and Marriott, plaintiffs have claimed that decentralized and highly discretionary management practices result in systematic gender or racial disparities in pay and promotion. At class certification, plaintiffs have relied in part on statistical analyses of the company's workforce showing companywide inequality. Defendants have responded with statistical presentations of their own, which frequently demonstrate widely varying outcomes for members of protected groups in different geographic areas of the company. These expert submissions usually suggest either that no problems exist, or that any discrimination is isolated and not attributable to institutional-level bias. In adjudicating between these competing visions, courts must referee what the Second Circuit terms "statistical dueling." As we show in this Article, sometimes at least one of the parties is dueling with unconventional weapons. Using simulated data, we show why courts should become more critical of statistical expertise purporting to test for subunit differences, particularly when offered at the class certification phase of the case. Under some circumstances, the statistical approach often used to oppose class certification in employment discrimination litigation is guaranteed to support the defendant's position, regardless of the actual facts of the case. Furthermore, some courts have improperly or unwittingly legitimized the use of this approach, even when it is demonstrably nonprobative of the issues before the court. Courts need new ways to think about these problems-approaches that better reflect the relevant legal framework and statistical principles.

INTRODUCTION

When statistical evidence is offered in a litigation context, the result can be bad law and bad statistics. Recent high-profile, high-stakes employment discrimination class actions bear this out.¹A series of similar cases litigated over the past several years involve potentially misleading statistical testimony purporting to show an absence of any pattern of discrimination.²Courts in these cases have not always understood the limitations of the statistical evidence before them or properly weighed its relevance to a ruling on class certification. Because the decision about whether to certify a class is critical to both sides, these errors may generate substantial costs.

In recent cases against large multinationals like UPS,³Wal-Mart,⁴and Marriott,⁵plaintiffs have claimed that decentralized and highly discretionary management practices result in systematic gender or racial disparities in pay and promotion. At class certification, plaintiffs have relied in part on statistical analyses of the company's workforce showing companywide inequality. Defendants have responded with statistical presentations of their own, which frequently demonstrate widely varying outcomes for members of protected groups in different geographic areas of the company.⁶These expert submissions usually suggest either that no problems exist or that any discrimination is isolated and not attributable to institutional-level bias. In adjudicating between these competing visions, courts must referee what the Second Circuit terms "statistical dueling."⁷

As we show in this Article, sometimes at least one of the parties is dueling with unconventional weapons. For-profit consulting companies and large defense firms are eagerly marketing unorthodox and unreliable statistical methods to employers anxious about how class actions multiply their potential liability.⁸Of course, plaintiffs can, and sometimes do, submit statistical evidence that is inconsistent with good social science practice and biased in favor of their position. However, a bias in the opposite direction is often lurking within the statistical approach of experts working for defendants, and courts and litigators have largely ignored this bias. Moreover, while the issue as outlined below will be immediately apparent to social statisticians, it has received little attention from either academic statisticians or consulting experts.

Using simulated data, we show why courts should become more critical of statistical expertise purporting to test for subunit differences, particularly when offered at the class certification phase of the case. Under some circumstances, the statistical approach often used to oppose class certification in employment discrimination litigation is guaranteed to support the defendant's position, regardless of the actual facts of the case. Furthermore, some courts have improperly or unwittingly legitimized the use of this approach, even when it is demonstrably nonprobative of the issues before the court. Courts need new ways to think about these problems-approaches that better reflect the relevant legal framework and statistical principles.

The conflicting expert submissions typical of contemporary Title VII class litigation reflect each side's distinct litigation strategy, as framed by the legal requirements for class certification. Rule 23 of the Federal Rules of Civil Procedure requires that courts facing class certification motions determine the existence and extent of common factual and legal issues.⁹The more that individual complaints of discrimination are part and parcel of a challenge to larger common institutional practices, the more appropriate it would be to certify a class.¹⁰The more idiosyncratic the claims are, the less reasonable class treatment becomes. Thus, when it comes to statistical evidence, plaintiffs and their experts focus on similarities, while defendants and their experts highlight differences. This debate over similarities and differences focuses on two frequently litigated and interrelated legal issues. The first is the use of aggregated versus disaggregated analyses, and the second is the potential commonality of subjective practices.

Disputes over aggregation played a central role in a number of recent class action decisions involving multi-facility classes.¹¹Where a case challenges company policies or practices across geographically dispersed worksites, be they divisions, regions, or individual retail stores, defendants typically claim that any statistical evidence must account for potential subunit differences.¹²

Mainly because of idiosyncrasies in the way case law has evolved, courts place more weight on "statistical significance" than on the magnitude of disparities between groups.¹³As a result, plaintiffs' statistical experts typically assess disparities with organization-wide data, pooled across geographic subunits.¹⁴

In contrast, defendants' experts take the stance that where the company employs a decentralized management structure, a unique employment system exists within each organizational subunit.¹⁵Therefore, statistical estimates of disparities must be computed separately by subunit. These different approaches frequently generate quite different results.

The issue of aggregation becomes more important in cases involving highly discretionary management practices. Increasingly, plaintiffs alleging systematic discrimination claim that the company-wide mechanism creating bias is discretionary and subjective decision making, which is implemented in the context of a decentralized personnel system. Plaintiffs further charge that this system lacks sufficient monitoring and oversight regarding the process and criteria used for making decisions about pay, promotion, and other conditions of employment.¹⁶While longstanding legal doctrine clearly permits plaintiffs to challenge these mechanisms as a common practice applicable to a class of employees,¹⁷defendants frequently use the plaintiffs' framing of the problem as a basis to argue against class certification. They maintain that if decision making is truly discretionary, then by definition there cannot be a common policy or practice causing the alleged bias. Further, they point to statistical evidence of subunit differences, generated through a disaggregated analysis, as support for that position.¹⁸

Many of the employment discrimination class actions currently being litigated in federal court exemplify what some legal scholars call "second- generation" employment discrimination.¹⁹While class action claims brought in the early days of Title VII often featured extreme levels of job segregation and anecdotal evidence of overtly racist and sexist management decisions, their progeny tell a more complex story. Today's major corporate targets typically have at least a token representation of men and women of color and white women at senior levels. They avidly describe their good faith steps to combat discrimination within the corporation and appear to lack significant documented instances of explicit discriminatory animus.²⁰Many scholars and advocates on both sides of the civil rights enforcement debate openly question whether the existing Title VII legal regime is adequate or appropriate for dealing with these cases.²¹

In this context, one might expect heightened judicial receptivity to arguments that discretionary decision making is not amenable to class treatment, especially when elaborate statistical presentations accompany that argument. In the context of a move to "second-generation" claims, defendants may be more successfully framing discrimination as individual deviance, rather than as stemming from structural or institutional factors. Thus, while in earlier cases a number of courts viewed "excessive subjectivity" as highly suspicious and easily linked to systemic practices of discrimination,²²more recent cases seem to treat such practices as neutral until...