Proxy Discrimination in the Age of Artificial Intelligence and Big Data

• Author: Anya E.R. Prince & Daniel Schwarcz
• Position: Associate Professor, University of Iowa College of Law/Fredrikson & Byron Professor of Law, University of Minnesota Law School
• Pages: 1257-1318

1257

Proxy Discrimination in the Age of

Artificial Intelligence and Big Data

Anya E.R. Prince & Daniel Schwarcz*

ABSTRACT: Big data and Artificial Intelligence (“AI”) are revolutionizing

the ways in which firms, governments, and employers classify individuals.

Surprisingly, however, one of the most important threats to anti-

discrimination regimes posed by this revolution is largely unexplored or

misunderstood in the extant literature. This is the risk that modern algorithms

will result in “proxy discrimination.” Proxy discrimination is a particularly

pernicious subset of disparate impact. Like all forms of disparate impact, it

involves a facially neutral practice that disproportionately harms members of

a protected class. But a practice producing a disparate impact only amounts

to proxy discrimination when the usefulness to the discriminator of the facially

neutral practice derives, at least in part, from the very fact that it produces a

disparate impact. Historically, this occurred when a firm intentionally sought

to discriminate against members of a protected class by relying on a proxy for

class membership, such as zip code. However, proxy discrimination need not

be intentional when membership in a protected class is predictive of a

discriminator’s facially neutral goal, making discrimination “rational.” In

these cases, firms may unwittingly proxy discriminate, knowing only that a

facially neutral practice produces desirable outcomes. This Article argues that

AI and big data are game changers when it comes to this risk of

unintentional, but “rational,” proxy discrimination. AIs armed with big data

are inherently structured to engage in proxy discrimination whenever they are

deprived of information about membership in a legally suspect class whose

predictive power cannot be measured more directly by non-suspect data

available to the AI. Simply denying AIs access to the most intuitive proxies for

such predictive but suspect characteristics does little to thwart this process;

instead it simply causes AIs to locate less intuitive proxies. For these reasons,

*

Anya E.R. Prince (anya-prince@uiowa.edu) is an Associate Professor, University of Iowa

College of Law. Daniel Schwarcz (Schwarcz@umn.edu) is the Fredrikson & Byron Professor of

Law, University of Minnesota Law School. For comments and suggestions on preliminary drafts,

we thank Ken Abraham, Ronen Avraham, Jessica Clarke, I. Glenn Cohen, James Grimmelman,

Jill Hasday, Claire Hill, Dave Jones, Sonia Katyal, Pauline Kim, Kyle Logue, Peter Molk, Chris

Odinet, Nicholson Price, Jessica Roberts, Andrew Selbst, Elizabeth Sepper, R ory Van Loo and

participants of the Consumer Law Conference at Berkeley Law School.

1258 IOWA LAW REVIEW [Vol. 105:1257

as AIs become even smarter and big data becomes even bigger, proxy

discrimination will represent an increasingly fundamental challenge to anti-

discrimination regimes that seek to limit discrimination based on potentially

predictive traits. Numerous anti-discrimination regimes do just that, limiting

discrimination based on factors like preexisting conditions, genetics,

disability, sex, and even race. This Article offers a menu of potential strategies

for combatting this risk of proxy discrimination by AIs, including prohibiting

the use of non-approved types of discrimination, mandating the collection and

disclosure of data about impacted individuals’ membership in legally

protected classes, and requiring firms to employ statistical models that isolate

only the predictive power of non-suspect variables.

I. INTRODU CTION ........................................................................... 1259

II. PROXY DISCRIMINATION BY HUMANS AND AIS ............................ 1267

A. PROXY DISCRIMINATION BY HUMAN ACTORS .......................... 1268

B. PROXY DISCRIMINATION BY AIS ............................................. 1273

C. UNDERSTANDING WHEN PROXY DISCRIMINATION BY AIS IS

LIKELY TO OCCUR ................................................................. 1276

1. Direct and Indirect Proxy Discrimination ................. 1276

2. The Difficulty of Identifying Causal, Opaque,

and Indirect Proxy Discrimination by AIs in

the Real World ............................................................. 1281

III. THE HARMS OF PROXY DISCRIMINATION BY AIS ......................... 1283

A. ANTI-DISCRIMINATION REGIMES AT RISK OF PROXY

DISCRIMINATION BY AIS ........................................................ 1283

1. Health Insurance ......................................................... 1284

2. Non-Health Insurance ................................................ 1285

3. Employment ................................................................ 1286

4. Other Legal Areas ....................................................... 1288

B. PROXY DISCRIMINATION BY AIS UNDERMINES THE INTENDED

GOALS OF IMPACTED ANTI-DISCRIMINATION REGIMES ............ 1289

1. Promoting Social Risk Sharing ................................... 1291

2. Preventing the Chilling of Socially Valuable

Behavior ....................................................................... 1292

3. Limiting or Reversing the Effects of Past

Discrimination ............................................................. 1295

4. Anti-Stereotyping......................................................... 1297

IV. RESPONDING EFFECTIVELY TO PROXY DISCRIMINATION ............. 1300

A. INEFFECTIVE SOLUTIONS ........................................................ 1300

1. Ban Discriminators’ Use of Obvious Proxies for

Protected Characteristics ............................................ 1300

2020] PROXY DISCRIMINATION 1259

2. Traditional Disparate Impact Liability ...................... 1304

B. POTENTIALLY EFFECTIVE STRATEGIES FOR COMBATTING

PROXY DISCRIMINATION BY AIS ............................................. 1306

1. Flipping the Default: Prohibiting Discrimination

Based on Non-Approved Factors ................................ 1306

2. Expanding the Information Used: Requiring

More Data to Limit Certain Types of Proxy

Discrimination ............................................................. 1310

3. Transparency-Oriented Reforms ............................... 1311

4. Ethical Algorithms that Explicitly Control for

Proxy Discrimination .................................................. 1313

5. Requirement of Potential Causal Connections ......... 1316

V. CONCLUSION .............................................................................. 1318

I. INTRODUCTION

Big data and Artificial Intelligence (“AI”) are revolutionizing the ways in

which firms, governments, and employers classify individuals.1 Insurers, for

instance, increasingly set premiums based on complex algorithms that process

massive amounts of data to predict future claims.2 Prospective employers

deploy AI and big data to decide which applicants to interview or hire.3 And

various actors within the criminal justice system—ranging from police

departments to judges—now use predictive analytics to guide their decision-

making.4

1. We use the term “artificial intellig ence” to encompass a broad array of computational

techniques for predicting future outcomes based on analysis of past data. These techniques

include “machine learning,” “deep learning,” “learning algorithms,” and many other terms.

While there are often important differences among these various types of AIs, these distinctions

are not pertinent to the analysis in this Article.

2. See Rick Swedloff, Risk Classification’s Big Data (R)evolution, 21 CONN. INS. L.J. 339,

340–44 (2014); Herb Weisbaum, Data Mining Is Now Used to Set Insurance Rates; Critics Cry Foul,

CNBC (Apr. 16, 2014, 11:29 AM), https://www.cnbc.com/2014/04/16/data-mining-is-now-

used-to-set-insurance-rates-critics-cry-fowl.html [https://perma.cc/MQ28-C8RA]; see also Ray

Lehmann, Why ‘Big Data’ Will Force Insurance Companies to Think Hard About Race, INS. J. (Mar. 27,

2018), https://www.insurancejournal.com/blogs/right-street/2018/03/27/484530.htm [https://

perma.cc/4GBZ-MBZZ] (“According to a 2015 survey conducted by Willis Towers Watson, 42

percent of executives from the property and casualty insurance ind ustry said they were already

using big data in pricing, underwriting and risk selection, and 77 percent said they expected to

do so within two years.”).

3. See Pauline T. Kim, Data-Driven Discrimination at Work, 58 WM. & MARY L. REV. 857,

860 (2017) (“Employers are increasingly relying on data analytic tools to make personnel

decisions . . . .”).

4. See Aziz Z. Huq, Racial Equity in Algorithmic Criminal Justice, 68 DUKE L.J. 1043, 1068–76

(2019); Elizabeth E. Joh, Policing by Numbers: Big Data and the Fourth Amendment, 89 WASH. L. REV.

35, 42–55 (2014); Sharad Goel, Ravi Shroff, Jennifer Skeem & Christopher Slobogin, The

Accuracy, Equity, and Jurisprudence of Criminal Risk Assessment 1 (Dec. 26, 2018)