THE ALGORITHM GAME.

• Author: Bambauer, Jane
• Position: ARTICLES

INTRODUCTION 2 I. THE GAME 5 A. What Is Game? 6 B. Who Got Game? 11 C. Where's the Game in Life? 12 Example 1: Policing and Probable Cause 16 Example 2: Employability Scoring 17 Example 3: Financial Tech Firms and Alternative Credit Scoring 19 Example 4: Corporate Reputation Management 21 II. GAMEABLE. SO WHAT? 22 A. Autonomy and Dignity 23 B. Accuracy 25 C. Distributional Fairness 28 D. Other Inefficiencies 32 III. LAW AND GAMING 33 A. Laws Promoting Gaming and Impeding Countergaming Strategies 34 B. Laws Impeding Gaming and Promoting Countergaming Strategies 43 C. Laws Eliminating the Need for Gaming 45 CONCLUSION 47 INTRODUCTION

Modern life is judgmental. For any modern person or business, scarcely a day goes by without some experience of being assessed and differentiated from their peers. Some of these judgments are relatively trivial, as when Google decides which ads to serve to which end users. But others are consequential and profoundly personal. They are carried out by both government and private entities, as when employers decide who to hire and how much to pay them, when creditors decide what interest rate to offer on a loan, when police officers decide who to search, or when dating websites decides who to recommend for courtship. Differentiating between people in order to allocate scarce resources is not new, but these assessments are increasingly made with the help of automated predictions based on exhaustive information collected from a variety of sources.

The shift from more organic, subjective, and noisy human-based decision-making processes to mechanical ones has motivated a large, diverse, and critical literature. Scholars have identified many problems that algorithmic decisionmaking can introduce or exacerbate, including opacity, (1) lack of accountability, (2) power imbalances, (3) discriminatory effects, (4) hassle to the people being judged, (5) indignity from being treated by a machine, (6) the lack of due process, (7) and an insatiable appetite for surveillance. (8) But so far, the legal literature has focused on the effects of algorithms in static mode. The policy literature has largely assumed that algorithmic systems dictate a score, and individuals accept the results. (9) This simplification is useful for initial exploration, but much can be learned by dispensing of it. Life is dynamic, and individuals change their behavior in anticipation of how they are judged and what the consequences will be. Within limits, people game the system for a range of altruistic and self-serving reasons. And algorithm designers game right back, using countermoves to discourage gaming or to reduce its effects.

Some scholars have analyzed particular aspects of gaming. Finn Brunton and Helen Nissenbaum have given a definition and rationale for obfuscation--that is, principled resistance and sabotage of assessment systems. (10) Rush Atkinson has explored how suspicion factors used to justify police stops and searches wind up altering human behavior. (11) Joshua Kroll and his coauthors have acknowledged that data subjects can engage in strategic behavior that could render algorithm transparency undesirable even if it were possible (which they doubt). (12) Computer scientists and business school professors have noted the perverse incentives that algorithms can create for motivated stakeholders. (13) More recently, computer scientists have developed the area of "Adversarial Machine Learning," which acknowledges the ability of adversaries to cause machine learning systems to make predictable errors and exploit them. (14) Scholars in the field of surveillance studies, including Gary Marx, have discussed various ways in which surveillance could be gamed and avoided. (15) But this paper is, we believe, the first to give sustained attention to gaming in all its forms.

The legal literature has developed very little in the way of theory that can help determine whether a person's ability to exploit the proxies used to judge him is normatively desirable. Nevertheless, existing law has frequently parachuted in, sometimes to support an individual's right to game the system, and sometimes to quash it. Existing law lacks a clear policy vision, or even a coherent language to foster a productive debate. This Article provides the language and vision. It identifies the competing values at stake in the algorithm game so that the law can be thoughtfully designed to promote the ones that are most important to policymakers.

Part I begins by defining gaming and setting out some assumptions. We then describe the gaming moves--tactics that people use to manipulate an algorithm's decisions, and the countertactics that algorithm designers use in response. The subjects of an algorithm can use avoidance, alteration, and obfuscation to exploit or confuse the algorithm. In response, an algorithmic designer can reduce transparency, or he can alter the decision-making model by collecting more data, making the model more complex, rapidly changing the model as it is used, or using less mutable factors.

In Part II, we identify four values that are affected by the algorithm game. Gaming can enhance the autonomy of those who do it by giving them some control over the measure by which they will be judged. Gaming and resisting computer processing can be understood as an exercise of liberty and autonomy, and by the same logic, countermoves used by algorithm designers can interfere with the ability to exercise those rights. But even if it enhances autonomy, gaming will often reduce the accuracy of a proxy since a gameable algorithm can more readily lead to the suboptimal distribution of resources. At the extreme, if gaming causes so much error that the results are arbitrary or capricious, it can cause the algorithm to fail minimum standards of fairness. The algorithm game also has important yet unintuitive distributional consequences. Some populations will be less willing or able to engage in gaming, and therefore both gaming and countermoves can have disparate effects on different subgroups. Finally, gaming can cause system inefficiency since the moves and countermoves take time, effort, and resources. (16)

Part III describes the legal landscape. Existing U.S. law tacitly promotes and demotes these values in various contexts. For example, the law sometimes facilitates gaming and frustrates algorithmic countermoves by requiring that decision-making processes be transparent, by limiting the use of certain immutable and statistically useful proxies, and by restricting the type or amount of information that can be collected about a subject. Labor, credit, and insurance law share many of these rules. These types of laws honor the autonomy value, but the implications for accuracy and distributional effects will depend on context. In other areas, U.S. law dampens gaming by compelling the disclosure of truthful information about a subject or by prohibiting avoidance. These laws, which are common in the areas of tax and criminal investigation, are probably meant to promote accuracy. But because the priority of these competing values is latent, the public policy debates are contentious and imprecise.

This Article hopes to add clarity to the debates about the proper role of algorithmic decisionmaking during these early years of big data innovation and regulation. The project's emphasis is taxonomical; our goal is to describe and organize the stakes involved without setting a priority between incompatible values. Thus, although we illustrate the concepts by applying them to specific examples such as credit scoring and criminal investigation, we make only modest policy recommendations.

1. THE GAME

   This Part lays the necessary groundwork for a deeper discussion of the law and ethics of algorithm design in light of gaming. Because it was too perfect to resist, we borrow a line from Public Enemy to separate our discussion into three Sections: "What is game," "Who got game," and "Where's the game in life?" (17) These Sections will define gaming, comment on who will do it (and why that matters), and provide a nonexhaustive set of tactics that can be used to exploit or confuse an algorithm as well as the countertactics that an algorithm producer may use in response.

   1. What Is Game?

      Gaming is intimately related to the use of proxies and estimators in decision-making processes, so we begin our discussion there. Decisions about scarce resources and penalties can be made in one of only two ways: by pooling potential recipients and distributing the resource using a neutral (or seemingly neutral) factor such as queues or lotteries, or by discriminating between them. The diversity visa lottery, for example, is a pooling system because it awards visas by randomly selecting a set number of visa applicants from a particular country. (18) A discriminating system would not use random selection, equal apportion, or queues. Instead, a discriminating factor could be premised on the individual's merit, need, or skill. (19)

      Pooling schemes are designed to treat all subjects in the pool the same without assessing the merits or costs associated with any person in the pool. Pooling would be unremarkable for homogenous pools, where everyone is more or less interchangeable. But pooling is also frequently applied to heterogeneous populations and reflects implicit policy choices to treat distinguishable people the same. For example, by prohibiting health insurers from considering preexisting health conditions when defining the terms and price of a health plan, the Affordable Care Act requires insurers to ignore factors that would be very relevant to predicted medical costs. (20) By doing so, it converted health insurance from a discrimination scheme to a pooling scheme. Even though we know ex ante that the pool could be separated into higher risk and lower risk subpools, the law forces the low-risk pool to cross-subsidize the high-risk pool in order to more broadly spread the...