UNDERSTANDING DISCRIMINATION IN THE SCORED SOCIETY
Tal Z. Zarsky(fn*)
INTRODUCTION: KEEPING SCORE OF THE SCORED SOCIETY
In The Scored Society ,(fn1) Professors Danielle Citron and Frank Pasquale introduce a troubling reality: a society in which a small yet powerful group of individuals makes crucial decisions regarding a broad segment of the public. Such decisions are reached on the basis of a scoring scheme the group members secretly develop in advance. This seemingly arbitrary and possibly automated scoring process(fn2) allows powerful entities(fn3) to quickly and seamlessly promote their objectives while treating similar individuals differently. The scoring process also unfolds in a manner which is incomprehensible to those whom it affects. The description above is not of a distant dystopia. Rather, it is of a society we are quickly entering-or perhaps, as a technology commentator recently noted in a similar context-“sleepwalking into.”(fn4)
Scoring has been carried out for years in the realm of consumer credit. Yet the age of big data is leading to the dissemination of these practices to many other contexts. The scoring practices are rendered a feasible option in business, governmental, and social settings due to a multitude of effects.(fn5) Scoring is made possible given the availability of vast quantities of personal information, collection of which is enabled by changes in business models, and the ever-lower price of digital storage facilities. It is further enhanced by advances in data analytics and the ability to effectively aggregate data.
The accelerating use of scoring brings about a variety of problems, which Professors Citron and Pasquale elegantly and eloquently detail. As they demonstrate, the problems related to the expansion of the scored society can be understood on a basic and deeper, analytical level. On the basic level, these processes are problematic given the vast amount of complaints they generate,(fn6) and the fact that they are used, at times, to the detriment of the individual.(fn7) Here, Professors Citron and Pasquale refer to use of credit scores to allocate loans as a case in point.(fn8) On an analytical level, the nature of these concerns could be linked to the way the process relies on biased and inaccurate datasets,(fn9) its inherent opacity,(fn10) or the lack of sufficient human review.(fn11)
Yet another important concern surfacing in discussions and analyses of the scored society and the troubles big data analytics bring about is that of discrimination . In this Article, I begin to examine the relation between discrimination-based arguments and the emergence of the scored society.(fn12) I also briefly examine what solutions could be applied to limit such concerns. This inquiry comes at a crucial time. The discrimination-based argument is already being examined and invoked when addressing the realm of scoring and big data by both academics and policymakers.(fn13) Specifically, a recent Request for Public Comment set out by the Department of Commerce (via the National Telecommunications and Information Administration) sought advice regarding the issues arising that involve discrimination and Big Data.(fn14) Yet there is much analytical work to be done before such arguments could be properly articulated in this ever-changing context. Perhaps most importantly, scholarship must connect the policy arguments and popular discontent noted in the context of scoring and big data analytics, with sound theoretical arguments voiced elsewhere while discussing discrimination.(fn15) Such work is critical, as it distinguishes between valid concerns and those that merely result from a Neo-Luddite sentiment or even manipulation by various interest holders.(fn16)
This Article draws out several antidiscrimination paradigms which on their face pertain to the dynamics discussed in The Scored Society, and big data in general. Such analysis allows for recognizing which discrimination-based concerns are especially acute in the scored society, as well as setting forth initial proposed responses for mitigating them, when possible. The Article proceeds as follows: after a brief Introduction mapping the confines of the debate and summarizing Professor Citron and Pasquale's contributions, the Article moves to Part I, where it generally addresses the notion of “discrimination” and its relevance to the issue at hand. Part II-the heart of this Article- identifies the discrimination-based concerns which relate to the mistreatment of “protected groups.”(fn17) There, the Article demonstrates the possible concerns while relying on race as a key example of a “protected group” and distinguishing between explicit discrimination, implicit discrimination, and instances of disparate impact. In Part III, the Article takes a brief look at selected discrimination concerns which go beyond protected groups. It generally finds these latter problems relatively easy to resolve. Finally, in the Conclusion, the Article argues that even though the scoring process is seemingly ridden with discrimination-based concerns, it certainly should not be categorically abandoned, as it might even promote antidiscrimination objectives when carried out properly.
The Scored Society provides an excellent point to begin an important discussion regarding the novel forms of discrimination the analysis of big data brings about. Indeed, Professors Citron and Pasquale make the crucial link between the emergence of scoring practices and a variety of troubling concerns-including discrimination. (fn18) Yet their article does not settle for merely pointing out problematic aspects. Professors Citron and Pasquale go even farther and offer innovative solutions. (fn19) They promote regulatory oversight through the introduction of licensing schemes, (fn20) audit logs (fn21) and mandatory test-runs of the scoring methods, using various data sets. (fn22) They also recommend the use of interactive modeling interfaces which allow those subjected to the scoring process to gain a greater understanding of its inner workings. (fn23) This objective could be achieved by allowing users to challenge the system with various hypothetical scenarios and examine the responses they receive. (fn24)
The various harms, concerns, and even solutions discussed throughout The Scored Society can be easily grounded in several existing moral and legal concepts. The most obvious choice is integrating the analysis of scoring into a broader discussion regarding privacy and data protection (or the lack thereof) in the digital age. Indeed, many of The Scored Society's concerns can be articulated through the terminology of the “Fair Information Practices”-or FIPs. FIPs are broad regulatory concepts set forth by several influential entities, most notably, the OECD.F (fn25) Both the European Union and the United States incorporated FIPs, to some extent, into their privacy laws.(fn26) FIPs spell out what obligations those collecting and using personal information have towards their users and “data subjects.” For instance, an individual's inability to view personal data collected about her while creating the score is related to the principle of “Individual Participation.”(fn27) The fear that said scoring-related information is inaccurate or incomplete is linked to the principle of “Data Quality.”(fn28) Finally, the overall theme which calls for greater disclosure in the scoring process is closely related to the principles of “Transparency” and “Openness.”(fn29)
Furthermore, the innovative solutions Professors Citron and Pasquale provide fit nicely with other regulatory proposals set forth in the privacy and data protection realm to achieve similar objectives. Most notable in this context is the European Union's Data Protection Regulation Proposal. This proposed legislation introduces novel and even controversial measures to enhance FIPs, such as “the right to be forgotten,” mandatory interoperability, a guarantee for human evaluation and hefty fines, among others.(fn30) To some extent, integrating these proposed solutions, especially those calling for enhanced disclosure and enforcement, into existing law will allow for overcoming the various concerns regarding inaccuracy and opacity which the scored society brings about.
Yet beyond the notion of privacy and data protection, the discussion at hand invokes an additional set of concerns, which call for a serious discussion regarding discrimination .(fn31) Indeed, Professors Citron and Pasquale note that the scoring process, especially when used to formulate credit scores, can prove discriminatory, have a disparate impact,(fn32) and even allow for “systemizing” discrimination.(fn33) They also note that scoring systems are stigmatizing(fn34)-a term often used in conjunction with discrimination.(fn35) These negative outcomes can result from deliberate or accidental actions of the scoring entities.(fn36) Thus, a solid link between the concerns stemming from the scored society and the broader concept of discrimination is apparent. However, policymakers and scholars must engage in additional work on the theoretical and policy level to further incorporate insights from the antidiscrimination discourse into the emerging realm of “scoring.” It is also important to recognize the limits of antidiscrimination theory, and the point at which referring to this concept will not prove helpful.
Before proceeding, note an important...
