Promoting Innovation While Preventing Discrimination: Policy Goals for the Scored Society

• Publication year: 2021

PROMOTING INNOVATION WHILE PREVENTING DISCRIMINATION: POLICY GOALS FOR THE SCORED SOCIETY

Frank Pasquale(fn*) and Danielle Keats Citron(fn**)

Professor Zarsky's response(fn1) is an erudite and thoughtful analysis of the discrimination concerns raised by our article, The Scored Society.(fn2) We particularly appreciate his connection of themes in our article with literature on discrimination law. This historical awareness and theoretical sophistication demonstrates the deep continuity between our concerns and those of other legal scholars.

Professor Zarsky has led us to realize that there are in fact several normative theories of jurisprudence supporting our critique of the scored society, which complement the social theory and political economy presented in our article. In this response, we clarify our antidiscrimination argument while showing that is only one of many bases for the critique of scoring practices. The concerns raised by Big Data may exceed the capacity of extant legal doctrines. Addressing the potential injustice may require the hard work of legal reform.

Before responding, though, we should acknowledge Professor Zarsky's contributions to the field, and explain how we believe our work advances inquiry along some of the trails he has blazed with his insightful analyses of data mining, privacy, and information law generally.

Professor Zarsky has done a great deal to explore the legal problems raised by the proliferation of networked identities and selves. We are all aware of the freedoms and dangers posed by pseudonyms, anonymous handles, and multiple roles online.(fn3) Viewed in light of Professor Zarsky's proposals about traceable anonymity to ensure accountability in a digital age, one can think of the scores explored in our article as deeply connected to those concerns: quantified identities imposed on individuals, often without their consultation, consent, or even awareness.

An attention economy has gradually developed on the internet, as companies collect information about the habits and demographics of those who visit their websites. Unlike the old broadcast model of simply exchanging programming content for (an easily avoided) obligation to watch commercials, the new online data collectors enjoy far greater powers to monitor the behavior and actions of users and to influence their online experience and reputation. The skillful use of that data is a large part of the success of online behemoths and is increasingly driving decisionmaking at companies ranging from banks to retailers.

But data collection and analysis raises serious concerns. Data collection practices range from the careful to the careless. The tradeoff between checking accuracy and speedy production can easily tilt toward the latter as competition increases.(fn4) Once primarily directed at marketing, data collection practices now figure into employment and credit opportunities.(fn5) The companies' digital stockpiles would delight a new Stasi or J. Edgar Hoover. Assurances to customers that data are anonymized mean little without audits-which are nonexistent for most firms, and rare and often cursory even when required. Should a company that observes a customer looking at a $10,000 ring on one site use that information to allow others to systematically raise prices for the customer on the assumption that he or she is wealthy?

What self-help measures can (and should) consumers take as they are observed online? Can contract and tort law address all of the potential violations of privacy that occur due to interferences with our settled expectations about how our data is used? Can law address the harms associated with data leaks due to insecure systems?(fn6) The collection of vast reservoirs of data raises difficult questions across the range of public and private, and statutory and common, law.

Professor Zarsky's essay Privacy and Data Collection in Virtual Worlds was an early effort to tackle these problems.(fn7) The term “virtual world” is usually associated with game-like activity in venues like Second Life or World of Warcraft. But the metaphor of the virtual world helps enrich our understanding of the degrees of freedom available to the legal system as it addresses questions like privacy and identity online. Professor Zarsky highlighted the vast extent of personal information that can be collected in virtual worlds. He discussed the heightened level of surveillance prevalent in the online environment. In our era of pervasive surveillance, tracking, and the internet of things, that world is our world, and its most gifted explicators have given us important insights into how pervasive data-gathering and processing should be governed.(fn8)

While many laissez-faire commenters have claimed that users can “take or leave” participation in virtual worlds if they find such surveillance oppressive, Professor Zarsky early on realized how many important activities are migrating to these virtual spaces and how individual user decisions are constrained. For example, someone opting out of Second Life, and creating their own “Third Life,” might well find that none of his friends follow him to his own virtual world, and that the creators of Second Life sue for copyright and trademark infringement to the extent the newer virtual world mimics their own. Those trying to defect to the alternative social networks Google+ and Ello have experienced this coordination problem directly: maybe they and some enterprising friends establish a presence there, only to find that ninety percent of the rest of their social network is too busy or uninterested to join them. Technical and practical challenges to creating a user experience sufficiently similar to attract users of the Second Life interface, while sufficiently different to avoid infringing intellectual property, may well be insuperable.

To deal with these dynamics, Professor Zarsky has recognized that TOSs and EULAs are, by and large, contracts of adhesion, and that only a public law solution can address the imbalances inevitable in these contracts, which in most cases amount to little more than private legislation of terms by the dominant party. We agree. Regulatory initiatives are essential to guard consumers who can hardly anticipate all potential uses of data on their own.

Professor Zarsky's attention to threats to the sensitive online ecology of social software continued in his work Law and Online Social Networks.(fn9) He analyzed the positive and negative social effects of interactions in these environments. Professor Zarsky is one of the first scholars to notice the importance of gaming in social networks-efforts to manipulate or fake the bonds of trust that create social capital and cooperation. We believe that, unregulated, scoring could lead to exactly the same issues: people pervasively manipulating their own...