LOOSE-LIPPED LARGE LANGUAGE MODELS SPILL YOUR SECRETS: THE PRIVACY IMPLICATIONS OF LARGE LANGUAGE MODELS.

• Author: Winograd, Amy

TABLE OF CONTENTS I. INTRODUCTION 616 II. PRIVACY VULNERABILITIES OF LARGE LANGUAGE MODELS 622 A. Privacy Attacks 623 B. Technical Solutions to Data Leakage 628 1. Data Sanitization 629 2. Differential Privacy 630 III. THE CHALLENGE OF INFORMED CONSENT 631 A. The Notice-and-Choice Paradigm 632 1. Privacy Interests in Public Training Data Are Underprotected 635 2. The Uncertainty of Downstream Uses Complicates Adequate Notice and Undermines Consent 640 3. The Permanence of Data Imprints Undermines Core Privacy Rights 641 IV: RECOMMENDATIONS 645 A. Clarify Existing Legal Obligations 646 B. Prioritize Publicly-Intended Training Data 649 C. Require Opt-Out Periods for Sensitive Nonpublic Personal Information 651 D. Improve Transparency: Training Datasets, Privacy- Preserving Mechanisms, and Data Collection Practices 652 E. Institute Oversight Bodies and Mandatory Audits 654 V. CONCLUSION 655 I. INTRODUCTION

On November 30, 2022, OpenAI--a leading artificial intelligence ("AI") research and deployment company--unveiled ChatGPT, an AI model designed to specialize in human-like, long-form conversation. (1) Within five days, more than one million people signed up to interact with the cutting-edge chatbot. (2) Just two months later, ChatGPT reached 100 million monthly active users, securing its position as the fastest-growing consumer application in history. (3) The world reacted with astonishment at ChatGPT's ability to produce cogent, creative, and occasionally magical responses: a seeming "mix of software and sorcery" (4) that some proclaim will fundamentally upend society and others dismiss as a high-tech parlor trick. (5)

Whether machine sentience looms in the near future (6) or recent advancements represent little more than illusions of meaning or vacant stochastic parroting, (7) one thing is for certain: the spellbinding digital magic conjured by ChatGPT reflects rapid progress in artificial intelligence, specifically large language models which have increasingly dominated the field of AI. (8) A large language model ("LLM") is a type of artificial neural network, (9) trained on an enormous amount of text data, which determines the probability of a word sequence. (10) In other words, given an input, LLMs essentially predict what word comes next. This deceptively simple yet powerful ability can be applied to a wide range of tasks such as text generation, question resolution, document summarization, sentence completion, protein sequence generation, language translation, and more. (11) For instance, OpenAI's ChatGPT can engage in open-ended conversations, write original prose and poetry, play complex games, generate computer code, design websites, and solve mathematical word problems. (12) In addition to processing image inputs, the recently released GPT-4 demonstrates unprecedented problem-solving and reasoning ability, scoring in the 90th percentile on the Uniform Bar Exam and the 88th percentile for the LSAT. (13) Given LLMs' broad capabilities, the potential applications of LLMs are diverse and expansive. (14)

LLMs' impressive baseline proficiency can be further enhanced through fine-tuning. After initial training on a large corpus of text data, LLMs can be fine-tuned, using far less training data, to improve performance on specific tasks. (15) For instance, ChatGPT, which has been optimized for dialogue, is a fine-tuned variant of OpenAI's GPT-3.5 family of large language models. (16) Capitalizing on this feature, a company can develop a general-purpose pretrained LLM and subsequently make the model commercially available via an API, enabling other organizations to fine-tune the model using custom data to optimize for their specific needs. (17) This practice has spurred a burgeoning industry. (18)

State-of-the art LLMs exhibit surprising versatility, even without fine-tuning. In a phenomenon known as "zero-shot learning," an LLM performs tasks for which it was never explicitly trained. (19) In "few-shot learning," the model's performance markedly improves with only a few example prompts. (20) Remarkably, a technique known as "chain-of-thought prompting"--which elicits a sequential thought process through structured reasoning examples or phrases like "let's think step by step"--significantly enhances few-shot and zero-shot performance on tasks that demand complex reasoning. (21) Unlike fine-tuning, zero-shot and few-shot learning do not require gradient updates (i.e., adjustments to the model's parameters) through additional training. (22)

The progress in this space has been astonishingly rapid. (24) A decade ago, in 2012, it was a groundbreaking feat when Google's neural network successfully identified cats in unlabeled images. (25) Five years later, Deepmind's AlphaGo model achieved super-human performance in Go, one of the world's most complex board games. (26) Today, OpenAI's DALL*E 2 can generate striking, original images from simple text prompts; (27) Google's AudioLM produces realistic speech and music continuations from brief audio prompts; (28) and Meta's Cicero ranks in the top ten percent of human players at Diplomacy, a conversational alliance-building strategy game requiring complex negotiation with multiple human players. (29) Since Google introduced a new neural network architecture in 2017, (30) LLMs have quickly become more capable and general purpose, trending toward single models that can complete thousands of different (and sometimes unpredictable) tasks. (31) In yet another breakthrough, the newest generation of multi-modal LLMs transcend text-based constraints by integrating a range of modalities, including audio, video, and images. (32) Increasingly sophisticated AI brings the promise of enhanced efficiency, elevated problem-solving, rapid scientific breakthroughs, improved quality of life, and other transformative social benefits. (33)

This tremendous technological progress is accompanied by the risk of wide-ranging social harms. LLMs are notoriously prone to learning the biases entrenched in their training data, and this toxicity appears to increase as they scale. (34) Deviant actors who develop toxic "mischief models" only exacerbate this issue. (35) While LLMs produce fluent and often impressive responses, they also have an alarming propensity for fabrication (dubbed "hallucination" by AI experts). (36) In other words, they are excellent at generating authoritative-sounding lies, a feature that can be easily exploited by bad actors who wish to spread disinformation. (37) And as conversations with LLMs are increasingly indistinguishable from those with humans, LLMs could be deployed to manipulate, deceive, and exploit vulnerable people. (38) With the assistance of LLMs, scammers might supercharge traditional schemes that already cost Americans billions per year. (39) Combined with access to voice simulation and deepfake technology, malicious actors have a disturbingly expansive arsenal of sophisticated tools for manipulation and harassment. (40) In addition to these short-term dangers, AI may eventually destabilize the economy, (41) and some experts worry that the alignment problem--the challenge of aligning superintelligent AI systems with human values and goals--poses an existential risk. (42)

This Note focuses on the novel, wide-ranging privacy harms posed by LLMs. (43) Part II outlines the privacy vulnerabilities presented by attacks that identify and extract sensitive information that an LLM has memorized from its training data, and briefly describes two privacy-preserving technical solutions--data sanitization and differential privacy--which aim to mitigate this issue. Although valuable tools, these technical solutions fail to adequately protect against the wide range of privacy harms posed by LLMs, which reach beyond data leakage. Remedying these harms is complicated by the enormous breadth of training data, the inscrutability of the models to both their architects and the data subjects, and the permanence of data imprints in the model.

Part III explores the deficiencies of the notice-and-choice paradigm that dominates privacy law, and examines the characteristics of LLMs that highlight these defects. For example, although publicly sourced datasets used to train LLMs contain personal information, U.S. law largely disregards the privacy interests in this data because it is exposed to the public. Additionally, even when a company provides notice to an individual whose data it has collected, the uncertainty of downstream applications of LLMs complicates adequate disclosure. Moreover, the permanence of data imprints embedded in LLMs compromises the fulfillment of core privacy rights. Most notably, those who wish to withdraw consent and remove their contributed data imprints from LLMs are left without recourse, due to the challenges of machine unlearning. These factors ultimately muddle an individual's privacy risk calculus and impede meaningful consent.

Part IV outlines preliminary recommendations for regulators, emphasizing that privacy protections must extend beyond individual choice. Regulators should clarify existing legal obligations, maximize transparency to encourage and clarify responsible development practices, embed privacy into the design and implementation of LLMs, and establish oversight and auditing frameworks that quantify privacy risk and curb abuse. Ultimately, an interdisciplinary effort between the legal and technical communities is necessary to address these issues.

2. PRIVACY VULNERABILITIES OF LARGE LANGUAGE MODELS

   The following Part describes the type of data used to train LLMs and explores why private information contained in the model's training data might be vulnerable to exposure. LLMs are prone to memorizing information in their training data, and adversaries can attack LLMs to exploit this vulnerability and elicit sensitive memorized information. This vulnerability is likely to intensify as LLMs continue to scale. This Part then explores...