BEING "SEEN" VERSUS "MIS-SEEN": TENSIONS BETWEEN PRIVACY AND FAIRNESS IN COMPUTER VISION.

• Author: Xiang, Alice

TABLE OF CONTENTS I. INTRODUCTION 3 II. DEFINITIONS 10 III. WHY WORRY ABOUT BEING MIS-SEEN? 17 IV. WHY COMPUTER VISION? 19 V. CHALLENGES TO ALGORITHMIC BIAS MITIGATION IN COMPUTER VISION 23 A. Why Is Collecting Images with Informed Consent So Difficult? 25 VI. PRIVACY LAWS 30 VII. HARMS OF BEING SEEN VERSUS MIS-SEEN 34 A. Harms of Being Seen 34 B. Harms of Being Mis-Seen 42 VIII. APPROACHES TO BALANCING PRIVACY AND BIAS MITIGATION 45 A. Carve-Outs from Privacy Law 45 B. Participatory Design 49 C. Trusted Third-Party Data Collection 50 D. Technological Advances 52 E. Right Against Being Mis-Seen 55 IX. CONCLUSION 60 I. INTRODUCTION

Human-centric computer vision ("HCCV") technologies, (1) including facial recognition, are some of the most controversial artificial intelligence ("AI") technologies. HCCV systems are among the few types of AI that have been subject to bans or moratoriums. Many U.S. jurisdictions have restricted the use of facial recognition technologies ("FRT") by government entities, particularly law enforcement. (2) The current version of the proposed EU AI Act categorizes all remote biometric identification ("RBI") systems as high-risk (and thus subject to extensive regulatory requirements), (3) and prohibits the use of real-time RBI by law enforcement in public spaces (with some narrow carveouts). (4) From a privacy perspective, the specter of mass surveillance, particularly by state actors, has led to significant criticism of the growing pervasiveness of FRT (5) and growing pushes for strengthening information privacy laws.

In recent years, there has also been a growing awareness of the issues of bias in HCCV. The highly influential Gender Shades paper showed that many of the major commercial gender classification algorithms performed less effectively on women than men and less well on individuals with deeper skin tones than lighter skin tones. (6) Since then, subsequent studies, including one by the National Institute of Standards and Technology ("NIST"), part of the U.S. Department of Commerce, have shown differences in performance based on skin tone and gender for different HCCV systems. (7) These studies have attributed these biases to a lack of diversity in the datasets used to train these commercial AI systems. (8)

Simultaneously addressing these concerns around privacy and fairness is difficult in practice. To address bias in HCCV, researchers at IBM created the "Diversity in Faces" ("DiF") dataset, (9) which was initially received positively for being far more diverse and balanced than previous face image datasets. (10) DiF, however, soon became embroiled in controversy once journalists highlighted the fact that the dataset consisted of images from Flickr. (11) The Flickr images had Creative Commons licenses, covering the copyright of the images, but the plaintiffs had not consented to having their images used in facial recognition training datasets. (12) In part due to this controversy, IBM announced it would be discontinuing its facial recognition program. (13) Microsoft, Amazon, and Google, which also used the DiF dataset to improve the fairness of their models, were also sued. (14) This example highlights a core tension in developing less biased HCCV: We want AI to recognize us, but we are uncomfortable with the idea of AI having access to data about us. While creating large, diverse human image datasets with informed consent is not impossible (as Section V.A discusses), there are challenges that require further research and regulatory guidance.

This tension is further amplified when the need for sensitive attribute data is considered. For example, to even discern whether a training dataset is diverse, we need a taxonomy of demographic categories, some notion of an ideal distribution across that taxonomy, and labels of these demographic categories. The methods that have emerged to address these necessities are often discomfiting and raise further privacy concerns. In designing DiF, the researchers did not have a variable for race or ethnicity, so they used various computational methods to derive labels for different facial features to indirectly capture differences across race, including metrics for skin color and craniofacial areas. (15) While these features were used in an effort to ensure racial diversity without access to direct data on race, these approaches do not reflect the sociological nature of demographic labels and could be misused, as we have seen in the pseudoscience of physiognomy, which focuses on quantifying physical differences across races. (16)

Other attempts at creating diverse face image datasets, like Fair-Face, (17) approach the challenge by having "Mechanical Turkers" ("MTurkers") (18) guess people's demographic attributes. If at least two MTurkers agree, then the label is considered ground truth; if there is no agreement, the image is discarded. (19) This approach is concerning because it relies on the ability of MTurkers to accurately assess people's demographic attributes, and it discards the images of people who might not fit neatly in the demographic taxonomy. This process could, for example, lead to fewer multiracial, non-binary, or transgender individuals being represented in the data. Designing a taxonomy for demographic classification often relies on stereotypes and can impose and perpetuate existing power structures.

Existing privacy laws address this issue primarily by erring on the side of hiding people's personal data unless there is explicit informed consent. In fact, privacy law and antidiscrimination law are often viewed as symbiotic, (20) under the assumption that preventing companies from collecting personal information helps to prevent discrimination. Evidence of bias in FRT, however, has contradicted this notion. Low representation of minority groups in the datasets used to train such models has led to biased performance, but that has not prevented the use of such systems to deprive minority individuals of their liberty. There have been several cases of Black men in the United States who were wrongfully arrested due to faulty facial recognition matches. (21) In 2019, for example, Nijeer Parks, a Black man, was arrested due to a faulty facial recognition match; he spent ten days in jail and paid close to $5,000 to defend himself before the case was dismissed for lack of evidence. (22)

To address such issues of bias in FRT, the policy response has centered around moratoriums on the usage of FRT by law enforcement and other public agencies. (23) While such moratoriums are reasonable given current problems with such technologies, they are limited to specific jurisdictions, do not apply to other domains for FRT, and do not address bias in other forms of HCCV. (24) The lack of stronger regulatory incentives to address bias in HCCV is concerning given the growing use of such technology. While there is limited data about the broader HCCV market, the FRT market alone is projected to grow from $4.45 billion in 2021 to $12.11 billion in 2028. (25) Even in North America, where FRT has been quite controversial, the market for FRT is expected to double by 2027. (26) While recent moratoriums on FRT for law enforcement suggest a strong discomfort with government use of the technology, the demand for private surveillance camera systems with FRT has continued to grow, (27) as has the use of this technology in everyday life. It is now common for people to open their phones with face verification (28) or to automatically sort photos based on the people in the photos. (29) Moreover, in regions where FRT has not faced as much controversy as in the United States or EU, (30) the technology is increasingly used by government authorities (31) and for everyday verification purposes, such as payment (32) and entering establishments. (33) Outside of FRT, the use of HCCV has become increasingly common, with cameras using eye, face, or body detection for autofocus (34) or for creating artificial bokeh effects (blurry background), (35) robots using human/object detection to navigate real-world spaces, (36) and computer-generated images ("CGI") employing AI to create new fantastical scenes. (37)

Thus, while privacy and bias concerns around FRT have manifested themselves in moratoriums on specific use cases in some jurisdictions, HCCV systems are unlikely to go away anytime soon. The focus of this Article is therefore not on the line-drawing exercise of which HCCV systems should be banned or permitted but rather on categorizing the relevant harms from such technologies and tensions that must be addressed first to judge what would constitute appropriate use cases. Policymakers and AI developers must assess these privacy, bias, and other ethical concerns in contexts where the technology is in use now or might be in use in the future. Unfortunately, as this Article will discuss, addressing both privacy and bias concerns in practice can be quite difficult--not only because each set of concerns entails addressing many sociotechnical challenges, but also because privacy and bias mitigation are often in tension in the algorithmic context, where addressing bias can be enabled by additional access to personal information. The goal of this Article is not to advocate for the increased or decreased usage of HCCV technologies but rather to characterize this tension between privacy and bias mitigation efforts and to propose potential paths forward that respect both.

Part II lays out important definitions used throughout the Article to facilitate more nuanced discourse about HCCV. Part III discusses the importance of considering the harms of being "mis-seen" in a world where HCCV is increasingly pervasive. Part IV explains what makes the HCCV context unique in terms of the privacy and fairness tensions it raises. Part V discusses current challenges to mitigating algorithmic bias in HCCV, focusing particularly on the difficulties with collecting large, diverse datasets...