Why Black Box Ai Evidence Should Not Be Allowed in Criminal Cases

• Publication year: 2024
• Citation: Vol. 37 No. 1
• Author: Written by Hon. Abraham C. Meltzer

WHY BLACK BOX AI EVIDENCE SHOULD NOT BE ALLOWED IN CRIMINAL CASES

Written by Hon. Abraham C. Meltzer^*

I. BLACK BOX AI

Artificial Intelligence (AI) systems make mistakes. This fact is not sufficiently acknowledged: AI systems will make errors a certain percentage of the time. That is inherent in how machine-learning algorithms are designed. How, then, should lawyers and judges think about AI-generated evidence — which, increasingly, parties are seeking to present in court?

AI-generated evidence might arise from, for example: identifications made by facial recognition software; recidivism risk-prediction algorithms used in determining a defendant's pretrial release status, and in sentencing; medical diagnoses made by AI trained to interpret X-rays and MRI's; and automated underwriting programs used to approve or deny bank loans. Such evidence may well be both valid and relevant. But we cannot assume that all AI evidence is true, because sometimes it is not.

A subset of AI evidence comes from black box AI systems. "Black box" refers to AI systems so complex that their inner workings, and how they generate specific outputs, are beyond human comprehension. With black box systems, it is impossible to recreate, precisely, how the AI arrived at any specific output or conclusion. Put bluntly, black box AI systems "do not explain their predictions in a way that humans can understand." (Rudin, Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead (2019) 1 Nature Machine Intelligence 206.)

This means that when a black box AI system makes a mistake — as it inevitably will — it is impossible to specify how and why the AI got it wrong. Thus, in a criminal case, a criminal defendant (or the prosecution) who is faced with erroneous black box AI evidence has no meaningful way to challenge or to cross-examine the mistaken black box evidence. Again, this is because we cannot understand the path by which the AI generated its output. In other words, black box systems necessarily eliminate meaningful examination and confrontation. Instead, their outputs are like pronouncements from a Delphic Oracle: they may be right or wrong, but their specific genesis remains an impenetrable mystery. This article, therefore, suggests that black box AI evidence should not be allowed in criminal cases.

[Page 20]

II. CALLING A DRAGONFLY A MANHOLE COVER: THE RESNET-50 IMAGE CLASSIFICATION SYSTEM

AI image classification systems provide ready examples of black box AI errors. Such systems look at photo images and say what the picture shows. Because humans are visual creatures, we are able to recognize when the AI mistakenly describes a photo, even if we do not understand how or why the AI made its error.

For example, ResNet-50 is an AI image classifier owned by MathWorks. According to its website, ResNet-50 is a "convolutional neural network that is 50 layers deep," and which was "trained on more than a million images from the ImageNet database." People use ResNet-50 to automatically scan photos and state what they show. And sometimes, it gets it wrong.

Dan Hendrycks, director of the Center for AI Safety, provides two examples of images that ResNet-50 misclassified with 99% confidence (and there are others). The first photo shows a squirrel next to a small fountain. ResNet-50 said this was a "Sea Lion." The second photo shows a dragonfly resting on a woven mat. ResNet-50 said this was a "Manhole Cover." (Hendrycks et al., Natural Adversarial Examples (2020) ArXiv:1907.07174 [Cs.Stat].)

In both instances, it appears that ResNet-50 mistakenly focused on background textures: an animal next to a wet rock surface becomes a "sea lion"; and a cross-hatched texture is a "manhole cover" regardless of anything else. Again, because humans are visually sophisticated, these errors are immediately apparent to us. It is obvious to our eyes that a dragonfly resting on a mat is not a manhole cover.

But what if you did not know the difference between a dragonfly and a manhole cover, and instead asked for detailed, step-by-step information on how ResNet-50 reached its erroneous conclusion that the dragonfly was a manhole cover? How, specifically, did it go wrong? You would be told that it is impossible to retrace how the algorithm made its determination.

This is not rhetorical exaggeration. ResNet-50 "has about 5 • 10⁷ trainable parameters and for classifying one image it needs to execute about 10¹⁰ floating point operations .... It is hardly traceable and not recalculate-able by humans." (Buhrmester et al., Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey (2019) ArXiv:1911.12116v1 [Cs.AI].) Thus, when ResNet-50 erroneously concluded the photo showed a manhole cover instead of a dragonfly, it first examined 50 million parameters and performed 10 billion operations. To recreate the specific path the AI took, and therefore to understand exactly how it made its error, is not humanly possible. That is what is meant by a black box system.

Nor is ResNet-50...