Machine learning, automated suspicion algorithms, and the Fourth Amendment.

• Author: Rich, Michael L.
• Position: IV. Including ASAs in the Totality-of-the Circumstances Analysis through Conclusion, with footnotes, p. 901-929

4. INCLUDING ASAS IN THE TOTALITY-OF-THE CLRCUMSTANCES ANALYSIS

   If an ASA's output alone cannot satisfy the individualized suspicion requirements of the Fourth Amendment, it must be considered as a part of the totality-of-the-circumstances analysis. As explained above, ASAs are unique data sources in that they aim to assist in the second step of the individualized suspicion analysis by providing information about when one should infer criminality from certain historical facts. (208) Though they can be the source of bad law, when it comes to new technologies and the Fourth Amendment, analogies to existing data sources are the currency of the realm. (209) A good analogy should help courts and police identify the factors that will help them separate reliable ASAs from unreliable ones. This Part will explore three potential analogies. (210) First, ASAs are similar to police profiles, such as 210 those frequently used to identify drug couriers, human traffickers, child abusers, or terrorists, (211) as they utilize historical information to identify traits that are commonly held by criminals with the goal of predicting future criminality. (213) Second, algorithms are akin to informants in that people outside of law enforcement are providing information to police, albeit indirectly through the ASA. (213) Third, algorithms are similar to drug-sniffing dogs in that both resemble "black boxes" that create outputs from known inputs and potentially uncertain processes. (214) This Part addresses each analogy in turn. The first two are ultimately unhelpful for substantive and procedural reasons. The third is more useful, though it is also imperfect. This Part concludes by identifying lingering challenges around incorporating ASAs into the totality-of-the-circumstances analysis.

   1. Algorithms as Police Profiles

      Traditional police profiles are "abstract[s] of characteristics thought typical of persons" engaged in certain criminal activity. (215) These characteristics often include traits or behavior that are legal and innocent when considered individually, but that become suspicious in a given context or when viewed together. (216) For instance, in United States v. Sokolow, the profile of a drug courier on an airplane included innocent facts such as: (1) paying for plane tickets in cash; (2) traveling under a name that does not match the name listed with one's phone number; (3) traveling from a "source city" for drugs; (4) staying in the "source city" for a brief period, particularly when compared to the length of the flight to get there; (5) appearing nervous; and (6) not checking luggage. (217) Profiles like this one formalize the traditional policing process of examining an individual's noncriminal characteristics and actions to determine whether, when taken together, they create a suspicion of criminal conduct. (218) Taken in their best light, profiles consolidate and perpetuate the experience of numerous officers, thus allowing even junior officers to be smart at detecting crime, much in the way that police training instills the experience of veterans in new recruits. (219)

      Though somewhat confusing, the Supreme Court's guidance on profiles suggests that they matter, but only indirectly, to the individualized suspicion analysis. First, the Court has been clear in saying that a profile qua profile does not justify an inference of individualized suspicion. In other words, a set of facts should receive neither any greater nor any lesser weight because those facts are contained in something that a certain law enforcement agency has called a profile of criminal activity. (220) Rather, a court must review de novo a police officer's determination that the facts contained in a police profile support the necessary individualized inference of suspicion. (221) However, to the extent that a profile is a distillation of police experience that the conjunction of certain facts is indicative of criminal conduct, that experience is entitled to some deference. (222)

      Requiring courts to engage in de novo reviews of traditional profiles makes sense. (223) We expect police to examine the facts in an individual case and use typical tools of reason and logic--induction, deduction, and the like--to decide whether they suggest possible criminal activity. Yet police are "engaged in the often competitive enterprise of ferreting out crime." (224) Thus, police have some incentive to push boundaries, which in this case might mean constructing a "chameleon-like" profile that can fit any situation. (225) A judge, on the other hand, should have no skin in the game. (226) Thus, when presented with the same facts that were available to the police at the time, including background on the relevant officer's training and experience, the judge can double-check the officer's logic dispassionately. Ideally this process ensures that the police reasoned logically and reached a defensible conclusion that the facts supported a sufficient inference of criminal suspicion. (227) Similarly, the court can ensure that enough facts exist regarding the specific target of the search or seizure to support a finding of suspicion that is sufficiently individualized. (228)

      On the surface, then, an ASA is like a police profile, in that it identifies likely criminals through the coexistence of multiple innocent facts gleaned from past experience. But substantial differences lie beneath this superficial analogy. First, ASAs derive their conclusions from hard data. In order to "learn" a correlation between certain conduct or characteristics and criminal activity, an ASA must process training data derived from real-life situations. (229) On the other hand, traditional profiles are frequently criticized for the absence of data demonstrating a person meeting the profile is likely engaged in criminal conduct. (230) Similarly, ASAs, like other machine learning algorithms, can examine exponentially more data points about a person or situation than could reasonably be listed in a traditional profile. (231)

      Second, ASAs can identify more complex relationships between observable data and criminal activity than the simple checklist of a traditional profile, which is often applied without clear standards. (232) An ASA that applies even basic machine learning algorithms can not only check for the existence of particular facts, but also assign a weight to each fact depending on the strength of its correlation to criminal activity. (233) Likewise, an ASA can assess the interdependency of variables. For example, an ASA can determine the extent to which the occurrence of criminal activity depends not just on the existence of a single variable, but on the concurrent existence or non-existence of multiple variables. (234) Thinking back to the profile in Sokolow, an ASA might reveal that paying for a ticket in cash and not checking a bag do not, each standing alone, predict drug trafficking with any particular strength, but that the concurrence of the two factors is highly predictive. Consequently, an ASA's capacity for examining a multitude of variables and identifying complex relationships between variables means that the rules generated by an ASA may not be interpretable, even to the ASA's programmers. (235)

      The differences between traditional profiles and ASAs make the Supreme Court's approach to traditional profiles unhelpful and counterproductive when applied to ASAs. To begin with, often no one will be able to explain to a reviewing court how or why the algorithm made its prediction. Thus, the court simply will be unable to double-check the ASA's work. Even when an ASA is programmed to be interpretable, the "logic" of an ASA is not of the sort that a judge can easily double-check. (236) One benefit of an ASA is its capacity to identify correlations within data that are not obvious but are statistically valid. (237) In other words, an ASA could identify a set of behaviors that correlate strongly to criminal conduct, even though the logical connection between the behavior and criminality--that is, why a criminal would engage in that behavior--is unclear to a human observer. The absence of a clear logical connection does not mean that the behavior is a bad predictor of criminality; rather, the logic explaining the correlation may be surprising, or the available dataset may fail to contain the information needed to understand it. (238) Yet a court treating the ASA like a traditional police-created profile, and therefore requiring a logical explanation for why certain facts predict criminality, might incorrectly reject the ASA's "illogical" prediction, notwithstanding the level of confidence the ASA has in the prediction.

      In sum, courts treating ASAs like police profiles may demand that the ASAs be interpretable, thus undermining their effectiveness, (239) and may reject accurate predictions as "illogical." At the same time, the profile analysis would ignore the real sources of ASA inaccuracy, which typically occur in the training drug trafficking with any particular strength, but that the concurrence of the two factors is highly predictive. Consequently, an ASA's capacity for examining a multitude of variables and identifying complex relationships between variables means that the rules generated by an ASA may not be interpretable, even to the ASA's programmers. (235)

      The differences between traditional profiles and ASAs make the Supreme Court's approach to traditional profiles unhelpful and counterproductive when applied to ASAs. To begin with, often no one will be able to explain to a reviewing court how or why the algorithm made its prediction. Thus, the court simply will be unable to double-check the ASA's work. Even when an ASA is programmed to be interpretable, the "logic" of an ASA is not of the sort that a judge can easily double-check. (236) One benefit of an ASA is its capacity to identify correlations within data that are not obvious but are statistically valid. (237) In...