REGULATING BLACK-BOX MEDICINE.

• Author: Price, W. Nicholson, II

Data drive modern medicine. And our tools to analyze those data are growing ever more powerful. As health data are collected in greater and greater amounts, sophisticated algorithms based on those data can drive medical innovation, improve the process of care, and increase efficiency. Those algorithms, however, vary widely in quality. Some are accurate and powerful, while others may be riddled with errors or based on faulty science. When an opaque algorithm recommends an insulin dose to a diabetic patient, how do we know that dose is correct? Patients, providers, and insurers face substantial difficulties in identifying high-quality algorithms; they lack both expertise and proprietary information. How should we ensure that medical algorithms are safe and effective?

Medical algorithms need regulatory oversight, but that oversight must be appropriately tailored. Unfortunately, the Food and Drug Administration (FDA) has suggested that it will regulate algorithms under its traditional framework, a relatively rigid system that is likely to stifle innovation and to block the development of more flexible, current algorithms.

This Article draws upon ideas from the new governance movement to suggest a different path. FDA should pursue a more adaptive regulatory approach with requirements that developers disclose information underlying their algorithms. Disclosure would allow FDA oversight to be supplemented with evaluation by providers, hospitals, and insurers. This collaborative approach would supplement the agency's review with ongoing real-world feedback from sophisticated market actors. Medical algorithms have tremendous potential, but ensuring that such potential is developed in high-quality ways demands a careful balancing between public and private oversight, and a role for FDA that mediates--but does not dominate--the rapidly developing industry.

TABLE OF CONTENTS INTRODUCTION I. WHAT ARE MEDICAL ALGORITHMS? A. Current Implementations B. Mobile Health C. Black-Box Medicine II. THE NEED FOR REGULATION A. Inherent Complexity and Opacity B. Lack of Procedural Information III. EXISTING APPROACHES A. FDA Regulation of Medical Devices B. Regulatory Authority over Algorithms C. FDA's Historical Approach 1. Algorithms and Software 2. Diagnostic Tests D. Current Approaches 1. Algorithms and Mobile Health 2. Laboratory-Developed Tests IV. CHALLENGES IN EXISTING APPROACHES A. Overregulation 1. Limits on FDA's Regulatory Ability 2. Limits on Innovation B. Underregulation V. Reform A. Command-and-Control Regulation B. Postmarket Surveillance C. Information Forcing and Collaborative Governance____ 1. Providers 2. Hospital Systems 3. Insurers D. Setting the Right Balance CONCLUSION INTRODUCTION

Trauma patients often die from catastrophic hemorrhages. (1) If doctors and nurses are nearby to intervene promptly, death can frequently be prevented. But there are many patients, and it takes an observing expert to know when to intervene. How can we reduce these deaths? An emerging solution relies on computation--sophisticated algorithms that can find patterns in continuously monitored vital signs and call providers before it's too late. (2)

Medicine increasingly turns to algorithms to solve complex health problems. What pattern among a set of thousands of genes predicts which lung tumors will respond to treatment? How should scarce resources like inpatient beds be allocated to optimize patient care among patients with different illnesses and prognoses? What facial features can identify genetic disorders and suggest early interventions? More generally, how can algorithms be used to save lives and reduce suffering by improving medical practice? Sophisticated techniques are being developed to examine vast troves of health data, including genetic sequences, metabolic screens, and electronic health records, in search of answers to these and other questions. Frequently, these algorithms are opaque even to their developers, who may know that something reproducibly works, but not how or why.

This Article builds on my previous work introducing and analyzing this form of "black-box medicine" by canvassing current implementations and then asking: How should algorithmic medicine--that is, the use of such algorithms to guide care--be regulated? If providers rely on a trauma-monitoring algorithm that misses signs a provider could have observed, trauma patients might die from hemorrhage before anyone notices. Algorithms that predict the wrong drug to treat a cancer could waste months and hundreds of thousands of dollars on ineffective treatments. And if algorithms allocate hospital beds ineffectively, patients that need the resources most may not get them.

Patients and providers must trust that algorithms are safe and effective to rely on them, but they lack the experience or knowledge to evaluate algorithms at the point of care, creating a need for systemic regulation. Regulation can help but must walk a fine line: demonstrating safety and efficacy without destroying the flexibility and ongoing innovation that drive algorithmic medicine's development.

FDA is moving to regulate medical algorithms but looks to be moving in the wrong direction. In 2014, FDA proposed that complex laboratory-developed diagnostic tests should be subject to the same preapproval regime currently used to evaluate commercially sold diagnostic kits. (3) Under this regime, diagnostic tests, including algorithmic medicine, would be categorized by risk and then subjected to according regulatory scrutiny; complex algorithms would typically face the heaviest scrutiny and would require clinical testing and preapproval. (4) This regulatory model is poorly suited to drive the accurate, rapid, and safe development of algorithmic medicine. Algorithms can be developed quickly and tailored to the particular needs of health systems and patient groups. Furthermore, at least some algorithms evolve as they incorporate new data and learn to more accurately predict relevant outcomes. Clinical-trial-based preapproval regimes--typically costly, slow, and designed for unchanging products--circumscribe these strengths. Rigidly imposing such a regime on algorithms would substantially slow their development and adoption.

This is not to argue that FDA should have no role in regulating algorithmic medicine. FDA has traditionally taken a command-and-control approach, exercising centralized authority to impose requirements for industry before allowing market access, and some elements of that approach are justified here. (5) In high-risk situations, moderate FDA preapproval requirements may be appropriate. In lower-risk situations, however, more modest registration requirements should suffice. In either case, robust postmarket surveillance will help ensure safety and quality as algorithms are developed and deployed in clinical practice. In general, more light-touch, iterative, adaptive regulation is likely to promote growth and innovation in the field. (6)

But command-and-control is not enough; FDA should consider adopting a collaborative governance approach in this area. Other evaluators, such as insurers, hospital systems, and providers, can and should be involved in helping to evaluate algorithms continually as they are implemented and used in clinical practice. (7) For these evaluators to help provide parallel oversight, they need information, and here FDA can play a central role by mediating the distribution of that information. FDA could require developers to disclose accurate information about their algorithms and could then mediate sharing of that information to insurers, hospitals, or providers. Algorithmic medicine demands a more flexible web of regulatory oversight to bring safe and effective algorithms to patients, providers, and the health system as a whole. In considering solutions, the new governance literature emerges as relevant, offering insights on how collaboration among multiple health care market participants may be useful for optimal oversight of complex medical algorithms.

This Article proceeds in five Parts. Part I describes medical algorithms, considers examples of current implementations, and lays out the overlapping subcategories of mobile-health algorithms and black-box algorithms. Part II asks why medical algorithms need to be regulated at all and, in the process, considers challenges faced by health-care market actors in evaluating medical algorithms in the absence of centralized regulatory authority. Part III begins by analyzing the relatively complex bases of FDA's regulatory authority over software. It then describes FDA's existing and current approaches to regulation of medical algorithms, first canvassing FDA's historical approaches to software and in vitro diagnostic device regulations, and then considering FDA's current approaches to mobile-health software and laboratory-developed diagnostic tests. Part IV critiques aspects of current approaches, noting the dangers of too-rigid as well as too-permissive regulation. Part V presents suggestions for reform. It draws on the new governance literature to suggest possible approaches involving information forcing, collaborative governance, and iterative flexibility, but also suggests that FDA should retain a good deal of centralized command-and-control authority. A few brief thoughts conclude.

1. WHAT ARE MEDICAL ALGORITHMS?

   Medical algorithms are, somewhat tautologically, algorithms used in medicine. But since many decisionmaking processes can be described as algorithms, (8) what I mean here is more specific: computer-based algorithms that help make medical decisions or analyze medical information. Examples include computer-aided diagnostics, such as a classifier that diagnoses melanoma from pictures of skin lesions; (9) a program that evaluates a magnetic resonance image (MRI) for the presence of a tumor; (10) predictive analytics programs that attempt to identify high-risk patients based on a host of...