Imperfect Tools: A Research Note on Developing, Applying, and Increasing Understanding of Criminal Justice Risk Assessments

• Published date: 01 August 2023
• DOI: http://doi.org/10.1177/08874034231180505
• Author: D. Michael Applegarth,Raven A. Lewis,Rachael M. Rief
• Date: 01 August 2023
• Subject Matter: Research Note

https://doi.org/10.1177/08874034231180505

Criminal Justice Policy Review

2023, Vol. 34(4) 319 –336

© The Author(s) 2023

Article reuse guidelines:

sagepub.com/journals-permissions

DOI: 10.1177/08874034231180505

journals.sagepub.com/home/cjp

Research Note

Imperfect Tools: A Research

Note on Developing,

Applying, and Increasing

Understanding of Criminal

Justice Risk Assessments

D. Michael Applegarth1, Raven A. Lewis2,

and Rachael M. Rief3

Abstract

This article shares considerations for designing, implementing, and understanding

risk assessments used to reduce recidivism of people under community supervision.

These insights are gleaned from 27 data scientists who participated in focus groups

during the National Institute of Justice’s Recidivism Challenge Winners Symposium.

Analyses revealed three primary themes: design considerations, implementation, and

increasing awareness and understanding of risk assessments. Critical aspects of the

design phase include validating the tool, incorporating field data that account for

real-time changes, and adopting strategies to address false positives/negatives and the

model’s complexity. Upon the tool’s development, practitioners are recommended

to devise an implementation plan, balance attention to risk with client-focused needs,

and exercise modest discretion while considering algorithmic results. Recognizing the

value predictive instruments bring to decision-making and identifying their limitations

is needed to increase understanding for all stakeholders. Collaboration and dialogue

between tool developers and practitioners are crucial at every stage.

Keywords

risk assessments, machine learning, design, implementation

1University of California Los Angeles, USA

2Rutgers University, Newark, NJ, USA

3University of Nebraska Omaha, USA

Corresponding Author:

D. Michael Applegarth, Luskin School of Public Affairs, University of California Los Angeles, 3250 Public

Affairs Building, Box 951656, Los Angeles, CA 90095, USA.

Email: applegarth@g.ucla.edu

1180505CJPXXX10.1177/08874034231180505Criminal Justice Policy ReviewApplegarth et al.

research-article2023

320 Criminal Justice Policy Review 34(4)

Introduction

Despite years of research, many important questions remain regarding risk assessment

development, use, and the communication of results to involved parties. Actuarial risk

assessments are imperfect tools meant to inform but not dictate appropriate responses.

An important area of inquiry is how to help criminal justice stakeholders use risk

assessments while avoiding causing harm to those subjected to their results. This arti-

cle seeks to move the discussion forward by drawing on conversations from the

National Institute of Justice’s (NIJ) Recidivism Forecast Challenge Winners

Symposium and examining recent literature in this area.

Actuarial risk assessments are commonly used across the criminal justice contin-

uum, from pretrial detainment to postincarceration community supervision. Within the

criminal justice system, risk assessments predict the likelihood that someone involved

with the system will engage in criminal activity in the future. These tools affect who is

incarcerated, how agencies spend their funding (e.g., service development and deliv-

ery), and how practitioners approach case management (Werth, 2019). While the use

of risk assessments in criminal justice is far from new, the frequency of use, methods

used to develop them, and the diversity of applications have grown rapidly over the

last few decades. Concerns regarding accuracy, ethical and legal implications, and the

potential of increasing racial and gender disparities have accompanied this practice.

Technological advancements have facilitated applying more sophisticated methods

(e.g., machine learning) to produce risk projections, but the aforementioned concerns

remain if they are not specifically addressed. Agencies must determine what process

and information will go into developing these tools, how to implement them best, and

appropriate means to effectively communicate their use to stakeholders and the public.

Users of these tools are also responsible for demonstrating their utility while being

transparent regarding their limitations.

Risk assessments, including ones informed through machine learning, need to be

accurate, transparent in their procedures, and responsive to possible errors. Formal

guidelines to achieve these goals are often lacking (Berk & Hyatt, 2015). This can

create challenges for criminal justice practitioners and administrators who are under

increasing pressure to comply with mandates requiring these tools, even while the

number of risk assessment tools to choose from continues to grow (Desmarais, 2017).

These tools vary in complexity and the rigor with which they have been tested.

Practitioners and administrators must also contend with the lack of a one-size-fits-all

approach or “best” tool and a paucity of knowledge to guide effective implementation

(Desmarais, 2017; Desmarais et al., 2016). Scholars have noted the lack of research

examining the implementation of risk assessments in the criminal justice field, the

absence of regulations on how to use them, and the lack of review and approval by

researchers and legal bodies (Garrett & Monahan, 2019, 2020; Starr, 2015).

Proponents of risk assessments view them as a means to improve the criminal jus-

tice system’s effectiveness, account for potential biases, reduce system involvement

for low-risk individuals, and decrease the number of people incarcerated. Critics warn

risk assessments may legitimize mass incarceration and further disadvantage