Risk Assessment Overrides: Shuffling the Risk Deck Without Any Improvements in Prediction

• Author: Christopher T. Lowenkamp,Thomas H. Cohen,Kristin Bechtel,Anthony W. Flores
• DOI: 10.1177/0093854820953449
• Published date: 01 December 2020
• Date: 01 December 2020
• Subject Matter: Articles

953449CJBXXX10.1177/0093854820953449Criminal Justice and Behaviorcohen et al. / Risk Assessment Overrides: shuffling the Risk deck
research-article2020
Risk Assessment OveRRides
shuffling the Risk deck Without Any improvements
in Prediction
THOMAS H. COHEN
CHRISTOPHER T. LOWENKAMP
Administrative Office of the U.S. Courts
KRISTIN BECHTEL
Arnold Ventures
ANTHONY W. FLORES
California State University, Bakersfield
In the federal supervision system, officers have discretion to depart from the risk designations provided by the Post
Conviction Risk Assessment (PCRA) instrument. This component of the risk classification process is referred to as the super-
vision override. While the rationale for allowing overrides is that actuarial scores cannot always capture an individual’s
unique characteristics, there is relatively limited literature on the actual effects of overrides on an actuarial tool’s predictive
efficacies. This study examines overrides in the federal system by assessing the extent to which risk levels are adjusted
through overrides as well as the impact of overrides on the PCRA’s risk prediction effectiveness. Findings show that nearly
all overrides lead to an upward risk reclassification, that overrides tend to place substantial numbers of persons under federal
supervision (especially those convicted of sex offenses) into the highest supervision categories, and that overrides result in a
deterioration of the PCRA’s risk prediction capacities.
Keywords: supervision overrides; risk prediction; risk assessment tools; professional discretion
intROductiOn
Actuarial risk instruments have become a crucial element of the evidence-based com-
munity corrections paradigm (Andrews & Bonta, 2010a, 2010b; Andrews et al., 1990).
Research has consistently shown that these tools provide a superior method of assessing the
risk of recidivism for persons under supervision (hereinafter referred to as supervisees)
AutHORs’ nOte: Please note that a longer version of this article is available on the ResearchGate website.
This article benefited from the careful editing of Ellen Fielding and Suzelle Fiedler. Correspondence concern-
ing this article should be addressed to Thomas H. Cohen, Probation and Pretrial Services Office, Administrative
Office of the U.S. Courts, One Columbus Circle, NE, Washington, DC 20544; e-mail: thomas_cohen@ao.
uscourts.gov.
CRIMINAL JUSTICE AND BEHAVIOR, 2020, Vol. 47, No. 12, December 2020, 1609 –1629.
DOI: 10.1177/0093854820953449
ht
ps:/
doi.
rg
Article reuse guidelines: sagepub.com/journals-permissions
© 2020 International Association for Correctional and Forensic Psychology
1609

---

1610 CRIMINAL JUSTICE AND BEHAVIOR
compared with unstructured clinical approaches (Ægisdóttir et al., 2006; Grove et al., 2000;
Schmidt et al., 2016). Moreover, actuarial tools are especially powerful because they pro-
vide community corrections professionals with a mechanism for making decisions about a
supervisee’s risk of recidivism that is both uniform and based on standardized sets of crite-
ria. In the absence of these standardized criteria, professionals are often left with their sub-
jective beliefs, notions, and biases to assess a supervisee’s likelihood of committing new
crimes, with the ultimate result being that professional judgment produces poorer risk pre-
dictions compared with actuarial instruments (Holsinger et al., 2001; McCafferty, 2015).
Although actuarial risk instruments outperform clinical prediction, most of these tools
provide officers with an escape hatch that essentially allows them to disregard the instru-
ment’s designated risk classification and assign their own supervision level (McCafferty,
2015). For example, a supervisee classified as low risk by an actuarial instrument could be
overridden to a higher supervision level should a corrections professional decide that, in
his or her judgment, the actuarial risk designation underrepresents a supervisee’s risk to
reoffend (Cohen, Pendergast, & VanBenschoten, 2016). This aspect of the risk classifica-
tion process is typically referred to as professional discretion or the supervision override
and is one of the key components of the risk classification process (Andrews et al., 1990;
McCafferty, 2015).
Those who favor using overrides assert that they are a necessary component when
assigning supervision levels, because actuarial tools cannot always capture the unique
characteristics of individuals that can be gleaned through an extensive investigation pro-
cess (Cohen, Pendergast, & VanBenschoten, 2016). Hence, supervision overrides allow
officers to depart from the actuarial risk classification when the totality of a supervisee’s
characteristics suggests that the person’s adjusted supervision levels should diverge from
the initial risk classification. By allowing officers to take into account the totality of a
supervisee’s characteristics, proponents of overrides assert that these mechanisms could
potentially increase an actuarial tool’s predictive accuracy (McCafferty, 2015). A growing
but still relatively limited body of research, however, suggests that overrides could have
the opposite result, diminishing a risk tool’s predictive efficacy (Cohen, Pendergast, &
VanBenschoten, 2016; McCafferty, 2015; Schmidt et al., 2016; Wormith et al., 2012).
This study attempts to augment the empirical literature on supervision overrides by ana-
lyzing this issue for the U.S. federal supervision system. Specifically, this study examines
the use of supervision overrides for 259,571 persons on federal post-conviction supervision
who were assessed with the Post Conviction Risk Assessment (PCRA). The PCRA is an
actuarial instrument used by federal probation officers to assess the risk of recidivism for
supervisees placed on federal terms of supervised release (i.e., TSR) or probation. By exam-
ining such a large population of supervised persons with risk assessments, this research can
further enhance our understanding of overrides and their effect on risk prediction.
LiteRAtuRe RevieW
Since the establishment of the risk-needs-responsivity model, risk assessments have
become a cornerstone for practitioners to target the risk and needs of their clients in an
effort to improve the likelihood of success (Andrews & Bonta, 2010a, 2010b; Andrews
et al., 1990). These tools are now infused into supervision practices expanding from juve-
nile through adult populations, and the process for implementing policies to inform

---

Cohen et al. / RISK ASSESSMENT OVERRIDES: SHUFFLING THE RISK DECK 1611
decision-making based on the risk assessment output presents a number of valid ques-
tions about whether or not these policies (in this case, overrides) impact their validity.
While overrides are not directly synonymous with using exclusively professional or clin-
ical judgment (presumably, practitioners are still following the risk assessment information
to inform decision-making), it is still important to take stock of over 60 years’ worth of
research comparing clinical judgment to actuarial risk assessment. Overall, there is substan-
tial and robust evidence to suggest that the use of unstructured clinical or professional judg-
ment consistently results in weaker predictive validity in comparison to actuarial tools (see
Ægisdóttir et al., 2006; Grove et al., 2000; Grove & Meehl, 1996; Harris, 2006). Despite
this evidence, the challenge of adhering to or even trusting risk assessments persists. Meehl
(1957), along with Dawes et al. (1989), described in great detail a number of the sources of
resistance that commonly perpetuate the determination to ignore the actuarial assessment
(Dawes et al., 1989, p. 11). These suspicious and perhaps critical viewpoints of risk assess-
ment certainly persist among community corrections professionals (Clem, 2003; Schneider
et al., 1996).
Some may suggest that this level of evidence should prompt policymakers to adopt solely
actuarial processes (see Borum et al., 1993; Grove & Meehl, 1996; Quinsey et al., 1998, for
more discussion). However, others argue that perhaps there is a need to strike a balance and
determine if there is empirical value to integrating clinical judgment alongside actuarial
decision-making (Gottfredson & Moriarty, 2006). One way community corrections profes-
sionals have attempted to weave discretion into actuarial decision-making is to allow offi-
cers authority to override or underride the results of the risk assessment when determining
actual supervision levels.
While overrides have become a central facet of many risk assessment regimes, there
is little empirical basis to support them (Cohen, Pendergast, & VanBenschoten, 2016;
Wormith et al., 2012). Similar to the research consistently demonstrating that statistical
prediction outperforms clinical judgment, the information on overrides, which could be
characterized as a form of professional judgment, is beginning to reveal a similar trend—
namely, the extensive use of overrides negatively impacts the predictive performance of
risk assessments.
One common use of an override on supervision is based on charge type. Specifically,
supervision clients who have convictions for sex crimes may receive an override that
increases the intensity of supervision conditions. One such study aimed to test professional
overrides and compare the predictive validity of the Level of Service/Case Management
Inventory (LS/CMI) on a Canadian supervision sample comprising those convicted of sex
crimes (N = 1,905)...