Preliminary Examination of the Impact of Program Factors on Summary Effect Sizes

• DOI: 10.1177/0306624X20964064
• Author: Faye S. Taxman,Avinash Bhati,Liana R. Taylor
• Date: 01 November 2021
• Published date: 01 November 2021
• Subject Matter: Articles

https://doi.org/10.1177/0306624X20964064

International Journal of

Offender Therapy and

Comparative Criminology

2021, Vol. 65(15) 1629 –1652

© The Author(s) 2020

Article reuse guidelines:

sagepub.com/journals-permissions

DOI: 10.1177/0306624X20964064

journals.sagepub.com/home/ijo

Article

Preliminary Examination

of the Impact of Program

Factors on Summary Effect

Sizes

Liana R. Taylor1, Avinash Bhati2,

and Faye S. Taxman3

Abstract

The Washington State Institute for Public Policy (WSIPP) uses meta-analyses to

help program administrators identify effective programs that reduce recidivism. The

results are displayed as summary effect sizes. Yet, many programs are grouped within

a category (such as Intensive Supervision or Correctional Education), even though

the features of the programs might suggest they may be very different. The following

research question was examined: What program features are related to the effect size

in the WSIPP program category? Researchers at ACE! at George Mason University

reviewed the studies analyzed by WSIPP and their effect sizes. The meta-regression

global models showed recidivism decreased with certain program features, while other

program features actually increased recidivism. A multivariate meta-regression showed

substantial variation across Cognitive-Behavioral Therapy programs. These preliminary

findings suggest the need to further research how differing program features contribute

to client-level outcomes, and develop a scheme to better classify programs.

Keywords

correctional programming, meta-analyses, cognitive-behavioral therapy, recidivism,

effect sizes

1Texas A&M University – Central Texas, Killeen, TX, USA

2Maxarth LLC, North Potomac, MD, USA

3George Mason University, Fairfax, VA, USA

Corresponding Authors:

Liana R. Taylor, Department of Social Sciences, Texas A&M University – Central Texas, 1001 Leadership

Place, Killeen, TX 76549, USA.

Email: liana.taylor@tamuct.edu

Faye S. Taxman, Department of Criminology, Law, & Society, George Mason University, 4087 University

Drive, Ste 4100, Fairfax, Virginia 22030, USA.

Email: ftaxman@gmu.edu

964064IJOXXX10.1177/0306624X20964064International Journal of Offender Therapy and Comparative CriminologyTaylor et al.

research-article2020

1630 International Journal of Offender Therapy and Comparative Criminology 65(15)

Introduction

Meta-analysis is a statistical method used to objectively summarize research findings

by consolidating a large body of evaluations and studies that may use diverse methods,

populations, and settings. It has been beneficial in changing the view of correctional

programming as largely useless and guiding the corrections field to determining “what

works” (Smith et al., 2009). Meta-analyses have been conducted to show the effective-

ness of various programs and treatment with different populations such as relapse pre-

vention (Dowden et al., 2003), sexual offender treatment (Lösel & Schmucker, 2005),

Cognitive-Behavioral Therapy (Landenberger & Lipsey, 2005), and juvenile correc-

tions (Wong et al., 2016), to name a few. The Washington State Institute for Public

Policy (WSIPP) uses meta-analyses and internally-developed methods to assist state

and local program administrators with identifying effective programs that reduce recid-

ivism in their local jurisdictions. A meta-analysis is conducted on all the obtained stud-

ies in a program category and the resultant summary effect size indicates the estimated

impact on recidivism (Wanner, 2018). Advances in meta-analyses include the use of

meta-regression which goes beyond the summary effect sizes to identify how different

features of the analyzed studies can influence those effect sizes (Borenstein et al.,

2009). The purpose of the current study is to report a subset of the preliminary findings

from a feasibility study conducted at the Center for Advancing Correctional Excellence!

(ACE!) at George Mason University (GMU) that examined which program features

affect the summary effect sizes of WSIPP program categories.

Despite the advantages of meta-analyses, there are two primary caveats that must

be considered when using these findings to identify effective program categories:

First, meta-analyses consolidate studies to reduce bias in the findings that may be

introduced due to methodological differences. The studies become the data points and

the researcher is able to look at the findings across studies to assess the overall effect

on the outcome of interest; for correctional programs the outcome is usually recidi-

vism. However, due to the rigor of these studies, the outcomes may not be easily rep-

licated within local jurisdictions (Goggin & Gendreau, 2006). Therefore, the results of

the meta-analyses could still be misleading due to variations in how the programs are

implemented in “real-world conditions”, including the features of a program.

Second, the resulting summary effect size is not sufficient to fully understand the

range of program impacts. It is usually reported without consideration of the variations

that could exist among the programs and studies in a given meta-analysis. For exam-

ple, Palmer (1995) recommended that research should examine “programs as compos-

ites. . .” (p. 122) and programs should not all be considered “unimodal” (p. 102).

Palmer (1995) notes even among generic categories used to compile “similar” pro-

grams, there are variations in successful and unsuccessful programs for reducing

recidivism. Possible sources of variation within the features of programs include type

of staff, characteristics of the target population, program mechanisms, and implemen-

tation fidelity. Borenstein et al. (2009) stated this heterogeneity in meta-analyses

should be reported and explored to determine how it impacts the findings. Solely rely-

ing on the summary effect size could give the impression that programs in the same

general category will produce similar recidivism reductions, regardless of whether the