Propensity Score Analysis

• Published date: 21 May 2012
• DOI: https://doi.org/10.1108/S1474-7863(2012)0000013012
• Date: 21 May 2012
• Pages: 173-198
• Author: Sarah Brown

PROPENSITY SCORE ANALYSIS

Sarah Brown

ABSTRACT

In the evaluation of most interventions in criminal justice settings,

evaluators have no control over assignment to treatment and control/

comparison conditions, which means that the treated and comparison

groups may have differences that lead to biased conclusions regarding

treatment effectiveness. Propensity score analysis can be used to

balance the differences in the groups, which can be used in a number of

ways to reduce biased conclusions regarding effectiveness. A review of

propensity scoring studies was conducted for this chapter, where the

limited number of evaluations of criminal justice interventions using

these methods was identified. Due to the small number of these studies,

research was also reviewed if propensity scoring had been employed

to evaluate interventions that are similar to those in criminal justice

systems. These studies are used as examples to demonstrate how the

methods can be used to evaluate criminal justice interventions, the

different ways propensity scores can be used to analyse treatment

and comparison group differences, and the strengths and limitations

of this approach. It is concluded that, while not appropriate for all

interventions/settings, propensity score analysis can be useful in

criminal justice arenas, at least to investigate the comparability of

treatment and comparison groups, with suspected non-comparability

being a common weakness of traditional quasi-experimental studies and

Perspectives on Evaluating Criminal Justice and Corrections

Advances in Program Evaluation, Volume 13, 173–198

Copyright r2012 by Emerald Group Publishing Limited

All rights of reproduction in any form reserved

ISSN: 1474-7863/doi:10.1108/S1474-7863(2012)0000013012

173

frequently cited limitation in terms of drawing efficacy conclusions from

such evaluations.

INTRODUCTION

Propensity scoring was developed by Rosenbaum and Rubin (1983),and

propensity score methods are becoming standard techniques for controlling

confounding variables in medicine and epidemiology (Brookhart et al.,

2006). In the evaluation of many interventions, and most interventions in

criminal justice settings, evaluators have no control over assignment to

treatment and control/comparison conditions. This means that the treated

and comparison groups may have differences that lead to biased conclusions

regarding treatment effectiveness. Propensity scores can be used to balance

the differences in the groups, which can be used in a number of ways to

reduce biased conclusions regarding effectiveness. These methods involve

collecting data on a range of variables that could be argued to be related to

assignment to treatment and/or treatment outcome. These data are used to

develop a model of ‘assignment to treatment’, which is employed to allocate

each participant a propensity score. These scores can then be used in a range

of ways to evaluate treatment outcome, as will be discussed in more detail

below. These methods are less commonly used in criminal justice

evaluations and for evaluations outwith the medical context, though they

have been used more frequently in recent years.

A review of propensity scoring studies was conducted for this chapter.

1

Evaluations of criminal justice interventions using these methods were

identified. Due to the small number of these studies (Berk & Newton, 1985;

Berk, Newton, & Berk, 1986;Jones, D’Agostino, Gondolf, & Heckert,

2004), research was also reviewed if these methods had been employed to

evaluate interventions that are similar to those in criminal justice systems

(e.g. foster care, alcoholics anonymous), or targeted similar populations (at

risk children, mothers) to criminal justice populations, and/or have a similar

format to criminal justice interventions (welfare-to-work programmes).

These studies were included as such interventions have similar problems/

difficulties employing randomised treatment control designs (for a more

detailed discussion, see Chapters 3 and 4, this volume). These studies, which

are discussed in more detail below, revealed that propensity scoring methods

are more frequently used in the United States than elsewhere: all but one of

SARAH BROWN174