Arguments about Methods in Criminal Justice Evaluation

• Pages: 49-69
• Date: 21 May 2012
• Published date: 21 May 2012
• DOI: https://doi.org/10.1108/S1474-7863(2012)0000013007
• Author: David Smith

ARGUMENTS ABOUT METHODS

IN CRIMINAL JUSTICE

EVALUATION

David Smith

ABSTRACT

The chapter considers the change of position of the Home Office on the

value of randomised controlled trials (RCTs) in England and Wales

which took place around 2003 after the end of the Crime Reduction

Programme (CRP). Before the CRP Home Office researchers had

shown little interest in RCTs; after it, they came close to arguing that no

other kinds of evaluation research were worth doing. This represented a

reversal of a position that had dominated Home Office thinking on the

issue for almost 30 years – that RCTs were in general impractical and

unlikely to produce clear-cut results. This view was based in part on the

experience of RCTs in the 1970s, which led influential researchers to

conclude that the method could not be transferred from medicine to

criminal justice. But, disappointed with the lack of definite results from

the CRP, the Home Office turned back to RCTs as a potential source of

certainty about what works. The chapter considers two recent scholarly

exchanges on the question, in relation to an evaluation of a community

crime reduction programme, for which an experimental design was

attempted but not achieved, and to Lawrence Sherman’s recent advocacy

of RCTs and his use of research on restorative justice as an example of

Perspectives on Evaluating Criminal Justice and Corrections

Advances in Program Evaluation, Volume 13, 49–69

Copyright r2012 by Emerald Group Publishing Limited

All rights of reproduction in any form reserved

ISSN: 1474-7863/doi:10.1108/S1474-7863(2012)0000013007

49

the successful use of the method. The chapter argues that the restorative

justice research, while of very high quality, does not provide as clear an

example of the use of an RCT as Sherman claims, and concludes with

some reflections on the inherent difficulties of criminal justice evaluation,

and on the lack of a predictable, rational relationship between research

quality and policy influence.

INTRODUCTION

Without seeking to re-open the ‘paradigm wars’ (Pawson & Tilley, 1998a;

see also Astbury, this volume) among competing schools of evaluation, I try

in this chapter to give an account of some recent arguments about the merits

or otherwise of randomised controlled trials (RCTs) in criminal justice

evaluation. The starting point for this is the change in attitude of Home

Office researchers between 2000 and 2004, when they shifted from

apparently accepting that RCTs would rarely be feasible to (in some

accounts at least) apparently claiming that no other kinds of research were

worth doing. After discussing this volte-face, the chapter briefly reviews the

history of criminological RCTs in Britain, before considering two recent

exchanges between enthusiasts for RCTs and advocates of more ‘realist’,

non-experimental methods. The example of research on restorative justice

schemes, claimed as experimental but (inevitably) rougher in some respects

than this designation implies, is used to suggest that the practicality of pure

RCTs in the field of criminal justice is limited, a point further illustrated

from my own experience. Finally, I offer some brief reflections on what

kinds of research have influenced or arguably ought to have influenced

policy and conclude that there is no reason to think that RCTs have a better

chance than other methods of achieving such influence (see also W. L.

Marshall & Marshall, this volume).

THE CRIME REDUCTION PROGRAMME

AND AFTER

The Crime Reduction Programme (CRP) which ran in England and Wales

from 1999 to 2001 emerged in a period of official optimism that it was

possible to identify (at least tentatively) what was likely to work in crime

prevention efforts, and that evaluative research could turn these tentative

DAVID SMITH50