Moving From Efficacy to Effectiveness

• DOI: http://doi.org/10.1111/1745-9133.12316
• Author: Michael Robbins,Allison J. Ober,Jessica Saunders
• Published date: 01 August 2017
• Date: 01 August 2017

RESEARCH ARTICLE

DRUG MARKET INTERVENTION

Moving From Efficacy to Effectiveness

Implementing the Drug Market Intervention Across

Multiple Sites

Jessica Saunders

Michael Robbins

Allison J. Ober

RAND Corporation

Research Summary

In 2012, the editors of CPP published an exchange about the Drug Market Intervention

(DMI) in High Point, NC, concluding that it may be a promising approach to crime

control but questioning whether it could be implemented across different settings. In

this effectiveness study, we followeda cohort of seven sites that par ticipated in a Bureau

of Justice Assistance–sponsored DMI training to assess implementation and outcomes.

Three sites were not able to implement, and implementation fidelity varied across the

four sites that did implement. Of the four sites that held at least one call-in, only one

was successful at reducing overalland drug crime (by 28% and 56%, respectively). This

works out to an implementation rate of 57% with an average overall crime reduction

of 16% (treatment-on-the-treated) or 4% (intent-to-treat). The results of this study

This work was made possible by Award 2010-DJ-BX-1672, granted by the National Institute of Justice, Office

of Justice Programs. The opinions, findings, and conclusions and recommendations expressed in this

publication are those of the authors and do not necessarily reflect the views of the Department of Justice. We

would like to thank the members of our expert panel: Anthony Braga, Rodney Brunson, Jonathan Caulkins,

Scott Decker, Anne Piehl, and George Tita. We would also like to thank the team from the Michigan State

University Training and Technical Assistance program for providing guidance and material reviews,

particularly Natalie Hipple, Ed McGarrell, Tim Bynum, and Heather Perez. Additionally, we would like to extend

our gratitude to each on-site research coordinator who collected implementation data: Zachary Bryan, Robert

Farley, Viola (Simone) May, Michael McCloskey, Alexis Norris, Mary Perrilloux, and Robert Stallings. We also

express our gratitude to each team for welcoming us and providing us unfettered access to its program

implementation. Finally, we thank our other project team members, specifically Beau Kilmer, Sarah

Greathouse, Greg Ridgeway, and Rob Davis. Direct correspondence to Jessica Saunders, RAND Corporation,

1776 Main Street, Santa Monica, CA 90407 (e-mail: jsaunder@rand.org).

DOI:10.1111/1745-9133.12316 C2017 American Society of Criminology 787

Criminology & Public Policy rVolume 16 rIssue 3

Research Article Drug Market Intervention

demonstrate the importance of replication and the careful study of implementation

fidelity prior to wide dissemination.

Policy Implications

When the findings of an evaluation reveal an effective crime reduction program,

particularly when it garners significant public attention, it is not uncommon to rush

to judgment that it should be widely implemented. DMI is a perfect illustration of

this shortsighted approach to evidence-based crime prevention—multiple trials across

a variety of contexts are necessary to understand whether a program is ready for broad

dissemination and scale-up. The DMI program was challenging for sites to implement

and resulted in significant reductions in crime in the site with the implementation

fidelity that was highest and most similar to the original site. Our findings echo earlier

concerns that the approach may be less effective across diverse settings and illustrate why

effectiveness studies are vital in the development of evidence-based policy.

The field of crime prevention and reduction has moved toward using high-quality

scientific research as a means of identifying and disseminating evidence-based pro-

grams and practices (Farrington, Gottfredson, Sherman, and Welsh, 2002; Welsh

and Farrington, 2006). The quality standards differ across clearinghouses, but in general,

there is no specific requirement for program replication or evidence of implementation

success or intervention effectiveness across multiple trials and settings for programs to be

rated as “promising” or as “effective.” Nevertheless, “successful” programs may not be ef-

fective when replicated elsewhere because the initial results were dependent on the specific

conditions under which the trial was undertaken. Without information from multiple tri-

als, we have an incomplete, and sometimes biased, understanding of the likely program

effectiveness in subsequent implementations.

Following the Society of PreventionResearch (SPR) model for evidence-based practice

development, prevention strategies, polices, and programs generally follow a three-stage

process: (1) development and testing in an efficacy trial or pilot, (2) effectiveness trials pro-

ducing consistent effects across multiple trials in appropriate settings, and (3) a dissemination

strategy that may include manuals and technical assistance strategies to ensure that replica-

tions are implemented with fidelity so they continue to produce desired results (Flay et al.,

2005; Kellam and Langevin, 2003). In fact, there are strict standards and requirements for

efficacy and effectiveness trials before programs should be scaled up (Gottfredson et al.,

2015), which few, if any, community crime prevention programs can currently meet. This

has real consequences for the field as many federal initiatives are designed to encourage or

even require the adoption of “evidence-based practices”that may not be able to produce the

desired or anticipated results because they are based on only one or a few trials. This point

was stressed by Papachristos (2011) in his essay “Too Big to Fail,”in which he described how

788 Criminology & Public Policy