Can Evidence-Based Policy Ameliorate the Nation’s Social Problems?

• Author: Gordon Berlin,Carolyn J. Hill,Virginia Knox
• DOI: 10.1177/0002716218769844
• Published date: 01 July 2018
• Date: 01 July 2018
• Subject Matter: Significance of the Evidence-Based Movement

769844ANN
The Annals of the American AcademyCan Evidence-Based Policy Ameliorate Social Problems?
research-article2018
This article updates the pipeline paradigm for evidence
building with a cyclical paradigm that encompasses
evidence building, implementation, and adaptation. A
cyclical paradigm for evidence-based policy and prac-
tice assumes that an intervention will be adapted over
time, across settings, and across populations. These
innovations and adaptations are encouraged and tested,
with periodic review of the need for impact assessment.
The cyclical paradigm also emphasizes service contrast
Can Evidence- at every stage, not just in the evidence-building stage
where it has traditionally been a focus. A continuous
Based Policy cycle of evidence building, implementation, and
adaptation—
looping back to further evidence
building— can help to ensure that the impacts of evi-
Ameliorate the dence-based policies and programs are sustained and
grow in new settings.
Nation’s Social Keywords: evidence; implementation; adaptation;
Problems?
evaluation; counterfactual; service contrast
In recent decades, policy-makers have grown
more likely to use research evidence to guide
their attempts to meet social and educational
goals. researchers have identified many
By
reforms worthy of broad expansion, including
VIrgINIA KNOx,
changes in the welfare system and programs
CArOLyN J. HILL,
that help low-income parents foster their chil-
and
dren’s early development. yet despite these
gOrDON BErLIN
successes, on the whole it remains hard to
implement large-scale interventions supported
Virginia Knox is director of the families and children
policy area at MDRC. She has led numerous evalua-
tions that provide information on both impacts and
potential program improvements, particularly inter-
ventions that build parents’ abilities to support their
children’s healthy development.
Carolyn J. Hill is a senior fellow at MDRC. Her
research interests are in implementation, public man-
agement, performance measurement, and program
evaluation. She is the coauthor of Public Management:
Thinking and Acting in Three Dimensions (CQ Press
2016).
Correspondence: virginia.knox@mdrc.org
DOI: 10.1177/0002716218769844
166 ANNALS,
AAPSS, 678, July 2018

---

CAN EVIDENCE-BASED POLICy AMELIOrATE SOCIAL PrOBLEMS?
167
FIgurE 1
The Prevailing Evidence-Building Pipeline Paradigm
by evidence. Even when interventions are grounded in current knowledge and
show positive effects in early tests, those effects are often modest, and they often
are not repeated when the programs expand to other settings.
The prevailing paradigm for building evidence about programs or policies is a
linear one (see the pipeline model shown in Figure 1). A new intervention is
developed in response to a social problem. It undergoes early tests. If those tests
find positive results relative to what participants would have otherwise experi-
enced, then additional impact studies are conducted in new locations (replica-
tion). If the replication studies are positive, funders will support further expansion
(scaling up), expecting to see similar effects as long as future versions implement
the core elements of the intervention faithfully (see, for example, Institute of
Education Sciences 2016). This pipeline paradigm is sometimes accompanied by
a tiered funding model, in which more funding is made available to expand a
program to a larger scale as it generates more—and more rigorous—evidence of
its continued effects in more locations.
In this article, we suggest an updated paradigm. Compared with the pipeline
model, this framework better reflects how social programs develop across loca-
tions and over time, and thus could more effectively address social problems. It
moves evidence-based policy forward by explicitly acknowledging the comple-
mentary nature of rigorous research and implementation. In a cycle of continu-
ous learning, implementation decisions are guided by evidence as much as
possible, and adaptations of evidence-based models are used to generate further
evidence about a program’s outcomes. It brings together three processes: build-
ing rigorous evidence produced from a pipeline paradigm; implementing policies
and programs that are grounded in evidence about how, for whom, and where an
intervention can be effective; and learning from adaptation in a systematic way.
The framework is cyclical, so the need for rigorous evidence building is continu-
ally assessed. And it emphasizes service contrast at every stage of the process
instead of only during the impact evidence-building stage.
Gordon Berlin is president of MDRC, a nonprofit, nonpartisan research organization dedi-
cated to learning what works to improve the lives of low-income people. Berlin has authored
numerous publications, including “Rewarding the Work of Individuals: A Counterintuitive
Approach to Reducing Poverty and Strengthening Families” (The Future of Children 2017).
NOTE: We acknowledge our MDrC colleagues, especially rekha Balu, William Corrin, and
John Martinez. Our discussions and collaborations with them have contributed substantially to
the ideas in this article.

---

168
THE ANNALS OF THE AMErICAN ACADEMy
This updated paradigm can be used by funders, researchers, and practitioners
who want to use evaluation to strengthen programs and their impacts. Ideally, the
paradigm can encourage conversations among these different contributors to the
improvement of the programs that are, after all, the foundational content of
evidence-based policy.
In this article, as in the rest of this volume, the term rigorous evidence refers to
evidence of impacts based on research designs that allow one to conclude that a
given social intervention caused the effects observed. As discussed elsewhere in
this volume, such studies primarily use high-quality experimental or quasi-
experimental designs. Experimental designs (also called randomized controlled
trials) compare the outcomes of a group of people assigned at random to receive
an intervention with the outcomes of a control group whose members may often
receive other services available in the community but do not receive the interven-
tion being studied. Because they were assigned at random, the two groups’ measur-
able and unmeasurable characteristics should be similar at the outset of the study,
so researchers can be highly confident that any differences in the outcomes of the
two groups are caused by the program group’s participation in the intervention.
Other studies or program improvement activities that evaluate program effec-
tiveness use methods that may incorrectly attribute improvements in partici-
pants’ outcomes to their participation in the intervention (for example, designs
that measure the outcomes of a group before and after it receives an interven-
tion, or designs that compare the outcomes of program participants with the
outcomes of a group of people who are similar but who were not chosen ran-
domly). However, applying some of these methods, or “design elements,” in
combination can result in more rigorous quasi-experimental designs; and under
certain conditions, some quasi-experimental designs can approach the rigor of a
randomized controlled trial (Corrin and Cook 1998; Shadish, Cook, and
Campbell 2002).
Successes and Limitations of the Pipeline Paradigm
The evidence-based policy movement is capitalizing on an unusual window of
opportunity. Politicians across the ideological spectrum increasingly support
using evidence to make decisions about funding, ending, or improving programs
(Berlin 2016). rigorous empirical studies give policy-makers confidence that
social programs can improve people’s lives and at the same time give them con-
fidence that programs can be held accountable for doing so. In recent years, early
successes led to efforts by the executive branch (principally in the Office of
Management and Budget) to demand and reward the production and use of evi-
dence in policy and budgeting. They have also led to efforts by the legislative
branch to write new laws with tiered evidence structures. These laws encourage
states or other service providers to implement evidence-based models, while also
supporting innovation by linking increasing funding to stronger evidence bases
(Institute of Education Sciences 2016; gueron and rolston 2013; Haskins and
Margolis 2014).

---

CAN EVIDENCE-BASED POLICy AMELIOrATE SOCIAL PrOBLEMS?
169
research has identified many effective approaches to social problems such as
poverty or poor health—and ineffective approaches as well. Early large-scale
social policy experiments such as the rAND Health Insurance Experiment
(Brook et al. 1984), the Abt Housing Allowance Demand Experiment (Merrill
and Joseph 1980), and the Negative Income Tax Experiment (Munnell 1986),
were not set up simply to test models with the idea of replicating them. Instead,
these studies tested theories and provided empirical evidence of parameters
underlying the design of health insurance programs, housing subsidies, and
income-support concepts. The National Supported Work Demonstration
(Manpower Demonstration research Corporation 1980) and the later Welfare-
to-Work experiments of the 1990s (greenberg, Deitch, and...