Randomized trials and quasi-experiments in education research.

• Author: Angrist, Joshua D.
• Position: Research Summaries

The 2001 No Child Left Behind (NCLB) Act promises a series of significant reforms. The hope is that these reforms will jump-start under-performing American schools. Most public discussion of the Act has focused on the mandate for test-based school accountability and the federal endorsements of charter schools and other forms of school choice. Other important provisions include changes in funding rules for states and a new emphasis on reading instruction. The NCLB Act also repeatedly calls for education policy to rely on a foundation of scientifically based research. Although this appears to be a bland technical statement, it strikes me as potentially at least as significant as other components of the Act.

What is Scientifically Based Research?

NCLB defines scientifically-based research as research using rigorous methodological designs and techniques, including control groups and random assignment. In a presentation made shortly after President Bush signed NCLB into law, the deputy director of the Office of Research in the Department of Education put studies involving randomized trials and quasi-experiments at the top of the methodological hierarchy.

Randomized trials are experiments in which the division into treatment and control groups is determined at random (for example, by tossing a coin). Quasi-experimental research designs are based on naturally occurring circumstances or institutions that (perhaps unintentionally) divide people into treatment and control groups in a manner akin to purposeful random assignment.

A reliance on control groups and random assignment indeed would mark a new direction for education research. For example, an important question on the education research agenda is the role of technology in schools. Most previous research on the use of technology in the classroom (computer-aided instruction or CAI) relies on uncontrolled measurements, such as the level of satisfaction experienced by technology users. Not surprisingly, teachers and students typically report that they enjoy using new computer equipment (as shown in a recent study of laptops in Maine's public schools). But this does not establish that students who use the laptops are learning more, or that the expenditure on computers meets a cost-benefit standard (after all, computer hardware and software is expensive).

Randomized trials provide the best scientific evidence on the effects of policies like educational technology, changes in class size, or school vouchers because differences between the treatment and control group can be attributed confidently to the treatment. A good quasi- or natural experiment is the next best thing to a real experiment. In some cases, quasi-experiments also involve random assignment, such as in the lotteries sometimes used to distribute school vouchers. In addition to comparing apples to apples, randomized trials and natural experiments also rely on assessments by disinterested non-participants and on clearly defined outcomes that other researchers can reproduce and interpret. This is what science is all about. In contrast, U.S. education policy has often relied on evidence that is fragmentary or anecdotal, uses subjective outcomes, and, most importantly, fails to make rigorous comparisons of treatment and control groups.

If successful, a shift to scientifically based research will move the study of education much closer to medicine, which has been experiencing a similar transition to scientifically based research over the last half-century. NBER researchers have been in the vanguard of this transition to scientifically based research on education. We have used natural experiments--and in some cases, actual randomized trials--to provide powerful evidence on issues ranging from the effects of compulsory attendance laws to changing class size. I describe some of this work below, focusing on my own efforts. I have used quasi-experiments--and in recent and ongoing projects, randomized trials--to make scientifically grounded inferences regarding the effects of achievement incentives and school choice, school resources, and macro education policy.

School Incentives and School Choice

The desire to help disadvantaged teens get through high school is a recurring theme of school reform proposals. Most anti-dropout efforts involve the provision of support services to low-achieving students. But the results from recent demonstration projects assessing...