Are social workers ignoring the 'cornerstone of science' by failing to replicate their research?

Author:Ezell, Mark
Position:GUEST EDITORIAL - Editorial
 
FREE EXCERPT

Aspirited debate is occurring in the popular press on social science and the scientific method. Jonah Lehrer (2010b) published an article in the New Yorker whose primary topic was the "decline effect," where initial estimates of interventions' effectiveness weaken when replicated. Lehrer's examples came from pharmacology, medicine, psychology, zoology, and more. A debate was ignited, largely on the Internet (for example, "Neurological blog," "Respectful Insolence," "Science Based Medicine," "Psychology Today," "ABC News"). In the next issue of the New Yorker, Lehrer (2011) responded to letters and e-mails in another article as well as an article in Wired (Lehrer, 2010a).

Issues raised by the debate deserve consideration by social work researchers. This editorial explains the decline effect and presents comments on replications, failure to submit, publication bias, and "a fundamental cognitive flaw" (Lehrer, 2010b). The term "decline effect" has not appeared in social work literature, nor have estimates of the number of replications, the extent of failing to report research, or evidence of journals' publication bias, with the exception of Dickersin (1997). I base my comments on almost 30 years in academic social work at four universities, extensive reading of our research literature, teaching research, reviewing manuscripts, and publishing.

THE DECLINE EFFECT

The "decline effect" is a phrase attributed to Rhine (1938) in research on extrasensory perception describing the situation in which a research participant guessed a hidden card at a rate beyond chance. Alter repeated testing, however, "the effect dramatically diminished" (Rhine, 1938). Lehrer (2010b) reported a similar pattern in research about antipsychotic medications. Early clinical trials documented the medications' positive impact on psychiatric symptoms, but recent research indicates that "the therapeutic power of the drugs appeared to be steadily waning" (Lehrer, 2010b, para. 2).

Another researcher, Jonathon Schooler, attempted to replicate Rhine's finding of a decline effect by studying precognition. At first, Schooler found higher than expected precognition, "'but then, as we kept on running subjects, the effect size'--a standard statistical measure--'kept getting smaller and smaller'" (Lehrer, 2010b, p. 6-7).

Lehrer (2010b) described similar results in research on language and memory, zoology, biology, and epidemiology. He described a study in which the researcher identified the 49 most frequently cited clinical research studies and reviewed the subsequent research literature to see if there might be a decline effect (Ioannidis, 2005).The sampled studies had to have 1,000 or more citations. In 45 of the sampled studies the intervention in question was found to be effective. (It is interesting to note that the four studies in the sample that showed no efficacy for the intervention being studied were contradicting earlier claims of efficacy). Thirty-four of the studies in the sample had been replicated: 21% were contradicted by subsequent research, and 21% had weaker effects. One should not conclude, Ioannidis (2005) pointed out, "that the original studies were totally wrong and the newer ones are correct simply because they are larger or better controlled" (p. 224).

To what degree has the decline effect been observed in social work? A literature search reveals no reference to it as such. The prior question, however, and the one of central concern here is this: How frequently are replication studies done in social work? Without replications, decline effects cannot be observed. Replication studies seem to be fairly rare in social work; if they occur, it is likely to happen with more heavily funded and broadly implemented interventions.

Failure to replicate intervention studies is costly for reasons far beyond documenting decline effects:

Replication, widely acknowledged as...

To continue reading

FREE SIGN UP