Broken science: what happens when cancer doctors, psychologists, and drug developers can't rely on each other's research?

• Author: Bailey, Ronald

NULLIUS IN VERBA is the motto of one of the earliest scientific associations--the Royal Society, founded in 1663. Broadly translated, the phrase means "Don't take anybody's word for it."

You know how it's supposed to work: A scientist should ideally be able to do the same experiment as any other scientist and get similar results. As researchers check and recheck each other's findings, the sphere of knowledge expands. Replication is the path to scientific advancement.

Some 15 million researchers published more than 25 million scientific papers between 1996 and 2011. Among them were several casting doubt on the veracity and reliability of the rest--suggesting that even studies published in gold-standard journals by researchers from top-tier institutions are far more likely than anyone previously realized to be false, fudged, or flukey. The upshot is that many researchers have come to believe that science is badly battered, if not broken.

Everything We Know Is Wrong? The Stanford statistician John Ioannidis sounded the alarm about our science crisis 10 years ago. "Most published research findings are false," Ioannidis boldly declared in a seminal 2005 PLOS Medicine article. What's worse, he found that in most fields of research, including biomedicine, genetics, and epidemiology, the research community has been terrible at weeding out the shoddy work largely due to perfunctory peer review and a paucity of attempts at experimental replication. Ioannidis showed, for instance, that about one-third of the results of highly cited original clinical research studies were shown to be wrong or exaggerated by subsequent research. "For many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias," he argued. Today, he says science is still wracked by the reproducibility problem: "In several fields, it is likely that most published research is still false."

Initially, some researchers argued that Ioannidis' claims were significantly overstated. "We don't think the system is broken and needs to be overhauled," declared New England Journal of Medicine editor Jeffrey Drazen in The Boston Globe in 2005. But his analyses have sparked a vast and ongoing reassessment of how science is done. Once other scientists started looking into the question, they found the same alarming trend everywhere.

In 2012, researchers at the pharmaceutical company Amgen reported in Nature that they were able to replicate the findings of only six out of 53 (11 percent) landmark published preclinical cancer studies. Preclinical studies test a drug, a procedure, or another medical treatment in animals as precursors to human trials. In 2011, researchers at Bayer Healthcare reported that they could not replicate 43 of the 67 published preclinical studies that the company had been relying on to develop cancer and cardiovascular treatments and diagnostics. Ioannidis estimates that "in biomedical sciences, non-replication rates that have been described range from more than 90 percent for observational associations (e.g., nutrient X causes cancer Y), to 75-90 percent for preclinical research (trying to find new drug targets)."

The mounting evidence that most scientific findings are false provoked a rash of worried headlines, including "How Science Is Broken" at Vox in 2015; "Why medical clinical trials are so wrong so often" in The Washington Post in 2015; "The Truth Is Many Scientific Studies Can't Be Trusted" at Business Insider in 2012; "Lies, Damned Lies, and Medical Science" in The Atlantic in 2010; and...