Analysis of neurological studies on animals reveals widespread bias.
The road to market for a promising new therapy can be notoriously long and treacherous. Before the first small-scale clinical trials in humans can even be contemplated, a new therapy (such as a drug or surgical procedure) must first pass muster in preclinical animal studies.
A study published in PLoS Biology has uncovered considerable bias in the reporting of results from animal studies into neurological diseases. The result is that some treatments could be progressing to human clinical trials on the back of flimsy preclinical evidence. The discovery could also help to explain why promising results in animals often fail to translate into promising results in humans.
Although logistically simpler than human clinical trials, animal studies are not without their limitations. For ethical and financial reasons, studies aim to keep animal numbers to a minimum. The problem with limiting animal numbers is that studies can end up with ambiguous results. A new drug might have improved outcomes for some of the animals, but the difference between the animals taking the drug and those taking the placebo could be too close to make a confident call.
To get around this problem, researchers can pool the results from several studies of the same treatment to perform what’s called a meta-analysis. Combining results in this way effectively creates a virtual study with far more animals – and far better statistical oomph – than any of the individual studies. If there really is a difference between a drug and a placebo, you can be confident that the effect is real.
The new analysis, conducted by an international team of researchers led by John Ioannidis at Stanford University, examined the results of meta-analyses of animal studies into a range of neurological disorders. The analysis included 160 meta-analyses, containing the combined data from 4445 individual studies into Alzheimer disease, Parkinson’s disease, stroke and spinal cord injury, among others.
The analysis found that far more studies claimed statistically significant results – meaning they were confident that their treatment was better than placebo – than would be expected. The authors calculated that of the 4445 studies, only 919 should be able to report statistically significant results, where they are 95% sure that the effect of the treatment is real. Instead, they found that nearly twice this number – 1719 studies – reported significant findings.
So where does this bias towards positive results come from? The authors point to a few contributing factors. Selectively publishing only studies that show positive results, or choosing creative statistical analyses that show results in a favourable light are likely contributors to the problem. But the authors also noted that studies reporting a conflict of interest were particularly prone to bias. And although the study looked only at neurological disease research, they suspect that similar bias towards positive results occurs in other areas.
In addition to distorting the academic literature and poisoning meta-analyses, too many positive reports about a treatment’s effectiveness in animals can lead to human clinical trials being commenced – at great risk to participants and great cost – on the basis of shaky evidence.
The authors suggest that pre-registering animal studies could help to ensure that both positive and negative results make their way to publication. This is already a requirement for human clinical trials. Greater transparency through open access to study methodology, raw results and statistical analyses would also help to reduce bias in the future.
Reference: Tsilidis KK, Panagiotou OA, Sena ES, Aretouli E, Evangelou E, Howells DW, Salman RA, Macleod MR & Ioannidis JPA. (2013). Evaluation of excess significance bias in animal studies of neurological diseases PLoS Biology DOI:10.1371/journal.pbio.1001609