In tumultuous times, it is easy to miss the fact that science is undergoing a quiet revolution. For several years now, concerns have been peaking in biomedicine about the reliability of published research – that the results of too many studies cannot be reproduced when the methods are repeated. Alongside growing discontent, the scientific community has answered by driving forward a raft of open science reforms. From initiatives to making research data publicly available, to ensuring that all published research can be read by the public, the aim of these reforms is simple: to make science more credible and accessible, for the benefit of other scientists and the public who fund scientific research.
Today one of these reforms takes hold for the first time in clinical medicine: a new type of journal article called a Registered Report in which the journal commits to publishing clinical trials regardless of their outcome. This might sound like common sense – because that’s exactly what it is – but in the competitive world of science and academia it represents a significant departure from the status quo.
The trouble with trials
All science is coloured by human imperfection, but you could be forgiven for thinking that clinical trials ought to be immune to low reliability. Trials are high stakes enterprises that can influence life and death, and have long been regarded as the gold standard in discovering whether a new treatment works. In a typical trial, you divide your sample randomly into two groups. You administer the new treatment to one group, and a control treatment to the other. You blind the patients in each group to whether they’re receiving the treatment or the control, and you blind the researchers too until the end of the trial. Finally, before the study even commences, you register the study design in a public forum together with a plan for analysing the data. These steps are intended to stop researchers from fooling themselves into seeing what they want to see.
In a perfect world, clinical trials would indeed be a bastion of scientific credibility. But the “publish or perish” mentality of academia collides violently with these ideals.
The first problem is publication bias – a tendency for certain types of results to be easier to publish than others, even when the quality of the underlying methods is the same. Suppose you were to test a promising new treatment and you find that it reliably outperforms the current treatment – this is what is known as a positive result. Depending on how important this outcome is, you would have relatively little trouble publishing your findings in a respected peer-reviewed journal. You might even have a shot at one of the more prestigious medical journals, which in turn could provide a great boost to your academic career, making it easier for you to get promoted and acquire more research funding.
But suppose the promising new treatment didn’t perform noticeably different from the current treatment – a so-called negative result. Now you’re in a world of academic pain. The most prestigious journals are unlikely to consider your results newsworthy enough to publish. You may find it impossible to publish the study within any respected journal, and you might even run out of funding while trying to find your study a home. After all this frustration you may well decide to abandon publishing it altogether, leaving the results in what scientists call the file drawer: a great attic in the sky for results that were too complex or ambiguous to see light of day.
Over time, these incentives teach researchers that positive results are good and negative results are bad, despite the fact that negative results – showing that a treatment doesn’t or may not work – are absolutely vital for advancing medicine. This publication bias is a major reason why somewhere between 33 to 60% of clinical trials never report results. Just imagine for a moment what this does to our understanding of which medical treatments really work and why.
The second problem with clinical trials is a form of cherry picking called hidden outcome switching. Back in the 1980s, the medical community decided that clinical trials should be registered in advance, pre-specifying the study design, outcome measures, and analysis plan. One of the reasons this was brought in was to stop cherry picking by researchers, either deliberately or unconsciously. In any study with a lot of variables, it is relatively easy to find a positive result by changing the main outcome measure after looking at the data or by using a different type of analysis to the one originally planned. Just as shifting the goal posts in a football match guarantees a goal from every kick, with enough data dredging a researcher is guaranteed to find something they can publish, even if that “something” is a false discovery.
Publication bias places researchers under enormous pressure to engage in cherry picking – all the incentives in academia point researchers toward cutting corners and fooling themselves (and others) in the interests of publishing in prestigious journals. So it comes as little surprise that cherry picking is rife in clinical trials: a 2014 analysis found that around 1 in 3 trials change their primary outcome measure after the trial is complete, and Ben Goldacre’s COMPARE project recently found that of 67 trials published by the most prestigious medical journals, 58 covertly altered their outcome measures from the registered protocol.
Registered Reports to the rescue
This all sounds pretty grim, but it turns out that the solution to both publication bias and hidden outcome switching is straightforward. The first step is to treat the publication process as we would treat a clinical trial itself and ensure that journals are kept blind to the results when they decide whether to accept or reject articles. This guarantees that positive and negative results are on equal footing in the published literature. And secondly, we need to ensure that researchers adhere to their registered trial protocols, or at least explain why deviations from protocol are required. For this, peer review of the protocol provides the answer.
A new type of article called a Registered Report provides both of these features. Unlike conventional scientific publishing, Registered Reports are reviewed in two stages. Researchers first submit their protocol to the journal before they collect their data. This undergoes peer review focusing on the study rationale and robustness of the methodology. Following any necessary revision, the protocol can then be accepted in advance by the journal, guaranteeing publication of the outcomes provided the researchers adhere to their protocol. As a condition of final publication, the researchers are also required to make any anonymised data from their study publicly available.
Registered Reports originated in psychology a few years ago as a way to address concerns about reproducibility, and have since been taken up by 70 academic journals covering a wide range of sciences. As well as preventing publication bias and cherry picking, they change the core incentives in science, liberating researchers from the pressure to report positive results because the results themselves become a dead currency. For a Registered Report, the results of a trial make no difference to whether the trial is published.
Today BMC Medicine becomes the first major medical journal to offer Registered Reports and all medical journals should follow suit. With the stakes in clinical research so high, they have little reason to refuse. Indeed, if as a society we agree that biased reporting in medicine is intolerable, shouldn’t all trials be published as Registered Reports?