When we launched Head Quarters five years ago, psychology was in a pretty dark place. The field was still reeling from the impact of the Diederik Stapel fraud case – the largest perpetrated in psychology and one of the greatest ever uncovered in science. At the same time, a cascade of failures to replicate major findings was just beginning, and as if to add insult to injury, one of psychology’s most prestigious journals published a study claiming to confirm, of all things, the existence of psychic powers.
Psychologists were faced with one inescapable conclusion: that the research culture in the field was fundamentally flawed and needed urgent attention. Five years later, the field has taken some big steps forward toward righting the ship. Let’s take a look at some of those improvements.
We’re now less likely to fool ourselves and others
Scientists are meant to be detectives searching for the truth rather than lawyers cherry-picking the evidence to fit an argument. Unfortunately, psychology has proven to be one of the most bias-prone sciences. Whether conscious or deliberate, it is all too easy for researchers to selectively report their study results in a way that is influenced by their preconceptions. And because of the noisy nature of evidence in psychological science (humans are messy things, after all), these biases can have a substantial impact on the conclusions that are drawn.
One of the best defences against bias is to prespecify or “preregister” details of a study, including the study predictions and how the data will be analysed. We’ve written before about the benefits of study preregistration in science. Preregistering designs helps us avoid fooling ourselves by recording our plans before they are implemented – in other words, drawing the target on the wall before rather than after firing the arrow. Since 2013, one of the major repositories for preregistered plans at the Center for Open Science has recorded thousands of new protocols registered by psychological scientists.
When preregistration is then combined with peer review, it can further help eliminate bias against null or undesirable results because the paper is accepted before the results are known. This initiative, called Registered Reports, was launched at about the same time as we launched Head Quarters and has now grown to include over 120 journals.
…and we’re sharing data and materials like never before
Compared with sciences like climatology, crystallography and genomics, psychology has a poor track record in making data openly available to other researchers and the public. Psychologists have traditionally regarded the data they collect as something akin to personal property, and most refuse to share even when asked directly by colleagues.
This too is changing. In 2015, the Center for Open Science launched the Transparency and Openness Promotion – or TOP – Guidelines, a certification scheme in which thousands of journals are raising standards, with many requiring authors to either publicly archive their data or state why archiving isn’t possible. In parallel, grassroots initiatives are leveraging the power of peer review to push for transparency. Signatories of the Peer Reviewers’ Openness initiative commit to only providing in-depth peer reviews of articles that either make data and materials available, or which state publicly why they can’t be.
Meanwhile, many journals are now offering Badges for Open Practices – among them, kitemarks indicating whether authors have made data or materials available. As these indicators become the norm, articles without them will begin to raise flags, and eyebrows.
We’re starting to do things bigger and better
If psychology were a country, much of it would still be stuck in the feudal period – a multitude of laboratories competing like petty fiefdoms for funding and fame by pumping out lots of small, unreliable research studies. This long-standing tradition is now changing, with alternative federalised models emerging. One impressive example is the Psychological Science Accelerator, launched in 2017, in which hundreds of laboratories across 45 countries are collaborating on joint projects. By conducting research on a much larger scale, the hope is to combine limited resources, increase scientific diversity, and produce more robust conclusions.
The next generation of psychologists is demanding better
As this debate has unfolded, the community of younger scientists that are the future of psychology has been vocal in supporting reforms, even when senior academics might think they know better. Social media – including Twitter, Facebook and blogging, has played a huge role in levelling the traditional academic hierarchy.
The message from younger scientists is clear: Psychology must get its house in order to be a career worth pursuing and a field worth funding. But open science and best practice doesn’t need to come at personal cost to early career researchers. New initiatives are cropping up to recognise open practices in hiring policies at universities – rather than simply valuing the number of prestigious publications an applicant has, or the amount of grant money they have won, universities are now beginning to consider the applicant’s history of open practices. At the same time, a network of open science working groups has emerged across university psychology departments in the UK and Germany, while draft guidance recently published by UK Research Excellence Framework (REF) has emphasised the importance of transparency and reproducibility in judging research quality. This is important because, in the UK, the REF decides every five years how much public funding UK universities receive. These initiatives will set a benchmark for good practice that will be instilled in generations of psychologists to come.
We’re also creating tools to help other sciences fight similar problems
Psychology is far from only science to suffer from a lack of transparency and reliability. Large parts of biomedicine suffer the very same problems, a point brought home in the last few weeks by the disturbing news that among a set of 50 high-profile studies in cancer biology, most could not be replicated. Tools like Registered Reports, the TOP guidelines, PRO initiative, and the Psychological Science Accelerator can just as easily be applied in other areas of science.
In some ways it is fitting that many of the solutions to bad science are coming from psychology. After all, shouldn’t it be the job of scientists who study the mind, and our various human biases, to provide solutions to doing better science? What took us so long?
Why this matters
It matters because psychology matters. Psychology is a remarkably broad discipline and you may be surprised to learn just how much it influences our everyday lives. Discoveries in psychology influence everything from designing traffic signs, to treating mental illness, to shaping policies for combatting climate change. Psychology is everywhere.
And that’s why it needs to be transparent and reliable. The last five years have shown what psychology can achieve by turning the scientific method on itself and embracing the need for reform. We still have a long way to go, but the destination is clear and we now have a road map. All aboard.
To contact the author of this story: theguardian