EXCITING results from a scientific study are in effect meaningless if they cannot be replicated. All too often, at least in psychology experiments, that seems to be the case. A new report by a scientist who looks at this area, Brian Nosek of the University of Virginia, has once again showed that a high proportion of psychology studies failed to replicate. And this time, Dr Nosek and his colleagues may have found a shortcut to identify which fall into this category.
In most circumstances, a study is considered to be significant if the odds are 5%, or lower, that the result would have occurred by chance. So for every 20 studies that get published, it is reasonable to expect that one will have results that are not correct. In 2015 Dr Nosek, working with a different team, found something alarming: that a whopping 64% of 97 psychology experiments that he re-ran failed to replicate.
Those experiments had appeared in specialist publications. For his new work, published this week in Nature Human Behaviour, Dr Nosek selected experiments that had appeared in Science and Nature between 2010 and 2015. He expected replications of work in these top-tier journals to be more successful. He also re-ran them using samples that were five times larger than those of the originals to reduce the possibility of getting a different result due to chance. Although the results were better than those of the 2015 study, eight of the 21 experiments failed to replicate. Among the 13 that did, a calculation of the size of the effect the studies were examining was, on average, 75% of that in the original experiment.
As a psychologist himself, Dr Nosek was curious whether the research community had a sense of which sorts of experiments were likely to replicate. To this end, he found 206 social scientists (mostly psychologists and economists) via social media. Given tokens, each with a nominal value of 50 cents, the experts were invited to trade on the outcome of the re-run experiments, buying into the “shares” of studies which they thought would replicate and selling or shorting those they thought would not.
The social scientists were on the money. The market prices for each share associated with work that replicated all ended up being worth more than those that did not. Those owning shares in the replicating studies got paid out according to the value of the tokens they held. Stakes in the non-replicants were worth nothing. All this suggests that experts had a decent inkling ahead of time of which of the studies would not replicate, despite the peer-review process used by scientific journals to weed out experiments that might not be robust. Perhaps, then, there is a market opportunity in testing scientific results.
This article appeared in the Science and technology section of the print edition under the headline "Betting on the result"