The problem with our data-driven world...
J Thoendell stashed this in Science
But then you see studies like the one that recently came out in Science, America’s leading scientific journal, that subjected 100 supposed high-quality psychology papers to a large-scale replication study. When new research groups replicated the experiments in the papers to see if they’d get the same results, they were only able to do so 36% of the time. Almost two-thirds of the papers’ effects couldn’t be replicated by other careful, professional researchers.
“This project provides accumulating evidence for many findings in psychological research and suggests that there is still more work to do to verify whether we know what we think we know,” concluded the authors of the Science paper.
In many fields of research right now, scientists collect data until they see a pattern that appears statistically significant, and then they use that tightly selected data to publish a paper. Critics have come to call this p-hacking, and the practice uses a quiver of little methodological tricks that can inflate the statistical significance of a finding. As enumerated by one research group, the tricks can include:
- “conducting analyses midway through experiments to decide whether to continue collecting data,”
- “recording many response variables and deciding which to report postanalysis,”
- “deciding whether to include or drop outliers postanalyses,”
- “excluding, combining, or splitting treatment groups postanalysis,”
- “including or excluding covariates postanalysis,”
- “and stopping data exploration if an analysis yields a significant p-value.”
Add it all up, and you have a significant problem in the way our society produces knowledge.