More and more experiments aren’t reproducible

More and more experiments aren’t reproducible August 31, 2015

Scientists are worried because more and more experiments are not reproducible.  A principle of the scientific method is that for an experiment to be valid, another scientist who performs it must get the same results.  In many cases today, that is not happening.  This is especially true in the social “sciences.”

From Pete Spotts, An emerging challenge to science’s credibility – CSMonitor.com:

Concerns are mounting that a pillar of modern science is showing cracks.

A key feature of science is researchers’ ability to reproduce experiments – to conduct a reality check on another group’s work by using its materials and following its methods, then comparing the results.

It’s a way to separate results worth building upon from those that aren’t, either because a research team was careless, overlooked something, misinterpreted data, or at worst, fabricated results.

During the past several years, however, worries have grown that many nonreproducible results are working their way into the scientific literature, lingering undetected and, importantly, unchallenged. Such results can feed into others’ work as they design their own experiments or pose their own research questions.

At stake, researchers say, is the credibility of science, especially when it is invoked to inform public policy on issues from climate change to new medical treatments. Investment decisions also hinge on credible, reliable research.

Concerns may grow further with the publication Thursday of what purports to be the most systematic effort to date to replicate others’ experiments. More than 270 scientists around the world banded together to replicate 100 social- and cognitive-psychology experiments whose results appeared in three prestigious psychology journals.

While 97 percent of the original studies showed statistically significant results, only 36 percent of the replicated studies did. Of the successful replications, 83 percent showed a much smaller effects than the original studies showed, says Brian Nosek, a social psychologist at the University of Virginia and executive director of the Center for Open Science in Charlottesville, Va.

The results come laced with caveats. Results are not necessarily wrong if they can’t be replicated, nor are they necessarily right if they can be replicated, the researchers note. Numerous variables can come into play, yielding different results even when the same general procedures are followed. Moreover, the new study represents an initial look at a small sample of experiments in one discipline.

Indeed, it’s unclear if these new results themselves could be reproduced, suggested Dr. Nosek, one of the study’s coauthors, during a briefing this week on the work.

“This should just be seen as a first step, an initial piece of evidence for establishing what reproducibility in general might be,” he said.

[Keep reading. . .]

Well, maybe what this particular study shows is that  “social- and cognitive-psychology” should not count as a “modern science.”  The social sciences have long tried to piggy-back on the success and prestige of the physical sciences, but human beings tend not to behave in the consistent and predictable fashion of chemicals and inert matter.

Still, the article suggests that this has become a problem in other fields as well.  But maybe it is time to give the social sciences another name.  These social “studies” are not without value, but it may be a stretch to consider them “scientific.”

"Once we abandon the notion of an objective reality there are no guard rails or ..."

Monday Miscellany, 3/18/24
"We wonder if we ever encounter alien life, would we be able to understand it ..."

Monday Miscellany, 3/18/24
"Hate speech, jealousy, envy, bitterness, anger, lying, frustration, accusation, blame, and an endless list of ..."

Monday Miscellany, 3/18/24
"Canada is a parasite nation. It hasn't reproduced a native generation in over 40 years. ..."

Monday Miscellany, 3/18/24

Browse Our Archives