Psychology experiments often can’t be replicated

There is currently what is being described as a “crisis” in the field of social/personality psychology.  It turns out, many psychological experiments, however heralded in the media and whose findings are made a big deal of, cannot be replicated by other researchers.  Is that due to fraud?  Statistical quirks? Or does it mean that psychology is not a science after all? [Read more...]

Psychologists admit to bogus research

Social science aspires to the status of natural science, never mind that human beings are not as consistent or predictable as inert matter.  But a new study has found that an alarmingly large percentage of experimental psychologists admit to using questionable, if not bogus, research methods:

Questionable research practices, including testing increasing numbers of participants until a result is found, are the “steroids of scientific competition, artificially enhancing performance”. That’s according to Leslie John and her colleagues who’ve found evidence that such practices are worryingly widespread among US psychologists. The results are currently in press at the journal Psychological Science and they arrive at a time when the psychological community is still reeling from the the fraud of a leading social psychologist in the Netherlands. Psychology is not alone. Previous studies have raised similar concerns about the integrity of medical research.

John’s team quizzed 6,000 academic psychologists in the USA via an anonymous electronic survey about their use of 10 questionable research practices including: failing to report all dependent measures; collecting more data after checking if the results are significant; selectively reporting studies that “worked”; and falsifying data.

As well as declaring their own use of questionable research practices and their defensibility, the participants were also asked to estimate the proportion of other psychologists engaged in those practices, and the proportion of those psychologists who would likely admit to this in a survey.

For the first time in this context, the survey also incorporated an incentive for truth-telling. Some survey respondents were told, truthfully, that a larger charity donation would be made by the researchers if they answered honestly (based on a comparison of a participant’s self-confessed research practices, the average rate of confession, and averaged estimates of such practices by others). Just over two thousand psychologists completed the survey. Comparing psychologists who received the truth incentive vs. those that didn’t showed that it led to higher admission rates.

Averaging across the psychologists’ reports of their own and others’ behaviour, the alarming results suggest that one in ten psychologists has falsified research data, while the majority has: selectively reported studies that “worked” (67 per cent), not reported all dependent measures (74 per cent), continued collecting data to reach a significant result (71 per cent), reported unexpected findings as expected (54 per cent), and excluded data post-hoc (58 per cent). Participants who admitted to more questionable practices tended to claim that they were more defensible. Thirty-five per cent of respondents said they had doubts about the integrity of their own research. Breaking the results down by sub-discipline, relatively higher rates of questionable practice were found among cognitive, neuroscience and social psychologists, with fewer transgressions among clinical psychologists.

via BPS Research Digest: Questionable research practices are rife in psychology, survey suggests.
HT: Joe Carter

More weird science

According to this article in The New Yorker, when scientists replicate an experiment, the results–proven initially–sometimes change with time.  Drugs that at first are shown to be effective often are shown to be ineffective when tested later.   The article cites one experiment whose results varied when it was performed in different locations.  In many different scientific fields, effects  decline with time.

While Karl Popper imagined falsification occurring with a single, definitive experiment—Galileo refuted Aristotelian mechanics in an afternoon—the process turns out to be much messier than that. Many scientific theories continue to be considered true even after failing numerous experimental tests. Verbal overshadowing might exhibit the decline effect, but it remains extensively relied upon within the field. The same holds for any number of phenomena, from the disappearing benefits of second-generation antipsychotics to the weak coupling ratio exhibited by decaying neutrons, which appears to have fallen by more than ten standard deviations between 1969 and 2001. Even the law of gravity hasn’t always been perfect at predicting real-world phenomena. (In one test, physicists measuring gravity by means of deep boreholes in the Nevada desert found a two-and-a-half-per-cent discrepancy between the theoretical predictions and the actual data.) Despite these findings, second-generation antipsychotics are still widely prescribed, and our model of the neutron hasn’t changed. The law of gravity remains the same.

Such anomalies demonstrate the slipperiness of empiricism. Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can’t bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

The decline effect and the scientific method : The New Yorker.

HT:James Kushiner


CLOSE | X

HIDE | X