Psychologists admit to bogus research

Social science aspires to the status of natural science, never mind that human beings are not as consistent or predictable as inert matter.  But a new study has found that an alarmingly large percentage of experimental psychologists admit to using questionable, if not bogus, research methods:

Questionable research practices, including testing increasing numbers of participants until a result is found, are the “steroids of scientific competition, artificially enhancing performance”. That’s according to Leslie John and her colleagues who’ve found evidence that such practices are worryingly widespread among US psychologists. The results are currently in press at the journal Psychological Science and they arrive at a time when the psychological community is still reeling from the the fraud of a leading social psychologist in the Netherlands. Psychology is not alone. Previous studies have raised similar concerns about the integrity of medical research.

John’s team quizzed 6,000 academic psychologists in the USA via an anonymous electronic survey about their use of 10 questionable research practices including: failing to report all dependent measures; collecting more data after checking if the results are significant; selectively reporting studies that “worked”; and falsifying data.

As well as declaring their own use of questionable research practices and their defensibility, the participants were also asked to estimate the proportion of other psychologists engaged in those practices, and the proportion of those psychologists who would likely admit to this in a survey.

For the first time in this context, the survey also incorporated an incentive for truth-telling. Some survey respondents were told, truthfully, that a larger charity donation would be made by the researchers if they answered honestly (based on a comparison of a participant’s self-confessed research practices, the average rate of confession, and averaged estimates of such practices by others). Just over two thousand psychologists completed the survey. Comparing psychologists who received the truth incentive vs. those that didn’t showed that it led to higher admission rates.

Averaging across the psychologists’ reports of their own and others’ behaviour, the alarming results suggest that one in ten psychologists has falsified research data, while the majority has: selectively reported studies that “worked” (67 per cent), not reported all dependent measures (74 per cent), continued collecting data to reach a significant result (71 per cent), reported unexpected findings as expected (54 per cent), and excluded data post-hoc (58 per cent). Participants who admitted to more questionable practices tended to claim that they were more defensible. Thirty-five per cent of respondents said they had doubts about the integrity of their own research. Breaking the results down by sub-discipline, relatively higher rates of questionable practice were found among cognitive, neuroscience and social psychologists, with fewer transgressions among clinical psychologists.

via BPS Research Digest: Questionable research practices are rife in psychology, survey suggests.
HT: Joe Carter

About Gene Veith

Professor of Literature at Patrick Henry College, the Director of the Cranach Institute at Concordia Theological Seminary, a columnist for World Magazine and TableTalk, and the author of 18 books on different facets of Christianity & Culture.

  • Carl Vehse

    In addition to discussing the to-be-published Psychological Science article, “Measuring the prevalence of questionable research practices with incentives for truth-telling,” by Leslie John, George Loewentstein, and Drazen Prelec, the linked Research Digest article also refers to recently published article, “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” by Joseph P. Simmons, Leif D. Nelson and Uri Simonsohn (Psychological Science 2011 22: 1359-1366, originally published online 17 October 2011).

    This article claims, in its abstract:

    1. “In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not.” (A variation of Maslow’s Law of Instruments: If all you have is a hammer, everything starts looking like a nail.)

    2. “[W]e suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.”

    The authors claim their findings apply to both frequentist and Bayesian approaches. However, one of the authors of the to-be-published article, Drazen Prelec, published an earlier article, “A Bayesian Truth Serum for Subjective Data” (Science 306, October 15, 2004, 462-466), in which Prelec presents a Bayesian-based “method of eliciting subjective information, designed for situations where objective truth is intrinsically or practically unknowable.”

  • Carl Vehse

    In addition to discussing the to-be-published Psychological Science article, “Measuring the prevalence of questionable research practices with incentives for truth-telling,” by Leslie John, George Loewentstein, and Drazen Prelec, the linked Research Digest article also refers to recently published article, “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,” by Joseph P. Simmons, Leif D. Nelson and Uri Simonsohn (Psychological Science 2011 22: 1359-1366, originally published online 17 October 2011).

    This article claims, in its abstract:

    1. “In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not.” (A variation of Maslow’s Law of Instruments: If all you have is a hammer, everything starts looking like a nail.)

    2. “[W]e suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.”

    The authors claim their findings apply to both frequentist and Bayesian approaches. However, one of the authors of the to-be-published article, Drazen Prelec, published an earlier article, “A Bayesian Truth Serum for Subjective Data” (Science 306, October 15, 2004, 462-466), in which Prelec presents a Bayesian-based “method of eliciting subjective information, designed for situations where objective truth is intrinsically or practically unknowable.”

  • Tom Hering

    Drazen Prelec? Sounds like a combination of two psychotropic drugs. :-D

  • Tom Hering

    Drazen Prelec? Sounds like a combination of two psychotropic drugs. :-D

  • SKPeterson

    It is like the law of precise imprecision in numbers. If I say “oh, about 40 percent” in answering a query about some subject, I immediately call my results into question, but if I say “40.37%” then I immediately obtain assent to my results, even if I’ve completely made them up. So, if I use an undoubtedly scientific name like “Drazen Prelec” I lull readers into assuming my results are unquestionable.

    - Drazen “SKPeterson” Prelec

    Never question the Drazen

  • SKPeterson

    It is like the law of precise imprecision in numbers. If I say “oh, about 40 percent” in answering a query about some subject, I immediately call my results into question, but if I say “40.37%” then I immediately obtain assent to my results, even if I’ve completely made them up. So, if I use an undoubtedly scientific name like “Drazen Prelec” I lull readers into assuming my results are unquestionable.

    - Drazen “SKPeterson” Prelec

    Never question the Drazen

  • Carl Vehse
  • Carl Vehse
  • Carl Vehse

    Dilbert has previously covered the subject of accurate numbers.

  • Carl Vehse

    Dilbert has previously covered the subject of accurate numbers.

  • DonS

    A topic of discussion yesterday concerned how research in the hard sciences can be impacted by politics, and the social sciences are certainly also, probably to a greater extent, influenced by the fact that funding, whether private or public, usually is granted with at least an implicit directive to find a certain predetermined result.

    In a world of situational ethics, integrity is a rare commodity.

  • DonS

    A topic of discussion yesterday concerned how research in the hard sciences can be impacted by politics, and the social sciences are certainly also, probably to a greater extent, influenced by the fact that funding, whether private or public, usually is granted with at least an implicit directive to find a certain predetermined result.

    In a world of situational ethics, integrity is a rare commodity.

  • steve

    Interesting that after the “shocking” revelations about Diederik Stapel “changed” our perceptions about the veracity of academic research, this study comes out bolstering our new cynical paradigm. What I mean to say is, what if the researchers in this new study fell into the same traps as the subjects of their research? How much faith do we have in any of it? We’re also left to do the same thing as they did: if something sounds good and fits within our own paradigm then we believe it and use it as one of the building blocks for our existing belief structure. In the end, we all live in an indestructible house of cards.

  • steve

    Interesting that after the “shocking” revelations about Diederik Stapel “changed” our perceptions about the veracity of academic research, this study comes out bolstering our new cynical paradigm. What I mean to say is, what if the researchers in this new study fell into the same traps as the subjects of their research? How much faith do we have in any of it? We’re also left to do the same thing as they did: if something sounds good and fits within our own paradigm then we believe it and use it as one of the building blocks for our existing belief structure. In the end, we all live in an indestructible house of cards.


CLOSE | X

HIDE | X