More weird science

More weird science January 28, 2011

According to this article in The New Yorker, when scientists replicate an experiment, the results–proven initially–sometimes change with time.  Drugs that at first are shown to be effective often are shown to be ineffective when tested later.   The article cites one experiment whose results varied when it was performed in different locations.  In many different scientific fields, effects  decline with time.

While Karl Popper imagined falsification occurring with a single, definitive experiment—Galileo refuted Aristotelian mechanics in an afternoon—the process turns out to be much messier than that. Many scientific theories continue to be considered true even after failing numerous experimental tests. Verbal overshadowing might exhibit the decline effect, but it remains extensively relied upon within the field. The same holds for any number of phenomena, from the disappearing benefits of second-generation antipsychotics to the weak coupling ratio exhibited by decaying neutrons, which appears to have fallen by more than ten standard deviations between 1969 and 2001. Even the law of gravity hasn’t always been perfect at predicting real-world phenomena. (In one test, physicists measuring gravity by means of deep boreholes in the Nevada desert found a two-and-a-half-per-cent discrepancy between the theoretical predictions and the actual data.) Despite these findings, second-generation antipsychotics are still widely prescribed, and our model of the neutron hasn’t changed. The law of gravity remains the same.

Such anomalies demonstrate the slipperiness of empiricism. Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can’t bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

The decline effect and the scientific method : The New Yorker.

HT:James Kushiner

"Oh...Well, I feel much better now."

How the Experts Say We Should ..."
"What, the right to believe just anything? What if we limit it to: you have ..."

How the Experts Say We Should ..."
"And who elects the "fact checkers"?I do. Next question."

How the Experts Say We Should ..."
"Repeal 26th Amendment. We have already backed off on letting teenagers engage in drinking, gambling ..."

How the Experts Say We Should ..."

Browse Our Archives

Follow Us!


TRENDING AT PATHEOS Evangelical
What Are Your Thoughts?leave a comment
  • Carl Vehse

    Following Occam’s Razor, it’s more likely that experimental sloppiness, albeit subtle, is involved as well as sometimes a little dash of what has been called “pathological science.”

    Breakthrough discoveries in science are still made, but with them also come flukes, aberrations, egos, publish-or-perish, politics, media hype, and, unfortunately, some occasional fraud. A few years ago the previously claimed discovery of two elements had to be withdrawn by a world-reknown laboratory when it was found that one of the research team had fudged the data.

    Peer review is supposed to act as a gatekeeper, but does not always work.
    The practice of science is not, well… a perfect science.

  • Carl Vehse

    Following Occam’s Razor, it’s more likely that experimental sloppiness, albeit subtle, is involved as well as sometimes a little dash of what has been called “pathological science.”

    Breakthrough discoveries in science are still made, but with them also come flukes, aberrations, egos, publish-or-perish, politics, media hype, and, unfortunately, some occasional fraud. A few years ago the previously claimed discovery of two elements had to be withdrawn by a world-reknown laboratory when it was found that one of the research team had fudged the data.

    Peer review is supposed to act as a gatekeeper, but does not always work.
    The practice of science is not, well… a perfect science.

  • Two explanations for this;

    1. The statistically significant result in an area may simply be the first result that is actually reported. In other words, people were looking there, but never found anything–we’re more or less building our science on violations of “Western Electric” rules, for the statisticians out there.

    2. Nobody ever gets tenure for choosing the null hypothesis–for saying that we don’t know anything new. You always get promoted for demonstrating (at least well enough to pass peer review) a new idea.

    Combine these two phenomena, and we would expect a decline in the statistical significance of virtually any experimentally derived results. It’s worth noting, by the way, that the law of gravitation was not experimentally derived, but rather mathematically, if I remember correctly. Newton’s Principia is a gorgeous display of geometry applied to science.

  • Two explanations for this;

    1. The statistically significant result in an area may simply be the first result that is actually reported. In other words, people were looking there, but never found anything–we’re more or less building our science on violations of “Western Electric” rules, for the statisticians out there.

    2. Nobody ever gets tenure for choosing the null hypothesis–for saying that we don’t know anything new. You always get promoted for demonstrating (at least well enough to pass peer review) a new idea.

    Combine these two phenomena, and we would expect a decline in the statistical significance of virtually any experimentally derived results. It’s worth noting, by the way, that the law of gravitation was not experimentally derived, but rather mathematically, if I remember correctly. Newton’s Principia is a gorgeous display of geometry applied to science.

  • WebMonk

    This article, in many ways, typifies news-science: grabs an interesting aspect of science and then proceeds to get it almost completely backward when it comes to the details.

    I’m barely familiar with the article’s main story about Schooler’s verbal overshadowing study. But if it is handled as accurately as his statements about the borehole gravity tests in Nevada and the weak coupling in neutrons, then the author has managed to get almost everything wrong.

    Probably he managed the main part of the story better than his side-examples, but how much better I can’t know, so I’m very loath to trust any of it.

  • WebMonk

    This article, in many ways, typifies news-science: grabs an interesting aspect of science and then proceeds to get it almost completely backward when it comes to the details.

    I’m barely familiar with the article’s main story about Schooler’s verbal overshadowing study. But if it is handled as accurately as his statements about the borehole gravity tests in Nevada and the weak coupling in neutrons, then the author has managed to get almost everything wrong.

    Probably he managed the main part of the story better than his side-examples, but how much better I can’t know, so I’m very loath to trust any of it.

  • What Carl and Bike said. And Webmonk.

  • What Carl and Bike said. And Webmonk.

  • sg

    The commenters on this blog have a better grasp of what is going on than this goofy journalist for the New Yorker. Just another attack on reason by an innumerate but verbally facile post modernist.

  • sg

    The commenters on this blog have a better grasp of what is going on than this goofy journalist for the New Yorker. Just another attack on reason by an innumerate but verbally facile post modernist.

  • WebMonk

    sg, you just made my day!

    “Just another attack on reason by an innumerate but verbally facile post modernist.”

  • WebMonk

    sg, you just made my day!

    “Just another attack on reason by an innumerate but verbally facile post modernist.”

  • This is what happens when people stuck in a modern worldview try to understand contemporary science. Our knowledge of light, particle physics, and chaos theory have demonstrated that what we call laws are the peak of a bell-curve. Photons are not described as natural-class particles any more, but as potential energy packets. Atoms are not described as electrons orbiting a nucleus, but as a cloud of “shells” – an area where an “electron” might be at a given time (actually distributed energy, not a particle). Chaos theory tells us that we need at least three differential equations in the imaginary number plane just to describe a leaf falling from a tree (and it took Lorentz six differentials!). The nature of science as descriptively normative has been commonly known among physicists for at least thirty years.

  • This is what happens when people stuck in a modern worldview try to understand contemporary science. Our knowledge of light, particle physics, and chaos theory have demonstrated that what we call laws are the peak of a bell-curve. Photons are not described as natural-class particles any more, but as potential energy packets. Atoms are not described as electrons orbiting a nucleus, but as a cloud of “shells” – an area where an “electron” might be at a given time (actually distributed energy, not a particle). Chaos theory tells us that we need at least three differential equations in the imaginary number plane just to describe a leaf falling from a tree (and it took Lorentz six differentials!). The nature of science as descriptively normative has been commonly known among physicists for at least thirty years.

  • Oh no! I just stood up, and stepped into a 2.5% gravity reduced zone! I can feel my spinal column decompressing as I type!

  • Oh no! I just stood up, and stepped into a 2.5% gravity reduced zone! I can feel my spinal column decompressing as I type!

  • If we appreciate the scientific process aright, we will welcome theses observed discrepancies because they are the currency of a closer approximation of what reality is. This will always be imperfect (discrepancies will always be found) but, if we do our jobs, we will be closer than we were before.

  • If we appreciate the scientific process aright, we will welcome theses observed discrepancies because they are the currency of a closer approximation of what reality is. This will always be imperfect (discrepancies will always be found) but, if we do our jobs, we will be closer than we were before.

  • MikeD

    Please do read the full article. It’s a thing of beauty… an internal critique of the flase religion of scientism. Science is only a godly endeavor as we have the proper persepective that it’s design is for dominion and the betterment of the lives of our neighbor to the glory of God. In other words, in so much as it works for holy purposes… it’s useful, let us thank God for it. But never a means of furnishing universals. This is from Scripture alone, for His word is truth.

  • MikeD

    Please do read the full article. It’s a thing of beauty… an internal critique of the flase religion of scientism. Science is only a godly endeavor as we have the proper persepective that it’s design is for dominion and the betterment of the lives of our neighbor to the glory of God. In other words, in so much as it works for holy purposes… it’s useful, let us thank God for it. But never a means of furnishing universals. This is from Scripture alone, for His word is truth.

  • WebMonk

    Spomer, in the cases about gravity and weak pairing mentioned in the article, the author got it entirely wrong or has totally misrepresented things. You’re right that observed discrepancies are welcome, but in this case, the author didn’t pick any sort of valid discrepancies as those examples.

    MikeD – I can’t understand how you can consider it a beautiful internal critique of the “flase religion of scientism” when it gets at least several major facts completely wrong.

    That said, it does point out some very real problems that plague all types of investigations, and it shows the value of continued testing of various findings to weed out the findings that are actually primarily statistical flukes.

    Fortunately, scientists are aware of this, and the scientific community as a whole takes this into account for new discoveries.

    News writers, though, never take this into account, which is one of the reasons there are so frequently stories about huge new discoveries that will rock the world!!!! but then turn out to not be so dramatic.

  • WebMonk

    Spomer, in the cases about gravity and weak pairing mentioned in the article, the author got it entirely wrong or has totally misrepresented things. You’re right that observed discrepancies are welcome, but in this case, the author didn’t pick any sort of valid discrepancies as those examples.

    MikeD – I can’t understand how you can consider it a beautiful internal critique of the “flase religion of scientism” when it gets at least several major facts completely wrong.

    That said, it does point out some very real problems that plague all types of investigations, and it shows the value of continued testing of various findings to weed out the findings that are actually primarily statistical flukes.

    Fortunately, scientists are aware of this, and the scientific community as a whole takes this into account for new discoveries.

    News writers, though, never take this into account, which is one of the reasons there are so frequently stories about huge new discoveries that will rock the world!!!! but then turn out to not be so dramatic.

  • WebMonk

    One of the other things notable about the article is that all the dramatic declines in discovered effects results were in the biological fields.

    It goes to show that biological systems are extremely complex and have millions of different ways in which results can change, even among what appear to gross observation to be identical tests.

    Now, if you start messing around with really simple systems (at least by comparison) such as individual atoms, electrons, photons, the declining discovery effect should be a LOT smaller. Considering that the author primarily used biology-based studies for examples, and the only to non-biology examples he used to support his view were horribly mangled, I think it would be a safe bet to say biological research is much more susceptible than other types of research.

    That’s hardly a solidly founded guess, but I think it would prove to be accurate if tested.

  • WebMonk

    One of the other things notable about the article is that all the dramatic declines in discovered effects results were in the biological fields.

    It goes to show that biological systems are extremely complex and have millions of different ways in which results can change, even among what appear to gross observation to be identical tests.

    Now, if you start messing around with really simple systems (at least by comparison) such as individual atoms, electrons, photons, the declining discovery effect should be a LOT smaller. Considering that the author primarily used biology-based studies for examples, and the only to non-biology examples he used to support his view were horribly mangled, I think it would be a safe bet to say biological research is much more susceptible than other types of research.

    That’s hardly a solidly founded guess, but I think it would prove to be accurate if tested.

  • Webmonk, I also observed the biological sciences – bias you mention @ #12. I would also like to observe that these changes in results were observed precisely because of the rigours of the scientific method, peer review, and subsequent re-verification efforts.

    But in general, the article had that unmistakeable air of the typical feeble journalisitc atttempt at a profundity about scientific matters well beyond his/her intellectual grasp.

  • Webmonk, I also observed the biological sciences – bias you mention @ #12. I would also like to observe that these changes in results were observed precisely because of the rigours of the scientific method, peer review, and subsequent re-verification efforts.

    But in general, the article had that unmistakeable air of the typical feeble journalisitc atttempt at a profundity about scientific matters well beyond his/her intellectual grasp.

  • Bryan Lindemood

    Having not read the latest scientific journals.

  • Bryan Lindemood

    Having not read the latest scientific journals.

  • WebMonk

    Bryan – ??
    Did your post get cut off or something?

  • WebMonk

    Bryan – ??
    Did your post get cut off or something?

  • I wasn’t really sure what the article meant when it referred to “the weak coupling ratio exhibited by decaying neutrons, which appears to have fallen by more than ten standard deviations between 1969 and 2001” so, doing my thing, I Googled it.

    I found this review of the New Yorker article over at MIT’s Knight Science Journalism Tracker, of which I know nothing, but assume it has some cred just from being at MIT.

    The review touches on what several here have also mentioned, with lines like

    As a physical sciences snob, it went down pretty easy with me to read that in the behavior and medical sciences the reliability of published findings is erratic at best.

    and

    Well! Really? By contrast to the extensive detail he provided from the softer sciences, this is all he has on physics and such. One is left wondering what the ten standard deviations in the fine coupling constant mean in absolute numbers, and what physicists say about changes in understanding of neutron decay physics and whether they are distressing.

    Then there is this amusing update at the bottom of the review:

    Not sure what “weak coupling ratio for neutron decay” means, I sent by email to a theoretical physicist prominent in public discussion and who has his eye on large issues, Arizona State U’s Lawrence Krauss, the graf in the New Yorker on that and gravity measurements.

    Here is a slightly amended rendition of his reply:

    “The physics references are (deposit scatological bovine expletive here) … the neutron data have fallen, reflecting under-estimation of errors, but the lower lifetime doesn’t change anything having to do with the model of the neutron, which is well understood and robust … And as for discrepancies with gravity, the deep borehole stuff is interesting but highly suspect. Moreover, all theories conflict with some experiments, because not all experiments are right.” / LMK

    FYI.

  • I wasn’t really sure what the article meant when it referred to “the weak coupling ratio exhibited by decaying neutrons, which appears to have fallen by more than ten standard deviations between 1969 and 2001” so, doing my thing, I Googled it.

    I found this review of the New Yorker article over at MIT’s Knight Science Journalism Tracker, of which I know nothing, but assume it has some cred just from being at MIT.

    The review touches on what several here have also mentioned, with lines like

    As a physical sciences snob, it went down pretty easy with me to read that in the behavior and medical sciences the reliability of published findings is erratic at best.

    and

    Well! Really? By contrast to the extensive detail he provided from the softer sciences, this is all he has on physics and such. One is left wondering what the ten standard deviations in the fine coupling constant mean in absolute numbers, and what physicists say about changes in understanding of neutron decay physics and whether they are distressing.

    Then there is this amusing update at the bottom of the review:

    Not sure what “weak coupling ratio for neutron decay” means, I sent by email to a theoretical physicist prominent in public discussion and who has his eye on large issues, Arizona State U’s Lawrence Krauss, the graf in the New Yorker on that and gravity measurements.

    Here is a slightly amended rendition of his reply:

    “The physics references are (deposit scatological bovine expletive here) … the neutron data have fallen, reflecting under-estimation of errors, but the lower lifetime doesn’t change anything having to do with the model of the neutron, which is well understood and robust … And as for discrepancies with gravity, the deep borehole stuff is interesting but highly suspect. Moreover, all theories conflict with some experiments, because not all experiments are right.” / LMK

    FYI.

  • SKPeterson

    Ah, yes, the “hard” sciences physics snobs. Very true. They think they’re the only “real” scientists. Which is fine, they have it easy. The rest of us scientists have to deal with the messy realities of the world, especially when you deal with biological and human systems. Such things often don’t lend themselves to easy measurement or observation. Biological organisms, especially humans, exhibit a wide variety of behaviors and adaptations to constantly changing environmental stimuli. As a result, the ability to isolate and measure cause and effect becomes difficult.

    However, that does not make biological or human sciences any less rigorous methodologically than the hard sciences, but that there are things observed in Nature that cannot be replicated in a controlled laboratory. Here, Popper and logical positivism have not done the sciences any great favors.

    Moreover, this constant physics-envy that has grabbed hold of the other sciences does not advance science – it cripples it. Instead we get biological scientists who try to make hard science claims for their findings, which cannot be supported by the available data, or by the rigors of replication in a laboratory setting. Economics is the worst of these sciences – it has capitulated wholesale to a scientism of physics worship; there is no physical science model that won’t be applied in some slipshod manner to identify and measure “constants” in human/economic behavior. That’s bad enough, but becomes particularly dangerous when these constants are held to be actually constant and policies derived thereby. Inevitably, the constants are not constant and the assertions, models, conclusions fail or are found wanting. In the aftermath, the biological sciences, the human sciences, and other scientific fields become increasingly discredited and valid scientific inquiry in these fields subject to extremes of a postmodern “post-scientific” scientism, or to an increasingly rigid clinging to the hard-science paradigm. Either way, not good.

  • SKPeterson

    Ah, yes, the “hard” sciences physics snobs. Very true. They think they’re the only “real” scientists. Which is fine, they have it easy. The rest of us scientists have to deal with the messy realities of the world, especially when you deal with biological and human systems. Such things often don’t lend themselves to easy measurement or observation. Biological organisms, especially humans, exhibit a wide variety of behaviors and adaptations to constantly changing environmental stimuli. As a result, the ability to isolate and measure cause and effect becomes difficult.

    However, that does not make biological or human sciences any less rigorous methodologically than the hard sciences, but that there are things observed in Nature that cannot be replicated in a controlled laboratory. Here, Popper and logical positivism have not done the sciences any great favors.

    Moreover, this constant physics-envy that has grabbed hold of the other sciences does not advance science – it cripples it. Instead we get biological scientists who try to make hard science claims for their findings, which cannot be supported by the available data, or by the rigors of replication in a laboratory setting. Economics is the worst of these sciences – it has capitulated wholesale to a scientism of physics worship; there is no physical science model that won’t be applied in some slipshod manner to identify and measure “constants” in human/economic behavior. That’s bad enough, but becomes particularly dangerous when these constants are held to be actually constant and policies derived thereby. Inevitably, the constants are not constant and the assertions, models, conclusions fail or are found wanting. In the aftermath, the biological sciences, the human sciences, and other scientific fields become increasingly discredited and valid scientific inquiry in these fields subject to extremes of a postmodern “post-scientific” scientism, or to an increasingly rigid clinging to the hard-science paradigm. Either way, not good.

  • WebMonk

    Biology usually is considered a “hard science”, along with genetics, and things like that. The “soft science” fields are things like anthropology, psychology, history (kind of), etc. Your entire premise in 17 is based on the false categorization of biology in the “soft sciences”.

  • WebMonk

    Biology usually is considered a “hard science”, along with genetics, and things like that. The “soft science” fields are things like anthropology, psychology, history (kind of), etc. Your entire premise in 17 is based on the false categorization of biology in the “soft sciences”.

  • SKPeterson

    WebMonk – if it came off that way – I missed. I was thinking in terms of how physicists view all other sciences. For many of them, biology and genetics are also “soft” as they can’t always lend themselves to the experimental rigors of physics, or use the right math. Biologists and geneticists have nothing in their scientific pursuits to be ashamed of. It is just more likely they will be dealing with natural conditions that can skew, bias or make contingent their findings. These are realities that are often missing from the experimental reality of physics.

  • SKPeterson

    WebMonk – if it came off that way – I missed. I was thinking in terms of how physicists view all other sciences. For many of them, biology and genetics are also “soft” as they can’t always lend themselves to the experimental rigors of physics, or use the right math. Biologists and geneticists have nothing in their scientific pursuits to be ashamed of. It is just more likely they will be dealing with natural conditions that can skew, bias or make contingent their findings. These are realities that are often missing from the experimental reality of physics.

  • sg

    “As a result, the ability to isolate and measure cause and effect becomes difficult.”

    This seems the main problem. It is much harder to isolate variables in life science than in physics, etc. Those slippery critters change what they are doing.

  • sg

    “As a result, the ability to isolate and measure cause and effect becomes difficult.”

    This seems the main problem. It is much harder to isolate variables in life science than in physics, etc. Those slippery critters change what they are doing.

  • Anybody with any understanding of the subtlety of science is going to look at that article and go “yeah, so what?” If everybody approached science with the grade school understanding the excerpt appears to have they are going to be thrown by the idea “repeating an experiment didn’t yield identical result, oh noes the sky is falling!” The variables involved in any setting are beyond imagination, even though labs try to carefully control all known variables variation exists. It is one of the reasons why in the lab you measure everything from humidity to ambient light.

    Maybe it is my background or maybe it is my perspective in life with science as a servant of the Gospel, but I just don’t see anything earth shattering or requiring a “decision” in this article. But then this is why we have peer review, to analyze methods, results and conclusions.

  • Anybody with any understanding of the subtlety of science is going to look at that article and go “yeah, so what?” If everybody approached science with the grade school understanding the excerpt appears to have they are going to be thrown by the idea “repeating an experiment didn’t yield identical result, oh noes the sky is falling!” The variables involved in any setting are beyond imagination, even though labs try to carefully control all known variables variation exists. It is one of the reasons why in the lab you measure everything from humidity to ambient light.

    Maybe it is my background or maybe it is my perspective in life with science as a servant of the Gospel, but I just don’t see anything earth shattering or requiring a “decision” in this article. But then this is why we have peer review, to analyze methods, results and conclusions.

  • Maybe getreligion.org needs a sister site, getscience.org. The media just doesn’t get science.

  • Maybe getreligion.org needs a sister site, getscience.org. The media just doesn’t get science.