Scientific Consensus and Probability

Scientific Consensus and Probability September 14, 2017

A recent thread erupted into a debate about the value of scientific consensus.

Here are some of the comments from said piece:

Scientists are human. They are swayed by those under whom they trained, politics, funding, etc. Consensus means nothing, even if it is consensus among experts. One expert position carries the same weight as 10 or 10,000. What matters is how the data relates to the theory.


There is no reason why consensus among scientists reasonably matches up to robustness of scientific theory. Until scientifically robust theory matching the two is provided, I dismiss the validity of consensus.

I like comments like this because they challenge assumptions and make you think a little about whether assumptions are justifiable. Fellow ATPer Jeremiah Traeger beat me to it in producing a good riposte to these claims. However, I want to add a little bit more.

This is not uncommon debate either (especially when concerning climate change). Here is what Futurism has to say about it:

This, however, brings into question the validity of a scientific consensus. How sure are we the consensus is indeed, scientific?

It’s always useful to remember that, first, a scientific paper is not a run-of-the-mill paper. It is a paper that is reviewed by a host of individuals in the community who scrutinize its limitations, its experimental processes, and its findings. Only after the paper has been through this peer review process is it accepted for publication. And each new paper builds on the information disclosed in the papers that came before.

Hence, the birth of a scientific consensus isn’t subject to a majoritarian rule. It actually signifies the fact that a great many scientists from different backgrounds have considered the question at hand and have reached similar conclusions.

That doesn’t mean that science is a panacea—it doesn’t mean that science is perfect or always 100% correct. It is important to remember that science is adaption; it’s change. But what it does mean is that we have a pretty good understanding of how things work, and it will take a mammoth amount of evidence to change our current understanding.

Moving slightly more closely to the area of probability, Climate Science Investigations has this to add:

To summarize, the pursuit of science focuses on establishing probability rather than certainty. Scientists accept the fact that they are not all knowing and must remain objective and open to other possibilities when conducting scientific research.

However, through the use of consensus-building activities — challenging others’ ideas, reexamining and retesting data, critiquing others’ work through the peer-review process — scientists have been able to build a body of knowledge about which we can be reasonably confident. Still, there are scientific mysteries that are not yet well understood; conclusions about those are likely to have a much lower probability of accuracy.

Science is, therefore, simultaneously durable enough to provide a reasonable basis for scientists to make logical conclusions about the world in which we live and still flexible enough to be revised and improved when newer, better evidence and findings are discovered.

For me, this is about probability. I know that humans are biased, and lay people in particular. scientists are biased too, to varying degrees. They are humans after all. But the important thing is that the scientific method is “designed” in such a way as to mitigate such bias. The whole process means that ideas are there to be torn down and should rest or fall on their merits. The process is a way of dealing with the vagaries of humans and the fact that scientists are human.

Yes, there might be political influences.

Yes, there might be perseverance biases.

Yes, there might be any number of other factors that can lead the journey towards truth momentarily astray.

The key is that this is in the word momentarily. The scientific process will eventually find such biases out and correct. After all, the power of the scientific process is that it is self-correcting. This is what separates it from, say, religious epistemology. There is no self-correcting mechanism built into religion. Indeed, it is rather dependant upon not changing or adapting. A holy book is often supposedly immutable, or in some way holding durable truth. It often sits there staunchly unchanging.

Each scientist, though human, is contributing towards this journey towards truth. It is important to note that they are (or should be) experts in their fields. These aren’t just any old humans but are trained, knowledgeable intellects who are conducting research that seeks to find explanations for phenomena.

If we take the probability of just one of these concluding correctly (in an objective sense), given the expertise that they have, we might assign any non-zero percentage. But if you look at one thousand such scientists in the same field, and if 999 of them agree, then the probability of them being right is somewhat higher than the reverse.

Yes, they might all be basing their work on an incorrect premise and thus be in some way incorrect (as I am sure has happened in the past, and might in some contexts be the case not, perhaps with hidden variable theories). But the nature of the process is that this will eventually be corrected, in all probability.

I suppose it comes down to whether you think an informed scientist in the relevant field has a greater or lesser than 50% chance of being right and how this probability shifts, the more scientists you add into the mix.

If I look at the field of scientists in a particular discipline or area and see an overwhelming consensus, what has a greater chance of being the case? That the vast majority are wrong, or that the small minority is wrong? I can understand a prima facie appraisal in saying, “The number of people believing something does not mean it has a greater chance of being true.”

This is correct. Just look at religion, or the flat earth in ancient times. Or any other such belief. But these are a different category of belief than a scientific belief based on tested and retested and challenged and re-challenged evidence.

I think this category error is what lies at the heart of the basis of criticism for consensus.

After all, the probability of that many people being wrong after all of that training, peer-review, testing, retesting and so on is surely lower than them being right. It’s rather like conspiracy theories, in this respect. The probability of that many people lying and being able to keep it quiet… You get the point.

And this is what makes comments like this so naive:

Real science doesn’t, or shouldn’t, give a damn about consensus.
Science is about truth-seeking.
Consensus is about popularity, and politics.

They might have the soundbite appeal but they don’t stand up to scrutiny. Consensus isn’t about popularity. Just because a position is held by a majority does not mean it is about popularity. These findings still have to be based on firm evidential bases.

Goodness, if the Daily Mail reports on ExxonMobil knowing the truth about climate change but covering up for financial gain, then you know they might be onto something. This is where politics and economics get in the way of seeking truth. But people like this previous commenter will suck it up. I would prefer to err on the side of the experts, especially if I am not an expert in the given field. As the DM stated recently:

US oil giant ExxonMobil knowingly misled the public for decades about the danger climate change poses to a warming world and the company’s long-term viability, according to a peer-reviewed study released Wednesday.

An analysis of nearly 200 documents spanning decades found that four-fifths of scientific studies and internal memos acknowledged global warming is real and caused by humans.

At the same time a similar proportion of hundreds of paid editorials in major US newspapers over the same period cast deep doubt on these widely accepted facts.

The study also cites ExxonMobil calculations that capping global warming at under two degrees Celsius (3.6 degrees Fahrenheit) — the goal enshrined in the landmark Paris climate accord — would impose sharp limits on the amount of fossil fuels that could be burned, and thus potentially affect the firm’s growth.

Both findings are relevant to ongoing investigations by state and federal attorneys general, along with the Securities and Exchange Commission, on whether the company deceived investors on how it accounts for climate change risk.

The take away point is this:

By discounting the consensus, you are either saying the minority has a greater chance of being right (in which case you have a terribly hypocritical approach that fails in shooting itself massively in the foot), or you are saying they both have equal probability.

The first position is nonsensical, primarily for the reasoning the denier gives in disputing consensus. They are claiming a particular number of people has no bearing on the truth, and then opt for the truth being more likely with a particular number (the smaller one).

More likely, though, is the claim that neither side has a greater probability for owning the truth, so to speak, and that it is only the content of the beliefs that should be evaluated.

But if an evaluation is to be valued at all, then surely more evaluations by a greater number of expert people will be more accurate than a single evaluation (by the commenter alone?)? That is how statistics usually works. We don’t poll just one person and pin our hopes on them.

Probability favours the consensus, here. It might not always work. But I would rather rely on probability than throw my hands up in the air and say, “Climate change? No idea. And I don’t trust any one else, either.”

What makes matters worse is that unless this commenter does all the research and experimentation himself, then he has to trust the work of others, and work in a collegiate manner. He has to trust the evaluations and the data of others. This is absolutely pragmatically necessary. Otherwise, he might as well give up on anything any other human is telling him, because what this approach indicates is that no other human is prima facie trustworthy and that all data and evaluations should be done personally.

How does he know cyanide is poisonous. A consensus of scientists believes this is the case. But he doesn’t have the time to read up and research this and to do hundreds of years of work just on this one claim. Because he would have to carry out and evaluate all evidence that this claim is itself based on. It’s turtles all the way down to the very first piece of observational and scientific data.

This is wildly stupid. Thus we need shortcuts. And the shortcut is? Relying on the claims of consensus experts.

I’ll take the advice of the health notice on the bottle of bleach, derived from scientific consensus, and not add it to my tea, thank you very much.

Browse Our Archives