How should skeptics view qualitative research?

A phrase I hear a lot from skeptics is “The plural of anecdote is not data”. I first heard it from my partner’s sister, and then in Ben Goldacre’s Bad Science. I laughed out loud the first time I heard it, and I’ve used it many times since.

It’s rubbish.

The usual way the phrase is deployed is like this:

Skeptic: There’s no evidence that homeopathy can cure eczema.

Woo Badger: Well you say that, but my friend’s nan went to a homeopathist and after that her symptoms cleared right up!

Skeptic: Yes, but the plural of anecdote is not data!

I am on the skeptic’s side here. But what the person means is that anecdotes are not a good way of establishing the medical effectiveness of treatments. Anecdotes absolutely can be data if you have a different research question. If the question was “What are some of the ways homeopathy users justify their belief in its effectiveness?” then the above would be a useful piece of data.

I’m interested in this because I’ve just completed a qualitative social science PhD, and in doing it I had to think quite a lot about research methods and what they are useful for. Early in my studies, I learned about the ‘paradigm wars’, fierce academic battles over the respective merits of qualitative and quantitative research, with each side attempting to discredit the other. As someone who didn’t live through those wars (at least not as a participant), I couldn’t comprehend them at first. It seemed obvious to me that these paradigms could complement each other, and were suited to different types of questions. (That’s still broadly my position, by the way).

Scientist

Scientist using a glovebox. Photo: O. Usher (UCL MAPS), Flickr.

What I hadn’t realised, which qualitative-advocates argue very well, is how quantitative research has a history of erasing minorities. There is a risk of “Well that’s only 10% of people, which is a small and unimportant number” leading to treating white, heterosexual, cisgender experiences as the only ones that count. It’s a shortcoming that qualitative research is well positioned to correct.

Most skeptics-with-a-k, skeptics of the variety who attend Skeptics in the Pub, tend to take a high view of quantitative studies, regarding these as the most scientific. I worry that we tend not to be quite skeptical enough about quantitative social research. During the paradigm wars, qualitative researchers advanced some extremely powerful critiques of quantitative research, which threatened the foundations it’s built on. These researchers did less well at solving the inherent difficulties with qualitative research. Nevertheless, what is now quite well established is that questionnaire surveys are excellent at telling us how people answer questionnaires. The assumption that these answers represent the reality about these people’s underlying beliefs and attitudes (or that these attitudes are stable over time) is much more in doubt.

Specific critiques of quantitative research are quite interesting. Sadly now I’ve completed I no longer have access to a university library, but if there’s enough interest in this I might try to expand on them in a future post. I rarely hear Skeptics discuss these critiques.

Anyway, there’s my first point: skeptics are sometimes insufficiently skeptical about quantitative research.

Onto the next thing: how should skeptics respond to qualitative research? Or, more specifically, how should skeptics respond to my PhD?

I’ve given talks on Accelerated Christian Education (ACE), my PhD topic, to Skeptics in the Pub groups around the country, and interestingly I’ve never come up against “Ah! But the plural of anecdote is not data!” in response to my own anecdotes. I’m curious about why that is. Is it because I’m talking about creationist schools, and skeptics are all so biased against creationism that they’re ready to believe whatever I tell them? If so, we are all less good at skepticism than we like to think. More optimistically, perhaps it’s because they recognise that there is value in firsthand accounts of experience, even given all we know about the vagaries of eyewitness testimony. Let’s hope it’s that one.

Lots of people favour mixed-methods research (ie qualitative and quantitative) to get around the pitfalls of each, and there’s a lot to be said for that. Unfortunately, it’s expensive and time-consuming, and generates so much data that it can be difficult to analyse. In my case, though, there’s a further problem. I don’t think there could ever be a high-quality quantitative study of a random, representative sample of ex-ACE students, for two reasons: 1) Nobody would fund it, and 2) even with the funding, I don’t think it would be possible to get a large random sample of graduates to participate.

This means that a skeptic can look at my research and say “Ah, but we just can’t know whether these findings are representative of ACE students more widely.” If you did say that, you would have a point. I’m not sure you would say that if you actually read it, mind, but you definitely could. The reason you wouldn’t say it, though, is that I think your conclusion is more likely to be “It doesn’t matter if these are the only people who had experiences like this. This was unacceptable.”

“But qualitative research is biased!”

Yup. But it’s a mistake to think that the natural sciences aren’t. Science is affected by the assumptions of those doing it, and historically, too many of those people have been sexist. See also: scientific racism. As Stephen Jay Gould wrote in The Mismeasure of Man, “it is obviously not accidental that a nation still practicing slavery and expelling its aboriginal inhabitants from their homelands should have provided a base for theories that blacks and Indians are separate species, inferior to whites.”

Not only has some quite bad science been the result of bias, but the science that corrected it often has too. We now know that women and black people are not biologically inferior thanks to the work of feminists and anti-racists. The idea that ideological commitments necessarily result in weak research is plain wrong.

My research touches on various types of abuse. I refer to sexual assault and a case in which a child was beaten to death. What, exactly, would a neutral account of those things look like, and would it be ‘better’ than the one I gave?

“Skeptics should be be empiricists”

I saw this objection raised on Facebook in a group I am not part of, and so couldn’t reply. Empiricists would have us believe that the only stuff we can know is the stuff we can observe. As a result, they would exclude anything not directly observable from science. In psychology, that gives you behaviorism. The trouble with behaviorism is that it has been much less successful than cognitive psychology in explaining human behaviour.

Example: Leon Festinger’s first cognitive dissonance experiment. Festinger gave different groups either large or small rewards for completing a boring task. According to (empiricist) behaviorism, the large-reward group should feel better about the task. That’s the opposite of what happened. You can’t explain that finding without referring to people’s (unobservable) private thought processes. Empiricism fails, and this means that skeptics should not be empiricists.

I don’t have a grand conclusion. Social research methodology is still a contested area and we are a long way from consensus on a single best way to do things (if such a thing is even desirable). Skeptics should inform themselves on what the strengths are of the different research schools. Whatever methodology is used, all that matters is that the evidence is strong enough to support whatever claims are being made.

Patheos Atheist LogoLike Leaving Fundamentalism and Patheos Atheist on Facebook!

About Jonny Scaramanga