It’s All In How You Ask the Questions (LSP #32)

It’s All In How You Ask the Questions (LSP #32) March 5, 2018

Most folks know that exactly how we ask questions shapes, in great part, how those questions are answered. That might go double when it comes to controversial topics! A survey released last year might shed some interesting light on how accurate surveyed responses about atheism are, and give us some new directions in finding out exactly how many atheists exist in America. We might have been counting wrong all along, it seems–and that’s both good and bad news in its way. Today, Lord Snow Presides over religious surveys: how they’re given, how they’re taken, and how they’re interpreted.

“Stańczyk during a ball at the court of Queen Bona in the face of the loss of Smolensk,” 1862. (By Jan Matejko by way of Wikimedia.) Stańczyk was a court jester in the sixteenth century and remains a popular figure aroundabouts Poland. Compare and contrast.

Hiding in Plain Sight.

I ran across this study from last May thanks to 538.com. The study itself is located here. It’s by Will Gervais, an Assistant Professor with the University of Kentucky (and no relation to Ricky Gervais), and Maxine Najle, a doctoral student of his. It’s called “How many atheists are there?”

The main idea of the paper is that because of the stigmatization of atheism, it’s entirely possible that many atheists don’t use the label–even when answering private questions, like the ones posed regularly to Americans by the likes of Gallup and Pew. However, at the same time the rates of Nones have absolutely soared, which leads us to think that atheism should be rising as well–and yet it isn’t rising as quickly, except among the youngest Americans.

These researchers sought, therefore, to create a series of questions about atheism that would be perceived as less threatening by respondents, and thus elicit more accurate answers than perhaps surveys are getting on that topic.

And I think there’s something to that idea. Non-believers are absolutely in the right in taking care about how and when they reveal their non-belief–and in what company. We’ve long known that atheism is seen as dangerous, even anti-American by a lot of people here–especially if they’re fervent Christians. That antipathy is part of Christianity’s general culture war, with leaders often callously (and hypocritically, not that they care in the least) drumming up adherents’ fear and dehumanization of atheists and pushing a variety of false ideas about what non-belief entails. Just being reminded that non-believers exist can send fundagelicals spiraling down into bouts of pearl-clutching and full-throated denunciations.

That stigma might be easing in recent years, but anybody who isn’t Christian knows that sometimes it’s just not safe to talk about their non-belief in a country still dominated by increasingly-polarized, increasingly-unstable, increasingly-belligerent TRUE CHRISTIANS™ all eager and ready to dole out some retaliatory “Christian love” to those who defy the controlling arm of their tribe.

Asking the Right Questions.

This paper’s authors wondered if the real number of atheists is being accurately counted by survey groups, and if maybe the way we ask questions on these surveys is skewing the results we get on them.

And here, too, I think that’s a damned fine question to ask. We’ve gotten some striking confirmation lately of the need to take extreme care in crafting survey questions.

In 2013, the NCSE (a science-teachers’ group ferociously opposed to Creationism) noticed that how researchers pose questions to survey respondents can have a huge impact on the number of people claiming to be Creationists. When they got away from the label itself and asked questions about the age of the universe and whatnot in a way that allowed people to think about the question away from the label of Creationism, they got a much more encouraging count of people who have a more reality-based view of the real world and universe.

In the same fashion, Vox recently highlighted the fact that how we ask questions on surveys gets us markedly different results regarding support for abortion rights. Employing well-known labels like pro-life and pro-choice and using recognizable talking points from either side of the debate makes people’s minds align with those talking points and labels–and they give responses based upon that alignment. Moving away from either the labels or the recognizable talking points gives us what may be a much more accurate sign of just how many people support and oppose this essential human right.

This whole tendency to align ourselves in particular ways and then answer questions that way shows up as one major criticism of personality inventories like the MBTI, that Myers-Briggs test. One’s assigned “type” will often vary depending on the mindset and mood in which the test is taken, which are factors that can change dramatically–and with those changes, the results achieved by the test change in turn.1

In terms of religion surveys, too, we’ve known for a while that when we ask people up-front about stuff relating to religion, like how often they attend church, we get a lot of aspirational and people-pleasing answers. One link we share a lot around here comes from ChurchLeaders.com, which tells us “7 Startling Facts” about church attendance–with the main one being that real church attendance is very much bad compared with reported church attendance. (Whoopsiedoodlenoodle! That had to smart.)

It Matters How We Ask Questions.

How we ask questions matters enormously if we want to get accurate answers from people. A number of social pressures and cognitive biases are in play in these surveys, and if researchers aren’t aware of those things and work with them, they’re going to get answers based on how people want to be seen or what they think it’s safest to say. Hell, even researchers who are aware of this stuff can be fouled up by it.

In the study we’re talking about today, 2000 people received a list of affirmations for them to agree or disagree with as part of the unmatched count technique. This questioning technique is used to assess other potentially controversial opinions, as well. The idea is that people will feel safer in affirming a controversial question in a list when it’s lumped in with everything else as a total count. It’s more indirect, and thus less threatening.

In keeping with the technique, the list contained a bunch of innocuous affirmations like “I own a dog” and “I am a vegetarian.” The lists were similar for both groups, except that the target (non-control) group got one extra question about whether or not they believed in God. The control and target groups were both asked to count up and write down how many of the listed affirmations in total were true for them. The folks running the test figured that they’d have equal numbers of dog owners and vegetarians in each group, so if the counts differed that would be a sign of the atheists peeking back at them.

When the researchers asked the question like this, they got ranges of atheism that went from 20% of respondents to 32% of them. They did an aggregate analysis that led them to think that about 26% of their total respondents were atheists.2 In their abstract, they mentioned the findings of the big-name survey houses: 3-11% of Americans are atheists. We know that much; we’ve talked about it around here and other confirming studies besides. But they think that’s way too low now:

Our data and model predict that atheist prevalence exceeds 11% with greater than .99 probability, and exceeds 20% with roughly .8 probability. Prevalence estimates of 11% were even less credible than estimates of 40%, and all intermediate estimates were more credible.

That’s some eye-opening stuff!

Now, it’s possible that this larger estimation is going too far. Greg Smith, Pew Research’s religious polling director, isn’t quite sure he’s ready to jump onto this new estimate. Ironically, he wonders if the way they phrased their questions necessarily accomplishes what they think it does. Plus, the line of questioning created here does introduce the idea of belief or nonbelief being a binary yes/no question, which we already know introduces some problems of its own. (Gervais and Najle did replicate their results from the unmatched count technique, though there is a ways to go yet–as we’d expect with anything new under the scientific method.)

I find this whole line of thinking fascinating, especially as researchers become more and more interested in gauging exactly where Americans are on the whole religion question–and as surveys themselves become more common. I need more replication and confirmation of it before I go all in on this new figure, but it’s an interesting new wrinkle in the topic of evangelical churn–not to mention a new scrutiny laid upon surveys themselves.

Whether it’s replicated and accepted or ultimately rejected, this study and the research it generates will increase human knowledge either way–which is a feat that the scientific method has accomplished far more often and far more reliably than religion’s ever managed! HEY-oh!

Next time, we take a look at Mike Huckabee’s latest tempest-in-a-teapot. Oh, it is cringeworthy. Oh, it is very much bad. My goodness. We’ll see you next time!


1 I’ve had that exact discovery myself. I’ve turned up all over the map of the MBTI types. I still couldn’t really tell you which of the 16 of them fits me best overall. (Some people have much better luck with it than others, I guess.)

2 I don’t know how to do one of these myself, but I couldn’t find anyone criticizing how they did it so I trust that it’s accurate. Here’s a brief explanation of the term and why it’s important.

Endnote: Maxine Najle has also written a Master’s thesis you might like: “Delicious Justice: Schadenfreude Toward Atheists Bound for Hell.” I’m suddenly hoping we see a lot more from her in the future. We might be witnessing the birth of a new Elizabeth Loftus!


Come join us on FacebookTumblrTwitter, and our forum at rolltodisbelieve.com!

If you like what you see, I would love to have your support. My PayPal is captain_cassidy@yahoo.com (that’s an underscore in there) for one-time tips, and I also welcome monthly patrons via Patreon with Roll to Disbelieve. Thanks! <3

Lord Snow Presides… is our weekly off-topic chat series. I’ve started us off with a topic, but feel free to chime in with whatever’s on your mind! Lord Snow is my sweet, elderly white cat, who presides over my household like Conan the Barbarian on his throne, wearing the jeweled crown of Aquilonia upon a troubled brow. It is I, his chronicler, who alone can tell thee of his saga. 

"And they really lose their shit if you have evidence to back your opinion!"

Xenos Christian Fellowship: Turnt Up For ..."
"I'm guessing you haven't run out yet?"

Christian Opportunists Love Strange Days (LSP ..."
"This is going to be an extra-special crown, ism't it?"

The Husband Hunt: An Evangelical Horror ..."
"No, you're the First Loser! You're number one!"

The Husband Hunt: An Evangelical Horror ..."

Browse Our Archives

Follow Us!


TRENDING AT PATHEOS Nonreligious
What Are Your Thoughts?leave a comment