Ethical Problem in the Turing Test?

Ethical Problem in the Turing Test? July 19, 2011

While we wait for the last responses to the Ideological Turing Test to trickle in, I wanted to address a concern that was raised by a couple of Christian bloggers. Stacy of Accepting Abundance felt she couldn’t ethically vote in the survey, and Elizabeth of Startling the Day shared some of her worries. Stacy wrote:

As I read the responses I found myself unable to make such a judgement. These are people, not computers trying to pass as people, but real people. I know this test was used to study economists and how well economists of opposing ideologies understood the other side; I get that. But this is about something much deeper than economics. It’s about a person’s faith, and faith is a tender thing. As I read the responses I kept wondering, “What if this is a ‘real’ Christian, possibly new in his faith, or struggling in his faith, and I judge him to be an atheist based on his responses?” That could be damaging to a person’s soul.

My most basic response is that the participants knew what they signed up for; people who decide to take their beliefs to the public sphere need know they’ll catch flak and have to be ready.  That said, if anyone (on either side) had told me that they were planning to use the votes as a referendum on the strength of their beliefs , I would have had a chat with that participant and might have recommended they withdraw. Christians who are beaten by atheists in this contest aren’t bad Christians and they aren’t even necessarily bad apologists. A low ranking for believability doesn’t necessarily indicate that you aren’t supporting your arguments, it just means that your interpretation of Christianity isn’t what most people expect to see.

Or it might be a lot less meaningful then that. I’ve been looking over the responses, and there’s a lot of questionable reasoning among voters. One commenter on Mark Shea’s blog declared that an entry that used the word ‘theodicymust be written by an atheist because it was such a nasty piece of “derogatory jargon,” which must come as a surprise to most Christian philosophers. Most criteria were better than this, and I’m looking forward to profiling the different types of theology-dar as a follow-up to the list of rankings.  The voters provided data at least as interesting as the participants they were evaluating.

I’m glad Stacy and Elizabeth shared their concern, because this wasn’t an ethical problem on my radar.  The biggest snag came when I was picking the questions for my Ideological Turing Test.  I tried to stay sensitive to the concerns of the Christian participants who had signed up. One person’s participation was contingent on a guarantee from me that s/he would not be forced to blaspheme in the answers. S/he was also concerned that, if the Christians had to write strong attacks on Christianity while they were trying to pass as atheists, they might do harm to someone’s faith. After all, they wouldn’t get the chance to rebut their arguments-as-atheists or explain why these claims weren’t enough to shake their personal faith.

To accommodate these concerns, I left the questions open for positive statements about atheism instead of compelling attacks on religion (the first question was “What’s your best reason for being an atheist?” not “What’s the best argument against religion”). This ended up satisfying everyone concerned that they weren’t endangering the salvation of others.

I certainly won’t say I hope no one’s confidence is shaken while reading this blog, but, as always, I hope people’s minds are being changed only for good reasons.  The percent of people who thought an entry was written by a Christian has no relation to the validity of the arguments in the entry, and I hope it’s the strength of arguments that move people toward truth.  I stand by the Litany of Gendlin:

What is true is already so.
Owning up to it doesn’t make it worse.
Not being open about it doesn’t make it go away.
And because it’s true, it is what is there to be interacted with.
Anything untrue isn’t there to be lived.
People can stand what is true,
for they are already enduring it.

Browse Our Archives

What Are Your Thoughts?leave a comment
  • Anonymous

    The big answer to the ethical issue is consent.This isn't some people that were tricked into agreeing to be judged, with small print being snuck in a license agreement. Everyone who participated in answering questions did so knowing that THE WHOLE POINT was to be judged – and yes, possibly misjudged. I can't imagine that anybody would go into this test without being okay with this. As to the issue of criteria – yeah, I know mine were very iffy. I was fairly confident in a lot of my picks on the Atheist Answers, but it had little to do with the "interpretation of Atheism isn't what most people expect to see." It was a lot about the words used – there are some common atheist memes, ways of phrasing things, that consistently cropped up among the Atheist responders but not among the Christian ones. All about style and not substance. Or so I thought. I'm well aware that once results come out, I might turn out to have done no better than chance…

  • KL

    I agree that style was more important than substance in my choices. This could still be troubling to someone who is overall "misjudged," I suppose, but as you and others have pointed out, the participants knew when they signed up what they were likely to be judged on. Also, you may be aware of this, but at least one person has claimed to be feeding false results into the answer survey: seems a little mean-spirited to me, even given the fact that the commenter seems to have a deep misunderstanding of the purpose of the exercise. I'm not terribly concerned about it skewing results, especially since you're not trying to create a methodologically rigorous, peer-review-level study, but I figured I should bring it to your attention.

  • Anonymous

    Ew. Annoying asshole is gonna try to mess up results. I guess we'll see if he succeeds. Might be a few ways of filtering out the garbage, depending on how he entered it and what information the google form gives you.

  • Bo

    Wow, that commenter at Volokh is a jerk.I participated to see how well I actually understand Christian viewpoints, not to prove that I'm better than Christians.Leah, not sure what information you got from responders or what you're planning on publishing, but could you include time stamps? I was going to ask for IP addresses, but you would want to obfuscate them for privacy reasons, but in a way that similar addresses produce similar obfuscated strings (i.e., poorly).Depending on how consistently people voted, we might be able to detect bogus responses clustered in a certain time. I presume he didn't leave thoughtful comments each time in the comment section either, which may help.

  • @ KL's link:"I previously dumped numerous bad-data responses into the atheist round of this “test”, and I intend to do the same on this round. For those who ask how could one put “bad data” in without knowing the answers, I respond that I simply put in completely randomized answers to the questions from numerous IP addresses. I’ve also had more than a few colleagues do the same. Thus, the “test” is utterly worthless as an experiment."Wow. Just… wow. People like this are statisticians worst nightmare.

  • Anonymous

    Bo, there's no need to suggest methods for filtering data while the voting is still open.

  • I saw that comment on Volokh, and I'm not too worried. There were well over 1000 responses in the round he tried to sabotage, so he should just be background noise. Not to mention that, to truly mess up the test, he'd have to know the right answers, in order to give false ones. If he fed in random data, it just makes small differences harder to spot.

  • Another way to weed out idiots like the guy on Volokh would be to only count answers that explained their rationale for selection (if there are enough people who did this to yield a meaningful result). Maybe you could publish two sets of numbers, one for the people who explained their thinking and one for those who didn't?

  • So, we have the commenter at Volokh's who is so sure that Leah has an ulterior motive that s/he felt compelled to try to screw up the project, and the commenter at Shea's who just assumed all the dickish sounding people were atheists. Nice.

  • KL

    @Leah,Yeah, I wasn't terribly concerned either, for the reasons you mentioned. But figured I should point it out so you could at least keep an eye out. It does make me sad that people feel the need to try to sabotage the thing, though, even if they don't succeed.

  • I'll also add that a pretty high proportion of respondents left comments, so unless he's really putting an obscene amount of time and effort into messing up a project that doesn't prove anything anyway, I doubt he did much damage.

  • I love that quote from Gendlin! It is awesome!I would make some rare exceptions to the part "People can stand what is true,for they are already enduring it" (like for young children).

  • Also, I have been trying to encourage people to come and vote. I hope it is helping. I am super excited to see the results.

  • Antrobus

    I think that the people who worry about damaging someone's faith are forgetting that there's going to be a lot of bluff and double-bluff going on here, because everyone wants to 'win' by being a successful imposter (subjects), or hard to fool (responders). Also, it's mainly the responders who are being tested, not the subjects. If someone is genuinely a Christian, but I reckon they're an atheist pretending to be a Christian, it's my judgment that's out, not the sincerity of their faith (and more or less vice versa if the essay of a genuine atheist strikes me as a caricature of how many atheists think).I've twice been a Turing Test Christian in real life – I'm an atheist, but I sometimes interact with charity projects where there's no benefit to be obtained by revealing this. So I don't, but neither do I lie and say I'm a Christian. I occasionally discuss things which I do truly believe, such as the impossibility of knowing that one is 'basically a good person'. The curious thing is that the more uncertain about morality and volition I appear, the more convinced religious people are that I'm deeply religious.

  • Antrobus, that is really funny: "The curious thing is that the more uncertain about morality and volition I appear, the more convinced religious people are that I'm deeply religious."I think they might be reacting to your obvious interest in the subject and, coupled with the (false) assumption that you are a Christian, infer that you are deeply religious. I am sure they would not make this inference if you demonstrated indifference instead of interest.

  • Hey Leah! Please excuse my poor choice of wording on my post today. I didn't mean "rebuttal" as much as I meant reaction. I respect your test and I love it as an effort for both "camps" to learn more about one another!

  • No worries, Elizabeth! The internet is a poorly-inflected place, so I wanted to make sure I explained.

  • Hi Leah, I've enjoyed reading through your test. Great blog! I just wanted to say hello!