Test Driving the Turing Test Poll

The Ideological Turing Test starts this week, and I’ve put together a generic Google Form so you guys can give feedback on the survey instrument.  You can test drive it here. Make sure you give me feedback in the comments of this post, not on the form.

I’d definitely like to know if you have any technical problems, and this is your last chance to give feedback on the questions.  I’m using a four item Likert scale for judging, since I’m guessing that upping the number of choices will not help people drill down.  I’ve been sticking with the “Very Likely Christian” “Likely Christian” “Likely Atheist” and “Very Likely Atheist” labels.  Do people still like these?

I’m adding a question about how compelling people find the entries.  Last year, some Christians deliberately tried to imitate “boring” atheists.  That’s fair tactics, but I would have liked to know if the atheist voters who passed them agreed that the answers were dull.  So this year, I want to add a question about how compelling/attractive the entries are, regardless of whether you accept their premises.  I’m not sure of the best way to go about it, though.

Currently, I’ve got category labels, but I’m not very happy with them.  Perhaps it would be better as just “Please rate how compelling you found this entry on a scale of 1 to 5″?  I’m concerned atheists and Christians may bring radically different ideas of what ‘compelling’ denotes to the survey.  I could break it up into two questions, one on how internally consistent and logical you found the entry and one of the attractiveness of the worldview being described, regardless of it’s truth value.  Advice?

Make sure you let me know if there are other demographic questions you want included.  Keep in mind that, since I’m doing one form for each entry, I don’t want the non-essential questions to be too long, since you’ll fill them out each time.

About Leah Libresco

Leah Anthony Libresco graduated from Yale in 2011. She works as a statistician for a school in Washington D.C. by day, and by night writes for Patheos about theology, philosophy, and math at www.patheos.com/blogs/unequallyyoked. She was received into the Catholic Church in November 2012."

  • Aaron M

    Leah,
    What’s your reasoning for limiting believers just to Christians? What specifically about the questions requires people to either accept that God does not exist or that Jesus died so that people could be saved?

    • Tim

      Out of interest, what was the rational behind compelling changing to attractive mid-question?

      • leahlibresco

        I wanted both words in the mix for people to react to and give feedback on. That question is very much still under revision.

    • leahlibresco

      Nothing about the question requires it. It’s easier to judge the entries against each other if they’re all aiming at a similar thing (and if we’ve got some guess of how the religious they’re impersonating actually looks). I mostly do Christianity vs atheism on this blog, so I don’t really have the expertise to recruit Hindus and Hindu impersonators, for example.

  • http://prodigalnomore.wordpress.com/ The Ubiquitous

    I found this question unsatisfying.

    Regardless of your own beliefs, how compelling did you find this entry?

    If we are putting aside our own beliefs, how can we honestly determine how personally compelling the entry is? Do you mean something like:

    Putting aside your own beliefs, how strong of a case did this entry make?

    • http://prodigalnomore.wordpress.com/ The Ubiquitous

      Now reading your post, I see you had similar feelings about the phrasing.

      I could break it up into two questions, one on how internally consistent and logical you found the entry and one of the attractiveness of the worldview being described, regardless of it’s truth value. Advice?

      I am against splitting attractiveness from truth value. We aren’t Nietzsche. Leave it as: How strong of a case did this post make?

      Alternatively, if you want to add a second half to the question, consider your professor’s question split into three:

      1. Could you live this philosophy?
      2. Would you want to?
      3. Is it true?

      Why reinvent the wheel?

      • http://prodigalnomore.wordpress.com/ The Ubiquitous

        You would have to carefully note on which of the three questions the reader would be allowed to draw on his own experience. Perhaps one way to do this would be to include two sets of the similar questions:

        For questions 1 through 3, draw on your own experiences:
        1. Could you live this philosophy?
        2. Would you want to?
        3. Is it true?

        Then:

        For questions 4 through 6, do not draw on your own experience. Consider the philosophy, or the case made, merely in the abstract:
        4. Is this philosophy livable?
        5. If this could be proven true, would you want to live by it?
        6. Is this a strong case?

  • @b

    If I’m right, the test is to spot the entry that advocates a position the author believes is mistaken.

    If so, I think Q1 is asking whether we trust the entry is (Very Likely) Genuine or (Very Likely) an Impersonation. I prefer that wording. Last year’s temptation was to match an entry to its theological best fit, ie. smells fishy so not “my” team.

    Q2 is then asking whether the entry has shifted your confidence level (noticably, trivially, or imperseptively). This will be very interesting. And I predict asymmetrical.

    • DelphiPsmith

      Yeah, I like the Genuine/Impersonation wording better too, since that is what’s being tested.

  • DelphiPsmith

    On item 1, don’t you need an option that falls right in the middle, “Don’t know” ? Unless you want to force people to make a decision…

    • leahlibresco

      I do want to force people. You’re allowed to make a weak guess, but you have to make a guess.

  • deiseach

    Well, since we’re all being critical, I would suggest maybe an “Other” option and box for the education question, since any non-American visitors may have different qualifications/attainments that don’t exactly map onto the choices (here in Ireland, for example, they’ve just fiddled with the qualifications system so that what was formerly a three-year diploma is now a pass degree and what was a four-year degree is now an honours degree), but it’s not a vital question.

    You might like to throw in a geographical one while you’re at it, for people to answer if they like (e.g. Do you currently reside in: U.S.A., Europe, Other)?

    Maybe for the religion one ask does the person answering consider themselves progressive or conservative, although the wording of that could be sticky – one person’s ‘orthodox’ is another person’s ‘so you’re calling me a heretic, are you????’.

  • Pingback: Refined the Turing Test Survey


CLOSE | X

HIDE | X