Love, Joy, Feminism
A Guest Post by Apodosis
Libby Anne posted recently about a new survey conducted by Brian Ray of the National Home Education Research Institute, a non-profit which conducts studies of questionable scientific validity on homeschooling. As a Ph.D. social scientist myself, I looked over the new survey with a critical eye and I’m sorry to say there won’t be much useable data gleaned from it because it is rife with methodological problems.
The survey has over 100 questions and nearly every question needs revisions, so I’m just going to summarize my main critiques of the study.
1. Institutional Review Board
When you are conducting a survey that collects personal information from the participants, every IRB in the country requires that you have signed informed consent forms for each of your participants. In online surveys, these tend to include a cover page describing the purpose of the study and what is asked of the participants, as well as a signature box to show you have understood your rights.These forms often include the phrases “You may stop the survey at any time without giving a reason” and “You are not obligated to answer any question you don’t want to”. They provide contact information for the PI and other associated researchers, as well as for the IRB that approved it. “If you feel your rights have been violated, call this number” etc.
In the Gen 2 survey, there is no such information. It is not clear who is being asked to participate (see #2), the goals of the study are not honestly stated (see #3), and there is no contact information for the PI or an IRB to contact if you feel your rights have been violated. I had to dig on NHERI’s website to find the email address of the PI, Dr. Brian D. Ray (it’s bray@nheri.org, btw, in case you feel your rights have been violated), and he actively discourages you from contacting him. Red flag.
A peer-reviewed publication will not publish any results from research that was not overseen by an IRB.
2. Target Population
It is not clear who is being asked to participate in the survey. In the “About” section, Ray says the participants are “those between the ages of 18-38 years old that grew up in religious homes”, but in the FAQ he says “Anyone between the ages of 18-38” may participate. I am between the ages of 18 and 38; I was raised in a moderate mainline Protestant family, and I am now a progressive mainline Protestant. I honestly cannot tell if he wants my data or not. Though both of those statements about who should participate apply to me, the questions on the survey indicate otherwise.
3. Goals
Which brings me to my most serious methodological critique of the study, which is that the goals are not honestly stated. There are two different studies conflated here which have entirely different goals. Dr. Ray even alludes to this in the “About” section: one goal is “to come up with data points of key influences that either encouraged or deterred the participants from practicing the same faith as their parents”; the other goal is to “use the statistics from this survey to help equip parents to make more informed decisions in the education and spiritual guidance of their children.” That is, in simpler terms, the goals are (A) to find out how young people’s religious views change as they reach adulthood, and (B) to figure out how to make sure young fundamentalists/evangelicals stay in the fold.
In fact, the main goal of the study seems to be (B) with a shallow veneer of (A) superimposed on it. Now, (A) is an interesting study whose results I would look forward to reading. (B) is not a scientific study. You do not perform a scientific study with the goal of achieving a certain result. Social science is about trying to describe and explain human behavior, not about trying to change it or attach value judgments.
Here are some examples of the problems that arise when the two studies are conflated.Dr. Ray pays lip service to the idea that young people may belong to a variety of faiths, as evidenced by his questions:
Generally, what kind of religious service did you attend as a child?
What kind of church or religion do you currently associate yourself with?
Available answers include Atheist, Agnostic, Buddhist, Hindu, Jewish, Muslim, and a variety of mainline Protestant denominations in addition to the exhaustive list of evangelical/fundamentalist groups. However, on the next pages he asks questions like
My current church is serious about applying the biblical principles of eldership, shepherding, mentoring, and church discipline.
Did you ever get any “worldview” training?
How often did your Father/Mother read the Bible to you?
These questions indicate that the primary audience of this study is people raised in a fundamentalist/evangelical/Christian patriarchy home. How are Buddhists and Hindus and Atheists supposed to answer these questions? Note that there is no “Not applicable” option available. This brings me to #4.
4. Not Applicable
It is important in surveys to allow your participants the freedom to answer the questions honestly, and not be forced to pick the “closest” answer. Many of the questions follow the “Have you stopped beating your wife” pattern, where e.g. a person who has never beaten their wife has no honest way to answer the question. “Not applicable” is only an available answer for one of 100 questions in the survey. This becomes a problem with questions like the following:
How distant or close do you feel to God most of the time?
What was the status of your parents’ marriage during your raising?
My father was very involved in our family home life.
How would an atheist answer the first question without “Not applicable”? How would someone answer the second question if they were raised by two parents in a committed relationship who were not married? And in the third case, what if you didn’t have a father?
“Other” followed by a fill-in-the-blank should also occur much more frequently as an available answer in questions such as:
Have you, or your significant other, ever chosen to have an abortion? Yes/ No
Do you think that people should wait to have sex until they are married, or not necessarily? Yes/ Not necessarily
How would you describe yourself when you were a child? I was very rebellious/I struggled with rebellion, but overcame it/I was always fairly obedient and honoring as a child
In the first question, it would be more appropriate to ask “Have you ever had an unwanted pregnancy?” and then to ask how it was dealt with. As it stands, a person who has never been pregnant and a person who has carried a rapist’s child to term would give the same answer. In the second and third questions, there are clearly more than just the possible answers given. My answers would be “No, unless they want to” and “My parents were supportive of what I chose to do and who I chose to be”.Participants in the study need a way to answer these questions honestly, so even if Dr. Ray wants to limit his variables it would be more methodologically sound to provide an “Other” response with a fill-in-the-blank to such questions.
5. Leading/Biased Questions
This brings me to another point: almost every question is leading or has other problems caused, for the most part, by faulty assumptions and a lack of imagination on the part of Dr. Ray.
“What kind of church or religion do you currently associate yourself with?” Some people consider themselves to be a member of a faith without having a specific faith community. Some people consider themselves to be members of multiple faith traditions. A better question would be “How would you describe your religion or faith?” with multiple check-boxes allowed. And he should list all the religions, not just the ones he could think of off the top of his head! He forgot at least (off the top of my head) Baha’i, Sikh, Wicca, Christian Science, Animism, Deist, Shinto, Unitarian Universalist, Not applicable, Other____, and he did not provide any different denominations of Judaism or Islam.In addition, he should have grouped the religions together by type rather than alphabetically—this list makes it very hard to see if your faith is represented. And groups like “Mormons” and “Congregationalists” should be called by their actual names—LDS and UCC, respectively. This list of religions, while it pays lip service to religious diversity, is actually offensive in its exclusions and ignorance. Plus, there is no acknowledgment in this survey of mixed-faith households. What if your parents were of different faiths, or what if you and your significant other do not share a faith?
Questions such as
What is your sex/gender? Male/ Female
How was your relationship with your Mother / Father when you were 16-17 years old?
How often did your Mother / Father explain biblical principles to you?
presume an oversimplified, heteronormative view of gender and family.Sex is not the same as gender; there are quite a few other genders than just male and female; you might have parents of multiple genders or the same gender (and therefore not have a “Mother” and a “Father”); and you might have been raised by other family members, or primarily by friends, or in foster homes.If this was really a survey about changes in young people’s religious views, it would try to get an accurate picture of their lives without limiting them to these binaries. Ray should also ask about people’s sexual orientations if he’s that interested in the status of their romantic lives; however, his reference to “homosexual” “encounters” in one of the questions indicates that he probably does not believe in sexual orientation as a concept (see #6).
6. Limited Mindset
Many of these questions portray a mindset that is isolated to the evangelical/fundamentalist/Christian patriarchy/Quiverfull/Purity movements–it’s possible that Dr. Ray does not even realize there are other ideologies out there. For instance, there is this question/answer set:
What statement most aligns with how many children would you like to have? I don’t want to have children / I want no more than a few children / As many children as God will provide / I don’t know
This really should be divided into multiple questions. First, he should ask “Do you plan your pregnancies?” If no, he should ask about “how many children you hope God will give you”, but if yes, he should ask how many you plan to have. I don’t think Dr. Ray realizes that some people plan their pregnancies. Then there is the following question/answer pair:
If you have—or were to have—children, what form of education do you plan to use for them? Christian school / Christian school and homeschool / Christian school and non-Christian private school / Christian school and public school / Homeschool / Homeschool and non-Christian private school / Homeschool and public school / Private school, non-Christian / Public school / Charter school/virtual charter / Other
At least this one has “Other” as an option, though it doesn’t let you write in your response. First of all, it’s a badly designed question—he should just list the types of school and let you check as many boxes as you want. Second, the question relies on the assumption that parents would choose their children’s manner of schooling before the children even exist. What if the child is gifted or disabled? Would that change the parents’ plans? And Dr. Ray does not even realize that people who would answer like me are out there—“It depends on the child’s needs and wants, as well as other considerations such as expense, distance, quality of education, etc.”
Then there’s this question/answer pair:
Did your parents use corporal discipline (spanking) with you?No / Yes, consistently and they were generally under loving control / Yes, consistently and often they were not under loving control / Yes, inconsistently and they were generally under loving control / Yes, inconsistently, and often they were not under loving control
These answers presuppose a worldview where the value of spanking is not open to debate (spoiler: we don’t live in that world). It would be more accurate to ask “What was your parents’ position on spanking?” “How often were you spanked?” “Are your attitudes toward it positive, neutral, or negative?” “Would you spank your own children?” etc.
And then there’s the part where he puts these two questions next to each other, clearly conflating them (shades of Libby Anne’s two boxes): “Have you had a sexual encounter or physical relationship with someone to whom you are not married?” and “Were you ever sexually abused before age 18?” Note that a girl who was repeatedly raped by her Christian patriarchy father for ten years and a girl who was raped once by an acquaintance at a party as a teenager would probably have different reactions to their faith traditions, though they would give the same “Yes” response. No effort is made to make this distinction in the survey. And nowhere does the survey ask about physical abuse, since Dr. Ray probably doesn’t think it exists.
Or how about the question “If you were unsure of what was right or wrong in a particular situation, how would you decide what to do?” with the available answers: Do what would make you feel happy / Do what would help you to get ahead / Follow the advice of a parent or teacher, or other adult you respect / Do what you think God or the scripture tells you is right / Something else. These answers reveal Dr. Ray’s belief that morality does not exist outside (his brand of) Christianity.
7. Exclusionary Language
The language Dr. Ray uses is exclusionary and often confusing. For instance, he persists in using terms like church, pastor, scripture, prayer, Bible, youth group, Sunday School when these terms are uniquely Christian and do not apply to people of other faiths–he ought to say faith community, religious leader, holy book, prayer/meditation, religious education if he wanted to get accurate data. He also fails to define several terms which I don’t understand because I was not raised in a fundamentalist environment: family-integrated, homeschooling-friendly, shepherding, church discipline, worldview training. In his questions about belief in “God” and “heaven”, he ought to ask separate questions about each property he wants to assign to these words’ meanings (e.g. “Do you believe in a deity or deities?” “If yes, do you believe the deity/deities is/are omniscient? Omnipresent? Omnipotent? Does it/do they have a gender? Does it/do they affect everyday events?” etc.). If Ray wants accurate data, he should define confusing and ambiguous terms.
You should not be able to tell someone’s political and religious beliefs from survey questions designed to elicit yours. You should not be asked to give dishonest answers to survey questions because your honest answers are unavailable as options. You should not have to infer how exclusionary language in the questions would apply to your situation. You should not be asked to give out your personal information without giving your informed consent. An IRB would not approve this study. I would not rely on any of the conclusions that come out of it.
Ad-Free & Subscriber Log-In