![\"\"](\"https:\/\/wp-media.patheos.com\/blogs\/sites\/166\/2013\/07\/survey.jpg\" "\\"survey\\"")
<\/a>A Guest Post by Apodosis\u00a0<\/em><\/strong><\/span><\/p>\nLibby Anne posted<\/a> recently about a new survey<\/a>\u00a0conducted by Brian Ray of the National Home Education Research Institute<\/a>, a non-profit which conducts studies of questionable scientific validity<\/a> on homeschooling. As a Ph.D. social scientist myself, I looked over the new survey with a critical eye and I\u2019m sorry to say there won\u2019t be much useable data gleaned from it because it is rife with methodological problems.<\/p>\n The survey has over 100 questions and nearly every question needs revisions, so I\u2019m just going to summarize my main critiques of the study.<\/p>\n 1. Institutional Review Board<\/strong><\/p>\n When you are conducting a survey that collects personal information from the participants, every IRB in the country requires that you have signed informed consent forms for each of your participants. In online surveys, these tend to include a cover page describing the purpose of the study and what is asked of the participants, as well as a signature box to show you have understood your rights.These forms often include the phrases \u201cYou may stop the survey at any time without giving a reason\u201d and \u201cYou are not obligated to answer any question you don\u2019t want to\u201d. They provide contact information for the PI and other associated researchers, as well as for the IRB that approved it. \u201cIf you feel your rights have been violated, call this number\u201d etc.<\/p>\n In the Gen 2 survey, there is no such information. It is not clear who is being asked to participate (see #2), the goals of the study are not honestly stated (see #3), and there is no contact information for the PI or an IRB to contact if you feel your rights have been violated. I had to dig on NHERI\u2019s website to find the email address of the PI, Dr. Brian D. Ray (it\u2019s bray@nheri.org, btw, in case you feel your rights have been violated), and he actively discourages you from contacting him. Red flag.<\/p>\n A peer-reviewed publication will not publish any results from research that was not overseen by an IRB.<\/p>\n 2. Target Population<\/strong><\/p>\n It is not clear who is being asked to participate in the survey. In the \u201cAbout\u201d section, Ray says the participants are \u201cthose between the ages of 18-38 years old that grew up in religious homes\u201d, but in the FAQ he says \u201cAnyone between the ages of 18-38\u201d may participate. I am between the ages of 18 and 38; I was raised in a moderate mainline Protestant family, and I am now a progressive mainline Protestant. I honestly cannot tell if he wants my data or not. Though both of those statements about who should participate apply to me, the questions on the survey indicate otherwise.<\/p>\n 3. Goals<\/strong><\/p>\n Which brings me to my most serious methodological critique of the study, which is that the goals are not honestly stated. There are two different studies conflated here which have entirely different goals. Dr. Ray even alludes to this in the \u201cAbout\u201d section: one goal is \u201cto come up with data points of key influences that either encouraged or deterred the participants from practicing the same faith as their parents\u201d; the other goal is to \u201cuse the statistics from this survey to help equip parents to make more informed decisions in the education and spiritual guidance of their children.\u201d That is, in simpler terms, the goals are (A) to find out how young people\u2019s religious views change as they reach adulthood<\/em>, and (B) to figure out how to make sure young fundamentalists\/evangelicals stay in the fold<\/em>.<\/p>\n In fact, the main goal of the study seems to be (B) with a shallow veneer of (A) superimposed on it. Now, (A) is an interesting study whose results I would look forward to reading. (B) is not a scientific study. You do not perform a scientific study with the goal of achieving a certain result. Social science is about trying to describe and explain human behavior, not about trying to change it or attach value judgments.<\/p>\n Here are some examples of the problems that arise when the two studies are conflated.Dr. Ray pays lip service to the idea that young people may belong to a variety of faiths, as evidenced by his questions:<\/p>\n Generally, what kind of religious service did you attend as a child?<\/p>\n What kind of church or religion do you currently associate yourself with?<\/p><\/blockquote>\n Available answers include Atheist, Agnostic, Buddhist<\/a>, Hindu, Jewish, Muslim, and a variety of mainline Protestant denominations in addition to the exhaustive list of evangelical\/fundamentalist groups. However, on the next pages he asks questions like<\/p>\n My current church is serious about applying the biblical principles of eldership, shepherding, mentoring, and church discipline.<\/p>\n Did you ever get any \u201cworldview\u201d training?<\/p>\n How often did your Father\/Mother read the Bible to you?<\/p><\/blockquote>\n These questions indicate that the primary audience of this study is people raised in a fundamentalist\/evangelical\/Christian patriarchy home. How are Buddhists and Hindus and Atheists supposed to answer these questions? Note that there is no \u201cNot applicable\u201d option available. This brings me to #4.<\/p>\n 4. Not Applicable<\/strong><\/p>\n It is important in surveys to allow your participants the freedom to answer the questions honestly, and not be forced to pick the \u201cclosest\u201d answer. Many of the questions follow the \u201cHave you stopped beating your wife\u201d pattern, where e.g. a person who has never beaten their wife has no honest way to answer the question. \u201cNot applicable\u201d is only an available answer for one of 100 questions in the survey. This becomes a problem with questions like the following:<\/p>\n How distant or close do you feel to God most of the time?<\/p>\n What was the status of your parents\u2019 marriage during your raising?<\/p>\n My father was very involved in our family home life.<\/p><\/blockquote>\n How would an atheist answer the first question without \u201cNot applicable\u201d? How would someone answer the second question if they were raised by two parents in a committed relationship who were not married? And in the third case, what if you didn\u2019t have a father?<\/p>\n \u201cOther\u201d followed by a fill-in-the-blank should also occur much more frequently as an available answer in questions such as:<\/p>\n Have you, or your significant other, ever chosen to have an abortion? Yes\/ No<\/p>\n Do you think that people should wait to have sex until they are married, or not necessarily? Yes\/ Not necessarily<\/p>\n How would you describe yourself when you were a child? I was very rebellious\/I struggled with rebellion, but overcame it\/I was always fairly obedient and honoring as a child<\/p><\/blockquote>\n In the first question, it would be more appropriate to ask \u201cHave you ever had an unwanted pregnancy?\u201d and then to ask how it was dealt with. As it stands, a person who has never been pregnant and a person who has carried a rapist\u2019s child to term would give the same answer. In the second and third questions, there are clearly more than just the possible answers given. My answers would be \u201cNo, unless they want to\u201d and \u201cMy parents were supportive of what I chose to do and who I chose to be\u201d.Participants in the study need a way to answer these questions honestly, so even if Dr. Ray wants to limit his variables it would be more methodologically sound to provide an \u201cOther\u201d response with a fill-in-the-blank to such questions.<\/p>\n 5. Leading\/Biased Questions<\/strong><\/p>\n <\/strong>This brings me to another point: almost every question is leading or has other problems caused, for the most part, by faulty assumptions and a lack of imagination on the part of Dr. Ray.<\/p>\n \u201cWhat kind of church or religion do you currently associate yourself with?\u201d Some people consider themselves to be a member of a faith without having a specific faith community. Some people consider themselves to be members of multiple faith traditions. A better question would be \u201cHow would you describe your religion or faith?\u201d with multiple check-boxes allowed. And he should list all the religions, not just the ones he could think of off the top of his head! He forgot at least (off the top of my<\/em> head) Baha\u2019i, Sikh, Wicca, Christian Science, Animism, Deist, Shinto, Unitarian Universalist, Not applicable, Other____, and he did not provide any different denominations of Judaism or Islam.In addition, he should have grouped the religions together by type rather than alphabetically\u2014this list makes it very hard to see if your faith is represented. And groups like \u201cMormons\u201d and \u201cCongregationalists\u201d should be called by their actual names\u2014LDS and UCC, respectively. This list of religions, while it pays lip service to religious diversity, is actually offensive in its exclusions and ignorance. Plus, there is no acknowledgment in this survey of mixed-faith households. What if your parents were of different faiths, or what if you and your significant other do not share a faith?<\/p>\n Questions such as<\/p>\n What is your sex\/gender? Male\/ Female<\/p>\n How was your relationship with your Mother \/ Father when you were 16-17 years old?<\/p>\n How often did your Mother \/ Father explain biblical principles to you?<\/p><\/blockquote>\n presume an oversimplified, heteronormative view of gender and family.Sex is not the same as gender; there are quite a few other genders than just male and female; you might have parents of multiple genders or the same gender (and therefore not have a \u201cMother\u201d and a \u201cFather\u201d); and you might have been raised by other family members, or primarily by friends, or in foster homes.If this was really a survey about changes in young people\u2019s religious views, it would try to get an accurate picture of their lives without limiting them to these binaries. Ray should also ask about people\u2019s sexual orientations if he\u2019s that interested in the status of their romantic lives; however, his reference to \u201chomosexual\u201d \u201cencounters\u201d in one of the questions indicates that he probably does not believe in sexual orientation as a concept (see #6).<\/p>\n 6. Limited Mindset<\/strong><\/p>\n Many of these questions portray a mindset that is isolated to the evangelical\/fundamentalist\/Christian patriarchy\/Quiverfull\/Purity movements\u2013it\u2019s possible that Dr. Ray does not even realize there are other ideologies out there. For instance, there is this question\/answer set:<\/p>\n What statement most aligns with how many children would you like to have? I don\u2019t want to have children \/ I want no more than a few children \/ As many children as God will provide \/ I don\u2019t know<\/p><\/blockquote>\n This really should be divided into multiple questions. First, he should ask \u201cDo you plan your pregnancies?\u201d If no, he should ask about \u201chow many children you hope God will give you\u201d, but if yes, he should ask how many you plan to have. I don\u2019t think Dr. Ray realizes that some people plan their pregnancies. Then there is the following question\/answer pair:<\/p>\n If you have\u2014or were to have\u2014children, what form of education do you plan to use for them? Christian school \/ Christian school and homeschool \/ Christian school and non-Christian private school \/ Christian school and public school \/ Homeschool \/ Homeschool and non-Christian private school \/ Homeschool and public school \/ Private school, non-Christian \/ Public school \/ Charter school\/virtual charter \/ Other<\/p><\/blockquote>\n At least this one has \u201cOther\u201d as an option, though it doesn\u2019t let you write in your response. First of all, it\u2019s a badly designed question\u2014he should just list the types of school and let you check as many boxes as you want. Second, the question relies on the assumption that parents would choose their children\u2019s manner of schooling before the children even exist. What if the child is gifted or disabled? Would that change the parents\u2019 plans? And Dr. Ray does not even realize that people who would answer like me are out there\u2014\u201cIt depends on the child\u2019s needs and wants, as well as other considerations such as expense, distance, quality of education, etc.\u201d<\/p>\n Then there\u2019s this question\/answer pair:<\/p>\n Did your parents use corporal discipline (spanking) with you?No \/ Yes, consistently and they were generally under loving control \/ Yes, consistently and often they were not under loving control \/ Yes, inconsistently and they were generally under loving control \/ Yes, inconsistently, and often they were not under loving control<\/p><\/blockquote>\n These answers presuppose a worldview where the value of spanking is not open to debate (spoiler: we don\u2019t live in that world). It would be more accurate to ask \u201cWhat was your parents\u2019 position on spanking?\u201d \u201cHow often were you spanked?\u201d \u201cAre your attitudes toward it positive, neutral, or negative?\u201d \u201cWould you spank your own children?\u201d etc.<\/p>\n And then there\u2019s the part where he puts these two questions next to each other, clearly conflating them (shades of Libby Anne\u2019s two boxes<\/a>): \u201cHave you had a sexual encounter or physical relationship with someone to whom you are not married?\u201d and \u201cWere you ever sexually abused before age 18?\u201d Note that a girl who was repeatedly raped by her Christian patriarchy father for ten years and a girl who was raped once by an acquaintance at a party as a teenager would probably have different reactions to their faith traditions, though they would give the same \u201cYes\u201d response. No effort is made to make this distinction in the survey. And nowhere does the survey ask about physical abuse, since Dr. Ray probably doesn\u2019t think it exists<\/a>.<\/p>\n