When Data Go Bad

When Data Go Bad November 12, 2012

Imagine if some evangelical social scientists set out 20 years ago to document how the children of “Christian” parents fare in life, and began the task by gathering a small sample of children even before they were born by recruiting married parents who attended Sunday School classes at New Life Church in Colorado Springs, the Wheaton Bible Church outside Chicago, and Saddleback Community Church in Orange County, California—all located in prosperous communities exhibiting above-average social support of families, children, and faith. And then these scholars interviewed and re-interviewed the parents—and later the kids themselves—off and on for 20 years, and called it the National Christian Family Study. Their findings were published in peer-reviewed social science journals with a good deal of regularity over the years, and were utilized by political and social movements to improve public perception of (all) Christian parents. The news media regularly run stories on this study and its updated findings. Occasionally those studies included a comparison sample of other parents and kids, pulled from other small datasets collected by other people, and an occasional comparison with nationally-representative data from large studies like the National Survey of Family Growth. The evangelical kids compared favorably. Year after year the kids are re-interviewed, regardless of their awareness of the media attention cast upon them—including cover stories in Christian magazines—and the obvious utility of the findings for the movement to enhance respect for Christians, especially evangelicals.

Would the social scientific community consider this study a solid one, employing high-quality sample selection methods and useful both for understanding the experience of Christian households in America and for comparing this group of children with those in other studies? Unlikely. Seriously unlikely. And I would concur.

Well, this is what’s happened in the social science of family, except the study wasn’t about evangelical parents and their children. Read about it here.

Before you presume I’m on the war path here, think again. Quality sampling yielding valid, reliable data is not threatening. It is what is: good information, subject to invariable limitations. I just want to see the social scientific study of children and families more rooted in good data in which we can have confidence. I’m not alone. Despite a summer of abuse, however, I continue to hold that accurate information on politicized topics is important to collect, even though it may not reveal what the conventional wisdom would suggest or what elites–including very many social scientists–wish to hear. So be it.

Browse Our Archives

Follow Us!

What Are Your Thoughts?leave a comment
  • Jimmy Green

    You are not doing yourself any favors by continuing to misrepresent the findings of your study. As you have admitted in respeonse to SSRJ’s commissioned audit and in mainstream media interviews, your sample group was not “children of gay parents” but children that self-reported one of their parents having had a same sex encounter at any point in their life. However, when addressing conservative audiences, you drop the nuance and make inaccurate comparisons to gay people for what seems to be political, not scientific, purposes. The way you continually frame your project as somehow invalidating decades of previous research by “liberal” sociologists, with the implicit suggestion that there is some all powerful gay conspiracy that has corrupted the science, is transparent to anyone who does not share your a priori religious conviction that gay people are “intrinsically disorderd” and thus their children must be tainted as well.

    If you really want to prove you that you had no political intent in this matter, you would stop giving interviews to Focus on the Family and other anti-gay political groups, as well as publically disclaim the use of your work in amicus briefs in same-sex litigation. Prior researchers, like Lisa Diamond and Carol Gilligan, have publically condemned and even given depositions in response to right wing groups distorting their research. You said in your Slate piece that you did not believe your research should be used to deny gay people the right to marry but that it has already been cited in ongoing litigation concerning DOMA and same-sex marriage, along with being cited in every right wing article or blog post that describe gay people in the most disturbing ways, specifically the slander about child abuse.

    Many people believe this is what you intended from the start and I don’t see much evidence to disprove that assumption.

    • Mark Regnerus


      I’m not looking to do myself favors, friend. In this blog post, I’m calling out a long-cited dataset for some significant problems. To be sure, the NFSS has limitations, ones that I’ve admitted. If organizations to whom I’ve given interviews have glossed over those limitations, they do so in spite of my selected words. While other scholars may leap to squash the possible interpretations that people make of their data analyses, I tend not to do that, not for Focus on the Family nor for the Human Rights Campaign, or anyone in between. If a media organization is interested in laying out the complexities of the data, I am open to talking with them. Most, however, have been looking to conduct a hatchet job. I’m not helping out that effort, dude. Finally, I don’t pay much attention to who all is talking about the study, and what they’re saying. (I have a day job that requires attention.) People may believe about me what they wish; they are free to. I will stick by the data, and offer plausible interpretations of it, together with spelling out its limits. Friends and enemies can attest to that. Moreover, I have made the data public–uploaded it to the ICPSR data repository at the University of Michigan in early October. The NLLFS is not public data; never has been, after 20+ years. You are free to wonder what’s going on inside my head (and perhaps wrestle over whether that should even matter) but making data public is a scientific value/principle.


  • George Yancey

    That is a great idea Jimmy. Therefore, I think that all scholars should stop giving interviews to all political organizations. That way no one would be able to challenge our political bias. That is what I think if I believe your contention that Regnerous should not be interviewed by any conservative organizations. So may I assume that the standard you want to impliment on Regnerus is the same one you want for everyone or do you want to only allow certain political organizations to use social science research. I know that is not what you want because we should be interested in doing science and not poltics right?
    By the way I have unfortuanatly had some of my reserach cited by a white supremacist group. Trust me as an African-American I am not a white supremacist. Pointing to others who cite our work is not a very convincing argument of a scholars intentions.

  • Paula Murchison

    There is yet no “quality sampling yielding valid, reliable data” about children raised by gay parents in the general. The study by Regnerus as well as any other study dominated by self-selecting “children” volunteers raised by heterosexually married or heterosexually failing opposite sex parents and further limited to only those self-selecting volunteers willing to report for whatever motivation that one of his/her opposite sex parents had some kind of extra-marital same-sex affair is by no means an adequate substitute. The day Regnerus or anyone can find a representative non-“compromised” (to use his term) sample, hell will freeze over.

  • Marco Luxe

    The better comparison for your example would not be to “Christian” parents, but to a marginalized, isolated and stigmatized minority group, let’s say the Hmong in rural Wisconsin. Now try to come up with a longitudinal study involving this formerly persecuted refugee group that doesn’t have some self-selection. Your comparison to the NLLFS is facile and disingenuous.

    • Ted Seeber

      Or Westboro Baptists- I truly cannot think of a group more reviled for their actions.

  • Paul McMichael

    Interesting choice of analogy. Choose a group from the normative tradition of western societies and you’d most likely find outcomes consistent with normative western societies. Choose from the non-normative tradition of same sex couples, however self-selecting the methodology, a group that has been told at times they are inhernetly disposed to child abuse, that having the children is intrinically abusive, a group that has their rights proscribed and prescribed for millenia, a group without wide poltical support until very recently, who are very often estranged from wider support networks such as family and church etc. then you find child outcomes as good if not better than their heterosexuals peers? Now that is truly a sociological study worth doing For the first time, no one can say homosexuals ‘can’t’ bring up kids, they can only say, ‘some’ homosexuals can bring up kids. Nevertheless it is a radical finding, unlike your study which suffers terribly from confirmation bias (what, you didn’t go looking for that outcome? Yur funders thought they would find LBGT parents are just as good? Don’t even go there).

    You need to realise – nothing is in a vacuum “I will stick by the data, and offer plausible interpretations of it, together with spelling out its limits. ” No-one is listening to the interpretations limitations. And they won’t listen until you speak out louder. “I’m not helping out that effort, dude.” Intellectual abdication, nothing less. Own it.