When Data Go Bad

Imagine if some evangelical social scientists set out 20 years ago to document how the children of “Christian” parents fare in life, and began the task by gathering a small sample of children even before they were born by recruiting married parents who attended Sunday School classes at New Life Church in Colorado Springs, the Wheaton Bible Church outside Chicago, and Saddleback Community Church in Orange County, California—all located in prosperous communities exhibiting above-average social support of families, children, and faith. And then these scholars interviewed and re-interviewed the parents—and later the kids themselves—off and on for 20 years, and called it the National Christian Family Study. Their findings were published in peer-reviewed social science journals with a good deal of regularity over the years, and were utilized by political and social movements to improve public perception of (all) Christian parents. The news media regularly run stories on this study and its updated findings. Occasionally those studies included a comparison sample of other parents and kids, pulled from other small datasets collected by other people, and an occasional comparison with nationally-representative data from large studies like the National Survey of Family Growth. The evangelical kids compared favorably. Year after year the kids are re-interviewed, regardless of their awareness of the media attention cast upon them—including cover stories in Christian magazines—and the obvious utility of the findings for the movement to enhance respect for Christians, especially evangelicals.

Would the social scientific community consider this study a solid one, employing high-quality sample selection methods and useful both for understanding the experience of Christian households in America and for comparing this group of children with those in other studies? Unlikely. Seriously unlikely. And I would concur.

Well, this is what’s happened in the social science of family, except the study wasn’t about evangelical parents and their children. Read about it here.

Before you presume I’m on the war path here, think again. Quality sampling yielding valid, reliable data is not threatening. It is what is: good information, subject to invariable limitations. I just want to see the social scientific study of children and families more rooted in good data in which we can have confidence. I’m not alone. Despite a summer of abuse, however, I continue to hold that accurate information on politicized topics is important to collect, even though it may not reveal what the conventional wisdom would suggest or what elites–including very many social scientists–wish to hear. So be it.