Apples to apples to oranges

statisticsIt’s always interesting to see which data analysis or surveys get covered by mainstream media outlets and which don’t. A bunch of media outlets chose to highlight a study this week comparing 289 teens (average age of 17 in 1996) who took abstinence pledges at that time with 645 teens who were otherwise similar but didn’t take abstinence pledges. That is, the two groups shared, among other things, similar religious, sexual and political views. The study did not in any way compare either of these groups with the 11,000 or so teens from the same data set who didn’t take pledges and weren’t similar in their political views or religious views.

So the study, whatever one thinks of it, is comparing two very specific and relatively small groups of people against each other — teens who took pledges with teens who were similar but didn’t take pledges. It may not make for as sexy of headlines but the best social science tends to have such limited scope. It enables you to control for variables much better and retain statistical significance. And it’s a great idea for a study — and one that has some pretty striking results. Research Janet Rosenbaum, who has done numerous studies in this field, found that there was no difference in the rates of sexual activity between these two groups of individuals, except that the pledge takers were somewhat less likely to use birth control. To me, this is exciting and significant enough for a news story. While there is much dispute over the whether this is the right way to measure such significance (some say that you must wait for 10 years of prolonged education to study effectiveness), the author is saying that her measurement determines the effectiveness of abstinence pledges when compared to teens with similar socio-religious outlooks.

Unfortunately, it wasn’t exciting enough for the media. Much of the coverage erred by intimating that the study compared those who took abstinence pledges with all other teens, which is not true.

Here’s the Daily Mail lede, for instance:

Teenagers who vow to remain virgins until they marry are just as likely to have pre-marital sex as other young people, a study has found.

BEEP. Wrong. MSNBC headlined its summary of the analysis this way:

Study: Teenage ‘virginity pledges’ are ineffective

Misleading. This study did not compare teens who take abstinence pledges with those who don’t and if it did, it certainly wouldn’t show abstinence pledges to be ineffective. Instead, the researcher attempted to compare highly similar groups who did and didn’t take virginity pledges — “apples and apples” as she has said.

My only problem is that while she was able to control for many variables, there was no reported control for the type of sex education the two groups received. It’s impossible to know, for instance, whether any member of either group received any sex education, much less formal abstinence education.

And yet that didn’t stop other media reports from saying that the study is an indictment of abstinence education. This was, in all fairness, probably helped along by the study’s author blaming abstinence classes for the difference in birth control rates.

Here’s Fox News, for instance, saying that this very limited study actually proves that “Abstinence-only programs do not delay the onset of intercourse.” This study didn’t even look at abstinence-only education programs. It looked at abstinence pledges, which may or may not be a part of abstinence-only education programs. Do reporters not understand what abstinence education is? As defined federally, at least, it doesn’t even mention pledges. And, again, this study only dealt with pledges. I’m not even sure — based on the reportage or study — where the 289 pledges back before 1996 came from. Were they taken as part of a school curriculum? Were they self-reported from the Baptist youth group the teens were part of? I have no idea.

Anyway, here’s the Washington Post (which used the same lede as most other outlets) saying that the study raises questions about abstinence education:

The study is the latest in a series that have raised questions about programs that focus on encouraging abstinence until marriage, including those that specifically ask students to publicly declare their intention to remain virgins.

If you’re going to indict abstinence education, it would be helpful for the reader to know whether abstinence pledges are part of formal abstinence curriculum, how much of a part, how little the study looked at types of sex education teens received, etc. There’s simply no discussion of that in the article.

The Post report does do a good job of explaining aspects of the study in laymen’s terms, but look at how quickly it broadens out to something completely beyond the scope of the study, which I’ll stop repeating:

The findings are reigniting the debate about the effectiveness of abstinence-focused sexual education just as Congress and the new Obama administration are about to reconsider the more than $176 million in annual funding for such programs.

The article quotes numerous people saying the study shows that abstinence education, which I believe fewer than a quarter of students receive, should be stopped. We also hear from someone who supports abstinence education:
apples

Proponents of such programs, however, dismissed the study as flawed and argued that programs that focus on abstinence go much further than simply asking youths to make a one-time promise to remain virgins.

“It is remarkable that an author who employs rigorous research methodology would then compromise those standards by making wild, ideologically tainted and inaccurate analysis regarding the content of abstinence education programs,” said Valerie Huber of the National Abstinence Education Association.

I do appreciate the inclusion of this quote, although I’d like to hear more about the other complaints she has about the study — other than that abstinence programs aren’t about one-time pledges.

While according to the study there were no significant differences in the sexual activity of pledgers and their very similar peers — they both significantly delayed sex compared to other teens but eventually had sex at the same age, had the same number of sexual partners, engaged in the same type of sexual activity, etc. — the percentage of the 289 pledge takers who did have sex before marriage were less likely to use birth control. The study’s author attributed the difference to “what youths learn about condoms in abstinence-focused programs.” While the story got a counterpoint from someone disputing Rosenbaum’s anti-abstinence education charge, it would be more helpful to know if there was any difference in sex education between the groups studied and, if so, what that difference was. Without knowing this information, the opposing allegations don’t mean much.

The author herself has some interesting discussion of media coverage of the report at her blog and this Time Q&A with Rosenbaum is also helpful.

Print Friendly

  • Pingback: OD Today: 1 January 2009 « Online Discernment Today

  • Julia

    This misunderstanding and misuse of facts reminds me of a widespread misunderstanding of a provision in our Constitution which I keep seeing and hearing on TV, in papers and on-line. Rather than being OT, I think it is a good example of how the real significance of facts can be misunderstood because people don’t think things through.

    I have otherwise intelligent friends who are so indignant that slaves were treated as 2/3 of a person in the Constitution that they will not listen to an explanation of what that fact actually means. Rather than a detriment to slaves, it was an attempt to limit the power of slave state politicians in Congress. Think it through with me.

    These slave state politicians were NOT going to be advocating the slaves’ interests no matter how slaves were counted; they would only be advocating the white citizens’ interest. But limiting a slave state’s official count of people – for the purpose of determining that state’s number of Representatives in the House - prevented slave states from having even more power to vote against slaves’ best interests.

    It was a compromise. Slave states actually wanted to count every slave; it was the Northern states who didn’t want slaves counted. A state like Virginia had many more non-voting slaves than white citizens; counting slaves in determining Virginia’s delegation to the House artificially inflated the number of congressmen advocating for Virginia’s white citizens – because none of them would actually advocate for the slaves. Even a 2/3 count gave white citizens in slave states more proportional representation than citizens in non-slave states. No wonder Congress could not get rid of slavery – even counting slaves as 2/3, slave states had a huge advantage in votes. It would have been better for slaves if they were not counted at all – the opposite of what many people think.

    It’s often fevered partisans who make these logical mistakes; they are so ready to believe what they know is true that they won’t even consider a different explanation of the facts.

  • Ben

    Julia, actually, 3/5s.

    Mollie, this seems like a perfectly good headline to me: Study: Teenage ‘virginity pledges’ are ineffective

    That’s exactly what the study showed. If the parents of a girl are deciding whether it would be helpful if their daughter took such a pledge, it would seem to me that this study would indicate it won’t matter either way. If she does take the pledge, she’d look like one group of apples, if she didn’t, she’d look like the other group of apples. Assuming of course she comes from the type of socio-religious background where this would even be a consideration.

  • http://www.getreligion.org Mollie

    Ben,

    Well, the study only demonstrates whether abstinence pledges are effective among a very select group.

    And it shows that they are just as effective as not taking pledges in various categories — sexual debut, number of sexual partners, etc. but a lower incidence of birth control albeit no change at all in STD rate. (which was the most surprising thing of all, in my view. No, the four year delay in sexual debut among conservative teens is probably the most interesting thing. I digress.)

    I appreciate the limited scope of the study but the headline — particularly in conjunction with the article — don’t make these distinctions and variables clear.

    Still, I completely agree with your point that the study was looking at the effectiveness of abstinence pledges. Just between two abnormal groups (statistically speaking). It’s a measurement of effectiveness — one that’s not conclusive — and it’s not the measure of effectiveness.

    For what it’s worth, I would recommend abstinence — but not abstinence pledges — for my own children. I have theological and moral objections to them and I think so little of them, in fact, that I’m surprised that they don’t “backfire” more. But I was the kind of teenager for whom a pledge would have been a recipe for DISASTER.

    This study, incidentally, disputed one of the regular claims of abstinence-pledge opponents, which is that pledge takers substitute oral and anal intercourse for vaginal intercourse — in an effort to “keep the pledge.”

    That was not evident in the sample group studied by Rosenbaum.

    Also, if you’re interested in the topic, she has authored a number of fascinating studies on pledges in particular and, it looks from my cursory review, done them particularly well.

  • Julia

    Julia, actually, 3/5s.

    I should have checked it out instead of relying on memory.

    Thanks.

  • http://perpetuaofcarthage.blogspot.com/ Perpetua

    This is fascinating. Do you have a link to the actual study?

  • Jerry

    This story illustrates that it’s way too easy to, as a friend says, leap to contusions. On the other hand, there is a story http://www.nytimes.com/2008/12/30/science/30tier.html?ref=science which shows that shows the truly religious are better at resisting temptation.

    So there’s a natural question: what are the religious beliefs of girls who take a virginity pledge? Are they doing out of devout religious beliefs or doing it because it’s the thing to do?

    It does make sense to me that the pledge is not very effective. After all, how many here have pledged to stop smoking or drinking or overeating? New Years resolutions are easy to make.

    There is a reason for all the structure around 12 step programs. I would not be at all surprised that it would take similar structures to help adolescents make responsible choices but that is a question for future research.

  • http://www.muchmorethanwords.com gfe

    Of course, one of the problems with studies of this type is the cause-and-effect thing. I have no reason to doubt the statistics that those who took pledges are somewhat less likely to use birth control or disease prevention measures. However, that does not mean that taking the pledge caused these young people to make unwise decisions. It might be that primarily those who were under a certain kind of parental pressure made the pledges, and that kind of parental pressure was a factor in the unwise behavior. Or perhaps it’s because the young people who took the pledges are the ones who didn’t have the courage to say no to their parents, and when they get older they don’t have the courage to say no under sexual pressure. I don’t know if either of these alternative explanations is true; the point is that from the data we can’t tell.

    In other words, the fact that B precedes C doesn’t mean that B caused C. A common explanation is that A caused both B and C.

    The same is true of many other health studies that appear in the news almost daily. Correlation is not the same as causation, and they are frequently confused in news stories. Sometimes, of course, there can be both, but not always.

  • http://TheEverwiseBoonton.blogspot.com Boonton

    Misleading. This study did not compare teens who take abstinence pledges with those who don’t and if it did, it certainly wouldn’t show abstinence pledges to be ineffective. Instead, the researcher attempted to compare highly similar groups who did and didn’t take virginity pledges — “apples and apples” as she has said.

    Well actually it did. You said it tried to find a group of kids with similiar demographics, religions, beliefs and so on. Some took the pledge and some didn’t. If the pledge was effective those who took it would show significant differences. If not, they wouldn’t.

    You seem to be advocating gaming the system. Compare a group of kids with strong beliefs against premarital sex who took the pledge to those who do not hold such beliefs & did not pledge. But this is not an apples to apples comparison. We’d expect the first group to have lower rates of intercourse, but that doesn’t tell us if the lower rate can be related to the pledge or exists independent of the pledge. You demand apples to apples but I’m not quite sure I understand why you don’t feel you got it.

    I have a similar beef with people who insist the evidence shows private and Catholic schools are better than public ones. Comparing parents who pay $2-10K per year out of their own pockets for education to parents who pay $0 is not an apples to apples comparison.

    My only problem is that while she was able to control for many variables, there was no reported control for the type of sex education the two groups received. It’s impossible to know, for instance, whether any member of either group received any sex education, much less formal abstinence education.

    I’m not clear how the study was set up but assuming the sample was choosen randomly this should not be a problem. Kids with formal abstinence education would have an equal chance of ending up in either the test group (those taking the pledge) or the control one (those who don’t take the pledge).

    Unless you have reason to believe that something else happened, you can’t really argue this point. Suppose, for example, the kids who didn’t take the pledge happened to have a high portion of kids who had ‘formal abstinence education’. This might make the numbers for non-pledgers look better than they really are for the general population thereby making the pledge look ineffective when in fact it might be so.

    So the study, whatever one thinks of it, is comparing two very specific and relatively small groups of people against each other — teens who took pledges with teens who were similar but didn’t take pledges. It may not make for as sexy of headlines but the best social science tends to have such limited scope.

    Fatal for advocates of the pledge and abstinence education? No but not good. Imagine if the opposite result had been found. Imagine the pledge sample, on average, delayed intercourse for 3-4 years compared to the non-pledge sample. Would you be complaining that pledgers were compared to similiar non-pledgers rather than comparing them to dissimiliar non-pledgers?

    Remove the element of sex from this study and imagine it was about something more neutral. Let’s say that we were asking if taking a pledge to alter your diet and exercise to become more healthy worked. We have a data set of 11,000 people. We pick about 1,000 people who are similiar in things like weight, health status, age, demographics etc. Some of those people take this pledge and others don’t. A few years later we see if the two groups show a difference in health that can be attributed to diet and exercise changes and we find nothing. We would conclude the pledge was ineffective……

    or in terms of stats 101 we would say the sample failed to cause us to reject our null hypothesis that the pledge made no difference in the population therefore we do not accept the alternative that the pledge did.

    You wouldn’t say this conclusion is incorrect because we should have compared high performance athlets who took the pledge to overweight couch potatos who didn’t!


CLOSE | X

HIDE | X