Abstract

Although the need to understand the efficacy of various sex education and abstinence programs has never been greater, evaluation data available are limited.  There are at least four reasons for this:  (1) the cost of good evaluation, (2) methodological difficulties in such research, (3) a social environment in which research on fertility-related topics may be seen as controversial, and (4) with abstinence programs, the possibility that a prejudice against an abstinence message for young people has limited the pool of funds available from the private sector for research on this approach.  All these issues must be addressed if we are to have a rich array of information on which to base programming. 

The 1990s may well be remembered as the "what works?" era.  The question is continually posed--by foundations, public officials, community activists, and the press.  The assumption is that, after several decades of research, there must be some idea of how best to address current social problems--violence, drug abuse, teen pregnancy, drunk driving, public safety, delinquency, and truancy, to name a few.  Money seems ever scarcer for community change, so the need is pressing to do only what has some glimmer of hope.

This essay focuses on why there is so little information about sex education and abstinence programs to answer the "what works?" question and on what might be done to improve matters so that five or six years from now the field will be in a far better position analytically.

As the director of the new National Campaign to Prevent Teen Pregnancy, I have spent many hours in conversation with experts on teen pregnancy to make certain that the campaign is well grounded in the best information available and is appealing to a wide variety of constituencies throughout the nation.  One thing is crystal-clear from these early months of work:  everyone agrees that policy should aim to reduce teenage pregnancy. Feelings on how to do so, however, strongly diverge.  Some emphasize the need to reduce sexual activity among teens, while others feel that this is a quixotic quest and that the real need is for more contraception.  Some focus mainly on changing the behavior and values of adults, claiming that teens only reflect what goes on around them.  Thus, a major teen pregnancy prevention campaign in California has as its motto "teen pregnancy is an adult problem." Others believe that more community coalitions are needed to improve matters or that the media must be reformed.  Some are interested in vigorous enforcement of the statutory rape laws to discourage men from having sex with minors, while others emphasize the use of adult mentors.

In the midst of this morass of strongly held views, a strong grasp of the facts is critically important.  Accordingly, one of the campaign s first steps was to establish the Task Force on Effective Programs and Research.  This group s mandate is to make certain that the campaign is as soundly based in science as possible, to ask if our activities match research findings, and to stimulate new research where needed.

The first task this group set for itself was to review what is known about effective programs to reduce teen pregnancy (Kirby, 1997; partial results of this review appear elsewhere in this volume).  For several reasons, the campaign has a deep and sincere interest in school-based programs, including sex education and abstinence curricula.   Schools are an obvious way to reach large numbers of adolescents.  But not only do school-based programs ensure a wide audience, they can reach different age groups with different messages.  The age of first intercourse has been steadily decreasing in recent years; educating young teens may help arrest this trend and perhaps even reverse it.  Finally, most adults support school-based education programs.

Although the consensus about abstinence programs begins to break down by the time 17-year-olds are being considered--and certainly 19-year-olds--there seems little disagreement about the importance of abstinence at younger ages.  Thus, the National Campaign to Prevent Teen Pregnancy is interested in abstinence.

Sex education and abstinence programs come in many forms.  Some sex education programs discuss only reproduction.  Others provide information about contraception or stress decision-making skills.  Some abstinence programs stress that avoiding sex is the only reasonable approach to preventing pregnancy and other unwanted outcomes, including AIDS and other sexually transmitted diseases (STDs).  If these abstinence programs discuss contraception, it is mainly within the context of its limitations its failure rates, its side effects, and its possible contribution to increased sexual risk taking.  Some call this the  abstinence-only approach.  Some abstinence programs stress abstinence until marriage, believing firmly that this value structure is a major contributor to family stability and stronger communities.  Others are less precise but prefer abstinence through high school or until "adulthood," variously defined.

Another set of programs--the abstinence-plus ones--combine an abstinence message with instruction on contraception and sometimes even direct access to contraceptive services.   These programs are relatively accepting of the fact that many young people are sexually active and that restricting access to or information about contraception will probably not result in sexually active teenagers returning to abstinence.  Therefore, their emphasis is on providing information about contraception (its benefits as well as its imperfections) and about the skills needed to use contraception successfully.

The federal government and the states are financially invested in school-based programs and thus are eager to know what works. The Adolescent Family Life Act continues to fund school-based programs.  More recently, the new welfare reform law offers the states access to $50 million annually over the next five years to support abstinence-only education that is precisely defined to include, among other provisions, teaching that abstinence until marriage is the expected standard of behavior  (see Bevan & Haskins in this volume).  In essence, this new law means that about $500 million of public funds will be spent over the next five years to support abstinence-only education of a particular type.

States are also involved with programs.  As of 1995, 26 states required abstinence education; 22 states and the District of Columbia required sex education programs; and  38 required HIV/STD education programs.
Thus, the issue of what we know about the effectiveness of sex education and abstinence programs--and, in particular, abstinence-only education--has become compelling.  Other essays in this volume detail what is known about the impact of such programs on teen pregnancy and on other markers of sexual activity, such as age at first intercourse, number of sexual partners, and use of effective contraception.  Suffice it to say, the total  picture is that research to date does not yield a simple answer to the important question of whether any of these programs has succeeded in delaying sexual activity among teenagers.  And, although several studies on abstinence-only programs have been published, the research they report is of sufficiently poor quality that clear answers are unavailable.  We simply do not know what the impact of these abstinence-only programs are:  they may delay intercourse, they may not.

Such an inconclusive response is not the kind of news that community leaders and policy makers need, especially when the lives of countless young people are touched by various curricula.  This is a most unfortunate situation.  It is also puzzling.  Why is it, after all these years of innovation and experimentation by deeply caring individuals who believe in the merit of their work, that we know so little about pregnancy prevention programs generally?  The question is of particular moment now that the federal government has put a half-billion dollars on the table for abstinence-only programs.

There are at least four reasons:  (1) the cost of good evaluation, (2) methodological difficulties in such research, (3) a social environment in which research on fertility-related topics may be seen as controversial, and (4) with abstinence programs, the possibility that a prejudice against an abstinence message for young people has limited the pool of funds available from the private sector for research on this approach.  The first three, which apply to all school-based programs, have been detailed by the Institute of Medicine (1995).

Cost.

With regard to cost, methodologically rigorous evaluations that incorporate random assignment or the development of a comparison group can be expensive.  It is, for example, often necessary to hire outside evaluators, especially for smaller programs with limited staff.  Few programs have the additional funding readily available in their budgets for such an undertaking, and program staff may be reluctant to spend program dollars on research evaluations that would not immediately translate into the ability to provide more or higher-quality service.  In some cases, evaluations are mandated by federal or state legislation, but no additional funding is provided in the legislation or funding  is set at an unrealistically low level.  This leaves the option of using funds designated for program services, much to the distress of program staff.  Sometimes additional funds for evaluation can be raised from local foundations, but success with such an approach is often limited.  This perennial problem in finding or being provided with adequate evaluation financing sets the stage for a particularly distressing sequence of events:  a program is put in place without adequate funds for evaluation, and then, when it is unable to prove its effectiveness, it is criticized for not knowing what impact it has had.

The cost problem helps, in part, to explain why so little good evaluation information has emerged from one of the major public programs supporting the abstinence-only approach--the Adolescent Family Life Program, which was established in 1981 and has been administered through the Office of Population Affairs within the Department of Health and Human Services.  This program has funded many interventions that offer services to pregnant and parenting teens and a few that emphasize teen pregnancy prevention, principally through the abstinence-only approach.  Although all programs funded by this grant mechanism have been required to evaluate their effects, the law requires that no more than 5% of total expenditures for each program be devoted to evaluation.  Given the cost of high-quality evaluation, which can sometimes equal the cost of the intervention itself, this cap has effectively eliminated the ability of individual programs to conduct a solid evaluation of their work unless they were able to raise additional funds from outside sources for this purpose--a hard task to accomplish,  especially for relatively small programs such as these.

Methodological Difficulties

As for research methods--the second issue--most programs target only a small number of people, generally a convenience sample such as students in a classroom or teen mothers receiving public support in a community program.  The sample size is usually limited, and often there is a selection bias toward people who want to participate in the program.  Small sample size makes it difficult to detect statistically significant differences between intervention and comparison groups.  And comparison groups can be difficult to select, in some cases because clinically oriented programs often provide basic health services that might be unethical to withhold.  Determining the intervention "dosage," or amount of time spent in a program, is also challenging and must be carefully tracked, because some participants attend all segments of the program and some attend only a few.  Similarly, the fact that pregnancy prevention programs often consist of many components makes it difficult to assess the relative effectiveness of each component. No single component may be the most effective piece, but, rather, the combination of components is effective. In addition, longitudinal follow-up of participants is difficult in general but is particularly challenging in reproductive health programs because of confidentiality issues.

Another problem faced in many program evaluations is that outcome measures are limited to self-reported sexual activity and contraceptive use.  Such reports may be unreliable, but alternative outcome measures are often not available, save the most conspicuous consequences such as sexually transmitted diseases (STDs) and pregnancy.  Even these obvious outcome measures can be difficult to assess precisely. Although births can be verified through the vital registration system, for example, there is no universal system for reporting abortions or miscarriages, a fact that leads, among other things, to chronic problems in documenting the actual number of abortions performed annually in the United States.

These considerations argue in favor of evaluating only a few large, multisite model programs relying on experienced evaluators with resources sufficient for the task.  Conversely, some have suggested that individual programs should focus attention on process evaluation (that is, the careful collection of data on client characteristics and service utilization) and that third parties should undertake well-funded impact evaluations (that is outcomes and long-term follow-up) of various program models that target different subpopulations.

Social Environment

A third factor that helps to explain the limited information available about abstinence-only approaches and others is the problem of the general social environment.  The past 10 or 15 years have not been hospitable to research that might be seen as sex-related and therefore controversial.  Involving adolescents in such research, particularly without parental consent, is especially controversial and can raise legal issues.

During the 1980s especially, the federal government severely curtailed systems of data collection that had been used to monitor a wide variety of programs related to fertility, such as the national family planning reporting system.  The view seemed to be that because such programs were seen by some public officials as objectionable, it was best to down play or ignore them altogether by, among other things, collecting little information on their activities or effects.  Thus, the fact that only 23 programs met the Institute of Medicine s evaluation criteria may reflect more the political climate within which pregnancy-related programs have recently operated than a disinclination among program leaders to evaluate their activities.  In addition, during the 1980s, the withdrawal of much federal funding from all but abstinence-only programs may have had a chilling effect on program directors and researchers who might otherwise have been inclined to evaluate their programs.

Prejudice

A fourth and final explanation for the limited information available about abstinence-only interventions may be that many private foundations active in teen pregnancy prevention have had a poor opinion of the abstinence-only message, with a decided preference for programs that combine abstinence with positive support for contraceptive information and services.  As a consequence, such funders many have been inclined to invest few resources either in programs that support the abstinence-only approach or in evaluation of their impact.

For the Future

If we want to move ahead in understanding how to do effective sex education, including education for abstinence, generally, we are going to have to address these hurdles.  Specifically,

1.  Public and private sector funders will need to invest substantially more money in program evaluation. The notion that a modest 5% set-aside for program evaluation can answer the tough questions about impact, for example, is naive.  As noted, sometimes as much must be spent on evaluation as on the actual program.  It may seem a bitter pill to swallow, but if we fail to make adequate investments in research, we will be here again in five years bemoaning the limited information available about how to improve matters.

2.  Researchers and program leaders alike need to adhere to strong standards of experimental design, with particular attention to addressing the pervasive problem of self-selection bias.  Findings must be submitted to peer review and published as widely as possible; in particular, we need to hear more about what did not work out and why.

3.  All of us need to help convince policy makers at all levels that their timidity about research in this area is ultimately self-defeating.  The only way we will know, for example, if a particular abstinence-program delays first intercourse will be to ask kids about that event before and after the intervention.  If we cannot ask, we are unable to develop effective interventions, whatever their makeup.

4.  If we want to learn more about the specific impact of abstinence-only approaches, those who fund these programs must intensify their investment in evaluation.  In particular, the large investment soon to be made in abstinence-only education under the new welfare reform legislation should be accompanied by adequate funds for strong evaluation, either by carving out  a substantial portion of monies for evaluation from the funds already earmarked for the abstinence-only programs or by additional funding.

With progress on all these fronts, several years from now we may have a few programs or curricula using the abstinence-only approach that have been shown to raise the age of first intercourse; if not, we will have an important piece of information that can be used in programming. We will probably also have more information on abstinence-plus approaches.  If so, communities will then have a greater range of options to choose from as they confront the challenge of pregnancy and STDs.  As alluded to, this is a big and complicated country, and communities vary enormously in their preferences and their values.  If we are serious about improving the lives of adolescents, we must be able to offer a variety of effective programs.  Some may be abstinence-based; some may be abstinence-only; some may even be youth development programs that deal very little with the mechanics of pregnancy prevention but instead help to create the basic motivation to avoid pregnancy in the first place.

We should be modest in our expectations about any of the educational programs being discussed, whether they are abstinence-only or abstinence-plus.  Teen pregnancy, the spread of AIDS and STDs, and teen sexual activity generally  have many antecedents and explanations.  It is foolish to think that small community- or school-based programs in isolation would be able to make major inroads in these areas.  Most have small budgets and therefore limited staff.  Many programs count their efforts in hours--five hours of such and such material in a classroom, for example, or two sessions on successive Saturdays.  How can these efforts compete with a media culture that saturates hours of every single day in an adolescent s life with messages that are typically in direct opposition to all our curricula?  How can we expect a small program to be an equal match for movies in which everyone is sexually active and unmarried and no one seems the worse for wear?  The answer points us to working as much at the cultural and institutional level of our nation as we do at the small program level.  Or, as we sometimes argue at the National Campaign to Prevent Teen Pregnancy, for every hour you spend working one-on-one with an at-risk young person, spend an hour also on changing the public conversation and culture in complementary ways.

References

Institute of Medicine. (1995).  The Best Intentions:  Unintended Pregnancy and Well-being of Children and Families. Washington, D.C.: National Academy Press.

Kirby, D. (1997).  No Easy Answers. Washington, D.C.: National Campaign to Prevent Teen Pregnancy.