EVENTS

Date:	Thursday, July 10, 2008
Time:	1:30 PM -- 3:00 PM
Location:	Wohlstetter Conference Center, Twelfth Floor, AEI 1150 Seventeenth Street, N.W., Washington, D.C. 20036

 

Holding Test-Based Accountability Programs Accountable

 

 

WASHINGTON, JULY 14, 2008--There is increasingly vocal concern, especially since the passage of the No Child Left Behind Act, that schools have started "teaching to the test." School districts around the country have created accountability programs--like Florida's A+ program--that sanction schools based on objective scores in "high-stakes" subjects like reading and math, and then assign schools a grade from A to F based on a continuous point scale that corresponds with achievement. While these programs have had positive effects on high-stakes subjects, there is little research on whether these programs have had negative effects on student performance in other important, so-called low-stakes subjects, like science, social studies, and the arts.

Though it seems intuitive that increased focus on reading and math would have negative effects on other subjects, a recent study by Marcus Winters of the Manhattan Institute--coauthored with Jay P. Greene and Julie Trivitt--found that the effects of "high-stakes" accountability programs may actually improve student proficiency in science. On July 10, Winters presented his findings at an American Enterprise Institute conference, explaining that while he "expected this crowd-out effect . . . what we're finding is that schools that face sanctions for not doing well on math and reading, students in those schools are making gains in science as well, even though the incentive is there for them to only focus on math and reading."

Winters described the methodology used to arrive at this "puzzling" result: "When you're studying the impact of any program in education, [you must] try and account for the heterogeneity of schools." Because measuring school achievement in different schools is hard, their paper, "The Impact of High-Stakes Testing on Student Proficiency in Low-Stakes Subjects," uses a regression-discontinuity model and acknowledges that the dichotomous letter-grade system drastically changes incentives for schools to perform. This study controls for the point system "so what we're really getting to is the effect the grade had on the school," Winters said

Beyond controlling for differences in schools, measuring proficiency in science is challenging because science tests are only administered in the fifth, eighth, and tenth grades. Due to the fact that a student did not take a science test in fourth grade, it is challenging to measure improvement in the fifth grade. To address this complex issue, Winters decided to use math and reading scores as a proxy for improvement in science. This approach has its limitations. As David Figlio of the University of Florida noted, even though reading and math scores are correlated, "it turns out that the contemporaneous correlations between reading and math and science aren't as big as we'd really love."

Limitations in the data and the relative newness of this topic leave room for future inquiry. While he acknowledged that Winters and his coauthors are "hands above the table" in terms of what they can do with the data, Figlio clarified that he is "not yet sure exactly what the takeaway message is." Since there is only one year of science data available, it is impossible to look at pre-treatment variables and long-term effects. What is also unclear is whether this "spillover" learning effect will apply to topics other than science, like social studies and the arts. It may be the case, as Winters explained, that "science equals reading plus math."

Additionally, belief that proficiency levels will continue to improve, simply based on these results, is misguided. Winters's study used a proficiency-output approach, essentially ignoring the behaviors that produced the results. "Are teachers changing their behavior? We don't care exactly what teachers are doing in classrooms; we care about the output at the end," he added.

Understanding what is contributing to these positive results is important. Right now "we don't have a good idea of what is driving these positive effects," said Jane Hannaway of the Urban Institute. There are valid disagreements on whether increased time on tested subjects is good or bad, whether curriculum has narrowed, and what the future holds under accountability programs. But, as Hannaway explained, "the bottom line . . . is that the incentives associated with test-based accountability systems are indeed powerful, and we had better get them right."

--JON FLUGSTAD

For video, audio, and event information, visit www.aei.org/event1748/. For more information about AEI's National Research Initiative, visit www.aei.org/nri/.

For media inquiries, contact Véronique Rodman at 202.862.4870 or vrodman@aei.org.

###