|Education Outlook logo 130|
No. 6, May 2009
Although the premise underlying "educational adequacy" lawsuits--that more court-ordered resources will result in higher outcomes--is accepted by many in the education community, almost no one has seriously examined the empirical evidence to determine its validity. Several of the adequacy remedies ordered by the courts have been in place for a decade or more, yet no one, to our knowledge, has compared the pre- and post-remedy student outcome data to determine whether student achievement has actually improved in response to the significantly increased funding and resources made available by virtue of the courts' orders. Instead, most commentators point out the obvious--that increased funding has led to additional programs and personnel and new and improved facilities. The most important question--has student achievement improved as a result of the court interventions?--remains a subject barely addressed by the education research community.
Nor have the courts seemed too interested in this vital question. While courts are often guided in their decisions by what judges in other states have done, rarely has any court considered whether the remedies in other states have actually produced improved student achievement. Rather than examining the relevant achievement outcomes of other states, judges tend to rely on the testimony of expert witnesses and school personnel who tout their own pet programs and predict significantly better outcomes if such programs are funded and properly implemented. Indeed, even when judging the effectiveness of their own previously ordered remedies, courts rarely examine the remedy's effect on student achievement. For example, the Wyoming Supreme Court, after more than a decade of unprecedented funding increases for Wyoming's public schools, declared that the state legislature was in compliance with the state constitution. Yet, in its related lengthy 2008 opinion, the court barely discussed whether student performance had improved. The court's only words on the subject were that "at the time of trial Wyoming ranked as one of the highest states in the nation for schools making adequate yearly progress under the federal No Child Left Behind programs." This was hardly surprising, given Wyoming's relatively advantaged student population. No inquiry was made to determine if achievement levels in Wyoming had actually improved during the course of the extensive remedy in any meaningful way.
A few scholars have attempted to determine the effect of court remedies on student achievement, but these studies are handicapped by changing state tests and typically concentrate on only a relatively brief period when such remedies have been in place. No attention has been given to the obvious question: how are students doing now compared with their performance prior to the implementation of the remedy?
Four states--Kentucky, Wyoming, New Jersey, and Massachusetts--implemented substantial court-ordered remedies a decade or more ago, providing an opportunity to evaluate the impact of significant judicial interventions over several years. It is nearly impossible to compare each state's achievement tests reliably. All four states, however, have participated in the National Assessment of Educational Progress (NAEP) testing program (known as the nation's report card) for enough years that we can examine their students' pre- and post-remedy achievement levels. Using NAEP test scores, we can compare changes in student achievement levels among these states and in relation to students nationwide. Moreover, scores on the federally administered NAEP tests are less susceptible to manipulation by state authorities eager to augment results.
During the period of time we examine--from 1992 to 2007--NAEP scores in elementary and middle schools increased in almost every U.S. state, regardless of whether the state undertook reforms mandated by adequacy lawsuits. Thus, improved scores by themselves tell us little; the more telling indicator of success is how well students in each of these four states performed on NAEP in comparison to other states in the national pool. Given the substantial funding increases in these four states, we might expect these students to have outperformed their "nonadequacy" peers by a significant margin.
In examining test scores, we are most concerned about the achievement scores of at-risk students--the economically disadvantaged and minority students who are the primary target of adequacy remedies. Although pre-remedy NAEP scores broken down for economically disadvantaged students are not available, NAEP does report 1992 scores for black and Hispanic students in fourth grade reading and math and eighth grade math. Using the NAEP data, we can determine whether the relative achievement of black and Hispanic children in reading and math has improved since the inception of the remedy in each state.
Kentucky has long been cited by proponents of judicial intervention as a model of what adequacy cases can accomplish. The 1989 Rose decision found that the Kentucky education system did not comply with the state constitution and ordered, among other things, structural changes and increased funding. This case also heralded in the golden age of successful adequacy litigation, lasting from 1990 to 2004. However, the plaudits showered on the Kentucky remedy relate mostly to the structural changes it brought about and not to success in improving the achievement of students. On this front, the NAEP scores reflect little or no progress.
We began our analysis with the state's largest minority group. Black students constitute about 11 percent of the state's public school enrollment and perennially score significantly below the state's white students on standardized tests. Unfortunately, the adequacy remedy has done little to reverse this persistent black-white achievement gap. If anything, Kentucky's black students have fallen even further behind the nation during the course of the remedy. As indicated in figure 1, from 1992, when the remedy began, through 2007, black students' scores increased on each of the three tests, but at a rate significantly lower than black students' scores nationwide. For example, in 1992, the state's fourth grade reading scores for black students were five points above the national average for black students; by 2007, they had dropped to the national average. In math, fourth graders were eight points above the national average for black students in 1992 but had fallen to three points below it by 2007. Kentucky's black eighth grade students fared no better. In 1992, they were noticeably above the national average for black students; by 2007, they had fallen below it.
EduO No. 6, May 2009 Figure 1
As shown in figure 2, the state's white students' scores did improve somewhat over the course of the remedy, but their scores on all three tests remain significantly below the national average for white children. From 1992 through 2007, their scores increased slightly more than the national average for white students on two tests (fourth grade reading and eighth grade math) and mirrored the national gain on one test (fourth grade math).
EduO No. 6, May 2009 Figure 2
A 2007 analysis of NAEP scores in Kentucky concludes that since such scores have increased in the state since 1998 in reading and since 2000 in math for all students--including black students and economically disadvantaged students--the remedy may be having a positive effect. However, as noted earlier, NAEP scores in virtually every state increased during this period, regardless of whether the state's students enjoyed significantly enhanced funding or other remedial measures; therefore, the mere fact that the scores increased tells us little about the success of the remedy. Moreover, using these same test scores, time period, and subgroups of children, we still find that Kentucky's gains were significantly below the national average on six tests, about the same on five tests, and above the national gains on only one test.
Perhaps the most dramatic court intervention occurred in Wyoming. In 1995, the Wyoming Supreme Court found the education funding system unconstitutional and decreed that the legislature provide whatever funds necessary to make education in the state the "best." A compliant legislature flush with tax revenues from the state's mineral interests responded with massive increases in K-12 education appropriations, which drove Wyoming's per-pupil funding from just over the national average to among the highest in the nation, particularly when adjusted for regional cost differences. Despite these unprecedented increases in school funding, the achievement of Wyoming's students has largely failed to keep up with the nation or even with its much lower-funded, although demographically similar, neighboring states.
One of the primary targets of Wyoming's judicial remedy was the significant achievement gap between white and Hispanic children, the state's largest minority group. As figure 3 indicates, from 1992 to 2007, the increases achieved by Hispanic students in Wyoming were less impressive than the increased achievement by Hispanic students in the rest of the country on two of the three tests. While fourth grade scores in both reading and math increased, they did so at only about half the rate for Hispanic students nationwide. In the eighth grade, the gains on the math test made by the state's Hispanic students surpassed the gains made nationally by Hispanic students.
EduO No. 6, May 2009 Figure 3
White students constitute almost 90 percent of the student population in Wyoming, so we also examined their performance during the course of the remedy to see if success could be claimed for the vast majority of the students in the state, regardless of the disappointing results for its largest minority group. However, as figure 4 shows, the state's white students failed to keep pace with white students nationally. On all three tests, the average test score increases in the nation for white students were greater than those achieved by Wyoming's white students. For example, in 1992, Wyoming's fourth graders were two points above the national average in reading; by 2007, they were two points below it.
EduO No. 6, May 2009 Figure 4
New Jersey has been involved in school finance litigation over its K-12 education system since 1970. In both duration and expense, New Jersey's remedial efforts stand out. New Jersey's most recent judicial remedy, developed in the mid-1990s, is different than the other three states we evaluate because it is not statewide; it applies to only thirty-one of the state's poorest school districts (out of over six hundred school districts in the state). These thirty-one "Abbott districts," named after the court case, are the only beneficiaries of over $1.5 billion a year in additional funding under the court order. As a result, they are by far the highest-funded school districts in an already high-spending state, outspending both the wealthiest and other poor districts by thousands of dollars per student.
The Abbott districts were chosen because they had concentrated poverty and because they were providing poor education as identified by test scores and other measures of school differences. Comparable districts in terms of poverty but that provided a better education did not receive the Abbott designation and were not entitled to the extraordinary funding. Thus, in New Jersey, the worse the education provided in a school district, other things being equal, the more money the district gets--hardly the kind of incentive that promotes high achievement.
NAEP does not report scores separately for the thirty-one Abbott districts. Nevertheless, we believe New Jersey's NAEP scores can tell us a lot about the success of the Abbott remedy. The Abbott districts educate approximately half of the black and Hispanic students in the state. Once again we can compare test scores between 1992, before the bulk of the remedial dollars began to pour into the Abbott districts, and 2007, after the districts had enjoyed many years of higher funding and when the most recent test results became available.
The picture we find is a mixed one, with little evidence that the state's black students have progressed much, if at all, relative to black students nationwide, but with some positive indications that its Hispanic students have made significant progress. As shown in figure 5, from 1992 through 2007, the scores for fourth grade black students in New Jersey on the NAEP reading test increased by fourteen points, slightly more than the twelve-point increase nationwide among black students during the same period. On the math test, the state's students gained thirty-four points, compared to a national gain of thirty points. On the surface, New Jersey's black fourth grade students appear to have done a little better than the national average over the fifteen years of the remedy. One would be hard-pressed, however, to argue from these numbers that the remarkable levels of funding enjoyed by the Abbott districts (in which spending per pupil can exceed $20,000 per year) have led to any dramatic improvements. At best, New Jersey's black students appear to be holding their own.
EduO No. 6, May 2009 Figure 5
However, even that modest assessment is problematic because it essentially depends on one data point--the 2007 test scores. Virtually all of the progress made by the state's black students has occurred in the last two years--from 2005 to 2007. Before 2005, the state's black fourth graders lost significant ground, gaining only a single point from 1992 through 2005 on the fourth grade reading test, compared to a national gain of eight points among black students during the same period. The same was true on the math test, in which the gains made by the state's black fourth graders also lagged behind the nation through 2005. On the 2007 test, reading scores increased by thirteen points, more than in any other state, and math scores by nine points, making up for the lack of progress made during the first thirteen years of the period and putting them slightly ahead of the progress made nationally. Proponents of the remedy rely on the 2007 scores to argue that the court remedies are finally having a positive effect. We hope that is the case, but even accepting the 2007 scores at face value, the NAEP tests indicate only modest progress by the black fourth grade students. Moreover, virtually no one claims any significant progress at the middle school level. As indicated in figure 5, the scores of the state's black eighth grade students increased twenty-two points from 1992 through 2007, while nationally their scores rose twenty-three points.
On a brighter note, New Jersey's Hispanic students, almost half of whom go to school in the Abbott districts, appear to have made more significant progress since the remedy began. As reflected in figure 6, on all three of the NAEP tests we examined, the state's Hispanic students registered higher gains from 1992 to 2007 than Hispanic students nationwide. The performance of the state's Hispanic students on the fourth grade reading and eighth grade math tests was especially notable. Moreover, unlike the black students' scores, there were no extraordinary gains on the 2007 test that cast doubt on the validity of the patterns we observed.
EduO No. 6, May 2009 Figure 6
In 2008, New Jersey enacted a new financing formula, and the case returned to the courts to determine whether the special funding status of the Abbott districts should continue. A May 29, 2009, ruling by the New Jersey Supreme Court appears to signal an eventual end to the Abbott case. Although retaining supervision for three years, the court has placed a limit on the increases that go to Abbott districts and has called for integrating the Abbott districts with other districts in the state under its new funding formula.
Massachusetts is, for the most part, the success story among the four states we examined. In 1993, the Massachusetts legislature enacted a host of reforms in response to the McDuffy case, declaring its system of education unconstitutional. Over the next decade, the legislature tripled the funding of the state's public schools--from approximately $3 billion to $10 billion--and adopted a number of other strong reform measures including "a rigorous regimen of academic standards, graduation exams, and accountability." Thus, Massachusetts is not an example of pure funding increases but instead of fundamental changes in schools along with increased resources. To us, this is very different, and the results confirm that perspective.
Over the course of these remedies, the achievement scores of white and Hispanic students in Massachusetts have outpaced the comparable national scores. As indicated in figure 7, the state's white students have done particularly well, posting test score increases well above the national average, particularly in eighth grade math. When the remedy began, the state's white eighth graders' scores were only one point above the national average; by 2007, they were fifteen points above the national average.
EduO No. 6, May 2009 Figure 7
Massachusetts's Hispanic students, one of the state's two large minority groups, also made impressive gains compared to Hispanic students nationwide. As figure 8 shows, the math scores of the state's Hispanic fourth graders and eighth graders moved from several points below the national average to several points above during the course of the remedy.
EduO No. 6, May 2009 Figure 8
This pattern of progress, however, does not hold true for the state's black students, its second largest minority group. While Massachusetts's black students' scores exceed the national average for black students, these scores were even further above the national average in 1992--when the remedy commenced on two of the three tests (fourth grade reading and eighth grade math). Therefore, it is difficult to see what, if any, progress black students have made as a result of the remedy relative to white students or to other states (see figure 9). In 1992, the state's white-black achievement gap on eighth grade math was thirty-four points, compared to a national gap of forty points. By 2007, the national gap had decreased to thirty-one points, while Massachusetts's gap had widened to forty-one points. This may not be quite as bad as it seems because part of the gap-widening was caused by significant gains in white students' scores, as opposed to decreased black students' scores, but the state's white-black achievement gap remains a serious problem that the remedy has failed to alleviate.
EduO No. 6, May 2009 Figure 9
The results in three of the states--Kentucky, Wyoming, and New Jersey--are disappointing for anyone hoping that court-ordered funding increases will lead to dramatic student achievement gains. This is especially true for black students: we found no evidence in these four states of significant progress by black students. Indeed, in both Massachusetts and Kentucky, black students have lost substantial ground as compared to national averages during the course of these judicial remedies.
The news is mixed for those states with large Hispanic student populations. In Wyoming, Hispanic students have not kept up with the nation as a whole, while in Massachusetts and New Jersey, their scores have improved relative to Hispanic students in the rest of the country. Results for white students are also varied. In Massachusetts, white students clearly did well compared with the white students across the country. In Wyoming and Kentucky, however, there is little evidence of significant progress among white students relative to the nation as a whole. (Understandably, few white students have benefited from the remedy in New Jersey because it is directed at the predominately minority Abbott districts.)
The remedies arising from adequacy lawsuits in these four states have been the most sweeping in the nation, both in terms of the time they spanned and the funding increases they generated. Advocates of court-ordered funding increases as a means to raising achievement, particularly among disadvantaged children, would logically turn to these states to find evidence for their cause. Unfortunately, the evidence does not support any substantial improvements in performance resulting from court-ordered or -induced remedies. Simply spending more on the existing system, whether brought about by court order or legislative action, has not yielded results. With more fundamental changes, however, Massachusetts indicates the possibility of obtaining true performance increases.
Eric A. Hanushek and Alfred A. Lindseth are the authors of Schoolhouses, Courthouses, and Statehouses: Solving the Funding-Achievement Puzzle in America's Public Schools (Princeton University Press, May 2009). This Outlook is based upon an analysis of the impact of judicial remedies more fully described in their book.
1. Campbell County School District, et al. v. State of Wyoming, et al., Docket No. 2008 WY 2 (S.C., Wyo. Jan. 8, 2008), p. 28.
2. See Margaret E. Goertz and Michael Weiss, "Money Order in the Court: The Promise and Pitfalls of Redistributing Educational Dollars through Court Mandates; The Case of New Jersey" (paper, Annual Meeting of the American Education Finance Association, Baltimore, MD, 2007).
3. The NAEP tests cannot be used for school accountability because they are given only to a sample of students in each participating state. Since 1992, however, the tests have been given to a representative sample of sufficiently large numbers of students so that they can be reliably used to compare states and subgroups of students within each state.
4. Rose v. Council for Better Education, 790 S.W.2d 186 (Ky. 1989).
5. Susan Perkins Weston and Robert F. Sexton, "Substantial and yet Not Sufficient: Kentucky's Effort to Build Proficiency for Each and Every Child" (working paper, "Equal Educational Opportunity: What Now?" symposium, Teachers College, Columbia University, New York, November 12-13, 2007), available at www.tc.edu/symposium/
symposium07/resource.asp (accessed May 28, 2009).
6. Campbell County School District v. Wyoming, 907 P.2d 1238 (Wyo. 1995).
7. A separate analysis, fully set forth in our recent book, Schoolhouses, Courthouses, and Statehouses: Solving the Funding-Achievement Puzzle in America's Public Schools, compares student performance using NAEP and a number of other outcome measures between Wyoming and three other states in the same region--Montana, South Dakota, and North Dakota. These states are very similar in most respects, being largely rural with demographically similar school enrollments. Yet, despite per-pupil expenditures approximately 20 percent higher than its three neighbors, the longer Wyoming's students are exposed to the state's schools, the further they lag behind the students of the other three states under almost any measure of achievement.
8. For a history of the New Jersey school finance litigation, see Abbott v. Burke, 710 A.2d 450 (N.J. 1998), pp. 455-60.
9. New Jersey typically ranks first, second, or third among the nation's fifty states in its per-pupil spending on K-12 education.
10. Although somewhat inconsistent with the NAEP scores, fourth grade scores on the state tests show some improvement in the Abbott districts in recent years. NAEP and state test scores continue to show little or no improvement at the eighth grade level. See Margaret E. Goertz and Michael Weiss, "Money Order in the Court: The Promise and Pitfalls of Redistributing Educational Dollars through Court Mandates; The Case of New Jersey."
11. McDuffy v. Secretary, Executive Office of Education, 615 N.E.2d 516 (Mass. 1993).
12. See the discussion in Robert M. Costrell, "The Winning Defense in Massachusetts," in School Money Trials: The Legal Pursuit of Educational Adequacy, ed. Martin R. West and Paul E. Peterson (Washington, DC: Brookings Institution Press, 2007), 278-304.
13. That white and Hispanic students have made significant strides forward in Massachusetts since the McDuffy decision seems clear; what is less certain is what role the increased funding had in such improvement. As in the other three states we examined, funding was substantially increased in the years following the court decision, but unlike those other states, Massachusetts also enacted and enforced a strong regimen of academic standards, a high-stakes graduation test, and strict accountability measures. Although these measures encountered stiff resistance, the Massachusetts legislature, using the window of opportunity opened by the McDuffy decision, found the political courage to pass them.
14. In the book on which this analysis is based--Schoolhouses, Courthouses, and Statehouses--we propose directly linking financing to important changes in policy and incentives through a system of performance-based funding.