Data: No deus ex machina

Standardized test by

  • Title:

    Cage-Busting Leadership
  • Format:

  • Paperback Price:

  • Paperback ISBN:

  • 256 Paperback pages
  • Buy the Book

Article Highlights

  • Data can be a powerful tool. But we must recognize that collecting data is not using data.

    Tweet This

  • It seems clear that would-be reformers have consistently overestimated the potential of data.

    Tweet This

  • How would a #cagebusting leader use data in school systems?

    Tweet This

  • Would-be reformers too often suggest that data is going to resolve long-standing problems.

    Tweet This


Data-based decision-making is all the rage. Secretary of Education Arne Duncan (2009) has emphatically declared, "I am a deep believer in the power of data to drive our decisions. Data gives us the roadmap to reform. It tells us where we are, where we need to go, and who is most at risk." In the past few years, all 50 states have adopted most or all of the Data Quality Campaign's framework for state data systems.

In important respects, this is a welcome development. Data expose inequities, create transparency, and help drive organizational improvement.

But something is amiss. Many educators regard talk of data-based decision-making as an external imposition, sensing new obligations and what they see as a push to narrow schooling to test scores and graduation rates. Districts remain hidebound and bureaucratic, with precious few looking like data-informed learning organizations. And the data—which are relatively crude, consisting mostly of reading and math scores—are unequal to the heavy weight they're asked to bear.

Despite these challenges, enthusiasts continue to make sweeping claims about the restorative power of data. Too often, as we talk to policymakers, system leaders, funders, advocates, and vendors, we get a whiff of deus ex machina, the theatrical trick of having a god drop from the heavens to miraculously save the day. (The phrase's literal meaning is "God in the machine.") Like a Euripides tragedy in which an unforeseen development bails out the playwright who has written himself into a corner, would-be reformers too often suggest that this wonderful thing called "data" is going to resolve stubborn, long-standing problems.

We need a more measured view. Data can be a powerful tool. But we must recognize that collecting data is not using data; that data are an input into judgment rather than a replacement for it; that data can inform but not resolve difficult questions of politics and values; and that we need better ways to measure what matters, rather than valuing those things we can measure.

We've Been Here Before

Data have long promised easy answers, sometimes with discomfiting results. Frederick Kelly created the first modern multiple-choice test in 1914 (Murdoch, 2007). Others quickly followed suit. Edward Thorndike and Charles Judd devised achievement tests in spelling, handwriting, arithmetic, composition, and more (Butts & Cremin, 1953). By 1923, more than 300 standardized scales were available (Cubberley, 1919).

Stanford's iconic dean of education, Ellwood Cubberley (1919), cheered such assessments, insisting, "We can now measure an unknown class and say, rather definitely, that, for example, the class not only spells poorly but is 12 percent below standard" (p. 694). Cubberley explained,

Standardized tests have meant nothing less than the ultimate changing of school administration from guesswork to scientific accuracy. The mere personal opinions of school board members and the lay public … have been in large part eliminated. (p. 698)

Consider the IQ test, created to help sort new recruits mobilized for World War I. The U.S. government asked elite psychology professors to develop a system for gauging intelligence. In hindsight, some of the results were unreliable. In one analysis, testing expert H. H. Goddard identified 83 percent of Jews, 80 percent of Hungarians, and 79 percent of Italians as "feeble-minded" (Mathews, 2006). In one 1921 study, Harvard researcher Robert Yerkes concluded that "37 percent of whites and 89 percent of negroes" could be classified as "morons" (Gould, 1981, p. 227). Yerkes had no concerns about the results because the tests were "constructed and administered" to address potential biases and were "definitely known to measure native intellectual ability" (Graham, 2005, p. 48).

In the 1960s and 1970s, proponents of data and accountability again insisted that they had it right. U.S. Office of Education Associate Commissioner Leon Lessinger (1970) promised,

Once we have standardized, reliable data on the cost of producing a variety of educational results … legislators and school officials will at last be able to draw up budgets based on facts instead of on vague assertions. Through knowledge gained in the process of management, we will also be able to hold the schools accountable for results. (p. 10)

Lessinger was hardly alone; more than 4,000 books and articles on data and education accountability were published in the late 1960s and early 1970s (Browder, 1975). Yet in 2001, No Child Left Behind's architects started from the bipartisan conviction that U.S. schooling was nearly bereft of good data.

In hindsight, it seems clear that would-be reformers have consistently overestimated the potential of data and have used new data in inappropriate and troubling ways. We'd do well to keep this in mind if we intend to do more than repeat past mistakes.

The full text of this article is available via subscription at Education Leadership.

Frederick M. Hess is director of education policy studies at the American Enterprise Institute and writes the Rick Hess Straight Up blog for Education Week. Jal Mehta is an assistant professor at the Harvard Graduate School of Education and coeditor, with Robert B. Schwartz and Frederick M. Hess, of "The Futures of School Reform" (Harvard Education Press, 2012).

Also Visit
AEIdeas Blog The American Magazine
About the Author


Frederick M.
  • An educator, political scientist and author, Frederick M. Hess studies K-12 and higher education issues. His books include "Cage-Busting Leadership," "Breakthrough Leadership in the Digital Age," "The Same Thing Over and Over," "Education Unbound," "Common Sense School Reform," "Revolution at the Margins," and "Spinning Wheels." He is also the author of the popular Education Week blog, "Rick Hess Straight Up." Hess's work has appeared in scholarly and popular outlets such as Teachers College Record, Harvard Education Review, Social Science Quarterly, Urban Affairs Review, American Politics Quarterly, The Chronicle of Higher Education, Phi Delta Kappan, Educational Leadership, U.S. News & World Report, National Affairs, the Washington Post, the New York Times, The Wall Street Journal, the Atlantic and National Review. He has edited widely cited volumes on the Common Core, the role of for-profits in education, education philanthropy, school costs and productivity, the impact of education research, and No Child Left Behind.  Hess serves as executive editor of Education Next, as lead faculty member for the Rice Education Entrepreneurship Program, and on the review boards for the Broad Prize in Urban Education and the Broad Prize for Public Charter Schools. He also serves on the boards of directors of the National Association of Charter School Authorizers and 4.0 SCHOOLS. A former high school social studies teacher, he teaches or has taught at the University of Virginia, the University of Pennsylvania, Georgetown University, Rice University and Harvard University. He holds an M.A. and Ph.D. in Government, as well as an M.Ed. in Teaching and Curriculum, from Harvard University.

    Follow AEI Education Policy on Twitter

    Follow Frederick M. Hess on Twitter.

  • Email:
  • Assistant Info

    Name: Sarah DuPre
    Phone: 202-862-7160

What's new on AEI

image The Census Bureau and Obamacare: Dumb decision? Yes. Conspiracy? No.
image A 'three-state solution' for Middle East peace
image Give the CBO long-range tools
image The coming collapse of India's communists
AEI on Facebook
Events Calendar
  • 21
  • 22
  • 23
  • 24
  • 25
Wednesday, April 23, 2014 | 12:00 p.m. – 1:30 p.m.
Graduation day: How dads’ involvement impacts higher education success

Join a diverse group of panelists — including sociologists, education experts, and students — for a discussion of how public policy and culture can help families lay a firmer foundation for their children’s educational success, and of how the effects of paternal involvement vary by socioeconomic background.

Thursday, April 24, 2014 | 12:00 p.m. – 1:30 p.m.
Getting it right: A better strategy to defeat al Qaeda

This event will coincide with the release of a new report by AEI’s Mary Habeck, which analyzes why current national security policy is failing to stop the advancement of al Qaeda and its affiliates and what the US can do to develop a successful strategy to defeat this enemy.

Friday, April 25, 2014 | 9:15 a.m. – 1:15 p.m.
Obamacare’s rocky start and uncertain future

During this event, experts with many different views on the ACA will offer their predictions for the future.   

No events scheduled today.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.