Improbable scholars, inappropriate conclusions

School by

Article Highlights

  • Improbable Scholars lacks the very humility and appreciation for nuance that the educators it chronicles possess.

    Tweet This

  • The plural of anecdote is not data. McShane reviews Improbable Scholars via @educationnext.

    Tweet This

  • I don’t think that we can draw any broader policy recommendations from Improbable Scholars.

    Tweet This

Improbable Scholars: The Rebirth of a Great American School System and a Strategy for America’s Schools
By David L. Kirp
Oxford University Press, 2013, $24.95; 272 pages

As reviewed by Michael McShane

The book jacket of David Kirp’s Improbable Scholars describes it as a “playbook—not a prayer book—for reform that will dramatically change our approach to reviving public education.” Kirp sees the beating-the-odds story of Union City, New Jersey—a high-poverty school district that turns out high test scores and graduation rates—as a challenge to the agenda of “education reformers.”

Union City, Kirp argues, is successful because it fosters a culture of continuous improvement of existing staff and schools, utilizes bilingual instruction, and guarantees high quality pre-school—eschewing high stakes accountability, Teach for America teachers, and school choice. He profiles in depth the efforts of school leaders to instill a culture of abrazos—caring—while still pushing students to achieve to their fullest potential.

Unfortunately, as the saying goes, the plural of anecdote is not data. Kirp selected a district that was already doing well, and then tried to back out practices that made it succeed where other fail. But what about other districts that are doing the same things as Union City but with poor results? What about districts that are doing the opposite of Union City and succeeding? Is Union City an outlier because of some other variable that this single case study fails to take into account? Is Union City the rule or the exception?

Kirp’s analysis lacks the power to answer these questions because it violates a central tenet of social science research—it selects on the dependent variable. Though he adds three more cities in his conclusion, they cannot shed any additional light on causality because of the way they were chosen.

I have zero doubt that the Union City school district is a great institution. The teachers Kirp profiles are rock stars and heroes and should be applauded for their work. I just don’t think that we can draw any broader policy recommendations from this book.

But even beyond that, Kirp’s clear political agenda leads to distracting asides. Early in the book, when describing the history and context of Union City, he sneaks in two paragraphs about how it is wrong to oppose the DREAM act, before settling back in to describe the number of students that receive subsidized lunches. A page later Kirp asserts that “you won’t find any Teach for America recruits on a mission to reinvent urban education” at Washington Elementary School, unnecessarily knocking folks who have nothing to do with this story. The author takes every chance he gets to slight New Jersey Governor Chris Christie, whom he says “made a name for himself by beating up on educators.” These comments are gratuitous and unnecessary.

Also, Kirp says some things that are just wrong.

On page 212, he inaccurately summarizes the research on school vouchers. First, he uses a 2002 GAO study to say that students who receive vouchers fare no better than comparable public school students, even though a veritable mountain of evidence to the contrary has been published since then. Also, he describes a 2012 study of the Milwaukee Parental Choice Program by John Witte and Patrick Wolf as the “best-designed” study of school vouchers. That just isn’t true. The Witte/Wolf study used a matched comparison design, following a cohort of students that participated in the program and comparing them to a similar set of students in the Milwaukee Public Schools. But in any evaluation where there is fear of selection bias, matched comparison designs are inferior to randomized control trials. There is a huge risk of selection bias in school voucher research, so while the Witte/Wolf study is useful, and used the best possible method given the limitations of the program as it was implemented, it is not the “best-designed” study of school vouchers. What’s more, there have been 10 random assignment studies of school vouchers (the best design possible) and 9 of them have found positive results for some or all participating students.

Kirp also says on page 6 that California is a “textbook example” of white voters refusing to tax themselves to pay for schools that primarily educate poor and minority students. I’m not sure how to square that with the fact that California passed Proposition 30 by a 55/45 margin in November, increasing both income and sales taxes to raise nearly $7 billion per year for schools, minority children and all.

Those complaints aside, what is most disheartening about this book is that it lacks the very humility and appreciation for nuance that the educators it chronicles possess. The assurance with which Kirp offers his policy recommendations, going so far as to say that they’d look familiar to “any educator with a pulse,” stands in stark contrast to the approach of the teachers and school leaders who adapt to the needs of their students. We can all learn from their example.

Michael McShane is a research fellow at AEI and co-author of President Obama and Education Reform: The Personal and the Political.

Also Visit
AEIdeas Blog The American Magazine
About the Author


Michael Q.

What's new on AEI

AEI Election Watch 2014: What will happen and why it matters
image A nation divided by marriage
image Teaching reform
image Socialist party pushing $20 minimum wage defends $13-an-hour job listing
AEI on Facebook
Events Calendar
  • 27
  • 28
  • 29
  • 30
  • 31
Monday, October 27, 2014 | 10:00 a.m. – 11:30 a.m.
State income taxes and the Supreme Court: Maryland Comptroller v. Wynne

Please join AEI for a panel discussion exploring these and other questions about this crucial case.

Tuesday, October 28, 2014 | 9:30 a.m. – 12:15 p.m.
For richer, for poorer: How family structures economic success in America

Join Lerman, Wilcox, and a group of distinguished scholars and commentators for the release of Lerman and Wilcox’s report, which examines the relationships among and policy implications of marriage, family structure, and economic success in America.

Tuesday, October 28, 2014 | 5:30 p.m. – 7:00 p.m.
The 7 deadly virtues: 18 conservative writers on why the virtuous life is funny as hell

Please join AEI for a book forum moderated by Last and featuring five of these leading conservative voices. By the time the forum is over, attendees may be on their way to discovering an entirely different — and better — moral universe.

Thursday, October 30, 2014 | 2:00 p.m. – 3:00 p.m.
A nuclear deal with Iran? Weighing the possibilities

Join us, as experts discuss their predictions for whether the United States will strike a nuclear deal with Iran ahead of the November 24 deadline, and the repercussions of the possible outcomes.

Thursday, October 30, 2014 | 5:00 p.m. – 6:15 p.m.
The forgotten depression — 1921: The crash that cured itself

Please join Author James Grant and AEI senior economists for a discussion about Grant's book, "The Forgotten Depression: 1921: The Crash That Cured Itself" (Simon & Schuster, 2014).

No events scheduled this day.
No events scheduled this day.
No events scheduled this day.
No events scheduled today.
No events scheduled this day.