Improbable scholars, inappropriate conclusions

School by Shutterstock.com

Article Highlights

  • Improbable Scholars lacks the very humility and appreciation for nuance that the educators it chronicles possess.

    Tweet This

  • The plural of anecdote is not data. McShane reviews Improbable Scholars via @educationnext.

    Tweet This

  • I don’t think that we can draw any broader policy recommendations from Improbable Scholars.

    Tweet This

Improbable Scholars: The Rebirth of a Great American School System and a Strategy for America’s Schools
By David L. Kirp
Oxford University Press, 2013, $24.95; 272 pages

As reviewed by Michael McShane

The book jacket of David Kirp’s Improbable Scholars describes it as a “playbook—not a prayer book—for reform that will dramatically change our approach to reviving public education.” Kirp sees the beating-the-odds story of Union City, New Jersey—a high-poverty school district that turns out high test scores and graduation rates—as a challenge to the agenda of “education reformers.”

Union City, Kirp argues, is successful because it fosters a culture of continuous improvement of existing staff and schools, utilizes bilingual instruction, and guarantees high quality pre-school—eschewing high stakes accountability, Teach for America teachers, and school choice. He profiles in depth the efforts of school leaders to instill a culture of abrazos—caring—while still pushing students to achieve to their fullest potential.

Unfortunately, as the saying goes, the plural of anecdote is not data. Kirp selected a district that was already doing well, and then tried to back out practices that made it succeed where other fail. But what about other districts that are doing the same things as Union City but with poor results? What about districts that are doing the opposite of Union City and succeeding? Is Union City an outlier because of some other variable that this single case study fails to take into account? Is Union City the rule or the exception?

Kirp’s analysis lacks the power to answer these questions because it violates a central tenet of social science research—it selects on the dependent variable. Though he adds three more cities in his conclusion, they cannot shed any additional light on causality because of the way they were chosen.

I have zero doubt that the Union City school district is a great institution. The teachers Kirp profiles are rock stars and heroes and should be applauded for their work. I just don’t think that we can draw any broader policy recommendations from this book.

But even beyond that, Kirp’s clear political agenda leads to distracting asides. Early in the book, when describing the history and context of Union City, he sneaks in two paragraphs about how it is wrong to oppose the DREAM act, before settling back in to describe the number of students that receive subsidized lunches. A page later Kirp asserts that “you won’t find any Teach for America recruits on a mission to reinvent urban education” at Washington Elementary School, unnecessarily knocking folks who have nothing to do with this story. The author takes every chance he gets to slight New Jersey Governor Chris Christie, whom he says “made a name for himself by beating up on educators.” These comments are gratuitous and unnecessary.

Also, Kirp says some things that are just wrong.

On page 212, he inaccurately summarizes the research on school vouchers. First, he uses a 2002 GAO study to say that students who receive vouchers fare no better than comparable public school students, even though a veritable mountain of evidence to the contrary has been published since then. Also, he describes a 2012 study of the Milwaukee Parental Choice Program by John Witte and Patrick Wolf as the “best-designed” study of school vouchers. That just isn’t true. The Witte/Wolf study used a matched comparison design, following a cohort of students that participated in the program and comparing them to a similar set of students in the Milwaukee Public Schools. But in any evaluation where there is fear of selection bias, matched comparison designs are inferior to randomized control trials. There is a huge risk of selection bias in school voucher research, so while the Witte/Wolf study is useful, and used the best possible method given the limitations of the program as it was implemented, it is not the “best-designed” study of school vouchers. What’s more, there have been 10 random assignment studies of school vouchers (the best design possible) and 9 of them have found positive results for some or all participating students.

Kirp also says on page 6 that California is a “textbook example” of white voters refusing to tax themselves to pay for schools that primarily educate poor and minority students. I’m not sure how to square that with the fact that California passed Proposition 30 by a 55/45 margin in November, increasing both income and sales taxes to raise nearly $7 billion per year for schools, minority children and all.

Those complaints aside, what is most disheartening about this book is that it lacks the very humility and appreciation for nuance that the educators it chronicles possess. The assurance with which Kirp offers his policy recommendations, going so far as to say that they’d look familiar to “any educator with a pulse,” stands in stark contrast to the approach of the teachers and school leaders who adapt to the needs of their students. We can all learn from their example.

Michael McShane is a research fellow at AEI and co-author of President Obama and Education Reform: The Personal and the Political.

Also Visit
AEIdeas Blog The American Magazine
About the Author

 

Michael Q.
McShane
  • Michael Q. McShane is a research fellow in education policy studies at AEI. He is the coeditor, with Frederick Hess, of "Common Core Meets Education Reform" (Teachers College Press, 2013). He is also the coauthor of "President Obama and Education Reform: The Personal and the Political" (Palgrave MacMillan, 2012). His analyses have been published widely in technical journals and reports including Education Finance and Policy. He has contributed to more popular publications such as Education Next, The Huffington Post, National Review, The Chronicle of Higher Education, and the St. Louis Post-Dispatch. He began his career as an inner-city high school teacher in Montgomery, Alabama.


    Follow Mike McShane on Twitter.

  • Phone: 202-862-5838
    Email: Michael.McShane@aei.org

What's new on AEI

image The money in banking: Comparing salaries of bank and bank regulatory employees
image What Obama should say about China in Japan
image A key to college success: Involved dads
image China takes the fight to space
AEI on Facebook
Events Calendar
  • 21
    MON
  • 22
    TUE
  • 23
    WED
  • 24
    THU
  • 25
    FRI
Wednesday, April 23, 2014 | 12:00 p.m. – 1:30 p.m.
Graduation day: How dads’ involvement impacts higher education success

Join a diverse group of panelists — including sociologists, education experts, and students — for a discussion of how public policy and culture can help families lay a firmer foundation for their children’s educational success, and of how the effects of paternal involvement vary by socioeconomic background.

Event Registration is Closed
Thursday, April 24, 2014 | 12:00 p.m. – 1:30 p.m.
Getting it right: A better strategy to defeat al Qaeda

This event will coincide with the release of a new report by AEI’s Mary Habeck, which analyzes why current national security policy is failing to stop the advancement of al Qaeda and its affiliates and what the US can do to develop a successful strategy to defeat this enemy.

Friday, April 25, 2014 | 9:15 a.m. – 1:15 p.m.
Obamacare’s rocky start and uncertain future

During this event, experts with many different views on the ACA will offer their predictions for the future.   

No events scheduled this day.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.
No events scheduled this day.