American Enterprise Institute

May 21, 2007

[Edited transcript from audio tapes]

8:30 a.m.	Registration
9:00	Introduction:	Frederick M. Hess, AEI
9:10		Panel I: The Evolving Relationship between Research and Policy
	Presenters:	Jeffrey Henig, Columbia University Teachers College
		Andrew Rudalevige, Dickinson College
	Discussants:	Gina Burkhardt, Learning Point Associates Michael Feuer, National Research Council
		Ellen Condliffe Lagemann, Harvard University
10:35	Break
10:45		Panel II: How Research Is Used—Teacher Quality and Reading
	Presenters:	Richard Ingersoll, University of Pennsylvania
		James Kim, Harvard Graduate School of Education
	Discussants:	Reid Lyon, Higher Ed Holdings and Whitney International University
		Lorraine McDonnell, University of California, Santa Barbara
12:00 p.m.	Luncheon
12:45 p.m.		Panel III: How Research Is Used—NCLB and School Choice
	Presenters:	Michael J. Petrilli, Thomas B. Fordham Foundation
		Andrew Rotherham, Education Sector
	Discussants:	David Driscoll, Massachusetts Commissioner of Education
		Roberto Rodriguez, United States Senate HELP Committee
2:00	Break
2:10		Panel IV: How Research Is Used by the Public, the Courts, and Educational Leaders
	Presenters:	Lance Fusarelli, North Carolina State University
		Joshua Dunn, University of Colorado at Colorado Springs
	Discussants:	Pascal Forgione, Austin Independent School District
		William Howell, University of Chicago
		Warren Simmons, Annenberg Institute for School Reform
3:30	Break
3:40		Panel V: Changing the Incentives for Researchers and Decision-Makers
	Presenters:	Dan Goldhaber, University of Washington
		Kenneth Wong, Brown University
	Discussants:	Michael McPherson, Spencer Foundation
		Kathleen McCartney, Harvard Graduate School of Education
		Grover J. (Russ) Whitehurst, Institute of Education Sciences
5:10	Reception
6:00	Adjournment

Proceedings:

[Start of Panel 1:  The Evolving Relationship between Research & Policy]

Frederick M. Hess:  I’m Rick Hess, director of Education Policy Studies at the American Enterprise Institute.  I would like to welcome you all here today for this conference, The Politics of Knowledge:  Why Research Does (or Does Not) Influence Education Policy. Before we get started – we are going to get started in about five or ten minutes – I would like to take a few moments to explain why we have organized this conference and what I hope we might accomplish today. 

Before we do that, I think it is important to suggest that when we deal with research or knowledge, we are always building upon prior efforts.  And for those of you who are confused, this is actually not simply a reenactment of Ellen’s book The Politics of Knowledge.  We are simply appropriating a title because that is how we do things in the academy. 

Now, what are we talking about today?  Well, we have spent a great deal of time in recent years discussing the merits of various research methodologies and ways to enhance the rigor and the reliability of research and evaluation.  We have also spent a good deal of time discussing how to communicate findings to practitioners, to schools and to classrooms so that new findings will actually be utilized.  We have spent far less time thinking seriously about our efforts to promote the rigor and relevance of research, our position within the political and policy environment, about how political pressures shape the research agenda, how research is communicated or fed into the policy-making process, or how relationships between researchers and public officials and advocates and educators and the media affect the utility and the use of research.  Instead, we have been more likely to hear or to voice broad-brushed enunciations of the educational research community, of think tanks, or of politicized decision-making. 

Our focus today is on the nexus of research in policy-making, on the soft tissue that links these two worlds, at the institutions, norms, and incentives that shape the respective activity.  The guiding assumption is that we can get all the technical and scientific questions right and still not benefit from scholarly activity and conversely, that the value of research will be dramatically heightened if the incentive, the norms, and the institutions are sensibly constructed. 

I do not anticipate that anybody is going to leave today with simple answers.  I do hope that we leave better-equipped to address the question: If we really believe that valid and reliable research findings can and should influence education policy in various ways, what arrangements or norms will help to make it so? 

Similarly, I hope we might also be prepared to say that, if we believe that research should be fair-minded, carefully constructed, and skillfully executed and we are willing to go to where the data points us, and able to stretch our understanding to what we ought to be doing and how we ought to do it, what arrangements or norms will encourage the production of such work?  If we leave today with useful insights into these two questions, I will consider the day a success. 

With that, let me quickly give you the rundown on how the day is going to go.  What we are going to have is five panels, starting with the panel you see seated here. 

The first panel – We are going to look at the evolving relationship between research and policy and talk a bit about how we have gotten to where we are today.  The second panel – we are going to talk about how research is used or has been used in the case of teacher quality and reading.  The third panel – we are going to discuss how research has been used in the case of NCLB and school choice.  Panel four – We are going to think about how research has been used by the public, the courts, and educational leaders, and finally, we are going to see if we can get some larger take-aways for the day.  We are going to think about how we can change the incentives that face researchers and policy-makers. 

Each panel is going to feature a couple of authors and the discussants, all of them speaking for about 10 minutes a piece.  We then are going to open it up for conversation among the panelists and for Q&A.  During Q&A, I would ask that you please wait for the microphone.  We will have two mics circulating.  And then do us the favor of identifying yourself by name and affiliation.  I’ll also ask because I always do that we please avoid the proud DC tradition of asking a question by making a speech from the floor and then appending a question mark and I’ll ask that we will actually ask questions which can promote and advance the conversation.  It is understood that in a day like this that there are a number of people in the room that could quite readily sit up here.  I see Jim Kohlmoos out there.  I see a variety of folks who thought deeply about these issues.  I find these days tend to be much more productive, however, if we avoid the temptation to orate from the floor. 

The papers that are being presented will be revised and then published as a collective volume in early 2008.  Because the drafts that you see presented here today are still very much works in progress, I would invite audience members during the day to please feel free to corner authors and suggest corrections, offer thoughts and please challenge analyses and interpretations.  The entire set of papers is available online at the web address www.aei.org/event1455.  That is event 1455.  It is also really easy to find if you go to the AEI website.  For those of you who prefer not to deal with the web, we have got a full set of papers and hard copies outside where you came in the foyer and we have also got CDs with the full set of papers available for those who prefer something small and in their hand. 

Finally, before I turn to the first panel, I would like to thank The Spencer Foundation for helping to make this conference possible and I would like to thank the generous donors who support AEI’s education program more generally.  I would also like to thank Juliet Squire standing up here with the mike for her outstanding efforts in quarterbacking this whole thing and organizing it and in getting all these people safely and in orderly fashion in DC, so thanks, Julie. 

If you have questions during the day, Julie, hopefully can be able to help you out.  Also, Morgan Goatley right here, you can corner.  And Rosemary, where are you?  There you are.  Rosemary.  And all of them are far more competent than I so if you have any questions or concerns, hopefully they can help you address them. 

With that, we are going to roll right into the first panel.  Let me briefly make the introductions.  . 

Andy Rudalevige is an associate professor of political science at Dickinson College where he has taught since 2000.  In 2004-05, he was a visiting research scholar at Princeton University’s Center for the Study of Democratic Politics.  His first book Managing the President’s Program published by Princeton University Press won the American Political Science Associations Richard E. Neustadt Award. 

Jeff Henig - seated to Andy’s right - is a professor of political science and education at Teachers College at Columbia University and a professor of political science at Columbia University.  He is the author or co-author of seven books including the Color of School Reform: Race Politics and the Challenge of Urban Education and Mayors in the Middle:  Race Politics in Urban School Reform, both published by Princeton University Press. 

Our discussants for the first panel – Speaking first will be Ellen Lagemann.  Ellen is a Charles Warren professor of the History of American Education at Harvard University.  In July, she will become a distinguished fellow at the Bard Center where she will direct the Bard Center for Education and Democracy.  A historian of education, Ellen is a former dean of the Harvard Graduate School of Education and former president of The Spencer Foundation.

Speaking second will be Gina Burkhardt.  Gina is chief executive officer of Learning Point Associates.  Prior to becoming CEO of Learning Point in 2003, Gina served as executive director of the North Central Region Educational Laboratory at Cornell. 

And speaking last on this initial panel will be Michael Feuer.  Mike is executive director of the Division of Behavioral and Social Sciences and Education in the National Research Council of the National Academies.  Formerly at the National Research Council, Mike was the first director of the Center for Education and the founding director of the Board on Testing and Assessment.

With that, Andy, will you please get started? 

Andrew Rudalevige:  Good morning!  My name is Andy Rudalevige.  It is a great pleasure to be here this morning.  My assignment here today was to talk about historical and political topography of education research as it has been carried out by the federal government, most specifically, within the Department of Education and its various organizational ramifications.  I will not talk so much about other places in the government where education research is carried out – NSF, NIH, especially, but we will talk about those touchstones for the way in which education research has been conceived by successive generations of reformers. 

As you can see the paper is grandiosely titled Truth versus Partisanship, and you will see where that comes from in a minute.  For a Washington audience, I do not think I need to tell the story of the three envelopes.  Anyway, given our Draconian or I should say Hessian limits on time, I think I will have to skip over it but most of you know the punch line, which is that when someone brought into office needs to get advice from his or her predecessor, the predecessor leaves some envelopes on the desk.  The first one says, “Blame your predecessor.”  The second one says, “Reorganize.  Reorganize.  Reorganize.”  And the third one says, “Prepare three envelopes.” 

The three envelopes have been in constant use in the federal education research bureaucracy forever, really, but especially since 1965. I have considered the sequence of organizational manifestations of this function from the Bureau of Research consolidated in the Office of Education back in the mid-60s, the creation of an independent National Institute of Education back in 1972 that was then moved under a new office – the Office of Educational Research and Improvement, OERI, in 1980, when the new Department of Education came into being.  Five years later NIE was wiped out off the organizational chart – OERI sorry – filled that void itself.  A major reorganization less than ten years later tried to model OERI along lines of the National Institutes of Health with specific subject matter institutes under its umbrella.  And then less than a decade later, the creation of the Institute of Education Sciences, IES, its current manifestation. 

Now, all those changes do have an underlying theme and that is the quest for science.  This has been something as a grail for politicians especially.  You can see there recent quotes just from the 2001–2002 debate over the education sciences reformat, the very name of which gives you an idea of what political actors were trying to accomplish in this latest reorganization.  Again, the idea is to insulate federal research from partisan or political influences.  To look into this institute when we have education questions the same way that we look to the NIH when we have medical questions.  In this striving for a scientific agency which will be free of partisanship and produce real facts has been a hallmark of this debate, not just in the 2002 debate but going back well into American history. 

The title of the paper comes actually from a section of a report, an advisory committee to President Hoover, back in the early 1930s.  Section entitled (Truth versus Partisanship), so you can see the idea of the committee strongly influenced by notions of scientific management and the idea that there was one right way to do pretty much anything argued that we need to get beyond mere differences of opinion tenaciously held and we can do this how?  Well, by finding out facts, facts that are established by the scientific method and presented in understandable terms.  Clearly, something that would fit in very nicely with the idea of evidence-based interventions in the current round of educational accountability in research.  This was to be done organizationally as well.  The research function needed more clout.  Right, it needed enough status that they could present those facts forcibly both to the Congress and to the president. 

Now, in point of fact, educational research has had a hard time using force in quite that way.  One question of the paper – and I will not go into a huge amount of detail here – is the question of whether structure can in fact serve science in this context.  Terry Moe, a Stanford political scientist, wrote some years ago about the politics of bureaucratic structure and the difficulty in achieving insulation and autonomy from political forces when agencies are created or reorganized, right?  Because structure matters to outcomes, we should expect that structure itself is a variable that is contested.  That political actors of all stripes, widely construed, ought to be interested in what an agency looks like and in fact, you have seen that striving over time. 

In education research, rarely have you seen the insulation that you have seen in the hard sciences – so I’ll come back to that.  Partly, of course, as we’ll talk about more, education research is hard to pull off in a “scientific” manner.  Right, it is hard to do controlled experimental design in a lot of cases and besides that it is value-laden.  Right?  So a lot of the research agendas are driven by values and things that move past the normal realm of determining facts and that makes it inherently political and that is also meant to be politicized in a lot of ways.

Going back to the earliest incarnation of the Department of Education, it was actually originally a department at least in name – As early as 1867, it was supposed to collect statistics and facts but in fact, in time, it collected all sorts of weird extraneous functions including the reindeer service in the Alaskan territory, which dealt with the purchase and the breeding of reindeer for native populations in Alaska.  They did not get rid of this until the end of the 20th century.

In the modern era, the Office of Education, NIE, OERI, there has been something of a vicious cycle where incentives to politicize have been more common than incentives to insulate.  There has not been necessarily a lot of organized interest support except with the regional labs in the R&D centers which we can talk more about.  Legislative interest has been, perhaps for that reason, somewhat sporadic, except for the occasional opportunities to look for ear-marking and chances to benefit one’s district. 

There has been lots of turnover within the personnel structure of the Federal Research Establishment and partly as a cause, and partly as a consequence of that perhaps, research really has been seen as a political arm of the administration and as “unscientific” any way.  Carl Kaestle of course wrote about “the awful reputation of education research;” that certainly has not helped the function gain support legislatively or even administratively in a lot of cases.

What we have seen really is institute-envy.  We have had a repeated desire for NIH and the debate is going back to the Office of Education, NIE, OERI, and now IES, again you see the same arguments made  – Why can this not be more like NIH?  Sometimes NSF but NIH more regularly, given its multiple umbrella organizations.  Can we insulate it from partisan-buffering?  Can we have appointments that are not political?  Can we have fixed terms for example?  Keep them out of the normal political crush?  Can we have some advisory board that is prestigious and serves to buffer the organization from politics?  We need some rigorous peer review.  How can we make that happen? 

And we need lots of time and patience, and frankly, money. I’m from Massachusetts, so I can say this, we have got something like New Hampshire – NH, not NIH – Live free or die.  And in fact that has done both, right?  Lived, and lived practically without money and as a consequence, or perhaps wither on the vine.  I can say other nasty things about New Hampshire later but that will do for now.

How might we lead to the politics of insulation?  Well, the last round in the early 2000s did seem to lead despite this checkered history of a condition where there have been efforts to move in steps down this road.  Partly perhaps, this is caused by desperation – the idea that we are running out of ways to this, ways to think about this.  Three strikes and perhaps we are out.  Our next effort might be the last one where the federal government seriously pays attention to this function even though it is perhaps the one function that everyone agrees ought to be federalized. 

There were at that time, of course, some high profile dividends that seemed to be flowing from education research, especially in the reading sphere.  Most of them not funded by the way by the Department of Education itself.  You had of course, an on-going, very salient debate about measurements and standards and the idea that you needed some scientific underpinning to that measurement and to those standards, which coincided of course, with the political window.  Before President Bush’s election and Congress preceding him, there was relatively bipartisan agreement over the things that would later go into No Child Left Behind. But add to that mix President Bush and his real focus on measurement especially and on accountability, and create a period where you could have something occur like No Child Left Behind, which stresses multiple times its goal of getting scientifically based research into the mix. 

So the ESRA, the Education Sciences Reform Act in a lot of ways is unique in this sequence or cycle as a bipartisan vehicle where members of Congress actually wanted to claim credit for doing something along these lines.  In the past, most members of Congress did not care very much.  NIE was achieved largely by the ideological entrepreneurship perhaps of John Brademas, a Congressman from Indiana.  There were more people interested this time around in part because it did provide opportunity and a vehicle for members of Congress to say, “Yes, we are doing something tangible towards accountability, towards all the nice rhetoric that we have talked about in No Child Left Behind.” So, within IES and the watchful gaze of Russ Whitehurst, we have science for the first time really in statute and a lot of definitions of what that might entail, again, with an emphasis on evidence-based interventions and the notion of controlled experimentation. 

A fixed term for a presidentially-appointed director and the NCES commissioner, also has a fixed presidential term that is staggered away from presidential elections.  The director has moved up to level 2 of the executive schedule, right up from level 4 and prior to the last organization before that, up from level 5.  Again, same as the NSF, an advisory board with a majority of researchers created.  This is actually new in this history and the mission of the agency streamlines somewhat to remove technical assistance, the ORAD function [sounds like] and adding evaluation more explicitly so that the idea that IES could in fact do this function in an isolated way, at least had a chance to take off. 

So far, there is obviously some criticisms – that I will come to in a minute – So far, so good seems to be at least the broad consensus of the education research community.  Do I wait for corrections on that front?  Whitehurst himself called this a tipping point.  That enough things had come together that we could finally move towards serious education research at the federal level.  There has been some independents [sounds like] of the agency vis-à-vis what works -- the whole reading recovery, controversy was resolved by the What Works Clearinghouse at least in a way of saying. 

Well, it did work despite the political tides against that finding.  Peer review has been revamped.  Implementation as I mentioned has been separated from evaluation, all things that least make the creation of science possible.  Others would argue perhaps that “Well, it is not a tipping point.  The whole thing is tipped over.”  There is too much science and what there is, is too narrowly defined, that education does not work this way, that there is too much centralized control, whether it is over the education statistics function over the labs and the centers, whose missions have been redefined centrally.  It is argued that the independence might lead to too little departmental support. 

Moving forward and you see a little bit of that perhaps in the departmental strategic plans that are moving forward and the idea that this might lead to administrative micromanaging.  You could also argue that the centralized control issue will lead to more legislative micromanaging as the labs and centers perhaps make their stand. 

So key questions looking forward and I will stop here – Will incentives continue to match the needs of the research community?  That is going to be the important thing moving forward and at this point, the jury is perhaps out.  And finally, politics have been amenable to the structure we have in place.  Will that continue to be true?   And even if the structure is a good one, will it have the resources that it needs to succeed?  And we end here with the question that eliminates any question of politics whether it is the politics of knowledge or whatever else, the politics of defining politics itself, who gets what?  When?  Where?  How?  Will education research get its, share?  That again will be a huge question moving forward.  Thank you very much. 

Frederick M. Hess:  Thank you Andy.  Jeff?  And again, while we get Jeff set up, let me mention that I do impose Draconian time limits on everybody.  It is just one of those things.  Let’s see.  I have got to be Draconian on some sense or other and – but you will find copies of all the papers available out on the foyer.  You will find again, as I said, CDs with the full set of papers if you are interested.  And as we mentioned, all the papers are available online at the AEI website.  Jeff.

Jeffrey Henig:  Okay.  So when it comes to the role of research and shaping public policy and debate, one might reasonably argue that this is the best of times.  NCLB with its frequent mention of evidence-based decision-making has authoritatively underscored the important role that objective knowledge could and should play in a democratic society.  As Andy suggested, IES with its grand policies and promotion of randomized field trials with the What Works Clearinghouse has provided a detailed road map of what strong research design might look like in education policy. 

Research findings and research debates get deep coverage in outlets like Education Week and instant coverage in the blogosphere.  Some researchers are sufficiently visible in fact to have become mini-celebrities in their own right with their pictures on the front page of the Wall Street Journal.  Or it might just as well be argued that this could be the worst of times.  While highly visible, research, as it appears in the public stage, often presents an unpretty face. 

I lead my paper with four mini-dramas—little research battles that have played out in the front pages of real life education policy research.  Paul Peterson versus John Witte on the front page of the Wall Street Journal, combating one another in terms of research on vouchers in Milwaukee, with Peterson calling Witte a snake and Peterson saying Witte’s research is lousy.  Second mini-drama, the AFT charter school study on the front-page of the New York Times.  A week later, Center for Education Reform taking out a full page ad in New York Times decrying both the research and Times coverage.  Front page of the Wall Street Journal again, Jesse Rothstein, Princeton economist, versus Caroline Hoxby, Harvard economist, combating one another on Hoxby’s earlier research on intrametropolitan school competition, again, with charges of name-calling and ideological bias. 

And the fourth mini-drama I start with relates to the Reading Wars and Reading First, which I label The US Department of Education versus itself.  I will not go to the heart-pumping details of these now.  I will leave it to say that each tells a story of prominent attention to the importance of high-quality research with a flipside portrayal of the research enterprise as personalized, partisan, and bombastic.  In a book that should be out early next year, I look at this in detail with particular attention to the Charter School debate and the polarized presentation of research around school choice and charter schools. 

In this paper, I try to distill some lessons from that analysis to highlight six broad structural changes that I think are potentially changing the demand for research, the availability and type of data, and the way research enters the public realm as part of on-going policy and political debates. 

Briefly, these are new technologies and the dissemination of research.  Think tanks, advocacy organizations and others have electronic newsletters.  You know all of these: PEN’s weekly Newsblast – which has an estimated 240,000 readers – Checker Finn’s Education Gadfly and Andy Rotherham’s blog, Eduwonk, which gets about 1,200-1,400 visitors everyday.  One indicator of the scope of the emerging virtual world of education policy communication:  Eduwonk includes 97 links to other education blogs, 12 links to sites providing education news and analysis and 30 policy and political blogs that cover education along with other issues. 

I speculate in the paper that this may have altered the relationship between research and policy in at least two ways.  One is speed.  There is a palpable sense of time pressure felt by some researchers that once one study is out and disseminated quickly, that in order to quickly provide an antidote, it is necessary for them to get their evidence out just as quickly.  I think there is an argument also that the speed contributes to an erosion of quality-control. 

In general, while these avenues of getting research directly from researchers to the public, bypassing journals, bypassing peer review, things like that, represents in some ways a refreshing democratization of the debate about research.  It matters at a point where some studies are circulated and cited in early drafts, not only before peer review, but before even careful checks of the data and analysis. 

The second broad structural change I talk about in the paper is the academy as a context and I discuss such changes as a shifting relationship between disciplinary and interdisciplinary programs, declining status of ed schools.  What I call hyperpluralism, multiple and competing journals and subfields and speculate in the paper about how these may contribute to a dysfunctional fragmentation of the landscape of academic and professional life and with consequences, I think, both for how the research unfolds within the scholarly community and more immediately to the point here, undermining the image of research, particularly academic research and its utility to journalists, policy-makers and citizens.  Academic research on education policy is quite literally all over the place. 

Third broad structural factor I talk about in the paper has to do with privatization and the growth of the corporate sector in K-12 education.  You all know the story:  increasing role of for profit education management organizations, publishing industry, for profit providers of professional development, curriculum testing, the whole new supplemental education services industry and direct delivery by for-profit and nonprofits through contracts and charter schools, and the like. 

I suggest that on the positive side – these provide a new market for education research.  If you are a young scholar coming out with the proper skills, there are a lot more places for you to look for jobs and for people who are interested in what you are doing than in the past.  But I also suggest that there is a risk that this puts new constraints on research and researchers.  For example, in terms of access to what is often regarded as proprietary data. 

Fourth broad structural factor I talk about is where Andy left off in terms of federal funding, I think, again, the emphasis on evidence based on stronger research designs is leading to an affirmation of the importance of research but government funding is tight and constrained.  For every hundred dollars in federal money going to research less than 41 cents goes to research in the Department of Ed.  There is little of that money – very little of that money as I talked about in the paper that goes to basic research, more open-ended theoretically-driven research, whereas for the applied and developmental end of the research – a large percentage, more than half in dollars, is going out in the form of contracts rather than grants, with tighter oversight as a result of the research design, measurement, analysis, and the dissemination of the results. 

Against this backdrop, at least in some high profile areas, researchers turn more to foundations for support of their research but that money is tight, too.  Many foundations prefer to put their funding into direct services.  Among those that are willing to fund research, there is increased emphasis within the research, within the foundation community, on the importance of making sure that what they do supports the foundation mission as distinct from, in some instances, from the attitude of some of the early foundations in terms of supporting social learning.  Increasingly the foundations -- because they are interested like anyone else is in bang for the buck, put their attention, even their research attention, into dissemination and advocacy. 

The fifth broad factor – and I know there is a panel later on the courts – I will be even more cursory here than I have been on the others – is the courts as a customer.  Researchers do respond to demand from government for different information and during the ‘60s, a lot of the attention from the courts was looking for data relevant to the cases on school desegregation.  I argue in the paper that arguably there is a shift in the action from the courts, from the feds, to the states – a shift in focus from race to finance and a shift from equity to adequacy to the extent that these put the focus on cost-benefit ratios and economic models. 

These may actually be lowering the temperature to some extent on research compared to the ‘60s and ‘70s when research was entangled in the hot issues of race and busing, but to the extent that they develop as they may into broad consideration of such issues as to what does it take to be a full citizen or to the extent that they lead to a strong judicial mandates for higher funding and subsequently higher taxes, these shifts have the potential to embroil research in a new set of controversies. 

The last structural change I talked about has to do with the dynamics of federalism, discussion about education, and education research is moving up the ladder of federalism from the locals to the states to the nationals.  I argue in the paper that at the local level, research is often largely constrained.  The local education bureaucracies have a tight control on data and as result, researchers find it often a thin environment unless they align their research with specific applied interest of the local district.  The local arena of discussion is often somewhat of a more pragmatic one than one that we see on the national level where the debate is highly visible, highly partisan.  Research is often seen as a weapon rather than as a tool for answering straight-forward questions about what works or what does not. 

So, Rick encourages us to have a bang-up conclusion.  This is the best I could do. 

Frederick M. Hess:  That is our boiler plate slide.  That is correct. 

Jeffrey Henig:  So, is it the best of times or is it the worst of times?  My view is that the currently mixed picture is partly a product of two good things.  Researchers have heard the message that they should descend from the Ivory Tower and engage the world.  The old model of speaking truth to power where a scholar whispers in the ears of the power that be, the elite leaders, I think is also past.  In the age of the internet, discourse about research is democratized, is more open, and for the most part, that is to the good.  But it is a volatile time when promising opportunities are twinned with definite dangers. 

Many of the very aspects of Ivory Tower research that are so frustrating to many, such as the abstract concern for theory, the deliberately unhurried pace, fascination with technical aspects of research design, reliance on an internal network of peer review that can be stuffy, conservative, and reactionary – intellectually reactionary at times – and on journals where scholars pluck to one another in terms no one else understands, many of these characteristics that are so frustrating also play a role in maintaining a socially-valuable set of distinctions --  distinctions between research and advocacy, between the pursuit of knowledge and the pursuit of advantage, between sounding good and being right, and it is an open question to conclude in my mind, “How far down the path of relevance researchers can travel without putting something of value at risk?”  Thank you.

Frederick M. Hess:  Thank you Jeff.  Ellen.

Ellen Lagemann:  Sorry.  Andy, thanks to Rick.  He may be a tough task minister but it is not easy to fill a room like this before 10 o’clock in the morning in Washington DC. 

As Rick has mentioned, I wrote a book almost 20 years ago called The Politics of Knowledge.  It was a history of the Carnegie Corporation of New York, which was chartered as Dan Fallon knows to promote the advancement and diffusion of knowledge.  In working on this book, I quickly realized and I probably should have known it before I wrote this book, but I do not think I thought about it, that the corporation was not simply a venture fund backed by philanthropic money.  Its founders, trustees, and staff, on the one hand, and its successful and unsuccessful grant seekers on the other – all had agendas and those agendas were not necessarily the same thing.  They sometimes conflicted and as a result, it was very clear to me very quickly that the Carnegie Corporation, like any funding agency, was engaged in a politics of knowledge. 

And I would like to build my comments about Jeff and Andy’s papers around the conception of the politics of knowledge that I developed in that now very out-of-print book and it should stay out of print.  At least as I formulated in that book, the politics of knowledge is comprised of three interrelated sets of questions. 

First, which fields of knowledge and approaches within different fields are more or less authoritative at different times and as a consequence, more or less relevant to policy-making?  Second, to what degree are our experts in different fields regarded and treated differently from non-experts?  That is a hugely important question in a democratic society.  And third, who can become an expert and how does that happen? 

Viewed from the perspective of the politics of knowledge, education I believe and I’m sad to say this but it is clear to me that education has come out on the short end of things.  I wish that were not the case – for obvious reasons, I think – but I think the evidence is very, very clear and some of this has been said.  Owing to its association with a feminized profession – teaching – and to its frequent association with schools that tend to be marginalized at the periphery of research universities, education research suffers from three major problems. 

It has not drawn sufficient resources and by resources, I do not just mean money.  It has not gained high levels of respect, even when respect is deserved.  And it has too often failed to display the rigor needed to unravel the complicated problems that education presents. 

Education research is, of course, an activity as several people have said that takes place in many different places, not just in schools and departments of education but all over universities, all over government agencies, all across think tanks, all across foundations.  People from many different disciplines, as well as education itself, also work in this field and as a result of all of this, I think education research has never developed a coherent set of standards or a clear consensus concerning central questions.  One must even wonder if it is accurate to speak of a distinct field called education research.

The intriguing question posed by this conference and by Jeff and Andy’s papers then is “Are things getting better?”  It has often been said that educational innovations tend to be polished by hope but unfettered by data.  Is work currently going on in the field - if it is a field - likely to change that sorry situation?  As I read their papers, both Jeff and Andy are guardedly optimistic.

Jeff, who I read as being the more guarded of the two, believes the future caliber of research and its potential relevance and value in policy-making will depend on the balance achieved among opposing forces in what he calls – in terms of six broad institutional arenas, which he just talked about. 

Andy seems encouraged by changes in the institutional structures for education research, especially the autonomy purposely built into IES at its creation.  He rightly notes however that politics can undermine structure and the jury is still out on IES, which is still we must remember, particularly from a historical perspective, a very, very young agency.  It is hardly a nursery school. 

I agree with most of what both Jeff and Andy have argued but I would also give more emphasis to two issues that have been raised but I do not think raised centrally enough, which in my view will have a hugely important impact on the future well-being of education research.  In my view, scholarship in education will or will not thrive depending on whether we can change what Carl Kaestle classically has called the awful reputation of education research.  To some extent, that will depend on the working out of the forces that Jeff and Andy have identified but it will also depend on larger matters of public perception.  So long as teachers are bashed and education is seen as a common-sense practice that pretty much anyone can do well, research about education will not be understood as the complicated, difficult, and very important undertaking that it is. 

How then do we change centuries-old social perceptions that lead to the demeaning of expertise in education?  Partly, I think by doing excellent work and by being willing to criticize work that is not excellent and partly too, by talking as widely as we can about education research – its importance, its demands, and its complexity.  Doing so can be extremely frustrating, as many people in this room know. 

I remember talking at a Congressional briefing some years ago where a leading member in the House walked out after I challenged his view that all we needed to improve education was better teachers.  He stood up quite literally and said, “Professor, why do you have to pretend it is all so complicated?”  And he turned on his heels and he left.  Even when I did not win that mind or that heart – that day, I probably would never win it – I think speaking truth to power, not in the whispering in the ear sense but in the publicly-declaring sense, about education research is vital. 

In addition, I think we need to admit that the canons of research that are likely to renew tenure or other credits in the academic and scholarly worlds are not the same as those that are likely to make your work valuable to policy-makers or to practitioners.  To actually be useable and maybe even useful in education, research must first be scientifically valid and reliable and then it must be translated to findings and recommendations that policy-makers can actually use or test toys or other things that practitioners can employ in their work. 

In my view, gaining a better understanding of what usability means in education research and then finding ways to support such work is critical to the future of this field.  Useable knowledge generated by research is not likely to be tenure-able research so there are many disincentives to doing such work and overcoming those disincentives will require fundamental reform of universities.  Hardly a minor project if you have ever had a position of responsibility in a university. 

But let me return to the two papers.  Let us hope the optimism, even the guarded optimism, in both papers is correct.  If it is, the likelihood of informing education policy-making with tested, disciplined knowledge that is based on solid, accumulating evidence will increase.  That said, as we all know or we would not be here, politics tends to make it very, very difficult to mount programs of high quality, reliable research in education that are useful and doing so will require sustained and determined effort.  It will also require what one of my favorite people, John Gardner, used to call loving critics who are willing to make tough judgments about the quality and the significance of the research that all of us do.  I hope that these papers and this conference can advance that project. 

Frederick M. Hess:  Thank you, Ellen.  Gina.

Gina Burkhardt:  So I took this task pretty seriously and came at it from a very concrete perspective so bear with me as I came up with a lot more questions than answers and I’m going to put them out to the group and hopefully there will be some conversation.

So I want to make four points.  I know you are only supposed to make three, but I could not get it down to three.  One is about politics.  One is about relationships.  One is about evolution.  And the last is about innovation. 

So when I read these papers, I thought what came out of them was more an indication that to move education research forward and have a relationship with policy, you really need more of a knowledge of politics than a politics of knowledge -- and it goes back exactly to what Ellen had talked about earlier that there are these three arenas where knowing politics helps advance your cause, especially when you are inside baseball, which in my mind is inside the research community.  So if you figure out how to be the best researcher on the block or the researcher of the month, then it is your knowledge or your information that is highlighted and then often gets the most attention, whether or not it is targeted at policy-making or policy makers. 

How you advance or move your research to the field is also politically, I think, dictated, so how it gets out mostly in scholarly journals – up until this point – where it is presented, how it is actually turned into application or utilization, which happens far less frequently than it should.

And the third one is about who is involved or who is let into this elite community of researchers and that as we watch the field right now, we are watching different organizations come into or penetrate this elite field in terms of producing research and understanding the political nature of getting funded and getting charged with doing research. 

The second point I would like to make is about relationships and as I look at relationships, I always think that it is best to have a relationship is with another person.  Having a relationship with yourself is good.  There are lots of self-help books out there that talk about loving yourself but I think the research community has gone a little bit overboard in terms of having a relationship with itself and less of a relationship with the outside world or the policymakers or practitioners that are highly dependent on the information to reform education as we know it. 

Also, I think that the research community has figured out how to have a strong relationship with funders or with the Department of Education in terms of soliciting funding but they have not figured out very well – respectfully, I say this – how to disseminate or get the information out into the public so that it is used and applied. 

And I would like to take issue a little bit with what was in the papers around regional education laboratories in the centers.  There has been this perpetuating myth that the regional education laboratories really were in a place of generating research over the last twenty years.  But up until this point, up until the new mandates from IES, they were really a set of research and development organizations and they were the one arm of the Department of Education that was really charged with doing the D in R&D and in our minds – and I can say ours because I think my adult life has been spent in these communities – in our minds, our job was to take what the research community put forward and actually work with policymakers at the state and local levels to create opportunities, to practice how the research was implemented, and to pilot how the research was implemented, so that we could watch policies in action.  What we did not succeed at and I put myself and our organizations among the many of you in the room, what we did not succeed at is the scale up and the cost of that, I think, has been significant to the education field. 

The third point that I want to make is about evolution.  Organisms mostly, in a changing environment, adapt and change and again, I think the structures that both papers talked about are structures that did not adapt to the changing outside environments.  So we see a lot of shuffling inside the Department of Education around how best to structure the relationship between the department and the world and the research community. 

And I think that the evolution process did not happen very well until very recently with the advent of IES and as part of again, a regional organization laboratory, I commend Russ Whitehurst and the group in IES for changing the conversation about the rigor of research and promoting research that is actually need-based.  If you look at the research being done now out of these centers and these labs, you will see that they are all mandated to be in response to needs that have been generated by the states and the locals. 

What I think IES is still struggling with is the relationship part and their idea of a relationship still sits inside the research community.  They are working on, and we are helping them figure out, how best to have a relationship with policymakers.  That is above and beyond the political community because if you look at states, you will see that most of the policy-making around changes in reform and around compliance to NCLB happens at the state level, probably independent of a lot of the research that is happening in the research community or at the national level. 

And my last point, which is really optimistic I think, is that I believe that innovation is going to trump all of these conversations and I’m really looking forward to that happening.  We have technology that is advancing far more quickly than any of us can imagine and it is a process that is getting information out and the demand for information is growing.  There is a community sitting outside of this conversation – businesses, entertainment, media – that are really on the verge of converging and forcing – really forcing change from the outside in. 

So as we have these conversations sitting inside baseball, there is a larger force outside that is demanding with a lot of money that different kinds of education and different kinds of systems develop in response to the needs that all our kids have.  So as we move forward and we struggle with the idea of scientifically-based research and whether or not methodologies are the right methodologies, there is a lot of money, which is going into making change and reform happen in schools that is not necessarily based on research.  That is not a good thing.  So the communication between these outside forces and these inside communities really need to happen so that as we move forward, we are working together to make the changes work. 

Frederick M. Hess:  Thank you, Gina.  Mike.

Michael Feuer:  Let me add my thanks to Rick and the organizers for a very interesting conference and a very interesting set of papers.  My compliments to Jeff and Andy for two very excellent papers.  I’m not going to try a page-by-page or a line-by-line commentary on these papers.  I might do that offline but I learned a lot from these papers.  I think they make very important contributions to the debate and to our continuing attempt to accumulate knowledge.

In my 10 minutes this morning, I would also like to make four points.  That means about three minutes per point.

Frederick M. Hess:  Did you try that line last night?

Michael Feuer:  Like most of the work that I do I have already applied for a no-cost extension here so – Point number one is about the several references to Carl Kaestle’s wonderful 1993 paper called “The Awful Reputation of Education Research.”  There are more references to it actually in the written papers than you heard this morning but I think there is just the possibility – a little bit of unintended sleight of hand going on with respect to the awful reputation of education research. 

It has been a while since I read Carl’s paper but as I remember it, Carl was quite distinctly making a difference – trying to make a point that there is a difference between the reputation of education research and the quality of education research and in a lot of the rhetoric surrounding the debate that we are participating in today, one gets a sense that either in Carl’s paper or in the way that it has been interpreted, that that reputation is deserved.  And I do not think actually that it is quite a correct reading of the original paper and it leads me to suggest two cautions about this business about the awful reputation.

First of all, people like us who are writing about the quality of education research need to be just a bit more meticulous in the way that we understand the possibility for language in being interpreted and misinterpreted.  I think that constitutes -- an example of good quality research is when one pays more attention to the research being criticized.  Maybe more important, this distinction between research quality and the reputation of research is, I think, a key to understanding some of the weak points in the education research system.  From the standpoint of what I take to be an underlying assumption in this conference, which is that good research can and should make a big difference to policy, we would probably be better off studying the opportunities and constraints that affect the way research is used in our democratic system rather than try to come up with blanket indictments of the entire education research enterprise.  More simply put, even if we all agreed that some education research met the highest possible standards of quality, I would argue that that is at best a necessary but by no means a sufficient condition for assuring that that research would be understood, utilized, and applied in the real world. 

Which brings me to my second point, much inspired by Jeff’s mini-dramas which I really enjoyed a lot.  I think this is a beautiful reminder of the world we live in and by the way, Jeff, this one, I could not help point out that in your paper you talked about - in the Rothstein-Hoxby mini-drama – something about the principal opponents and I think you are on to a new branch of statistics here called principal opponents analysis, and I commend you for that.  These are clearly stories about respected researchers doing very high quality, rigorous research that triggers debate, invites replication, and even in some cases, which I think is true at least in the case of Witte v. Peterson, the gradual acceptance of some common ground findings as the basis for collaboration and continued knowledge accumulation. 

So that suggests for me two hypotheses that I would offer to Jeff and Andy that they may want to consider as friendly amendments in their papers.  The first is: if one measure of the maturity and quality of the scientific field is the extent to which scholars apply rigorous methods and models, publish their results, and argue the implications of their findings, it would seem that the awful reputation of education research is indeed not deserved, at least not uniformly. 

The second is: if high-quality education research produces conflicting and controversial results, it is more interesting to explore the conditions under which those results find their way into policy discourse and become the basis for actual policy improvements.  I’m suggesting here that if education researchers are frustrated by the slow pace of adoption of their findings, let me remind them all, let me remind us all that we are in good company – one of the very first recommendations of the National Academy of Sciences in about 1865, I believe, was that the United States convert to the metric system.  Although that report has now cleared review, it has not yet been implemented. 

Third point is about what might be called the cognitive psychology of the politics of knowledge.  That is a mouthful and I know Rick will take care of this in some blog or another.  I suggested earlier that even if research is good by some objective standards and the findings are even considered to be fairly robust, those are not sufficient conditions for their acceptance and adoption, and I just want to suggest that there is a macro-level analogy to something that people who study individual-level decision-making have known for a long time.  And that is even if one has very intimate, sophisticated knowledge of certain processes that does not lead necessarily to rational behavior. 

One of the great decision theorists and probability risk analysts of our century, as I was told this story, refused to leave his home in the suburbs in Washington until they caught the sniper.  Now when you think about that, it is very interesting because this gentleman actually knew all of the statistics associated with rational decision-making here and yet decided to stay home.  Of course, after they caught the sniper, he went back on to the beltway, instantly increasing his chance of violent death about a hundredfold.  The point being – rational thought does not necessarily produce rational behavior. 

My final point is about rhetoric.  For reasons that would take a little too long to elaborate here, I think the condition of education research is subject to a similar amount and intensity of rhetorical excess as the condition of education itself. 

I was not going to get into this except that Jeff reminded us in his paper of Rick Hess’s post-AERA catharsis, which, for anyone who has been around long enough must have looked and sounded eerily like the good old days of Senator Proxmire and his Golden Fleece Awards.  Just to remind us that although the sex life of the screwworm did not sound like a particularly promising or smart use of public funds, even the good Senator acknowledged after giving that research project a Golden Fleece Award that the study actually had been a key to unraveling some of the major biological aspects of pest control and ultimately contributed to major gains in agricultural safety and productivity. 

It is just too easy, even now it is obviously a lot of fun, to ridicule research just because of its goofy titles but it is not scientific, Rick, and had you sampled from the AERA catalogue a little bit less randomly, maybe a little bit more randomly, you might have noticed a large number of sessions dealing with evidence on the effects of high stakes test-based accountability, the implications of brain science for education, the cognitive basis for science education reform, just to name a few. 

So, to conclude, which I noticed our time-keeper is warning me about, let me quote something that Lawrence Cremin of blessed memory, a great historian of American education, once wrote about education and then I’m going to ask you to substitute in where he had the words education and schools, substitute education research.  Cremin wrote, “If there is a crisis in American schooling, it is not the putative mediocrity and decline but rather the crisis inherent in balancing tremendous variety of demands Americans have made on their schools.”  Substitute education research for schools and schooling and I think we will end up with a healthier, more useful, albeit perhaps a little bit less dramatic place in which to explore and ultimately improve the quality of our craft.

Frederick M. Hess:  Thank you, Mike.  Andy, Jeff, any initial thoughts or comments before we open it up? 

Andrew Rudalevige:  Just very quickly on the awful reputation issue. I think it is a fair point.  The title has become almost a substitute for the paper.  Though having read the paper recently, I can say that there is a – certainly a comprehensive treatment of some of the reasons for that reputation whether I would agree that Dr. Kaestle does not think that it is entirely deserved.  As you know though, it is drawn from a much larger set of oral histories for which its own title becomes useful marker perhaps. 

Everybody has been to fourth grade as the title of his larger work which draws together a lot of very valuable oral history.  It is a fascinating piece but the notion there is “Yeah, everybody has been to fourth grade therefore everybody knows what needs to be done in education.”  This gets to some of the comments that were made about – well, if it is so easy to do, why should we bother to do it in a rigorous way and I think that is worth just adding that to the mix. 

Frederick M. Hess:  Jeff?

Jeffrey Henig:  Just one quick point for now also on this issue of the actual quality of research versus the uses of it in debate.  In the – I’m doing this -- both being embarrassed by being a shameless huckster, but in terms of my forthcoming book but just to show that I’m not very good at it, I still do not have a title.  But in the book, I’m looking very much at the uses of research and public debate but my argument is that the research itself in charter schools, despite the polarization in the public debate has been converging on a number of findings that are not consensual but are increasingly accepted and that research is working in arc pretty much like we would like to see it which is we know a lot more about charter schools now than we did 10 years ago but in the public debate that is not reflected.

Frederick M. Hess:  Jeff, I wanted to ask you – one of Gina’s points was the acceleration of technology and the implications for communication.  It strikes me that it seems reading your paper that you actually seem to have a different take on that – that you seem actually concerned that that accelerating communication can erode the protections and the insulation of the academy.  So what some of us see – what some might see is anachronistic, you seem to see as actually the unique strength of the academy.  Could you talk a bit about that? 

Jeffrey Henig:  Yeah, well, I’m not a historian like others at the table so I can only reflect in an amateurish way on the folly of arguing against the onslaught of technology.  Things are going to change and I know they are going to change and you do not know how exactly that is going to play out but – I do think without overstating it, without seeming like a crusty academic, which I am, I do think that there are some advantages in broad public knowledge accumulation to occasionally slowing down, taking a beat, waiting for more studies to accumulate, seeing where the center of gravity in multiple conflicting studies are, learning what replication shows and does not show, and that that part of academia is of value and could save us from a lot of hyper-flip-flops on issues as we cycle back from this-is-great, this-is-bad.  What do we know?  We do not know anything. 

Frederick M. Hess:  Okay, let’s open it up.  We should have two microphones out there.  Please catch Morgan’s eye over here or Shane’s eye over here.  Do we have a first question?  Susie?  And again, be kind enough to identify yourself by name and affiliation. 

Susan Sclafani:  Susan Sclafani.  Chartwell Education.  I just wanted to pick up on that last point because when Gina was speaking I was thinking not just of technology in the sense of dissemination of research but the changes that technology is making in the way in which we are educating children, whether it is the concept that Negroponte has that give a child a laptop and they will teach themselves, or the use of simulations and virtual environments as a way to engage children in probably the most constructivist view of learning that we have ever seen as opposed to looking at direct instruction versus a little bit added-in discovery learning. 

And I think the point that Gina was making is we are not doing any research it seems in those areas, though Michael’s point about some of the sessions at AERA are getting at that a little bit, but where is the field in terms of looking at what is coming as opposed to what has been the dominant forms of education?

Frederick M. Hess:  Gina.

Gina Burkhardt:  I have this vision that the research community has started a lot of these conversations about what works and what might work under specific circumstances and then if we apply some of this to policy recommendations or policy changes, it actually could change some of the structures, the continuous learning of kids so that we are not confined with day-long, four hours or six hours or whatever.  Within that, then, the second level comes with the connection of practice to research.  They actually can implement some pilots where you watch how technology or how changes in structures really affect and the implementation side and the development side, it is not funded anymore, it is not happening.  So I worry like you, Susan, that we are moving to a place that we do not know is coming and we are not prepared in terms of collecting evidence or information without a future, rather a past. 

Frederick M. Hess:  Jim?

Jim Kohlmoos:  Jim Kohlmoos from the Knowledge Alliance, which is formerly called NEKIA.  I appreciate Gina’s comments too, particularly folks in innovation and what Susan just said – perhaps R&D becoming a stimulus for innovation, looking ahead rather than looking backwards, I want to ask Ellen though, looking at the history of education research, whether you think that there could be this potential shift towards the futures and R&D in education becoming an incubator for innovation.  Is that possible now based upon what you have seen in the past? 

Ellen Lagemann:  Well, my first answer is I have no idea.  It is a very interesting question.  I was listening to Susan and to Gina and remembering – and I’m sure there are people in this room who will remember this better than I – but there was a movement in the social sciences -- was it in the ‘80s? -- about social forecasting.  There were big issues of datalys and various things that were notoriously, I think, off the mark.  It is incredibly difficult to do but I would rather take up the point that Michael was making about – you know, you can see this glass is half full or half empty. 

I happen to share the opinion that I think he was putting forth – and he can correct me if I’m wrong – that there is a lot of good work out there and generally it gets tarnished with the paintbrush of – what was your example?  The screwworm or whatever it was.  So I guess, I’m not – If you look at the history, if you look at the founding of the Department of Education in the late 1850s and you look forward over time, I would say things are better but that has to be modified here or there – caveats need to be added to the effect that there is so much more going on and it is spreading.  Jeff’s numbers about the dissemination outlets is staggering and we have no standards in this field. 

I remember when I was president of the National Academy of Education trying to work with AERA and how do we do something about this?  I think that is a huge problem.  And authority and power is so dispersed in education, we have such a chaotic, even an archaic system of governance in education.  If you look at all the local levels, let alone work that goes on in the federal apparatus, I do not know how we get our hands around that. 

Frederick M. Hess:  Mike, your points before are being interpreted – are you not concerned of the quality control today in education research?

Michael Feuer:  No, no, I do not want to be misinterpreted as saying that everything is fine and we should go home, there is a – I think an incredible amount of very healthy introspection on the part of the education research community about quality.  Look, if you pick up the latest bulletin of the American Economics Association annual meeting, I bet you could come up with just tons of really, really riveting examples of research that very few people would either understand or think has much practical application and yet economics as a discipline, as a scientific community is considered -- whether you like it or not – the jewel of the social sciences. 

So all I’m suggesting here is that the education research community, which by the way has to its credit the development of a number of findings and methodologies and results that have been adopted by many other fields, should be grateful that we are doing all of this kind of discussion and constantly thinking about how we can continuously improve this craft. 

The example I have in mind here is of course – it is always funny to me that the people who are most interested in the so-called medical model are not exactly remembering that one of the great contributions to medical research - in epidemiology anyway - is meta-analysis.  Where do you think that was invented?  That was invented in education research and we do not get enough, we do not take enough and we are not offered enough credit for some of the very, very fine work that they takes place in education research. 

Now, that said, this is not a defense for the status quo or an apology for all the – shall we say, postmodern incomprehensibility of some research that goes on – but I do think that we tend to undervalue ourselves perhaps a little bit more than we should, which goes along with the cycles of despair and exuberance that we feel about education itself and a little bit calming of the passions here would not be a bad idea. 

Ellen Lagemann:  I just wanted to add one thing to what my pal here said.  Economics is a discipline.

Michael Feuer:  Uh-oh.

Ellen Lagemann:  Economics is a discipline and education is a field of research.  It is a field of practice.  It is a field of research and I think – I happen to think that people who are involved in education research – I do not care what discipline you are from or what school or institution you are sitting in – I think all of us have something of a special burden, which is the end result ultimately somewhere down the line, you have to remember there are kids in classrooms who need to learn to read and count and think and it is hugely important for all of us, so I just think that it is hard sometimes to compare a discipline, where elegance in a tenure review can be sufficient, to some of the other fields.  Michael and I have talked about this, not 24/7, but often. 

Gina Burkhardt:  So I think that comes back to the conversation about the community of contract researchers.  I think that was what Jeff called them.  People outside the academic research community that are now being brought into the mix of doing research on a market-driven, client-focused basis and while the community can have these arguments about rigor and relevance, they may not be addressing – they may be – but they may not be addressing the needs that have been identified by the policy-making or practitioner community just as you suggest, Ellen. 

So that, I also wanted to take a little bit of debate around the idea that these contract researchers’ methodology may not be as rigorous or as relevant when I think it is, and I’m not sure that it is as biased as you indicate in your paper in terms of being pushed to have the right answer to the response that a contractor wants but rather that they are available, they are affordable, and that they are willing to be client-focused and market-driven.

Frederick M. Hess:  Jeff, do you want to speak?

Jeffrey Henig:  I’ll do it but I’m going to indirectly respond to Gina, but not directly and I’m going to do it by starting with Susan’s question about a particular research area and my reflection on that was if we went around this room even and then recognizing that education policy is made here but it is also made in a lot of other places and went around those rooms too, the number of researchable questions that are important is huge.  And I start to think – but the way we tend to deal with this in Washington is to elevate two or three or four of those at a time and try big banged-up studies. 

Ultimately, I think, my way around this – and this is my indirect response – I personally feel the more research, the better and that the only way we are going to address many needs, especially when we cannot predict what is coming down the road is have a lot of folks doing studies, some of which people are going to think at that time are a waste of time and it is going to turn out some more of it. 

To me, if that is right, if what we want is a multiplicity of studies, we are concerned about quality and sometimes there is reason to do big, big expensive studies but what we want is a multiplicity of studies.  There are policies that would work to build the infrastructure of research, that reduce the marginal cost of research and make it more possible for people to do studies and to link together on scaling up issues to similar databases. 

I’m going on too long but I’ll just wrap it up. 

Frederick M. Hess:  -- Jeff, wait.  Give an example, for instance.

Jeffrey Henig:  Okay.  So I pointed out one of the problems with scaling up and with fragmented research is that people are making up their variables, making up their measurements all the time because there has not been good data.  Some of that is beginning to change.  It is also a technology issue but as the states are building these administrative databases, student level data  – for all its limitations – NAEP and NELS and some of these others where there is a common research where the cost of building the data infrastructure, at least the key – deep-handed variables, the outcomes that we are interested in has been collectivized that makes it a lot more possible for researchers who may not be highly funded, who may have somewhat idiosyncratic research questions to come and do those studies and I think that is a plus, it makes studies that are done in Texas more relevant to studies that are done in New York.  I think we are moving in that direction but I think we could be more self-conscious about it. 

Gina Burkhardt:  I agree. 

Gregory McGinity:  Hi, I’m Gregory McGinity with The Broad Foundation in Los Angeles.  Mike, you talked a little bit about the importance of reputation and Jeff, in your paper, you talked about one of the things that I think is really interesting is that unlike in medicine where you have JAMA and other prestigious journals, there is sort of the one everybody looks at, we do not have that in education.  Is one needed and could you actually get to the point that one was developed and do you think that would change the way the field was viewed? 

Jeffrey Henig:  Well, see, I can predict now that Ellen will – after I say this remind me that education is not a discipline, it is a field – and so it would be hard to do it.  But I do think there would be an advantage in having a clear hierarchy of journals within the education arena and a high-quality journal of the JAMA status that publishes really good stuff. 

I have talked in my research with journalists and a lot of times, they feel a disadvantage by not having an authoritative checkpoint on quality that they can rely on.  I think it would be possible to do it, to write it in a way that is not down to the public but is meant for audiences beyond the experts and would have to probably have a speeded-up peer review process in order to keep a certain sense of currency, but I think that can be done if people are recognizing that this is the most important journal or the most important couple of journals in the field.

How do you do it?  I do not – I have thought a little bit about it – I do not know who has the incentive to fund and create such a journal and make it happen.  I see it as a public good in the classic sense.

Frederick M. Hess:  Did you want to stand again?

Ellen Lagemann:  I just want to add just one thing very quickly.  I was on a committee of the National Research Council that looked at some of the issues around journals and I think – I do not remember the report well enough to remember the details of it.  I’m sure it is available.  But we certainly looked at journals and thought about what their impact could be and sadly, I failed to do anything about this, the most read journal is the Harvard Education Review, which is not peer-reviewed.  It is a student journal and the students are the ultimate arbiters of what goes in there and that is a huge problem, I think, for the field.  It is absolutely wonderful to have students learning to do all sorts of things but in my view it would be very complicated – but it certainly worth thinking about. 

Jim Farmer:  Jim Farmer.  Georgetown.  In the past month, I have been working with education technologists in the UK.  One refers specifically to the IES report on education technology and Ellen, I think this one will straighten your point, the—

Frederick M. Hess:  Would you speak up a bit, please?  Thank you.

Jim Farmer:  I think this would illustrate Ellen’s point about practitioners.  None of them had read the IES report but everyone knew about it from either a blog or from an online description of it and their conclusion of it was that the US has decided that education technology is not significant and no investment should be made in it.  What do you do about this question of dissemination?

Ellen Lagemann:  I’m going for two of it in just a minute.  I’m not totally clear to what your question is.  Are you talking about what should people involved in education research be concerned about technology or people in the practice of education?

Jim Farmer:  There is a difference between providing information to the practitioner versus writing a journal for tenure. 

Ellen Lagemann:  Yes, yes, yes.  Well, my own personal view is we have not even begun to understand the processes that need to be involved to translate the findings of rigorous scientific research into application, whether that application is clear recommendation, it is for policymakers that are backed up with the kind of backing up that they need to have, or whether it is the little widget that can help Johnny learn to read. 

So, I do not think we have an understanding of the engineering involved, let alone will to do this.  We seem to be more interested in generating knowledge than in worrying about its use, which is why this conference is so important, I think.  And to me, I have spent my life in the university.  I deeply believe in research.  I believe in research which has a value even when you cannot see its immediate value.  I do not mean to be denigrating that but I do believe that use in a field like this is hugely, hugely important.

Andrew Rudalevige:  I was just going to say having the Clearinghouse which evidently is not being looked at as directly as was hoped, was a very self-conscious effort to disseminate that knowledge in a way that could be translated into little green and red colored symbols.  It is a very simplified effort.  Well, this one gets the seal – the Good Housekeeping seal – this one does not.  This one works.  This does not.  To the extent that is not being utilized by the education research community, that is, I think, going to be very problematic.  I do not know what the answer is.  Maybe somebody from IES here can talk about how they hope to build on that.

Frederick M. Hess:  Gina?

Gina Burkhardt:  I once heard Michael McPherson speak eloquently on this exact subject about translation and whether or not that was the right word.  Maybe when he is up here, you can ask him that question. 

Frederick M. Hess:  Okay.  Yes, sir?  Shane?  Oh, there we go. 

Larry Snow:  Larry Snow with Houghton Mifflin Company.  Following up on that question, there is a translation of the research, not just to the research community and the policymakers but the intermediary of the media and as you have discussed the media now is much broader through technology like blogs and immediate discourse, if you will.  At the same time there are fewer education writers.  There are fewer bureaus in Washington and so there is also a constricting of some of the outlets and in talking about translation of research, there have been distinctions between the research and the press releases that came out of the Department of Education and the nuances of – for example, the educational technology study as a first year study and many other limitations get lost. 

And we are also now in an era where there is greater media attention to research.  Again, the educational technology study gets the front page in New York Times.  And then a follow-up of school districts dropping one on – individual computers and so there is – the question would be the responsibility or the awareness of the researchers and how the research gets described for broader public consumption.

Jeffrey Henig:  Well, I think these are really very important aspects of what I would refer to as the ecology of research use that go well beyond first order approximation that research is lousy so that is why we do not see more of it in practice.  I think what you are getting to here is that this is a very complex story and there are these intermediate organizations that – for better or worse are doing some form of – I do not really like the word translation either, but they are certainly distilling out of scientific studies what they perceive to be the nuggets of knowledge that may be most applicable to specific decisions that people want to be making better. 

That said, again, I think we just need to recognize a couple of things – that even when the research is pretty much above reproach – now, look, I do not want to get into a whole big fight here and – but most of the scientific community, the respectable scientific community has pretty much figured out that evolution is where it is.  This has not made it –

Frederick M. Hess:  -- do not go and get controversial with us now.

Jeffrey Henig:  Well, but I just want to suggest that no matter how well you translate the theory of evolution, you are not going to necessarily convince various school boards in the country that – and you know where I’m going with this.  So I think, with respect to some of the work we do, there is a big opportunity here for setting our own standards and setting our own sights on a somewhat more gradual, incremental appreciation of what this research means and getting away from the “Here’s the result.  Why don’t you knuckleheads in the government just apply it?”  Because that is not the way the world works. 

There is a – not to – I have historians all around me here, but I think it was in Piers Plowman, which is a 14th century bit of English literature, that the first reference comes up to the phrase L’homme propose, Dieu dispose, which means “Man proposes and God disposes.”  And you find periodic reminders of this all through history.  I would offer as a metaphor for what we are talking about that Science propose and politique dispose.  With that I think we can have, maybe have a better mutual understanding of what is possible in this field. 

Frederick M. Hess:  Andy.  Jeff?

Andrew Rudalevige:  Well, I was just going to say but he beat me to it.  I thought it was “The president proposes and the Congress disposes.”  One thing I think that the structural debates here may give education research the time to do is give us some space and to back off the demand for immediate results with the possible and optimistic result perhaps that these big questions about the conditions under which things work can actually be explored in some detail.  I tell my research methods classes and I do want to thank my colleagues here for not picking on political science as they well could have and when they do turn to economics instead, it was a – in any case, I tell my research methods classes that there are no interesting yes-or-no questions, in fact. 

All the questions come to “it depends.”  That is the answer but that is not the last answer and that answer in the final exam is not going to get you very far.  What will get you further is the idea of what it depends upon and when and so I think it comes back to Gina’s point about the notion of conditionality and the space we might be able to give the research community to get that.

Frederick M. Hess:  Well, I think that is an outstanding closing note for this first panel.  What I hope we have done effectively in this panel is get on the table a number of the questions that we want to come back to during the day.  On the question, for instance, of what is to researchers’ obligation to ensure that the work is represented accurately, I hope that everybody will push Richard on this in the next panel when Richard Ingersoll is up here.  On these questions of disseminational blogging, while Andy Rotherham is up here talking about school choice, I think, talking about these issues and the dissemination of research and analysis will be entirely appropriate. 

I think Michael, as always, summarized nicely what we, I think, want to keep focused on all day: is that regardless of the quality of research, rigorous and relevant research is not enough.  It is not clear, necessary or obvious that it will be used even when it is available.  The real question for today is how does one increase the likelihood that when rigorous and relevant research is available that it will be used in a way which we think is sensible and appropriate and we will come back to that.  We are going to take a ten-minute break and then we will start with the next panel.  Thank you. 

[Start of Panel 2: How Research is Used-Teacher Quality & Reading]

Frederick M. Hess:  Let’s all go ahead and get seated, please and get started.  Okay, let’s all go ahead and take our seats, please.  Okay.  Take another one or two moments but let’s go ahead and get started.  Alright, we are going to go ahead and get started.  Well, this is the second panel of the day, the panel which is going to look at how research is used in the case of teacher quality and the case of reading.  We have with us two authors and two discussants on this panel. 

Speaking first will be James Kim.  Jimmy is an assistant professor of education at the Harvard Graduate School of Education.  Jimmy’s research interests include the use of quantitative methods to assess the effectiveness of compensatory education policies for disadvantaged students and the impact of reading programs on adolescent learning. 

Speaking second will be Richard Ingersoll.  Richard, a former high school teacher, is currently a professor of education and sociology at the University of Pennsylvania.  Over the past decade, Richard has done extensive research on the problems of teaching shortages and under-qualified teachers.  His research on these issues has been widely reported in the media and featured in numerous major education reports. 

Speaking third will be Lorraine McDonnell.  Lorraine is a professor of political science at the University of California at Santa Barbara.  Prior to joining the UCSB faculty, Lorraine was a senior political scientist at RAND for 16 years where her research focused on the design and implementation of education policies and their effects on school practice.  Lorraine is also, I guess, in about a year, the incoming president of the American Educational Research Association. 

And finally, we have Reid Lyon.  Reid is Executive Vice President for Research and Evaluation at Higher Ed Holdings and Best Associates and Whitney International University headquartered in Dallas, Texas.  Prior to joining Best Associates, Reid was a research psychologist and the chief of the Child Development and Behavior Branch within the National Institute of Child Health and Human Development at the National Institutes of Health or NIH.  With that, Jimmy, would you please get us started?

James S. Kim:  Good morning, everyone.  My name is Jimmy Kim and the title of my paper is Research and the Reading Wars because -- I do not know why.  I have a fairly straightforward agenda.  I want to do four things in my talk today.  First is to answer this question: What is the Reading War about? Two is: How has the war been fought? Three is: What are the consequences for the research and policy and practice community?  And then I want to conclude with some suggestions rather than recommendations, because I’m not sure how good they are and I would love to get feedback on it. 

So, what has the Reading War been about?  My sense in reading the history is it is really about two things.  One is a fairly specific question about the efficacy of different types of instructional practices in beginning reading.  And Jeanne Chall, in her classic book Learning to Read: The Great Debate in 1967, basically summed up the question quite well.  She said, “Do children learn better through a code emphasis where teachers instruct children about the specific relationships between letters and sound, or do they learn better through a meaning-based emphasis?”  And the strategy that she specifically looked at was “look-say,” just teaching children to look at whole words and then to say them and to focus on meaning. 

But I think the second thing that the Reading War has been about is this question about professional autonomy.  That is, who should govern curriculum and instruction in the classroom?  Frank Smith, who is a whole-language leading theorist, started writing in the ‘70s about the importance of having teachers be the ones who make these curricular decisions.  Since the Reading War is a debate, I think, about an instructional question and a political question, it is not surprising that the war has been fought in two different venues.  To address the instructional question, what scholars have done, like in other fields of education where there is debate, is to conduct experiments, publish in peer-reviewed journal articles and then, occasionally, every five to ten years, we convene expert panels to tell us what the research says.  In reading, the theoretical debate has focused on this question of “Does context help children to decode new words?”  If context helps, maybe we do not need to focus so much on letter-sound relationships. 

And so what we see starting in the late 1960s after Chall’s publication is a series of scholars who debate as adversaries and publish in these journals.  So we have Chall saying that “In looking at 30 experimental studies from 1900 to 1965 comparing code and meaning-based strategies, 27 favored a code emphasis.”  Two years later, Ken Goodman publishes an article saying, “That is actually incorrect.  Children seem to read words quicker in context and sentences than in isolated lists.”  Then in the 1970s, this debate is picked up by other scholars until there is a scientific consensus that starts to form.  We have been very involved in this research and we begin to resolve some of these questions.  So, that is just one way that the war has been fought and the importance of this venue that is typically used to adjudicate disputes in science. 

To promote teacher autonomy, the whole language theorist at the same time disseminated ideas through professional organizations and state local boards of education.  And what I talk about in my paper is that this debate has been a political debate about whether policy mandates promote teacher autonomy.  And so whole language theorists started communicating directly to teachers arguing for the need to change practice and, most important of all, arguing that teachers should be at the vanguard of policy making, and it should not be driven entirely by results published through peer-reviewed journals.  What are the consequences of fighting the Reading Wars through these two venues? 

This is an actual reviewer’s reaction to a paper that a very distinguished literacy scholar had submitted to the Journal of Reading Research Quarterly in the late 1970s.  And you see kind of the Reading Wars being fought through this journal, and I’m not going to go through all of these comments; you all can read it.  But I will go to the last comment on suggestions.  This is what the reviewer said: “When will we get to the real issues?  When will we try to look at real kids reading real language and when will we lift our eyes from the word to meaning?”  So, this theoretical dispute is being fought in this peer-reviewed journal, and this has a couple of very important consequences. 

One is that, as we all know, peer review is a very slow process and evidence accumulates in a very evolutionary fashion.  And so, the consensus about early reading instruction and what is most effective begins to build in the ‘80s and ‘90s and it is not surprising that the two most influential expert panels occurred in the late 1990s, 20 years after these theoretical disputes were first fought over in journal articles.  And what is important about this process is that scholarship – what makes ideas valued in the scholarly community is that findings have to replicate over time, over different laboratories, over different samples of students.  And a basic idea is that no single study or single scholar should ever dictate policy recommendations;