American Enterprise Institute

September 22, 2006

[Edited transcript from audio tapes]

8:15 a.m.	Registration & Breakfast
8:30	Keynote Address:	Congressman Rush Holt (D-N.J.)
9:00		Panel I: Progress and Pitfall in Voting Technology
	Panelists:	Steven Hertzberg, Election Science Institute Deborah Markowitz, Vermont Secretary of State
		Michael I. Shamos, Carnegie Mellon University
		Charles Stewart III, Massachusetts Institute of Technology
	Moderator:	Norman J. Ornstein, AEI
11:00		Panel II: HAVA—What Has Been Done? What Remains to Do?
	Panelists:	Donetta L. Davidson, Election Assistance Commission
		Edward B. Foley, Ohio State University
		R. Doug Lewis, The Election Center
	Moderator:	Thomas E. Mann, Brookings Institution
12:30 p.m.	Adjournment

Proceedings:

[Start of Panel I: Progress and Pitfall in Voting Technology]

Norman Ornstein:  Thank you all for coming.  We are going to start this session as soon as I activate it with my mini-bar key here and take my little smart card which I happened to find in the back seat of a cab in Cuyahoga County.  And I have erased the 300 votes that are on it so we can use it here as well.  I'm Norm Ornstein of the American Enterprise Institute, joined by Tom Mann of the Brookings Institution.  We are co-directors of the AEI Brookings Election Reform Project.  John Fortier [sounds like] who is at the other side of the room, and Tim Ryan, Matt Weil and Molly Reynolds from AEI and Brookings have helped to put this particular conference together which has…  The 2006 Elections: Are We Ready seems to have more questions and more answers to those questions emerging everyday, and not just in places like Chicago or Louisiana where traditionally we might have had questions about elections but in bastions of good governance, at least previously known as such. 

We are particularly pleased to kick off this conference this morning with Congressman Rush Holt of New Jersey, a physicist by training.  He had a distinguished career at Princeton before moving on to represent a district that includes Princeton, New Jersey.  He is one of the core members of Congress who from the beginning, post-2000 elections and through the passage of HAVA and beyond, has shown a strong and abiding interest in issues of our elections and the integrity of our elections, the efficiency of our elections and the need for further diligence, vigilance in election.  So, Rush Holt.

Rush Holt:  Thank you, Norm.  Thank you for putting this together, and I'm delighted to see so many people here, a number of people well-versed in the subject of elections.  I am pleased to see Kenny Gil here, and Ray Martinez, formerly from the Election Assistance Commission, and a number of others.  And I thank you, Norm, for keeping the focus on this subject as part of your good government efforts over the years.  I thank Tim Ryan also and, of course, Tom Mann for this, and I am honored to be here with you.  There are a number of real experts that you will be hearing from shortly. 

When I came to Congress I did not intend to be an expert on the administration and technology and philosophy of elections, but I quickly was thrown into that in my very first election, when I was elected to Congress.  When one precinct went by 9,000 votes to my opponent I realized there was something wrong because there were not 9,000 people who lived in that city, let alone that precinct.  And it was in this case a simple arithmetical error.  And, of course, all of America got, I think, a series of reminders in recent years that elections are not so straight-forward.  And in the usual American way the public has said this is too important to leave to the experts, and so there is now a very lively citizens’ activist network, or a series of networks, somewhat to the discomfort, I suppose, of the professional officials.  But despite all of their efforts, I would have to give the answer to today's question, “Are we ready,” as a pretty clear no. 

We have had the benefit of two presidential election reform commissions, the Ford-Carter National Commission on Election Reform following the 2000 election, and the Carter-Baker Commission on Federal Election Reform following the 2004 election.  We have had in Congress sweeping and historic legislation providing the sum of almost $4 billion to help modernize our election equipment and improve the administration.  And yet the reputation of American democracy on the world stage is, to some extent, in tatters.  I think we should be intensely focused on how well our elections can be expected to run in this year, and in coming years. 

It is interesting to note that the title of the 2001 Ford-Carter Report was “To Assure Pride and Confidence in the Electoral Process.”  Three years later, the Carter-Baker report was simply “Building Confidence.”  Have we given up on “Pride?”  The best we could hope for now is, maybe, “Believability.”  But of course that is a pretty high standard.  As I always say, it is the loser who has to believe the results of the election.  It is the loser and the loser's camp and supporters.  The point becomes more stark when you consider that the June 2006 report from the Brennan Center for Justice at NYU, working in conjunction with the National Institute of Standards and Technology, was entitled “The Machinery of Democracy: Protecting Elections in an Electronic World.”  It seems they were no longer looking for pride, or even confidence.  We are looking for protection from our own voting machines.  To answer the question posed by your conference, no, we are not ready for 2006. 

The passage of the “Help America Vote Act” in 2002, HAVA, although unintentionally, in some ways created problems.  It did a number of good things and it funded the wide-scale replacement of obsolete voting equipment without mandating any meaningful standards.  Consequently, it pushed jurisdictions all over the country to hurry out and buy electronic equipment, not exclusively electronic but, by and large, the jurisdictions purchased electronic equipment that provides a method of voting that is clear, simple, easy to use, and totally unverifiable.  Fair elections, need we say, are the very cornerstone of democracy.  And without the ability to ensure the integrity of the system, it raises questions about the basis of our democracy. 

A number of people in the last day have said, "Well, what about Maryland?"  It appears to me to be a simple case of poor administration, and a certain amount of buck passing and finger pointing about who was responsible for that poor administration.  It was, by itself, not a failure of the machines.  Clearly, if you do not have the mechanism for activating the machines the election cannot go forward, and some say, “If they had only been more low-tech it could have gone forward.”  You might also say, “If only they had had enough emergency ballots and provisional ballots at each precinct it could have gone forward.”  The technology provides some advantages, but it also provides a greater risk of catastrophic failure. 

My principal point is that the process of voting should be fair, accessible and auditable.  For years we have said elections should be fair, above-board, without corruption.  Starting a few decades ago we began to realize that it could not really be fair if people with physical disabilities and other limitations were excluded, either directly or indirectly, s.  So accessibility has become, as it should, one of the principles to be observed in elections.  But auditability, verifiability has not until recently been talked about as a core principle of elections. 

Anything of value should be auditable and audited.  Votes, we know, are valuable.  You can find instances throughout history where they have been bought and sold.  I suppose you could even establish a price here at the AEI for a fair exchange of votes.  In any case, because they are valuable, they should be auditable.  We should not have it any other way.  The process of voting in a democracy has always been intended to belong to the voter who makes her or his decision and casts the ballot, not really to the election official, and certainly not to the vendor of the equipment. 

The system should be publicly auditable.  It is not auditable if the vendor says it is, or even if the election official says it is.  It must be the voter because only the voter in our system of secret ballots can verify whether her or his vote is recorded as intended, no one later.  No election official, no vendor can determine whether the vote is recorded the way the voter intended.  So if the voter casts a vote on an electronic voting machine and walks away asking "How do I know that that vote was recorded properly…?"  In nearly all electronic machines currently in use, the so-called touch screen or DRE machines, the voter cannot know, nor can the election official, nor can the vendor know if the vote was recorded the way the voter intended.  The voter casts the vote in secret and therefore it must be the voter who verifies whether it is recorded properly. 

I recognized immediately… I mean, as I say, I got involved in election technicalities sooner than I intended, or even beyond my intentions, when I ran for Congress.  But then as I got involved in the Help America Vote Act and I asked that there be a provision of auditability, I was assured that that was taken care of, only to find in the final legislation when changes were not permitted anymore …  tThat is a more and more common practice in Congress right now, that legislation may not be amended when it comes to the floor.  It was considered an audit to have a printout at the end of the day from these machines.  So whatever slipped between the cup and the lip, between the casting of the vote and the recording in the electronic memory may have occurred, it will be permanently recorded then in a neat printout at the end of the day from each of the machines. 

My legislation on the subject, the Voter Confidence and Increased Accessibility Act of 2005, which is actually the latest in some legislation that goes back over three or four years now, HR550 in the current Congress, would not only remedy the problem by requiring that a voter-verified paper record of all votes cast be kept, but it would also require random audits in a percentage of machines in a percentage of precincts.  And it would ban the use of wireless devices, and it requires software to be available for inspection. 

Someone once asked, “Why do the companies need to keep the software proprietary when all it does is count?”  It is not as if it is sophisticated, crypto-logic software.  The first bill was written to be effective in time for the 2004 elections, would that as it was, t.  The current bill, for the 2006 elections.  Although we now have nearly half of the House of Representatives as co-sponsors, the hearing on this bill will only be held at the end of next week, the day before our target adjournment for this session.  I regret that Congress has not sufficiently focused on this cornerstone of our American democracy.  In 2001, the Caltech/MIT study, “Voting:  What is and What Could Be,” reads: "In the 2000 Presidential Election, the state of Florida conducted an enormous audit of its voting machines. 

It is extremely important to be able to conduct such an audit.  Paper ballots have a high degree of auditability.  The votes cast on a broken machine can never be reclaimed.  Most new electronic machines produce an internal paper tape, like a cashiers' tape, and an electronic recording of every voting session.  While this is an improvement over the older machines, it is not a direct recording of the voter's intention.  If the machine fails between the touch-screen and the tape, or some time during the day, or counts backwards, as evidently some machines have done in some jurisdictions in recent years, the voter's intentions are still lost. 

We feel that new voting standards must require a minimum level of auditability."  So that was the Caltech/MIT study.  We knew this in 2001, before HAVA passed, that independent auditability was critical.  But HAVA did not mandate it.  The touch-screen machines currently in use, nearly all of which produce no voter-verified paper record, counted nearly 50 million votes in 2004.  And this November more than 40 percent of voters will be casting their votes on unverifiable electronic machines. 

Is there a possibility that votes cast on some of these machines could be recorded incorrectly due to a software error?  Yes.  Due to malicious hacking?  Yes.  Some of you have read about the study just released a week ago by the computer scientists at Princeton.  At Felton, one of the, really, world’s experts on computer security, with a couple of graduate students, got hold of a machine, the kind of machine that has been in common use in the United States.  He will not say where he got it, or how.  But it is evidently programmed with software that has in the past, at least, been used for recording a significant percentage of votes in the United States.  He and his students could hack it in 60 seconds. 

And the scary thing is that if one of these machines is infected with a virus, and the election official comes by with his smart card or her smart card to prepare all of the machines for this years' election, putting this years' candidates on the screen and in memory by going from machine to machine, inserting the smart card with the new programming information, that virus will be transmitted to every one of those machines.  I asked whether this would be easily detectable, and he said not easily.  It could be detected just as they were able to reverse-engineer the software itself, but when you hear about various irregularities, whether it is in Fairfax County or in another state elsewhere in the country, that things did not go quite right in this election, and unfortunately there is no paper trail to reconstruct what actually happened that day, that is troublesome. 

But I will tell you what is more troublesome.  Our tens of thousands of election officials closing up their precincts at 9:15 at night, having read their results and sent their smart card back to the county clerk's office, and everyone has gone home thinking it worked just fine, and it did not - but we will never know.  That troubles me a great deal more.  Let me hasten to say I believe more votes in recent years by far have been lost through traditional disenfranchisement, monkeying with the registration list, intimidating voters, having inadequate supplies so there are long lines. 

There is a young man who works on my staff who recently graduated from Canyon College in Ohio where the students had to stand in line for 12 hours to vote at the last election.  So there are a number of things we need to do with regard to the administration of elections to make sure that there are adequate provisional ballots, adequate machines, well-trained poll workers, and that the activating cards are distributed to the precincts, as did not happen in Maryland a few days ago.  Nevertheless, with all of those obvious, or problems that should be obvious, there is this nagging absence of auditability.  In April of this year, more than 200 citizens from 30 states came to Capital Hill, held meetings with members of Congress.  This was on their own nickel. 

Citizens are worried about this.  There was something of a coup in the League of Women Voters - and I see we have a representative here - where the leadership kind of brushed off the idea of auditability, and said that they liked HAVA, they had worked hard on HAVA, and they did not really see the need for a paper trail.  There was a coup [??] that came from the grassroots that said, as the League of Women Voters now says that we need …  iIn fact, the Leagues says “the League supports only voting systems that are designed so A) that they employ a voter-verifiable paper ballot and other paper records, said paper being the official record of the voter's intent.  B), that the voter can verify, either by eye or with the aid of suitable devices for those who have impaired vision, that the ballot accurately reflects his or her intent, that the paper ballot is used for audits and recounts, that routine audits of the paper ballot in randomly selected precincts can be conducted in every election and the results published in the jurisdiction.” 

A couple of dozen states now have some version of the paper trail bill, some better than others.  Only a few of them actually require audits.  Having auditability without a procedure for random audits gives, I think, false confidence.  And so I think a feature of my federal legislation that is critically important is that it requires audits as well as auditability.  I think HR550 would implement all of the major recommendations of the Carter-Baker report, the Brennan Center report, the League of Women Voters resolution. 

But as Norm and others of you know, having a majority of the House as co-sponsors of the legislation does not guarantee passage of the legislation.  So we are continuing to keep up the effort.  But actually it is not we, the members of Congress, who support this legislation who are keeping up the effort.  It is, really, millions of Americans who are, in their own way, with their own money, have a grassroots organization or a network of grassroots organizations quite active.  The way we got the number of co-sponsors we did was from the outside in, the way, in textbooks, legislation is supposed to work. 

It is remarkable, and it is encouraging that the public feels so strongly about this and that the grassroots movement is what is driving it.  That gives me hope for our democracy.  It gives me hope that some of these grassroots activists will also be involved in this election to ask questions of the election officials to make sure that they are doing everything they can to be ready for this year’s election, even in the absence of regulations and legislation that would mandate that they be ready for the election.  I hope this will be widespread.  It was not widespread enough in advance of the primary in Maryland.  Perhaps for the general it will be - citizens engaged with the electoral process, not just when they show up to vote on Election Day, but in preparation for that vote.  And if so, we will be just-in-time ready for this years' election. 

Thank you.

Norman Ornstein:  Thanks very much, Rush.  That gives us, I think, a framework for starting out as we move forward.  As you leave, let me just ask you one question.  Governor Ehrlich yesterday suggested that Maryland's system is so broken that they should scrap all their machines, go back to paper, and that people should be discouraged from voting on Election Day, and move to absentee ballots.  What do you think of that for this year?

Rush Holt:  As I say, one of the principles is fairness.  And something that troubles me about the move toward pre-voting, or, in other words, voting from your home, whether it is by computer or by absentee ballot, is that it runs the risk of an unfair loss of secrecy.  Intimidation and coercion, whether it is from your boss or your spouse, is something to be guarded against, it seems to me.  And not just the ritual, but the practice, of secret ballot on Election Day in a secret voting booth, I think, is something to be preserved.  Now, whether it is done low-tech with paper ballots, or high-tech with touch screen machines, is something for, I think, local election officials to debate. 

It is worth noting that private enterprise, private vendors have been inserted into this process, into the decision about which method to use, in a way that has not always been constructive.  But however that decision is made locally, I think we need to keep in mind the principles of fairness and reliability and accessibility and auditability.  Electronic machines have the advantage of preventing over-voting, in other words voting twice for two different candidates in the same election by mistake, or leaving half of your ballot blank because you did not see the other page.  The electronic machines will alert you to elections you have not voted in, or over-votes.  It will make it possible for people who are blind or otherwise voting with disabilities to find their way through the ballot.  So the electronic machines have some advantages.  Those advantages are lost if there is not auditability. 

And that is what I think the local officials have to keep in mind, and there is right now a real fight - and I think that is not an understatement - between the governor and the election officials in the state about how this is going to be carried out.  And as part of this fight, there are aspersions cast by one side or the other that there is an attempt to depress the vote, or to otherwise manipulate the turnout.  And that just goes to show what you get into if you are not ready.  Thanks.

Norman Ornstein:  Thanks again.  I should note that, speaking of absentee and early voting, we have a publication that John Fortier has written that will be out in just a month with a comprehensive look at the extent of and problems with that voting.  And John Fund, who is here, has also written quite eloquently about that.  You have given us a good kick-off.  Go catch your plane, and thank you again.  And in three minutes we will start our first panel where we will talk more specifically about a lot of the technology.  And I should note that we have actually, as you can see, added Deb Markowitz to this panel, instead of the second one, which is in your program.  So three minutes.

[Long audio gap]

Norman Ornstein:  All right, let us move the panelists up and get this session going.  Take your seats, please, and we will get underway.  As you can see, the first panel that we have is called “Progress and Pitfalls in Voting Technology.”  We have an extraordinary group of people here.  I had mentioned that Deb Markowitz had moved to this panel, but she is not yet here. Probably a little bit of a flight problem, so we will move here in as she arrives.  Steve Hertzberg, who will talk first, is Project Director of the Election Science Institute, which is an organization founded to promote transparency in elections, as you know.  In particular, and most notably now, he directed a very strong and very disturbing study of the primary election in Cuyahoga County, Ohio that demonstrated a number of the human and other errors that now are being showcased, at least, in our elections. 

We will have some interesting discussion, including the reaction of Diebold, which has now become a typical reaction from Diebold.  Deb Markowitz, who will soon be here, we hope, is Secretary of State in Vermont, and now is President of the National Association of Secretaries of State, and on the executive board of the National Election Standards Board, has been extraordinarily active in modernizing the administration of elections in Vermont, a state which has very, very high turnout. She has been in conferences we have done before and has a wonderful grasp of the nature of the issues and the problems here. 

Michael Shamos is Distinguished Career Professor in the School of Computer Science at Carnegie Mellon, Director of the E-Business Degree Program, has a very strong background in election voting systems and has taught a course at NIST, the National Institute of Standards and Technology, on voting system testing, so he certainly knows whereof he speaks.  As does Charlie Stewart, head of the Department of Political Science at MIT, a founding member of the Caltech/MIT Voting Technology Project who has worked on and written extensively about the nature of voting and reliability of various voting machines. 

So let us start with Steve Hertzberg.  We will have presentations, some discussion among the panelists, and then we will open up for a broader discussion.  Turn the lights down a touch so people can see…

Steven Hertzberg:  Just one housekeeping matter.  I have printed out some copies of the statement that I plan to read, if anyone would like to take it.  It is on this chair up here.  If you would like to pass it around…

Norman Ornstein:  And without objection it will be made a part of the record.

Steven Hertzberg:  I just thought it would feel more comfortable reading from something.  If we can go through the first slide…?  And what I would like to do here is, I think to many of those in the audience we may be a newcomer.  I just want to spend a couple minutes telling you a little bit about us, and what we have been doing over the last several years so you can become more familiar with us.  Also, I just want to introduce Tracy Warren who is in the front of the room who worked with us closely on the Cuyahoga Project who is also part of the team.  Election Science Institute is a non-partisan, non-profit election science organization which seeks to improve the election process using rigorous science-based approaches done in collaboration with election officials to develop real-world solutions and best practices that can be employed immediately. 

We have been quietly working towards this goal since 2002.  Many of you may be familiar with some of those who ESI has worked closely with, people such as Michael Alvarez at the Caltech voting technology project, Warren Mitofsky - God bless his soul - of Mitofsky International, and Fritz Scheuren, President of the American Statistical Association.  However, there are many more who have been working diligently behind the scenes for several years whose names I'm sure you will hear in the not-too- distant future. 

Further, I would like to take a moment to review some of ESI's accomplishments.  During the 2002 mid-term elections, ESI was the first to collect detailed observations and feedback directly from voters around the nation through its innovative website.  Our goal was to collect data from the field in an effort to objectively evaluate election system performance.  Two, in the 2003 California special election, ESI test-piloted a multi-faceted observation program around California, triangulating exit polling results with precinct- structured observation and web-based surveys completed by voters.  This was an infinitely more complex operation and this pilot educated us on the logistical complexities of such an endeavor. 

During the 2004 Presidential Election, ESI worked with the Common Cause Educational Fund and members of the American Statistical Association in New Mexico and Ohio to replicate and expand the program started in California in 2003.  However, this time, through the heavy use of technology, we developed systems to collect and analyze our data in close to real-time.  Immediately after the 2004 Presidential Election, ESI worked with leading statisticians to audit the results of 41 of Ohio's 88 counties.  This is something that was not widely reported.  We found no statistical evidence of vote bias for either presidential candidate in any of the counties that we sampled.  Just to take a note here, when you look at the actual data that we sampled in Ohio for the 2004 election, it basically looks like a circle, which means there is no correlation.  There was no bias in the data whatsoever.  We could not find any evidence of vote-shifting whatsoever, even with all of the discussions that has been going on. 

Moreover, in Ohio, when we were alerted that a state-wide recount was mandated, ESI joined forces with the American Statistical Association, offering to assist each of Ohio's election inspectors [sounds like] with the random selection of precincts to be recounted.  Unfortunately, none of Ohio's 88 election directors accepted this offer.  We were very disappointed.  In early 2005, ESI advisers conducted an independent review of Ohio's 2004 exit poll conducted by Edison/Mitofsky. 

Our study confirmed Mitofsky's findings, namely that the discrepancies in the Ohio exit poll were a result of error within the exit poll, and not the election system.  This report has been published in Chance Magazine.  Later in 2005, ESI worked closely with election officials in Franklin County, Ohio, where we entered into an unprecedented partnership – this, again, not widely reported - with ES&S, Election Systems & Software, and Franklin County to conduct a review of the iVotronic source code.  ESI team members worked closely with ES&S engineers at ES&S' facilities in Omaha, Nebraska over the course of six months.  We conducted a comprehensive source code review of that system before Franklin County took possession of that system. 

Finally, and more recently, we conducted a review of the May 2nd Primary Election at Cuyahoga County, Ohio's largest county.  Could you turn to the next slide please?  Okay, on to the Cuyahoga County report.  In April of this year, prior to the May 2nd Primary, the Cuyahoga County commissioners - these are the county commissioners who oversee the board of elections - to a certain extent also shared responsibility with the Secretary of State in overseeing the board of elections.  However, the county commissioners controlled the budgets for the county.  They contracted with ESI to conduct a comprehensive review of Cuyahoga County's election system performance on Election Day.  ESI understood, going into this engagement, that an election system is not simply a piece of equipment.  An election system cannot be separated from the people who run the system, including the election officials, the booth workers, the voters, as well as the procedures for administering the system. 

These things that we are all discussing - the poll workers, the voters, the machine, the procedures – are inextricably linked, and we do not typically talk about those separately.  We talk about them as subsystems of a larger system.  So I think it is important to note that that was our view going in.  While the Diebold AccuVote-TSX voting system, which was used in Cuyahoga County, passed both federal and Ohio state certification requirements, ESI sought to analyze the entire election system in the real world in sometimes extreme conditions of an actual election.  Specifically, we designed a seven-part review so that we could garner a holistic view of the election process. 

Specifically, our work included … I'm just going to go through the seven major sub-areas of our work:  An exit poll of Cuyahoga County voters in an effort to evaluate their voting experience with the new electronic voting equipment. 

Second, a survey of poll workers and election-day technicians shortly after Election Day in order to understand their perspective on the election system.  In Cuyahoga, they not only have poll workers but they had specially trained folks who received additional training on the technology, and they typically had one of those in every polling location.  And we surveyed both the poll workers and these election-day technicians. 

Three, an in-depth content analysis of approximately 6500 election-day incident reports called into the Cuyahoga Board of Elections command center by poll workers, and written on paper forms by poll workers at each precinct.  Actually, there was something like 6800 of these incident reports that were created on Election Day.  We have reviewed all of them. 

Four, a functional threat analysis that leveraged standard six Sigma methods to prioritize functional weaknesses within the election process. 

Five, a manual count of paper ballots for a statistically significant sample of precincts.  In this case we counted paper ballots from 50 of the 569 polling locations.  Warren Mitofsky did the sample for us to determine which 50 locations we should choose. 

Six, a comparison of election results stored electronically and on paper record.  This is the audit that everyone is talking about that should happen.  We actually struggled our way through it and were able to finish one, although very challenging.  Seven, lastly, building on our voting machine allocation work from Franklin County, Ohio.  ESI continued development of its simulation model to help optimize the allocation of voting machines.  So here we were building on some work that we had done in Columbus where we were trying to address the long lines and the vote wait times.  And we were trying to figure out how do you best allocate voting machines so that you minimize wait times.  And if there are wait times, you distribute them equally across all polling locations.  In conducting this research, ESI's sole intention was to provide assistance to Cuyahoga County.  The report does not affix blame to the vendor, the election staff, the booth workers or the voters.  Rather, the report describes what ESI found from its surveys after examining the data provided. 

Let me talk a little bit about the results.  The good news from ESI's report is that approximately 90 percent of those voters surveyed liked the new system, and had confidence that their votes would be recorded correctly, although 10 percent did have problems.  Two, over 90 percent of the booth workers liked the system, but wanted better training.  So, in general, we found that the folks that are using the system most were satisfied with the system from a usability perspective. 

However, numerous issues were uncovered during the study.  I'm going to go through some of them here.  There is a myriad of issues that we described in the report.  It is an over-200-page report, and I suggest you take a look at it if you would like further details.  Number one, serious deficiencies in inventory control exists.  I think we are now seeing the same impact of this in Maryland, at least from what I am reading in the Post.  These deficiencies led directly to the loss of what I call mission-critical election components - memory cards, paper rolls and machines.  For example, in Cuyahoga, of the 50 polling locations that we sampled, 87 V-pack cartridges, 87 rolls, were missing out of 50 polling locations. 

We believe that there should have been somewhere around 485 rolls; 87 of them we cannot find.  Twenty-nine TSX units…  We are talking about $3000 voting devices.  Twenty-nine of them cannot be located in the warehouse.  And this is after their team spent three or four days… It used to be 45 that we could not find, and then they narrowed it down to 29.  So, not only do we have a budget issue here because we got so many machines missing that cost so much, we have other problems.  Number two:  The paper ballots, the VVPAT [sounds like] ballots did not always match the electronic results, something else that surprised us.  Specifically, we found that election results in 32 out of 48 … s So we surveyed 50, we could not find the paper ballots for 2.  So, 32 out of the 48 that we were able to find, 66 percent of the polling locations showed a net difference between the VVPAT summary total and the election results. 

Let me just make sure I am clear here.  There are two parts to the paper trail.  There is the part that is verified by the voter, that their ballot that gets rolled up through the window, and then the next voter comes and they cannot see what the previous voter voted on.  But at the end of the day, the poll workers are instructed to print out a summary, which should be just a tally of everything that machine recorded on Election Day.  Those summaries, the calculated summaries in 66 percent of the cases did not match the electronic data we have.  Three, auditing of election result is extraordinarily challenging, taking almost eight weeks to assemble, compare, and analyze all the data.  So, while there is a lot of talk about doing an audit, I challenge anyone to go do one.  We had a team of 20 people, 14 PhDs, and we had a tough time.  Further, transparency is severely constrained by reporting limitations within the Diebold GEM System.  For example, while the paper-based system outputs records on a machine level, electronic data is reported on a precinct or polling location level.  Therefore, one cannot conduct a simple one-to-one comparison between both mediums of results. 

So, we want to just take a paper tape from a machine, go to the electronic results and compare them.  It is not that straightforward.  You cannot easily produce that report from the Diebold GEM System, even if the data is in an SQL database.  The good news is these are things that are easily corrected.  The bad new is, at least for this November, all election officials are going to have to deal with this challenge if someone wants to audit the election results. 

Next, in Ohio, where the printed ballot is the legal ballot of record, not all paper ballots could be counted during the manual recount.  For example, 10 percent of the VVPAT cartridges or the rolls counted were either destroyed, blank, legible, missing, taped together, or otherwise compromised.  Next, a detailed manual count process is not in place.  At least, not in Cuyahoga and nowhere else that I have looked at have I seen a detailed manual count process. 

During our study, ESI not only developed its own manual count procedures, but also produced its own prototype of a manual recount fixture in order to handle the V-pack cartridges.  I mean, trying to handle a register tape that is on a table is not an easy thing to do.  We built fixtures so we could do it easily, but I do not know of any other counties that have these fixtures.  We built 14 of them, and we still have them all. 

Next, some of the things we found in reviewing the incident reports that we got from poll workers.  Voter registration issues are the most reported incidents by booth workers, with almost 1/3 reporting items such as incorrect addresses, misspelled names, or inconsistent signatures.  Next, poll worker training and familiarity with the system are a significant issue.  I think we probably all know that already, with approximately 1/3 of booth workers saying they had difficulty setting up the machines, and 45 percent having difficulty closing out the machines at the end of Election Day.  Clearly we have some challenges to get the polling places open if the 1/3 of the poll workers cannot get the machines operational.  In summary, the value of doing a critical evaluation is learning how well parts are working and what needs improvement.  From this standpoint, the ESI study fulfilled its purpose. 

Now, let us talk about where we are in the big picture and what else should be done.  I have created a rough approximation of a life cycle chart here just for illustration.  I refer to the primary elements of a product’s life cycle, as basically what I call “define, design, develop, test, and deploy.”  I have broken this out a little differently in the illustration, but I think I wanted…  And maybe this is not appropriate, but I wanted to ask the audience here. 

You look at these different points here.  There is the first point, ideation.  Then you develop requirements, start to develop specifications.  Then you design a product, you develop it, you start to test it, you roll it out in a small pile and you test it some more.  And once you are confident it performs according to specs, you roll it out nationally, or whatever your market reach is.  I look at that chart and I wonder:  Can anyone tell me where we are?  Because I do not knew either, because in this, we are still doing requirements development, and driving new specs that are coming out mid-next year.  It still appears that we are doing some sort of testing out there with Princeton doing its security reviews, is doing the tests that we are doing in Cuyahoga.  I mean, it seems like information is still being collected about what is required to provide a system to election officials and how these systems perform.  But yet we have rolled out the system nationally. 

I mean, typically, I come from the commercial world.  The non-profit business is new to me.  This is not how you do this in the commercial world.  So, the point I wanted to make with this slide is that we are in several different phases within a product's life cycle at the same time.  While the voting system has been rolled out throughout the country, we are also still trying to conduct testing.  Moreover, we have almost no way right now of testing this system's performance during an election.  We have almost no definable quality control mechanisms in place.

 This is another point I wanted to make.  Specifically, we have yet to define the variables that one should measure to determine performance or the metrics that these variables should perform.  So not only do you deploy a complex system, but you do not deploy it without having some ability to figure out how it performs once you have deployed it.  It is called feedback and control mechanisms.  I mean, this is standard practice in the engineering world, and it feeds into quality control.  We really have no meaningful quality control built into what has become a very complex system.  Not only have we not defined how this system should perform, we do not even know what variables to measure.  And if we knew what variables to measure, we do not even know how to get the data.  We do not know if the systems can provide the data that will allow us to measure them.  That is how far away we are from having a system that I would call fit for use. 

To sum up my assessment, this system is simply not yet fit for its intended use.  The good news is we can create a system that is fit for use, but we will need to take some steps backward through the life-cycle to achieve this end.  I will wrap up just briefly here.  I just want to talk about November, and where I think we can go after that.  This November, ESI will again travel to Ohio, this time in Franklin and Summit Counties where we will be working with BYU and local Ohio universities to conduct another exit poll.  Our goal is to learn how best to teach our exit poll and oversight model to others, so that we can commence a replication program.  But much more work for this coming November needs to be done. 

Given our findings in Cuyahoga, I suggest that we also invest energies in the following three areas.  Number one, and we have heard it here already today, an audit procedure handbook that communities can utilize if required.  I think we now have the knowledge on how to write an audit handbook for the Diebold TSX System, and that is something we are working on.  By the way, just as a side note, I have approached Diebold on three separate occasions asking for their collaboration on that auto procedure and they have not come back to me.  Number two, the creation of a manual count process can be utilized for paper-based recounts.  I would expect that we are going to have a number of paper-based recounts, and to my knowledge again, I do not believe those procedures are in place. 

Lastly, I think we need to develop a test plan for observing critical procedures during the election process, such as the performance of the poll worker/VVPAT printer system.  We do not really understand why 10 percent of the paper ballots had problems.  We do not understand to what extent printers jammed and to what extent poll workers misfed paper.  How much of it is human, how much of it is machine, and how much of it is the interface?  Without having folks in the polling place to observe that, you cannot really know it.  We just know what the impact of that was in the election.  It is my understanding that there is great interest in developing requirements from many election subsystems.  It seems to me that November provides an excellent opportunity for additional discovery. 

So, we have got a great opportunity here in November, and I hate to miss it.  I mean, we are going to be doing some extra polling again in Ohio, but there is much more that needs to be done.  And quite frankly, there is still time, as long as election officials will acquiesce.  After November, I believe it is time to revisit the life cycle illustration I referred to in my earlier slide, and follow a well-defined path in an effort to deploy a system that is fit for its intended use. 

Specifically, it is ESI’s suggestion that in 2007 we focus on conducting intensive field test and requirements gathering in a limited number of pilot jurisdictions, perhaps six jurisdictions.  And the reason I chose six is because basically there are six major vendor systems out there.  Three major vendors each produce two.  We can go to six jurisdictions and cover basically most of the market place.  I then suggest we use these pilot efforts in 2007 to do the following:  Extensively field test the election system, using well-defined test plans and data collection strategy.  You have to design this stuff upfront before you go out there and start to test it. 

Next, significantly progress the requirements-gathering process, completing the next round of requirements/documents based upon this testing.  So, this testing needs to feedback into new requirements documents and those feedbacks into changes in the system.  Next, develop procedural handbooks to provide in detail all processes and procedures for operating the given system.  We cannot simply develop a new round of specs for the technology and not also develop complementary procedures for managing that technology.  I mean, the fact that these election officials do not have tools that allow them to do inventory control of these mission-critical assets is ridiculous, and can be easily solved. 

Lastly, define quality-control mechanisms, which I mentioned before, and baseline metrics, so that election officials and the public can monitor election system performance against these metrics.  Of course, this plan of action will take significant resources to accomplish.  Just to wrap up, I would like to finish by reflecting on my most recent letter to the Cuyahoga County Commissioners.  In that letter, I stated that while the challenges facing Cuyahoga County's election system are considerable, ESI remains confident that the most significant constraint for election system improvement is the will to achieve, meaning improvement in the election system with the cooperation of the voting system vendor, public officials, the public, the philanthropic community, and the media. 

The serious issues uncovered today can be addressed employing widely used engineering practices, management science methods, and process improvement techniques.  I think I stressed this point over and over again when I was in Cuyahoga.  The biggest challenge here is the will of the parties.  Once the parties will come to the table collaboratively, the science community can address these issues, I believe quite quickly.  Thank you.

Norman Ornstein:  Thanks, I am reassured.  I did have one question.  How many of the people working on this in Cuyahoga County had come straight from the Iraqi Provisional Authority?  One little housekeeping detail, which is that Deb Markowitz - we took a vote and we have audited it and it is verified - will actually be in the second panel now.  So, we will now move on.  Michael?

Michael Shamos:  I have a question for Steven, actually two questions.  In your study you found a discrepancy between the internally-recorded ballot images, I take it, and the totals.  There was some discrepancy there.  Was there any systematic shift?  Did it appear to be random?  And why are we not looking at the software that was used in those machines to determine what the source of this discrepancy is?

Steven Hertzberg:  I have been trying for four years to get access to software, and I have not gotten very far.  I think that is the essential next step.  But to answer your question, no, there was no powder.  Just to update you, Diebold has offered…  We went to Diebold after this [audio glitch] “Please provide us with possible explanations as to why we are having these discrepancies.”  The only discrepancies they have addressed so far is what Michael is referring to, the electronic to the electronic.  There is data stored in the memory card and there is data stored on the machine as, supposedly, a backup.  That should not be a mismatch. 

Computers are very good at copying data.  That is one of the things they do very well, as well as addition.  So, that was very surprising to see that.  I think what we are seeing from the explanations that Diebold has provided yet - we have not validated these explanations - is that human error can cause error in the machine count.  I think that is what I am seeing but I still cannot validate.  I do not think we have enough information.  Some of the forensic study that we asked to continue had to be shutdown because we got too close to the November election and they needed to get ready for that.  Last point:  I agree with you, the source code absolutely needs to be reviewed.

Michael Shamos:  I think you said that in Cuyahoga County in Ohio that the VVPAT is the official ballot, in those cases, where you had precincts with missing VVPAT rolls.  Do those voters not get to vote or where the unofficial electronic totals actually used in the canvas?  And on what legal basis, if you know?

Steven Hertzberg:  I will answer this way.  The electronic… the data from the memory cards was what was used except… I will just add a little bit more information.  They did have 70 memory cards missing at one time.  At least, that is the rumor.  I never validated that.  We did not look in that, but my understanding is they were able to restore those memory cards from the data on the machines, and they used that data to produce the official election results. 

As to your legal question, I think the good news there is I just saw an article in the Cleveland Plain Dealer about 10 days ago.  It has entered into the public debate.  Dave Lambert, who is the prosecutor in Cuyahoga County, says that it would be illegal to use the electronic ballots.  If paper ballots are missing, you have to rely on the paper ballot.  So, if they are missing, they are missing.  So, I do not think it has been resolved.  I think the court will ultimately resolve it.  But at least the debate started.

Michael Shamos:  Okay, thank you.  We will get to some give- and-take later on, but the memory cards that were missing, as I understand it, in many cases they hired cabs to take these memory cards and a lot of them just never showed up.  The cabs disappeared somewhere.  Is that accurate?

Steven Hertzberg:  Yes, I had heard reports that in Chicago when they had problems with their election recently that they had used cabs to bring the memory cards back to the Board of Elections.  At about 1 a.m. on election night, all the media is in the front of the Board of Elections Office and so we went around to the back.  And I was with a colleague and I said "I have a hunch.  Let us go take a look and see how are these cards getting delivered?" 

And, sure enough, there was a line of cabs.  Actually, I honestly do not have a big problem with cabs.  What I did have a problem with is that every single one - we stood there for 15 minutes - was unaccompanied.  They were all alone.  They all had bags of what are called transfer cases in the back.  I do not know if any of the cabs went missing, or what happened there, but just the fact that this mission-critical data was being delivered in cabs unaccompanied surprised me.

Norm Ornstein:  I would ask all of you, if you happen to be in a cab in Cleveland, please reach underneath the seat, and if you find something bring it over to Steve.  Thanks.

Michael Shamos:  Okay, if I can have the presentation.  I did not want to focus on the question for today, which is re we ready for 2006?  Of course, we are not ready and it does not matter at this point because there is nothing we can do about it between now and the election. 

Okay, I have answered that question.  We get on to the talk.  I want to raise this issue and I want to view it from an engineering viewpoint, which is the way I have always dealt with election systems.  What is the best method of voting?  HAVA basically has restricted us to a very small number of possibilities now. 

And I want to look at a systematic mechanism for developing an answer to this question.  I have an answer.  But much more important than your accepting my answer is accepting the methodology behind it.  You apply your own analysis to the methodology, you will come up with possibly different conclusions, but at least it will not be based on emotion and fear-mongering, which is a lot of what we have got these days.  My background is I did a lot of computerized voting system examination starting in 1980.  I have done 118 voting system examinations.  I am doing the 119th one next week.  Most of the critics of electronic voting have done zero examinations of voting systems.  A few critics have actually done one.  I know of one guy who has done more than 10.  I do not know anybody who has done 100.  I have been an expert witness in five electronic voting cases. 

The question, “What is the best voting method,” is constrained by HAVA and state requirements.  I have listed some of the technical requirements of HAVA.  For those legal squirrels, I have given you the actual citation to the particular sections, subsections, sub, sub, sub, subsections of the act.  The voter has to be able to… verify before casting the vote that the voters vote has been understood correctly by the machine, and the voter has to have an opportunity to correct it if the machine has not understood the voter before casting the vote.  HAVA requires over-vote warning.  It does not require under-vote warning.  It requires that a permanent paper record of each voter's voter be kept.  It does not require a contemporaneous voter-verified paper trail, but it does require a permanent paper record.  It requires disabled accessibility and alternative language accessibility. 

It is silent on these other issues down here, but various states in their way require all of those four things, require voter secrecy.  In most states, that is a state Constitutional requirement.  It is not a federal Constitutional requirement.  Some states actually require security of their voting systems.  Not every State does require that, although I do not think that there is any active effort to buy deliberately insecure voting systems.  Nonetheless, sometimes that provision is absent from their statute.  States require a certain level of reliability in voting systems.  It is not specified numerically.  It just usually says that the system has to be reliable. 

And usability, another requirement … if the voter is unable to learn how to operate the machine during the three minutes that they are allowed to be in the voting booth, then it is really no good to provide them with any kind of a machine at all.  So, what I decided to do was abstract a way from that previous slide's list of requirements, five really critical things that we need to have in election systems.  I have listed security first because that is the main topic of debate around the United States these days. 

Every other week there are some revelations about some horrible insecurity that has been found in the voting system and many of those flaws are indeed horrible.  What is not often mentioned is that many of those flaws are easily repairable.  That does not mean that the flaws did not exist and that they were not bad.  And so I ask you, in other areas of life, for example, the eating of spinach or the trust that we place in airplanes or in banks and other systems, when a security flaw is found, there is never a recommendation that we ban spinach. 

There is never a recommendation that we stop using banks.  What we do when we find a security flaw is we fix it.  Now, there has not been enough attention paid to what the remediations are that are necessary when we find flaws.  And every system that there is of any nature has a security flaw, has undiscovered security flaws.  It has security flaws we cannot even conceive of now because we do not have the infinite vision into the future.  What we have to do is be prepared when we find security flaws to fix them. 

Secrecy:  Secrecy, as I said, is a constitutional requirement in most states. 

Accessibility statutory, usability and reliability statutory.  Up here at the top, I have listed the only four voting methods I know of that are rationally usable in the United States:  the DRE without a voter-verified paper trail, the DRE with a voter-verified paper trail…  And the reason I put the word “current” there is I am basing the evaluation I have not given you yet, I am going to be basing it on the existing commercial systems that are now out there in the market, not systems of any conceivable design that you might develop next year.  It says what is there currently. 

Precinct count op scan.  Precinct op scan refers to a process in which the voter takes a sheet of paper similar to the answer sheet on an SAT, makes marks on it, and then the voter themselves goes over to a precinct counter and stands there and feeds the ballot into the machine, and that is the point at which the voter can receive the over-vote warning and things like that, as opposed to the old central count method in which all that was done at the precinct was collecting these pieces of paper, and then we had to trust that they would make it securely back to some central place where they would be securely and reliably counted.  And that is essentially unacceptable under HAVA. 

There are some exceptions where it is possible, but counties that are doing it are strongly encouraged to stop doing it.  And then precinct count op scan with a ballot marking device.  The ballot marking device is similar in most cases to a DRE machine, except it does not count, or record, or total any votes.  Its only function is to make correct marks of the prescribed blackness in the correct places on the ballot.  That avoids the situation were voters regard that piece of paper as a free-form kind of thing where they can cross out names, encircle names, and underline names and write them. 

And the variety of stuff that voters do is completely unbelievable.  It was attempted to be fixed by HAVA, which required every state to develop a definition of what constituted a vote.  And, of course, they have all done that, and the same thing is either a vote or not a vote, depending on what state you are in.  So, I do not believe there is one voter in 10,000 that has actually read the state document on what constitutes a vote, so they have no idea.  So, obviously by using a ballot marker, there are going to be certain kinds of problems that will disappear. 

Okay, what I decided to do was to rank these different voting methods, and this is not with reference to any particular manufacturer, but to the method itself.  I decided to rank them on a scale of zero to 10.  Nothing is ever going to get a 10 because 10 is perfection, and we professors never think anything is perfect.  So we never give hundreds, we never give tenths.  So the security of VVPAT I have ranked as seven.  There are people who would not agree with that. 

There are people who want to rank it as zero.  Robert Kennedy Jr. will want to rank it as zero because he wrote in the current Rolling Stone that the DRE machine is a hacker’s dream, believing that it is a siren that invites hackers to simply go in and tamper with elections.  Never mind the fact that in the 25 years that we have been using DRE machines in the United States, there has never been a verified incident of tampering with the DRE machine.  So, if it is a dream, it is a really uphill dream, but nonetheless … s So, you might not rank security of DREs without VVPATs as seven.  You might rank them as three, but at least give them a ranking.  With respect to secrecy, DREs without VVPATs are really pretty good.  There have been occasional programming errors once in a while that have left trails around from which you might be able to reconstruct how someone's ballot was cast.  These days, I know of no such flaws.  I did not give it a 10 simply because I do not know whether there might be such flaws, but I gave it a nine. 

Accessibility, there is nothing that is more accessible than a DRE because you have audio interfaces, sip-and-puff interfaces.  Anything that you can conceive of by way of communicating with a disabled person, it is possible to implement that on a DRE. 

Usability.  I cannot evaluate that independently since I have been voting on them for an incredibly long time.  In a typical voting system exam, I will spend six hours casting votes.  A very diligent voter who begins to vote the first year of their life that they are eligible and continues to vote through age 75 will spend approximately six hours’ total in voting booths throughout their lives.  I spend six hours in the voting booth every time I do an exam. 

So, I cannot tell you independently about usability.  I am basing this ranking on reactions that voters have had to the actual experience of voting on a DRE.  In this regard I give great weight to the opinion of my mother, who is 86, and voted on the ES&S iVotronic System for the first time in May of 2006 after having spent a life on lever machines.  And even though the iVotronic is not a full-phase ballot, is not really analogous to a lever machine, she loved it.  That is not the only data, but that is an important data point for me. 

Reliability.  I do not consider six a particularly good ranking for reliability.  There is anecdotal evidence that approximately 10 percent of DRE machines fail during the election.  When I questioned that on an internet bulletin board, I was informed by somebody that the current federal standards actually permit 10 percent of machines to fail on Election Day, although the reliability requirement is not phrased that way.  If it were phrased that way, there would be public outcry:  “What do you mean 10 percent?  Maybe 0.01 percent would be more rational.” 

So, how did this slip in?  It is because there is a requirement in the 2002 voting system standards that voting machines must exhibit a mean time to failure of 160-something hours… I think it is a 162 hours.  What that sounds like is that the average life of a machine from the time you turn it on until it fails is going to be 160-something hours.  That actually is not correct.  Well, maybe it is correct in an average sense, but it does not say anything to you about the number of machines that are going to fail during the election.  And if you use the standard exponential failure model that statisticians do for such things, you find that it is perfectly permissible to have 10 percent of the machines fail in the first 12 hours, so long as there are a lot of machines that go on for a year without failing.  Then you still get the mean time to failure, to come out right at 162 hours.  So reliability is not particularly good on DREs, but nonetheless within federal guidelines. 

Let see what happens if we add a VVPAT to the DRE machine.  Yes, security goes up and the reason why security goes up is that there are certain attacks on a voting machine that will not be detected in the DRE without VVPAT but will be immediately apparent in the DRE with VVPAT.  Whether you rank this a 7, and this is 9 or this is zero and this is 10 will depend on your risk assessment as to what you think the practicality is of such an attack being attempted and being successful.  We have virtually no data on that. 

Secrecy.  The current DREs with VVPATs, I should have given them a zero, but you would think I was prejudiced.  I give them a two on secrecy.  The reason for that is that on the sequential paper roll that is being kept in the machine, the first voters who showed up at the polls that day, theirs is the first one on the tape.  The last person, theirs is the last on the tape, and everybody else is in sequential order in between.  So, if there is ever an audit and we pulled the paper tape out of the machine, I now know how absolutely everyone in the precinct voted if I have access to the record of the order in which they showed up at the polls. 

Well, in Pennsylvania, that is called the poll list and it is required to be kept at every polling place.  As you show up to vote, the judge writes down a sequential number next to your name that tells who was first and who was last.  Even if you did not write everything down in the order in between, all I have to do is see who the first voter is who goes in, and I know his is going to be the first on the tape, and that is illegal.  You cannot have a voting system in which I can tell how somebody voted, unless the voter wants me to know. 

Accessibility.  DREs with VVPATs, I give them a five.  That is a compromise verdict.  They are accessible to people with certain disabilities but they are completely inaccessible to the blind because it may seem obvious, but a blind person cannot read the voter-verified paper trail, and therefore is not able to verify it.  So, whether you give it a zero or something else, I gave it a five.  Usability, six.  The reason that it has a usability of six is the paper trail itself is not particularly easy to use.  The reason is that the voters do not fully understand it.  That may change with a learning curve, but also it is often extremely difficult to see and interpret because the format of data that appears on the paper trail does not match the beautiful visual stuff that you are seeing on the screen. 

And so the process of actually comparing the paper trail to the screen creates a lower degree of usability.  The reliability of DRE with VVPAT is based on statistics.  Ten percent of DREs without VVPATs fail on Election Day.  Twenty percent of DREs with VVPATs fail on Election Day.  So, adding this new physical device with moving parts and paper and things to run out of doubles the failure rate.  So, I cut down the reliability from six to three. 

Next, precinct count op scan.  Precinct count op scan I ranked very low in security, and the reason for that is in precinct count op scan that your physical SAT answer sheet is the official ballot.  We have a very long history in the Unites States of people being very skillful at tampering with paper ballots.  They know how to stuff ballot boxes.  They know how to replace ballot boxes.  They know how to replace portions of ballots in ballots boxes.  They know how to throw ballot boxes in the garbage, in the lake, in the river, in the bay.  In 2004, three weeks after the election, in San Francisco, ballot boxes were found floating in San Francisco Bay.  And so if the belief is that we have a piece of paper and everything is okay, it was not true in 1850 and it is not true now. 

And the reason is that we have a highly distributed mechanism in the United States with hundred of thousands of voting precincts.  We have to have a dependable system for getting those paper records to some central place for audit purposes without people fiddling around with them, like cab drivers, for example.  Secrecy, I give precinct count op scan only slightly less than DRE without VVPAT.  The reason that it is not completely secret is that people write write-ins on the optical scan ballot and often they can be identified from their handwriting. 

And there are other problems.  They have to stand in line while they are putting this ballot into the machine and a tall person standing behind them gets a chance to look at their ballot.  It is not awful, but it is not a 10.  Accessibility, zero.  A blind person cannot mark a paper op scan ballot.  Usability, five.  The problem there is that voters find them to be incredibly usable.  The problem is the voter is not marking the ballot properly.  He does not realize that it was not usable.  So, if you write in somebody's name but you fail to mark the oval next to the write-in, it does not count.  Yet, it all looked really easy to you.  You just wrote somebody's name on the ballot. 

Reliability, very high.  There is not much that goes wrong with a piece of paper.  The scanning machine, it is very old technology especially at the slow speeds that precinct count machines go at.  It is extremely easy to make a paper feed mechanism that is reliable.  So, they have no problem there.  The ballot marker, we solved a whole bunch of problems with the precinct count op scan.  Security goes up.  The reason that security goes up is that in order to stuff the ballot box you have to stuff them with ballots that have actually been marked by the ballot marker, because, if not, we can easily tell that those ballot were not marked by the ballot marker. 

Secrecy goes slightly up.  Accessibility goes way, way, way up because for accessibility purposes, the ballot marker is the equivalent of a DRE.  It has all the same interfaces, audio, and sip-and-puff, and all of that.  Reliability goes a little bit down from precinct count without the ballot marker because then you have the reliability problem of the ballot marker itself, which can run out of ink.  It can mechanically fail, et cetera.  You add them all up.  So, what I did for this study… I did not want to call it a study.  For this off-the-cuff impression, I ranked all of these categories equally.

 Now, if you give security a weight of a million, and you give everything else a lower weight than that, then of course DRE with VVPAT is going to win because the 9 here is going to swamp everything else.  But let us not forget that these other things are also statutory in constitutional requirements.  So, you cannot just rank security as a million and everything else as one.  But I ranked them equally, and so I just get the total by adding these things up. And I did this honestly, though I had an idea in advance what might win.  I did not fudge the numbers to have it come out right, or I never would have had these two be tied - 40 and 40.  But they just came out that way.  And so, with this ranking, the equal weighting, DRE without a VVPAT is demonstrably better than a DRE with a VVPAT, and is tied with precinct count op scan with a ballot marker. 

And these two things are kind of wallowing way behind.  You have to do a tremendous amount to bring them up to snuff.  And so, what I am urging, in summary, is that, as it always has been, when you try to apply technology to anything, it becomes an engineering problem.  You give the engineers the constraints, they attempt to design and build a system that satisfies the constraints.  Whether they succeed or not should be measurable on scientific and engineering principles, not on people going around screaming about the effect that Hugo Chaves may have on American elections because of his supposed ownership in some parent of a parent of another company.  That is not an engineering argument. 

What I want somebody to do is to show me exactly what Hugo is going to do to affect one of these categories, then maybe we can worry about.  So, you do not have to agree with my numbers.  You do not have to agree with my list here.  You do not have to agree with my rankings.  But before you come to a different opinion, I want you to make a spreadsheet of your own like that in order to evaluate these systems.  Thank you.

Norman Ornstein:  Let me ask you a question.  First, I should note I went by a Sequoia Systems machine the other day and there was a distinct smell of sulfur.  So, I am not sure about Hugo Chaves.  The question is this.  If you have an election which is extremely close, and one which requires a recount, or at least in which the loser demands a recount and which in many states has a recount requirement because you are within a certain percentage of the votes, use that criterion to evaluate the DREs with no paper trail and the latter cases. 

Michael Shamos: I assume the latter case is the… is it anything other than the ES&S AccuVote, the PCOS with the ballot marker?

Norm Ornstein:  Auto mark?

Michael Shamos:  Yes.

Norm Ornstein:  Its auto mark.  So, evaluate that with that criterion.

Michael Shamos:  He is not a shill.  We did not agree in advance that he was going to ask me this question, but I am really happy to have a chance to answer it.  Representative Holt made an important point about auditability and transparency, and the question is if you look at all kinds of other systems that we have got in our lives, how do we audit them?  When an airplane crashes, we need to perform an audit to find out what happened.  What were the conversations that took place between the pilot and the tower?  What were the settings of the L-ronds [sounds like] and all the other instrumentation so we can determine whether it was a mechanical failure, pilot error or whatever? 

Now, while the passengers are riding in the plane, they do not see a paper print-out of the data that is being written to the black box.  They could.  Of course, it would scare the hell out of them, but they could.  You have the screen where they are showing the movie.  They could have a little inset that shows all the data that is being recorded.  So, when a plane crashes and we do the post-crash audit, how do we know that the audit trail was accurate and it had not been tampered with by terrorists or there was no programming error, et cetera? 

It is extremely simple.  You test to see whether the audit mechanism is working.  Now, we can debate a long time how much testing you have to do, exactly what you have to do, et cetera, but you can test it.  Now, if the audit mechanism is working, then the internal print-out at the end of the day, we know it corresponds to the ballot images of the voters as they voted because the audit mechanism is working.  If so, the election becomes fully auditable.  Not unauditable, not paperless.  These are fabrications.  Thank you for the opportunity to answer that question. 

Now, what about the auditability of [audio glitch] with ballot marker?  It is a big problem.  Because when you go and you try to take a look at those voter marked paper ballots later on sometime, it is not going to be on election night, it is going to some later time.  How do you know that those were the ones that were actually cast by the voters?  And the problem with all document ballot systems - the only kind we have now is op scan, but it would include plain paper ballots, it would include punched cards, et cetera -  is there exists only one copy of the voters’ vote.  If you tamper with that copy, it is over.  You do not know what to do. 

And in the precinct count op scan with the ballot marker, specifically the ballot marker is not totaling the votes because somebody does not trust it to.  So, if anybody tampers with or substitutes the op scan ballots, there is no point in going back to the ballot marking machine.  It is not going to tell you anything.  You have no data from which to reconstruct the action. 

Norm Ornstein:  [Audio glitch] it would be possible to create a machine that counts internally and that also has that trail?

Michael Shamos:  Yes, voter verification is a wonderful thing.  The problem is that the kindergarten methods that the vendors have developed up to this date do not actually solve the problem.  They make it worse.  And the reason I am sort of well-known as an opponent of paper trails… I’m not actually an opponent of paper trails; I’m an opponent of a statutory requirement for paper trails, and the reason for that is that once you freeze the technology and require its use, there is no incentive for anybody, maybe outside of the academic environment, there is no incentive for a vendor for example to build something better, because if he does not have a paper trail he would not be able to sell his machine. 

So, I think we are still in the drawing-board stage.  In fact, if we take Steven's chart of where we ought to be on the development schedule, okay, we have had ideation, sort of.  We think there needs to be some kind of permanent record, but we do not know exactly what it has to be.  Does it have to be paper?  What happens if the paper disappears, as in Cuyahoga County?

Norm Ornstein:  I have to ask you one more question.  You started by talking about how security is a problem everywhere.  It is a problem with the airlines.  It is a problem in so many other places.  But if the airlines reacted to questions or challenges to the security of their systems the way the manufacturers of election machines have reacted, we might have a very different dynamic in the country, including the questions about the security of flying.  Have you seen any other manufacturer of any other systems that has the lack of independent verification, or the response of vendors, comparable to what we have seen with voting machines?  And do you see any of these manufacturers as better than others when it comes to trying to be responsive to some of these questions and challenges?

Michael Shamos:  Let us take the issue of what vendors say and separate that outcome.  If an independent verification which is an ITA-NIST [sounds like] thing, which is in flux, and I have testified about that to Congress, and I do not need to say more about it today.  If you expect me to defend the ridiculous things that the vendors of voting system say about their systems when challenged, I will not.  If I had any cartooning ability, I would develop political cartoons around that very topic, okay?  This is beyond hoof-and-mouth disease.  This entire lower-half- of-body-and-mouth disease, what is going on? 

Now, is there any other industry in which folks behave that way?  Well, it actually used to be that way in the airline industry.  Whenever there was a crash of an airplane… and the reason we do not have it much is that airplanes do not crash much anymore.  But in the 1960s, they were crashing all the time, and what would happen is a massive finger-pointing exercise would take place where the manufacturer of the air frame blamed the manufacturer of the engine.  The manufacturer of the engine blamed the pilot.  The pilot blamed the tower.  The tower blamed somebody else. 

And so, I believe there is always an attempt by organizations.  Their first thought is "Let’s see if we can deflect the blame away."  The smart ones who realize quickly that they cannot do it own up to it and tell you what they are going to do about it.  There are classic cases in American corporate history of companies that have handled disasters well - the Tylenol case is the classic example taught in business school - versus companies that have not handled it well, and have gone out of business because of it.  So, it is ridiculous what they say.  I mean, they do not get it.

Norm Ornstein:  Okay, let us move on for the moment to Charles Stewart.

Charles Stewart:  Thank you, Norm and thanks to everybody for… especially Norm and Tom for running this great project, and for Tim for actually doing the electronics here and getting us all here today.  I think it was Doug Chapin who coined the phrase “election geeks,” and I think this is a panel of election geeks, folks who like to get down and dirty with data.  What I want to do is… we have heard the 10-foot view and the 100-foot view of elections.  What I want to do is to talk about performance of election machines in a kind of 30 thousand-foot view, which is the thing that I do, and talk about found votes since 2000.  We have a picture here in Florida of trying to find votes in Palm Beach County in 2000. 

One reading of the Help America Vote Act is that it was attempting to respond to the problems in Florida by which people went to the polls to vote, or woke up in the morning and wanted to vote and could not, and that one performance measure, not all performance measure, but one performance measure is to see how many of those voters actually succeed in getting their votes counted by the end of the day, or by the end of the vote-counting process. 

What I want to do is talk about the improvements in elections since 2000 in terms of found votes, that is, votes that would not have been counted in 2004 had it not been for changes in voting equipment and other practices in the time between 2000 and 2004.  And by a little bit of hand-waving suggesting that this at least gives us a sense about how things will be in 2006.  So, it might be the case that we are not ready for 2006 in the way that we should be, but we are not as unready as we could have been, which is maybe on faith, praise.  But it could have been worse.  I guess that is one of the take-home messages here. 

So, I have been involved with the CalTech/MIT Voting Technology Project, or as you call it, the MIT/CalTech Voting Technology Project.  Our URL is votingtechnology.org.  Go check out our reports.  This is a graphic that contains the information that Representative Holt made reference to earlier.  It has become sort of part of the lexicon of voting reform.  It talks about “Were our votes lost near and around Election Day?”  These are some estimates we made in 2001 about ways in which people who wake up on Election Day - I guess [indiscernible] not include election period - might discover if they want to vote, at the end of the day they have not had a vote counted for them. 

There is this process.  There is this chain of behaviors that have already been made reference to.  There is the registration of voters that might fail.  There is a verification of voters, getting voters into the precincts. That might fail.  There is actually the counting and rendering of ballots that is getting a voter the right ballot and then getting it counted in the precinct.  Things might go wrong there. 

And then finally, and this is something the security debate has really brought to our attention over the last two years and is one, I think … o One of the good things about the security debate is it has brought to mind that we actually could have problems with actually aggregating these votes after they have been counted in the precincts.  There are some estimates that we have used.  The data are as good as they get, which is not necessarily good, but it is the data we have. 

So for our estimate in 2000, we thought that between probably four and six million people having to vote uncounted on Election Day because of problems at these various stages.  The middle box is really where kind of voting machines and known technologies can fail. And by doing some estimation, it looked something like 1.5 million to 2 million votes there will be lost.  One, if you are just thinking about voting machines themselves and just thinking of the problem, which is not the only problem, an important problem is things like butterfly ballots and hanging chads and machines just breaking down and not working, we have a couple of million votes that we could extract out of the system by doing better at the voting machine level.  That is point one. 

Point number two is that is only the beginning.  And for instance as we are now moving into a security debate, or debates about IDs and registration rolls and the rest, we are actually I think moving into the area where there are even more votes to be recovered on Election Day.  But let us focus for the moment on machines.  So very quickly, there has been a lot of talk about this already and we are going to get to the Q&A.  To just all remind us, voting machines can lose votes for many reasons.  Sometimes there are just bad confusing interfaces and sometimes those are because of the features of specific machines.  There is blinking lights in some of these full-faced DREs that confuse the hell out of voters.  There is ballot layouts, things like butterfly ballots, et cetera. 

So, it could be the machines, there is something wrong with that.  It could be state election laws, right?  For instance, for most states the font specifications in the election laws about what ballots are suppose to look like or a how-to in how to poorly design a human machine interface.  That is how you do not do it, and that is written into many laws.  In Massachusetts, for instance, we have had the interaction of the state requirement that ballot questions be actually printed on the ballot was interacted with the Voting Right Act requirement that they be printed in English and Spanish, so that when we still had lever machines, you had to put … t The description of the questions was in four-point type, which for anybody over 25 is probably illegible.  And it was over most people's heads, too. 

So, that was by design.  As they used to say about Microsoft software, that was not a bug, it was a feature.  So, sometimes these problems are not bugs, they are features.  There are also performance problems and degradations.  Sometimes these machines break.  So, we had HAVA and we have been trying to figure out …  sSo HAVA started a path to try to upgrade some of these machines, and in some of these poor practices that were mentioned in their previous slide. 

I see Kim Brace [sounds like] back there.  His reports on the upgrade of voting machines… It looks something like 82 million voters will be using different machines in 2006 than they used in 2000?  At least registered voters, although, of course, all will not vote.  It looks like about half of that upgrading or changing happened between 2000 and 2004, and another half of that change has happened in the last two years. 

So, there is a lot of fast changing right now.  So, there is a lot of changing going on.  One of the things we did at the CalTech/MIT Voting Technology Project was try to come up with a fast and dirty, easily accessible and calculable measure of voting machine performance that is widely used, probably too widely, because there need to be other measures of performance.  But it is called residual vote and it is just simply the number of blank and over vote ballots in a jurisdiction.  And if you divide that by the turnout you have the percentage of all ballots that are residual votes. 

There is probably a floor on this.  There are people actually who turn out and do not vote.  There is a long line of public opinion research that suggests that something like half of one percent of people in presidential elections will admit to not voting for president.  So, you will never get residual vote down to zero.  Probably you will get down to something like 1/2 of one percent.  And in some races, you may not even get it down that close, like judicial retention races often times have 25-50 percent blank ballots.  People get tired, they do not know, they do not vote sometimes.  But in presidential elections at the top of the ballot, we have a feeling that this number should be "pretty low". 

But in any case, the residual vote … sSomeone asked me the other day, emailed me and wanted to know what the right residual vote rate is.  And the answer was there is no right residual vote rate.  This measure and other measures are really most valuable in making comparisons between machines, between versions of machines, between points in time within individual jurisdictions, as kind of a summary measure whether things are getting better or things are getting worse. 

So, let us see what happened between 2000 and 2004, and this is based on some research that I published in the Election Law Journal in May of this year.  In 2000, almost two percent of all ballots cast for president in the US were blank or over-voted.  That is roughly two million votes in 2000.  In 2004, that was just a hair over one percent.  So, to a first approximation, that is about one million ballots that were additionally counted for president between the two elections because of the reduction in residual vote rate. 

There are some states with really big reductions.  Florida, Illinois and Georgia were the three best.  [Indiscernible] recognized them as three states that had a lot of problems that were recognized in the newspapers.  Georgia … wWell, we want to get to the end, so I will not start telling stories.  But nonetheless, there are three states that have a lot of problems, did a lot better in 2004 by this measure.  There are three states at the other end, Connecticut, Nebraska and Indiana, states that actually were not known for having problems, really did not change a whole lot in the ensuing four years, and actually saw the residual vote rates go up a little bit.  Not a whole lot, except in the case of Connecticut, where they really held out for the lever machines for a long, long time.  To which, by the way, there is now kind of an affect if you look out in the states. 

I think this is kind of a general sense of things, is that it is a sort of “the first shall be the last and the last shall be first” sense among different states, that if you are lucky in 2000, a lot of states that were really bad, have done a lot of work.  And a lot of states that were not so bad did not do a whole lot.  And now that some of the rankings just in terms of attention to voting system irregularities or problems, in some cases, not all of them, but in some cases, have kind of reversed.  And so there are some states that [indiscernible] the ball. 

Finally, there are some states that did not report turnout in 2000 or 2004, or have something that looks like turnout but when you look at it, it really is not a turnout.  So, we cannot measure how they have done with respect to residual vote rating.  Here is the baker’s dozen of states that are in that category.  So, I am making the mistake of showing a lot of data in a presentation like this.  You can read the article.  I will just report that it is possible to begin using the residual vote rate, to begin to look at counties between any two points in time, in this case, between 2000 and 2004.  And seeing what happened to residual vote rates in those counties, which are presumably more or less stable in things like their demographics, partisanship, et cetera.  And see how the performance of their voting machines by this measure changed across those four years. 

And so I have given you kind of the most common changes like counties that used to have punch cards now have DREs, saw a residual vote rate drop of 1.6 percent between the four years.  At the other end, the very small number of jurisdictions that went from paper to op scans actually basically stayed flat with respect to DREs.  The takeaway here is two takeaways:  One, almost everywhere residual vote rates went down.  And I will say a few words about that in just a second.  The big thing was everyone knew this was a tough election and people were really careful about marking their ballots, so that is part of it.  People knew that there were problems and so they were paying particular attention.  By and large, jurisdictions that changed their voting machines did better than the nation as a whole. 

So, just to reiterate what I said before, between 2000 and 2004, there are about seven million more votes counted on Election Day.  Most of that was probably due to the surge in voters because of the competition over the election.  And in fact if you look even closer, it was the battleground states that had the biggest surges of all.  But if you do a little bit of statistical analysis and then try to run the counterfactual of what would have happened had none of these counties changed their voting equipment between 2000 and 2004 and kind of do some back-of-the-envelope calculations, it looks like the voting machine changes are associated with a million of those new votes.  So it is not most of the surge and turnout between 2000 and 2004, but it is a nontrivial number. 

In my book, a million is still a lot of people.  So, that was a lot of folks.  We usually notice, by the way, from year to year fluctuations of the size of a million people in turnout in presidential elections.  So, where do these improvements come from?  To be very clear, some of these improvements were due to the machines.  As Michael was saying, there are improved interfaces in the DREs.  We have things like hanging chads that are going the way of the lever machine.  So the machines aren’t getting better overtime, at least in terms of the interface.  That is pretty clear.  But there are other things going on as well, which makes me hesitant to say it is all the machines, per se. 

There is an interesting thing when you have controversy over elections, and when you change machines.  First of all, you might train your poll workers better.  This is certainly the case in Georgia, for instance, where I looked at it pretty clearly in the 2002 election.  It was not only that they instituted new DREs, but they instituted also a massive poll worker education process.  So, was it the machines?  Was it the poll workers?  Was it some combination of the two?  Probably some combination of the two, but we really cannot tell that at the 30,000-foot level.  [Audio glitch] by voters. 

I remember talking to Connie McCormick in LA County, who runs elections there.  We talked about the next election.  They still had these punch cards and she talked about how the very next election date… they educated voters about making sure there are no dimple chads, and people were taking the styluses and just  jamming them into the machines, and they were breaking the styluses.  They were making sure that there were no ambiguities.  So, voters know they need to be more careful, and maybe they need to know to be more careful in places that had the most problems in the past. 

Finally, there is the response of election officials to being scrutinized.  There was one state… I try to gather this data as soon as I can, and there was one state that used to report its turnout figures pretty close after the election, and it had been now a couple of months and there was no turnout. So, I called the person in the state election office that I developed a relationship with, and I said "So, when are you going to get the turnout stuff?"  And he said "Well, you know, we've decided that if there's anything wrong with the numbers we publish, the verified voting people are going to crucify us. 

And so we are checking and double-checking and making sure that everything is exactly right."  It is likely that some of the problems that we've have documented in the distant past…  In fac