Saving Mr. Data

I am beginning to think that enthusiasts of standardized testing and data in education accountability are feeling nervous these days.  Senator Lamar Alexander of Tennessee is the new chair of the Senate committee on Health, Education, Labor, and Pensions, and he has set about the long overdue process of renewing and/or revising the No Child Left Behind Act which required annual testing of all students in every state.  The outcome of that process and of the House’s parallel bill which left committee already and which failed to adopt a Democratic sponsored amendment to require states to adopt “college and career ready standards” and to use standardized test results in accountability systems, will play a significant role in the current policy environment that is best summarized as “test and punish”.

However, it is not just a Republican controlled Congress that is threatening federal mandates for universal and annual standardized testing.  An unusual coalition of small government conservatives and anti-testing progressives have joined with growing numbers of parents concerned with how test based accountability is consuming their children’s education.  The once unthinkable is now being thought out loud and in the open — Congress could reauthorize the Elementary and Secondary Education Act without provisions requiring that all states test all students in all grades.

This would, obviously, be a blow to a cornerstone provision of NCLB that once enjoyed bipartisan support as a necessary measure to ensure that states did not try to duck being accountable for all students.  It would also throw a huge monkey-wrench into favored policies of the Obama administration promoted by Secretary of Education Arne Duncan. While the Common Core State Standards might survive in some form without annual standardized testing, the testing consortia, Partnership for Assessment of Readiness for College and Careers (PARCC) and Smarter Balance Assessment Consortium (SBAC), began their work with the support of federal grants almost as soon as the standards were being adopted thanks to financial support from the Bill and Melinda Gates Foundation and federal incentives from the Race to the Top grant program.  If annual testing requirements were scrapped by Congress, it is an open question how many states would keep Common Core and stay with the testing programs created for it.  Another policy threatened by removing annual testing requirements is the assessment of teachers by the test scores of their students.  Despite wide criticism from the research community, Secretary Duncan remains firmly committed to tying teacher evaluations to students’ annual progress on standardized examinations, and without annual examinations of all students, you cannot run their results through discredited and unstable statistical models to determine if teachers deserve their jobs.

So defenders of annual testing have work to do in public if they are going to save their baby,

One such recent effort appeared in the New York Times on February 6th. Penned by Chad Aldeman, a partner at Bellwether Education and former adviser to Secretary Duncan’s Department of Education, it is simply titled “In Defense of Annual Testing”, and it lays out what are becoming familiar efforts to shore up presumably left leaning support for keeping NCLB’s annual testing requirements.  These are not arguments that should be casually dismissed, and they have the moral authority of some of the nation’s most distinguished civil rights organizations that originally signed on for the accountability measures in NCLB when it passed and who have reiterated their support.

They are, however, arguments that don’t stand up to serious scrutiny.

Aldeman opens with a brief assertion of a now familiar claim:

But annual testing has tremendous value. It lets schools follow students’ progress closely, and it allows for measurement of how much students learn and grow over time, not just where they are in a single moment.

This claim suggests that without ANNUAL standardized testing of ALL students then we will not know how INDIVIDUAL students are progressing through school.  It has echoes in Secretary Duncan’s anecdote of how he tutored a great kid in his youth who had been tricked into thinking he was progressing towards college but who was barely reading at a third grade level.  According to Secretary Duncan’s telling, this young man was the proverbial “child left behind” for whose education nobody had ever taken any real responsibility.  Annual testing was the only effective means to catch how he was being poorly served and demand that someone do something about it, and Secretary Duncan even presented annual testing of all students as a parental tool: “Will we work together to ensure every parent’s right to know every year how much progress her child is making in school? Or is that optional?”

The problem is that this line of thinking shows a staggering lack of imagination.

As an argument, it fails to acknowledge that there are many other, and far more interesting, points of data that can be used by teachers, parents, and schools to keep far more compelling tabs on student progress throughout the year.  Locally designed and implemented ongoing assessments such as portfolios and project based learning can provide teachers with ongoing and meaningful insights into how children are learning, and report cards can be reformed to provide parents and guardians with far more nuanced information.

It is even possible to use externally designed assessments in more interesting ways that help inform teachers, students, and parents in real time.  Bruce Baker of Rutgers University notes that computer-based “adaptive assessments” given individually and with no stakes attached can be the basis of a system of formative assessment that gives immediate feedback to teachers about student progress that can be used to craft individual learning plans for those who struggle.  In coordination with portfolios and project based learning (and with full disclosure of how they use data and substantial privacy safeguards), computer based assessments could become a valuable tool based on data.  Dr. Baker further notes such data-driven and formative assessments would be vastly more useful than mass administered tests whose results are distributed months after the fact when students have already moved on to their next grade levels.

Aldeman dedicates most of his Times piece, however, on the belief that NOT testing every child in every year will allow states and localities to wiggle out from being accountable to all students all the time:

The grade-span approach would eviscerate the ability to look at particular groups of students within schools. Instead of having multiple grades over which schools could compile results, each school would be held responsible only for the performance of students in a single grade. Not only would this lower the quality of the data, but it would also raise the stakes of the tests: If you think the stakes are too high now, imagine being a fifth grader in a school where your score determines the results of the entire school.

Worst of all, under this approach, far fewer schools would be on the hook for paying attention to historically disadvantaged groups of students. A school with 10 Hispanic students in each grade would no longer be held accountable for whether those students were making sufficient progress, because the 10 fifth graders wouldn’t be enough to count as a meaningful population size.

Let me state that as a liberal with an eye for history, this argument is certainly intriguing.  We are a nation that only 12 years before my birth required the National Guard to let nine African American students attend a high school ordered to desegregate.  In 1969, the year I was born, the Supreme Court issued a ruling calling for Southern states to cease delays in desegregating their schools — a full 15 years after Brown v. Board found such arrangements unconstitutional.  Federal legislation and federal court cases were also instrumental in holding states and municipalities responsible for ending gender discrimination, for providing students with disabilities access to an education, and for providing support for students learning English.  It is no doubt this record of positive intervention at the federal level and of state delays in implementing equality of opportunity that motivated civil rights groups to endorse annual testing in NCLB and to stand with it today.

That does not change that such testing is unnecessary, is unacceptably disruptive to learning, and is narrowing curricula nationwide.

Mr. Aldeman is suggesting that eliminating annual testing will mean huge swaths of children will be hidden from the test and that the test stakes will be raised enormously with only exam being used.  The stakes argument hinges on a mistaken impression of what the exams say and what should be done with the data they produce.  For the stakes on gradespan testing to be even higher than they are today, one has to assume that such testing is used not only to monitor the education system but also to actively punish schools with low test results.  While few would argue that schools with poor results should be permitted to languish, the kinds of punitive measures embodied in NCLB are not a necessary result of monitoring student test scores.  Under the leadership of Superintendent Tony Alvarado, New York City’s Community District 2 implemented a complex and interconnect culture of reform that included standards and assessments.  However, data from the assessments were used to monitor how schools in the district were doing and to allocate resources for improvement and innovation where they were most needed and with the constant goal of instructional improvement.  Again, Dr. Baker of Rutgers makes a salient observation:

Here’s the really important part, which also relates to my thermometer example above. The testing measures themselves ARE NOT THE ACTIONABLE INFORMATION. Testing provides information on symptoms, not causes or underlying processes. It is pure folly to look at low test scores for a given institution, and follow up with an action plan to “improve test scores,” or close the school if/when test scores don’t improve, without ever taking stock of the potential causes behind the low test scores. TEST SCORES ARE SYMPTOMS, NOT CAUSES, NOT ACTIONABLE IN AND OF THEMSELVES.

Recognition of that fact and crafting policy responses to low test scores with that in mind would necessarily lower the stakes on the tests themselves.

Further, while there might be some argument for an annual test that could contribute to closer monitoring of those symptoms, there is no argument that convincingly says that such tests must be given to every student in every grade in order to get a good picture of how schools and school systems serve historically disadvantaged children.  First, a low stakes system of formative assessments, both qualitative and quantitative, could apply to all children and would conform to accommodations for children with special needs.  So there can readily be ways for teachers, schools, and parents to know how ALL students are doing during the course of the year.

Once we’ve set aside the issue of having a meaningful, formative assessment system for all students that can actually assist teachers, there’s no truly compelling argument against properly devised sampling of students for standardized testing.  Implemented correctly, sampling would not leave substantial numbers of children invisible as Mr. Aldeman fears, and we would stop spending inordinate time trying to ferret out distinctions in performance within schools when, as Dr. Baker once again notes, the greatest and most consequential differences in test measured achievement exist between schools and districts, not within them.  Insisting upon keeping annual testing of every student in every grade keeps an unnecessarily disruptive system in place as part of an accountability system that, in fifteen years, has not yielded sufficient results to justify the sacrifices in teacher autonomy over instruction and the sacrifices in non-tested subjects being shunted aside in favor of test preparation.  In fact, the only people to “benefit” from this system are private test designers like Pearson, who are being handed not just lucrative contracts but also terabytes of data to mine for new products, and advocates of firing as many teachers as possible based upon student test scores.

This is especially frustrating to me because data, when used with a clear understanding of what it can and what it cannot do, is a tool, an important tool at that.  It can help us develop broad pictures of what is happening in schools, and it can direct our attention to places that require more careful and nuanced study.  The persistent overreach and abuse of its capabilities is building a backlash that makes it much harder to successfully advocate for more judicious and appropriate use of what can be learned.  If we wish to SAVE data and its uses in school, it would be best to set aside NCLB and begin again sensibly.

I believe I see the problem, Captain. My head’s been severed.