Tag Archives: Big Data

October 29, 2015 · 5:04 pm

When is a Pledge to Decrease Testing Not a Pledge to Decrease Testing?

Apparently, when President Obama makes it.

Honestly, at this point in his administration, expecting President Obama to well and truly take action to reverse the damage of the “test and punish” era of school accountability is like expecting the Bush administration to not start unnecessary wars. That, however, did not prevent the national media from declaring that President Obama’s weekend call for reducing the burden of standardized testing in public schools a major departure from previous policies. David Dayen of Salon gushed that the President was breaking “with twenty years of precedent,” and Mother Jones’ Julia Lurie wrote that “the announcement represents a significant change in course for the Obama administration.” Nearly every major news outlet declared the announcement a move to limit the time spent on standardized testing in school, and American Federation of Teachers President Randi Weingarten hopefully declared the announcement a move towards fixing an urgent problem in education today:

Important Course Correction+hopefully,just the beginning,Obama Administration Calls 4 Limits on Testing in Schools https://t.co/fyXoE7GRJ1

— Randi Weingarten (@rweingarten) October 24, 2015

People deeply informed on the issue of high stakes testing and its warping impact on our schools are far less hopeful than President Weingarten and not remotely as gushing as the national press. Peter Greene of Curmudgucation held no punches over the weekend, flatly declaring that the Obama plan “sucks and changes nothing.” His key points are entirely accurate and properly cut through the smoke and mirrors of the announcement to a purpose more aimed at trying to trick anti-testing advocates into complacency:

The fact that the administration noticed, again, that there’s an issue here is nice. But all they’re doing is laying down a barrage of protective PR cover. This is, once again, worse than nothing because it not only doesn’t really address the problem, but it encourages everyone to throw a victory party, put down their angry signs, and go home. Don’t go to the party, and don’t put down your signs.

Anthony Cody of Living in Dialogue noted, quite correctly, that President Obama has sounded this note before and utterly failed to follow through with anything that would diminish the punishing role of current testing policies. The administration apparently hopes the announcement and some minor shifts will allow them to bide their time while changing very little:

First, President Obama remains unaware of the very limited educational value of standardized tests, and second, the administration remains absolutely committed to tests playing a key role in America’s classrooms. As some have pointed out, now that the PARCC and SBAC tests are here, and have plainly failed to deliver on Duncan’s 2010 promise that they would measure creativity and critical thinking so much better than any previous test, now we are looking forward to the NEXT generation of tests, which will be “competency-based.” Cue the test vendors for another multi-million dollar development project.

No matter how bad the current tests are, the new and better tests are always just around the corner. And anyone who dares to question this optimistic projection is a Luddite afraid of accountability.

Dr. Audrey Amrein-Beardsley, an expert on value added measures at Arizona State University, was not impressed with the announcement either, noting that the proposed 2% limit on time spent on testing would still mean 18 hours of annual standardized test taking time for most students. She further observed:

In addition, all of this was also based (at least in part, see also here) on new survey results recently released by the Council of the Great City Schools, in which researchers set out to determine how much time is spent on testing. They found that across their (large) district members, the average time spent testing was “surprisingly low [?!?]” at 2.34%, which study authors calculate to be approximately 4.22 total days spent on just testing (i.e., around 21 hours if one assumes, again, an average day’s instructional time = 5 hours). Again, this does not include time spent preparing for tests, nor does it include other non-standardized tests (e.g., those that teachers develop and use to assess their students’ learning).

So, really, the feds did not decrease the amount of time spent testing really at all, they literally just rounded down, losing 34 hundredths of a whole. For more information about this survey research study, click here.

Interestingly, the 2% idea apparently comes from Secretary Duncan’s slated replacement, former New York Commissioner and current senior adviser, Dr. John King Jr. who puts such a limit in place in New York in order to placate growing concerns over the dominant role of standardized testing in the state.

Well, we all know how that turned out, right?

Perhaps most damning was the scathing response penned by Robert Pondiscio for US News and Word Report. Mr. Pondiscio is a senior fellow at the Thomas B. Fordham Institute, a conservative think tank that has been highly supportive of the Common Core and associated testing, an adviser to the Democracy Prep chain of no-excuses charter schools, and while he is generally well disposed to the data from standardized testing, he has also been willing to question to impact of the stakes attached to them in the current environment. That questioning was in overdrive in his commentary:

But one would have to be cynical or naive not to understand that the moment you use tests, which are designed to measure student performance, to trigger various corrective actions and interventions effecting teachers and schools, you are fundamentally shifting tests from providing evidence of student performance to something closer to the very purpose of schooling. This is precisely what has been occurring in our schools for the last decade or more. When parents complain, rightfully so, about over-testing, what they are almost certainly responding to is not the tests themselves, which take up a vanishingly small amount of class time, but the effects of test-and-prep culture, which has fundamentally changed the experience of schooling for our children, and not always for the better.

The Obama talk on testing seeks to curry favor with parents and teachers (and their unions) while doing nearly nothing to change the fundamental role of testing and its effect on schooling. It’s all well and good to “encourage” states, districts and schools to limit testing, but as long as test-driven accountability measures, which are driven substantially by federal law, are used not to provide feedback to parents and other stakeholders but to trigger corrective measures in schools, it won’t matter if children take two tests or 2000; the effects will be the same.

While I question the degree of positives that Mr. Pondiscio lavishes upon standardized testing data (“the life-blood that courses through the arteries” – really?), I am not, myself, against limited standardized testing being part of a comprehensive system of school monitoring and being the very beginning point of school improvement efforts. What is most striking to me is how clearly, however, that Mr. Pondsicio has identified the problem with the perverse incentives testing has placed upon our schools in the era of No Child Left Behind and Race to the Top: The stakes placed upon the tests have transformed their purpose from being “in the background” monitors of schools, school systems, and state performance into being objects unto themselves. The tests and “adding value” to student performance on them have become a substantial purpose of education instead of a by product of a rich and meaningful educational program.

That’s a problem, and it is good that someone prominent in education reform circles has noted it for some time now and is willing to go on record in a major publication to call President Obama and his education team to the mat for it. Mr. Pondiscio, who says test based measures are the most reliable and objective teacher evaluation tool, appears willing to give that up because its side effects have driven teachers away from the Common Core and from any testing whatsoever. I disagree vigorously with the idea that test based measure are either reliable or objective (and the bulk of the research evidence is on my side on this), but I actually sympathize with Mr. Pondiscio’s predicament and his apparent frustration that the administration steadfastly refuses to get it. I have written on this before, urging reformers who really want a chance at building support for common standards and who value the use of standardized testing at all to decouple them from high stakes before popular revulsion violently swings the pendulum out of their reach for the next two decades. Common standards, done thoughtfully and carefully (the Common Core were not) and disseminated by genuine common interest among states entering fully voluntary partnerships (the states in Common Core did not) and offered to teachers with appropriate time for development of their own knowledge and curricula with high quality materials (teachers in Common Core states never got that) is a defensible proposition. Comprehensive system monitoring that uses standardized test data limited to the purposes for which it can work well is also entirely defensible.

It is also swirling in the drain reserved for ideas that end up flushed out of the education system, and Mr. Pondiscio appears aware that he has many of his own allies to blame for it, and, hence, his frustration. The problem, however, is one that his allies in Washington and various state capitols also seem unwilling to acknowledge, and unless, they do acknowledge it, they have little incentive to back off of testing policies tied to high stakes.

The problem is that they are lazy.

School accountability and improvement is difficult and often uncertain work. When used honestly, standardized test score data can tell you where to begin, but it should never be confused with evidence of what needs to happen in a school. Are there schools with low test scores and low value added that are Dickensian nightmares that should be closed as soon as possible? Sure. There are 98,000 public schools in the country. But there are also schools with low test scores and low value added that are full of devoted teachers, strong school leaders, and committed parents, but who need resources to provide genuine educational opportunities for all learners and to do so in a way that does not cheat them of a well-rounded and holistic education. For that matter, there are schools that boast of their great test scores and high value added, but they get there by being Victorian work houses worthy of Scrooge where children are basically beaten into submission.

The point is that you do not know until you go to the school and actually investigate.

But the Arne Duncans and the John Kings do not want to do that. They want to sit in offices in Albany and Washington, look over spreadsheets, and make sweeping judgements about which schools are winners and which schools are losers. They cannot really give up the high stakes attached to the standardized tests because that would mean they would have to do the hard of work of accountability and renewal, the work that actually can inform smart choices based upon community input.

And we can’t have that, now, can we?

3 Comments

Filed under Arne Duncan, Common Core, John King, Testing, VAMs

Tagged as Arne Duncan, Big Data, Common Core, reporting, testing, Thomas B. Fordham Institute

July 24, 2015 · 5:02 pm

Dear Senator Gillibrand: Public Schools Need Advocates, Not More Punishments

Dear Senator Gillibrand:

I am writing to you today wearing a number of different hats that I hope you will respect. First, I am a constituent living in New York City who has been pleased to vote for you in the past. Second, I am a life long educator, having studied education at our mutual alma mater, Dartmouth College and having taught at every level of school from junior high school to graduate school since 1993. Third, I am a scholar of public education, having earned my doctorate in 2002 and currently serving as the director of secondary education preparation programs at Seton Hall University. Fourth, and most importantly, I am the father of two public school students whose future education depends heavily upon the incentive systems that you and your fellow lawmakers vote upon in Washington, D.C.

I am writing to you for two reasons in particular. As Jon Stewart noted on your recent appearance on The Daily Show, you have a reputation for working across the aisle on various issues and an ability to find common ground where few believe it exists. I am also writing because you recently voted, along with almost all other Democrats, for Amendment 2241 to the “Every Child Achieves Act” introduced by Senators Coons, Murphy, Booker, Warren, and Durbin. According to Senator Coons’ announcement of the amendment:

Specifically, the amendment would require state accountability systems to provide additional resources and support to local schools identified as any of the following:

In the bottom five percent of public schools as according to the state accountability system

A public high school where two-thirds or fewer students are graduating on time

Any public school where economically disadvantaged, disabled, minority, or English Language Learner students are not meeting state-set goals for achievement.

The sponsors asserted that the amendment was a “serious departure” from the No Child Left Behind accountability system as it mandated no federal consequences and left it to states to determine the interventions and consequences for schools that continue to struggle. Despite these assurances, there remained significant reasons to oppose the amendment, reasons that nearly every member of the Democratic Caucus appeared to discount.

1. The amendment baked test and punish into the ESEA re-authorization. While the announcement made a big deal about about states determining their accountability systems and interventions, the language of the amendment continued to emphasize test based systems and even echoed the “college and career readiness” language that is emblematic of the Common Core State Standards and their accompanying tests. So while the amendment may have been presented as increasing state control of education and accountability, the actual language had significant emphasis on standardized testing, student growth measures, and statewide (standardized) measure(s) “which is consistent with progress toward readiness for postsecondary education or the workforce without the need for postsecondary remediation.” Informed readers of the amendment recognize a continuation of the Common Core State Standards and annual testing with the state’s required to base their accountability upon such testing.

The emphasis on quantified measures is also present in the language requiring states to create interventions for schools in the “bottom 5 percent”. While there is little doubt that many states have significant numbers of schools that struggle and which struggle for years at a time, the need to identify the “bottom 5 percent” each and every year is a kind of trap that means no matter how well a state manages to improve its schools, there will always be a portion of them labeled as failures in need of extra interventions. Further, by emphasizing the quantity, the amendment would have guaranteed the further primacy of testing in accountability.

It may be well-intentioned for you and your Democratic colleagues to insist that states not neglect their most distressed schools and student populations, but it is well past time to move away from annual standardized testing. We are almost a decade and a half in the No Child Left Behind era, and the data could not be clearer: high stakes testing and consequences do not work to substantively improve schools. Kevin Welner and William Mathis of University of Colorado at Boulder brilliantly called for a sharp move away from test based accountability:

The ultimate question we should be asking isn’t whether test scores are good measures of learning, whether growth modeling captures what we want it to, or even whether test scores are increasing; it is whether the overall impact of the reform approach can improve or is improving education. Boosting test scores can, as we have all learned, be accomplished in lots of different ways, some of which focus on real learning but many of which do not. An incremental increase in reading or math scores means almost nothing, particularly if children’s engagement is decreased; if test-prep comes at a substantial cost to science, civics, and the arts; and if the focus of schooling as a whole shifts from learning to testing.

The way forward is not to tinker further with failed test-based accountability mechanisms; it is to learn from the best of our knowledge. We should not give up on reaching the Promised Land of equitable educational opportunities through substantially improved schooling, but we must study our maps and plan a wise path. This calls for a fundamental rebalancing —which requires a sustained, fair, adequate and equitable investment in all our children sufficient to provide them their educational birthright, and an evaluation system that focuses on the quality of the educational opportunities we provide to all of our children. As a nation, we made our greatest progress when we invested in all our children and in our society.

2. We don’t need annual testing of all children to find the problems we know are there. Bruce Baker of Rutgers University makes it very clear that testing for accountability does not need to be a disruptive or annual affair. Using sampling methods it is entirely possible for states to get a very accurate view of what is going on in their schools, and the insistence that we need to test everyone in every school every year, is based upon false premises of how our students are distributed and about how accurately testing can reflect upon individual classrooms. Worse, we already know how tightly coupled test results and the demographic characteristics of a community are so that we likely do not need to test in order to know which schools are likely in need of more assistance. My colleague, Dr. Chris Tienken of Seton Hall University, very neatly demonstrated this recently with a sophisticated regression analysis of different social capital indicators that accurately predicted test scores.

Standardized testing, then, is an endeavor that is best done is the least intrusive ways possible to keep one very broad eye on the community and to use the results to see if further, more detailed, analysis is necessary or not. By attempting to retain them as a tool of high stakes accountability, Senate Democrats sought to maintain a lever which has failed to create significant results for an entire generation of students.

3. Resources matters — Democrats’ language on that was weak. Just as we know that schools with high percentages of students in poverty indicate schools likely to struggle, we also know that our communities with poor families tend to have large percentages of them and lack community resources as well. The language of the amendment called for states to identify struggling schools and to ensure “identified schools have access to resources, such as adequate facilities, funding, and technology” but the federal role in assisting this remains weak even as the federal government makes requirements upon states and municipalities. While the amendment had references to many grant programs, it lay primary responsibility with the states while leaving one of the core inequalities in American education intact: how we fund schools and distribute resources.

Local funding by property taxation is an inherently unequal form of funding, and we rely upon state aid to provide additional funding, aid that is inadequate to the task. Consider our home state of New York. Commissioner Elia has identified 144 schools statewide that are struggling or persistently struggling as measured by state test scores. These include schools in Albany, Buffalo, Hempstead, Mount Vernon, New York City, Rochester, Utica, and Yonkers. It should come as no surprise that all of these districts are on the list of our most underfunded high need school districts in the state. Based on the state’s own, inadequate, foundational aid formula, Hempstead should be getting $6,426 per pupil MORE this year than it is getting, and such accounting trickery has been played on every district in the state for years.

If the federal government were truly interested in helping our schools by holding states accountable, it would do better to focus upon how the different 50 states raise and distribute funds to our highest needs schools.

4. Test based accountability misses the real issues. We can test and test and test some more. We can gather as much data from those tests as we like. But they will never tell us the underlying reasons for the gaps in test performance among our population. Assuming that the tests are measuring worthwhile skills and knowledge, the existence of a gap in test measured performance tells us nothing about why it exists. At its best, testing gives us an idea of where to examine more closely, but when raising the test score become a paramount concern for schools and districts, the consequences are not inherently desirable.

Can the federal government assist states and municipalities in the pursuit of an equitable education for all? Certainly, but it would mean shifting the focus to resources and funding and away from test scores. For example, the federal government could finally fulfill its promise of providing 40% of the cost to implement the Individuals With Disabilities in Education Act which it has never done. The federal government could listen to its own data that suggests our nation’s schools need $197 billion in capital improvements and that a full quarter of schools with more than half of students qualifying for free and reduced lunch are in either fair or poor condition. The federal government could focus improvement efforts on questions of teacher retention in our most struggling schools which, contrary to the rhetoric of those opposing teacher tenure, is a much greater problem than teachers staying too long.

The federal government could also learn from history. After 14 years of testing and punishment, some tiny gains in the National Assessment of Educational Progress can be seen, but those gains are dwarfed by the closing of the student achievement gap measured through the 1970s and into the early 1980s: $13 year old math NAEP$

As you can see, from 1973 until 1986, the gap between White and Black 13 year-olds in mathematics shrank by 22 percentage points, at which point it began to rise again, slowly over the next decade, then decreased slightly after the passage of NCLB but at nowhere near the rate we saw from 1978 until 1986. What explains this? There are, of course, many possible reasons, but one that stands out in policy is that by the early 1980s we largely abandoned efforts to integrate our schools and to integrate our communities via fair housing policies. Since the 1980s, our communities have become even more segregated by income levels, and our schools have re-segregated as well so that today, a typical African American student in 2007 attended a school where 59% of his peers were low income, up from 43% in 1989.

For almost 4 decades we have increasingly concentrated children with very great needs within communities that struggle to provide basic services and in schools that are consistently deprived to the resources and personnel they need to give the children in their care what they need to thrive. We do not need more federal accountability measures of this. We require action aimed at the opportunity gap.

You have a deserved reputation for fairness and for finding ways to work with colleagues when others prefer to fight. I challenge you to research these issues and bring them to your fellow lawmakers in bipartisan fashion. I challenge to craft a federal education policy that emphasizes support and growth over test and punish. Use federal leverage with states to make sure state aid to local schools is up to their needs. Propose the full funding of IDEA for the first time in its history. Challenge colleagues to invest in the capital improvement needed so our children learn in buildings that are well equipped and safe. Find federal resources that will help urban schools with recruitment, development, and retention of teachers.

And recognize that threatening schools with standardized test results cannot overcome our society-wide abandonment of integrated schools and communities. Our public schools need advocates in Washington, not an entire caucus ready to reassert policies that distort education’s focus and ignore the real funding needs of our children’s schools.

Sincerely,

Daniel Katz, Ph.D.

2 Comments

Filed under Common Core, Funding, NCLB, politics, Testing

Tagged as Andrew Cuomo, Big Data, College and Career Readiness, opportunity, Poverty, testing

March 15, 2015 · 8:25 am

Pearson’s Intellectual Property — Why Is This Even a Thing?

Bob Braun, a five decade veteran of the Newark Star Ledger and currently an independent blogger, blew up a portion of the internet on Friday by reporting that Pearson, the international education giant responsible for the PARCC examinations currently underway, was “spying” on students’ social media activity. According to a letter from Watchung Hills Regional High School District Superintendent Elizabeth Jewett, the district test coordinator got a late night phone call from New Jersey DOE after Pearson initiated a “priority one alert” for a breech of test security within the district. NJDOE informed the district that they believed Pearson’s alert was for a student who took a picture of a test item during testing and posted it to Twitter, and the state suggested that the district should discipline the offending student. However, upon examination, the district ascertained that a student had tweeted a comment well after testing was over and included no picture at all. The tweet has since been deleted by the student, but given the 140 character limit on Twitter, it is extremely unlikely that any significant breech of test security could have possibly occurred. However, the incident revealed that Pearson is monitoring social media for any and all references to the testing going on and is prepared to initiate state level investigations of individual students (how else would NJDOE know the district and student involved?) over very flimsy circumstances.

The story took off very quickly as did Mr. Braun’s accusation that Pearson is “spying” on students’ social media. The web site was loading very slowly on Friday night likely due to very high traffic, but by later that night it was completely inaccessible and Mr. Braun reported on Facebook that his web host informed him a denial of service attack was underway from an as of yet unidentified sources. Meanwhile, outraged parents and anti-testing/anti-PARCC sentiments took off in social media:

#parcc #pearsonisbigbrother #pearsoniswatching #monitoredbypearson pic.twitter.com/wKN0DEEJ53

— Sarah Manville (@manville71) March 14, 2015

Social media monitoring with #Tracx may not be illegal Big Brother #pearson1984, but it's still creepy. #PARCC #pearsoniswatching #Pearson

— Alicia Buck (@alleybLFP) March 14, 2015

@StephanieSimon_ Hoping & looking forward to reading a story from you on #PeepingPearson #pearsoniswatching #pearson1984 #PARCC #WhyIRefuse

— Allison M. White (@allionthemove) March 14, 2015

On what planet does a multinational monolithic testing company have any right to demandithat minors acquiesce to its code of conduct?#PARCC

— Phyllis Bush (@qbgone) March 14, 2015

Let me state that I am unconvinced that “spying” is exactly the correct word over “monitoring.” The reality is that most corporations of any size are monitoring social media routinely to check on their reputations and potential scandals. In a world where social word of mouth is genuinely a thing, it makes business sense for them to do so, and social media is not communication in the private space. If you don’t believe me, wait until you have a bad customer experience with your cable company and then take to Twitter about it — If you don’t get a response from someone in corporate within 24 hours, I owe you a coffee.

However, even from a “monitoring” social media perspective, Pearson’s actions are troubling. I will concede that the company — and participating PARCC states — have an interest in test security while a standardized test is being deployed (although I also agree with Peter Greene that this level of test security does not bode well for the quality of these exams), but what, exactly, causes Pearson to raise a “priority one alert” and contact a state department of education with sufficient information to locate a district and specific child in question? What information about a minor’s social media use does Pearson consider its business to pass along to the top education officers in a state? To what depth does Pearson consider itself able to impose a gag order on other people’s children and use state capitols to enforce it?

Remember — the child in question did not send out a photograph of the exam, merely a single tweet limited to 140 characters AFTER testing for the day was over. For that, Pearson initiated contact with the NJDOE that sent Trenton thundering into the student’s social media account and alerting district officials when frankly, nothing should have happened at all. Thankfully, Superintendent Jewett is reasonable and knowledgeable about social media; it could have easily gone south really quickly.

Pearson’s hyperactive attitude towards test security is disturbing not only because of how it is being enacted without concern of proportion, privacy, and the implications of initiating state level investigations into unremarkable student speech. It is also disturbing because of its connection to Pearson’s larger perspective on its intellectual property and the allowance the public sector gives them in defense of it. While discussing this on Twitter, I encountered a user who stated that he “applauded” Pearson “defending its intellectual property,” which led me to a single question:

Why is Pearson’s intellectual property even a thing after it delivers a exam to be used for public education?

Considering the following:

PARRC was seeded with part of a federal grant worth over $300 million to create examinations for the Common Core State Standards.
Pearson was the only bidder for the contract to write the examinations for PARCC.
That makes the Pearson written PARCC examinations the only CCSS examination in 12 states and the District of Columbia — Pearson writes CCSS aligned examinations for other states such as New York.
Pearson’s contract with New Jersey alone is worth more than $100 million over 4 years.
The examination is high stakes – with implications for teacher evaluation and a possible future role in graduation requirements.
The examination is used by the state to fulfill federal requirements under the No Child Left Behind Act that all students in all schools between grades 3-8 and in grade 11 be tested in English and Mathematics. Unlike other standardized examinations students takes, these exams are mandated by state and federal laws.
Pearson has no intention of releasing complete copies of this year’s exams even after they have been fully deployed and assessed.

This isn’t even like copyright rules preventing photocopying textbooks — textbooks publishers rightly expect that schools will buy enough copies of their texts for students using them, and they are in direct competition with other potential text providers. Pearson has an exclusive contract to provide examinations for millions of students (a contract it did not exactly sweat bullets to obtain). These examinations are used for high stakes purposes. The examinations fulfill federal mandates for testing in our public schools, and they inform personnel decisions locally, administrative decisions at the district and state levels, and federal actions nationally. The company is providing a contracted service in our public education system which is, itself, compulsory and, for the time being at least, democratically controlled.

Once they are done writing the exams, why isn’t Pearson required to turn the entire kit and kaboodle over to the state and thus to the voters and tax payers who provide the vast majority of decision making and funding to public education?

I am unaware of a construction company that, after delivering a highway project, reserves lanes for its own use or to pull up and recycle in other projects. Generally speaking, government buildings do not have entire floors blocked off for use of the contractors who built them. When Northrop Grumman delivered the USS Ronald Reagan to the Navy, they did not block off sections of the ship that the Navy cannot access. If such companies create or develop a process of construction or tool for use in construction, they can protect that via patents, but once the contracted item is finished, we generally understand it as belonging to the public who paid for it.

But when it comes to items that are not physical in nature, we accept an arrangement where the public foots enormous costs to only lease the product in question. Think of electronic voting machines. I can think of few things as important as protecting public confidence in the integrity of their vote, but companies are not required to make the code for voting machines open source and the public depends upon leaks to inform us of potential security holes in the devices. Similarly, Pearson is providing a mandated service for our compulsory public education system, and the results of that service will have actual consequences not just for the individual teachers and students involved, but also for the entire system. Confidence in what they are providing and informed decision making about whether or not what they are providing is desirable requires open and informed discussion and debate — such discussion and debate is impossible while Pearson’s intellectual property is valued more highly than the public purposes it allegedly serves.

In a small way, you cannot even blame Pearson. They made contracts with states that allowed them to behave this way, and they are a publicly traded company with $17.75 billion in market capital. Doing everything to maximize their revenue and return to investors is what they do and not a secret. However, we elect governors who appoint leaders to state education departments; they represent us. Craven obsequiousness in making contracts worth 100s of millions of taxpayers’ dollars is unnecessary and unacceptable. It is possible, I suppose, that if our elected leaders and their appointees insisted upon reasonable contracts and the full disclosure of all test materials after the tests are over, then the cost would go up, perhaps to a level states could ill afford and leading to pulling back of the test and punish regime that is currently driving education policy and warping curriculum into test preparation.

Heavens. That would be terrible.

18 Comments

Filed under Common Core, Corruption, PARCC, Pearson, Testing

Tagged as Big Data, PARCC, Pearson, shenanigans, Social Media, testing

February 18, 2015 · 1:02 pm

Saving Mr. Data

I am beginning to think that enthusiasts of standardized testing and data in education accountability are feeling nervous these days. Senator Lamar Alexander of Tennessee is the new chair of the Senate committee on Health, Education, Labor, and Pensions, and he has set about the long overdue process of renewing and/or revising the No Child Left Behind Act which required annual testing of all students in every state. The outcome of that process and of the House’s parallel bill which left committee already and which failed to adopt a Democratic sponsored amendment to require states to adopt “college and career ready standards” and to use standardized test results in accountability systems, will play a significant role in the current policy environment that is best summarized as “test and punish”.

However, it is not just a Republican controlled Congress that is threatening federal mandates for universal and annual standardized testing. An unusual coalition of small government conservatives and anti-testing progressives have joined with growing numbers of parents concerned with how test based accountability is consuming their children’s education. The once unthinkable is now being thought out loud and in the open — Congress could reauthorize the Elementary and Secondary Education Act without provisions requiring that all states test all students in all grades.

This would, obviously, be a blow to a cornerstone provision of NCLB that once enjoyed bipartisan support as a necessary measure to ensure that states did not try to duck being accountable for all students. It would also throw a huge monkey-wrench into favored policies of the Obama administration promoted by Secretary of Education Arne Duncan. While the Common Core State Standards might survive in some form without annual standardized testing, the testing consortia, Partnership for Assessment of Readiness for College and Careers (PARCC) and Smarter Balance Assessment Consortium (SBAC), began their work with the support of federal grants almost as soon as the standards were being adopted thanks to financial support from the Bill and Melinda Gates Foundation and federal incentives from the Race to the Top grant program. If annual testing requirements were scrapped by Congress, it is an open question how many states would keep Common Core and stay with the testing programs created for it. Another policy threatened by removing annual testing requirements is the assessment of teachers by the test scores of their students. Despite wide criticism from the research community, Secretary Duncan remains firmly committed to tying teacher evaluations to students’ annual progress on standardized examinations, and without annual examinations of all students, you cannot run their results through discredited and unstable statistical models to determine if teachers deserve their jobs.

So defenders of annual testing have work to do in public if they are going to save their baby,

One such recent effort appeared in the New York Times on February 6th. Penned by Chad Aldeman, a partner at Bellwether Education and former adviser to Secretary Duncan’s Department of Education, it is simply titled “In Defense of Annual Testing”, and it lays out what are becoming familiar efforts to shore up presumably left leaning support for keeping NCLB’s annual testing requirements. These are not arguments that should be casually dismissed, and they have the moral authority of some of the nation’s most distinguished civil rights organizations that originally signed on for the accountability measures in NCLB when it passed and who have reiterated their support.

They are, however, arguments that don’t stand up to serious scrutiny.

Aldeman opens with a brief assertion of a now familiar claim:

But annual testing has tremendous value. It lets schools follow students’ progress closely, and it allows for measurement of how much students learn and grow over time, not just where they are in a single moment.

This claim suggests that without ANNUAL standardized testing of ALL students then we will not know how INDIVIDUAL students are progressing through school. It has echoes in Secretary Duncan’s anecdote of how he tutored a great kid in his youth who had been tricked into thinking he was progressing towards college but who was barely reading at a third grade level. According to Secretary Duncan’s telling, this young man was the proverbial “child left behind” for whose education nobody had ever taken any real responsibility. Annual testing was the only effective means to catch how he was being poorly served and demand that someone do something about it, and Secretary Duncan even presented annual testing of all students as a parental tool: “Will we work together to ensure every parent’s right to know every year how much progress her child is making in school? Or is that optional?”

The problem is that this line of thinking shows a staggering lack of imagination.

As an argument, it fails to acknowledge that there are many other, and far more interesting, points of data that can be used by teachers, parents, and schools to keep far more compelling tabs on student progress throughout the year. Locally designed and implemented ongoing assessments such as portfolios and project based learning can provide teachers with ongoing and meaningful insights into how children are learning, and report cards can be reformed to provide parents and guardians with far more nuanced information.

It is even possible to use externally designed assessments in more interesting ways that help inform teachers, students, and parents in real time. Bruce Baker of Rutgers University notes that computer-based “adaptive assessments” given individually and with no stakes attached can be the basis of a system of formative assessment that gives immediate feedback to teachers about student progress that can be used to craft individual learning plans for those who struggle. In coordination with portfolios and project based learning (and with full disclosure of how they use data and substantial privacy safeguards), computer based assessments could become a valuable tool based on data. Dr. Baker further notes such data-driven and formative assessments would be vastly more useful than mass administered tests whose results are distributed months after the fact when students have already moved on to their next grade levels.

Aldeman dedicates most of his Times piece, however, on the belief that NOT testing every child in every year will allow states and localities to wiggle out from being accountable to all students all the time:

The grade-span approach would eviscerate the ability to look at particular groups of students within schools. Instead of having multiple grades over which schools could compile results, each school would be held responsible only for the performance of students in a single grade. Not only would this lower the quality of the data, but it would also raise the stakes of the tests: If you think the stakes are too high now, imagine being a fifth grader in a school where your score determines the results of the entire school.

Worst of all, under this approach, far fewer schools would be on the hook for paying attention to historically disadvantaged groups of students. A school with 10 Hispanic students in each grade would no longer be held accountable for whether those students were making sufficient progress, because the 10 fifth graders wouldn’t be enough to count as a meaningful population size.

Let me state that as a liberal with an eye for history, this argument is certainly intriguing. We are a nation that only 12 years before my birth required the National Guard to let nine African American students attend a high school ordered to desegregate. In 1969, the year I was born, the Supreme Court issued a ruling calling for Southern states to cease delays in desegregating their schools — a full 15 years after Brown v. Board found such arrangements unconstitutional. Federal legislation and federal court cases were also instrumental in holding states and municipalities responsible for ending gender discrimination, for providing students with disabilities access to an education, and for providing support for students learning English. It is no doubt this record of positive intervention at the federal level and of state delays in implementing equality of opportunity that motivated civil rights groups to endorse annual testing in NCLB and to stand with it today.

That does not change that such testing is unnecessary, is unacceptably disruptive to learning, and is narrowing curricula nationwide.

Mr. Aldeman is suggesting that eliminating annual testing will mean huge swaths of children will be hidden from the test and that the test stakes will be raised enormously with only exam being used. The stakes argument hinges on a mistaken impression of what the exams say and what should be done with the data they produce. For the stakes on gradespan testing to be even higher than they are today, one has to assume that such testing is used not only to monitor the education system but also to actively punish schools with low test results. While few would argue that schools with poor results should be permitted to languish, the kinds of punitive measures embodied in NCLB are not a necessary result of monitoring student test scores. Under the leadership of Superintendent Tony Alvarado, New York City’s Community District 2 implemented a complex and interconnect culture of reform that included standards and assessments. However, data from the assessments were used to monitor how schools in the district were doing and to allocate resources for improvement and innovation where they were most needed and with the constant goal of instructional improvement. Again, Dr. Baker of Rutgers makes a salient observation:

Here’s the really important part, which also relates to my thermometer example above. The testing measures themselves ARE NOT THE ACTIONABLE INFORMATION. Testing provides information on symptoms, not causes or underlying processes. It is pure folly to look at low test scores for a given institution, and follow up with an action plan to “improve test scores,” or close the school if/when test scores don’t improve, without ever taking stock of the potential causes behind the low test scores. TEST SCORES ARE SYMPTOMS, NOT CAUSES, NOT ACTIONABLE IN AND OF THEMSELVES.

Recognition of that fact and crafting policy responses to low test scores with that in mind would necessarily lower the stakes on the tests themselves.

Further, while there might be some argument for an annual test that could contribute to closer monitoring of those symptoms, there is no argument that convincingly says that such tests must be given to every student in every grade in order to get a good picture of how schools and school systems serve historically disadvantaged children. First, a low stakes system of formative assessments, both qualitative and quantitative, could apply to all children and would conform to accommodations for children with special needs. So there can readily be ways for teachers, schools, and parents to know how ALL students are doing during the course of the year.

Once we’ve set aside the issue of having a meaningful, formative assessment system for all students that can actually assist teachers, there’s no truly compelling argument against properly devised sampling of students for standardized testing. Implemented correctly, sampling would not leave substantial numbers of children invisible as Mr. Aldeman fears, and we would stop spending inordinate time trying to ferret out distinctions in performance within schools when, as Dr. Baker once again notes, the greatest and most consequential differences in test measured achievement exist between schools and districts, not within them. Insisting upon keeping annual testing of every student in every grade keeps an unnecessarily disruptive system in place as part of an accountability system that, in fifteen years, has not yielded sufficient results to justify the sacrifices in teacher autonomy over instruction and the sacrifices in non-tested subjects being shunted aside in favor of test preparation. In fact, the only people to “benefit” from this system are private test designers like Pearson, who are being handed not just lucrative contracts but also terabytes of data to mine for new products, and advocates of firing as many teachers as possible based upon student test scores.

This is especially frustrating to me because data, when used with a clear understanding of what it can and what it cannot do, is a tool, an important tool at that. It can help us develop broad pictures of what is happening in schools, and it can direct our attention to places that require more careful and nuanced study. The persistent overreach and abuse of its capabilities is building a backlash that makes it much harder to successfully advocate for more judicious and appropriate use of what can be learned. If we wish to SAVE data and its uses in school, it would be best to set aside NCLB and begin again sensibly.

I believe I see the problem, Captain. My head’s been severed.

12 Comments

Filed under Data, NCLB, politics, teaching, Testing, VAMs

Tagged as Arne Duncan, Big Data, New York Times, testing

January 13, 2015 · 9:33 pm

Liberal Apostasy: Is It Time To Downgrade the Federal Department of Education?

Washington has seen recent jockeying for positions on the debate to repeal or revise the No Child Left Behind (NCLB) as Republican Senator and former Secretary of Education Lamar Alexander takes over the Senate Committee on Health, Education, Labor & Pensions. Senator Alexander has signaled that he intends to engage in significant overhauls of the 2001 law which updated the Elementary and Secondary Education Act of 1965 and which has been due for re-authorization since 2007. He will likely find atypical allies from traditionally Democratic leaning teacher unions who will join more conservative advocates in calling for a decrease in standardized testing and walking back Obama Administration initiatives that have pushed states to evaluate individual teachers and make tenure and dismissal processes tied to such scores.

Current Secretary of Education Arne Duncan laid out some of his priorities for an overhaul of the legislation in a speech on January 12th, and he was joined by a group statement from a number of venerable civil rights organizations that, among their other priorities for federal education law, called on lawmakers to preserve the annual standardized testing requirements from NCLB. Given the historic problems in many state with both resistance to integration and with neglecting urban and rural student populations which led to more vigorous federal education laws in the first place, it is not surprising that such organizations would want a new education law to retain tight oversight provisions for the states. Secretary Duncan, in his planned remarks, emphasized accountability:

I believe parents, and teachers, and students have both the right and the absolute need to know how much progress all students are making each year towards college- and career-readiness. The reality of unexpected, crushing disappointments, about the actual lack of college preparedness cannot continue to happen to hard-working 16- and 17-year olds – it is not fair to them, and it is simply too late. Those days must be over.

That means that all students need to take annual, statewide assessments that are aligned to their teacher’s classroom instruction in reading and math in grades 3 through 8, and once in high school.

As he continued, he framed the need for accountability in civil rights language:

Will we work together to ensure that every public school makes a real priority of the educational progress of minority students, those living in poverty – be there rural, urban, or someplace else — those with disabilities, those learning English, or other groups that have struggled in school in the past? Should unacceptable achievement gaps require action? Or is that simply optional?

Secretary Duncan’s questions were poignant, and the moral authority of the NAACP and the ACLU should have reminded listeners that we have historically done a poor job improving educational opportunity for students minority students and students in poverty. However, the insistence that annual, standardized test-based accountability is the only solution to making certain that we are accountable to all of our students is deeply flawed. It is flawed because the testing regimen that Secretary Duncan supports has already narrowed the curriculum received by huge swaths of our student population, even though Secretary Duncan declared that non-tested subjects like science and the arts “are essentials, not luxuries.” It is flawed because even though Secretary Duncan stated that teachers and principals deserve support, the national reality is that most states are still spending less on education today than they did before the Great Recession even as the federal government has pushed those states to demand much more from those same teachers and principals. It is flawed because while Secretary Duncan said he believes “teachers deserve fair, genuinely helpful systems for evaluation and professional growth that identify excellence and take into account student learning growth,” his favored metric is the value-added model (VAM) of teacher performance, and the research simply does not support using VAMs as either “fair” or “genuinely helpful” and they certainly cannot “identify excellence” with reliability.

Secretary Duncan once famously said that “We should be able to look every second grader in the eye and say, ‘You’re on track, you’re going to be able to go to a good college, or you’re not,’” so his faith in power of standardized testing data is long lasting and probably sincere, but it has led his department, under the guise of relieving states from the most punishing aspects of NCLB, to push states in educational directions that are legitimately damaging to the public’s trust in education and which incentivize schools and teachers to further narrow their curriculum in search of higher test performance.

Which leads me to a question: Is it time to downgrade the federal department of education?

This is not an idle question because while I believe that the federal legislation from the 1960s and 1970s was a necessary beginning to address systemic inequalities in educational opportunity for the poor, for minorities, for women, and for people with disabilities, the Cabinet level role of the Department of Education has become highly problematic in today’s hyper-focus on standardized testing. The federal DOE was actually created by Congress in 1979 in order to strengthen the federal commitment to public education and to increase coordination and accountability for the various federal laws that have direct impact on schools. The department was immediately under fire from conservative activists interested in a smaller federal government, and President Reagan, riding the conservative wave that put him in office, pledged to abolish the fledgling department. This did not happen, and by the 1990s, the department was secure under the successive Presidencies of George H.W. Bush and Bill Clinton. With the bipartisan passage of No Child Left Behind, the department was firmly entrenched in pushing states to hold schools accountable to student achievement on standardized test scores, and while the Obama administration Race to the Top grants allowed states to apply for waivers on NCLB requirements that all children read and do math “at grade level,” states had to agree to adopt common standards, expand charter schools, and use standardized test scores to evaluate teachers. Secretary Duncan made it clear that he would hold states receiving waivers to those agreements when in April of last year, he stripped Washington state’s NCLB waivers for not meeting the federal department’s “requirements for reform.”

The federal government provides roughly 12% of the annual national spending of $550 billion on elementary and secondary education. While this does not represent a sum that comes close to helping states meet federal requirements, it has proven enough for the federal government to use the Department of Education to push policies like test-based accountability and rapid expansion of charter schools upon states and locales that might seek other ways to improve their schools given more flexibility. Title 1 funds, for example, reach 56,000 schools serving 21 million children, but since NCLB those funds have been tied towards demonstration of student annual progress via standardized testing and during the Obama administration states were required to use standardized tests to evaluate teachers if the received waivers from other NCLB provisions. The federal government can use its funding to enforce specific policy priorities on the states, but it rarely funds those priorities enough to help school districts implement them effectively. For example, for 40 years the federal government has failed to fully fund the Individuals with Disabilities in Education Act (IDEA), covering roughly 17% of the cost of the legislation even though it has long promised states to cover a full 40%.

Questioning the Cabinet level role of the DOE does not mean abandoning the landmark legislation in education that proceeded the department’s formation, and it is important to recognize the significant and needed impact of that legislation. Congress passed the Education for All Handicapped Children in 1975. In the 1969-1970 school year, a total of 2,677,000 children representing 5.9% of all children in public school received special education services, but none of them were identified with specific learning disabilities. In the 1979-1980 school year, 4,005,000 children representing 8.9% of all students received special education services, including almost 1.3 million with specific learning disabilities that were being accommodated. Title IX was passed in 1972 when 386,683 women received bachelors degrees, representing 46% of degrees conferred. By 1979-1980 school year, that percentage had risen to 49%. In 1960, five years before the Elementary and Secondary Education Act was passed, the median years of school attended by an African American male was 7.9, and by 1980, that had increased to 11.9 years — more important, the gap in the median number of years in school between white and black males closed from 20 percentage points.

All of these gains occurred in the wake of landmark federal legislation, but before the Department of Education was created in its current form.

What makes the current federal DOE so problematic in 2015 is not the role that it was created to play, but the fact that most initiatives out of it since Congress passed No Child Left Behind resemble a classic case of regulatory capture by the for profit charter school sector, the testing industry, and the data mining industries. Unlike other cases of regulatory capture, there appears to be no prospect for partisan realignment as administrations change and monied interests involved in the Executive branch shift — successive two-term Republican and Democratic administrations have charged down the current path, prioritizing testing with increasingly higher stakes.

While eliminating the Cabinet level DOE would impede some of these forces, I am also mindful of important considerations. First, the civil rights organizations that have signed on supporting Secretary Duncan’s priorities are not wrong in their concern that states have historically neglected and have even actively discriminated against certain populations, and that states and localities must be held accountable for providing equal access and equal opportunities for all of their students. While I disagree that yearly high stakes examinations are the way to ensure that, nobody can reasonably look at our history and dismiss the issue. Second, demoting the federal DOE might complicate the plans of the interests who have worked to monetize public education, but it is not as if they are absent from state level government. When it comes to adding requirements to teachers while cutting funding and when it comes to turning public schools over to charter corporations, there is little daylight between Democratic star politicians like New York Governor Andrew Cuomo, Chicago Mayor Rahm Emmanuel, and former Newark Mayor and now United States Senator Cory Booker and Republican counterparts such as Scott Walker of Wisconsin and Chris Christie of New Jersey.

Finally, as Arthur Camins notes here, our problem of the past 15 years can be described as the federal DOE “reaching for the wrong things:”

The problem over the last several decades of education policy is not overreach. It is that the federal government has been reaching for the wrong things in the wrong places with the wrong policy levers. For example, the nation has largely abandoned efforts to end segregation, arguably a prime driver of education inequity. The large-scale, community-building infrastructure and WPA and CCC employment efforts of the Great Depression have given way to the limited escape from poverty marketing pitch of education policy following the Great Recession. Whereas the 1960s War on Poverty targeted community resource issues, current education efforts target the behavior of individual teachers and pits parents against one in other in competition for admission to selected schools.

Professor Camins’ points are well-taken, but advocates for returning public education to the public’s care need ideas for addressing how education policy has been captured in both Washington D.C. and in state capitols. Consider the case of Andrew Cuomo who raised over $40 million between his inauguration in 2010 and reelection in 2014 — more than half of which came from just 341 donors, donors who expect influence upon the governor commensurate with their investment. In essence, this is a question of rooting up corruption and the circumvention of democratic processes, but as Fordham Law Professor Zephyr Teacher demonstrates, there are no easy answers.

But we must seek answers, even difficult ones. What has happened at the federal DOE is dangerous for quality and equitable public education, but it is also a symptom of a problem endemic in our politics. Mere handfuls of extremely wealthy people can override the wishes of millions of voters and circumvent public debate on crucial issues.

We cannot afford it any longer.

4 Comments

Filed under Chris Christie, Corruption, Cory Booker, Funding, Gates Foundation, politics, Social Justice

Tagged as Andrew Cuomo, Arne Duncan, Big Data, Bill Gates, Corruption, testing

December 5, 2014 · 4:23 pm

Bride of VAMenstein: No Bad Idea Gets Left Behind

When I was much younger, my grandfather, a carpenter and engineer, had an expression he was fond of saying whenever we drove through a particularly poorly designed intersection or highway interchange. He’d grunt in disgust and comment, “Whoever built this should do the world a favor. Design ONE more and then drop dead.”

There are times when I’d like the economists who keep insisting they can design value added models of teacher effectiveness to consider following the same advice.

On November 25th, the U.S. Department of Education released newly proposed regulations for teacher preparation in the over 1200 programs that exist across the country. The press release stated:

“It has long been clear that as a nation, we could do a far better job of preparing teachers for the classroom. It’s not just something that studies show – I hear it in my conversations with teachers, principals and parents,” U.S. Education Secretary Arne Duncan said. “New teachers want to do a great job for their kids, but often, they struggle at the beginning of their careers and have to figure out too much for themselves. Teachers deserve better, and our students do too. This proposal, along with our other key initiatives in supporting flexibility, equity and leadership, will help get us closer to President Obama’s goal of putting a great teacher in every classroom, and especially in our high-need schools.”

This is not a new subject for research and policy speculation. In 1984, Judith Lanier of Michigan State University contributed a comprehensive chapter on teacher education for the 3rd Handbook of Research on Teaching. Dr. Lanier concluded that while many spoke of the importance of teacher preparation, there were no entities willing to take robust authority for making sure its many parts worked, and that its quality remained highly spotty and often quite poor. Since then, there have been numerous proposals to change and improve teacher preparation from the Holmes Group Reports, to the Carnegie report on teacher preparation, to John Goodlad’s proposals for preparing teachers, to the original report of the National Commission on Teaching and America’s Future. In the 30 years since Dr. Lanier wrote her chapter, there have been numerous proposals, programs, and practices that have worked upon teacher preparation in the United States.

Now it is the turn of the Data Junkies.

The DOE announcement says states will be required to report on the performance of teacher preparation programs based upon the following:

Employment outcomes: New teacher placement and three-year retention rates in high-need schools and in all schools.
New teacher and employer feedback: Surveys on the effectiveness of preparation.
Student learning outcomes: Impact of new teachers as measured by student growth, teacher evaluation, or both.
Assurance of specialized accreditation or evidence that a program produces high-quality candidates.

Some of this is benign, some of it is deceptive, and some of it is rank foolishness. The fact that Secretary Duncan’s statement specifically cited Relay “Graduate School of Education” as an example of an innovation in teacher preparation to be held up does not lead me to a great deal of confidence. Relay, for those who do not know, is a teacher training “graduate school” that has no actual professors of education and is not attached to an institution of higher learning. Rather, it is an alternative program housed in North Star Academy Charter School in Newark, NJ using its own teachers to train new hires in the methods of teaching used in North Star and allowing them both to be credentialed and to “earn” graduate degrees. Relay and its supporters defend this because the charter school has externally impressive scores on standardized tests, but those scores come, as Dr. Bruce Baker of Rutgers University demonstrates, at the expense of more than half of the students who enroll at North Star – because they never make it to graduation. North Star enrolls over 14% fewer students on free lunch than Newark Public Schools in general, less than half as many students with disabilities, and the students with disabilities at North Star are vastly more likely to be mild or low cost to the school, including no students with autism, no emotionally disturbed students, no intellectually disabled students, and no students with multiple disabilities. Between 5th grade and 12th grade, half of students attending North Star leave the school, and 60% of African American boys leave.

Just to be clear: The Secretary of Education for the United States of America announced new teacher preparation regulations by praising the “innovation” of a “Graduate School of Education” that does no serious graduate study, has no qualified educational researchers, and that prepares its graduates to teach the methods espoused by a charter school where an African American male student only has a 40% chance of reaching his senior year of high school.

Components of these regulations are puzzling. The DOE wants states to keep track of teacher retention rates, presumably because of the long known problem of early career teachers leaving both assignments or the profession in high numbers. Such a requirement raises staggering logistical challenges, as states do not readily have ways to track the careers of teachers certified in their states who teach in other states, teachers who switch teaching in a public school for a position in a private or parochial school, and teachers who take up full time graduate studies — all of which are very different than leaving because of feeling overwhelmed and under-prepared.

More troubling, such data would be largely indicative of the professional cultures and environments of the schools in which teacher preparation graduates teach. While teacher education has worked in the past three decades to provide prospective teachers with quality experiences to reduce the long recognized “reality shock” experienced by novice teachers, such work is frequently difficult, time and resource intensive, and requires significant rethinking of the relationship between universities and schools where prospective teachers are prepared. However, significant research also exists that demonstrates that teacher turnover is deeply tied to school factors in initial job placements that are entirely outside university control. In no place in these regulations on preparing teachers do I see anything related to how states and communities support the local schools to promote collaborative environments that support early career educators. What I do see is a potentially perverse incentive for teacher preparation programs to steer their graduates as far away from struggling schools as possible.

Worse than this provision by far, however, is the proposal to take the already invalid concept of Value Added Measures (VAM) of teacher performance and to use the VAMs of teachers to evaluate their teacher preparation programs. A VAM is a statistical model based on student standardized test performance that takes a student’s previous year’s test scores, claims to predict how that student will perform given a year of effective teaching, and then generates the teacher’s “value added” based on how well students do based on those predictions. The American Statistical Association issued a clearly worded statement this year detailing the problems with VAMs, citing both the lack of tests that are valid for the purpose and the very limited impact that teachers have on student variability on standardized test performance. Research generally agrees that teachers are a very important if not the most important in school factor for students, but research also agrees whatever teachers’ impact is, standardized tests are an exceedingly poor measure of it, accounting for only 1-14% of student variability on the tests.

Despite these inherent flaws, VAMs remain highly popular with the federal DOE which has been influenced by the Gates Foundation funded “Measures of Effective Teaching” study which claims that VAMs can be used as a component of teacher evaluation. Jesse Rothstein of University of California at Berkeley, however, notes that the data used to justify that claim is strikingly weak, and that teachers who are effective by some measures show up as ineffective by others and vice versa. Dr. Baker of Rutgers illustrates here that teachers whose students score high in one year (called “Irreplaceables” by Michelle Rhee’s New Teacher Project “thought leaders”) are not all “irreplaceable” in subsequent years (and in fact most drift all over the map), making it absolutely necessary to consider that factors outside of the classroom play significant roles in student test performance. VAMs also potentially damage teachers whose students, far from being low performers, work at an accelerated curriculum that is several years past the material directly tested on the exams used to generate VAMs. The New York Times reported in 2011 of the tribulations of Ms. Stacy Isaacson, who was universally regarded as an outstanding mathematics teacher whose students got excellent scores on state examinations and over two dozen of whom went on to New York City’s highly selective high schools, got ranked in the 7th percentile of teachers in the city by the VAM formula used that year:

Ms. Isaacson’s low percentile could not be explained to her by anyone in her administration, and the fault lay at the opaque statistical formula used to rank her based on students’ tests. Given the inherent flaws with VAMs, my explanation is as follows. In the New York City Value Added Model, what is circled in this picture is a real number:

Everything circled here is the result of misapplying statistical tools used to model entire national economies to a single teacher’s classrooms:

Anyone who knows children and their development should be troubled by VAMs because in order to believe that they work with such small samples as a single teacher’s classroom, we have to believe that the VAM can adequately account for every factor outside of a teacher’s instruction that can impact how students do on a test. Did Johnny get an Individualized Education Plan this year that finally provides support for his dyslexia? Are Johnny’s parents reconciling after a period of separation and his home life is stabilizing? Has Johnny’s cognitive development reached a point where he is ready for more complex learning and will outpace previous years of instruction because children do not actually develop in straight lines? All of these are factors that can boost a teacher’s value added score without the teacher actually having done anything especially different for Johnny. There are as many factors not directly related to a single teacher that can negatively impact a value added score.

So let’s review: Research supporting VAMs ignores its own contradictory research. No current standardized test is sufficiently well designed for the purpose of generating VAMs. VAMs measure teacher input on student variability in standardized test scores which is as low as 1% and only as high as 14%. Teachers whose students score in very high percentiles in one year can have students who score far differently in subsequent years. Teachers who are effective by every other measure possible can be placed in the very bottom tier of teachers using VAMs. This is not the kind of stuff that inspires much confidence, but the federal DOE is going to push ahead anyway.

Have really terrible measures of teacher effectiveness on your hands? Never mind! If you are Secretary Duncan, you have Bill Gates backed research and advocacy, and seriously flawed “research” from Michelle Rhee’s pet group to tell you otherwise. Full speed ahead.

Of course, if you are going to blatantly ignore what a growing body of genuine research tells you about your favored reforms, it stands to reason that you will double down on them and try to push them even further into the system by measuring teacher preparation programs by the VAMs their graduates generate. There is a lesson here that Secretary Duncan, Bill Gates, Michelle Rhee, and an entire platoon of corporate reformers seem incapable of learning, and it has to do with learning humility when beloved projects turn out to be far more complicated and fraught with failure than anticipated.

In the 1935 sequel “Bride of Frankenstein,” the badly wounded but recovering Henry Frankenstein initially renounces his creation but is forced by his former mentor, Dr. Septimus Pretorius, to assist a project creating a “bride” for the monster. The monster is excited by the chance to have a companion like himself, but is quickly devastated by her immediate, terrified, rejection of him and destroys himself, Henry’s laboratory, Dr. Pretorius, and the bride, proving again that the power of life and death is not a toy to be trifled with.

I could save Secretary Duncan quite a lot of trouble if he’d just ask.

Well, that didn’t go as planned, did it?

6 Comments

Filed under Gates Foundation, schools, teacher learning, Testing, VAMs

Tagged as Arne Duncan, Big Data, Bill Gates, Michelle Rhee, testing

June 10, 2014 · 9:54 pm

Gates’ Money and Privacy Activism — Opposition to Ed. Reform Hits the Mainstream

A few stories caught a lot of eyes over the weekend. None of that is good news for education reformers who have banked on stealth and little reporting.

The first is a major article and interview regarding the role Bill Gates’ money has played in the development of, promotion of and adoption of the Common Core State Standards in the Washington Post. The story is not entirely complete. For starters, it fails to disclose that David Coleman already had funding from the Gates Foundation for his Student Achievement Partners, so a meeting with Gates in 2008 is not their first intersection. Also, it does not explore the heavy hand that Gates has also had in the push for more high stakes testing to evaluate teachers via his funding of the highly flawed Measures of Effective Teaching study, nor does it examine the role that Gates has played in enabling technology entrepreneurs to mine the data generated from those tests without parental consent.

Regardless, the article is both informative and important for several reasons. First, it is one of the first times anyone in a major news outlet has provided a portrait of the diverse opposition to current reform efforts in education that doesn’t make it sound like mostly the work of Alex Jones style cranks. The article even quotes academics who question whether the standards are based on sound research on how children learn or even if there is a connection between quality standards and learning. Second, this is an article in a major outlet that does not equivocate in the slightest about how much influence one very rich man has had in trying to control the entire course of American public education. While it does not editorialize on the question, it is hard to read how many federal, state, nonprofit, academic and corporate entities were lobbied by, influenced by or funded by Gates and not wonder what role democracy has anymore when it comes to our public schools.

Finally, Gates himself comes across as defensive and dismissive:

Gates is disdainful of the rhetoric from opponents. He sees himself as a technocrat trying to foster solutions to a profound social problem — gaping inequalities in U.S. public education — by investing in promising new ideas.

This would be more convincing if Gates displayed the slightest interest in testing new ideas before unveiling them in 45 states at once before parents and teachers have a chance to understand them, indeed, before anyone has a chance to understand if they are a net positive or not. From Gates’ point of view and experience, this must make sense. He has compared common standards to standardized electrical outlets and computer code as a means of allowing innovation, and certainly getting DOS on most desktop computers in the world led to a lot of software developers having a common platform. But education is not consumer electronics, and bypassing the entirety of stakeholders who value public education for a variety of reasons was going to lead to push back, and even today, Bill Gates does not demonstrate awareness of that.

The second article appeared in Politico and was dedicated to parent activists working to protect their children from data mining operations tied to public education. This represents another public airing of activities whose proponents would prefer to avoid being seen in the open. The report quotes New York’s Leonie Haimson of Class Size Matters about how parents have reacted when informed about the plans of technology firms to use pretty much every bit of data they can get their hands on, and it quotes worried data entrepreneurs coming to grips with parental opposition:

Many said they had always assumed parents would support their vision: to mine vast quantities of data for insights into what’s working, and what’s not, for individual students and for the education system as a whole.

“People took for granted that parents would understand [the benefits], that it was self-evident,” said Michael Horn, a co-founder of the Clayton Christensen Institute, an education think tank.

Instead, legitimate questions about data security have mixed with alarmist rhetoric in a combustible brew that’s “spreading like wildfire” on social media, said Aimee Rogstad Guidera, executive director of the Data Quality Campaign, a nonprofit advocacy group for data-driven education.

That fear, Guidera said, “leads to people saying, ‘Shut it down. No more.’”

Guidera hopes to counter the protests by circulating videos and graphics emphasizing the value of data. But she acknowledges the outrage will be hard to rein in.

Could the parent lobby scuttle a data revolution that’s been championed by the White House, pushed by billionaire philanthropists and embraced by reformers of both parties as the best hope to improve public education? “I do have that concern,” Guidera said. “Absolutely.”

The article doesn’t go into detail about just how much money is thought to be at stake and takes the data mining firms at their word that they only want to help, but I came away from reading it with one resounding message: this damage is entirely self inflicted, but the data miners see the parent activists as the problem. They did not want to do the hard marketing work of convincing people that they were doing something valuable, and they did not anticipate that parents might see the data generated by their children’s public educations as something they’d want to protect rather than just shovel over for free. It is hard to sympathize here, especially when they have avoided openness from the beginning.

Which leads to the third article from today: a call from the Gates Foundation for a two year “moratorium” on high stakes decisions based upon Common Core aligned testing. This is the first official wavering from the Gates camp since the standards and testing drive began in earnest, and it is highly significant as an indication of concern that the whole enterprise is in trouble. It may also be a miscalculation — yes, teacher opposition to using value added measures of their effectiveness based on standardized tests is strong, and yes, teachers have barely had time to adjust to the new standards. But two years will not fix the flaws in VAMs, and it will not assuage parental concerns about the role of testing and data mining. It will potentially take a chunk out of testing companies and data mining companies who were making business plans based upon all Common Core states embarking on wide scale testing next year, and I find it interesting that the Gates Foundation is willing to have them cool their heels while the standards’ supporters try to do something they have avoided all along: talk in public.

Data Mining in Education — Incredible Potential, Incredible Arrogance

When inBloom fell apart, I expected that the argument would continue. Politico reported yesterday on the scope of data mining in education — and the stunning arrogance of some technology entrepreneurs. According to the report, some analysts expect that data mining in education could lead to 300 billion a year in economic growth, and I find that credible. I already use “small data” in my classroom via social media feeds that let me know what my students are thinking about as they read and to craft my planning around that input. The potential of “big data” to craft learning tools for use by teachers and students is likely incredible.

And it will be thoroughly undermined by the secrecy and arrogance of its proponents.

Jose Ferreira, the CEO of Knewton, a leading “adaptive education” firm, is portrayed as frustrated and dismissive:

When parents protest that they don’t want their children data-mined, Ferreira wishes he could ask them why: Is it simply that they don’t want a for-profit company to map their kids’ minds? If not, why not? “They’d rather the NSA have it?” he asked. “What, you trust the government?”

Ferreira said he often hears parents angrily declaring that their children cannot be reduced to data points. “That’s not an argument,” Ferreira said. “I’m not calling your child a bundle of data. I’m just helping her learn.”

He goes on to say:

“It just helps children,” Ferreira said. “That’s all it does.”

But Ferreira misses the point by acting as if all he is facing is knee-jerk and reactionary parents misconstruing his work. Data analytics may be powerful, but the firms involved in it have not be upfront or informative to either school districts or parents and guardians of school aged children. While it is true that businesses have always made money via our public schools, I can walk into my daughter’s first grade classroom and I see the publishers of her textbooks and other classroom materials. The money spent is part of a public budget. I can engage in her curriculum, and I can consult with her teachers about her strengths and struggles. Even more to the point, I get to know the teachers who know her school work the best and my voice and perspectives are heard.

But when the federal government alters education privacy law to suit the interests of data mining and when the firms themselves are unclear even to school districts how data is safeguarded and used, parents have legitimate concerns that cannot be dismissed. With the rush to implement Common Core and the accompanying testing poised to create vast amounts of data from 10s of millions of students, it is even more important for technology entrepreneurs to spend time actually engaging parents and teachers about the potential of these tools and the safeguards on student privacy they intend to use. If they are unwilling to do that, and Mr. Ferreira’s statements are not encouraging on that front, then the pushback is both inevitable and deserved.

They are not selling iPhones to adult consumers. If I purchase a smart phone that has the potential to generate data about myself, I am capable of wrestling with the implications of that both legally and morally. My two public school children, however, are legally obligated to be in school, and their public education is matter of concern for both our family and society as a whole. inBloom’s major error was only engaging state level DOEs when setting up a system that would have effected 10s of millions of children. It looks like the big data firms that are eager to get in on the 8 billion dollar market in software and digital materials are not learning from that.

These are my children. They are my responsibility. You need to sell these products to me. You need to give me an opportunity to opt my children out of your system.

inBloom Shuts Down. This is My Told You So Face

inBloom lost New York State and is shutting down.

The technology firm, funded largely by the Gates Foundation, was poised to take advantage of Obama administration changes to the Federal Education Rights and Privacy Act (FERPA) to create a mass data “cloud” by becoming a multistate repository of student data. In the past states might contract a database firm to create an in house system for their exclusive use. What makes inBloom different was the scale of the data collection, the multistate nature of the project, and that instead of simply housing data for the state and districts to analyze, inBloom intended to let vendors use that data to make education products.

Activists and parents protested on two grounds: first, they were concerned about just how secure data that is supposed to be protected by federal law would be and second, the nature of the deals made with inBloom that allowed students’ educations to become ongoing revenue stream and that were made with no public input whatsoever.

Technology firms should learn the right lesson from this. Individual learning products tailored by big data analysis are coming to public schools, and they have potential. But they should not come via back room deals that adjust federal privacy law and make contracts that fundamentally change the way states store and safeguard children’s private records without public input.

If technology firms repeat inBloom’s mistake and act as if they only have to market to 50 state education departments instead of to the parents and guardians of the 74 million children in this country, this fight will only happen again.

1 Comment

Filed under Activism, Gates Foundation, politics, Privacy, schools

Tagged as Big Data, Privacy

Tag Archives: Big Data

When is a Pledge to Decrease Testing Not a Pledge to Decrease Testing?

Dear Senator Gillibrand: Public Schools Need Advocates, Not More Punishments

Saving Mr. Data

Bride of VAMenstein: No Bad Idea Gets Left Behind

Gates’ Money and Privacy Activism — Opposition to Ed. Reform Hits the Mainstream

Data Mining in Education — Incredible Potential, Incredible Arrogance

inBloom Shuts Down. This is My Told You So Face

Subscribe to Blog via Email

Recent Posts

Archives

Categories

Blogs I Follow

Meta

Recent Posts

Archives

Categories

Meta