Audrey Amrein-Beardsley of Vamboozled just added the official statement from the American Statistical Association to her list of top articles on Value Added Models (VAMs) for education assessment. The full statement can be found here, and it is a doozy. Well, I say “doozy” because I know quite a few people who specialize in quantitative analysis and quotes like this count as pretty harsh condemnation:
VAMs are generally based on standardized test scores, and do not directly measure potential teacher contributions toward other student outcomes.
That means VAMs are limited.
VAMs typically measure correlation, not causation: Effects – positive or negative – attributed to a teacher may actually be caused by other factors that are not captured inthe model.
That means that VAMs used as the main means of teacher assessment are missing things…potentially very important things.
Under some conditions, VAM scores and rankings can change substantially when a different model or test is used, and a thorough analysis should be undertaken to evaluate the sensitivity of estimates to different models.
That means some VAMs are not exactly reliable. Page four lays it out in pretty clear language to me:
The measure of student achievement is typically a score on a standardized test, and VAMs are only as good as the data fed into them. Ideally, tests should fully measure student achievement with respect to the curriculum objectives and content standards adopted by the state, in both breadth and depth. In practice, no test meets this stringent standard, and it needs to be recognized that, at best, most VAMs predict only performance on the test and not necessarily long-range learning outcomes. Other student outcomes are predicted only to the extent that they are correlated with test scores. A teacher’s efforts to encourage students’ creativity or help colleagues improve their instruction, for example, are not explicitly recognized in VAMs.
VAM scores are calculated from classroom-level heterogeneity that is not explained by the background variables in the regression model. Those classroom-level differences may be due in part to other factors that are not included in the model (for example, class size, teaching “high-need” students, or having students who receive extracurricular tutoring). The validity of the VAM scores as a measure of teacher contributions depends on how well the particular regression model adopted adjusts for other factors that might systematically affect, or bias, a teacher’s VAM score.
“In practice, no test meets this stringent standard.” Keep in mind, EVERY state that accepted “Race to the Top” grants HAD to agree to use these tests to evaluate teachers. And from page five:
When used appropriately, VAMs may provide quantitative information that is relevant for improving education processes. For example, the models can provide information on important sources of variability, and they can allow teachers and schools to see how their students have performed on the assessment instruments relative to students with similar prior test scores.
Teachers and schools can then explore targeted new teaching techniques or professional development activities, while building on their strengths.
There is a key difference between this observation from the ASA and what is happening in schools: VAMs are NOT being used to provide data “relevant for improving education processes.” They are being used to label teachers as effective or not effective. They are being used to label schools failures or successes. So when the ASA says on page seven:
Statistical science has an important role to play in raising the quality of education, through developing and refining statistical models for use in education, providing guidance on designing experiments and interpreting statistical results, and applying quality and process improvement expertise to help guide judgments in the presence of uncertainty. The ASA promotes sound use of statistical methodology for improving education.
You know that they think this is NOT what is happening.
And just remember: if you live in a “Race to the Top” state, this is happening to your children’s teachers.
Daniel Katz, Ph.D. 