Click on the pics to appreciate the full ugliness.
The problem with evidence as defined by test scores is that data can always be manipulated. In honest hands, this manipulation does not usually produce misleading results — although not all data handlers know the meaning of the numbers they crunch and some may boldly assert “facts” unsupported by their numbers — but many stakeholders in education are under pressure to produce results. It’s a short step from optimistic interpretation to deceit. When the results show annual growth of 1.05 years from a benchmark test, that may be presented as “has shown dramatic improvement until our 2nd grade is exceeding expectations and producing over a year’s academic improvement now!” Ummm… That 0.05 growth above the 1.0? That 0.05 may not be statistically meaningful. There may be no growth or slightly less than one year’s growth.
Other problems with data:
Many teachers are forced to compile, record and keep data that is never ever used. Somebody’s great idea creates days of extra work throughout a school, but then no administrator ever finds time to sit down with the resulting forms and spreadsheets to figure out what the numbers reveal.
None of the spreadsheets from my last year before retirement affected instruction. We were kept on the common lesson plan whether our students could read and understand the questions or not. I proved and proved that my students could not read the Common Core tests I was obliged to give, explained the problem with giving 7th grade tests in English to bilingual students who were reading English at a third grade level and sometimes Spanish at an even lower level. But nothing changed and the tests kept being handed to me along with threats if I resisted those useless tests and quizzes.
I have shown a few of these tests. I’ll insert one more.
The cost of data gathering goes unremarked too often, especially now that most data lives out its life electronically. Those old-fashioned dollar losses from stacks of paper and ink at least highlighted wastage sometimes, as recycling bins and waste paper baskets filled up. The paper was visible. The hours spent at computers and in subsequent meetings and trainings are harder to track. The opportunity costs are impossible to track. For example, essay tests have mostly become a thing of the past. After complying with data requirements, many teachers don’t have time to grade such tests. The shift toward multiple choice has come about in part because those tests are good standardized test practice, but also because data requirements frequently don’t leave a whole evening or day to grade students’ essays properly — or even to grade piles of essays at all.
Eduhonesty: The opportunity costs from gathering data are kneecapping education. Time is stolen from lesson preparation all up and down the line, until buying lessons from Teachers Pay Teachers becomes some teachers’ only hope, while others use required lesson plans that they know are not as good as what they might be able to prepare themselves — if given back the time stolen by Spreadsheet #42.
I am by no means against gathering and analyzing educational data. Data is required so educators can determine how well instruction is working. But data demands have been exploding in the recent past, and I wrote this post to highlight one point: Data demands have opportunity costs. The time to prepare data is taken out of lesson preparation, grading, tutoring, materials preparation, and other student-centered activities.
And to what end? Our international test scores remain fairly stagnant. In some locations, scores have been declining over time despite this full court data press. I strongly suspect that excessive demands for data not only reflect this lack of progress — THEY CREATE A PORTION OF THE LOST LEARNING WE ARE BUSY DOCUMENTING.
Specials teachers complain that they are forced to create the same data as all the other teachers in their school, sometimes multiple, huge binders full of data, but then no one gives that data more than a cursory look. It’s not English or mathematics and it’s not on the state standardized test, so it’s considered unimportant or even essentially irrelevant in the larger scheme of things — but specials teachers are still expected to compile, record, and preserve the numbers. Sometimes they even have to find ways to quantify instructional results that are not fundamentally quantifiable, such as “artistic progress.”
But all teachers are usually expected to create and save that data. The Principal has to be able to produce data if the Assistant Superintendent asks for the data. What if the Assistant Superintendent asked for the data and it wasn’t there!?! But that does not mean anyone will ever review the “useless” specials data they may or may not demand to see. Although, unfortunately, if administrators ever decide the budget requires getting rid of a random music or art teacher, they may be able to use a teacher’s data for this purpose. I am reminded of a favorite saying by Ronald Coase: If you torture the data enough, it will always confess.