“On the dangers of being data-driven: Imagine driving from A to B ignoring the road, the weather, the traffic around you… only staring at the gauges on the dashboard.”
“Make your instruction data-driven.”
If you’re a public school teacher, you’ve probably heard this a hundred times.
In the last week.
Principals and administrators use that word – “data-driven” – as if it were inscribed over the front doors of the schoolhouse in stone.
The idea goes like this: All lessons should be based on test scores.
Students take the federally mandated standardized test. Your job is to make sure they get the best possible score. Your class is nothing but a way station between standardized tests.
Pretest your students and then instruct them in such a way that when they take the test again, they’ll get the best possible score.
It’s total nonsense. And it doesn’t take much to see why.
No teacher should ever be data-driven. Every teacher should be student-driven.
You should base your instruction around what’s best for your students – what motivates them, inspires them, gets them ready and interested in learning.
To be sure, you should be data-informed – you should know what their test scores are and that should factor into your lessons in one way or another – but test scores should not be the driving force behind your instruction, especially since standardized test scores are incredibly poor indicators of student knowledge.
No one really believes that the Be All and End All of student knowledge is children’s ability to choose the “correct” answer on a multiple-choice test. No one sits back in awe at Albert Einstein’s test scores – it’s what he was able to do with the knowledge he had. Indeed, his understanding of the universe could not be adequately captured in a simple choice between four possible answers.
As I see it, there are at least six major problems with this dependence on student data at the heart of the data-driven movement.
So without further ado, here is a sextet of major flaws in the theory of data-driven instruction:
-
The Data is Unscientific
When we talk about student data, we’re talking about statistics. We’re talking about a quantity computed from a sample or a random variable.
As such, it needs to be a measure of something specific, something clearly defined and agreed upon.
For instance, you could measure the brightness of a star or its position in space.
However, when dealing with student knowledge, we leave the hard sciences and enter the realm of psychology. The focus of study is not and cannot be as clearly defined. What, after all, are we measuring when we give a standardized test? What are the units we’re using to measure it?
We find ourselves in the same sticky situation as those trying to measure intelligence. What is this thing we’re trying to quantify and how exactly do we go about quantifying it?
The result is intensely subjective. Sure we throw numbers up there to represent our assumptions, but – make no mistake – these are not the same numbers that measure distances on the globe or the density of an atomic nucleus.
These are approximations made up by human beings to justify deeply subjective assumptions about human nature.
It looks like statistics. It looks like math. But it is neither of these things.
We just get tricked by the numbers. We see them and mistake what we’re seeing for the hard sciences. We fall victim to the cult of numerology. That’s what data-driven instruction really is – the deepest type of mysticism passed off as science.
The idea that high stakes test scores are the best way to assess learning and that instruction should center around them is essentially a faith based initiative.
Before we can go any further, we must understand that.
-
It Has Never Been Proven Effective
Administrators and principals want teachers to base their instruction around test scores.
Has that ever been proven an effective strategy for teachers planning lessons or the allocation of resources? Can we prove a direct line from data to better instruction to better test scores?
The answer is an unequivocal NO.
In a 2007 study from Gina Schuyler Ikemoto and Julie A. Marsh published in the Yearbook for the National Society for the Study of Education, data driven instruction actually was found to have harmful effects on educator planning and, ultimately, student learning.
Researchers looked at 36 instances of data use in two districts, where 15 teachers used annual tests to target weaknesses in professional development or to schedule double periods of language arts for English language learners. The result was fewer instances of collective, sustained, and deeper inquiry by groups of teachers and administrators using multiple data sources – test scores, district surveys, and interviews – to reallocate funds for reading specialists or start an overhaul of district high schools.
Teachers found the data less useful if it was not timely – standardized test scores are usually a year old by the time they get to educators. Moreover, the data was of less value if it did not come with district support and if instructors did not already buy into its essential worth.
In short, researchers admitted they could not connect student achievement to the 36 instances of basic to complex data-driven decisions in these two districts.
But that’s just one study.
In 2009, the federal government published a report (IES Expert Panel) examining 490 studies where schools used data to make instructional decisions.
Of these studies, the report could only find 64 that used experimental or quasi-experimental designs. Of these it could find only six – yes, six – that met the Institute of Education Sciences standard for making causal claims about data-driven decisions to improve student achievement.
And when examining these six studies, the panel found “low evidence” to support data-driven instruction. They concluded that the theory that data-driven instructional decisions improve student test scores has not been proven in any way, shape or form.
-
It’s Harmful – The Stereotype Threat and Motivation
Data-driven instruction essentially involves grouping students based on their performance on standardized tests.
You put the low scorers HERE, the students on the bubble who almost reached the next level HERE, and the advanced students HERE. That way you can easily differentiate instruction and help meet their needs.
However, there is a mountain of psychological research showing that this practice is harmful to student learning. Even if you don’t put students with different test scores in different classes, simply informing them that they belong to one group or another has intense cognitive effects.
Simply being told that you are in a group with lower test scores depresses your academic outcomes. This is known as the stereotype threat.
When you focus on test scores and inform students of where they fall on the continuum down to the percentile – of how far below average they are – you can trigger this threat. Simply tracking students in this way can actually make their scores worse.
It can create negative feelings about school, threatening students’ sense of belonging, which is key to academic motivation.
But it’s not just the low scorers who are harmed. Even the so-called “advanced” students can come to depend on their privileged status. They define themselves by their achievement, collecting prizes, virtual badges and stickers. These extrinsic rewards then transform their motivation from being driven by the learning and the satisfaction of their curiosity to depending on what high achievement gets them, researchers have found.
In short, organizing all academics around tests scores is a sure way to lower them.
-
The Data Doesn’t Capture Important Factors
Data-driven instruction is only as good as the data being used. But no data system can be all inclusive.
When we put blinders on and say only these sorts of factors count, we exclude important information.
For instance, two students do the same long-term project and receive the same grade. However, one student overcame her natural tendency to procrastinate and learned more than in past projects. The other did not put forth his best effort and achieved lower than his usual.
If we only look at the data, both appear the same. However, good teachers can see the difference.
Almost every year I have a few students who are chronically tardy to class. A good teacher finds out why – if this is because they aren’t making the best use of the class interval or if they have a greater distance to travel than other students. However, if we judge solely on the data, we’re supposed to penalize students without considering mitigating factors. That’s being data-driven – a poor way to be a fair teacher.
It has been demonstrated repeatedly that student test scores are highly correlated with parental income. Students from wealthier parents score well and those from more impoverished families score badly. That does not mean one group is smarter or even more motivated than the other. Living in poverty comes with its own challenges. Students who have to take care of their siblings at home, for instance, have less time for homework than those who have nothing but free time.
A focus solely on the data ignores these factors. When we’re admonished to focus on the data, we’re actually being told to ignore the totality of our students.
-
It’s Dehumanizing
No one wants to be reduced to a number or a series of statistics.
It is extremely insulting to insist that the best way for teachers to behave is to treat their students as anything other than human beings.
They are people with unique needs, characteristics, and qualities, and should be treated accordingly.
When one of my students does an amazing job on an assignment or project, my first impulse is not to reduce what they’ve done to a letter grade or a number. I speak my approbation aloud. I write extensive comments on their papers or conference with them about what they’ve done.
Certainly, I have to assign them a grade, but that is merely one thing educators do. To reduce the relationship to that – and only that – is extremely reductive. If all you do is grade the learner, you jeopardize the learning.
Every good teacher knows the importance of relationships. Data-driven instruction asks us to ignore these lessons in favor of a mechanistic approach.
I’m sorry. My students are not widgets and I refuse to treat them as such.
I am so sick of going to conferences or faculty meetings where we focus exclusively on how to get better grades or test scores from our students. We should, instead, focus on how to see the genius that is already there! We should find ways to help students self-actualize, not turn them into what we think they should be.
At this point, someone inevitably says that life isn’t fair. Our students will have to deal with standardized tests and data-driven initiatives when they get older. We have to prepare them for it.
What baloney!
If the real world is unfair, I don’t want my students to adjust to that. I want to make it better for them.
Imagine telling a rape victim that that’s just the way the world is. Imagine telling a person brutalized by the police that the world is unfair and you just have to get used to it.
This is a complete abdication not just of our job as teachers but our position as ethical human beings.
Schools are nothing without students. We should do everything we can to meet their needs. Period.
-
It’s Contradictory – It’s Not How We Determine Value in Other Areas
Finally, there is an inherent contradiction that all instruction must be justified by data.
We don’t require this same standard for so many aspects of schooling.
Look around any school and ask yourself if everything you see is necessarily based on statistics.
Does the athletic program exist because it increases student test scores? Does each student lunch correlate with optimum grades? Do you have computers and iPads because they have a measureable impact on achievement?
Some administrators and principals DO try to justify these sorts of things by reference to test scores. But it’s a retroactive process.
They are trying to connect data with things they already do. And it’s completely bogus.
They don’t suddenly believe in football because they think it will make the team get advanced scores. They don’t abruptly support technology in the classroom because they think it will make the school achieve adequate yearly progress.
They already have good reasons to think athletics helps students learn. They’ve seen participation in sports help students remain focused and motivated – sometimes by reference to their own lives. Likewise, they’ve seen the value of technology in the classroom. They’ve seen how some students turn on like someone flipped a switch when a lesson has a technological component.
These aren’t necessarily quantifiable. They don’t count as data but they are based on evidence.
We come to education with certain beliefs already in place about what a school should do and others are formed based on the empiricism of being there, day-in, day-out. “Data” rarely comes into the decision making process as anything but a justification after the fact.
And so we can firmly put the insistence on data-driven instruction in the trash bin of bad ideas.
It is unscientific, unproven, harmful, reductive, dehumanizing and contradictory.
The next time you hear an administrator or principal pull out this chestnut, take out one of these counterarguments and roast it on an open fire.
No more data-driven instruction.
Focus instead on student-driven learning.
Don’t let them co-opt you into the cult of numerology. Remain a difference-maker. Remain a teacher.
Like this post? I’ve written a book, “Gadfly on the Wall: A Public School Teacher Speaks Out on Racism and Reform,” now available from Garn Press. Ten percent of the proceeds go to the Badass Teachers Association. Check it out!
WANT A SIGNED COPY?
Click here to order one directly from me to your door!
gadflyonthewallblog 