“On the dangers of being data-driven: Imagine driving from A to B ignoring the road, the weather, the traffic around you… only staring at the gauges on the dashboard.”
“Make your instruction data-driven.”
If you’re a public school teacher, you’ve probably heard this a hundred times.
In the last week.
Principals and administrators use that word – “data-driven” – as if it were inscribed over the front doors of the schoolhouse in stone.
The idea goes like this: All lessons should be based on test scores.
Students take the federally mandated standardized test. Your job is to make sure they get the best possible score. Your class is nothing but a way station between standardized tests.
Pretest your students and then instruct them in such a way that when they take the test again, they’ll get the best possible score.
It’s total nonsense. And it doesn’t take much to see why.
No teacher should ever be data-driven. Every teacher should be student-driven.
You should base your instruction around what’s best for your students – what motivates them, inspires them, gets them ready and interested in learning.
To be sure, you should be data-informed – you should know what their test scores are and that should factor into your lessons in one way or another – but test scores should not be the driving force behind your instruction, especially since standardized test scores are incredibly poor indicators of student knowledge.
No one really believes that the Be All and End All of student knowledge is children’s ability to choose the “correct” answer on a multiple-choice test. No one sits back in awe at Albert Einstein’s test scores – it’s what he was able to do with the knowledge he had. Indeed, his understanding of the universe could not be adequately captured in a simple choice between four possible answers.
As I see it, there are at least six major problems with this dependence on student data at the heart of the data-driven movement.
So without further ado, here is a sextet of major flaws in the theory of data-driven instruction:
The Data is Unscientific
When we talk about student data, we’re talking about statistics. We’re talking about a quantity computed from a sample or a random variable.
As such, it needs to be a measure of something specific, something clearly defined and agreed upon.
For instance, you could measure the brightness of a star or its position in space.
However, when dealing with student knowledge, we leave the hard sciences and enter the realm of psychology. The focus of study is not and cannot be as clearly defined. What, after all, are we measuring when we give a standardized test? What are the units we’re using to measure it?
We find ourselves in the same sticky situation as those trying to measure intelligence. What is this thing we’re trying to quantify and how exactly do we go about quantifying it?
The result is intensely subjective. Sure we throw numbers up there to represent our assumptions, but – make no mistake – these are not the same numbers that measure distances on the globe or the density of an atomic nucleus.
These are approximations made up by human beings to justify deeply subjective assumptions about human nature.
It looks like statistics. It looks like math. But it is neither of these things.
We just get tricked by the numbers. We see them and mistake what we’re seeing for the hard sciences. We fall victim to the cult of numerology. That’s what data-driven instruction really is – the deepest type of mysticism passed off as science.
The idea that high stakes test scores are the best way to assess learning and that instruction should center around them is essentially a faith based initiative.
Before we can go any further, we must understand that.
It Has Never Been Proven Effective
Administrators and principals want teachers to base their instruction around test scores.
Has that ever been proven an effective strategy for teachers planning lessons or the allocation of resources? Can we prove a direct line from data to better instruction to better test scores?
The answer is an unequivocal NO.
In a 2007 study from Gina Schuyler Ikemoto and Julie A. Marsh published in the Yearbook for the National Society for the Study of Education, data driven instruction actually was found to have harmful effects on educator planning and, ultimately, student learning.
Researchers looked at 36 instances of data use in two districts, where 15 teachers used annual tests to target weaknesses in professional development or to schedule double periods of language arts for English language learners. The result was fewer instances of collective, sustained, and deeper inquiry by groups of teachers and administrators using multiple data sources – test scores, district surveys, and interviews – to reallocate funds for reading specialists or start an overhaul of district high schools.
Teachers found the data less useful if it was not timely – standardized test scores are usually a year old by the time they get to educators. Moreover, the data was of less value if it did not come with district support and if instructors did not already buy into its essential worth.
In short, researchers admitted they could not connect student achievement to the 36 instances of basic to complex data-driven decisions in these two districts.
But that’s just one study.
In 2009, the federal government published a report (IES Expert Panel) examining 490 studies where schools used data to make instructional decisions.
Of these studies, the report could only find 64 that used experimental or quasi-experimental designs. Of these it could find only six – yes, six – that met the Institute of Education Sciences standard for making causal claims about data-driven decisions to improve student achievement.
And when examining these six studies, the panel found “low evidence” to support data-driven instruction. They concluded that the theory that data-driven instructional decisions improve student test scores has not been proven in any way, shape or form.
It’s Harmful – The Stereotype Threat and Motivation
Data-driven instruction essentially involves grouping students based on their performance on standardized tests.
You put the low scorers HERE, the students on the bubble who almost reached the next level HERE, and the advanced students HERE. That way you can easily differentiate instruction and help meet their needs.
However, there is a mountain of psychological research showing that this practice is harmful to student learning. Even if you don’t put students with different test scores in different classes, simply informing them that they belong to one group or another has intense cognitive effects.
Simply being told that you are in a group with lower test scores depresses your academic outcomes. This is known as the stereotype threat.
When you focus on test scores and inform students of where they fall on the continuum down to the percentile – of how far below average they are – you can trigger this threat. Simply tracking students in this way can actually make their scores worse.
It can create negative feelings about school, threatening students’ sense of belonging, which is key to academic motivation.
But it’s not just the low scorers who are harmed. Even the so-called “advanced” students can come to depend on their privileged status. They define themselves by their achievement, collecting prizes, virtual badges and stickers. These extrinsic rewards then transform their motivation from being driven by the learning and the satisfaction of their curiosity to depending on what high achievement gets them, researchers have found.
In short, organizing all academics around tests scores is a sure way to lower them.
The Data Doesn’t Capture Important Factors
Data-driven instruction is only as good as the data being used. But no data system can be all inclusive.
When we put blinders on and say only these sorts of factors count, we exclude important information.
For instance, two students do the same long-term project and receive the same grade. However, one student overcame her natural tendency to procrastinate and learned more than in past projects. The other did not put forth his best effort and achieved lower than his usual.
If we only look at the data, both appear the same. However, good teachers can see the difference.
Almost every year I have a few students who are chronically tardy to class. A good teacher finds out why – if this is because they aren’t making the best use of the class interval or if they have a greater distance to travel than other students. However, if we judge solely on the data, we’re supposed to penalize students without considering mitigating factors. That’s being data-driven – a poor way to be a fair teacher.
It has been demonstrated repeatedly that student test scores are highly correlated with parental income. Students from wealthier parents score well and those from more impoverished families score badly. That does not mean one group is smarter or even more motivated than the other. Living in poverty comes with its own challenges. Students who have to take care of their siblings at home, for instance, have less time for homework than those who have nothing but free time.
A focus solely on the data ignores these factors. When we’re admonished to focus on the data, we’re actually being told to ignore the totality of our students.
No one wants to be reduced to a number or a series of statistics.
It is extremely insulting to insist that the best way for teachers to behave is to treat their students as anything other than human beings.
They are people with unique needs, characteristics, and qualities, and should be treated accordingly.
When one of my students does an amazing job on an assignment or project, my first impulse is not to reduce what they’ve done to a letter grade or a number. I speak my approbation aloud. I write extensive comments on their papers or conference with them about what they’ve done.
Certainly, I have to assign them a grade, but that is merely one thing educators do. To reduce the relationship to that – and only that – is extremely reductive. If all you do is grade the learner, you jeopardize the learning.
Every good teacher knows the importance of relationships. Data-driven instruction asks us to ignore these lessons in favor of a mechanistic approach.
I’m sorry. My students are not widgets and I refuse to treat them as such.
I am so sick of going to conferences or faculty meetings where we focus exclusively on how to get better grades or test scores from our students. We should, instead, focus on how to see the genius that is already there! We should find ways to help students self-actualize, not turn them into what we think they should be.
At this point, someone inevitably says that life isn’t fair. Our students will have to deal with standardized tests and data-driven initiatives when they get older. We have to prepare them for it.
If the real world is unfair, I don’t want my students to adjust to that. I want to make it better for them.
Imagine telling a rape victim that that’s just the way the world is. Imagine telling a person brutalized by the police that the world is unfair and you just have to get used to it.
This is a complete abdication not just of our job as teachers but our position as ethical human beings.
Schools are nothing without students. We should do everything we can to meet their needs. Period.
It’s Contradictory – It’s Not How We Determine Value in Other Areas
Finally, there is an inherent contradiction that all instruction must be justified by data.
We don’t require this same standard for so many aspects of schooling.
Look around any school and ask yourself if everything you see is necessarily based on statistics.
Does the athletic program exist because it increases student test scores? Does each student lunch correlate with optimum grades? Do you have computers and iPads because they have a measureable impact on achievement?
Some administrators and principals DO try to justify these sorts of things by reference to test scores. But it’s a retroactive process.
They are trying to connect data with things they already do. And it’s completely bogus.
They don’t suddenly believe in football because they think it will make the team get advanced scores. They don’t abruptly support technology in the classroom because they think it will make the school achieve adequate yearly progress.
They already have good reasons to think athletics helps students learn. They’ve seen participation in sports help students remain focused and motivated – sometimes by reference to their own lives. Likewise, they’ve seen the value of technology in the classroom. They’ve seen how some students turn on like someone flipped a switch when a lesson has a technological component.
These aren’t necessarily quantifiable. They don’t count as data but they are based on evidence.
We come to education with certain beliefs already in place about what a school should do and others are formed based on the empiricism of being there, day-in, day-out. “Data” rarely comes into the decision making process as anything but a justification after the fact.
And so we can firmly put the insistence on data-driven instruction in the trash bin of bad ideas.
It is unscientific, unproven, harmful, reductive, dehumanizing and contradictory.
The next time you hear an administrator or principal pull out this chestnut, take out one of these counterarguments and roast it on an open fire.
No more data-driven instruction.
Focus instead on student-driven learning.
Don’t let them co-opt you into the cult of numerology. Remain a difference-maker. Remain a teacher.
Like this post? I’ve written a book, “Gadfly on the Wall: A Public School Teacher Speaks Out on Racism and Reform,” now available from Garn Press. Ten percent of the proceeds go to the Badass Teachers Association. Check it out!
WANT A SIGNED COPY?
Click here to order one directly from me to your door!
33 thoughts on “The Six Biggest Problems with Data-Driven Instruction”
Reblogged this on David R. Taylor-Thoughts on Education.
The biggest problem is equating “data” with “test scores”. As Peter Greene has pointed out, everything that goes on in a class is data. The contributions Jane makes to the Twelfth Night discussion. The fact that Sally and Susie come in the door glaring daggers at each other. The fact that Jason has another black eye. The fact that only one kid got an A on the last quiz. The fact that Pat put his head down on his desk at the beginning of class and hasn’t lifted it since. The fact that the students are buzzing about the latest administrative decree about the dress code. Everything that happens, every behavior, whether directly related to academics or not is data. Test scores are just a convenient, if wildly inaccurate and basically useless, way to boil all that data into a single number. All teachers should be data driven. That does not mean they should be test-score driven.
Dienne, I think we both agree but are quibbling over language. The standardizers have turned “data” into their creature and I don’t think we should use that word. It’s all empiricism but much of what we, teachers observe is not quantifiable in the same way.
AMEN!!! As a recently retired public school teacher, in special ed. for the last 14 years, elementary classrooms before that, I found that data is interesting, and can give clues to the problem, but it cannot diagnose or prescribe. If the teacher is not allowed and encouraged to use all her stored knowledge from classes and an understanding of the student, it just does not work. That’s why teachers are still the most important component of the school system. Personally I chose to stay in the classroom for 43 years, because I feel that most administrators are paper pushers who have forgotten that we teach students, not books. And because I would have NO patience for teachers who teach from the book, grounds for firing in my opinion!
LikeLiked by 1 person
May I never become one of those teachers who value “data” more than children.
[…] Singer writes here about the mechanistic, anti-child implicationsand consequences of data-driven Instruction. He identifies six issues. I offer only the first of […]
LikeLiked by 1 person
[…] and Pennsylvania teacher Steven Singer has a post out on the distinction between being “data-driven” and “data-informed”. Needless to say, it’s a big […]
LikeLiked by 1 person
[…] Source: The Six Biggest Problems with Data-Driven Instruction | gadflyonthewallblog […]
[…] 5) The Six Biggest Problems with Data-Driven Instruction […]
[…] we practice. That pattern is necessary to reinforce almost any skill acquisition. I know it’s trendy to flip the classroom a la Khan Academy with learning done through videos watched on-line at home and practice done in school. But when […]
[…] So let me take you back to Boston in the middle of the 19th Century and show you exactly where things first went wrong and how they still go wrong in nearly the same way. […]
[…] that isn’t learning. It’s likemindedness. It’s the ability to conform to one particular way of thinking about […]
[…] learning how to take a multiple choice test on reading is not the same as learning how to read and u…. It is not the same as interacting with, comprehending and forming an opinion about that […]
[…] I’m told that my worth as a professional is mainly defined by student test scores – that I should use those scores to drive my entire class, that my major goal should be increasing the scores and my every waking moment should be spent […]
[…] I am constantly being told not to pay attention to the scores. Instead, I’m told to pay attention to growth – how much this year’s scores have improved from last year’s scores. And the best way to do this, I’m told, is by paradoxically examining the scores in the most minute detail and using them to drive all instruction in the classr…. […]
[…] you think about it, data is one of the most financially valuable things schools […]
[…] Singer, D. (25 de septiembre de 2018). The Six Biggest Problems with Data-Driven Education [Entrada del Blog]. Recuperado de https://gadflyonthewallblog.com/2018/09/25/the-six-biggest-problems-with-data-driven-instruction/ […]
It’s an honor to be cited in your paper.
[…] Since most administrators have drunk deep of the testing Kool-Aid, they now force teachers to use test scores to drive instruction. So since the tests don’t focus on the most essential parts of Reading, Writing, Math, and […]
[…] needs to understand that Common Core cannot be separated into curriculum and testing. The testing drives the curriculum. You can’t say you’re against testing being used to make high stakes decisions and then have […]
[…] now after 16 years in the classroom, I’m routinely told by my principal to use student testing data to drive my instruction. And, moreover, to document how I am doing so in […]
[…] asked us what we noticed about these two sets of scores and I almost spit out my […]
[…] I don’t like tests. I think they can too easily become cruel games of “guess what the testmaker … […]
[…] I don’t like tests. I think they can too easily become cruel games of “guess what the testmaker … […]
[…] policy makers will tell educators to manage everything with a clipboard and a spreadsheet – for example, to increase the percentage of positive interactions vs negative ones in a […]
[…] Technology permits the collection of more data, raising student privacy concerns. Read Steven Singer’s The Six Biggest Problems with Data-Driven Instruction. Data can be unscientific, unproven, and dehumanizing. Those are just some of the […]
[…] They were replaced with standardized testing, test prep for the standardized testing, testing before the testing, and workbooks about how to do the testing right. […]
[…] We’ll let data drive the instruction instead driving it based on the actual living, breathing, hum… […]
[…] quarantined in greater numbers. I don’t love explaining to my administrators or principals about how useless standardized tests are and being told that my opinion is wrong. I don’t love how my educated viewpoints based on decades of classroom experience are always […]
[…] more teaching to the test. No more narrowing the curriculum. No more pressure to increase test scores. […]
[…] deep of the testing Kool-Aid, they now force teachers to educate in just this manner – to use test scores to drive instruction. So since the tests doesn’t focus on the most essential parts of Reading, Writing, Math, and […]
[…] They think it’s to be a toady to the Superintendent or higher level administrators. They think they have to demonstrate their performance to their bosses with whatever data is available at every turn. (This is also what they expect teachers to do for them.) […]
[…] certainly true that NCLB collected a mountain of data. However, such information conceivably could have been gathered without making standardized testing the fulcr…. It could just as easily have been collected based entirely on classroom grades. It’s just that […]