Aaron Regunberg: Behind the Standardized Test Curtain
Friday, April 20, 2012
Standardized tests. It’s hard nowadays to think about education without them—they’ve become the metric, the substance, the rationale for schooling. We use them to identify success and failure, to tell us which schools to shut down, which teachers to fire, and which students to let graduate. I think it’s safe to say standardized tests carry more weight today than literally any other aspect of our education system.
But what do these tests actually consist of? I recently got a window into this issue after reading the Kafkaesque account of a man who spent 15 years of his life working in the K-12 testing business for some of the biggest players in the industry (Pearson Education, Educational Testing Service, American Institutes of Research, etc.—names any teacher or school administrator would be very familiar with). The man, Todd Farley, wrote, “While I did enjoy the career (good money, nice people, fun trips), it also left me completely convinced of the utter folly of entrusting decisions about American students, teachers, and schools to the for-profit industry that long employed me. I don’t know how anyone who’s seen what I’ve seen could feel any differently.”
Farley did not develop this position out of any pedagogical or ideological concerns; he’s not an educator and he doesn’t seem to be invested in this debate one way or the other. Rather, he came to his position for the same reason a guy who works in the kitchen of a not-so-reputable restaurant might choose to never frequent that eatery during off-hours—he knows what goes on back there and will be damned before he’ll put that food in his own mouth.
Farley’s not talking about the major disasters that get publicity, like the recent cheating scandals in Atlanta or the one currently being investigated in D.C.. In Farley’s words, “Any Google search will result in many similar testing tragedies, but I’d say the scandals that make the news are only the tip of the iceberg. In fact, I’d say there aren’t scoring problems on some standardized tests—my experience suggests there are scoring problems on all of them.”
Those problems come from a few different sources, according to his account. First are the scorers themselves, whom he described as “a motley crew of temporary employees earning low hourly wages…while many of those people are earnest and conscientious employees, many others are not.” He goes on to tell a number of recollections about the folks he’s worked with at test-scoring centers, many of whom only take the jobs because they can’t get real employment elsewhere. From those who come in and leave drunk to those who clearly don’t care what they’re doing (he tells one story of a scorer who gave every student response the score of two one day just for the hell of it), he does not paint a flattering picture of the folks we are allowing to shape the futures of students, teachers, and schools.
But even more disturbing are the stringent scoring rules that are utilized unwaveringly in order to get these large groups of temporary employees to score the tests in a standardized way. Take, for example, the rubric for the very first standardized test Farley scored—if the fourth-grade students gave an example of bicycle safety in a drawing (wearing a helmet, stopping at a red light, etc.) they would get a point; if not, they’d get zero. Farley picked up the first test and saw the student had indeed included one example: the rider in the drawing was wearing a helmet. He was also doing an Evel Knievel leap over a chasm of fire. Puzzled, Farley consulted his supervisor; he was told the rider was wearing a helmet and that was enough to indicate the child understood the basics of bike safety. He was also told to credit any bike at a stop sign, “and soon I was being instructed to give full credit to a poster showing a bike flying through the air in front of a stop sign; a bike in the back of a pick-up truck in front of a stop sign; and a bike busted into pieces in front of a stop sign.” On the other hand, when Farley read a beautifully moving and well-constructed essay by one high school girl about “A Special Place,” he could only give it three out of four points because her piece did not include the words “a special place.”
But there’s an even darker side to Farley’s account: “Perhaps most importantly,” he writes, “the test-scoring industry cheats. It cheats on qualification tests to make sure there is enough personnel to meet deadlines/get tests scored; it cheats on reliability scores to give off the appearance of standardization even when that doesn’t exist; it cheats on validity scores and calibration scores and anything else that might be needed. I don’t want to just point fingers here, because I am guilty too, and over the years I fudged the numbers like everyone else. Statistical tomfoolery and corporate chicanery were the hallmark of my test-scoring career, and while I’m not proud of that, it is a fact. Remember, I was never in the testing business for any reason other than to earn a pay check, just like many of the testing companies are in it solely to make a buck.”
I understand that the bigger-picture critiques being adopted by many educators and community members across the country do not resonate with everyone. For me, standardized tests are dangerous because I don’t think they can actually measure a student’s ability to think, to be a critical learner, to work well with others, to have emotional intelligence—in short, all the things that actually matter in life and that one would hope an adequate education would provide. But for those readers who don’t share this pedagogical perspective, I hope Farley’s nuts-and-bolts condemnation means something. After all, he was behind the curtain. If my own experience in food retail taught me anything, it’s that when a waiter tells you not to get a particular item on the menu, it’s a big mistake to get that item. And it’d be an even bigger mistake to base your education system around it.
- Aaron Regunberg: Achievement First has Little Support in Providence
- Aaron Regunberg: Reform the Business Sector in 2012
- Aaron Regunberg: History and Education Reform
- Aaron Regunberg: Be Fair to Those Who Care
- Aaron Regunberg: Shame on You, Speaker Fox
- Aaron Regunberg: Stop Calling Them “Illegals”
- Aaron Regunberg: Brown Needs to Pay Up
- Aaron Regunberg: Teaching Evaluations Another Education Reform Failure
- Aaron Regunberg: The Case Against Achievement First
- Aaron Regunberg: Do Mayoral Academies Have Something to Hide?
- Aaron Regunberg: The Importance of Student Attrition Rates
- Aaron Regunberg: The Downside To Teach For America
- Aaron Regunberg: Let’s Make Rhode Island Competitive
- Aaron Regunberg: The Story Achievement First Doesn’t Want You to Hear
- Aaron Regunberg: Voodoo Economics is No Good for RI
- Aaron Regunberg: MLK was Cooler than You Thought He Was
- Aaron Regunberg: Think Again on Pension Reform
- Aaron Regunberg: Who’s Being Selfish?
- Aaron Regunberg: RI Students Left Out of Education Reform Debate
- Aaron Regunberg: Time for Brown to Pay its Fair Share in Providence
- Aaron Regunberg: Why 16-Year-Olds Know Better than You
- Aaron Regunberg: A Tale of Two Ed Reforms
- Aaron Regunberg: RI-CAN Goes too Far with School Report Cards
- Aaron Regunberg: Behind the Standardized Test Curtain