Aaron Regunberg: RI-CAN Goes too Far with School Report Cards

Friday, April 13, 2012

Aaron Regunberg, GoLocalProv MINDSETTER™

 

Earlier this week RI-CAN, the well-funded ‘education reform’ lobbying group, released a set of school report cards that, according to their sleek website, “provide Rhode Island families with detailed information about all 300 public schools in our state based on the academic achievement of their students” in order to “create transparency and awareness about how public schools perform” and “protect great schools.” Unfortunately for RI-CAN, they have been getting a lot of pushback from teachers unions, charter schools, and even Commissioner Gist (folks who don’t normally agree with one another too much)—pushback that I think can best be described using the words of my statistics professor when I showed the report cards to him, who asked in confusion, “Are they serious?”

For a more detailed explanation of RI-CAN’s faulty methodology, I urge everyone to take a look at this brief and articulate response from the Learning Community, a community-based charter school in Central Falls, which begins by stating that “The methodological deficiencies of the RI-CAN report cards render them at best useless and, at worst, harmful to our state’s efforts to support the education of every child.” It then lays out the five most obvious problems with RI-CAN’s system, which I’ll summarize here for those who don’t want to read the whole response.

View Larger +

First, the Learning Community points out that in these report cards, “Every school’s ‘overall student performance’ score is really based on the scores of only one grade level. For example, RI-CAN’s measure for an elementary school is based on the performance of the 5th grade only. This means that the scores of 40 students might represent a school where 300 students were actually tested.”

GET THE LATEST BREAKING NEWS HERE -- SIGN UP FOR GOLOCAL FREE DAILY EBLAST

Second, “RI-CAN’s approach to measuring school performance is dated and has been discredited nationally as too narrow. RI-CAN’s report cards rely solely on state standardized test data and do nothing to portray the context of each of our state’s schools.” This differentiates RI-CAN’s tool from other report card systems like those in DC and New York, which utilize a range of data points in their calculations.

Third—and probably most embarrassing for the folks at RI-CAN—the Learning Community’s response outlines how “RI-CAN misuses basic math. RI-CAN’s ‘student subgroup performance,’ ‘achievement gap,’ and ‘performance gains’ measures rely on combining the performance of multiple groups of students into a single score. Instead of doing the basic math to determine an accurate score, RI-CAN took a short cut and averaged a series of averages into a composite score,” ignoring the fact that “basic mathematics requires us to weight the scores based on the number of students in each group.” This is the part that so puzzled my stats professor, who informed me that if I ever handed in a project based on math like this he’d give it an F, no questions asked.

Of course, there’s more. The Learning Community goes on to explain how RI-CAN ranks schools on faulty numbers. “When researchers release data, they identify a margin of error. NECAP data includes these ranges, to let us know that a school that scored a 54 might actually be within the same range as a school that scored a 52. RI-CAN, however, chose to rank schools based on a single number—identifying distinctions even when there is no statistically significant difference between the schools’ performance.”

And fifth, “The RI-CAN system is unfair to urban schools. Because the RI-CAN report cards are based on a single grade level, schools with few students in that grade who are low-income or Black or Hispanic will not have a score for those subgroups. Many suburban schools are not given any score for ‘student subgroup performance,’ whereas every urban school is. By contrast, the RI Department of Education, recognizing the unfairness of this approach, is preparing to hold nearly all schools accountable for subgroup performance.”

I think the Learning Community deserves a great deal of credit for speaking out against RI-CAN’s faulty methodology, despite the risk of angering one of the best-funded and most well-connected lobbying forces in the state. And they deserve it all the more so considering their school received high ratings in RI-CAN’s system, which ranked the Learning Community in the ‘Top 10’ on 7 of the report cards’ 14 indicators (in other words, this response isn’t sour grapes, it’s a sincere desire to point out blatant misinformation).

The only major criticism I would add to that of the Learning Community deals with these‘Top 10’ lists that RI-CAN included with their report cards. Of RI-CAN's ‘Top 10’ schools for low-income student performance, five have a poverty rate (measured by the percent of kids receiving free or reduced price lunch in the grade level analyzed) below 20 percent. Eight have a poverty rate below the state average, and all 10 have rates below 62 percent. When you compare that to a Providence public school average rate of 86 percent, it becomes clear that the conclusions RI-CAN is trying to get us to draw are simply meaningless.

View Larger +

So…what was RI-CAN thinking? Why create a system that would clearly be dismissed as either stupid or dishonest by pretty much everyone who knows anything about the topic? I can’t pretend to know the answer, though I’m finding it uncomfortably difficult to rule out the possibility that the whole thing’s an elaborate ruse to trick more Rhode Islanders into visiting RI-CAN’s website and joining their “membership,” as well as a tool to aggrandize RI-CAN’s favored schools (hint—they’re probably mayoral academies). At any rate, I hope the widespread pushback they’re facing will teach RI-CAN that folks here are a little smarter than they apparently thought we were.

But there’s one more thing that troubles me about the report card issue, and it’s this: how does ranking—even ranking that is more accurate and less obviously designed for political gain—actually improve the public school system here in Rhode Island? Needless to say, data is incredibly important, and having accurate information about what is working and what is not working well in our schools is a crucial prerequisite for identifying strategies for progress. But any system of school report cards seems destined to have two fundamental flaws beyond the dumb methodological faults of RI-CAN’s model.

First of all, how can one quantify all the different components of diverse school communities in such a standardized and reductive way as this and still provide useful information? How, for example, can one accurately compare a school with an arts-based curriculum to a college preparatory program using one letter grade? I don’t think it can meaningfully be done.

But perhaps even more importantly, I fail to see what even the best possible public rankings of schools can achieve other than to stigmatize and deprive the schools that are in most need of assistance. RI-CAN talks a lot about how every child in Rhode Island deserves a great education, but what about the children who remain in poorly graded schools after their school’s most well informed families pull out? How does this system in any way help these students gain a great public education? That, of course, is the fundamental inconsistency in RI-CAN’s logic. Like all schemes based on a free market ideology, these policies create winners and losers. And when those losers are children who are finding it even more difficult to get the great education they deserve, how can RI-CAN claim to be working for every Rhode Island child?


 

If you valued this article, please LIKE GoLocalProv.com on Facebook by clicking HERE.

 

Related Articles

 

Enjoy this post? Share it with others.

 
 

Sign Up for the Daily Eblast

I want to follow on Twitter

I want to Like on Facebook