Data-Driven Decisions, No Data

Posted by on August 20, 2010

According to an article in yesterday’s Washington Post, the outcome of the upcoming D.C. mayoral primary may depend in large part on gains in students’ “test scores” since Mayor Adrian Fenty appointed Michelle Rhee to serve as chancellor of the D.C. Public Schools (DCPS).

That struck me as particularly interesting because, as far as I can tell, Michelle Rhee has never released any test scores to the public. Not an average test score for any grade level or for any of the district’s schools or any subgroup of its students. None.

DCPS only announces the percentage of students (by school and subgroups) whose scores fall within the four categories of below basic, basic, proficient, and advanced. As I discussed in a previous post, these numbers are not test scores – they are one big step removed – and year-to-year changes in these rates don’t necessarily tell us very much about the typical DCPS student. For one thing, none of these changes, whether scores or rates, are actually gains per se, they are cohort-to-cohort changes (e.g., the rates of third graders one year are compared with rate of a different group of third graders the next year). Moreover, as that post explained, it’s possible for a school’s proficiency rate to increase even when there is a decline in average test scores across grades (and vice-versa). Consequently, changes in proficiency rates are a very poor measure of student progress.

The week after that previous post went up, New York State illustrated the point in an unusual, rather dramatic fashion. The state revised the “cut point” scores above which students are called basic, proficient, and advanced. Literally overnight, proficiency rates dropped dramatically. In New York City, we found out that the “proficiency gap” between white and minority students hadn’t narrowed anywhere near as much as we had previously thought – actually, not at all. The proficiency rates for poor and special education students saw particularly steep declines, as did the city’s charter schools. And all of this without a single score moving an inch.

New York, however, was not left in the dark. Like virtually all states, it releases average test scores along with proficiency rates. This means that parents, researchers, and the public can make a more accurate assessment of cohort-to-cohort progress by checking both the changes in rates as well as the changes in scores, which are a better (though still limited) measure of progress.

As the New York situation illustrates, changes in proficiency or other benchmark rates are, by themselves, not only a poor way to gauge performance over time, they are potentially very misleading.

Yet, in DC, the rates are all we are given. For all we know, average test scores in the district have been flat for years, even though proficiency rates were rising until this year. It’s also entirely possible that the proficiency rate increases in D.C. over the past few years mask far more modest growth in test scores – or that this year’s proficiency decreases mask rising average scores. The rates can change a great deal even if scores are only marginally different between two cohorts of students.

The significant increases in DC’s NAEP scores between 2003 and 2009 (in fourth grade math and reading and eighth grade math) only provide information about a tiny slice of DCPS students (two grades) every other year – that’s why DCPS’ own results are so important. Even if they don’t track individual students over time (NAEP doesn’t either), they at least tell us – every year – how the typical student in each grade (or the typical student at each school) tested, and how this compares with previous cohorts over time.

The point is that we don’t really have that much reliable information about how well D.C. students are doing, because average test score data aren’t publicly available (at least not that I could find, at either DCPS or the Office of the State Superintendent of Education).

So, while education certainly appears to be one of the most important issues in this year’s Democratic Party primary, any vote based on actual test scores will have to be cast by absentee ballot.


5 Comments posted so far

  • [...] This post was mentioned on Twitter by Diane Ravitch, Shanker Blog. Shanker Blog said: NEW POST: WaPo says DC mayoral primary hinges on test scores, but DCPS has never released test scores — http://shankerblog.org/?p=592 [...]

    August 20, 2010 at 9:13 AM
  • [...] all of this, add the fact that we don’t have any actual test score data from DC-CAS, and must rely on completely inadequate proficiency rates to assess performance (our [...]

    November 1, 2010 at 11:49 AM
  • [...] Of course, it contains the usual problems that pervade our NCLB-style misunderstanding of education statistics: The mistaken idea that you can easily compare districts in different states with different tests and different definitions of proficiency; the false concept that cohort-to-cohort changes are “growth” (you’re comparing two completely different groups of students, especially at the grade level); and the common mislabeling (and misinterpretation) of proficiency rates as “scores” (DCPS does not release its actual test score data). [...]

    February 15, 2011 at 4:05 PM
  • [...] they are for use in value-added models). It’s also worth noting that a few states and districts, including Washington, D.C., release only the rates and not the [...]

    August 23, 2011 at 1:06 PM
  • [...] fact that, each year, a different set of students takes the test. I’ve discussed the fact that proficiency rates are not test scores – they only tell you the proportion of students above a given line – and that the rates and [...]

    September 26, 2011 at 11:26 AM

Sorry, the comment form is closed at this time.

Disclaimer

This web site and the information contained herein are provided as a service to those who are interested in the work of the Albert Shanker Institute (ASI). ASI makes no warranties, either express or implied, concerning the information contained on or linked from shankerblog.org. The visitor uses the information provided herein at his/her own risk. ASI, its officers, board members, agents, and employees specifically disclaim any and all liability from damages which may result from the utilization of the information provided herein. The content in the shankerblog.org may not necessarily reflect the views or official policy positions of ASI or any related entity or organization.

Banner image adapted from 1975 photograph by Jennie Shanker, daughter of Albert Shanker.