A New Idea For Test-Based Accountability In DC: Actual Test Scores

Posted by on July 14, 2014

The Washington Post reports on an issue that we have discussed here on many occasions: The incompleteness of the testing results released annually by the District of Columbia Public Schools (DCPS), or, more accurately, the Office of the State Superintendent of Education (OSSE), which is responsible for testing in DC schools.

Here’s the quick backstory: For the past 7-8 years or so, DCPS/OSSE have not released a single test score for the state assessment (the DC-CAS). Instead, they have released only the percentage of students whose scores meet the designated cutoff points for the NCLB-style categories of below basic, basic, proficient and advanced. I will not reiterate all of the problems with these cutpoint-based rates and how they serve to distort the underlying data, except to say that they are by themselves among the worst ways to present these data, and there is absolutely no reason why states and districts should not release both rates and average scale scores.

The Post reports, however, that one organization — the Broader, Bolder Approach to Education — was able to obtain the actual scale score data (by subgroup and grade) for 2010-2013, and that this group published a memo-style report alleging that DCPS’ public presentation of their testing results over the past few years has been misleading. I had a mixed reaction to this report and the accompanying story. Read More »


An Education Hearing I’d Like To See

Posted by on May 7, 2014

At the end of February, the District of Columbia Council’s Education Committee held its annual hearing on the performance of the District’s Public Schools (DCPS). The hearing (full video is available here) lasted over four hours, and included discussion on a variety of topics, but there was, inevitably, a block of time devoted to the discussion of DCPS testing results (and these questions were the focus of the news coverage).

These exchanges between Council members and DCPS Chancellor Kaya Henderson focused particularly on the low-stakes Trial Urban District Assessment (TUDA).* Though it was all very constructive and not even remotely hostile, it’s fair to say that Ms. Henderson was grilled quite a bit (as is often the case at these kinds of hearings). Unfortunately, the arguments from both sides of the dais were fraught with the typical misinterpretations of TUDA, and I could not get past how tragic it is to see legislators question the superintendent of a large urban school district based on a misinterpretation of what the data mean – and to hear that superintendent respond based on the same flawed premises.

But what I really kept thinking — as I have before in similar contexts — was how effective Chancellor Henderson could have been in answering the Council’s questions had she chosen to interpret the data properly (and I still hold out hope that this will become the norm some day). So, let’s take a quick look at a few major arguments that were raised during the hearing, and how they might have been answered. Read More »


Is Selective Admission A School Improvement Plan?

Posted by on January 23, 2014

The Washington Post reports that parents and alumni of D.C.’s Dunbar High School have quietly been putting together a proposal to revitalize what the article calls “one of the District’s worst performing schools.”

Those behind the proposal are not ready to speak about it publicly, and details are still very thin, but the Post article reports that it calls for greater flexibility in hiring, spending and other core policies. Moreover, the core of the plan – or at least its most drastic element – is to make Dunbar a selective high school, to which students must apply and be accepted, presumably based on testing results and other performance indicators (the story characterizes the proposal as a whole with the term “autonomy”). I will offer no opinion as to whether this conversion, if it is indeed submitted to the District for consideration, is a good idea. That will be up to administrators, teachers, parents, and other stakeholders.

I am, however, a bit struck by two interrelated aspects of this story. The first is the unquestioned characterization of Dunbar as a “low performing” or “struggling” school. This fateful label appears to be based mostly on the school’s proficiency rates, which are indeed dismally low – 20 percent in math and 29 percent in reading. Read More »


Teacher Turnover In DCPS

Posted by on January 15, 2014

Teacher turnover – the rates at which teachers leave the profession and switch schools – is obviously a very important outcome in education. Although not all turnover is necessarily a “bad thing” – some teachers simply aren’t cut out for the job and leave voluntarily (or are fired) – unusually high turnover means that schools must replace large proportions of their workforces on an annual basis. This can have serious implications not only for the characteristics (e.g., experience) of schools’ teachers, but also for schools’ costs, cohesion and professional cultures.

According to the most recent national data (which are a few years old), annual public school teacher turnover is around 16 percent, of which roughly half leave the profession (“leavers”), and half switch schools (“movers”). Both categories are equally important from the perspective of individual schools, since they must replace teachers regardless of where they go. In some subsets of schools and among certain groups of teachers, however, turnover is considerably higher. For instance, among teachers with between 1-3 years of experience, turnover is almost 23 percent. Contrary to popular opinion, though, the relationship between school poverty (i.e., free/reduced-price lunch rates) and turnover isn’t straightforward, at least at the national level. Although schools serving larger proportions of lower-income students have a larger percentage of “movers” every year, they have a considerably lower proportion of “leavers” (in part due to retirement).

This national trend, of course, masks considerable inter-district variation. One example is the District of Columbia Public Schools (DCPS). Read More »


DC School Growth Scores And Poverty

Posted by on July 15, 2013

As noted in a nice little post over at Greater Greater Washington’s education blog, the District of Columbia Office of the State Superintendent of Education (OSSE) recently started releasing growth model scores for DC’s charter and regular public schools. These models, in a nutshell, assess schools by following their students over time and gauging their testing progress relative to similar students (they can also be used for individual teachers, but DCPS uses a different model in its teacher evaluations).

In my opinion, producing these estimates and making them available publicly is a good idea, and definitely preferable to the district’s previous reliance on changes in proficiency, which are truly awful measures (see here for more on this). It’s also, however, important to note that the model chosen by OSSE – a “median growth percentile,” or MGP model, produces estimates that have been shown to be at least somewhat more heavily associated with student characteristics than other types of models, such as value-added models proper. This does not necessarily mean the growth percentile models are “inaccurate” – there are good reasons, such as resources and more difficulty with teacher recruitment/retention, to believe that schools serving poorer students might be less effective, on average, and it’s tough to separate “real” effects from bias in the models.

That said, let’s take a quick look at this relationship using the DC MGP scores from 2011, with poverty data from the National Center for Education Statistics. Read More »


Describing, Explaining And Affecting Teacher Retention In D.C.

Posted by on November 11, 2012

The New Teacher Project (TNTP) has released a new report on teacher retention in D.C. Public Schools (DCPS). It is a spinoff of their “The Irreplaceables” report, which was released a few months ago, and which is discussed in this post. The four (unnamed) districts from that report are also used in this one, and their results are compared with those from DCPS.

I want to look quickly at this new supplemental analysis, not to rehash the issues I raised about“The Irreplaceables,” but rather because of DCPS’s potential importance as a field test site for a host of policy reform ideas – indeed, the majority of core market-based reform policies have been in place in D.C. for several years, including teacher evaluations in which test-based measures are the dominant component, automatic dismissals based on those ratings, large performance bonuses, mutual consent for excessed teachers and a huge charter sector. There are many people itching to render a sweeping verdict, positive or negative, on these reforms, most often based on pre-existing beliefs, rather than solid evidence.

Although I will take issue with a couple of the conclusions offered in this report, I’m not going to review it systematically. I think research on retention is important, and it’s difficult to produce reports with original analysis, while very easy to pick them apart. Instead, I’m going to list a couple of findings in the report that I think are worth examining, mostly because they speak to larger issues. Read More »


Fundamental Flaws In The IFF Report On D.C. Schools

Posted by on February 6, 2012

A new report, commissioned by the District of Columbia Mayor Vincent Gray and conducted by the Chicago-based consulting organization IFF, was supposed to provide guidance on how the District might act and invest strategically in school improvement, including optimizing the distribution of students across schools, many of which are either over- or under-enrolled.

Needless to say, this is a monumental task. Not only does it entail the identification of high- and low-performing schools, but plans for improving them as well. Even the most rigorous efforts to achieve these goals, especially in a large city like D.C., would be to some degree speculative and error-prone.

This is not a rigorous effort. IFF’s final report is polished and attractive, with lovely maps and color-coded tables presenting a lot of summary statistics. But there’s no emperor underneath those clothes. The report’s data and analysis are so deeply flawed that its (rather non-specific) recommendations should not be taken seriously. Read More »


Burden Of Proof, Benefit Of Assumption

Posted by on January 20, 2012

** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post

Michelle Rhee, the controversial former chancellor of D.C. public schools, is a lightning rod. Her confrontational style has made her many friends as well as enemies. As is usually the case, people’s reaction to her approach in no small part depends on whether or not they support her policy positions.

I try to be open-minded toward people with whom I don’t often agree, and I can certainly accept that people operate in different ways. Honestly, I have no doubt as to Ms. Rhee’s sincere belief in what she’s doing; and, even if I think she could go about it differently, I respect her willingness to absorb so much negative reaction in order to try to get it done.

What I find disturbing is how she continues to try to build her reputation and advance her goals based on interpretations of testing results that are insulting to the public’s intelligence. Read More »


Success Via The Presumption Of Accuracy

Posted by on July 27, 2011

In our previous post, Professor David K. Cohen argued that reforms such as D.C.’s new teacher evaluation system (IMPACT) will not by themselves lead to real educational improvement, because they focus on the individual rather than systemic causes of low performance. He framed this argument in terms of the new round of IMPACT results, which were released two weeks ago. While the preliminary information was limited, it seems that the distribution of teachers across the four ratings categories (highly effective, effective, minimally effective, and ineffective) were roughly similar to last year’s – including a small group of teachers fired for receiving the lowest “ineffective” rating, and a somewhat larger group (roughly 200) fired for having received the “minimally effective” label for two consecutive years.

Cohen’s argument on the importance of infrastructure does not necessarily mean that we should abandon the testing of new evaluation systems, only that we should be very careful about how we interpret their results and the policy conclusions we draw from them (which is good advice at all times). Unfortunately, however, it seems that caution is in short supply. For instance, shortly after the IMPACT results were announced, the Washington Post ran an editorial, entitled “DC Teacher Performance Evaluations Are Working,” in which a couple of pieces of “powerful evidence” were put forward in an attempt to support this bold claim. The first was that 58 percent of the teachers who received a “minimally effective” rating last year and remained in the district were rated either “effective” or “highly effective” this year. The second was that around 16 percent of DC teachers were rated “highly effective” this year, and will be offered bonuses, which the editorial writers argued shows that most teachers “are doing a good job” and being rewarded for it.

The Post’s claim that these facts represent evidence – much less “powerful evidence” – of IMPACT’s success is a picture-perfect example of the flawed evidentiary standards that too often drive our education debate. The unfortunate reality is that we have virtually no idea whether IMPACT is actually “working,” and we won’t have even a preliminary grasp for some time. Let’s quickly review the Post’s evidence. Read More »


Evaluating Individual Teachers Won’t Solve Systemic Educational Problems

Posted by on July 26, 2011

** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post

Our guest author today is David K. Cohen, John Dewey Collegiate Professor of Education and professor of public policy at the University of Michigan, and a member of the Shanker Institute’s board of directors.  

What are we to make of recent articles (here and here) extolling IMPACT, Washington DC’s fledging teacher evaluation system, for how many “ineffective” teachers have been identified and fired, how many “highly effective” teachers rewarded? It’s hard to say.

In a forthcoming book, Teaching and Its Predicaments (Harvard University Press, August 2011), I argue that fragmented school governance in the U.S. coupled with the lack of coherent educational infrastructure make it difficult either to broadly improve teaching and learning or to have valid knowledge of the extent of improvement. Merriam-Webster defines “infrastructure” as: “the underlying foundation or basic framework (as of a system or organization).” The term is commonly used to refer to the roads, rail systems, and other frameworks that facilitate the movement of things and people, or to the physical and electronic mechanisms that enable voice and video communication. But social systems also can have such “underlying foundations or basic frameworks”. For school systems around the world, the infrastructure commonly includes student curricula or curriculum frameworks, exams to assess students’ learning of the curricula, instruction that centers on teaching that curriculum, and teacher education that aims to help prospective teachers learn how to teach the curricula. The U.S. has had no such common and unifying infrastructure for schools, owing in part to fragmented government (including local control) and traditions of weak state guidance about curriculum and teacher education.

Like many recent reform efforts that focus on teacher performance and accountability, IMPACT does not attempt to build infrastructure, but rather assumes that weak individual teachers are the problem. There are some weak individual teachers, but the chief problem has been a non-system that offers no guidance or support for strong teaching and learning, precisely because there has been no infrastructure. IMPACT frames reform as a matter of solving individual problems when the weakness is systemic. Read More »


Settling Scores

Posted by on March 21, 2011

In 2007, when the D.C. City Council passed a law giving the mayor control of public schools, it required that a five-year independent evaluation be conducted to document the law’s effects and suggest changes. The National Research Council (a division of the National Academies) was charged with performing this task. As reported by Bill Turque in the Washington Post, the first report was released a couple of weeks ago.

The primary purpose of this first report was to give “first impressions” and offer advice on how the actual evaluation should proceed. It covered several areas – finance, special programs, organizational structure, etc. – but, given the controversy surrounding Michelle Rhee’s tenure, the section on achievement results got the most attention. The team was only able to analyze preliminary performance data; the same data that are used constantly by Rhee, her supporters, and her detractors to judge her tenure at the helm of DCPS.

It was one of those reports that tells us what we should already know, but too often fail to consider. Read More »


The Legend Of Last Fall

Posted by on February 15, 2011

The subject of Michelle Rhee’s teaching record has recently received a lot of attention. While the controversy has been interesting, it could also be argued that it’s relatively unimportant. The evidence that she exaggerated her teaching prowess is, after all, inconclusive (though highly suggestive). A little resume inflation from a job 20 years ago might be overlooked, so long as Rhee’s current claims about her more recent record are accurate. But are they?

On Rhee’s new website, her official bio - in effect, her resume today (or at least her cover letter) - contains a few sentences about her record as chancellor of D.C Public Schools (DCPS), under the header “Driving Unprecedented Growth in the D.C. Public Schools.” There, her test-based accomplishments are characterized as follows:

Under her leadership, the worst performing school district in the country became the only major city system to see double-digit growth in both their state reading and state math scores in seventh, eighth and tenth grades over three years.

This time, we can presume that the statement has been vetted thoroughly, using all the tools of data collection and analysis available to Rhee during her tenure at the helm of DCPS.

But the statement is false. Read More »


Michelle Rhee’s Testing Legacy: An Open Question

Posted by on November 1, 2010

** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post.

Michelle Rhee’s resignation and departure have, predictably, provoked a flurry of conflicting reactions. Yet virtually all of them, from opponents and supporters alike, seem to assume that her tenure at the helm of the D.C. Public Schools (DCPS) helped to boost student test scores dramatically. She and D.C. Mayor Adrian Fenty made similar claims themselves in the Wall Street Journal (WSJ) just last week.

Hardly anybody, regardless of their opinion about Michelle Rhee, thinks that test scores alone are an adequate indicator of student success. But, in no small part because of her own emphasis on them, that is how this debate has unfolded. Her aim was to raise scores and, with few exceptions (also here and here), even those who objected to her “abrasive” style and controversial policies seem to believe that she succeeded wildly in the testing area.

This conclusion is premature. A review of the record shows that Michelle Rhee’s test score “legacy” is an open question. 

There are three main points to consider: Read More »


Data-Driven Decisions, No Data

Posted by on August 20, 2010

According to an article in yesterday’s Washington Post, the outcome of the upcoming D.C. mayoral primary may depend in large part on gains in students’ “test scores” since Mayor Adrian Fenty appointed Michelle Rhee to serve as chancellor of the D.C. Public Schools (DCPS).

That struck me as particularly interesting because, as far as I can tell, Michelle Rhee has never released any test scores to the public. Not an average test score for any grade level or for any of the district’s schools or any subgroup of its students. None. Read More »


A Below Basic Understanding Of Proficiency

Posted by on July 23, 2010

Given our extreme reliance on test scores as measures of educational success and failure, I’m sorry I have to make this point: proficiency rates are not test scores, and changes in proficiency rates do not necessarily tell us much about changes in test scores.

Yet, for example, in the Washington Post editorial about the latest test results from the District of Columbia Public Schools, at no fewer than seven different points (in a 450 word piece) do they refer to proficiency rates (and changes in these rates) as “scores.” This is only one example of many.

So, what’s the problem? Read More »


Disclaimer

This web site and the information contained herein are provided as a service to those who are interested in the work of the Albert Shanker Institute (ASI). ASI makes no warranties, either express or implied, concerning the information contained on or linked from shankerblog.org. The visitor uses the information provided herein at his/her own risk. ASI, its officers, board members, agents, and employees specifically disclaim any and all liability from damages which may result from the utilization of the information provided herein. The content in the shankerblog.org may not necessarily reflect the views or official policy positions of ASI or any related entity or organization.

Banner image adapted from 1975 photograph by Jennie Shanker, daughter of Albert Shanker.