** Reprinted here in the Washington Post
I’ve written many times about how absolute performance levels – how highly students score – are not by themselves valid indicators of school quality, since, most basically, they don’t account for the fact that students enter the schooling system at different levels. One of the most blatant (and common) manifestations of this mistake is when people use NAEP results to determine the quality of a state’s schools.
For instance, you’ll often hear that Massachusetts has the “best” schools in the U.S. and Mississippi the “worst,” with both claims based solely on average scores on the NAEP (though, technically, Massachusetts public school students’ scores are statistically tied with at least one other state on two of the four main NAEP exams, while Mississippi’s rankings vary a bit by grade/subject, and its scores are also not statistically different from several other states’).
But we all know that these two states are very different in terms of basic characteristics such as income, parental education, etc. Any assessment of educational quality, whether at the state or local level, is necessarily complicated, and ignoring differences between students precludes any meaningful comparisons of school effectiveness. Schooling quality is important, but it cannot be assessed by sorting and ranking raw test scores in a spreadsheet.
Read More »
Every two years, the release of data from the National Assessment of Educational Progress (NAEP) generates a wave of research and commentary trying to explain short- and long-term trends. For instance, there have been a bunch of recent attempts to “explain” an increase in aggregate NAEP scores during the late 1990s and 2000s. Some analyses postulate that the accountability provisions of NCLB were responsible, while more recent arguments have focused on the “effect” (or lack thereof) of newer market-based reforms – for example, looking to NAEP data to “prove” or “disprove” the idea that changes in teacher personnel and other policies have (or have not) generated “gains” in student test scores.
The basic idea here is that, for every increase or decrease in cross-sectional NAEP scores over a given period of time (both for all students and especially for subgroups such as minority and low-income students), there must be “something” in our education system that explains it. In many (but not all) cases, these discussions consist of little more than speculation. Discernible trends in NAEP test score data are almost certainly due to a combination of factors, and it’s unlikely that one policy or set of policies is dominant enough to be identified as “the one.” Now, there’s nothing necessarily wrong with speculation, so long as it is clearly identified as such, and conclusions presented accordingly. But I find it curious that some people involved with these speculative arguments seem a bit too willing to assume that schooling factors – rather than changes in cohorts’ circumstances outside of school – are the primary driver of NAEP trends.
So, let me try a little bit of illustrative speculation of my own: I might argue that changes in the economic conditions of American schoolchildren and their families are the most compelling explanation for changes in NAEP. Read More »
** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post
Tomorrow, the education world will get the results of the 2011 National Assessment of Educational Progress (NAEP), often referred to as the “nation’s report card.” The findings – reading and math scores among a representative sample of fourth and eighth graders – will drive at least part of the debate for the next two years, when the next round comes out.
I’m going to make a prediction, one that is definitely a generalization, but is hardly uncommon in policy debates: People on all “sides” will interpret the results favorably no matter how they turn out.
If NAEP scores are positive – i.e., overall scores rise by a statistically significant margin, and/or there are encouraging increases among key subgroups such as low performers or low-income students – supporters of market-based reform will say that their preferred policies are working. They’ll claim that the era of test-based accountability, which began with the enactment of No Child Left Behind ten years ago, have produced real results. Market reform skeptics, on the other hand, will say that virtually none of the policies, such as test-based teacher evaluations and merit pay, for which reformers are pushing were in force in more than a handful of locations between 2009 and 2011. Therefore, they’ll claim, the NAEP progress shows that the system is working without these changes.
If the NAEP results are not encouraging – i.e., overall progress is flat (or negative), and there are no strong gains among key subgroups – the market-based crowd will use the occasion to argue that the “status quo” isn’t producing results, and they will strengthen their call for policies like new evaluations and merit pay. Skeptics, in contrast, will claim that NCLB and standardized test-based accountability were failures from the get-go. Some will even use the NAEP results to advocate for the wholesale elimination of standardized testing. Read More »
The subject of Michelle Rhee’s teaching record has recently received a lot of attention. While the controversy has been interesting, it could also be argued that it’s relatively unimportant. The evidence that she exaggerated her teaching prowess is, after all, inconclusive (though highly suggestive). A little resume inflation from a job 20 years ago might be overlooked, so long as Rhee’s current claims about her more recent record are accurate. But are they?
On Rhee’s new website, her official bio - in effect, her resume today (or at least her cover letter) - contains a few sentences about her record as chancellor of D.C Public Schools (DCPS), under the header “Driving Unprecedented Growth in the D.C. Public Schools.” There, her test-based accomplishments are characterized as follows:
Under her leadership, the worst performing school district in the country became the only major city system to see double-digit growth in both their state reading and state math scores in seventh, eighth and tenth grades over three years.
This time, we can presume that the statement has been vetted thoroughly, using all the tools of data collection and analysis available to Rhee during her tenure at the helm of DCPS.
But the statement is false. Read More »
** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post.
Michelle Rhee’s resignation and departure have, predictably, provoked a flurry of conflicting reactions. Yet virtually all of them, from opponents and supporters alike, seem to assume that her tenure at the helm of the D.C. Public Schools (DCPS) helped to boost student test scores dramatically. She and D.C. Mayor Adrian Fenty made similar claims themselves in the Wall Street Journal (WSJ) just last week.
Hardly anybody, regardless of their opinion about Michelle Rhee, thinks that test scores alone are an adequate indicator of student success. But, in no small part because of her own emphasis on them, that is how this debate has unfolded. Her aim was to raise scores and, with few exceptions (also here and here), even those who objected to her “abrasive” style and controversial policies seem to believe that she succeeded wildly in the testing area.
This conclusion is premature. A review of the record shows that Michelle Rhee’s test score “legacy” is an open question.
There are three main points to consider: Read More »
In a previous post, I presented a simple tabulation of NAEP scores by whether or not states had binding teacher contracts. The averages indicate that states without such contracts (which are therefore free of many of the “ill effects” of teachers’ unions) are among the lowest performers in the nation on all four NAEP exams.
The post was largely a response to the constant comparisons of U.S. test scores with those of other nations (usually in the form of rankings), which make absolutely no reference to critical cross-national differences, most notably in terms of poverty/inequality (nor to the methodological issues surrounding test score comparisons). Using the same standard by which these comparisons show poor U.S. performance versus other nations, I “proved” that teacher contracts have a positive effect on states’ NAEP scores.
As I indicated at the end of that post, however, the picture is of course far more complicated. Dozens of factors – many of them unmeasurable – influence test scores, and simple averages mask them all. Still, given the fact that NAEP is arguably the best exam in the U.S. – and is the only one administered to a representative sample of all students across all states (without the selection bias of the SAT/ACT/AP) – it is worth revisiting this issue briefly, using tools that are a bit more sophisticated. If teachers’ contracts are to blame for low performance in the U.S., then when we control for core student characteristics, we should find that the contracts’ presence is associated with lower performance. Let’s take a quick look. Read More »
** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post.
Please check out the “sequel” to this post, which includes a multivariate analysis, and, more importantly, two other posts (here and here), which present summaries and discussions of the actual evidence on the relationship between unions and test scores.
For years, some people have been determined to blame teachers’ unions for all that ails public education in America. This issue has been around a long time (see here and here), but, given the tenor of the current debate, it seems to bear rehashing. According to this view, teachers unions negatively affect student achievement primarily through the mechanism of the collective bargaining agreement, or contract. These contracts are thought to include “harmful” provisions, such as seniority-based layoffs and unified salary schedules that give raises based on experience and education rather than performance.
But a fairly large proportion of public school teachers are not covered under legally-binding contracts. In fact, there are ten states in which there are no legally binding K-12 teacher contracts at all (AL, AZ, AR, GA, LA, MS, NC, SC, TX, and VA). Districts in a few of these states have entered into what are called “meet and confer” agreements about salary, benefits, and other working conditions, but administrators have the right to break these agreements at will. For all intents and purposes, these states are free of many of the alleged “negative union effects.”
Here’s a simple proposition: If teacher union contracts are the problem, then we should expect to see higher achievement outcomes in the ten states where there are no binding teacher contracts.
So, let’s take a quick look at how states with no contracts compare with the states that have them. Read More »
