Policy And Research: A Shotgun Wedding In New Jersey

Posted by on January 18, 2011

Earlier today, New Jersey Governor Chris Christie announced his plan to open 23 new charter schools in his state.  Just hours before this announcement, the NJ education department issued an analysis of new data on the performance of charter schools in the state (during the 2009-10 school year).   In an accompanying press release, the department claims that “the data affirms [sic] the need for Governor Christie’s reform proposals to grow the number of high-quality charter schools…” 

The release also contains several other extremely bold assertions that the results support expanding the state’s charter sector.  The title of the actual report, which contains only tables, is: “Living Up to Expectations: Charter Schools in New Jersey Outperforming District Schools.”

Unfortunately, however, the analysis could barely pass muster if submitted by a student in one of the state’s high school math classes (charter or regular public).

It seems that the department compared the overall proficiency rates of each of the state’s 70 or so charter schools with the rates of the district in which they are located, as well as with the overall state average (for the record, the tables and press release incorrectly call the rates “scores,” though it is an important distinction).  They then tallied up the results to see how many did “better” and how many did “worse.”  The overall conclusion: 79 percent of charters outperformed their “host” districts on language arts exams, and 69 percent did so in math.

There are so many monumental limitations of this analysis – not the least of which being that it makes absolutely no effort, even crudely, to control for student characteristics – that they’re really not worth enumerating.  Suffice it to say that the results, by even the most generous policy research standards, demonstrate very little about charter schools’ relative performance (versus comparable regular public schools in their areas).  New Jersey charters may have done better this year, or they may have done worse, but these results, as presented, cannot be used to judge it either way. The bold, sweeping conclusions in the press release are, at best, misleading and, at worst, absurd.

Look – I’m all about people and organizations, including government departments, producing their own analyses, and states’ education departments cannot be held to the same standard as universities or large research organizations.  So, needless to say, I have absolutely no problem with the fact that this analysis is purely descriptive. Quite the contrary – by themselves, the tables represent exactly the kind of information that these departments should be producing. Even the simplest comparisons can be very useful if interpreted correctly.

But drawing grand, completely unsupportable conclusions from the results, and then using these conclusions to justify policy decisions scheduled to be announced later that day, is alamaringly poor research practice. Moreover, regardless of your opinion about the governor’s decision to expand charter schools, this type of misleading, coordinated roll-out is unbecoming of an education department, and it does not serve the people of New Jersey well.


4 Comments posted so far

  • Finally, a critical analysis of the data. Everything in this article is so accurate. It is very unnerving to me that the public just so readily accepts the headlines and “bold assertions” rather than asking the tough questions that this article poses.

    Comment by Linda Martensen
    January 19, 2011 at 4:19 PM
  • [...] problematic, and well, quite honestly, pretty darn meaningless. I could not have said it better than Matt DiCarlo of Shanker Blog did here: “Unfortunately, however, the analysis could barely pass muster if submitted by a [...]

    January 19, 2011 at 6:48 PM
  • [...] ED! @GovChristie no, you did not. Here is a nice blog explaining just how flimsy that release was: http://shankerblog.org/?p=1646 5 hours [...]

    January 20, 2011 at 2:36 PM
  • [...] this month were his overview of the “Year in Research on Market-Based Reform” and this week’s commentary on the Christie administration’s recent claims that the New Jersey’s charters are doing a better job than their district schools, which the [...]

    March 1, 2012 at 2:55 PM

Sorry, the comment form is closed at this time.

Disclaimer

This web site and the information contained herein are provided as a service to those who are interested in the work of the Albert Shanker Institute (ASI). ASI makes no warranties, either express or implied, concerning the information contained on or linked from shankerblog.org. The visitor uses the information provided herein at his/her own risk. ASI, its officers, board members, agents, and employees specifically disclaim any and all liability from damages which may result from the utilization of the information provided herein. The content in the shankerblog.org may not necessarily reflect the views or official policy positions of ASI or any related entity or organization.

Banner image adapted from 1975 photograph by Jennie Shanker, daughter of Albert Shanker.