On Education Polls And Confirmation Bias

Posted by on September 5, 2013

Our guest author today is Morgan Polikoff, Assistant Professor in the Rossier School of Education at the University of Southern California. 

A few weeks back, education policy wonks were hit with a set of opinion polls about education policy. The two most divergent of these polls were the Phi Delta Kappan/Gallup poll and the Associated Press/NORC poll.

This week a California poll conducted by Policy Analysis for California Education (PACE) and the USC Rossier School of Education (where I am an assistant professor) was released. The PACE/USC Rossier poll addresses many of the same issues as those from the PDK and AP, and I believe the three polls together can provide some valuable lessons about the education reform debate, the interpretation of poll results, and the state of popular opinion about key policy issues.

In general, the results as a whole indicate that parents and the public hold rather nuanced views on testing and evaluation. Read More »


No Presentation Without Representation

Posted by on June 26, 2013

I tend to comment on newly-released teacher surveys, primarily because I think the surveys are important and interesting, but also because teachers’ opinions are sometimes misrepresented in our debate about education reform. So, last year, I wrote about a report by the advocacy organization Teach Plus, in which they presented results from a survey focused on identifying differences in attitudes by teacher experience (an important topic). One of my major comments was that the survey was “non-scientific” – it was voluntary, and distributed via social media, e-mail, etc. This means that the results cannot be used to draw strong conclusions about the population of teachers as a whole, since those who responded might be different from those that did not.

I also noted that, even if the sample was not representative, this did not preclude finding useful information in the results. That is, my primary criticism was that the authors did not even mention the issue, or make an effort to compare the characteristics of their survey respondents with those of teachers in general (which can give a sense of the differences between the sample and the population).

Well, they have just issued a new report, which also presents the results of a teacher survey, this time focused on teachers’ attitudes toward the evaluation system used in Memphis, Tennessee (called the “Teacher Effectiveness Measure,” or TEM). In this case, not only do they raise the issue of representativeness, but they also present a little bit of data comparing their respondents to the population (i.e., all Memphis teachers who were evaluated under TEM). Read More »


Teacher Surveys And Standardized Tests: Different Data, Similar Warning Labels

Posted by on February 28, 2013

As a strong believer in paying attention to what teachers think about policy, I always review the results of MetLife’s annual teacher survey. The big theme of this year’s survey, as pushed by the press release and reiterated in most of the media coverage, was that job satisfaction among teachers is at “its lowest level in 25 years.”

It turns out that changes in question wording over the years complicates straight comparisons of responses to the teacher job satisfaction over time. Even slight changes in wording can affect results, though it seems implausible that this one had a dramatic effect. In any case, it is instructive to take a look at the reactions to this finding. If I may generalize a bit here, one “camp” argued that the decline in teacher satisfaction is due to recent policy changes, such as eroding job protections, new evaluations, and the upcoming implementation of the Common Core. Another “camp” urged caution – they pointed out that not only is job satisfaction still rather high, but also that the decline among teachers can be found among many other groups of workers too, likely a result of the ongoing recession.

Although it is more than plausible that recent reforms are taking a toll on teacher morale, and this possibility merits attention, those urging caution, in my view, are correct. It’s simply not appropriate to draw strong conclusions as to what is causing this (or any other) trend in aggregate teacher attitudes, and it’s even more questionable to chalk it up to a reaction against specific policies, particularly during a time of economic hardship. Read More »


Are Teachers Changing Their Minds About Education Reform?

Posted by on December 14, 2012

** Reprinted here in the Washington Post

In a recent Washington Post article called “Teachers leaning in favor of reforms,” veteran reporter Jay Mathews puts forth an argument that one hears rather frequently – that teachers are “changing their minds,” in a favorable direction, about the current wave of education reform. Among other things, Mr. Mathews cites two teacher surveys. One of them, which we discussed here, is a single-year survey that doesn’t actually look at trends, and therefore cannot tell us much about shifts in teachers’ attitudes over time (it was also a voluntary online survey).

His second source, on the other hand, is in fact a useful means of (cautiously) assessing such trends (though the article doesn’t actually look at them). That is the Education Sector survey of a nationally-representative sample of U.S. teachers, which they conducted in 2003, 2007 and, most recently, in 2011.

This is a valuable resource. Like other teacher surveys, it shows that educators’ attitudes toward education policy are diverse. Opinions vary by teacher characteristics, context and, of course, by the policy being queried. Moreover, views among teachers can (and do) change over time, though, when looking at cross-sectional surveys, one must always keep in mind that observed changes (or lack thereof) might be due in part to shifts in the characteristics of the teacher workforce. There’s an important distinction between changing minds and changing workers (which Jay Mathews, to his great credit, discusses in this article).*

That said, when it comes to the many of the more controversial reforms happening in the U.S., those about which teachers might be “changing their minds,” the results of this particular survey suggest, if anything, that teachers’ attitudes are actually quite stable. Read More »


Surveying The Teacher Opinion Landscape

Posted by on October 30, 2012

I’m a big fan of surveys of teachers’ opinions of education policy, not only because of educators’ valuable policy-relevant knowledge, but also because their views are sometimes misrepresented or disregarded in our public discourse.

For instance, the diverse set of ideas that might be loosely characterized as “market-based reform” faces a bit of tension when it comes to teacher support. Without question, some teachers support the more controversial market-based policy ideas, such as pay and evaluations based substantially on test scores, but most do not. The relatively low levels of teacher endorsement don’t necessarily mean these ideas are “bad,” and much of the disagreement is less about the desirability of general policies (e.g., new teacher evaluations) than the specifics (e.g., the measures that comprise those evaluations). In any case, it’s a somewhat awkward juxtaposition: A focus on “respecting and elevating the teaching profession” by means of policies that most teachers do not like.

Sometimes (albeit too infrequently) this tension is discussed meaningfully, other times it is obscured – e.g., by attempts to portray teachers’ disagreement as “union opposition.” But, as mentioned above, teachers are not a monolith and their opinions can and do change (see here). This is, in my view, a situation always worth monitoring, so I thought I’d take a look at a recent report from the organization Teach Plus, which presents data from a survey that they collected themselves.

Read More »


Who Has Confidence In U.S. Schools?

Posted by on October 24, 2012

For many years, national survey and polling data have shown that Americans tend to like their own local schools, but are considerably less sanguine about the nation’s education system as a whole. This somewhat paradoxical finding – in which most people seem to think the problem is with “other people’s schools” – is difficult to interpret, especially since it seems to vary a bit when people are given basic information about schools, such as funding levels.

In any case, I couldn’t resist taking a very quick, superficial look at how people’s views of education vary by important characteristics, such as age and education. I used the General Social Survey (pooled 2006-2010), which queries respondents about their confidence in education, asking them to specify whether they have “hardly any,” “only some” or “a great deal” of confidence in the system.*

This question doesn’t differentiate explicitly between respondents’ local schools and the system as a whole, and respondents may consider different factors when assessing their confidence, but I think it’s a decent measure of their disposition toward the education system. Read More »


A Big Open Question: Do Value-Added Estimates Match Up With Teachers’ Opinions Of Their Colleagues?

Posted by on February 9, 2012

A recent article about the implementation of new teacher evaluations in Tennessee details some of the complicated issues with which state officials, teachers and administrators are dealing in adapting to the new system. One of these issues is somewhat technical – whether the various components of evaluations, most notably principal observations and test-based productivity measures (e.g., value-added) – tend to “match up.” That is, whether teachers who score high on one measure tend to do similarly well on the other (see here for more on this issue).

In discussing this type of validation exercise, the article notes:

If they don’t match up, the system’s usefulness and reliability could come into question, and it could lose credibility among educators.

Value-added and other test-based measures of teacher productivity may have a credibility problem among many (but definitely not all) teachers, but I don’t think it’s due to – or can be helped much by – whether or not these estimates match up with observations or other measures being incorporated into states’ new systems. I’m all for this type of research (see here and here), but I’ve never seen what I think would be an extremely useful study for addressing the credibility issue among teachers: One that looked at the relationship between value-added estimates and teachers’ opinions of each other. Read More »


Why Do Most Americans Support “Assistance To The Poor” But Oppose “Welfare”?

Posted by on November 17, 2011

Politicians and other public figures spend a great deal of resources – time and money – on crafting their messages so as to elicit a desired response. A famous example is the effort to relabel the estate tax as the death tax – the former conjures images of very wealthy people paying their fair share, whereas the latter obscures this limited applicability, and invokes outrage at being “taxed just for dying.”

As everyone knows, words matter, and these efforts pay off. You don’t need to look at the results of too many surveys or polls to realize that people respond very differently depending on what you call something or how you describe it (e.g., see this post on attitudes toward teacher tenure).

One other particularly interesting – and important – example of this description-based divergence of attitudes toward social programs for the poor. Read More »


Americans Do NOT Want To Cut Government Programs

Posted by on November 9, 2011

Conservatives sometimes assert and often imply that Americans want to cut government spending on social assistance and other programs. This is a myth.

In fact, when it comes to the types of programs that get most of the attention in our national debate, almost nobody supports spending reductions and, in many cases, there is strong support for increases.

Take a look at the figure below, which presents General Social Survey data for 2010. Each bar presents the distribution of responses to questions of whether the U.S. spends too much (red), about the right amount (yellow) or too little (green) on several different types of programs and public resources. Read More »


Do Americans Think Government Should Reduce Income Inequality?

Posted by on October 24, 2011

With all the recent coverage of Occupy Wall Street and President Obama’s jobs bill, we’ve heard a lot of polling results showing that a large plurality of Americans supports raising taxes on high earners, and that this support is strong among both Democrats and Republicans.

The campaign to raise taxes on high-income households is part of a larger ideological notion that reducing inequality by such means as taxation and welfare programs is a proper function of government. Supporters (e.g., Democrats) argue that progressive taxation helps to ensure that high earners pay their “fair share” in supporting the public resources, such as schools, roads and law enforcement, that are necessary (but not sufficient) for their success. Republicans, on the other hand, tend frame the issue directly in terms of government intrusion – the government is unfairly “picking winners and losers,” and stifling innovation and risk-taking. The assumption seems to be that many Americans don’t care for the generic idea of government taking an active role in reducing the gap between rich and poor, even though they tend to support many of the specific means by which this occurs, including not only raising taxes on high earners, but also public education and programs like Medicaid.

So, it might be interesting to see what Americans think of the broader idea that government has a legitimate role in reducing income inequality. Let’s take a quick look. Read More »


Character Education

Posted by on October 4, 2011

I’m always uncomfortable with personal accusations in our education debate, and they come from both “sides.” For instance, I don’t like hearing accusations that market-based reformers are “profiteers.” The implication is that these people seek to dismantle or otherwise alter the public education system for their own economic advantage.

It’s true that a significant proportion of market-based reformers support various forms of privatization, such as vouchers, and that this support is in part based on the power of competition and the profit motive to increase efficiency. It’s also true that there are some who stand to profit personally off certain policy changes. But the overwhelming majority of people on the “reform side” have no financial skin in the game, and even those who do might actually still care about education and children. You can and should disagree with them, if you’re so inclined, but accusing them of being motivated solely by personal financial gain, or even implying as much, could well be unfair, but, more importantly, it contributes nothing of substance to the debate.

On the flip side of that coin, however, is the endlessly-repeated “we care about children, not adults” narrative. This little nugget is a common message from the market-based reform crowd. Most recently, Ben Austin, head of a pro-charter school group, was on a panel at NBC’s Education Nation, and repeated the talking point several times. In fact, there’s now a small confederation of advocacy groups nominally based on the “children over adults” accusation – Students First, Stand for Children, etc. Read More »


Serious Misconduct

Posted by on September 15, 2011

A recent Monmouth University poll of New Jersey residents is being widely touted by Governor Chris Christie and his supporters as evidence that people support his education reform plans. It’s hardly unusual for politicians to ignore the limitations of polling, but I’d urge caution in interpreting these results as a mandate.

Others have commented on how some of the questions are worded in a manner that could skew responses. These wording issues are inherent to polling, and that’s one of the major reasons why they must be interpreted carefully. But one of the questions caught my eye – the question about teacher tenure – and it’s worth quickly discussing. Read More »


What Americans Think About Teachers Versus What They’re Hearing

Posted by on August 25, 2011

The results from the recent Gallup/PDK education survey found that 71 percent of surveyed Americans “have trust and confidence in the men and women who are teaching children in public schools.” Although this finding received a fair amount of media attention, it is not at all surprising. Polls have long indicated that teachers are among the most trusted professions in the U.S., up there with doctors, nurses and firefighters.

(Side note: The teaching profession also ranks among the most prestigious U.S. occupations – in both analyses of survey data as well as in polls [though see here for an argument that occupational prestige scores are obsolete].)

What was rather surprising, on the other hand, was the Gallup/PDK results for the question about what people are hearing about teachers in the news media. Respondents were asked, “Generally speaking, do you hear more good stories or bad stories about teachers in the news media?”

Over two-thirds (68 percent) said they heard more bad stories than good ones. A little over a quarter (28 percent) said the opposite. Read More »


A List Of Education And Related Data Resources

Posted by on March 1, 2011

We frequently present quick analyses of data on this blog (and look at those done by others). As a close follower of the education debate, I often get the sense that people are hungry for high-quality information on a variety of different topics, but searching for these data can be daunting, which probably deters many people from trying.

So, while I’m sure that many others have compiled lists of data resources relevant to education, I figured I would do the same, with a focus on more user-friendly sources.

But first, I would be remiss if I didn’t caution you to use these data carefully. Almost all of the resources below have instructions or FAQ’s, most non-technical. Read them. Remember that improper or misleading presentation of data is one of the most counterproductive features of today’s education debates, and it occurs to the detriment of all.

That said, here are a few key resources for education and other related quantitative data. It is far from exhaustive, so feel free to leave comments and suggestions if you think I missed anything important. Read More »


Extra Time: More From The Magazine’s Education Poll

Posted by on September 14, 2010

** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post.

A recent education poll conducted by Time Magazine has gotten a lot of attention. Many of the questions are worded so badly that the results are rather meaningless. The question on merit pay, for example, defines the practice as “paying teachers according to their effectiveness” (who would oppose that, if it could be accurately measured?). Other questions are very interesting, such as the one asking whether respondents would pay higher taxes to improve public schools (56 percent would). Or the finding that, when asked what will “improve student achievement the most,” more than twice as many people choose “more involved parents” (54 percent) over “more effective teachers” (24 percent).

But, as is sometimes the case, a few of the survey’s most interesting results were not included in the published article, which highlighted only 11 out of 40-50 or so total questions (the full set of results is available here). Here are three or four unpublished items that caught my eye (the sample size is 1,000, with a margin of error of +/- 3 percent): Read More »


Disclaimer

This web site and the information contained herein are provided as a service to those who are interested in the work of the Albert Shanker Institute (ASI). ASI makes no warranties, either express or implied, concerning the information contained on or linked from shankerblog.org. The visitor uses the information provided herein at his/her own risk. ASI, its officers, board members, agents, and employees specifically disclaim any and all liability from damages which may result from the utilization of the information provided herein. The content in the shankerblog.org may not necessarily reflect the views or official policy positions of ASI or any related entity or organization.

Banner image adapted from 1975 photograph by Jennie Shanker, daughter of Albert Shanker.