Patterico's Pontifications


Advocacy polls are real polls

Filed under: General — Karl @ 10:35 am

[Posted by Karl]

Jonathan Chait disagrees. He’s wrong, as he is about many things.  But he’s wrong in ways worth discussing.

Chait’s target is the recent Third Way poll of independent voters, which he doesn’t like because “Third Way is an intra-party lobbying group that urges Democrats to adopt moderate, pro-business policies” and its poll tends to support its positions.  He notes that if you frame poll questions differently you can get findings like those from Greenberg Quinlan Rosner (taken for the Center for American Progress), such as 81% agreeing that “[r]egular people work harder and harder for less and less, while Wall Street CEOs enjoy bigger bonuses than ever.”

To be sure, the wording of poll questions matters.  However, Chait does not bother to quote the questions in the Third Way poll.  It appears he does not like questions such as:

I’m going to name some topics that have angered some people in America.  For each one, please tell me if it makes you very angry, somewhat angry, not too angry, or not angry at all.

The poll then asks about: Congressional gridlock; the national debt; Wall Street bailouts; America falling behind its global competitors; the wealthy not paying enough in taxes; the next generation’s ability to achieve the American Dream; corporate profits; and China’s economic rise.  Another question that seems to bother Chait is: “What do you think would be the most effective way to strengthen our economy?”, giving reducing the deficit, reducing taxes and regulations, and reducing income inequality as options.  Had Chait actually quoted the poll’s questions, it might have occurred to his readers that those questions sound much more neutral than the GQR questions he did quote.  Indeed, the basic Third Way findings on economic opportunity vs. economic fairness are not much different from those of Gallup and Pew.

However, the issue of question neutrality goes to a larger problem with Chait’s general concept of “real” public opinion polling:

Pollsters understand that very slight differences in the wording of a question, or even in the ordering of questions, can produce dramatically different results. Polls that are actually designed to measure public opinion take great precautions to avoid tilting answers one way or another. They try to frame questions in as neutral fashion as possible, and when they do ask questions that gauge people’s ideological views, they measure it by looking at changes.

So, for instance, a poll might ask if you prefer a larger government with more services, or a smaller government with fewer services. That is a classic polling question. It’s not an accurate snapshot of public opinion, though, because even though it’s posed in a completely neutral way, in frames the question in abstract terms rather than specific terms. Its value as a measuring tool is simply that polls as the same question in the same way every year, and the changes in response to the same question can help tell you how public opinion is changing.

This is a wildly reductive view of public opinion polling, and especially reductive of political polling.

The information gathered from the sorts of polling Chait describes is valuable — even if the polls generated for the establishment media and by entities like Gallup and Pew often fall short of the ideal.  However, it does not logically follow that “advocacy” polls are not “real” polls.  The issue is the quality of a given poll for its purpose.

For example, another poll from Greenberg Quinlan Rosner — this time for Democracy Corps — looks at several of the messages Pres. Obama and Democrats have been putting out and tests them against hypothetical GOP messages (which are debatable, but beyond my scope today).  It is not remotely neutral, but highly informative about public opinion for those actually conducting campaigns.  It is advocacy polling like this (which the White House or the DNC surely conducts internally) that explains why Obama is mostly attacking the GOP instead of leading with claims that America is back or has made progress on job creation.

Election campaigns are not waged solely in the editorial bullpens of the New York Times and Washington Post, or the offices of Gallup.  Rather, beyond the fundamentals of peace and prosperity, they are driven by candidates and their messages.  The candidates, their supporters and their messages are not neutral.

In short, to suggest that advocacy polling is not “real” is in some senses exactly backwards.  And to compare the recent Third Way poll to the Center for American Progress poll is laughable.  Indeed, Third Way’s “advocacy” here rests primarily on the general, neutral approach of its poll.


17 Responses to “Advocacy polls are real polls”

  1. Ding!

    Karl (6f7ecd)

  2. You have a way of making me see things I haven’t thought about before, or see things I have thought about but in a new way. Thanks, Karl.

    DRJ (a83b8b)

  3. DRJ,

    Aw. Thanks!

    Karl (6f7ecd)

  4. Thought-provoking. Rather than just dismiss the polls’ subjectivity and their programmed results, tease information from the effort.

    gary gulrud (d88477)

  5. Interesting post, Karl! I must say that I find it heartening that liberals usually word and ask their poll questions in a way that best positions them to receive the answers they want to hear.

    It makes the days that they are nutted by reality that much sweeter.

    Colonel Haiku (e961ad)

  6. Col., especially apropos given the nature of the Chait piece and the whole purpose of Third Way. Although Chait is biased by his dislike of the group, Third Way is an intra-party group of Democrats which was formed to identify where the Democratic Party was out of touch with the American people and to bring the party in line with the electorate so Democrats would be less likely to repeat 2004. Chait & Hardline Democrats, the American people have to change their attitudes to match ours vs. Third Way, the Democratic Party has to change their attitudes to match American people.

    max (131bc0)

  7. I took a course on polling as part of my Masters in medical outcomes research. The wording of the poll, no matter the topic, is all important. A couple of major considerations are 1. poor and less well educated people will tend to give the answer they think the pollster wants.

    2. If you want a really accurate survey, which is just a poll, you should asked the question in several different ways. Use slants of the questions tending to give a positive and a negative spin. See if the answers change.

    None of this happens in most political polls.

    Mike K (326cba)

  8. Racists.

    ps, at 200-plus comments I’m not even going to try to read the Charles Johnson/LGF thread; the only thing I have to say is… Wow, are there still people who pay attention to Charles Johnson? It’s as passe in its way as reading the NYT.

    Patterico, surely you have much better things to do with your time.

    d. in c. (cae88c)

  9. d. in c., would you like to tell Karl that he is wasting his time?

    Icy (70876f)

  10. Nah, not really. To be honest I don’t quite grok what this post is even really about (no offense to Karl, I’m just not focused on it). I just wanted to have a bit of snark at Chuckles and LGF, but I didn’t want to scroll through that monster thread.

    On a side note, I’m a bit puzzled by these monster sites like Hot Air or Ace that have hundreds or thousands of comments per thread. Who could possibly keep track of that, or make any sense of it, or have any feeling of a commenting community? At least sites like this and Althouse and Sailer have a sort of human scale, where you’re able to process the flow of the conversation…

    d. in c. (ac417f)

  11. Planned Parenthood did what were really push polls at least around 1986. They didn’t want to know what people thought – they wanted to argue.

    Sammy Finkelman (19c914)

  12. Racist.

    Why are you racists all being so racistly racist?

    d. in c. (ac417f)

  13. More polling discussion from CAC:

    Here focussing on voter ID as percentage of actual turnout.

    Per 7. and MK, political polling wallows in the ambiguity associated with their craft’s terminology.

    Of the eligible population Indies maintain a plurality.

    gary gulrud (d88477)

  14. Chaitred is foolish, but it’s to be expected, the
    slanting of the question, by GQR is a little more

    narciso (8d0f34)

  15. narciso (14),

    The slanting of questions by GQR isn’t really all that distressing. In the first example for CAP, it’s admittedly a bit foolish because — unless there is unreleased data — it’s so obviously slanted as to be ineffective as propaganda to anyone but the converted. In contrast, the Third Way poll is much more like a poll commissioned by the establishment media, and Third Way’s advocacy gains strength from that aspect (which is what really irks Chait).

    In the second GQR example, the entire point is to test various messages, as a business might do for a marketing campaign. Part of my point here is that such polling is not distressing, but an entirely legit form of public opinion polling. Campaigns do it, so it’s actually a plus to see some transparency in this area.

    Karl (6f7ecd)

  16. Well they are distressing, because the resuly is given as neutral, when it is a ‘push poll’

    narciso (8d0f34)

  17. narciso,

    GQR-type stuff usually doesn’t get the publicity of the Gallup/Pew/NYT/WaPo type stuff. Political junkies see it, but probably not persuadable voters. And any advocacy in the establishment polls comes from more subtle stuff — sampling, question order, etc.

    Karl (6f7ecd)

Powered by WordPress.

Page loaded in: 0.3004 secs.