Unless you’re asking about something people care about, avoid “Agree or Disagree” questions

In a recent Roast, I noticed a client had drafted an “agree-disagree” question.

The respondents were real-time shoppers, incentivized to find a product in a grocery store, submit a picture of it, and answer questions via a mobile app.

After locating the client’s product, they’d first read a positioning statement and then answer, “How strongly do you agree with this statement?”

(If the statement was written for Tostitos Chips, it might read, “Where every crunch connects friends and celebrates authentic moments.”)

I pointed out that agree-disagree questions typically measure topics people care about, such as the statement, “The best years of my life are ahead of me.”

However, since apathy is the norm for most products, there’s usually nothing substantial enough for people to agree or disagree with.

As George Gallup said, “It’s difficult to elicit attitudes [that] have not had the chance to form.”

My recommendation?

Ask respondents about the degree to which their initial impression of the product fits the positioning statement.

That works because it probes an existing relationship: the relationship between the shopper’s impression and the positioning statement.

It also leverages the context of the survey better. Because respondents must submit an on-the-spot photo of the product, an impression of the product will have formed, making its measurement possible.

A good survey question usually measures something that exists.

You can design a methodologically flawless survey — and even draw praise from the purists and methodology nerds — but if it asks about topics that mean nothing to someone, so what?

And while I realize how metaphysically loaded the phrase “measure something that exists” might sound, trust me that I’m sharing a very practical rule of thumb.

Actually, I’m just paraphrasing Gallup, who in 1940 told pollsters to avoid questions that “seem to require consultation with a crystal ball or a bearded prophet.”

Follow Gallup’s wisdom and your best surveys will be ahead of you.