How you phrase a question doesn’t matter. Here’s what does.
A question I get a lot is…
"How do I know we're not biasing the respondent?"
But something I’ve unexpectedly discovered is that tweaking a question to be more “neutral” or precise almost certainly won’t impact your results.
Just last week, I tested three ways to ask respondents about their perception of youth sports coaches*, and the results were statistically identical across all versions:
v1: In a conversation, someone mentions they coach youth sports. What's your reaction to this person?
Avg: 68.64
v2: In a conversation, someone mentions they coach youth sports. What's your honest reaction to this person?
Avg: 77.7
v3: When someone mentions they "coach youth sports" what's your reaction?
Avg: 75.4
So what actually impacts your results?
When I asked a second group of respondents, "How do you feel about coaching youth sports?" I found an interesting contrast: while people viewed coaches positively, they had more negative feelings about coaching themselves.
On its own, this gap between admiration and action isn't groundbreaking.
But maybe talking about what we admire in others yet avoid ourselves is exactly the story your client wants to hear.
Maybe it isn’t.
The point is that survey design isn't about perfecting a single question, but using different questions to shape a narrative.
How many survey platforms have helped you do that?
When was the last time a vendor talked about storytelling instead of sample size?
The quant world is filled with perfectionists trapped in methodology – you need someone who turns data into stories that move people.
Want help designing surveys from someone who can do that?
Consider booking a Survey Roast.
• 15-minute personalized Loom video
• Copy-and-paste edits ready to implement
• Expert suggestions to uncover better stories through your survey
• All for $145
Cheers,
Sam
*I even changed the scales from “Positive” “Negative” “Neutral” to “I find myself genuinely interested” “I quietly look for an exit from the conversation” and “I have no particular reaction.”