Wait, you *wouldn’t* have sex with Isaac Newton?
Something to ponder:
There’s an inconvenient truth about survey questions that I’ve discovered from years of trial and error.
They’re surprisingly stubborn.
You can twist their wording, even transform them into seemingly different questions—yet respondents will often give you the same answers.
Earlier in my career, I would have vehemently denied this.
I assumed that tweaking a question would impact the results, as revealed by some nifty a/b test.
(this was back when I was watching too many Ted Talks, and actually believing that small changes – nudges, hacks, power poses, etc. – had these huge effects)
But survey after survey showed the same thing: word changes barely moved the needle.
At first I was paranoid.
If I didn’t have “counterintuitive” results, would I still be valuable to colleagues and clients?
But then, I read something that fundamentally changed my approach: George Gallup’s The Pulse of Democracy.
I distinctly remember the chapter (“Behind the Scenes”) and section (“Wording the questions”) that knocked some much-needed sense into my dumb brain: "The wording of the question is of relatively minor importance.”
Page 102.
Go look it up.
His point wasn't that phrasing never matters—sometimes it does—but that each question must be clear and relevant.
If you’re confusing people—or asking about something they simply don't care about—then debating this word or that is a complete waste…
akin to grammar purists debating the oxford comma in a sentence that lacks clarity, meaning, and purpose.
I took Gallup’s point seriously and stopped chasing “counterintuitive” results.
My surveys yielded more value.
Clients noticed.
I had mostly forgotten about this “revelation” (which just amounted to wising up, ignoring trends, and learning from history) until a few weeks ago.
I was scanning X when I came across something that sounded eerily similar to page 102: a tweet from Aella, a former cam girl turned data-driven researcher with over 100,000 subscribers to her Substack, Knowingless.
Her tweet: "Questions, in general, are weirdly resilient to surveys. People answer differently phrased questions the same.”
Keep in mind Aella surveys her subscribers about kinks, phobias, if they love “being railed,” if they’d have sex with Isaac Newton.
Exactly the kinds of topics you’d expect wording to dramatically affect responses.
Yet she's found exactly what Gallup was describing: change the words all you want, people's answers usually won’t budge.
Being clear and relevant is pound-for-pound the best advice I’ve got for you – and the foundation of every suggestion I provide in my Survey Roasts.
And I dunno if it’s the thought of banging Isaac Newton or what, but I’m feeling generous today.
Enter the discount code “IdbangNewton” and get 20% off a $145 Survey Roast. (The code is good through March 31st).
I won't waste your time with methodological nitpicking or semantic lectures (except, of course, in instances where that stuff really does make a difference).
Instead, I'll focus on ensuring your questions are… clear and relevant.
When you get those things right, you’ll get the insights you need.
So consider booking a Survey Roast. Send me your survey draft, and for $145, I'll make a 15-minute Loom video with copy-and-paste edits and suggestions to improve your survey data quality.
Cheers,
Sam