Survey hack for better pitch insights
If you’re an agency strategist you’ve probably fielded a survey for a pitch.
You fired up Survey Monkey or 1Q, asked questions about awareness or usage, hoping to find a compelling insight.
Want a methodology that delivers every time?
Split your respondents into two groups. Ask one group about doing something and the other about hearing someone talk about doing that same thing.
You’re pitting personal judgment against social judgment, and you’ll find something interesting regardless of the result.
Here’s an example prompted by a question that’s been nagging me ever since two former colleagues accused me of being “Basic.”
What do people actually think about Live, Love, Laugh' signs – the kind you’d hang in your kitchen?
To find out, I recruited 296 Americans to a survey (balanced for age, gender, and ethnicity) and randomly split them into the two aforementioned groups. One shared how they'd feel about putting a Live, Love, Laugh sign in their own kitchen. The other told me how they’d react to someone mentioning that they have one in their kitchen. Respondents used a slider labeled "Positive," "Neutral," and "Negative."
Here are the questions as they appeared in the survey (which you can take here).
Group 1: How do you feel about putting a "Live, Love, Laugh" sign in your kitchen?
Group 2: In a conversation, someone mentions that they have a “Live, Love, Laugh” sign in their kitchen. What’s your honest reaction to this person?
The result?
I found that people were significantly* more negative about the idea of hanging the sign themselves compared to hearing about others who have one. We're harsher critics of our own potential kitchen decor choices than we are of others'.
This methodology works for any behavior or choice.
I tested it across several examples: Live, Love, Laugh signs, watching reality TV, coaching youth sports, and taking a break from social media.
The patterns varied across these cases.
Sometimes people are harder on themselves than others (like with Live, Love, Laugh signs and watching reality TV). Sometimes they view others more favorably (as with coaching youth sports – which you can see visualized below). And sometimes there's no gap at all (like with "taking a break from social media," where personal and social judgments were equally positive).
Any pattern gives you something interesting to work with. Whether people are hypocritical, self-critical, or surprisingly consistent, these question-pairs will uncover a story worth telling.
*Note on significance
I ran t-tests to determine if the differences between groups were statistically significant (i.e., unlikely to have occurred by chance). For stats nerds: these were two-tailed, homoscedastic t-tests with sample sizes of 156 and 143. The Live, Love, Laugh comparison yielded a p-value of 1.51e-07, meaning there's less than a 0.0001% chance this difference occurred randomly.
Two other comparisons were also significant: youth sports (p=0.0002) and reality TV (p=0.02). I did not find a significant difference within the “taking a break from social media” group.
Want to run your own t-test? If you're using Excel, use the T.TEST function. For Google Sheets, it's TTEST. For survey data using rating scales (like 1-5 or -50 to +50), use "2" for two-tailed (because you're not predicting which group will be higher) and "2" for homoscedastic (because both groups are using the same scale with similar variance).
Resources:
Data Visualization One and Two (re-use and edit this template with a free DataWrapper account)
The Survey I used to collect data
Survey Questions in a Doc + more question-pairs I experimented with.
Survey Roasts
Speaking of designing surveys that elicit valuable insights — consider booking a Survey Roast.
Send me your survey draft, and for $145, I'll make a 15-minute Loom video with copy-and-paste edits and suggestions to improve your survey data quality.
Cheers,
Sam