By Stephanie Vance
We here at Troubadour are fans of the DIY market research movement. We love the idea of companies (particularly those who might not otherwise have the budget for custom research) being empowered to conduct surveys and make data-driven decisions about their products and services. With that in mind, we want to be a resource to those companies as they navigate the (often complex) waters of survey design, data analysis, and interpretation.
Anytime you field a survey, you want to get the very best intelligence possible. No research is without cost, whether its purchasing sample or asking your customers to take time out of their busy schedule to answer your questions, so you want to make sure that money (or customer good will) is well spent. You want quality data, and quality data begins with quality questions. Research insights are only as good as the questions asked.
With that in mind, below I’ve identified 4 common mistakes I often see in DIY market research, as well as alternatives to those mistakes that will lead to more consistent, accurate, and meaningful insights.
Don’t #1: Beware the Double-Barreled Question
DIY Mistake #1:
|Not at all||Not very||Somewhat||Very||Extremely|
This question is problematic because a video can be entertaining and not very informative or vice versa. If a respondent indicates a 3 for “moderately” how do we interpret that? Is the video moderately entertaining, moderately informative, or both? Or maybe the respondent found the video not at all entertaining (1) but very informative (5), and just averaged his/her response (3). The reality is, you just can’t know the true answer. A better way to ask this question is to break out each descriptor into its own question or, better yet, consider adding even more descriptors and using a grid.
DIY Solution #1:
|Not well at all||Not very well||Somewhat well||Very well||Extremely well|
|Relevant to me||O||O||O||O||O|
Don’t #2: Botch multiple-selects!
DIY Mistake #2:
Don’t assume your respondents know you want them to select all the possible options that apply to them. Respondents often work through surveys at a fairly quick pace, and in order to ensure they’re providing accurate data, we need to be clear and explicit in our questioning.
When it comes to multiple selects, be explicit and specific!
DIY Solution #2:
(Select all that apply.)
|None of these||O|
Don’t #3: Ask Wobbly Behavior Frequency questions
DIY Mistake #3:
|Very Infrequently||Very Frequently|
When asking about behavior frequency, ask in concrete, objective terms and resist the urge to use subjective scales.
This type of scaled response for behavior frequency DOES NOT yield very useful data, as answers are completely subjective. What is very frequent to some people might be middle of the road or even infrequent to others. It can even be different for the same person, depending on when you ask him/her.
For instance, i’m a runner. Now, if you had asked me in April how often I run, I would have said 7 – Very frequently meaning I ran 3 days a week. Now, 6 months later, I’ve trained for and run (and finished) the Marine Corps Marathon in DC. So, If you’d asked me yesterday how often I run, I would have said 7 – Very Frequently, but that has a whole new (and much more extreme) meaning now. So instead of using scales, ask about behavior in concrete, objective terms!
My Marine Corps Marathon Finish!
DIY Solution #3:
|Never||Less often than once a month||About once a month||2-3 times a month||About once a week||2-3 times a week||4-5 times a week||Every Day|
Don’t #4: Fall into the Gaps
DIY Mistake #4a:
This is a classic mistake that can cause serious problems with data analysis and interpretation, particularly for questions about income or spending.
The problem in the above example is that many data points aren’t covered – what if you estimate that you spend about $25/month at the movie theater? What if you don’t go to the movies or spend more than $50?
DIY Mistake #4b:
|$50 or more||O|
What if you spend $10? Do you choose option 2 or option 3?
DIY Solution #4:
(including theater tickets, snacks, etc.)?
(I don’t typically go to the movies)
|More than $50||O|
Make sure you cover all the ground, but don’t cover anything twice. Include a zero option when appropriate, as well as an option for those who top out of your response options. In other words – Mind the gaps!
*Bonus tip: need help on the finished charts for these survey answers and insights? Check out our Chart Templates.
Key takeaways and pro tips:
- Keep things simple – the more basic the question, the more conclusive the answer.
- Keep it intuitive – make it easy for your respondents to answer with confidence.
- Be explicit – make sure you clearly state what information you want from respondents. Otherwise, you’re making them guess, and you can’t assume that everyone will guess correctly.
- Don’t always default to scales – scales definitely have their place, but if you’re asking about concrete behaviors, ask in concrete terms
- Mind the gaps! And cover all ground when creating questions regarding income, spending, or numbers of any kind.
Stephanie Vance is Vice President of Research at Troubadour Research and Consulting, a research and analytics consulting company that emphasizes the story over the data. She can be found on LinkedIn.