DIY Market Research: 4 Common Mistakes

DIY Market Research: 4 Common Mistakes

By Stephanie Vance
 
We here at Troubadour are fans of the DIY market research movement. We love the idea of companies (particularly those who might not otherwise have the budget for custom research) being empowered to conduct surveys and make data-driven decisions about their products and services. With that in mind, we want to be a resource to those companies as they navigate the (often complex) waters of survey design, data analysis, and interpretation.
 
Anytime you field a survey, you want to get the very best intelligence possible. No research is without cost, whether its purchasing sample or asking your customers to take time out of their busy schedule to answer your questions, so you want to make sure that money (or customer good will) is well spent. You want quality data, and quality data begins with quality questions. Research insights are only as good as the questions asked.
 
With that in mind, below I’ve identified 4 common mistakes I often see in DIY market research, as well as alternatives to those mistakes that will lead to more consistent, accurate, and meaningful insights.
 

Don’t #1: Beware the Double-Barreled Question

 

DIY Mistake #1:

 

How entertaining and informative did you find this video?
Keyboard Cat DIY Market Research tips

Not at all Not very Somewhat Very Extremely
O O O O O

 
This question is problematic because a video can be entertaining and not very informative or vice versa. If a respondent indicates a 3 for “moderately” how do we interpret that? Is the video moderately entertaining, moderately informative, or both? Or maybe the respondent found the video not at all entertaining (1) but very informative (5), and just averaged his/her response (3). The reality is, you just can’t know the true answer. A better way to ask this question is to break out each descriptor into its own question or, better yet, consider adding even more descriptors and using a grid.
 

DIY Solution #1:

 

How well do each of the following words or phrases describe the video you just watched?

  Not well at all Not very well Somewhat well Very well Extremely well
Entertaining O O O O O
Informative O O O O O
Relevant to me O O O O O
Inspirational O O O O O
Boring O O O O O

 
 

Don’t #2: Botch multiple-selects!

 

DIY Mistake #2:
What cereal have you purchased?

Cheerios
Lucky Charms
Cap’n Crunch
Cookie Crisp
Fruity Pebbles

 
Don’t assume your respondents know you want them to select all the possible options that apply to them. Respondents often work through surveys at a fairly quick pace, and in order to ensure they’re providing accurate data, we need to be clear and explicit in our questioning.
 
When it comes to multiple selects, be explicit and specific!
 

DIY Solution #2:

 

Which of the following cereals have you purchased at least once in the past year?
(Select all that apply.)

Cheerios
Lucky Charms
Cap’n Crunch
Cookie Crisp
Fruity Pebbles
None of these O

 
 

Don’t #3: Ask Wobbly Behavior Frequency questions

 

DIY Mistake #3:

 

On a scale of 1 “Very infrequently” to 10 “Very frequently,” about how often do you run for exercise?

Very Infrequently                 Very Frequently
O O O O O O O O O O

 
When asking about behavior frequency, ask in concrete, objective terms and resist the urge to use subjective scales.
 
This type of scaled response for behavior frequency DOES NOT yield very useful data, as answers are completely subjective. What is very frequent to some people might be middle of the road or even infrequent to others. It can even be different for the same person, depending on when you ask him/her.
 
For instance, i’m a runner. Now, if you had asked me in April how often I run, I would have said 7 – Very frequently meaning I ran 3 days a week. Now, 6 months later, I’ve trained for and run (and finished) the Marine Corps Marathon in DC. So, If you’d asked me yesterday how often I run, I would have said 7 – Very Frequently, but that has a whole new (and much more extreme) meaning now. So instead of using scales, ask about behavior in concrete, objective terms!
 

VP Research, DIY Market Research

My Marine Corps Marathon Finish!

 

DIY Solution #3:

 

About how frequently do you run for exercise?

Never Less often than once a month About once a month 2-3 times a month About once a week 2-3 times a week 4-5 times a week Every Day
O O O O O O O O

 
 

Don’t #4: Fall into the Gaps

 

DIY Mistake #4a:

 

How much do you spend at the movies each month?

$1-$10 O
$20-$30 O
$40-$50 O

 
This is a classic mistake that can cause serious problems with data analysis and interpretation, particularly for questions about income or spending.
 
The problem in the above example is that many data points aren’t covered – what if you estimate that you spend about $25/month at the movie theater? What if you don’t go to the movies or spend more than $50?
 

DIY Mistake #4b:

 

How much do you spend at the movies each month?

Zero O
$0-$10 O
$10-$20 O
$20-$30 O
$30-$40 O
$40-$50 O
$50 or more O

 
What if you spend $10? Do you choose option 2 or option 3?
 

DIY Solution #4:

 

In a typical month, about how much money do you spend at the movie theater
(including theater tickets, snacks, etc.)?

Zero
(I don’t typically go to the movies)
O
$1-$10 O
$11-$20 O
$21-$30 O
$31-$40 O
$41-$50 O
More than $50 O

 
Make sure you cover all the ground, but don’t cover anything twice. Include a zero option when appropriate, as well as an option for those who top out of your response options. In other words – Mind the gaps!
 
*Bonus tip: need help on the finished charts for these survey answers and insights? Check out our Chart Templates.
 

Key takeaways and pro tips:

  • Keep things simple – the more basic the question, the more conclusive the answer.
  • Keep it intuitive – make it easy for your respondents to answer with confidence.
  • Be explicit – make sure you clearly state what information you want from respondents. Otherwise, you’re making them guess, and you can’t assume that everyone will guess correctly.
  • Don’t always default to scales – scales definitely have their place, but if you’re asking about concrete behaviors, ask in concrete terms
  • Mind the gaps! And cover all ground when creating questions regarding income, spending, or numbers of any kind.

 

DIY Market Research Contact Us

Stephanie Vance Troubadour Research and Consulting
Stephanie Vance is Vice President of Research at Troubadour Research and Consulting, a research and analytics consulting company that emphasizes the story over the data. She can be found on LinkedIn.

By |2015-02-17T19:48:44+00:00November 12th, 2014|Research Best Practices|0 Comments

About the Author:

Leave A Comment