Idea Filter: Evidence of a sensitive and easy to answer approach to idea and concept testing

Author: Andrew Grenville

1

Developing Idea Filter Our aim was to develop an approach to testing ideas that is:

• quick and easy to answer; • mobile friendly; • sensitive to differences between ideas; • and leads to the same conclusion as traditional methods.

We conducted a three arm study in which we asked two types of binary questions and compared those results to a traditional multipoint scale. See methodology section for specific questions used.

Comparing a 5-point scale to alternative binary measures We know that binary measures tend to be much faster to answer than a scaled questioni. We also know that they are easy to answer on a mobile device, so we sought to understand how binary measures compare to the more common scales typically used in testing ideasii.

We used five different concepts in the test and each person only evaluated a random three of the five. The concepts ranged from well-known products - M&M Peanut and Hershey’s Cookies ‘n’ crème - to products that exist elsewhere - Appletiser - to products not yet released - Mobile Wallet and Wireless Laptop.

The three types of questions returned different percentages for interest in buying, with selection returning lower numbers and yes/no higher numbers. The pattern, however, was the same for each measure—what was highest or lowest for each type of measure was identical. In fact, the rank order was identical for all five concepts.

37%45%

61%

13%

28%

45%53%

71%

26%36%

50%

63%

77%

32%

51%

0%10%20%30%40%50%60%70%80%90%

Appletiser Cookie's 'nCreme

M&M Peanut Mobile Wallet wFacial Login

Wireless Laptop

I'd buy it/purchase intent

Selection Top 2 Yes

2

The pattern was very similar for the other measures - liking, need, value, etc. - in terms of the relevant percent selecting an item. This is also consistent with what we have seen when comparing selection, yes/no and top 2 box in message filteringiii.

The rank order of the concepts on each measure was very similar, if not identical, across all measures. That means we would come to the same conclusions with each of the measures.

Sensitivity—measuring differences between messages and attributes One of the biggest challenges researchers face is being able to differentiate between ideas in terms of which one is the “winner”. We value a measure that discriminates between ideas and that also shows differences across the attributes.

So we measured the degree of difference between concepts and across attributes. In order to level the playing field across the measures, we looked at the differences between the highest and lowest scores, as a percentage of the highest score.

Selection showed more sensitivity on both the difference between messages and across attributes. Yes/no also appears to be somewhat more sensitive when comparing between ideas, compared to a top 2 box analysis. We can conclude that selection is more sensitive than the more typical top 2 box analysis of scaled data.

35%

48%

60%

0%10%20%30%40%50%60%70%

Selection Top 2 Yes

Average score across all measures

3

Time to answer We also measured how long it took respondents to complete idea filtering for the scaled and selection arms of the study. We used the median time to complete for the comparison, because sometimes people pause the completion of a survey part way through and that distorts the average—sometimes quite dramatically.

The selection method took less than a minute per idea—twice as fast as the scaled approach.

Conclusion Our results indicate it is possible to use a selection approach to idea screening that is quick and easy to answer, reliable and sensitive to differences between ideas and leads to the same conclusion as the traditional scaled approach. Given these results, and in light of how mobile friendly and quick to answer the question is, we recommend using a selection approach to idea filtering.

5:15

2:27

0:00

1:12

2:24

3:36

4:48

6:00

Scaled Questions Selection Question

Median time to complete concept questions for three concepts

4

Methodology We conducted three studies using sample from our Springboard America market community, which has been proven to be reliable.iv The samples were representative of the American population and surveys were in English. The sample sizes were: Scaled Arm 505; Selection Arm 503; Yes/No Arm 641.

Questions asked Selection Which of the following describe this idea? You can choose as many or as few as you feel describe it. I’d buy it I like it It’s new and different I need this type of product I want this type of product It offers good value It’s believable None of the above Yes/No Based on the product information you just saw; which statement below best describes how you feel about buying this product? Yes No I’d buy it I like it It’s new and different I need this type of product I want this type of product It offers good value It’s believable Scaled Based on the product information you just saw; which statement below best describes how you feel about buying this product? Definitely would buy Probably would buy Might or might not buy Probably would not buy Definitely would not buy

Which statement best describes how much you like or dislike this product?

Like extremely Like very well Like quite well Like somewhat Like slightly Not like at all Considering the price of the product, as given in the information, which statement below best describes how you feel about the value for the money of this product?

Very good value Fairly good value Average value Somewhat poor value Very poor value How would you rate this product in terms of being new and different from other products currently available?

Extremely new and different Very new and different Somewhat new and different Slightly new and different Not at all new and different How much do you agree or disagree with the following statement about this product? [ROWS][RANDOMIZE] I NEED this type of product I WANT this type of product [COLUMNS] Agree strongly Agree somewhat Disagree somewhat Disagree strongly Which statements listed below best describes how you feel about the believability of the statements made about this product?

Completely believable Somewhat believable Neither believable nor unbelievable Somewhat unbelievable Completely unbelievable

5

About Maru/Matchbox Maru/Matchbox, a Maru Group company, has been pushing the boundaries of the customer market insights space for over a decade. We combine deep sector expertise with cloud-based customer insights communities, and take a results-focused, consultative approach to helping clients better understand what motivates their customers and influences their markets, so they can act decisively and win.

i A Grenville, What Works Better, Scaled or Binary Brand Ratings? Vision Critical University ii A Grenville, Changing times demand rethinking old approaches: a case for quick, reliable, easy to answer questions. iii A Grenville, Message Filter: Evidence of a Sensitive and Easy to Answer Approach to Message and Claims and Testing. iv See for example R. Berger and A. Grenville, Wine, Cheese, Scotch and Sample: Know the Source and R. Berger and A. Grenville, Art or Science? The Perils and Possibilities of Survey Sampling in the Evolving Online World.