Jeffrey henning main stage - 2011

Main Stage, November 3, 2011

Crowd-Shaped Surveys: Adapting the Experience Based on Prior Respondents Jeffrey Henning – Affinnova

A Presenta*on from the Fes*val of NewMR Main Stage – November 3, 2011

Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT)

Crowd-Shaped Surveys: Adapting the Experience Based on

Prior Respondents Jeffrey Henning

Chief Marketing Officer


THE SPIRIT OF THE WEB


Metal Umlaut – 15 Apr 03






Metal Umlaut – 17 May 03


Metal Umlaut – 25 May 03


Metal Umlaut – 24 Jun 04


Metal Umlaut – 4 Dec 08


Metal Umlaut – 12 Oct 11


NOW IMAGINE THIS AS AN ONLINE SURVEY


Define “Metal Umlaut”

•  The "heavy metal umlaut" is over the ö in the names of Motörhead, and the Blue Öyster Cult



•  The "heavy metal umlaut" is over the ö and ü in the names of Motörhead, Mötley Crüe



•  The heavy metal umlaut was ridiculed by the spoof band Spinal Tap by putting the umlaut over the letter 'n'



•  The heavy metal umlaut can be seen in the term "nü-metal” to describe nu-metal with added umlaut-ness.



•  The heavy metal umlaut is in the name of the band Assück.



•  The heavy metal umlaut is in the name of the band Assück.



•  The use of an umlaut over some of the letters in the names of heavy metal bands like Queensrÿche.


YOUR MISSION: GIVE ONLINE SURVEYS THE SPIRIT OF THE WEB


An Old Trick That Deserves a New Look

21 CONFIDENTIAL MATERIALS


•  99% of online surveys unfold the same way for a respondent whether they are the first or last respondent

•  They provide little in the way of qualitative exercises and less in the way of co-creation

Where We Are Today

22


Trends Rise of Online

Qual

Surveys with Huge Sample Sizes

Pressure for Shorter, More

Engaging Surveys

23


Your Mission

How can we make our surveys adapt to early responses to better serve later respondents or to provide better data?



7 Examples of Crowd-Shaped Surveys

Ques>on Buckets

Crowd-‐Sourced Choice Lists Crowd-‐Sourced

Laddering Comment Explora>on

Idea Explora>on Concept Explora>on

Crowd-‐Sourced Ques>onnaires

25


Question Buckets What Likes

• Prompt if fewer than 100 responses

What Dislikes


Reasons for Usage


Sugges>ons • Prompt if fewer than 100 responses




Ques>on Buckets






28

In the past, why have you thought the Atkins Diet was not right for you? (Choose all that apply.)

q Love pasta too much to go without q Love bread too much to go without q Love sweets too much to go without q Try to avoid ea>ng red meat q Worried about health problems that Atkins might cause

q Other _________________________________

Develop Choice Lists for Closed-‐Ended Ques>ons


Crowd-Sourced Choice Lists

•  The “Other – please specify” entries are added to the list of options •  Over time, only the 10 most frequently selected choices are

presented •  To protect against choices entered early always being selected, the

10 choices can be selected probalistically •  The “Other” text box uses autocomplete to suggest the unshown

choices •  Analysis ramifications: track display




Ques>on Buckets





30


BrainJuicer MindReader





Ques>on Buckets


Laddering Comment Explora9on



34


Comment Exploration: GMI Consensus Question Component

•  Respondents read and evaluate other respondents’ comments

•  Comments consistently scoring below a certain level can be screened out if required


Respondents Select the Comments to Rate


Management Portal for Comment Moderation



Ques>on Buckets



Idea Explora9on Concept Explora>on


38


Idea Exploration

•  Idea voting communities do a poor job of quantifying the appeal of ideas

•  Large idea spaces are problematic •  Alternative is to show 7 ideas 10 times,

each time having respondent choose the one they’d must like to see done




Ques>on Buckets



Idea Explora>on Concept Explora9on


40


Large Innovation Spaces

Number of Variants per AUribute

64

117,649

46,656

15,625 4,096 729

1 2 3 4 5 6 7

Content example: Concept with 6 attributes

Innovation space example:

Number of unique concepts increases exponentially.

Insight

Product Benefits

Clinical Endpoint

Positioning

Product Features

Product Image

1


Optimization vs. Measurement

Simulated Best Concept #3 Simulated Best

Concept #2

Simulated Best Concept #1

42

•  Traditional methods are measurement focused – Build a preference model to map a landscape – Search the map to find preferred concepts


Optimization vs. Measurement (continued)

Optimized Top Concept #3



43

•  Evolution is Optimization Focused – Directly searches the landscape – Does not need to estimate a model

Jeffrey Henning, Affinnova, USA Festival of NewMR 2011 – Session 1 Schedule: 00:00am-02:00am (GMT) 44

Evolutionary Optimization Preferred Concepts New Concepts

Random Jumps


Real-Time Optimization

45



Ques>on Buckets




Crowd-‐Sourced Ques9onnaires

46


Crowd-Sourced Questionnaire

•  Challenge online community to come up with questions to meet a specific research objective 1. Brainstorm on questions 2. Vote on questions 3. Have researcher vet, validate and create final

questionnaire - Source: Pat Molloy, ConfirmIt



Adaptive Questionnaires

•  Imagine EFM questionnaires that improve their respondent engagement over time

•  Length optimization – Shortening themselves if abandonment is high – Adding questions if abandonment is low

•  Question optimization – Hiding open-ended questions once sufficient

verbatims collected – Hiding questions that cause abandonment



Ques>on Buckets





49


Your Mission •  Currently most surveys don’t adapt or learn based on the

behavior of other respondents

•  Crowd-shaped surveys solve interesting qualitative and quantitative issues that static surveys can’t

•  While two of the examples premiered in 2001, crowd-shaped surveys are an important technique to explore further!

50


Your Mission

Can you create online surveys that better embody the Spïrït of the Wëb?


Thank you

Jeffrey Henning


53

Q & A

Jeffrey Henning Affinnova

Greg Coops Asian Strategies


Jeffrey Henning

Email Jeffrey @ [email protected] Read Jeffrey’s blog @ hUp://www.affinnova.com/blog/ Follow Jeffrey’s tweets at @ hUp://twiUer.com/#!/jhenning

Main Stage, November 3, 2011

Crowd-Shaped Surveys: Adapting the Experience Based on Prior Respondents Jeffrey Henning – Affinnova

A Presenta*on from the Fes*val of NewMR Main Stage – November 3, 2011