Embed Size (px)
DESCRIPTION
Most surveys use data quality techniques such as red herrings, speeding, straightlining, and jibberish monitoring to determine which completes represent good responders or poor responders. This paper, presented at the 2013 NewMR online festival, demonstrates which techniques properly identify good and poor responders.
Citation preview
Festival of NewMR 2013 – Main Stage
Sponsor logos will go here
Angels and Demons: Are you penalizing the right people?
Annie Pettit, Chief Research Officer
Peanut Labs
Canada
Festival of NewMR 2013 – Main Stage
Raise your hand if right now you are…
04/11/2023 2
Festival of NewMR 2013 – Main Stage
Lazy Bum Larry
3
Festival of NewMR 2013 – Main Stage
Good Faith Gail
Festival of NewMR 2013 – Main Stage
Method to our madness
• 8000 people answered a survey
• Every survey measured against each of 30 data quality measures
• Calculated number of errors for each person (from 0 to 30)
• Grouped people into Angels and Demons based on number of “fails”
5
Festival of NewMR 2013 – Main Stage
Are responders angels or demons?
23%
77%Zero errors
One or more errors
Festival of NewMR 2013 – Main Stage
1 Red Herring
• 5.8% of angels chose 1 red herring
• 14.5% of demons chose 1 red herring
Why?• Sounds like a brand they know• It IS a local brand they know• Babies biting ankles
(
1
Festival of NewMR 2013 – Main Stage
2 Red Herrings
• 1.4% of angels chose 2 red herrings
• 40.2% of demons chose 2 red herrings
Why?• They didn’t notice it. Twice.• They aren’t reading it. At all.• They need to finish it. NOW!
Festival of NewMR 2013 – Main Stage
Tier 1 Speeding (<50% of the mean)
• 8.4% of angels speed • 40.7% of demons speed
Why?• They’re computer literate.• They read well. Really well.• They know surveys. Really well.
(
2
Festival of NewMR 2013 – Main Stage
Tier 2 Speeding (Fastest 1.5%)
• 0% of angels speed • 28% of demons speed
Why?• They’re good at clicking.• They aren’t reading. At all.• They know surveys. Really well.
Festival of NewMR 2013 – Main Stage
Straightlining
• 14.2% of angels straightline
• 60.7% of demons straightline
Why?• Every item was phrased the same way• It’s their true opinion
(
3
Festival of NewMR 2013 – Main Stage
Good vs. Bad Grids
• Tastes great• Looks appetizing• Smells nice• Well packaged• Interesting logo• Pretty colours
• Tastes terrible• Looks appetizing• Smells strange• Well packaged• Odd logo• Pretty colors
What is your level of agreement with each of the following statement? This product:
Strongly Agree
Somewhat Agree
Neutral Somewhat Disagree
Strongly Disagree
Festival of NewMR 2013 – Main Stage
Straightlining on A Good Grid
• 2.8% of angels straightline
• 58.4% of demons straightline
Why?• They aren’t reading. At all.• They know surveys. Really well.
Festival of NewMR 2013 – Main Stage
Rank From 1 through 6
• 7.9% of angels rank incorrectly
• 50% of demons rank incorrectly
– Choose same number twice– Choose numbers outside the
limits– Straightline
Why?• They aren’t reading. At all.• They click really well.
4
Festival of NewMR 2013 – Main Stage
Contradictions
• 28.4% of angels contradict themselves only once
• 39.5% of angels contradict themselves twice
• 7.9% of demons contradict themselves only once
• 73.8% of demons contradict themselves twice
Why?• Because it’s a valid answer• Babies biting ankles
(
5
Festival of NewMR 2013 – Main Stage
Verbatim Length
• 0.9% of angels share less than 10 characters
• 56.5% of demons share less than 10 characters
Why?• asdf• aaaa• don’t know
6
Festival of NewMR 2013 – Main Stage
Non-traditional behaviours
• 12.5% of angels buy non-grocery items at grocery stores
• 37.4% of demons buy non-grocery items at grocery stores
Why?• Because it’s a valid answer• Babies biting ankles
(
7
Festival of NewMR 2013 – Main Stage
High Incidence Awareness
• 3.5% of angels knew <3 of 8 big name brands
18
• 48.1% of demons knew <3 of 8 big name brands
Why?• Immigrants• Non consumers• Non shoppers
8
Festival of NewMR 2013 – Main Stage
Conclusions
Angels don’t exist
Perfection doesn’t exist
It’s not fair to expect perfection from responders
acting in good faith
Festival of NewMR 2013 – Main Stage
Conclusions
To Penalize Angels:• Use 1 red herring• Use only positive grids• Measure only one or two
contradictions• Don’t permit 1 or 2 mistakes
To Penalize Demons:• Use 2 red herrings• Use mixed grids• Use a rank question• Measure short verbatims• Focus on multiple mistakes
Festival of NewMR 2013 – Main Stage
Thank you!
Contact me with any questions.
Annie Pettit, PhDChief Research OfficerPeanut Labs
twitter.com/LoveStats
facebook.com/AnniePettit
ca.linkedin.com/in/AnniePettit
Festival of NewMR 2013 – Main Stage
From sampling to surveys to listening, you need to get your research done quickly and accurately. Peanut Labs tools help you build your own surveys and select targeted sample from our profiled panel. And, our Evolisten social media listening tool helps you gain a broader understanding of your consumers.
Use our DIY, permission based, profiled sampling tool to select and launch targeted sample
Use our DIY survey design tool to craft your surveys
Obtain quick poll results with our DIY targeted polling tool. Answers in mere minutes!
Listen to millions of social media opinions about people, places, brands, and more.
www.peanutlabs.com
