Embed Size (px)
Citation preview
ESRA Conference July 2011
Response Bias in Global
Online Panels: Attrition Effects
Presented by:
Elaine Trimarchi, Executive Vice President
Mktg, Inc. New York , USA
A True Story
Ron Gailey, when at Washington Mutual,
presented the results of 29 online
research studies, a total of 40,000
interviews, conducted from 2006 to 2007.
Problem: demand for financial products
was decreased over time; a phenomenon
not supported by experience in the
market.
2 ©2011 Mktg, Inc. All Rights Reserved.
“ In every study examined…
….people with more panel tenure
gave lower demand.”
Ron Gailey, 2008
3 ©2011 Mktg, Inc. All Rights Reserved.
The Grand Mean Project
We began with 17 American panels in 2008
We are now collecting data in 35 countries and 200+ panels
The Grand Mean is the average value of available panel data for a country or region.
4 ©2011 Mktg, Inc. All Rights Reserved.
Grand Mean Project
Selected demographic quotas were used to simulate census (age, income, gender and in the U.S. ethnicity).
Average length 18 minutes.
Each sample completed 500 -1000 interviews.
Questions covered: Technology and the media, Participation in market research, Values and lifestyle, Demographics.
We ask self reporting questions on survey participation including panel tenure, panel membership, etc.
5 ©2011 Mktg, Inc. All Rights Reserved.
6 ©2011 Mktg, Inc. All Rights Reserved.
Segmentations
From the Grand Mean Survey, we create ten segmentations.
Three main segmentations: Buyer Behavior
(37 variables) , Sociographics (31 variables) and Media (31 variables).
Seven other segmentations are: automotive, appliances, banking and Insurance, consumer electronics, clothing, entertainment, grocery shopping.
7 ©2011 Mktg, Inc. All Rights Reserved.
8 ©2011 Mktg, Inc. All Rights Reserved.
VIC----Replace
9 ©2011 Mktg, Inc. All Rights Reserved.
10 ©2011 Mktg, Inc. All Rights Reserved.
Now, let’s look at some
other Measures of
Hyperactivity
Belong to >4 panels
Surveys everyday
>30 surveys per month
11 ©2011 Mktg, Inc. All Rights Reserved.
15
.......so how can we
create consistency?
….we need standards.
22 ©2011 Mktg, Inc. All Rights Reserved.
©2011 Mktg, Inc. All Rights Reserved. 23
Solution
Using pre-profiled respondents, we are
able to target the Grand Mean Standard.
We select respondents based on their
demographic and behavioral profile in
order to meet the target.
©2011 Mktg, Inc. All Rights Reserved. 25
Column 1 - The Grand Mean – Census Balanced standard
of 20 American panels 26 ©2011 Mktg, Inc. All Rights Reserved.
Column 2 – First batch of Sample Sent – Segmentations for
the first 2818 invites. 27 ©2011 Mktg, Inc. All Rights Reserved.
Column 3 - Segmentations for the first 471 completes 28 ©2011 Mktg, Inc. All Rights Reserved.
Column 4 – Augment sample sent - although the segmentations
are balanced, a sample correction was needed for the imbalance of
the demographics. 29
©2011 Mktg, Inc. All Rights Reserved.
Column 5 – Non-responders to the e-mail invite (2364)
30 ©2011 Mktg, Inc. All Rights Reserved.
Column 6 – Segmentations for those who Broke-off during
the survey. 31 ©2011 Mktg, Inc. All Rights Reserved.
Column 7 – The Total sample sent (3914 invites)
32 ©2011 Mktg, Inc. All Rights Reserved.
Column 8 – Segmentations for the final quota of 600
completes 33 ©2011 Mktg, Inc. All Rights Reserved.
Conclusion Panels are not interchangeable.
Hyperactivity, such as panel membership, the number of surveys per month and panel tenure, creates inconsistencies.
Panels must maintain stable respondent tenure profiles or they will be unstable through time.
Consistency is important. To achieve consistency, we need a standard.
Researchers need to know
……. if the shifts they see in their data are real or due to changes in the sample frame.
34 ©2011 Mktg, Inc. All Rights Reserved.
36 ©2011 Mktg, Inc. All Rights Reserved.
