SMARTER ONLINE PANELS FORSMARTPHONE USERS:EXPLORING FACTORS ASSOCIATED WITH MOBILE
PANEL PARTICIPATION
Lauren A. Walton, M.S.Trent D. Buskirk, Ph.D.
Thomas Wells, Ph.D.
The Nielsen Company
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
2
SURVEY PARTICIPATIONSmartphones
We have observed…• Online surveys are being completed over
smartphone (deliberately orunintentionally).
• Penetration continues to rise across alldemographics
Imagine a scenario asking respondents toparticipate in an online panel where allsurveys are optimized and completed on asmartphone’s browser.
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
3
SURVEY PARTICIPATION
Smyth, J. (Director) (2010, November 9). Paper Self-Administered Surveys. SRAM 818: Data Collection Methods. Lecture conducted from Universityof Nebraska-Lincoln, Lincoln.
Social Exchange TheoryRespondents trust the benefits of respondingwill outweigh the costs of responding or notresponding.
What panel participation factors “cost” ourrespondents the most?
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
4
SMARTPHONE PANEL PARTICIPATION QUESTIONSQuestion Stem
Suppose you have been asked to join a survey panel inwhich you would use only your smartphone tocomplete surveys. How influential would each factorbe in you decision to agree to participate in the panel?
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
5
SMARTPHONE PANEL PARTICIPATION QUESTIONSPanel Participation Factors (9)
1. Data consumption associated with panel activities
2. How often the survey requests are made (e.g. weekly, bi-weekly, monthly, etc.)
3. How long it takes to complete a single survey
4. Amount of personal information required to join the panel
5. Incentive amount per completed survey
6. Level of GPS location sharing required
7. Requirement to download software/app to complete survey
8. Content of panel surveys
9. How long the panel lasts (e.g. one week, one month, etc)
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
6
SMARTPHONE PANEL PARTICIPATION QUESTIONSPanel Participation Factors
• Data consumption associated with panel activities
• How often the survey requests are made (e.g. weekly, bi-weekly, monthly, etc.)
• How long it takes to complete a single survey
• Amount of personal information required to join the panel
• Incentive amount per completed survey
• Level of GPS location sharing required
• Requirement to download software/app to complete survey
• Content of panel surveys
• How long the panel lasts (e.g. one week, one month, etc)
SurveyContent
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
7
SMARTPHONE PANEL PARTICIPATION QUESTIONSPanel Participation Factors• Data consumption associated with panel activities
• How often the survey requests are made (e.g. weekly, bi-weekly, monthly, etc.)
• How long it takes to complete a single survey
• Amount of personal information required to join the panel
• Incentive amount per completed survey
• Level of GPS location sharing required
• Requirement to download software/app to complete survey
• Content of panel surveys
• How long the panel lasts (e.g. one week, one month, etc)
Costsvs.
Benefits
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
8
SMARTPHONE PANEL PARTICIPATION QUESTIONSPanel Participation Factors• Data consumption associated with panel activities
• How often the survey requests are made(e.g. weekly, bi-weekly, monthly, etc.)
• How long it takes to complete a single survey
• Amount of personal information required to join the panel
• Incentive amount per completed survey
• Level of GPS location sharing required
• Requirement to download software/app to complete survey
• Content of panel surveys
• How long the panel lasts (e.g. one week, one month, etc)
PersonalInfo
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
9
SMARTPHONE PANEL PARTICIPATION QUESTIONSPanel Participation Factors
• Data consumption associated with panel activities
• How often the survey requests are made (e.g. weekly, bi-weekly, monthly, etc.)
• How long it takes to complete a single survey
• Amount of personal information required to join the panel
• Incentive amount per completed survey
• Level of GPS location sharing required
• Requirement to download software/app to complete survey
• Content of panel surveys
• How long the panel lasts (e.g. one week, one month, etc)
Time
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
10
RESEARCH QUESTIONS
1. How do the survey specific factors affect participation in ahypothetical online smartphone browser panel?
2. What are the measurement differences between Likertratings scale and MaxDiff conditions?
Benefits>Costs
Equivalent Measures?
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
11
METHODOLOGYData Source and Experiment
• 1,013 Respondents from GFK KnowledgePanel• Probability based internet panel• Nationally representative
• Screened for Smartphone and Tablet ownership
• Randomly assigned into one of two conditions:• 520 MaxDiff Scaling
• 51.3%• 493 Likert Ratings Scale
• 48.7%
• Checked randomization to conditions• 11 demographic variables• No Significant differences
• 8 dimensions of potential smartphone panel participation
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
12
SMARTPHONE PANEL PARTICIPATION QUESTIONSLikert Scale: 1 question
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
13
SMARTPHONE PANEL PARTICIPATION QUESTIONSLikert Scale: 1 question
Importance ofeach itemseparately
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
14
SMARTPHONE PANEL PARTICIPATION QUESTIONSMaxDiff Scaling: 7 sets of questions
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
15
SMARTPHONE PANEL PARTICIPATION QUESTIONSMaxDiff format: 7 sets of questions
MostInfluentialvs. Least
Influential
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
16
METHODOLOGYMeasurement Differences
Mora, M. (2010) “Making The Case For MaxDiff”; http://relevantinsights.com/making-the-case-for-maxdiff
Ratings Scale (Likert) MaxDiff
Each item is rated on ascale independently
RSPN select from a setof items the most and
least importantRating Items
MeaningUser scale bias(acquiesces or
extreme responding)
RSPN forced intoproviding relative
importance
Precision Lack ofdiscrimination:
respondents oftenrate everything as
preferred orimportant
Strongdiscrimination
power: RSPN forcedinto relative choice
Provides ratio dataand a measure of
magnitude
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
17
RESULTSTime to Complete: MaxDiff>Likert (Time difference 134.4)
Max Diff Section Likert Section
Average Section Timein seconds
201.2 66.8
Standard Deviation 435.21 53.97
MaxDiffaverage
3.5minutes.
On average,RSPN took over3 times as long
to completethe MaxDiff
section.
HighlySignificantDifference.
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
18
RESULTSTime to Complete: MaxDiff>Likert (Time difference 134.4)
3 m
inut
es 3
0 se
cond
s
3 m
inut
es 3
0 se
cond
s
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
19
RESULTSAverage Participation Influence
0
1
2
3
4
5
6
Frequency ofSurveys
Panel Duration Downloads Incentive Personal Info GPS Content ofSurveys
DataConsumption
Survey Length
Likert Rating Scale: Average Importance
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
20
RESULTSAverage Participation Influence
0
1
2
3
4
5
6
Frequency ofSurveys
Panel Duration Downloads Incentive Personal Info GPS Content ofSurveys
DataConsumption
Survey Length
Likert Rating Scale: Average Importance
#1: Incentive amount per completed survey
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
21
RESULTSAverage Participation Influence
0
5
10
15
20
25
Frequency ofSurveys
Panel Duration Downloads Incentive Personal Info GPS Content ofSurveys
DataConsumption
Survey Length
MaxDiff: Average Importance
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
22
RESULTSAverage Participation Influence
0
5
10
15
20
25
Frequency ofSurveys
Panel Duration Downloads Incentive Personal Info GPS Content ofSurveys
DataConsumption
Survey Length
MaxDiff: Average Importance
#1: Incentive amount per completed survey
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
23
Likert Rating Scale MaxDiff
1. Incentive amount per completedsurvey
Incentive amount per completedsurvey
2. How long it will take to completea single survey
How long it will take to completea single survey
3. Amount of personal informationrequired to join the panel
Amount of personal informationrequired to join the panel
4. Requirement to downloadsoftware/app to completesurveys
Requirement to downloadsoftware/app to completesurveys
5. How long the panel lasts (e.g. oneweek, one month, etc)
Level of GPS location sharingrequired
6. How often survey requests aremade (e.g. weekly, bi-weekly,monthly, etc.)
How often survey requests aremade (e.g. weekly, bi-weekly,monthly, etc.)
7. Level of GPS location sharingrequired
How long the panel lasts (e.g. oneweek, one month, etc)
8. Content of panel surveys Data consumption associatedwith panel activities
9. Data consumption associatedwith panel activities
Content of panel surveys
5 & 7 flip flop
Top 4 is thesame acrossconditions
8 & 9 flip flop
6 same
5 & 7 flip flop
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
24
CONFIDENCE BANDS• Measurement Techniques
Likert Scale
MaxDiff
Overlap inconfidenceintervals
Morevariation*
*some CI overlap butnot ENTIRE CIs
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
25
CONFIDENCE BANDSMaxDiff: Less Confidence Interval Overlap
Incentive
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
26
KEY TAKEAWAYSCosts of Participation
Incentive
Personalinformation
DownloadRequirements
Survey Length
PanelDuration
Frequency ofRequests
GPS
SurveyContent
DataConsumption
PersonalInformation
Length
DownloadRequirements
PanelDuration
Frequency ofRequests
GPS
SurveyContent
DataConsumption
Incentive
1 2 3Likert RatingsScale
MaxDiff
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
27
KEY TAKEAWAYSCosts of Participation
Top 4 Participation Attributes
• Incentive amount per completed survey
• How long it will take to complete a single survey
• Amount of personal information required to join the panel
• Requirement to download software/app to complete surveys
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
28
KEY TAKEAWAYSCosts of Participation
Incentive
Personalinformation
DownloadRequirements
Survey Length
PanelDuration
Frequency ofRequests
GPS
SurveyContent
DataConsumption
PersonalInformation
Length
DownloadRequirements
PanelDuration
Frequency ofRequests
GPS
SurveyContent
DataConsumption
Incentive
1 2 3Likert RatingsScale
MaxDiff
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
29
Tim
e MaxDiff onaveragetakes 3timeslonger
Scre
ens MaxDiff
required 7screens to1 LikertGridQuestion
KEY TAKEAWAYSMeasurement Techniques
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
30
Scre
ens MaxDiff
required 7screens to1 LikertGridQuestion
KEY TAKEAWAYSMeasurement Techniques
If we moved theexperiment tomobile environment
Likert may take asmany as 9 screens toavoid the grid
Time could be moresimilar betweenmeasurementtechniques
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
31
RESEARCH QUESTIONS1. How do the survey specific factors affect participation in a
hypothetical online smartphone browser panel?
2. What are the measurement differences between Likertratings scale and MaxDiff conditions?
• 3 panel participation factor influence groupings
• #1 Incentive amount per completed survey
• Similar panel participation factor influencerankings
• More stability in estimates from MaxDiff
• MaxDiff takes RSPN more time
Special Thanks to: Joseph Curry, Ph.D, Sawtooth TechnologiesMark Little , Sawtooth Technologies
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
33
CONFIDENCE BANDSLikert Rating Scale: Little differentiation (up to sampling error)
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
34
CONFIDENCE BANDSLikert Rating Scale: Little differentiation (up to sampling error)
Personalinformation
Incentive
DownloadRequirements
Survey Length
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
35
CONFIDENCE BANDS
PanelDuration
Frequency ofRequests
GPS
Content
DataConsumption
Likert Rating Scale: Little differentiation (up to sampling error)
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
36
CONFIDENCE BANDSMaxDiff: Less Confidence Interval Overlap
Incentive
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
37
CONFIDENCE BANDSMaxDiff: More differentiation in the estimates
PersonalInformation
Length
DownloadRequirements
Copy
right
©20
13 T
he N
ielse
n Co
mpa
ny. C
onfid
entia
l and
pro
prie
tary
.
38
CONFIDENCE BANDSMaxDiff: More differentiation among estimates Panel
Duration
Frequencyof Requests
GPS
Content
DataConsumption