Workshop: 'Self-administered Mobile Survey Workshop' - Dr Michael Bosnjak, Free University of Bozen-Bolzano (Mobile Research Conference 2011)

Self-Administered Mobile Surveys

MRC 2011 Workshop (Part 1)

London (UK)April 18th, 2011

Michael Bosnjak, PhD, Assoc. Prof. Free University of Bozen-Bolzano, School of Economics and Management

1

• Definitions of mobile surveys– Interviewer-administered surveys

• Interviews among mobile phone users• Interactive voice response surveys among mobile phone

users

– Self-administered surveys• SMS (text messaging) surveys• Browser-based surveys on mobile devices (e.g., mobile

phones having mobile Internet-access, Smartphones, etc.)

• Our focus: – Self-administered surveys AND– using a mobile phone AND– browser-based.

2

Mobile Surveys?Self-Administered Mobile Surveys?

Selected Applications3

Directly at point of sale: in shopping malls and at points of service

At public venues, such as concerts

At schoolyards, in universities & recreational facilities

At trade fairs

In training seminars

In workplaces without internet

access

En-route with bus or

train & at the airport

En-route with bus or train & at the airport

B2BB2C

Insights from difficult to

reach target groups, event/incident-based

surveys, immediacy

Overall Goal

• Providing a very brief introduction into the methodological foundations of self-administered mobile surveys (esp. sources of biases known from survey methodology)

• Summarizing key findings of an own methodological study series conducted between 2008-2011

• Discussing practical, evidence-based recommendations (esp. on measurement and nonresponse issues)

Agenda

• Background– Survey research: Overall aims and scope– The ´Total Survey Error´ concept

– Factsheet: Mobile Survey Study series (2008-2011)

• Measurement issues– What can be presented/assessed?

– How usable are mobile question formats?– Voice capturing/recognition: Why and how?

– Acceptance of GPS positioning?

• Nonresponse issues– Industry perceptions on mobile survey (non)participation?

– Reasons for (non)participation: What do mobile survey participants tell us?

– ´True´ reasons for (non)participation?– Speed of participation?

– Optimal length of mobile surveys?

• Take-home messages and discussion

Background: Overall Aim of Surveys• Measuring ´true scores´, i.e. yielding unbiased

estimates for facts and/or latent variables.– Examples of factual questions to measure facts:

• Household-level income/expense estimates > Disposable income

• Behavioral frequency estimates > Behavior

– Examples of indicators supposed to measure latent variables:• Evaluative judgments > Attitudes• Behavioral likelihood scales > Intentions• Brand/product related attributes > Image

• Sources of errors in surveys:• Representation-related biases: Coverage, Sampling,

Nonresponse• Measurement-related biases/errors

6

Background: Total Survey Error7

Construct

Measurement

Response

Population

Sampling Frame

Sample

Respondents

Survey estimate

Measurement Representation

Coverage

Sampling

Nonresponse

Measurement

Measurement

Inappropriate implementation into a specific mode: Undesired design-related effects

Inappropriate operationalization (range restriction, reliability, validity)

Representative for the population in question?

Representative (valid) for the construct in

question?

Background: Survey Errors/Biases

• Coverage ErrorMembers of the target population have no chance of being selected in the sample (e.g., no access to the Internet, incomplete lists etc.). Error due to the fact that not every unit in the population is represented on the frame.

• Sampling Error... arises from the fact that not all members of the frame population are measured.

• Nonresponse ErrorThe responses of people who have not been surveyed are different from those who actually have participated in a survey.

• Measurement ErrorDeviation of the answers of respondents from their true values on the measure, e.g. due to inappropriate operationalizations of (latent) constructs, design features and context effects.

8

Mobile Survey Methodology: Study Series9

Mobile Study I(1.7.-2.9.08)

1. Web: Item development: Determinants of the willingess to participate in mobile surveys (Sozioland Web-Panel)2. Pre-Testing: Expert usability assessment at YOC 3. Web: Determinants of the willingess to participate S4 (YOC Mobile-Panel; 979 panelists, 272 participants)4. Olympic Games 2008 Mobile Survey(YOC Mobile-Panel; 979 panelists, 413 participants)5. Web: Usability of S4 from participants´ perspective(YOC Mobile-Panel; 413 panelists from S4, 187 completes)

Mobile Study II(29.9.-18.10.09)

Mobile Study III(March/April 2011)

8. Usability of voice capturing/recognition technology (presentation of results at tomorrow at MRC 2011, April 19, 2011)

6. Mobile survey: Evaluation of last vacation(Respondi Web-Panel; 3270 panalists, 540 completes)7. Web: Usability of S6 from participants´ perspective(Respondi Web-Panel; 540 panelists from S6, 318 completes)

www.mobileresearchconference.com10

http://www.mobileresearchconference.com







































































Agenda

• Background– Survey research: Overall aims and scope – The ´Total Survey Error´ concept










What can be presented/assessed? (I)12

Single choice

Multiple choice

Drop-Down menu

What can be presented/assessed? (II)13

TextfieldMatrix / Polarity profile

Voice / image /video capturing

How úsable´ are standard formats?14

Einfachauswahl untereinander

Mehrfachauswahl untereinander

Geschlossene Auswahlliste

Textfeld einzeilig

Fragetyp mit Bild

65,00 73,75 82,50 91,25 100,00

87,9

74,7

82,7

87,3

89,2

Usability score (Range: 0-100 Punkte)

Frag

etyp

Subjective Usability AssessmentPost-hoc survey (Web) one week after mobile survey completion

Indicators for usability score: fluency, simplicity, ease of use

Observed

Item-NR

Drop-Out

45%

9%

9%

23%

Multiple choice

Single choice

Drop-Down menu

Textfield

Image map

Voice recognition / capturing ?

Sources: MS I and MS II combined

15

Technical Implementation: iPhone App

16

Technical Implementation: Android

GPS positioning: Privacy concerns?17

91%

9%

Yes (willing to disclose)No (not willing to disclose)

Acceptance of GPS-Location

among participants with

an iPhone (MS II; n=45)

Agenda











Nonparticipation: Industry perceptions?

• Survey among 327 market researchers about acceptance and use of mobile surveys in D/A/CH

• Top 3 advantages of mobile surveys:– 51%: Independence of time/location– 49%: Context-sensitive, fast surveys– 43%: Reachability of hard-to-reach, mobile target groups

• Top 3 barriers for mobile surveys:– 35%: Costs incurred to survey participants (data traffic)– 35%: Difficulties entering information (esp. open-ended

questions)– 33%: Software/platform heterogeneity

19

Mobile Research Mobile Research

Barometer

Februar 2011

Mobile Research Barometer 2/2011

Nonparticipation: Self-Reports?20

Post survey in Wave 3, open responses, N= 63

´True´ Reasons for (Non)Participation I

• What is the influence of the following potential determinants of the willingness to participate?1.Attitude towards participating2.Hedonic aspects (perceived enjoyment)3.Social aspects (subjective norm)4.Image and perceived self-congruity5.Perceived benefits and costs

• Hypothetical modelExtended technology acceptance model (Venkatesh et al., 2003)

• Prospective study design (MS I)– S1: Developing and optimizing measurement models– S3: Assessing all above mentioned determinants– S4: Olympic games mobile survey (non)participation

21

´True´ Reasons for (Non)Participation II22

Discussion and Overview

The results of this study indicate that the extended TAM we propose offers a suitable heuristicframework for explaining both intentions to participate in mobile surveys and actual participation.Of the six factors we propose as influential, the hedonic, affective, self-expressive, and trust-relatedones emerge as important determinants of the propensity to participate. Utilitarian aspects, such ascost considerations and the perceived usefulness of using the mobile mode for surveys, as well asconsiderations involving the perceived social pressure, surprisingly do not appear to exert a signif-icant influence.

These results therefore offer some suggestions about ways to influence people’s decision to par-ticipate in mobile surveys. Because the propensity to respond seems primarily a matter of hedonic,affective, self-expressive, and trust-related factors, survey researchers must address these four con-structs through persuasive appeals. For example, to enhance hedonic and affective factors, messagesmight focus on the positive consequences of participation, such as enjoying the survey itself. Self-expressive appeals mainly stress the lifestyle and value attributes of stereotypical idealized persons(Johar & Sirgy, 1991); therefore, testimonials by aspirational spokespersons could be effective in

Figure 3. Structural model of participation in mobile surveys with standardized path coefficients (N ! 272).Notes: Solid arrows indicate significant paths; dashed arrows represent insignificant influences. For clarity, thisfigure does not depict the measurement models and cross-correlations between exogenous variables. A fullcorrelation matrix appears in Appendix B.

357

Bosnjak et al. 357

at SSC MASTER on August 22, 2010ssc.sagepub.comDownloaded from

*std.β, sig. at α=.05, N= 272

Fit Indices (robust)SB-Χ²=407; df=296p<.05, Χ²/df= 1.37NNFI=.98RMSEA=.04(.03-.05)

Highest influences:

> Hedonic aspects> Self-congruity

Not relevant:

> Expected costs (!)> Opinions of others

´True´ Reasons for (Non)Participation III

• If hedonic factors outperform cost/benefit-related, then– éxciting´ incentives (lottery drawing) should

increase participation rates – compensation for incurred costs should undermine

hedonic motivation (salience of costs is increased)

• MS I experiment, manipulating basic compensation (1 EUR, yes/no) and announced prize draw (100 EUR voucher, yes/no)

• Results confirmed our expectations (see Appendix):– highest access and participation ratesfor „lottery & no incentive condition“

23

Geschwindigkeit des Zugriffs auf Welle 1 und 2

Kum

ulie

rtert

proz

entu

aler

Ant

eil

Stunden seit Einladung

Speed of participation? (MS I)24

Faster responses for Mobile compared to Web:

approx. 35% Mobile versus aaprox. 10 % Web

For about 4.5 hours, Mobile response rates are higher

compared to Web

Speed of participation? (MS II)25

0

2

3

5

6

8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00

Mean response speed in hours for different contact/invitation time points (sent out via SMS)

MS I

Current context of participation? (MS I)26

At home busy with my PC

Watched TV

Worked at home

Read

In the office / at work

Preparing / eating a meal

On the move

Nothing disrupted 23,47%

8,16%

7,14%

11,22%

8,16%

10,20%

14,29%

17,35%

„Were have you been taking part in the

survey?“(Wave 3; N=116)

At home

In the office / at work

In a car

At the bus or train station

Using public transport

On the move (other reasons) 5,17%

2,59%

4,31%

6,90%

17,24%

63,79%

"Which activity did you have to disrupt to take part in the mobile survey?/ What have you done in

that very situation?“ (Wave 3; N=98; open responses)

Optimal length of mobile surveys? (MS II)27

0 %

25 %

50 %

75 %

100 %

non iPhone iPhone

68,9 %

10,3 %

31,1 %

89,7 %

onlinemobile

„Do you want to continue answering the survey mobile or online (in this case you

will get a link via email)?“

Participants: n= 540

0

7,5

15,0

22,5

30,0

Total iPhone Non iPhone

20,119,319,8

5,25,15,2

Min

nute

s

Part 1: Mobile- Initial SurveyPart 1 & 2: Mobile Survey

Agenda











Take-Home Messages & Discussion

• ... on mobile survey measurement:– Most ćlassical´ closed-ended item formats can be included and

are in many cases sufficiently usable– Various measurement options ´beyond´ the usual self-

administered formats do exist (e.g. GPS positioning, multimedia upload)

– Open-ended text may need to be replaced by voice capturing/recognition (to be discussed tomorrow)

• ... on mobile survey (non)response:– Industry perceptions and self-perception of potential mobile survey

participants on the reasons for nonresponse may be misleading– Most probable motivators: anticipated enjoyment, image– Boomerang effects for (over-)compensation– Fast responses, given in various contexts– Óptimal length´ may not exist, various factors appear to

influence the willingness to spend time on (mobile) surveys

29

Thank you!

[email protected]://www.bosnjak.eu

Michael Bosnjak, PhD, Assoc. Prof. Free University of Bozen-Bolzano, School of Economics and Management

30

mailto:[email protected]

mailto:[email protected]

http://www.bosnjak.eu

http://www.bosnjak.eu

Appendix

31

Acceptance and users‘ behaviorInfluencing participants‘ behavior: design

Basic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile surveyBasic compensation (1 €): participation in mobile survey

yesyes nono

Prize draw(100 € Amazon voucher)




yes no yes no

in theSMS Group 1 Group 2 Group 3 Group 4

on the survey landing page

Group 5 Group 6 Group 7 Group 8

Ince

ntiv

e in

form

atio

n (ti

min

g)


yesyes nono





yes no yes no

in theSMS

Group 1

8,9%Group 2

17,3%Group 3

21,2%Group 4

12,3%


Group 5 Group 6 Group 7 Group 8

Acceptance and users‘ behaviorInfluencing participants‘ behavior

Ince

ntiv

e In

form

atio

n (T

imin

g)

Landing page

access

Groups not relevant, first contact on landing page


Reminder

Response rates maximized with price draw (group 3), additional compensation undermines motivation (see group 1: 1€ and price draw).

Response rates in wave 2 against time

Cum

ulat

ed re

spon

se ra

te

Hours since SMS invitation

SMS informationGroup 1 (1 EUR + price draw)Group 2 (1 EUR)Group 3 (prize draw)Group 4 (no incentive information)


yesyes nono





yes no yes no

in theSMS

Group 1

5,9%Group 2

12,8%Group 3

14,4%Group 4

9,2%


Group 5

9,8%Group 6

9,6%Group 7

9,1%Group 8

10,5%


Ince

ntiv

e In

form

atio

n (T

imin

g)

All questions answered

Nonresponse issues: Background

Why increasing response rates to surveys?

true difference nonresponse

error

nonresponse

rate

Black Boxyr ! = statistic of interest for respondentsyt ! = statistic of the total sampleynr ! = statistic of interest for nonrespondents 36

Nonresponse: Background:

Types of nonresponse

Source: Bosnjak (2001)37

Nonresponse: Background:

Generic reasons for nonresponse

• Failure to deliver the survey request• Spam guards

• Unused or infrequently checked e-mail addresses

• Non-availability during fielding period

• Inability to provide the requested data• Lack of knowledge

• Insufficient information readily available

• Noncompliance: Refusals to survey requests

38

Nonresponse: Theory: Why do people (not) respond to surveys?

• Economic exchange view

• Human needs and values

• Compliance heuristics

• Transactional view

• Planned behavior approach

• Leverage-salience theory

• Social exchange theory

39









Rationale:Respondents are motivated by the monetary benefits promised/expected.

Actionable recommendations:„Pay respondents“ according to the time/effort invested

Caveats:• Peoples´ price points vary greatly and are unknown a-priori• May largely increase non-response bias •Undermines intrinsic motivation and may increase measurement error (low survey involvement)• Promised monetary incentives NOT consistently effective (!)

40









Rationale:Some values are systematically related to the propensity to respond (higher order needs, civit duty orientation, etc.)

Caveats:• Effects small (if any)• Actionable recommendations?

41









Rationale:Certain aspects of the survey announcement and survey implementation do induce compliant behavior:1. Reciprocity2. Scarcity3. Authority4. Consistency5. Consensus6. Liking

Actionable recommendations:Can be derived from persuasion literatures, but specific prescriptive models on how to tailor them toward survey situations are rare.

Groves, Cialdini & Couper (1992); Cialdini (2008);http://www.influenceatwork.com/

42

http://www.influenceatwork.com

http://www.influenceatwork.com









Rationale:Larger response propensity if communication style reflects positive regard and avoids adult-to-child communication styles.

Caveats:• Limited scope• Empirical evidence scarce• Covered by other theories (compliance heuristics, social exchange)

Comley (2006)43









Rationale:The propensity to respond to surveys is primarily a function of three factors:• Attitude to participate• Subjective norms• Perceived behavioral control • Moral obligation

Actionable recommendations:If weights are known for a specific population/sample: Enables the researcher to design survey participation requests

Caveats: Restricted to optimize survey announcements

Bosnjak (2002); Bosnjak, Tuten & Wittmann (2005)44









Rationale:Respondents are differentially motivated by • different aspects of the survey (leverage, e.g. type of incentives) and by • how much emphasis is put on each aspect by the surveyor (salience, e.g. preference for certain incentives )

Actionable recommendations:Because of the interaction between leverage*salience, improving response rates is not always desirable! Nonresponse bias may be influenced by leverage*salience interaction.

Groves, Singer & Corning (2000)45









Rationale: Survey participation as social exchange: The likelihood of responding is greater when the respondent trusts that the expected rewards will outweigh the anticipated costs of responding.

Actionable recommendations:Tailored Design Method, a well-developed set of practical recommendations on all aspects of survey design/implementation, aimed at:•establishing trust•increasing participation benefits•decreasing participation costs

Dillman, Smyth & Christian (2009)46

Nonresponse: Theory: TDM-based recommendations (selection)

To establish trust To increase benefits of participation

To decrease costs of participation

•Obtain sponsorship by legitimate authority•Provide a token of appreciation in advance•Make the task appear important•Ensure confidentiality and security of information

•Provide information about the survey•Ask for help or advice•Show positive regard•Say thank you•Support group values•Give tangible rewards•Make the questionnaire interesting•Provide social validation•Inform people that opportunities to respond are limited

•Make it convenient to respond•Avoid subordinate language•Make the questionnaire short and easy to complete•Minimize requests to obtain personal or sensitive information•Emphasize similarity to other requests or tasks to which a person has responded

Dillman, Smyth & Christian (2009, p. 38) 47

Nonresponse: Evidence: Mail surveys:

Effective methods & procedures I

• Most effective factors in mail surveys (only factors under the researchers full control listed):

• Personalization of requests to participate(Dillman, 1978, 2000; Edwards et al., 2007; Fox, Crask, & Kim, 1988; Heberlein & Baumgartner, 1978; Yammarino et al.,1991; Yu & Cooper, 1983)

• Prepaid monetary incentives(Church, 1993)

• Number of contacts made (esp. if prenotifier is included)(Armstrong & Lusk, 1987; Edwards et al., 2007; Fox et al., 1988; Heberlein & Baumgartner, 1978; Yammarino et al.,1991; Yu & Cooper, 1983)

➡ Integrated and refined within the Total-Design-Method (Dillman, Smyth & Christian, 2009)

48

Nonresponse: Evidence: Mail surveys:

Effective methods & procedures II

• Effective, but not covered because of limited control:

• Survey topic / topic involvement

• Length

• Sponsorship (University / commercial)

• Factors reducing response rates (1, 2: Edwards et al., 2007; 3: Singer, Hippler & Schwarz, 1992):

1. Starting with the most general question (e.g. demographics)

2. Opportunity to opt-out of the study

3. Over-emphasizing data protection/confidentiality

•Partly covered later for Web surveys:

• Questionnaire design effects on nonresponse

49

Nonresponse: Evidence: Web surveys:

Effective methods & procedures III

• Personalization:

• Personal salutation (name) is effective (esp. for powerful sender) (e.g., Heerwegh, et al., 2005; Joinson & Reips, 2007)

• (Monetary) Incentives:

• In general effective but small overall effect (Göritz, 2006)

• Pre-paid monetary incentives need to be tangible to be effective(Birnholtz et al., 2004; Bosnjak & Tuten, 2003)

• Lotteries esp. effective, timing important (immediate notification)(Bosnjak & Tuten, 2003; Tuten, Galesic & Bosnjak, 2004)

•Contact features:

• No of contacts very effective (Cook, Heath & Thompson, 2000)

• SMS prenotifier very effective (Bosnjak et al., 2008)

50

Business

Workshop: 'Self-administered Mobile Survey Workshop' - Dr Michael Bosnjak, Free University of Bozen-Bolzano (Mobile Research Conference 2011)