View
1
Download
0
Category
Preview:
Citation preview
TakingSurveystoPeople’sTechnology:Implica:onsforFederalSta:s:cs
andSocialScienceResearch
Frederick G. Conrad Michael F. Schober
Collaborators: Christopher Antoun, David Carroll, Patrick Ehlen, Stefanie Fail, Andrew L. Hupp, Michael Johnston, Courtney Kellner, Kelly F. Nichols, Leif Percifield, Lucas Vickers, H. Yanna Yan, & Chan Zhang
Support:NSFgrantSES-1025645(Methodology,Measurement,andSta?s?csprogram)
Mobilemul?modalphones(smartphones)
• Peopleincreasinglycommunicateviasmartphones– >68%ofUSadultsasof2015– Con?nuinggrowth
• Peopleincreasinglyuseandswitchbetweenmul?plemodes(manyna?vetosmartphone)forinterac?ng– Voice– Text(SMS)– Email– Videochat– Webaccessforcommunica?on(e.g.,blogposts)– Specializedappsforcommunica?on(e.g.,Facebook)
Peopleincreasinglyexpectcapabilityto:• communicatewhilemobileand/ormul?tasking• chooseamodethatfitstheircurrentseWngandneeds
– e.g.,urgentvs.canwait,publicvs.private,noisyvs.quiet,brightvs.dim• switchmodeswhilecommunica?ng• respondinadifferentmodethancontacted
– e.g.,respondtovoicemailwithatext
Newpossibili?esandchallengesforsurveydatacollec?on
• AsR’sexpectmul?plemodesonthesamedevice,mayexpectthatsurveysaremul?modal– Poten?altointeractviaSMStextwhenconvenient– Poten?altorespondinmodethatisappropriatetocurrentseWng
(e.g.,textinnoisyenvironment,voicewhenthereisglare,etc.)– Moregenerally,tobeabletorespondinanymode,any?me,
anywhere
• Dopeoplerespondtoconven?onalsurveymodes(e.g.,telephoneinterviews)inthesamewayonsmartphonesasonlandlines?
• Howdopeoplerespondtolessconven?onalsurveymodesthatusesmartphonecapabili?es?
Newop?onsforsurveymodechoice• Nowpossibletochooseamodeonasingledevice,immediatelyandconveniently
• Quitedifferentfrompriorimplementa?onsofsurveymodechoice– WhenRinvitedbymailcompleteseitheronpaperorweb,thisrequiresextrastepoftypingURLintobrowser
– Canreduceresponserates(e.g.,Fulton&Medway,2012)
• Choiceonsingledevicemayleadtodifferentoutcomes
Modecomparisonstudy(Schoberetal.,2015,PLOSONE)
• examines– dataquality(sa?sficing,disclosure)– comple?onrates– respondentsa?sfac?on
• fourexis?ngorplausiblesurveymodesthatworkthroughna?veappsontheiPhone– Asopposedtospeciallydesignedsurveyapps– Asopposedtowebsurveyinphone’sbrowser– UniforminterfaceforallRs
• Asopposedtomixofplacorms(Android,Windows,etc.)
Experiment:4modesoniPhone
Medium
Voice SMSText
Interviewing
Agent
Human
Humanvoice
(RspeakswithI)
Humantext
(RtextswithI)
Automated
SpeechIVR
(Rspeakswithsystem)
AutomatedText
(Rtextswithsystem)
Surveysviatextmessaging?• Moreandmorepeopleareembracingtextmessagingforpersonalandprofessionalcommunica?on– ontheirmobilephones(smartphonesornot)– onotherdevices(e.g.,tablets,desktops)
• Tex?ngisbecomingapoten?allyimportantwaytoreachrespondents– somemayaeendtotextmorethantoemailsorvoicemails
– respondentsmayexpecttobeabletopar?cipateinasurveyviatext
• Someorganiza?onsarenowincludingSMStextintheirsuiteofmodesformobilesurveys– e.g.,GeoPoll,PollEverywhere,iVisionMobile,etc.
Textasamodeofinterac?on• Turn-by-turn
– Threaded(onasmartphone)
• Responsesdon’tneedtobeimmediate– Allowsmul?tasking
• Worksevenwithintermieentnetwork/cellservice– unlikevoice
• Doesnotrequirewebcapacityondevice– unlikemobilewebsurvey
Property Voice Text Synchrony Fully synchronous Less or asynchronous Medium Auditory Visual Language Spoken/heard Written/read Conversational structure
Turn-by-turn, with potential for simultaneous speech
Turn-by-turn, rarely but possibly out-of-sequence
Persistence of turn No Yes Persistence of entire conversation No Yes, threaded
Social presence of partner
Continuous (auditory) presence
Intermittent evidence (when texts arrive)
Character of multitasking
Simultaneous, especially when hands free, unless other task involves talking
Switching required between texting and other tasks
Impact of environmental conditions
Potential interference from ambient noise
Potential interference from visual glare
Impact of nearby others
Others may hear answers; potential audio interference from others’ talk
Others unlikely to see text and answers on screen, though possible
t
Measuresofdataquality• Conscien/ousresponding(lesssa?sficing)
– Rsareknowntotakeshortcuts—to“sa?sfice”—differentlyindifferentsurveymodes
• e.g.,Chang&Krosnick(2009),Heerwegh&Loosveldt(2008)– Weexamine
• roundednumericalresponses(e.g.,mul?plesof10)– Unroundedanswersaremorelikelytoresultfromdeliberate,memory-basedthoughtprocessesthan
es?ma?on(Brown,1995;Conrad,Brown,&Dashen,2003)– morelikelytobeaccurateinanswerstoobjec?vefactualques?ons(Holbrooketal.,2014)
• straightlining(nondifferen?a?on)– givingsameanswertobaeeryofQs
• Disclosure(moreisbeeer)– Rsopendisclosemoresensi?veinforma?onwhentheyself-administeraques?onnaire(websurveys,ACASI)
• e.g.,Kreuteretal.(2009),Tourangeau&Smith(1996)
• Par/cipa/onandcomple/on
Possibleoutcomes:Conscien?ousresponding
• TEXTVS.VOICE– Rsmightbelessconscien>ousintextbecausetheyimport“leasteffortstrategy”fromhowtheyusuallytext
– ORRsmightmoreconscien>ousintextbecausetheyfeelless?mepressuretorespondthaninspokeninterviews
– andcananswerwhentheyareready• HUMANVS.AUTOMATED
– Rsmightbelessconscien?ouswithautomatedinterviewer(selfadministra?on)becausethereisnohumantomo?vatethemtobeconscien?ous
Possibleoutcomes:Disclosure• TEXTVS.VOICE
– Rsmightdisclosemoreintextbecauseoffewersocialcuesintheinterac?on
• lessevidenceofreac?ontoanswers?• more?metobecomfortablewithanswers?• nooneelsecanheartheques?onsoranswers?
– ORRsmightdiscloselessintextbecause• theyworrythatothersmightseevisuallypersistentanswers?• theyworrythatanswersarepermanentlystored?• theycantake?metoanswerinwaysthatgivethebestimpression?
• HUMANVS.AUTOMATED– Rsmightdisclosemorewithautomatedinterviewer,asinACASIorwebsurvey
Items• First,safe-to-talkorsafe-to-textques?on• 32QstakenfrommajorUSsocialsurveysandmethodological
studies– E.g.,BRFSS,NSDUH,GSS,PewInternet&AmericanLifeProject– Formost,knowntohaveproduceddifferencesinsa?sficingor
disclosurebetweenconven?onalmodes• Yes/no,numerical,categorical,baeeryitems(seriesofQs
withsameresponseop?ons)• Ra?onaleforinclusion
– Qswithmoreandlesssociallydesirableanswers• e.g.,sexualhistory,druguse,newspaperreading
– Qsforwhichfrequencyreportscouldbepreciseores?mated(rounded)
• e.g.,numberofmoviesseenlastmonth,numberofappsoniPhone– BaeeryQ’sthatcouldproducestraightlining(non-
differen?a?on)
Implementa?on:Humanvoice• 8interviewers(Is)fromUMichsurveyresearchcenter
• customdesignedCATIinterfacethatsupportsvoiceandtextinterviews(PAMSS)
Implementa?on:Humantext• Same8IsfromUMichsurveyresearchcenter• SamecustomdesignedCATIinterface
– Iselects,edits,ortypesques?ons/prompts,andclickstosend• Textmessagessentthroughthirdparty(Aerialink)• Rscananswerwithsinglecharacter:Y/N,leeer(a/b/c),ornumber
Implementa?on:SpeechIVR
• Custombuiltspeechdialoguesystem• UsesATT’sWatsonspeechrecognizer,Asterisktelephonygateway
• Recordedhumaninterviewer,speechresponses(nottouchtone)
Implementa?on:Auto-text• Custombuilttextdialoguesystem• Textmessagessentthroughthirdparty(Aerialink)• Rscananswerwithsinglecharacter:Y/N,leeer(a/b/c),or
number
Respondents:634iPhoneusers• n=157to160randomlyassignedtoeachmode• RecruitedfromCraigslist,Facebook,GoogleAds,andAmazonMechanicalTurk– Webscreenerverifiedage(>21years)andUSareacode– iPhoneusageverifiedviatextmessagetodeviceanduseragentstringinresponse
• $20iTunesgipcodeincen?ve,providedaperpost-interviewwebques?onnaire
• Age,gender,ethnicity,income,educa?onnotreliablydifferentinfourmodes
• SomewhatyoungerandlessaffluentthanUSna?onaliPhoneusers
TextRespondent
22
HumanTextInterviewerInterface
23
Datacollec?on
• InterviewscarriedoutMarch-May2012
• Resultsbasedonspeech-IVRsystemrecogni?on– 95.6%correctrecogni?onaccuracybasedontranscripts
– Samepaeernofresultsifweusehumanannota?ons(Johnston,etal.,2013)
Percent respondents reporting rounded numbers of… Human Auto Human Auto Estimate SE
Odds ratio Estimate SE
Odds ratio Estimate SE
Odds ratio
Movies seen in theaters in past 12 months 24.4% 18.2% 17.1% 12.1% -0.463* 0.211 0.630 -0.383† 0.210 0.682 -0.035 0.425 0.965
Songs on iPhone 66.9% 61.8% 45.2% 51.6% -0.655*** 0.163 0.520 0.026 0.163 1.026 0.478 0.326 1.612
Apps on iPhone 80.6% 78.6% 47.5% 54.1% -1.332*** 0.179 0.264 0.112 0.175 1.119 0.391 0.358 1.479
Text messages sent and received on iPhone in current billing cycle 91.1% 90.1% 73.2% 70.5% -1.331*** 0.233 0.264 -0.125 0.214 0.882 -0.019 0.466 0.981
Times they ate spicy food in last month 46.5% 41.2% 38.6% 52.2% -0.325 0.228 0.640 -0.218 0.229 0.804 0.771* 0.323 2.162
Movies watched in any medium in last month 30.6% 40.9% 32.9% 30.6% -0.179 0.168 0.836 0.179 0.168 1.196 -0.557† 0.338 0.573
Times they shopped in a grocery store in last month 33.8% 41.0% 29.1% 35.0% -0.236 0.168 0.790 0.293† 0.168 1.340 -0.039 0.336 0.961
Times they ate in restaurants in last month 39.4% 36.7% 35.4% 36.9% -0.080 0.165 0.923 -0.025 0.165 0.975 0.178 0.329 1.195
†=p
Conscien?ousresponding:Straightlining
• Q:supportforvariousdietaryprac?ces(ea?ngredmeat,limi?ngfastfood,etc.)
» stronglyfavor» somewhatfavor» neitherfavornoroppose» somewhatoppose» stronglyoppose
• Wedefineanswersinbaeeryas“straightlining”whenatleast6of7responsesarethesame
• Significantlylessstraightliningintextthanvoice
37
Table 5. Disclosure effects for each question.
Percent people reporting… Human Auto Human Auto Estimate SEOdds ratio Estimate SE
Odds ratio Estimate SE
Odds ratio
Having smoked at least 100 cigarettes in their entire life 39.2% 34.0% 42.4% 50.3% 0.404* 0.162 1.497 0.054 0.162 1.055 0.547† 0.326 1.727
Exercising less than 1 time per week in a typical week 13.1% 12.6% 21.5% 29.3% 0.838*** 0.212 2.312 0.239 0.206 1.270 0.462 0.425 1.587
Having had 3 or more sexpartners in the last 12 months 7.6% 10.1% 13.6% 14.3% 0.520* 0.257 1.681 0.160 0.254 1.174 -0.261 0.518 0.770
Personally watching television for five or more hours on the average day 10.7% 9.5% 15.9% 15.3% 0.499* 0.243 1.647 -0.083 0.239 0.921 0.084 0.486 1.087
Having had one or more drinks of analcoholic beverage on morethan 15 days of the past 30
10.6% 11.4% 8.2% 19.1% 0.247 0.244 1.280 0.546* 0.248 1.727 0.890† 0.504 2.436
Never attending religious services 32.7% 44.7% 37.6% 44.0% 0.088 0.163 1.092 0.385* 0.164 1.469 -0.243 0.327 0.785
Never reading the newspaper 16.9% 29.6% 14.6% 27.4% -0.134 0.194 0.875 0.759*** 0.198 2.136 0.069 0.397 1.071
Smoking every day 13.8% 13.2% 9.5% 16.6% -0.040 0.235 0.960 0.283 0.236 1.327 0.684 0.477 1.892
Having ever, even once,used marijuana or hashish 58.8% 54.7% 65.0% 61.9% 0.281† 0.163 1.324 -0.148 0.163 0.862 0.034 0.326 1.034
Having had 5 or more drinks on the same occasion on more
than 3 days of the past 3010.6% 12.0% 8.9% 11.5% -0.115 0.257 0.892 0.202 0.258 1.224 0.147 0.517 1.159
Having had more than 30 female partners since their 18th birthday (among straight men
and homosexual or bisexual women)16.1% 11.0% 10.3% 9.3% -0.366 0.346 0.694 -0.299 0.344 0.741 0.334 0.698 1.396
Having had more than 25 male partners since their 18th birthday (among straight women
and homosexual or bisexual and men)9.7% 9.1% 10.5% 12.0% 0.203 0.381 1.224 0.046 0.380 1.047 0.222 0.763 1.248
Having had sex 4 or more times a week during the last 12 months 3.9% 9.7% 9.7% 9.7% 0.391 0.297 1.479 0.391 0.297 1.479 -0.999 0.628 0.368
Describing themselves as homosexual,gay, lesbian, or bisexual 9.5% 10.8% 7.1% 10.9% -0.134 0.272 0.875 0.297 0.274 1.345 0.338 0.551 1.403
†=p
Whataccountsfortextvs.voicedifferencesinprecisionanddisclosure?• Couldbeanyorallofthemanydifferencesin/mingandbehaviorbetweentextandvoiceinterviews– aloneorincombina?on
• Plausiblecontribu?ngfactorsinclude:– Textreducesimmediate?mepressuretorespond,soRhasmore?metothinkorlookupanswersàCouldexplaingreaterprecision(lessrounding)intext
– Textreduces“socialpresence”• ReducedsalienceofI’sabilitytoevaluateorbejudgmental?• NoimmediateevidenceofI’sreac?on?àCouldexplainmoredisclosureintext
Experimentaldesignhelpsruleinorruleoutaccounts
• e.g.,maybeR’sroundlessintextbecausetextI’sneverlaugh(noLOL’sorhaha’s)– Maybelaughterinvoiceinterviewssuggeststhatcasualresponsesaresufficient
– Butthatcan’tbeitbecauseR’sroundjustasmuchinHumanandAutoVoiceinterviews,andautomated“interviewer”neverlaughed
0"
0.5"
1"
1.5"
2"
2.5"
3"
3.5"
Text" Voice"
Human"
Automated"
Examples:Textvs.voiceinterac?ons
HUMANTEXT HUMANVOICE
1 I: Duringthelastmonthhowmanymoviesdidyouwatchinanymedium?
1 I: Duringthelastmonth,howmanymoviesdidyouwatchinANYmedium.
2 R: 3 2 R: OH,GOD.U:hman.That’salot.HowmanymoviesIseen?Like30.
3 I: 30.
Totalelapsed>meun>lnextQ:1:21 0:12
Examples:Textvs.voiceinterac?onsHUMANTEXT
1 I: Duringthelastmonthhowmanymoviesdidyouwatchinanymedium?
2 R: Medium?
3 I: Here’smoreinforma?on.Pleasecountmoviesyouwatchedintheatersoranydeviceincludingcomputers,tabletssuchasaniPad,smartphonessuchasaniPhone,handheldssuchasiPods,aswellasonTVthroughbroadcast,cable,DVD,orpay-per-view.
4 R: 3
Totalelapsed>meun>lnextQ:2:00
HUMANVOICE
1 I: *Duringthelast*
2 R: Huh?
3 I: Oh,sorry.Um,duringthelastmonth,howmanymoviesdidyouwatchinANYmedium.
4 R: Oh!Let’ssee,whatdidIwatch.Um,shouldIsayhowmanymoviesIwatchedorhowmanymovieswatchedme?[laughs]Allrightlet’s-letmethinkaboutthat.IthinkyesterdayIwatchedu:m,notinitsen?retybutyouknow,comingandgoing.Mykidsarewatchingin.Um,Idon’tknowmaybe2or3?mesaweekmaybe?
Examples:Textvs.voiceinterac?onsHUMANVOICE
5 I: Uh,sowhatwouldbeyourbestes?mateonhowmany,um,yousawinthewholemonth.
6 R: [pause]Um,Idon’tknowI’dsaymaybe3moviesifthatmany.
7 I: 3?
8 R: Isthatgoingtothemoviesorwatchingthemoviesontv.Likeyousaid*anymedium*right?
9 I: That’s*anymovies.*Yep.
10 R: Maybe1or2amonthI’dsay.
11 I: 1or2amonth?[breath]Uh,sowhatwouldbe*closer*
Examples:Textvs.voiceinterac?onsHUMANVOICE
12 R: *Yeah,because*Iuh,um,occasionallyItakethekidsonaTuesdaytoseeamovie,dependingonwhat’splaying.SoI’dmaybeonceortwiceamonth
13 I: Whichwouldbecloser,onceortwice.
14 R: Iwouldsaytwice.
15 I: Twice?
16 R: R:Mhm.Becauseitruns4Tuesdayswhichischeapertogo
17 I: Right
18 R: R:soI’dsaytwice,yah.BecauseIdotakethemtwice.Notlastmonthbutthemonthbefore
Totalelapsed>meun>lnextQ:ß
1:36
ExamplesfromAutomatedVoiceI:Howopendoyoureadthenewspaper?'Everyday','afew?mesaweek','onceaweek','lessthanonceaweek',or'never'?R:EverydayI:Gotit.I:Thinkingaboutthe?mesinceyoureighteenthbirthday(includingtherecentpastthatyou'vealreadytoldusabout),howmanymalepartnershaveyouhadsexwith?R:NoneI:Ithinkyousaid'9'.Isthatright?YesorNo.R:NoI:Thinkingaboutthe?mesinceyoureighteenthbirthday(includingtherecentpastthatyou'vealreadytoldusabout),howmanymalepartnershaveyouhadsexwith?R:ZeroI:Thanks
Numerical
FirstHypothesis:“Nine”
LastHypothesis:“Zero”
LastAnnota?on:“Zero”
Categorical
ExplicitConfirma>on
Median?meperques?onandanswer
Mul?tasking(self-reported)
Cannonresponseornoncomple?onexplainthepaeerns?
• Werepeoplewhoweremoreconscien?ous(lesslikelytoroundtheiranswers)ormorewillingtodisclosesensi?veinforma?onactuallylesslikelytostartorfinishinterviewsinvoicemodesthanintextmodes?
• Couldourmodeeffectsresultnotfromthecontribu?onofrespondersandcompleters,butinsteadfromthenon-contribu?onofnon-respondersandnon-completers?
Studydesignallowslookingatthisinafocusedway
• alloursamplemembershadalreadyindicated,byscreeningintothestudy,interestinandatleastsomecommitmenttopar?cipa?nginaninterviewontheiriPhone(inanunspecifiedinterviewmode).
• Thefactthatourpar?cipantswererandomlyassignedtoaninterviewingmodemeansthattheirini?a?vewasunlikelytohavedifferedacrossthemodes.
Nonresponse?
• noevidencethatdifferentkindsofpeople(age,gender,ethnicity,race,educa?on,income)fromoursamplewereanymoreorlesslikelytostarttheinterviewsinthedifferentmodes
• Implausiblethatanotherfactorcouldexplainpaeern:– wouldrequirethattendencyofRstogiveimpreciseanswersandreluctancetoengageinatextinterview(butwillingnesstoengageinavoiceinterview)wouldhavethesameorigin
Noncomple?on?• Comple?ongreaterinhumanthanautomatedinterviews
• Nodifferencebetweentextandvoice• àUnlikelytoaccountforvoicevs.textdifferences• Fornoncomple?ontoaccountfordisclosure,wouldrequireasystema?creversalofthepaeernofdisclosureobservedforthosewhocompletedandthosewhobrokeoff– thosewhobrokeoffwithautomatedinterviewerswouldhavetobethosewhohadlesstodisclose
– Butonewouldthinkthatpeoplewhobreakoffwouldbethosewithmoretodisclose
Prefertext(vs.voice)forfutureiPhoneinterview?
0
10
20
30
40
50
60
70
80
90
100
Text Voice
Percen
t Human
Automated
Othersa?sfac?onmeasures
• MostRsfoundinterviewveryorsomewhateasy– Morefoundspeech-IVRsomewhathard
• Futureinterviews:– TextRsoverwhelminglypreferredfutureinterviewintextvs.voice
– VoiceRspreferredvoice,butlesssoifspeech-IVR
Summary:Voicevs.Text• Textinterviewsproducehigherdataquality:greaterdisclosure,lesssa?sficing,highsa?sfac?on
• Eventhough(orbecause?)theytakelonger• Eventhoughdataarelesssecure(morepersistentandtraceable)thanvoice– Perhapsbecauseofdifferent?mepressurethanvoice?– PerhapsbecauseofconvenienceofansweringwhenandhowRwants?
– Perhapsbecauseofgreatersocialdistancewithinterviewer?
• Caveat:weimplementedtextinterviewsinonepar?cularway,withsingle-characterresponses
Summary:Humanvs.AutomatedInterviewer
• Automatedinterviewsonasmartphone(inthesemodes)canleadtodataatleastashighinqualityasdatafromhumaninterviewsinsamemodes– Nomoresa?sficingthanwithhumaninterviewers!– Moredisclosure
• Tradeoffs– Fieldperiodcanbeshorter– interviewscantakelonger– Higherbreak-off– requireaddi?onaldevelopmenteffort,especiallyspeech-IVR
• Caveat:weimplementedonepar?cularversionofspeech-IVR;otherscoulddiffer
ModeChoiceStudyConradetal.(underreview)
• Ismessage– Urgentorcanitwait?– Sensi?veornot?– Shortvs.long?
• WillIbemul?tasking?Ifso,whatelsewillIbedoing?• Whatmodewillbeeasiestorleastdisrup?veforpartner?• IsseWngpublicvs.private,noisyvs.quiet,brightvs.dim?• Whatismygenerally(chronically)preferredwayofcommunica?ng?
– e.g.,talkingvs.tex?ng
• Sopeoplecanusethesamedevice,forexample,torespondto– avoicecallwithatextmessage– atextmessagewithaFacebookpost– emailwithavoicecall
48
Onsmartphones,peoplechooseandswitchmodestofitneeds
Implica?onsforSurveyPrac?ce
• Nowpossibleformembersofpublictochooseoneofmanysurveymodesonasingledevice– immediatelyandconveniently
• Notofferingachoicecoulddeterpar?cipa?onbysmartphoneusers– orreducemo?va?onwhenansweringques?ons
49
Inothertasks,choiceseemstohelpandhurt
• Choiceenhancesintrinsicmo?va?on(byincreasingautonomy)andperformance– Patalletal.(2008)meta-analysis:78of91effectsofchoiceonintrinsicmo?va?onareposi?ve
• Toomanyop?ons(overload)leadstonochoice(paralysis)andreducedsa?sfac?onwithchoices– IyengarandLepper(2000):par?cipantsmorelikelytopurchasegourmetjams/chocolatesortocompleteop?onalassignmentswhenoffered6vs.24or30choices
• Howdoeschoiceaffectsurveypar?cipa?on?
50
SurveyModeChoice• Toincreasepar?cipa?on,researchersofferpoten?alrespondentsachoiceofmodes– e.g.,mailpaperques?onnaireandgiverandomhalfchoiceofcomple?ngonline;requiresextrastepoftypingURLintobrowser
• Butthiskindofchoiceseemstoreducepar?cipa?on:– Fulton&Medway(2012)meta-analysisof19mail/webchoicestudiesfindsthat,comparedtonochoice,modechoicereliablyreducespar?cipa?onby3.8%
– suggestcouldbeduetoParadoxofChoice(Schwartz,2009)orcostsofswitchingfrominvita?ontointerviewmode
• Choiceonsingledevicesimplifieschoiceimplementa?on
51
Currentstudy
• Examineshowmodechoiceonasingledeviceaffects– Par?cipa?on– Dataquality(rounding,straightlininganddisclosure)
– Rsa?sfac?on
• Same4modes,same32items
52
Possibleoutcomes:Par?cipa?on
• IfRscanChoose– Mightreducepar?cipa?onbecause
• Increasedcomplexity(Schwartz,2004;FultonandMedway,2012)• Breakinresponseprocess(Fulton&Medway,2012)
– Mightincreasepar?cipa?onbecause• Canchooseamodethatissuitablegiventheircurrentenvironmentandotherdemands(e.g.,whethertheycantalknow)
Possibleoutcomes:Conscien?ousresponding
• IfRscanChoose– mightprovidefewerconscien?ousanswersbecausetheychooseamodeinwhichit’seasiertotakeshortcuts
• e.g.,anautomatedmodebecausenohumaninterviewertopressthemtoworkhard
– mightprovidemoreconscien?ousanswersbecausebeingabletochoosemayincreasetheircommitmenttothetask
• Mayincreasemo?va?on
Possibleoutcomes:Disclosure
• IfRscanChoose– mightdisclosemorebecausechoosemoreprivatemodewithfewersocialcues
• e.g.,Automatedtext– mightdiscloselessbecausechoosemoreconvenient,fastermodewithmoresocialcues
• E.g.,humanvoice
Possibleoutcomes:Sa?sfac?on
• IfRscanChoose– Mightreducesa?sfac?onbecause
• Addingop?onsincreasesR’sexpecta?ons(Schwartz,2004)• Leadstoregretovernotchoosingimaginedalterna?ve(Schwartz,2004)
– Mightincreasesa?sfac?onbecausepeopleperceivethechosenalterna?veasmoreaerac?ve(Fes?nger,1948;Cooper,2007)
– Orjustbecausemoreconvenientandeasier!
ExperimentalCondi?ons
1. AssignedMode(NoChoice)• Rsrandomlyassignedtoamode• Contactedandinterviewedinsamemode
2. Choice• Rsrandomlyassignedtoacontactmode• Requiredtochooseinterviewmode
– Couldchoosecontactmodeoranyofotherthree– Makesexplicittheirmodechoiceinten?on
57
ModeComparisonExperiment
ModeChoiceDesignandImplementa?on(2)
• ModeChoiceintroduc?on:“Togetstarted,weneedyoutochoosehowyouwanttobeinterviewed--whateverworksbestforyou.Therearefourchoicesandanychoiceisfinewithus.Doyouwantto‘talkwithaperson’,‘talkwithanautomatedinterviewer’,‘textwithaperson’,or‘textwithanautomatedinterviewer’?
• Withineachcontactmode,orderofinterviewmodeop?onsrotatedacrossRs(16orders)
58
Respondents:1260iPhoneusers• AssignedMode(NoChoice):n=634
– n=157to160permode– InterviewedMarch–May,2012
• Choice(AbletoChooseInterviewMode):n=626– n=149to170permodeofcontact– InterviewedJuly–September,2012
• RecruitedfromCraigslist,Facebook,GoogleAds,andAmazonMechanicalTurk– Webscreenerverifiedage(>21years)andUSareacode– iPhoneusageverifiedviatextmessagetodeviceanduseragentstringinresponse
• $20iTunesgipcodeincen?ve,providedaperpost-interviewwebques?onnaire
• Age,gender,ethnicity,income,educa?onnotreliablydifferentbetweenAssignedModeandModeChoicegroups
• 13Umich/SRCinterviewers:– 5onlyinAssignedModecondi?on,3onlyinModeChoice,5inbothcondi?ons
59
Par?cipa?on• Doessimplybeingpresentedwithachoicereducepar?cipa?on?
– SlightlyfewerRsinModeChoicecondi?on(52.1%)choseamodethanansweredthefirstques?oninAssignedModecondi?on(55.9%).
• Doeschoosingamodereducecomple?on?– Overall,46.4%ofRsinChoicecondi?onvs.50.5%inAssignedMode
condi?oncompletedques?onnaire(RR1)
• Inallcases?– WhenRschosetostayincontactmode48.3%completedinterview,
notdifferentfrom50.5%AssignedMode– Modechoiceinautomatedmodeshasnoimpactoncomple?on
(43.4%vs.44.0%)
• Howdoesmodechoiceaffectbreakoffs?– MoreRswhochoseaninterviewmodecompletedtheinterview
(94.9%)thanthosewhowereassignedamode(90.4%)
YES
NO
ITREDUCESBREAKOFFS
YES
Par?cipa?on
55.9%
90.4%
48.9%
94.9%
0102030405060708090
100
Startinterview(answerQ1) Completeinterviewoncestart
Percen
t
AssignedMode
ModeChoice
• Overallcomple?onhigherwithout(50.5%)thanwithchoice(46.4%)• Noimpactonkindsofpeoplewhopar?cipatesochoiceprobablydoesnot
introducenonresponsebias 61
Breakoffaperchoicebutbeforeinterview
0.7%
11.1%
0
2
4
6
8
10
12
StayinMode(n=301) SwitchMode(n=388)
%don
’tan
swerQ1aW
er
choo
singm
ode
62
Bothgroupsmakechoicesoincreasedbreakoffswhenchoicerequiresswitchingmodesduetotoswitchingcosts,notParadoxofChoice
HumanVoice HumanText
Whatmodeswerechosen?
0
50
100
150
200
250
300
SwitchintoMode
StayinMode
AutomatedText
n=170
n=150 n=157 n=149
OriginalSampleSize(beforemodechoice)
AutomatedVoice
Num
bero
fRs
63
DataQuality:Rounding
• Wedefineroundinghereasnumericalanswersdivisibleby10– HowmanysongsdoyoucurrentlyhaveonyouriPhone?
• Exampleroundedanswer:1100• Exampleunroundedanswer:1126
64
AverageNumberofRoundedNumericalAnswers
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
AssignedMode ModeChoice
Voice
Text
Human HumanAutomated Automated
Effectofchoicenotduetopar?cularchoiceofmode:lessroundingwithchoicethanwithoutapercontrollingformode,p=0.008
2.58
p
Rounding:“Numberof?mesea?nginrestaurants”
0
5
10
15
20
25
AssignedMode ModeChoice
Percen
tRsrep
or>n
groun
ded
answ
er
*During the last month, how many times did you eat in restaurants?
p<0.01
66
Rounding:“NumberofsongsonyouriPhone”
40
42
44
46
48
50
52
AssignedMode ModeChoice
Percen
tRsrep
or>n
groun
ded
answ
er
*How many songs do you currently have on your iPhone?
p=0.02
67
PercentofRsstraightlining
0%
2%
4%
6%
8%
10%
12%
AssignedMode ModeChoice
Voice
Text
Human HumanAutomated Automated
Effectofchoicenotduetopar?cularchoiceofmode:marginallylessstraightliningwithchoicethanwithoutapercontrollingformode,p=0.085
6.78%
p=0.029
3.99%
68
AveragenumberofSociallyDesirableAnswers
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
AssignedMode ModeChoice
Voice
Text
Human HumanAutomated Automated
2.71 2.83
Num
berofAnswers
1
Figure4.Disclosure:Averagenumberofresponsesdeemedmostsociallyundesirable:abovethetopdecileforconKnuousnumericalresponsesorthemostextremecategoricalresponseopKoninthesKgmaKzeddirecKon.
Responsesdeemedmostsociallyundesirable:abovetopdecileforcon?nuousnumericalresponsesormostextremecategoricalresponseop?onins?gma?zeddirec?on
Sa?sfac?onhigherwithmodechoice
0
10
20
30
40
50
60
70
80
AssignedMode ModeChoice
%Verysa>sfie
d
p<0.001
Overall,howsa?sfiedwereyouwiththeinterview?
70
Reasonsforchoosingmodes
*Why did you choose this interviewing method?
Mostcommoncategories %ProvidingReason
Ease/simplicity 33.8%Convenience/flexibility 22.8%Quickness(shortestinterview?me) 10.3%Privacy 9.8%Liketex?ng 9.0%Environment--loca?on 8.8%
Threecoders;Agreement=98.1%
• Codedopen-endedanswersinto29categories
71
Reasonsforchoosingmodes*(examples)
• Humanvoice:– “Morecomfortablespeakingwitharealperson”
• Humantext:– “IchosetotextbecauseIhadasmallchildwithmeinmyhomeduringthe
interviewandcouldnothaveconcentratedontheques?onsifitwasonthephone.”
– “Toavoidbackgroundnoiseandtoclearlyunderstandtheques?onandtakemy?metoanswerit.”
• Automatedtext:– “Iamatworkandwouldn'talwaysbeabletoanswerques?onsifIspoketo
someoneonthephone.”– “BecauseIdidn'twanttotalkonthephoenordidIwanttotextaperson
simplybecausIknewsomeofmyresponseswouldhavebeenalielelate”• AutomatedVoice:
– “ididn'twanttotalktoanyonebut,IwasdrivingsoIcouldn'tlookatascreen”
– “Talkingtoanautomatedpersonwaslesspersonal”
*Why did you choose this interviewing method? 72
Summary• Modechoiceproduced:
– lessrounding– lessstraightlining– fewerbreakoffs– higherRsa?sfac?on
• Choicedidnotaffectpaeernsoftextvs.voiceforrounding,straightlining,disclosure
• Par?cipa?on– Lowerstartandcomple?onrateswithchoicethannot– Mostlyduetowhetherchoiceinvolvesmodeswitch– Rswhostartaperchoosingmodemorelikelytocomplete
73
Manyques?onsremain(overall)• Dodifferentdemographicsubgroups(e.g.,age,income,educa?on)varyindisclosure,effort,preferences?
• Generalizabilitytoothermobileplacorms?Tolesssmartmobilephones?
• Generalizabilitytoanon-convenienceornon-incen?vizedsample?
• Dorespondentswanttobeabletoswitchmodesmid-interviewwhencircumstanceschange(mobile,noisy,private,etc.)?
• HowmanyQscanbeaskedviatextinterviews?
Implica?ons
• Tex?ngisworthexploringfurtherasamodeofsurveydatacollec?onforFederalsta?s?csandsocialscienceresearch
• Asynchronous,less-?me-pressuredrespondingmayreallybebeeerthanusualmodes– Raisesques?onofwhetherFTFortelephonemodesshoulds?llbeconsideredthegoldstandardinasmartphoneera
– Andwhether“best”modevariesfordifferentresearchques?onsorpar?cipants
Implica?ons(2)
• Mul?taskingwhileansweringsurveyques?onsdoesnotnecessarilyleadtopoorerdataquality
• Maywellenhancerespondents’sa?sfac?onandwell-beingbyallowingthemtorespondwhereandwhentheyfinditconvenient
Implica?ons(3)
• Poten?albenefitsofautoma?onforsocialmeasurementextendtotheuseofapersonalportabledevicesdespitethevaryingcontexts(publicandprivate)inwhichthedeviceisused
Implica?ons(4)
• Offeringrespondentsamodechoiceonasingledevicemayhaveimportantbenefits
• Butnotallmodetransi?onsarethesame• Differentdesignsolu?onswillbeneededfordifferentmodetransi?ons
78
Specula?on:ANewTakeonStandardiza?on
• Shouldourtradi?onalone-size-fits-allapproachtocollec?ngself-reportdataberethought?
• Maybedifferentmodesfordifferentpeopleondifferentoccasionscanincreasecomparabilityoftheirresponses
• Maybewhatisneededisstandardizingpar?cipants’experience– enhancingeveryone’sabilitytofocusonthetaskinawaythatsuitstheirpreferencesandcircumstances
• Smartphones–mul?modal,mobiledevices–maybeforcingustothinkthisway
Thankyou!
80
Recommended