Voice recognition in self-administered mobile surveys' - Free University of Bozen-Bolzano and Globalpark (Mobile Research Conference 2011)

Globalpark. Manage what matters.April 19, 2011 Mobile Study III1

VOICE RECOGNITION IN SELF-ADMINISTERED MOBILE SURVEYS

LORENZ GRÄF, SVEN SCHERRER, MARC HAUPT, MICHAEL BOSNJAK, MARIO CALLEGARO


AGENDA

1.Background2.Research Questions3.Method4.Results5.Discussion and Outlook


AGENDA


Globalpark. Manage what matters.April 19, 2011 Mobile Study III

USABILITY OF MOBILE SURVEYS: PAST FINDINGS

4

Einfachauswahl untereinander

Mehrfachauswahl untereinander

Geschlossene Auswahlliste

Textfeld einzeilig

Fragetyp mit Bild

65,00 73,75 82,50 91,25 100,00

87,9

74,7

82,7

87,3

89,2

Usability score (Range: 0-100 Punkte)

Frag

etyp

Subjective Usability AssessmentPost-hoc survey (Web) one week after mobile survey completion

Indicators for usability score: fluency, simplicity, ease of use

Observed

Item-NR

Drop-Out

45%

9%

9%

23%

Multiple choice

Single choice

Drop-Down menu

Textfield

Image map


AGENDA



OVERALL GOAL Improving mobile input formats requiring rich, non-standardized information, which is usually captured using text input in other self-administered survey modes. Alternative to (typed) text-input formats:

Embedding voice input into self-administered mobile surveys, similar to CATI or esp. IVR surveys

Today: Subjective usability assessments as indicators of a potential improvement text versus voice Next step: Analyzing objective / unobtrusive data on nonresponse (item-nonresponse, drop-out)


TECHNICAL IMPLEMENTATION: IPHONE APP


TECHNICAL IMPLEMENTATION: ANDROID


RESEARCH QUESTION(S) TESTEDDifferences between

iPhone-App-based voice capturingversusAndroid-based voice recognitionversus

Classical (touchscreen-based) text inputon the following dependent variables:

usability assessment immediately following a specific voice/text input format, (objective indicators of nonresponse, esp. item-nonresponse and drop-out,)

retrospective usability assessement Ø 5 days following participation?


AGENDA


11 Globalpark. Manage what matters.

MINGLE PANEL – MOBILE USERS

04/11/11 Mobile Study III

N=10518


MINGLE PANEL – SMARTPHONE OS


N=3136


PARTICIPANTS


PARTICIPANTS

79,2 %

14,2 %6,6 %Sales

EmployedStudentUnemployed

Employment Status


PROCEDUREPart 1: Pre-Survey (23.03. – 28.03.2011, 6 Days, time of invitation: 5 PM GMT)• Device Screening

• Android: Technical requirements check

• iPhone: Instructions App installation

• Pre-commitment

Part 2: Main survey (28.03. – 03.04.2011, 7 Days, time of invitation: 5 PM GMT)

• Test drive: Voice recording / voice recognition

• Three text/voice input items (short: single number, medium: list three items, long answers: elaborated text/answer)

• Usability assessment following each text/voice input item

Part 3: Post survey (05.04. – 09.04.2011, 5 days, time of invitation: 8 AM GMT)

• Retrospective usability assessment

• Evaluation of Main-Survey Drop-Out• Preferred input method short / long entries


PROCEDURE (PARTS 2 AND 3)

16

Respondi panel members (N= 235)

iPhone App Voice Capturing Group

(n1.1= 126)

Android Voice Recognition Group

(n2.1= 60)

Android Text Input Group (n3.1= 49)

Immediate usability assessment for three corresponding questions.

(n1.2= 87) (n2.2= 36) (n3.2= 38)

Retrospective usability assessment (after Ø 5 days, Web-based).

(n1.3= 79) (n2.3= 36) (n3.3= 34)


DEPENDENT MEASURESImmediate usability assessment: Mean score (transformed to 0 -100 range) of three identically worded, closed-ended single items evaluating the extent to which (iPhone) voice recording / (Android VR) voice recognition / (Android text) text input was appropriate to capture the intended response („How do you evaluate the performance of the voicerecognition/-capturing /touchscreen keyboard feature?“, scaled from ´very good´ thru ´very poor´).

Retrospective usability assessment (reminder screenshots were displayed):Mean score (transformed to 0 - 100 range) of the following four indicators: ease of use, fluency, usability, unintended entries .

Non-response (work in progress):

Item-nonresponse: Aggregated item-nonresponse percentage for all three questions with experimentally manipulated input formats (per experimental condition)

Drop-Out: Overall drop-out rates (per experimental condition)

17


AGENDA



USABILITY ASSESSMENT; IMMEDIATE

Immediate usability assessment best and most homogeneous for Android-based

text input (!)


USABILITY ASSESSMENT: RETROSPECT

Retrospective usability assessment best for

Globalpark´s iPhone-App-based voice capturing

technology.


MISC: INPUT FORMAT PREFERENCES

21

Physical KeyboardVoice Capt/RecTouchscreenDon't Know

Short Entry Long Entry0 %

50 %

100 %

Short Entry Long Entry

57 %76 %

9 %

9 % 33 %13 %

iPhone App Android


SUMMARY OF RESULTSImmediate usability assessment:

Android touchscreen > better than > iPhone App VC > better than > Android VR

Android touchscreen ratings most homogeneous (largest agreements)

Retrospective usability assessment:iPhone App VC > better than > Android touchscreen > better than > Android VR

Android VR shows largest dispersion among ratings (largest disagreement)

Additional result:The larger the amount of text information to be entered, the more preferred are voice input formats on the expense of the touchscreen-based version, but only for iPhone participants (not for Android participants).


AGENDA



DISCUSSION AND OUTLOOKFurther analyses using data from the current study:

Item-Nonresponse analysis

Drop-Out analysis (partly done, see Appendix)

Quality of content between groups?

Technology and/or selection effects? For instance,iPhone users are more experienced with Apps and more ´attached´ to their Smartphone, ´causing´ more favourable post-hoc evaluations of App-based input formats;men versus women systematically overrepresented in certain groups, other systematic differences?

Next steps:

Results consistent over time and in the context of ´real life´ applications?

Please feel free to use VC/VR in your own studies!


THANK YOU!


Appendix


DROP-OUT (WORK IN PROGRESS)Voice

CapturingVoice

RecognitionRegular

Test Drive Instructions

9 (9,3%) 5 (10,4%) -

Test Drive 1 (1,0%) 0 (0%) 0 (0%)

Single figure 0 (0%) 0 (0%) 0 (0%)

List 0 (0%) 0 (0%) 0 (0%)

Free text 0 (0%) 0 (0%) 0 (0%)Drop Out on other pages

0 (0%) 7 (14,6%) 2 (5%)

Drop Out Overall 10 (10,3%) 12 (25%) 2 (5%)

N= 97 (100%) 48 (100%) 40 (100%)


Comments and Data Samples


Comments Voicecapturing„Da ich nicht sprechen konnte, konnte ich den Fragebogen nicht zu Ende durchgehen.“

„Ich war krank, böse Erkältung und konnte mich nicht motivieren, zumal mir das Sprechen superschwer fiel.“ Comments Voicerecognition

„Spracherkennung ist bei meinem Handy für solche Felder nicht verfügbar, nur für die Google-Suche“

„Ich hatte nur die englische Spracheingabe. Habe im Internet nach Hilfestellungen gesucht um auf deutsche Spracheingabe umzustellen, allerdings nichts gefunden.“

„Spracheingabe funktionierte nicht richtig. Spracheingabe ging nur als englischer Text. Alles Einstellungen meines HTC waren auf Deutsch und ich habe keine Möglichkeit gefunden die Spracheingabe auf Deutsch zu berichtigen.“

Comments Touchscreen Input

„Gute Erhebungstechnik, gerne wieder über smartphone...“

„Würde mich sehr freuen weitere Umfragen mit dem Handy beantworten zu dürfen.“ Data Samples Voicerecognition – Testphrase „Diese Befragung ist sehr interessant“

„Die herbe frau geküsst interessant“

“These are The 5 sense sons”

“disappear.com sia into the sun”

„Die befragung sea of sand“

„wiese befragung ist sehr interessant“


Demo - Audio Samples iPhone


Apple Quicktime required

0: Testtrial

mingle61 (optimal)

0: Single Word

mingle 99 (Optimal)

mingle 166 (Background noise)

mingle 173 (Additional info)

2: List

mingle31 / 19 (optimal)

mingle16 (Background noise)

mingle8 (Real List)

3: Open answer

mingle198 (Slight saxon dialect)

mingle187 (Whole sentence)

mingle192 (slightly annoyed)