Fishing or a Z?: Investigating the Effects of Error onMimetic and Alphabet Device-based Gesture Interaction"

Abdallah  ‘Abdo’  El  Ali      Johan  Kildal  

Vuokko  Lantz    

Oct.  23,  2012

h6p://staff.science.uva.nl/~elali/  

2

Outline"

I.  Introduc5on  

II.  Methods  

III.  Results  

IV.  Discussion  

V.  Future  Work  

3

4

Introduction

Introduction"

  Device-­‐based  gestures:      Gesturing  by  moving  a  smartphone  

device  in  3D  space    Research  seGngs,  home  

environments,  everyday  mobile  interac5on  

  Alterna5ve  to  when  users  are  encumbered  (e.g.,  manual  mul5tasking)  

  Natural  

  Promising  alterna5ve  to  mobile  touchscreen/keyboard  input  under  situa5onal  impairments  

5

Motivation"

  But  errors  are  an  inevitable  part  of  interac5on  with  technology…  

  Many  gesture  classes  are  available  (e.g.,  iconic,  symbolic,  deic5c)  for  use  in  smartphones,  but  which  have  minimum  user  frustra5on  when  recogni5on  errors  occur?  

  We  inves5gate  user  error  tolerance  for  two  iconic  gesture  sets  used  in  HCI:  mime5c  and  alphabet  gestures  

6

Related Work"

  Gesture-­‐based  Interac5on    Gesture-­‐Task  mapping  (Khnel  et  al.,  2011;  Ruiz  et  al.,  2011)    Social  acceptability  (Rico  &  Brewster,  2010)    Gesture  Taxonomies:  Deic5c,  symbolic,  physical,  mime5c/pantomimic,  abstract  (‘  \m/  ’)  

(Rime  &  Schiaratura,  1991;  …)  

  Recogni5on  Errors    Speech:  Repeat  (Suhm  et  al.,  2001)  and  hyperar5culate  (Oviad  et  al.,  1998)    Mul5modal:  Modality-­‐switching  (“spiral  depth”  of  6)  (Oviad  &  van  Gent,  1996)    Touch-­‐less  Vision-­‐based  Gesture  Recogni5on:  How  many  errors  before  switching  to  

keyboard  input?  40%  user  error  tolerance  before  modality  switching  (Karam  &  Schraefel,  2006)  

  Lidle  work  on  which  gesture  sets  are  most  robust  to  errors  during  gesture-­‐based  interac5on!  

7

Iconic Gestures"

8

  Mimetic / Pantomimic Gestures: natural, familiar, easy to learn, varied by activities   e.g., Fishing, calling, …

  Alphabet Gestures: familiar, easy to learn, varied by stroke   e.g., letter “C”, letter “S”, …

Research Question"

 What  are  the  effects  of  unrecognized  gestures  on  user  experience,  and  what  are  the  differences  between  mime5c  and  alphabet  gestures  (under  varying  error  rates:  0-­‐20%,  20-­‐40%,  40-­‐60%)?  

 Hypotheses:    

   Mime5c  gestures  →  users  less  familiar  with  ideal  shape  →  more  gesture  varia5on  under  high  error  rates  →  but  lower  subjec5ve  workload  due  to  higher  degrees  of  freedom    

   Alphabet  gestures  →  users  more  familiar  with  ideal  shape  →  more  rigid  gestures  under  increasing  error  rates  →  but  higher  subjec5ve  workload  due  to  lower  degrees  of  freedom  

9

10

Methods

Mimetic Gesture Design"

11

Alphabet Gesture Design"

12

Study Design"  Qualita5ve  Study;  Automated  Wizard-­‐of-­‐Oz  method  (Fabbrizio  et  al.,  2005)  

  24  subjects  (16  male,  8  female)  aged  between  22-­‐41  (M=  29.6,  SD=  4.5)  

  Mixed  between-­‐  and  within  subject  factorial  design:  2  (gesture  type:  mime5c  vs.  alphabet)  x  3  (error  rate:  low  vs.  med  vs.  high)  

  Experiment  in  Presenta5on®,  Wii  Remote®  interac5on  using  GlovePie™  

  Random  error  distribu5on  across  trials  (20  prac5ce,  180  test)  

  Tutorial  &  videos  given  of  how  to  'properly'  perform  each  gesture  

  Data  collected:  1.  Modified  NASA-­‐TLX  workload  [0-­‐20  range]  ques5onnaire  data  (Hart  &  Wickens,  1990;  Brewster,  

1994)  2.  Experiment  logs    3.  Video  recordings  of  subjects’  gesture  interac5on  4.  Post-­‐experiment  interviews  13

14

!"#$%$&'()*$+,

!" !#

-./&%$0,

'1)23#4&5'$*67.%*6

!" !# !$ !% !& !'

8'()*$+'89':".#),'2&"'()*$+'

;*<'=""*"

-&7'=""*"

>.?3'=""*"'

@'()*$+,'2&"'=""*"'$*67.%*6'A&""*"'$*67.%*6'B'4)*$+'*"7&"'$*C65&"4#)#6$&7'D'"#67*/.E#%*6F

Setup & Procedure"

15

Fishing Gesture

Personal Information

Form

t

≈1 hr

Instructions& Tutorial

PracticeBlock

Test Block

NASA-TLXQuestionnaire

Exit Interview

Reward

x3

Test Block

Perform Fishing Gesture

Perform Fishing Gesture

[...]

Experiment Session

Automated Wizard-of-Oz

16

Low High

MENTAL DEMAND

Low High

PHYSICAL DEMAND

Low High

TIME PRESSURE

Low High

EFFORT EXPENDED

Poor Good

PERFORMANCE LEVEL ACHIEVED

Low High

FRUSTRATION EXPERIENCED

Low High

ANNOYANCE EXPERIENCED

Low High

OVERALL PREFERENCE RATING

#: ____

(Hart & Wickens, 1990)

(Brewster, 1994)

NASA-TLX

17

Results

!"#$%&

'()*+,%&

-../0$

'"0./01%#,"

203*$0%$+/#

'0"."0"#,"

4356",$+7"8/09&/%:

;/<=-00/0=>%$" !":+31=-00/0=>%$" ?+@(=-00/0=>%$"

!!!

!"#$%"&'($)%*+$)

!!!!

!!!!

!!!!!

Workload"

18

!"#$%&

'()*+,%&

-../0$

'"0./01%#,"

203*$0%$+/#

'0"."0"#,"

4356",$+7"8/09&/%:

;/<=-00/0=>%$" !":+31=-00/0=>%$" ?+@(=-00/0=>%$"

!"#$%&'()*'+(,-'+

!!!

!!

!! !!

!!!!!!!!!

!!!!!!

Workload"

19

Workload"

  Mime5c  gestures  are  beder  tolerated  up  to  error  rates  of  40%  (cf.,  Karam  &  Schraefel,  2006),  compared  with  error  rates  of  up  to  only  20%  for  alphabet  gestures    

  From  a  usability  perspec5ve,  mime5c  gestures  more  robust  to  recogni5on  failures  than  alphabet  gestures  

20

Observations"

  Mime5c  gestures  evolve  into  real-­‐world  counterparts  under  error,  alphabet  gestures  tend  to  become  more  rigid  and  well  structured  

  Canonical  Varia5ons  via  posi5ve  reinforcement:  Survival  of  the  fidest  gesture  varia5ons    E.g.,  S12  exhibited  real-­‐world  varia5ons  on  both  the  Fishing  and  Trashing  

gestures  

  Varia5ons  develop  as  low  as  spiral  depth  of  2  (i.e.,  min.  2  recogni5on  errors)  

21

User Feedback"  Perceived  Canonical  Varia5ons  

  S9:  “The  shaking,  that  was  the  hardest  one  because  you  couldn’t  just  shake  freely  [gestures  in  hand],  it  had  to  be  more  precise  shaking  [swing  to  the  leA,  swing  to  the  right]  so  not  just  any  sort  of  shaking  [shakes  hand  in  many  dimensions]”  

  Cultural  and  Individual  differences      S10:  “For  the  glass  filling,  there  are  many  ways  to  do  it.  SomeFmes  very  fast,  someFmes  

slow  like  beer.”  

  Perceived  Performance  

  Ad  hoc  explana5ons  (e.g.,  fa5gue)  given  why  there  were  more  errors  in  some  blocks  

  S18:  “[Performance]  between  the  first  and  second  blocks  [baseline  and  low  error  rate  condiFons],  it  was  the  same...  10-­‐15%.”  

  Social  Acceptability    Alphabet  gestures  less  socially  acceptable  when  they  fail    S16:  “When  it  doesn’t  take  your  C,  you  keep  doing  it,  and  it  looks  ridiculous.”  

22

23

Discussion

Limitations"

  Lab-­‐study  

  Task  Independence  

  Random  error  distribu5on  

24

Implications for Gesture Recognition"

  Mime5c  gestures  evolve  into  real-­‐world  counterparts  under  error,  alphabet  gestures  tend  to  become  more  rigid  and  well  structured    

   One  shot  recogni5on  for  mime5c  gestures  important!  

  Interes5ng  explana5ons  (e.g.,  canonical  varia5ons)  and  causes  (e.g.,  fa5gue)  given  why  there  were  more  errors  in  some  blocks  

   Transparency  in  gesture  recogni5on  technology  may  beder  support  users  in  error  handling  strategies  

25

Implications for Gesture-based Interaction"

  40%  error  tolerance  in  line  with  previous  work  (Karam  &  Schraefel,  2006),  which  shows  usability  of  gesture-­‐based  interac5on  

  Mime5c  gestures  overall  have  beder  user  experience,  more  use  cases,  and  thus  more  suitable  for  device-­‐based  gesture  interac5on  (even  under  high  recogni5on  error!)  

26

27

Future Work

Future Work"

  Quan5ta5ve  assessment  of  how  many  errors  precisely  before  canonical  variant?  

  Other  classes  of  gestures  (e.g.,  manipula5ve)  

  Influence  of  device  form  factor  

28

29

Questions

h6p://staff.science.uva.nl/~elali/  

References"Brewster,  S.  Providing  a  structured  method  for  integra5ng  non-­‐speech  audio  into  human-­‐computer  interfaces.  PhD  

thesis,  University  of  York,  1994.  

Fabbrizio,  G.  D.,  Tur,  G.,  and  Hakkani-­‐Tur,  D.  Automated  wizard-­‐of-­‐oz  for  spoken  dialogue  systems.  In  Proc.  INTERSPEECH  2005  (2005),  1857–1860.  

Karam,  M.,  and  Schraefel,  M.  C.  Inves5ga5ng  user  tolerance  for  errors  in  vision-­‐enabled  gesture-­‐based  interac5ons.  In  Proc.  AVI  ’06,  ACM  (NY,  USA,  2006),  225–232.  

Khnel,  C.,  Westermann,  T.,  Hemmert,  F.,  Kratz,  S.,  Mller,  A.,  and  Muller,  S.  I’m  home:  Defining  and  evalua5ng  a  gesture  set  for  smart-­‐home  control.  Interna5onal  Journal  of  Human-­‐Computer  Studies  69,  11  (2011),  693  –  704.  

Hart,  S.,  and  Wickens,  C.  Manprint:  an  Approach  to  Systems  Integra5on.  Van  Nostrand  Reinhold,  1990,  ch.  Workload  Assessment  and  Predic5on,  257–292.  

Oviad,  S.,  Maceachern,  M.,  and  Anne  Levow,  G.  Predic5ng  hyperar5culate  speech  during  human-­‐computer  error  resolu5on.  Speech  Communica5on  24  (1998),  87–110.  17.  

Oviad,  S.,  and  Van  Gent,  R.  Error  resolu5on  during  mul5modal  human-­‐computer  interac5on.  In  Proc.  ICSLP  ’96,  vol.  1  (oct  1996),  204–207.  

Rico,  J.,  and  Brewster,  S.  Usable  gestures  for  mobile  interfaces:  evalua5ng  social  acceptability.  In  Proc.  CHI  ’10,  ACM  (NY,  USA,  2010),  887–896.  

Rime,  B.,  and  Schiaratura,  L.  Fundamentals  of  Nonverbal  Behavior.  Cambridge  University  Press,  1991,  ch.  Gesture  and  speech,  239–281.  

Ruiz,  J.,  Li,  Y.,  and  Lank,  E.  User-­‐defined  mo5on  gestures  for  mobile  interac5on.  In  Proc.  CHI  ’11,  ACM  (NY,  USA,  2011),  197–206.  

Suhm,  B.,  Myers,  B.,  and  Waibel,  A.  Mul5modal  error  correc5on  for  speech  user  interfaces.  ACM  Trans.  Comput.-­‐Hum.  Interact.  8  (2001),  60–98.  

30