MAPPING SPEECH RESEARCH IN HCI: TRENDS, THEMES AND CHALLENGES

LEIGH CLARK UNIVERSITY COLLEGE DUBLIN

@lmhclark@cogsis@hci_ucd

The CogSIS Project

CUI 2019: Conversational User Interfaces conference

22nd-23rd August 2019; Dublin, Ireland

cui2019.com

Mapping theoretical and methodological perspectives for understanding speech interface interactions

CHI 2019 part-day workshop: 4th May; Glasgow, UK

http://speech-interaction.org/chi2019/

UPCOMING EVENTS

Ben Cowan Philip Doyle

Diego Garaialde Justin Edwards

Emer Gilmartin

Trinity College Dublin

Stephan Schlögl

MCI Centre Innsbruck

Jens Edlund

KTH Stockholm

Matthew Aylett

CereProc Ltd

João Cabral

Trinity College Dublin

Cosmin Munteanu

University of Toronto Mississauga

https://www.amazon.com/AmazonBasics-Microwave-Compact-Works-Alexa/dp/B07894S727

MAP OUT:

PUBLICATION TRENDS RESEARCH METHODS RESEARCH THEMES

RESEARCH AIMS

speech interface; voice user interface; voice system; human computer dialog*; human machine dialog*; natural language dialog* system; natural language interface; conversational interface; conversational

agent; conversational system; conversational dialog* system; automated dialog* system; interactive voice response system; spoken

dialog* system; spoken human machine interaction; human system dialog*; intelligent personal assistant

+

ABSTRACT, TITLE & KEYWORD SEARCH

SEARCH TERMS & DATABASES

INCLUSION/EXCLUSION CRITERIA

1181

68

INCLUDE EXCLUDE

Speech focused

Full conference / journal papers

English

Embodiment

No interaction evaluation

Non-full / non- peer reviewed

RESEARCH METHODS

DIRECTION OF COMMUNICATION

User-system dialogue (44)

User input only (16)

System output only (12)

User attitudes 36

Task performance 33

Lexis & syntax 20

Perceived usability 18

System usage 15

User recall 7

Physiological data 3

Other 11

CONCEPTS MEASURED

RESEARCH THEMES

Synthesis

Content

SYSTEM SPEECH PRODUCTION

Keyboard and/or mouse

Digital pen

MODALITY COMPARISON

General production

Addressee identification

Alignment

USER SPEECH PRODUCTION

ASSISTIVE TECHNOLOGY & ACCESSIBILITY

https://www.nationaldeafcenter.org/topics/assistive-technology

Tabletop designs - physicians, deaf patients & interpreters

Mobile interface - limited hand dexterity

Voiced-based browser plugin - blind users

DESIGN INSIGHT

Early design insight - speech to access GUI-based software

Interface for a large-scale game

IPA EXPERIENCE

Disparity between people’s mental models of IPAs & reality of interaction

Human likeness can negatively affect IUX

Embarrassment of public use

Structure of multiple user interaction w/ Siri

CHALLENGES & ONGOING RESEARCH

CHALLENGES & ONGOING RESEARCH

MORE THEORETICAL UNDERSTANDING FOR:

1. LANGUAGE PRODUCTION TO SYSTEMS 2. PERCEPTION OF SYSTEMS 3. DESIGN IN LIGHT OF THESE

Proliferation of humanlike voices in non-human artefacts can create

unrealistic expectations of capabilities

Moore, R. K. (2017). Appropriate Voices for Artefacts: Some Key Insights. In 1st International Workshop on Vocal Interactivity in-and-between Humans, Animals and Robots.

POLITENESS & FACE

Politeness linked to concept of face (Goffman, 1952; 1967)

Social self-image dependent on societal norms and rules

Usually best interest to save face

No politeness

Connect… Give each piece a twist… Attach… …so it’s in line with the feet Locate… …so the end is closest to the top of the body

Politeness

Just connect…. Just give each piece a little bit of a twist… Basically, attach…. …so it’s more or less in line with the feet Now just locate…. …the end should be closest to the top of the body

POLITENESS EXAMPLE

https://fineartamerica.com/featured/i-dont-care-if-she-is-a-tape-dispenser-i-love-sam-gross.html

How similar is appropriate?

KEY POINTS

1. Speech HCI fragmented

2. More theoretical development/application can help improve cohesion in the field

3. Theories can help explain & understand, but can also be redefined and re-conceptualised in HCI

4. Current work at HCI @ UCD looking at UCD looking design choices affecting partners models, language production, user perception; humanlike ≠ always good aim

leigh.clark@ucd.ie

Clark, L., Pantidi, N., Cooney, O., Doyle, P., Garaialde, D., Edwards, J., ... & Cowan, B. R. (2019). What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents. arXiv preprint arXiv:1901.06525.

Clark, L., Doyle, P., Garaialde, D., Gilmartin, E., Schlögl, S., Edlund, J., ... & Cowan, B. (2018). The State of Speech in HCI: Trends, Themes and Challenges. arXiv preprint arXiv:1810.06828.

Murad, C., Munteanu, C., Clark, L., & Cowan, B. R. (2018). Design guidelines for hands-free speech interaction. Mobile HCI 2018 Adjunct (pp. 269-276). ACM.

Clark, L. (2018). Social boundaries of appropriate speech in HCI: a politeness perspective. BCS HCI 2018.

Clark, L., Cabral, J. & Cowan, B.R. (2018). The CogSIS Project: Examining the Cognitive Effects of Speech Interface Synthesis. BCS HCI 2018.

Large, D. R., Clark, L., Quandt, A., Burnett, G., & Skrypchuk, L. (2017). Steering the conversation: a linguistic exploration of natural language interactions with a digital assistant during simulated driving. Applied Ergonomics, 63, 53-61.

Clark, L., Ofemile, A., Adolphs, S. & Rodden, T. (2016). A Multimodal Approach to Assessing User Experiences with Agent Helpers. ACM Transactions on Interactive Intelligent Systems (TIIS), 6(4) 29.

@lmhclark@cogsis@hci_ucd