SAI User-System Interaction U1, Speech in the interface: 6. Human communication1 Module u1: Speech in the Interface 6: Human Communication Jacques Terken

SAI User-System Interaction U1, Speech in the interface: 6. Human communication 1

Module u1: Speech in the Interface

6: Human Communication

Jacques Terken


Motivation

provide knowledge about basic concepts of human communication relevant to the design of speech interfaces

explore use of knowledge about human communication as model for man-machine interaction


Contents

basic scheme for communication the structure of messages the structure of communication “how to do things with words”1: communication as process implications for man-machine interaction key points human-human communication as model for human-computer

interaction

1 J.L. Austin: “How to do things with words”, 1962


basic scheme

Context– Situational context situated communication: time, place, objects– Social context – Cultural context– Dialogue context

contextsender channel receiver

feedback

context


intentions and effects

Sender: intention to convey a message

message needs to be encoded Receiver: decoding effect

Ideal communication: intention = effect

Distortions underway: intention effect’

Furthermore: effect is more than intention (sender also communicates unintentional information)

complex messages require sequence of steps

communication as structured action / collaboration


Contents

basic scheme for communication the structure of messages the structure of communication communication as process implications for man-machine interaction key points human-human communication as model for human-computer

interaction


formal aspects (1)

Simple case: messages containing a single dialogue act (inform, request, confirmation etc)

Sentence/utterance– put the red block before the yellow block next to the ball– I herewith pronounce you man and wife– What time does the next flight to Boston leave?

Sentences made up of word groups Word groups made up of words Words made up of syllables and phonemes (~40) or

graphemes (~25) / @, e, a, r, … / ; { e, e, a, r, … }


formal aspects (2)

rules determine well-formedness of representations at different levels:– pragmatics: sentence meaning in context (speech act,

dialogue act) contextual meanings

*the king of France is bald (but may be okay in a story)– semantics: sentence meanings possible meanings

*the 50 year old bachelor surprised his wife– syntax: sentence structure possible sentences– morphology: word structure possible words– Dutch: loopje; *loopetje; kommetje; *kompje– Phonology: sound structure possible sound sequences

Dutch: herfst; Polish: przewalski


formal aspects (3)

mapping between levels not fully specified (not one - to - one):

ambiguity Mapping sounds onto words:

/Aiscri:m/ {I scream} or {Ice cream}

Mapping words onto sentence structure:

Syntactic ambiguities

i saw the girl with the binoculars


put the red block before the yellow block next to the ball

(1) put the red block

before the yellow block [ that is ] next to the ball

(2) put the red block before the yellow block,

next to the ball


Contents


interaction


levels of analysis

topic … topic … topic

transaction

exchange exchange exchange

turn turn turn

move move move


transaction

(part of) conversation dealing with a single topic consists of one or more exchanges


exchange

smallest interactive unit consists minimally of two turns produced by two different

speakers involves the negotiation of a single piece of information examples:

request - inform - confirm

presentation - acceptance


adjacency pairs:

question answer

request accept / turn down

invitation accept / decline

greeting greeting

apologysmoother

Interactive pressure: opening move induces obligation to answer with closing move (conversational obligation)


nesting of adjacency pairs:s: question X what time is the next train to amsterdam

a: question Y amsterdam central?s: answer Y yes

a: answer X in seven minutes at platform 5

mixed initiative vs. system initiative


turn

Turn: everything the speaker says before the next speaker takes over

Rules for turn-taking behaviour

1. Speaker selects next speaker

2. Next speaker selects self

Window of opportunity for turn switching (minimizes overlap and interruption; interruptions only permissible in particular circumstances

Turn consists of one or more moves


temporal aspects of turn-taking

dialog not as a sequence of well-formed non-overlapping turns:– time window for turn taking– overlap, interruptions


move/dialog act

no fixed inventory, but dependent on application Common types: Primary dialogue acts

– request for action – inform– question

Game theory conceives of communication as a sequence of movesExpresses the assumption that communication is governed by a set of rules

Dialogue control acts– acknowledge– check– confirm– greeting– thanks


Indirect communication

No one-to-one mapping between form and function: indirect communication (politeness forms)

Questions:– Can you tell me what time it is (yes/no question) – I would like to know what time it is (statement)– Do you have a watch (yes/no question with indirect

communication )

Direct communication:– What time is it (wh-question)– Please tell me the time (request)


spoken dialog systems

Identify dialog act performed by user in order to decide what dialog act to produce as an an answer

Machine learning techniques in order to learn best continuation in context x (and to learn best path)


Contents


interaction


spoken communication as joint action/cooperation Speech is volatile Mechanisms to cope with volatile nature of speech co-operation

receiver must be in receiving mode, i.e., be open to message

sender must adjust to channel and receiver characteristics


Goal achievement is usually accomplished through a number of steps involving both participants: dialogue

Monitoring of transmission success by sender on the basis of conscious and unconscious feedback from receiver

Immediate action in case of problems either by user or sender

– Role of eye gaze in staying in touch, monitoring and turn taking

spoken communication as joint action/cooperation (2)


speech generation

two simultaneous processes:– conceptualisation (“what to say”: content)– formulation (“how to say it”: converting content into

utterance)

simultaneous, but formulation lagging behind

( spontaneous speech phenomena)

– restarts, self-corrections, hesitations (silence, ehm), disfluencies

– grammar of spoken language


message formulation: issues

continuity, coherence: what we say usually links to what was said before:

context provides search space for interpretation and allows for efficient expression

– pronouns, anaphora – prosody– bridging: establishing a relation between expressions in

consecutive utterancescan you take the picnic supplies out of the van? the beer is probably warm, so please put it in the fridge.


grice’s cooperativity principle

human communication (supposed to be) governed by grice’s cooperativity principle, consisting of the following maxims

(guidelines for communication): quantity: make your contribution as informative as is required

I want a flight; I want a flight from A to B because …. quality: do not say what you believe is false or for which you

lack adequate evidence relation: be relevant manner: be perspicuous, brief and orderly; avoid obscurity

and ambiguity


speech comprehension

Stages in comprehension:– Perception: perceiving sounds and mapping sounds onto

words– Interpretation– Evaluation– From a processing point of view perception and

interpretation are closely interlinked

keywords: on-line processing, hypothesis testing and integrating


backchannelling

“utterances” and other communicative signals guiding the progress of the communication, i.e., serving a control function– nodding– yes, hm-hm– “madrid?”


error recovery

on-line property allows fast and efficient recovery:– problems noticed immediately– cause of problem usually easily identified

“what did you say?”

“did you say …”

“what do you mean”


Grounding

agreeing about current state of the dialogue

Important role of feedback Shared knowledge

+ knowledge that speaker knows is available to hearer

+ world knowledge (“living memory”)

+ knowledge concerning context of communication and what has been established by previous part of discourse

common ground


Contents


interaction


human communication no neat succession of turns: overlap in time: provision for speech recognition needed

Indirect communication: message function and form no 1:1 relation

interpretation dependent on domain knowledge: requires domain modelling: successful for restricted domains

interpretation dependent on context: requires discourse model


supporting the interaction

Prompting: system initiative Zooming: increase amount of feedforward in system prompt

– How may I help you– Which connection– Are you interested in connections or prices– Please respond with yes or no

Cooperative replies– User: “Do you have flights between 7:15 and 7:45?”– S: “No, but there is a flight at 7:55”


robustness issue

Lack of robustness creates need for verifcation

Explicit verification:– Did you say “amsterdam”?

Implicit verification– U: “I want to travel from .. to rotterdam”– S: “when do you want to travel to rotterdam?”

– Cf: S: “when do you want to travel to amsterdam?”


Verification is costly when you got it right Feedback (positive feedback or implicit verification) is

misleading when you good it wrong

Use verfication when uncertain Use feedback when confident


confidence-based verification

High confidence: just accept or implicit verification Medium confidence: explicit verification Low confidence: explicit request for repetition

Precision/recall issue for correctness and confidence measure:– Confidence measure may not be right: either too high or too

low


Contents


interaction


communication as joint action (cooperation)– taking into consideration listener needs– listening for intention

rule-governed behaviour at many different levels– structure of messages– structure of conversations (e.g. turn-taking behaviour)

grounding feedback (back-channelling)


Contents


interaction


cooperative assistant

Characteristics of cooperative assistant– Interprets messages in terms of intentions not in terms of

form

(“c.a. is not a politician or lawyer”)– Cooperative replies

(not blunt “no” but offers alternatives, e.g. through constraint relaxation)

– Context-sensitive replies and help

No global help manual– Proactive, triggered by implicit cues

Does not await explicit request for help but anticipates


discussion

Is this a good model for human computer interaction? Considerations:

– Schneiderman c.s.: user should stay in control

– Maes/Sidner: • computers should be able to understand our intentions

so that we don’t have to specify all the actions explicitly ourselves

• Computers should monitor our actions so that they can learn how to anticipate our needs and preferences

Documents

SAI User-System Interaction U1, Speech in the interface: 6. Human communication1 Module u1: Speech in the Interface 6: Human Communication Jacques Terken