View
216
Download
0
Category
Tags:
Preview:
Citation preview
Interaction, Media and Communication Research Group
Drawing Things Together:Integrating Modalities in Dialogue.
MAGIC: Multimodality and Graphics in Interactive Communication.
EPSRC/ESRC PACCIT Initiative: People at the Centre of C&IT.
Pat Healey, James King, Charlie Peters.
Information, Media and Communication Research Group, QMUL.
John Lee, Jon Oberlander.
Human Communication Research Centre, University of Edinburgh.
Simon Garrod, Nick Fay.
Department of Psychology, University of Glasgow.
Ichiro Umata, Yasuhiro Katagiri,
ATR Media Integration and Communications Laboratories, Kyoto.
Interaction, Media and Communication Research Group
Outline
1. Drawing-in-interaction: ethnographic observations• turn-taking and topic management
2. Graphical Language Games: Musical ‘Pictionary’• Experiment 1: Community-specific Graphical Languages • Experiment 2: Mechanisms of Interaction
3. The mechanisms of interaction available to people directly constrain the form and organisation of shared symbol systems – not just individual cognitive-computational abilities
4. Augmented Human Interaction.
Interaction, Media and Communication Research Group
Design for Human-Human Interaction
Interaction, Media and Communication Research Group
Design for Human-Human Interaction?‘Single User’ Baton Design:
‘Multi-User’ Baton Design:
designed for exchange:• symmetric
• smooth• centrally balanced
Example: Architectural Design
Interaction, Media and Communication Research Group
Example: Architectural Design
Interaction, Media and Communication Research Group
Ethnography of Architectural Design• 6 staff working on a design competition
– mixed expertise and responsibilities• Prepare 4 A2 presentation boards addressing:
– use of site, environmental concerns, building use, open-space use • 58 project interactions video taped over 4 weeks
– 40 two-party interactions – 13 three-party interactions – 5 interactions > 3-party
• One 12 minute, 3 party interaction transcribed for analysis– coded for overlaps, pauses, run-throughs, stress etc.– coded for gesture and drawing activity
Interaction, Media and Communication Research Group
Basic Observations• Complex variety of drawing spaces (in 12 minutes)
– 6 physical drawing spaces • pieces of paper or regions of a piece
– 3 gestural drawing spaces • drawing in the air with pen or finger
– verbal ‘spaces’
Interaction, Media and Communication Research Group
016 R: jus shoa few people lukkin ova the: edge and e:mm a cuppla people R: RIGHT-HAND BY SIDE: MOVES HAND FORWARD AND POINTS ON 1. KEEPS FINGER ON 1, MOVES IT FROM L-R
wandri n up ( 1. 0) en / / meb be: : .R: (CO NT’D.)
.R: POINTS AT 1.
017 J : bu’ i ’ * if i tsa f ree form .J : BEGINS TO DRAW IN THE AIR ABOVE DRAWING BOARD
obj ect l i ke that // i ts bounta have a rai l round i t / / o r some .J : (CO NT’D.)
R: Yeah Yeah
ki ndav (post u m) so I thought shoul d I j us= .J :(CO NT’D)
.J : AND STARTS TO DRAW ON 1.
018 R: =J UST put the rai l en nothin el se .R: BRINGS HAND FORWARD ABOVE 1,
.FLICKS S-HAND ABOVE 1:
.J : (CO NT’D.)
Excerpt 1
Interaction, Media and Communication Research Group
Turn TakingDrawing activity does not automatically claim the floor:
1. Drawing activities continue across turn and speaker changes– overlapping speech is avoided (e.g., Levinson, 1983)– gestures are used to bid for and maintain floor control (Bavelas,
et. al. 1995) - but see Furuyama.
2. Where competition for the floor occurs during drawing:
drawing is suspended
gesture & speech used to compete for floor
Interaction, Media and Communication Research Group
Excerpt 2
143 J: =dimension R: Draws (cont’d…)
( 3. 0)R: Draws (cont’d…)
144 J : [so thas145 R: [SO I thi nk* j ust do a hal f a fence ri gh j us doa hal f as though
.R: RIGHT-HAND
FLICKS UP FROM
DRAWING
.R: RESUMES DRAWING ON 5. (CONT’D…)
Interaction, Media and Communication Research Group
Topic ManagementDrawing activity is used to manage topic changes: ceases or shifts
location if topic changes• In excerpt 1 topic changes from rail to space.
– J moves hand to draw in air above the board– reference to ‘space’ is harder to resolve
• When topic returns to rail drawing resumes on the page
(Earlier in interaction J draws same ‘space’ over the drawing)
Interaction, Media and Communication Research Group
ObservationsDrawing, gesture and speech are integrated into composite
communicative signals (cf. Neilson and Lee, 1994)• often treated as separate channels (e.g., Netmeeting)
Drawing activities often serve interactional functions• not only representation of domain or computational aid. • use of space to manage topics
• cf. topographic and referential gesture spaces
Interaction, Media and Communication Research Group
Graphical Dialogue
50% of everyday drawings are produced as part of interaction
(van Sommers, 1994)
Examples of Graphical Dialogue: • Routine interactions
– e.g., sketch maps, explanatory diagrams, games • Auxiliary mode of Communication
– e.g., cross-linguistic communication, aphasia• Specialised interactions
– e.g., design interactions
Does interaction affect representation?
Interaction, Media and Communication Research Group
Graphical Language Games: Musical ‘Pictionary’What do we do if we don’t share a symbol system?
Criteria for Task: • communication task• structured, regular, domain• few (or no) established representational conventions• exclusively graphical interaction
Typical Set-up:• Subjects seated in separate (soundproof) rooms
– communication via shared whiteboard application– 30 sec piano piece each
• Task: draw picture of target: no letters or numbers– SAME or DIFFERENT
Interaction, Media and Communication Research Group
Room A Room B
• One piano piece each: same or different?• Communicate by drawing: no letters or numbers
Interaction, Media and Communication Research Group
Music Drawing Types:
1. Abstract: – Graph-like representation of domain structure e.g., pitch, intensity,
rhythm
2. Figurative: – Ad hoc associations: faces, figures, objects or situations
3. Composite:– Mixture of Abstract and Figurative
(independent classification by 2 judges: Kappa = 0.9, N =287, k= 2)
Sequence of ‘Figurative’ Trials:
Sequence of ‘Abstract’ Trials:
Interaction, Media and Communication Research Group
Abstract / Figurative Contrast:
Abstract drawings provide a more complex representational system for the task. Specifically:
1. Systematicity: support direct comparison within and between items
2. Proto-Compositionality: distinct parts of the drawing refer to distinct parts of the music
Figurative drawings are more holistic and more ad hoc.
Interaction, Media and Communication Research Group
What affects use of Abstract or Figurative?
Previous (non) findings:
1. No effect of target on drawing type:– genre / tempo / mode
2. No effect of medium on drawing type– stylus vs. mouse
3. No difference in drawing effort– same average quantity of lines and ink
Interaction, Media and Communication Research Group
Effects of Interaction: 1. Repetition promotes abbreviation -but only if participants can signal
understanding– c.f. grounding in dialogue
2. Dialogue partners tend to use drawings of the same type.– c.f. accommodation/entrainment
3. Level of communicative interaction:• Concurrent Drawing
– 60% Abstract• Alternate Drawing:
– 60% Figurative
Interaction, Media and Communication Research Group
Hypotheses: 1. AUTONOMOUS CO-ORDINATION (truth wins): • Participants independently migrate toward the representational
scheme that is most efficient for the Concurrent task.– co-ordination of time-based axis? more comparative?– co-ordination emerges as aggregate individual experience
• (e.g., Clark, Lewis…)
2. COLLABORATIVE CO-ORDINATION• Participants use the opportunities afforded by concurrent interaction
to establish a co-ordinated 'sub-language'.– collaborative revision and refinement of conventions – co-ordination emerges through local histories of interaction
Interaction, Media and Communication Research Group
Experiment 1: Community Sub-Languages? Does interaction contribute anything, in addition to individual expertise, to
the co-ordinated use of the Abstract drawings?
Experimental Design:
Phase 1: develop several communities with equivalent task experience but different interaction histories
Phase 2: compare interaction within and between communities.
Note: community membership is hidden from participants.
Interaction, Media and Communication Research Group
Subject 2
Subject 1 Subject 6
Subject 5
Subject 3 Subject 4
Round 1 = Round 2 =Round 3 = Round 4 =
Phase1: Community Development
Interaction, Media and Communication Research Group
Phase 1:• 10 ‘communities’ (66 people)
– seated round edge of large PC lab• Music Task: one piece each, same or different?
– 4 rounds of 12 trials – different partner on each round
During Phase 1:• common ‘interaction history’ accumulates
– reliable increase in speed: 53 sec. to 43 sec. – reliable increase in accuracy 37% to 52%
Interaction, Media and Communication Research Group
Between Group =
Within Group =
Phase 2: Experimental Manipulation
Interaction, Media and Communication Research Group
Results: Chi2(2) =19.0, p=0.00
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Within Community Between Community
Proportion
Abstract
Figurative
Composite
Interaction, Media and Communication Research Group
Results:
Interaction, Media and Communication Research Group
Conclusions: Experiment 1Cross-group interaction de-stabilises use of Abstract drawings• independently of
– indivdual expertise– what is being represented– individual cognitive-computational abilities– explicit knowledge of community membership
• (cf. Healey 1997).
Support for Collaborative Co-ordination Hypothesis:• localised patterns of interaction lead to community-specific (graphical)
‘dialects’
Why?
Interaction, Media and Communication Research Group
Why does interaction matter?
Graphical Interaction Mechanisms:•localisation •alignment.
• Abstract drawings are ‘proto-compositional’: interaction allows participants to co-ordinate meaningful elements of each other’s drawings
Interaction, Media and Communication Research Group
Hypotheses:3. ‘MUTUAL MODIFICATION: coordinated use of Abstract drawings
depends on ability to use interaction devices to annotate and modify elements of each other's drawings.• (e.g., circling, underlining, and arrows)
Experiment 2: interfere with use of interaction devices (localisation and alignment) and assess effects on communication.
Interaction, Media and Communication Research Group
Manipulation 1: Block Localisation
Room AScreen
Room BScreen
Annotation of other’s drawing
blocked.
Interaction, Media and Communication Research Group
Manipulation 2: Block Alignment
Room AScreen
Room BScreen
Transpose
Interaction, Media and Communication Research Group
Experimental Design:
• Pairs seated in separate soundproof rooms • Communicate via shared whiteboard- no letters or numbers
Subject A’s View Subject A’s ViewSubject B’s View Subject B’s View
+ BLOCKING- BLOCKING
- T
RA
NS
NP
OS
ITO
N+
TR
AN
SP
OS
ITIO
N
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
+B +T + T - B -T + B -T -B
Level of Interaction
Proportion of Drawings
Figurative
Abstract
(Blocking: Chi2(3) = 96.70, p =0.00, Transposition: Chi2(3) = 81.61, p =0.00)
Effects of Interference with Interaction:
Interaction, Media and Communication Research Group
Conclusions Experiment 2: • Blocking and Transposition cause change in graphical conventions
– independently of what is being represented– Independently of individual cognitive-computational abilities
• Complexity of emergent symbol system depends on mutual modification (localisation, alignment)– co-ordinated manipulation of external representations– participants’ ability to modify each other’s representations
• not editing / annotation / revision per se
Interaction, Media and Communication Research Group
General Conclusions:Significant parallels between verbal and graphical dialogue
– grounding, accommodation, turn-taking, – modality independent, interactional constraints on representation
How do people co-ordinate despite differences in interpretation?• Local, surface-based, manipulations of external representations.• Verbal and graphical repair mechanisms
– Localisation• specficity, reprise fragment clarifications
– Alignment• sequential relevance, embedded repair
Interaction, Media and Communication Research Group
General Conclusions:Drawing supports transactional and interactional functions• not just a representational medium
Not explained by physical / perceptual / computational properties per se • alternate vs. concurrent drawing
Modalities combine to form composite communicative signals applications often treat them as separate channels (e.g., net-meeting)
Generic interaction mechanisms are apparent across modalities• turn-taking, topic management, repair, grounding, accommodation
Augmented interaction mechanisms richer more expressive languages?
Interaction, Media and Communication Research Group
Augmented Human Interaction:Beyond Face-to-Face
1. Capture: gesture, expression, attention, engagement, understanding…
2. Provide augmented cues for interaction
3. Enable richer, more robust, forms of human communication– gestural languages, musical languages, dance …
Interaction, Media and Communication Research Group
Augmented Human Interaction:Beyond Face-to-Face
1. Capture: gesture, expression, attention, engagement, understanding…
2. Provide augmented cues for interaction
3. Enable richer, more robust, forms of human communication– gestural languages, musical languages, dance …
Interaction, Media and Communication Research Group
Augmented Human Interaction Lab
• multi-user, real-time, full body, motion capture.• integrated 3D audio display
Interaction, Media and Communication Research Group
Augmented Human Interaction Lab
Recommended