Engaging Spect-actors with Multimodal Digital Puppetry [NordiCHI 2010]

Engaging Spect-actors with Multimodal Digital Puppetry

Céline Coutrix (presenter),

Giulio Jacucci, Anna Spagnolli, Lingyi Ma, Matti Helin,

Gabriela Richard, Lorenza Parisi, Stefano Roveda, Prayag Narula

Introduction

•  (Digital) Puppetry

Introduction

•  Field study

-  Describe an example of how a computer can be used as a medium for engagement

-  Understand why would this be a promising field for HCI applications

Outline

1. The system studied

2. Data collection and analysis

3. Findings

a. Multimodality

b. Performative structures

c. Construction of engagement

4. Conclusion

Outline

3. Findings

a. Multimodality

4. Conclusion

The system: Euclide

pupeteer audience, in front of puppet

The system: Euclide

Euclide system

The system: Euclide

•  Has been successfully used for engagement in the Science museum in Naples, Italy

•  Why? How could it be improved? What can we learn for future puppetry systems?

The system: Euclide

•  Puppeteer

•  Audience

The system: Puppeteer

pupeteer

puppet on distant screen

•  Puppeteer

-  Voice

-  Face

-  Body

·  Global movement

·  Local movement

-  Scenery

•  Voice

•  Face

•  Body: Global movement

•  Body: Local movement

•  Body

•  Scenery

transparency

•  Scenery

orientation

•  Scenery

Zoom+Position

The system: Euclide

•  Puppeteer

•  Audience

The system: Euclide

audience, in front of puppet

distant pupeteer

The system: Euclide

•  Compromise between

-  Museum’s requirements for cheap, robust, generic and flexible hardware

-  Puppeteers’ requirements for large number of functionalities

-  Usability

·  e.g. mouth control moved from index to thumb to avoid tiredness

Outline

3. Findings

a. Multimodality

4. Conclusion

Data Collection

•  Video+audio recordings

-  Over several months

-  Four viewpoints

-  Three different pupeteers

•  Interview of head pupeteer

Data Analysis •  Constant comparison analysis

•  Top-down and Bottom-up approach

-  Findings originates from both the data and our research questions

•  Coding scheme built through three steps: open, axial, selective coding

•  Clips selected together, then divided between coders

•  Discussion and revision of coding until agreements > 0.6 35

Outline

3. Findings

a. Multimodality

4. Conclusion

Findings: Multimodal Puppetry

•  Most used features are controlled through more accessible devices, due to design

•  Mostly animation of face,

•  second comes body,

•  last comes scenery

face body scenery

•  Important multimodal use

Simultaneous use of modalities

Single modality

1. Speech ⇒ synchronization’s problem

2. Speech+lips

3. Lips

•  Lot of very short speeches and silences in order to maintain engagement

•  Puppet active even when silent

Speech durations Silence durations

Outline

3. Findings

a. Multimodality

4. Conclusion

Findings: Emergence of Performative Structure

•  Engaging the Audience Throughout the Sessions: Evidences

-  People in the audience:

·  Don’t talk to each other 99% of the time

·  Don’t pay attention to the outside 98% of the time

·  Talk to the puppet 65% of the time

·  Pleasure and Arousal annotated as positive all the time

·  Number increase, then decrease

•  Rather short, but:

-  Sessions interrupted by teachers

-  Group of children continue interacting from the next puppet station

0 2 4 6 8

< 1 min 1 min < < 2 min 2 min < < 9 min > 9 min

Number and length of sessions (mins)

•  Structure of sessions

-  Approach: enter the interaction area; observe

-  Testing: Try to interact to find out which actions have an effect

-  Playing: Interact in an aware, active, involved way; climax

-  Ending: attention diverted; leave

•  Structure of sessions

-  Approach, Testing, Playing, Ending

-  Long playing phases, with scenery animations

•  the audience starts interacting happily but calm,

•  then gets excited,

•  finishes the interaction happily and calm again

Outline

3. Findings

a. Multimodality

4. Conclusion

Findings: Construction of Engagement

•  The Puppet’s Multimodal Resources: Effects (face, body and scenery animations), like natural gestures, in relation to speech

-  Pure special effects

-  Non-verbal turns

-  Verbal accompaniements

-  Virtual gesturing

•  Pure special effects

-  Non verbal ressources are predominant modality for interacting with the audience

-  Used to attract visitors that cannot be reached by speech (babies, far away visitors)

•  Non-verbal turns: Contribution in an interaction that includes speech, e.g.

-  wearing accessory like glasses on special request from the audience and subsequently commenting on it

-  Explaining a fact by displaying a picture of it

•  Verbal accompaniements: Non-verbal actions are accompagnied by speech, e.g.

-  Acrobatic movement with onomatopeic sound

→ Communication relies on the nonverbal resource, which dictates its structure and meaning

•  Verbal accompaniements: Example

Puppet: Ho una molla I’ve got a spring Mi serve per saltare I need it to jump ⎡ Dong:: dong:: ⎤

Dong dong ⎣((Starts jumping))⎦

•  Virtual gesturing: priority is given to speech; visual features are used in synch with speech, following its structure and duration, e.g.

-  Head movement, like pointing gesture

•  Virtual gesturing: Example

Visitor: Dove sei? Where are you?

Puppet: Sono⎡::((pause))⎤ I’m

⎣((Looking down))⎦ ⎡qui dentro,⎤

inside here ⎣((Looking around))⎦ non mi vedi? can’t you see me

•  Multimodal interaction for puppeteer

is essential

-  Helps avoiding the low status of a conversational agent, e.g. verbal abuse

•  Response to verbal abuse: Example

Child: (Scemo) Dumb

Puppet: Vabbeh adesso basta. All right; let’s stop this

Child: ((turns back, surprised)) Puppet: Ogni volta che scemo, ogni volta che scemo.

All the times dumb, all the times dumb. E tu invece come sei? And what about you instead, what are you?

Child: ((goes away))

•  Talk = Main ressource,

Multimodal animation = instrument

•  Combination attracts visitors effectively

Outline

3. Findings

a. Multimodality

4. Conclusion

Conclusion

•  Why is digital puppetry a promising application field for HCI?

-  On the contrary to a fully computer-driven system

·  Engagement stems from combination of multimodal ressources by puppeteer

·  Users from the audience inspire the narrative by the puppeteer and feel like they have an active role

Conclusion

•  Why is digital puppetry a promising application field for HCI?

-  Performative Structures for Brief Interactions ⇒ repetitive element of an entertainment interface could be further taken into account

-  Multimodal Use of the System by the Puppeteer ⇒ possibility to introduce advanced interface techniques

·  Easing lip/speech sync,

·  Mapping expressive hand gestures to puppet expressions,

·  Etc.

Engaging Spect-actors with Multimodal Digital Puppetry

Engaging Spect-actors with Multimodal Digital Puppetry [NordiCHI 2010]

Documents

Water puppetry in Vietnam

cardiaco spect

Fun Puppetry

Puppetry New

Spect Methods

What is Puppetry? - COURSESamandapowellsellars.weebly.com/.../introduction_to_puppets.pdf · What is Puppetry? Puppetry is a ... The creation of puppets involves the manipulation

Gated Spect

RADIOFARMASI- SPECT

ACM NordiCHI 2012: Exploring the Design of Hybrid Interfaces for

SPECT SYSTEM

13-Introduction to Puppetry

Voice Puppetry

Water Puppetry

Karagiozis Greek Shadow Puppetry

Live Facial Puppetry

Shadow puppetry final

Puppetry. What is puppetry? Puppetry is a form of entertainment that dates back thousands of years. Though puppetry may be viewed today in our culture

QUALITY ASSURANCE TESTS FOR SPECT-CT /SPECT/GAMMA CAMERA

Spect Final

Puppetry in the Netherlands