Engaging Spect-actors with Multimodal Digital Puppetry [NordiCHI 2010]

Engaging Spect-actors with Multimodal Digital Puppetry

Céline Coutrix (presenter),

Giulio Jacucci, Anna Spagnolli, Lingyi Ma, Matti Helin,

Gabriela Richard, Lorenza Parisi, Stefano Roveda, Prayag Narula

Introduction

•  (Digital) Puppetry

1

Introduction

•  Field study

-  Describe an example of how a computer can be used as a medium for engagement

-  Understand why would this be a promising field for HCI applications

2

Outline

1. The system studied

2. Data collection and analysis

3. Findings

a. Multimodality

b. Performative structures

c. Construction of engagement

4. Conclusion

3

Outline



3. Findings

a. Multimodality



4. Conclusion

4

The system: Euclide

5

pupeteer audience, in front of puppet

The system: Euclide

Euclide system

6

The system: Euclide

•  Has been successfully used for engagement in the Science museum in Naples, Italy

•  Why? How could it be improved? What can we learn for future puppetry systems?

7

The system: Euclide

•  Puppeteer

•  Audience

8

The system: Puppeteer

pupeteer

puppet on distant screen

9


•  Puppeteer

-  Voice

-  Face

-  Body

·  Global movement

·  Local movement

-  Scenery

10


•  Voice

11


•  Face

12


•  Face

13


•  Face

14


•  Face

15


•  Face

16


•  Face

17


•  Face

18


•  Face

19


•  Body: Global movement

20



21



22


•  Body: Local movement

23



24



25


•  Body

26


•  Scenery

27

transparency


•  Scenery

28

orientation


•  Scenery

29

Zoom+Position

The system: Euclide

•  Puppeteer

•  Audience

30

The system: Euclide

audience, in front of puppet

distant pupeteer

31

The system: Euclide

•  Compromise between

-  Museum’s requirements for cheap, robust, generic and flexible hardware

-  Puppeteers’ requirements for large number of functionalities

-  Usability

·  e.g. mouth control moved from index to thumb to avoid tiredness

32

Outline



3. Findings

a. Multimodality



4. Conclusion

33

Data Collection

•  Video+audio recordings

-  Over several months

-  Four viewpoints

-  Three different pupeteers

•  Interview of head pupeteer

34

Data Analysis •  Constant comparison analysis

•  Top-down and Bottom-up approach

-  Findings originates from both the data and our research questions

•  Coding scheme built through three steps: open, axial, selective coding

•  Clips selected together, then divided between coders

•  Discussion and revision of coding until agreements > 0.6 35

Outline



3. Findings

a. Multimodality



4. Conclusion

36

Findings: Multimodal Puppetry

•  Most used features are controlled through more accessible devices, due to design

37


•  Mostly animation of face,

•  second comes body,

•  last comes scenery

face body scenery

38


•  Important multimodal use

Simultaneous use of modalities

Single modality

39


1. Speech ⇒ synchronization’s problem

2. Speech+lips

3. Lips

40


•  Lot of very short speeches and silences in order to maintain engagement

•  Puppet active even when silent

Speech durations Silence durations

41

Outline



3. Findings

a. Multimodality



4. Conclusion

42

Findings: Emergence of Performative Structure

•  Engaging the Audience Throughout the Sessions: Evidences

-  People in the audience:

·  Don’t talk to each other 99% of the time

·  Don’t pay attention to the outside 98% of the time

·  Talk to the puppet 65% of the time

·  Pleasure and Arousal annotated as positive all the time

·  Number increase, then decrease

43


•  Rather short, but:

-  Sessions interrupted by teachers

-  Group of children continue interacting from the next puppet station

0 2 4 6 8

10

< 1 min 1 min < < 2 min 2 min < < 9 min > 9 min

Number and length of sessions (mins)

44


•  Structure of sessions

-  Approach: enter the interaction area; observe

-  Testing: Try to interact to find out which actions have an effect

-  Playing: Interact in an aware, active, involved way; climax

-  Ending: attention diverted; leave

45


•  Structure of sessions

-  Approach, Testing, Playing, Ending

-  Long playing phases, with scenery animations

46


47


•  the audience starts interacting happily but calm,

•  then gets excited,

•  finishes the interaction happily and calm again

48

Outline



3. Findings

a. Multimodality



4. Conclusion

49

Findings: Construction of Engagement

•  The Puppet’s Multimodal Resources: Effects (face, body and scenery animations), like natural gestures, in relation to speech

-  Pure special effects

-  Non-verbal turns

-  Verbal accompaniements

-  Virtual gesturing

50


•  Pure special effects

-  Non verbal ressources are predominant modality for interacting with the audience

-  Used to attract visitors that cannot be reached by speech (babies, far away visitors)

51


•  Non-verbal turns: Contribution in an interaction that includes speech, e.g.

-  wearing accessory like glasses on special request from the audience and subsequently commenting on it

-  Explaining a fact by displaying a picture of it

52


•  Verbal accompaniements: Non-verbal actions are accompagnied by speech, e.g.

-  Acrobatic movement with onomatopeic sound

→ Communication relies on the nonverbal resource, which dictates its structure and meaning

53


•  Verbal accompaniements: Example

54

Puppet: Ho una molla I’ve got a spring Mi serve per saltare I need it to jump ⎡ Dong:: dong:: ⎤

Dong dong ⎣((Starts jumping))⎦


•  Virtual gesturing: priority is given to speech; visual features are used in synch with speech, following its structure and duration, e.g.

-  Head movement, like pointing gesture

55


•  Virtual gesturing: Example

56

Visitor: Dove sei? Where are you?

Puppet: Sono⎡::((pause))⎤ I’m

⎣((Looking down))⎦ ⎡qui dentro,⎤

inside here ⎣((Looking around))⎦ non mi vedi? can’t you see me


•  Multimodal interaction for puppeteer

is essential

-  Helps avoiding the low status of a conversational agent, e.g. verbal abuse

57


•  Response to verbal abuse: Example

58

Child: (Scemo) Dumb

Puppet: Vabbeh adesso basta. All right; let’s stop this

Child: ((turns back, surprised)) Puppet: Ogni volta che scemo, ogni volta che scemo.

All the times dumb, all the times dumb. E tu invece come sei? And what about you instead, what are you?

Child: ((goes away))


•  Talk = Main ressource,

Multimodal animation = instrument

•  Combination attracts visitors effectively

59

Outline



3. Findings

a. Multimodality



4. Conclusion

60

Conclusion

•  Why is digital puppetry a promising application field for HCI?

-  On the contrary to a fully computer-driven system

·  Engagement stems from combination of multimodal ressources by puppeteer

·  Users from the audience inspire the narrative by the puppeteer and feel like they have an active role

61

Conclusion

•  Why is digital puppetry a promising application field for HCI?

-  Performative Structures for Brief Interactions ⇒ repetitive element of an entertainment interface could be further taken into account

-  Multimodal Use of the System by the Puppeteer ⇒ possibility to introduce advanced interface techniques

·  Easing lip/speech sync,

·  Mapping expressive hand gestures to puppet expressions,

·  Etc.

62

Engaging Spect-actors with Multimodal Digital Puppetry

Documents

Engaging Spect-actors with Multimodal Digital Puppetry [NordiCHI 2010]