Mixed Reality: A Model of Mixed Interaction Céline Coutrix, Laurence Nigay User Interface...

Mixed Reality: A Model of Mixed InteractionCéline Coutrix, Laurence Nigay

User Interface Engineering TeamCLIPS-IMAG Laboratory, University of Grenoble

Introduction: Mixed Systems

•Merging Physical and Digital Worlds

•In many application fields

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

1

Introduction: Designers Needs

A framework for :

Understanding and capitalizing on existing Mixed Systems

Descriptive power

Creating designs Generative power

Assessing multiple design alternatives and increasing usability

Comparative power

2

•Mixed Interaction Model

•Descriptive Power

•Generative and Comparative Powers

•Conclusion and Future Work

Outline

3

•Mixed Interaction Model

• Mixed object

• Mixed interaction

•Descriptive Power

•Generative and Comparative Powers

•Conclusion and Future Work

Outline

4

ExampleQuickTime™ et un

décompresseur codec YUV420sont requis pour visionner cette image.

5

Mixed Interaction Model: Mixed Object

mixed object

tool task object

6

Mixed Interaction Model: Mixed Object

physical object

physical properties

digital object

digital properties

mixed object

physical properties

digital properties?

• Interaction modality m = (d, l)

• speech = (microphone, natural language)

7

Mixed Interaction Model: Mixed Object

•Linking modality

mixed object

physical properties

digital properties

inputdevice

outputdevice

output language

acquiredphysical

data

inputlanguage

generatedphysical

data

8

Mixed Interaction Model: Mixed Object

mixed drawing

physical properties shapes

camera

projector selection

image computervision

image

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

9

Mixed Interaction Model: Mixed Object

a music bottle

physical properties

musicpart

loudspeakers identityaudio

sequence

screentable

lightingcolored

sequence

pitch andvolumeanalysis

QuickTime™ et undécompresseur codec YUV420

sont requis pour visionner cette image.

10

Mixed Interaction Model: Mixed Object

mixed object

physical properties

digital properties

Linkingmodalities

tool task object

11

tool

task object

action

reaction

elementarytask

response

feedback

Mixed Interaction Model: Mixed Interaction

12

• Interaction modality m = (d, l)

Mixed Interaction Model: Mixed Interaction

13

mixed tool

physical properties

digital properties

inputdevice

outputdevice

output language

acquiredphysical

data

inputlanguage

generatedphysical

data

interactionoutput

language

interactioninput

language

task object

Mixed Interaction Model: Mixed Interaction

14

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

Mixed Interaction Model: Mixed Interaction

15

mixed tool

paperbutton

buttonpressedword,place

on drawing

camera imagecomputer

vision

translation

drawing

mixed tool

Mixed Interaction Model: Mixed Interaction

16

•Mixed Interaction Model

•Descriptive Power

•Generative and Comparative Powers

•Conclusion and Future Work

Outline

17

•Schemes for classification and better understanding of mixed systems

Descriptive Power

18

•Mixed Interaction Model

•Descriptive Power

•Generative and Comparative Powers

•Conclusion and Future Work

Outline

19

RAZZLE: Case Study

20

RAZZLE: Case Study

mixed piece

physical properties

IdentifierLocation

Orientation

locationsensor

head mounteddisplay

3Dprojection

numericvalues

geometrictransforma°

image

orienta°sensor

numericvalues

geometrictransforma°

21

Generative Powermixed tool

? ?

? ?

?

22

RAZZLE: Gestural Modality

QuickTime™ et undécompresseur DV - PAL

sont requis pour visionner cette image.

23

RAZZLE: Gestural Modalitymixed tool

physicalproperties

recognizedgestures

camera image computervision

translation

24

RAZZLE: Tactile Modality

QuickTime™ et undécompresseur DV - PAL

sont requis pour visionner cette image.

25

RAZZLE: Tactile Modalitymixed tool

physicalproperties

click

touchpad data inputlanguage

translation

26

•Mixed Interaction Model

•Descriptive Power

•Generative and Comparative Powers

•Conclusion and Future Work

Outline

27

Comparative Power

• Criteria within the model for assessing multiple design alternatives

• Observability

• Spatial continuity

28

Comparative Power: Observability

task object

physical properties

digital properties

inputdevice

outputdevice

output language

acquiredphysical

data

inputlanguage

generatedphysical

data

mixed tool

physical properties

digital properties

inputdevice

outputdevice

output language

acquiredphysical

data

inputlanguage

generatedphysical

data

interactionoutput

language

interactioninput

language

29

Comparative power: Observabilitymixed tool

physicalproperties

recognizedgestures

camera image inputlanguage

translation

30

Comparative Power: Observabilitymixed tool

physicalproperties

click

touchpad data inputlanguage

translation

31

task object

physical properties

digital properties

inputdevice

outputdevice

output language

acquiredphysical

data

inputlanguage

generatedphysical

data

Comparative Power: Spatial Continuity

mixed tool

physical properties

digital properties

inputdevice

outputdevice

output language

acquiredphysical

data

inputlanguage

generatedphysical

data

interactionoutput

language

interactioninput

language

32

Comparative Power: Spatial Continuitymixed tool

physicalproperties

recognizedgestures

camera image computervision

translation

physical properties

IdentifierLocation

Orientation

head mounteddisplay

3Dprojectionimage

mixed piece

locationsensor

numericvalues

geometrictransforma°

orienta°sensor

numericvalues

geometrictransforma°

33

Comparative Power: Spatial Continuitymixed tool

physicalproperties

click

touchpad data inputlanguage

translation

physical properties

IdentifierLocation

Orientation

head mounteddisplay

3Dprojectionimage

mixed piece

locationsensor

numericvalues

geometrictransforma°

orienta°sensor

numericvalues

geometrictransforma°

34

•Mixed Interaction Model

•Descriptive Power

•Generative and Comparative Powers

•Conclusion and Future Work

Outline

35

•Definition of a Mixed Interaction Model which

•Capitalizes existing approaches to describe mixed systems

•Allows designers to create new interaction modalities

•Allows them to choose between multiple design alternatives

Conclusion

36

•Improving comparative power by identifying more criteria

Future Work

37

•Providing a development environment based on ICARE

Future Work

38

Mixed Reality: A Model of Mixed Interaction

User Interface Engineering TeamCLIPS-IMAG Laboratory, University of Grenoble

Céline Coutrix, Laurence Nigay