1 1 Contribution to the study of visual, auditory and haptic rendering of information of contact in virtual environments 9/12/2008 Jean Sreng Advisors:

1

1

Contribution to the study of visual, auditory and haptic rendering of information of contact in virtual environments

9/12/2008

Jean Sreng

Advisors:Claude AndriotAnatole Lécuyer

Director:Bruno Arnaldi

2

2

Introduction

• Context: Manipulation of solid objects in Virtual Reality

• Example applications: industrial virtual assembly / disassembly / maintenance

• Focus: perception, simulation, rendering and of contacts between virtual objects

3

3

Outline

• State of the art on perception, simulation and rendering of contact

• Contributions• Integrated 6DOF multimodal

rendering approach

• Visual of rendering of multiple contacts

• Spatialized haptic rendering of contact

• Conclusion

Integrated 6DOF Multimodal rendering

Visual rendering of multiple contacts

Spatialized haptic rendering

4

4

Human perception of contact

• Visual perception of contact:• Stereoscopy (Hu et al. 2000)

• Motion parallax (Wanger et al. 1992)

• Shadows (Wanger et al. 1992, Hu et al. 2002)

• Auditory perception of contact:• Contact properties can be directly perceived (Gaver 1993)

• Contact sounds conveys information about shape and material (Klatzky 2000, Rochesso 2001)

Perception of contactSimulation of contactRendering of contact

5

5

Haptic perception

• Haptic perception provides an intuitive way to feel the contact (Loomis et al. 1993) :• Tactile perception (patterns at the surface of the skin)

• Kinesthetic perception (position and forces)

• Physical properties can be perceived through the contact (Klatzky

et al. 2003) :• Shape / Texture / Temperature

• Weight / Contact forces

• Perception of contact features through vibrations• Material (Okamura et al. 1998)

• Texture (Lederman et al. 2001)


6

6

Multimodal perception of contact

• Known Interaction between modalities• Visual-Auditory interaction

- Ex: Sound can shift the perception of impact (Sekuler et al. 1997)

• Visual-Haptic interaction - Ex: Pseudo-haptic feedback (Lécuyer et al. 2002)

• Auditory-Haptic interaction- Ex: Sound can modulate the perception of roughness (Peeva et al. 1997)

“Stiff”

“Soft”


7

7

Simulation of contact

• Multiple contact models (impact/friction)• Rigid (Newton/Huygens impact law, Coulomb/Amontons friction law)

• Locally deformable (Hunt-Crossley impact law, LuGre friction law)

• Multiple simulation methods• Collision detection

- VPS (McNeely 1999)

- LMD (Johnson 2003)

• Physical simulation- Constraint based (Baraff 1989)

- Penality (Moore 1988)


8

8

Visual rendering

• Visual rendering of the information of contact• Proximity :

- Color (McNeely et al.2006)

• Contact :- Color (Kitamura et al.1998)

- Glyph (Redon et al. 2002)

• Force :- Glyph (Lécuyer et al. 2002)


9

9

Auditory rendering

• Realistic rendering of contact sounds• Specific: Impact/Friction/Rolling

• Different techniques: FEM (O’Brien 2003), modal synthesis (Van den

Doel 2001)

• Symbolic rendering (Richard et al. 1994, Massimino 1995, Lécuyer et al. 2002)

• Associate an information to a sound effect- Information: Distances / Contact / Forces

- Sound effect: Amplitude / Frequencies


10

10

Haptic display of contact

• Haptic devices (Burdea 96) :• Force feedback

• Tactile feedback

• Haptic rendering of contact :• Closed loop (McNeely et al. 1999, Johnson 2003)

tradeoff between stability / stiffness

• Open loop (Kuchenbecker et al. 2006)

improve the realism of impactDevice Virtual

Env.

Impact


11

11

Objectives of this thesis

• Improve the simulation, rendering, and perception of contacts in virtual environments

Integrated 6DOF multimodal rendering



• Protocol :

• Integrated 6DOF approach for multisensory rendering techniques

• Study of rendering techniques to improve the perception of contact position

- Hypothesis of improvement

- Experimental implementation

- Experimental evaluation

12

12

Outline




13

13

Objectives

• Multiplicity of techniques:

• Contact simulation

• Sensory rendering

• How can we integrate all techniques seamlessly together independently from the simulation ?

• Contribution / Overview• Contact formulation (states / events)

• Example of contact rendering based on this formulation : Visual, Auditory, Tactile, Force feedback


14

14

Contact formulation

• Simple contact formulation based on :

• proximity points a, b

• Force f


15

15

Contact formulation

• From this formulation :

• Contact states

• Temporal evolution of contact states (such as events) :– Higher level information

– Adapted to many specific rendering techniques

Free motion ContactImpact Detachment


16

16


Determination of states and events

• The contact condition :

• The events are defined by :• Impact :

• Detachment :

Local linear velocity

Normal

ContactImpact Detachment

17

17

Multimodal rendering architecture


18

18

Multimodal rendering architecture

• Superimpose states and events information:


19

19

Example of Visual rendering

ContactImpact Detachment


20

20


Example of Auditory rendering

21

21

Multimodal rendering platform

• Multimodal:

• Visual

• Auditory

• Tactile

• force-feedback

© Hubert Raguet / CNRS photothèque


22

22

Preliminary conclusion

• We proposed a contact formulation (proximity/force) :

• Contact states and events

• We developed a multimodal rendering architecture :

• Visual (particles / pen)

• Auditory (modal synthesis / spatialized)

• Tactile (modal synthesis)

• 6DOF Haptic enhancement (openloop)


23

23

Outline




24

24

Visual rendering of multiple contactsObjectives

• Context: complex-shaped objects• Multiple contacts

• Difficult interaction Help the user by providing position information

• Contribution / Overview• Display the information of proximity / contact / forces

- Glyphs

- Lights

• Subjective evaluation

25

25



• Visualizing multiple proximity / contact / forces positions

Proximity Contact Contact forcesFAR NEAR LOW HIGH

26

26

Visual rendering using glyphs

Proximity Contact Contact forcesFAR NEAR LOW HIGH


27

27

Visual rendering using glyphs


28

28

Glyph filtering

• Reduce the number of displayed glyphs

• determine “relevance” based on the movement


29

29

Glyph filtering




30

30

Glyph filtering




31

31

Glyph filtering

vd

• The relevance is determined by comparing :

• The local velocity v• The local normal d


32

32

Visual rendering using lights

• Two types of lights :

• Spherical lights

• Conical lights


33

33

Visual rendering using lights


34

34

Subjective evaluation

• Objective: Determine user’s preferences about the different techniques

• Procedure: Participants were asked to perform an industrial assembly operation

• Without visual cues

• With each visual cues

• Conducted on 18 subjects

• They had to fill a subjective questionnaire

• Which effect : glyph / light / color change / size change / deformation

• For which info : forces / distances / blocking / focus attention


35

35

Results

• The visual effects were globally well appreciated

• Significant effects :

• Glyph Size effect globally appreciated (distance / force)

• Glyph Deformation effect appreciated to provide force information

• Light effect appreciated to attract visual attention

“Lower is better”


Mean ranking

36

36


• We proposed a visual rendering technique to display multiple contact information• Display Proximity / Contact / Force

• Using Glyphs/Lights

• We presented a filtering technique to reduce the number of glyphs displayed

• We conducted a subjective evaluation• Glyph size: proximity / force

• Glyph deformation: force

• Lights: focus the visual attention


37

37

Outline




38

38

Objectives

• Context: Complex-shaped objects Help the user by providing position information

• Provide contact position information using :• Visual rendering (particles/glyphs/lights)

• Auditory rendering (spatialized sound) Can we provide this position information using haptic rendering ?

• Contribution / Overview• Haptic rendering technique based on vibrations

- Perceptive evaluation in a 1DOF case

• 6DOF Haptic rendering technique- Perceptive evaluation to determine rendering parameters

- Subjective evaluation in a 6DOF case


39

39


Haptic rendering of contact position

• The impact between objects :• A reaction force of contact

• A high frequency transient vibrations

• This high-frequency transient vibrations depends on :• The object’s material (Okamura et. al 1998)

• The object’s geometry

• The impact position

• Is-it possible to perceive the impact position information using these vibrations?

40

40

Spatialized haptic renderingVibrations depending on impact positions

• Examine the vibrations produced by a simple object : A cantilever beam

• The vibrations depend on the impact position

41

41


Simulation of vibrations (Euler-Bernouilli)

• General solution :

Euler Bernoulli model (EB)

42

42

Simplified vibration patterns

• Simplified patterns based on the physical behavior• Maybe easier perception ?

• Simplified computation

• Chosen model : exponentially damped sinusoid• Amplitude changes with impact position

• Frequency changes with impact position

• Both amplitude and frequency changes


43

43

Simplified vibration patterns

Am

Fr

AmFr (Consistent)

AmCFr (Conflicting)

Near impact Far impact

44

44

Evaluation

• Objective: “Determine it is possible to perceive the impact position using vibration”

• Population: 15 subjects

• Apparatus :• Virtuose6D device

• Sound blocking noise headphones


45

45

Procedure

• Task: Do two successive impacts. “Between these two impacts which one was the closest one from the hand ?”

• 6 models- 2 realistic models (Euler-Bernoulli) (EB1, EB2)

- 4 simplified models (Am, Fr, AmFr, AmCFr)

• 4 impact positions

• 8 random repetitions

• Total of 576 trials (40 min)


46

46

Results of quantitative evaluation

• “How well was the subject able to determine the impact position by sensing the vibration ?”

• Overall performance :• ANOVA Significant (p < 0.007)

• Paired t-tests (p < 0.05) :– Am -

– EB1– EB2– AmCFr

– Fr -– EB1– EB2

RealisticEuler-Bernoulli

Simplified


47

47

Results of qualitative evaluation

• “How was the subjective feeling of realism ?”

• Rate the impact realism :• Paired t-tests (p < 0.05):

– Am– EB1– EB2– Fr– AmFr

– EB2– Fr– AmCFr

RealisticEuler-Bernoulli

Simplified


48

48

• Many participants inverted the interpretation of the vibrations

Sensed Perceived

Normal Inverted


Quantitative evaluation and inversion

49

49

Discussion

• Global weak inter – subject correlation : • Each subject seems to have his/her own interpretation (inversion or not)

• Strong intra – subject consistency :• Subjects seem to be very consistent within his/her interpretation

• Several strong inter – subject correlations between models• Several models interpreted the same way

• Vibrations can be used to convey impact position information


50

50

6DOF Spatialized haptic rendering

• Generalize the previous result for 6DOF manipulation :

• Virtual beam model


51

51


Manipulation point and circle of confusion

• Different impact positions can generate the same haptic feedback

52

52

Perceptive evaluation

• Objective: “Is it possible to perceive such complex vibrations: Is it possible to perceive the vibration direction ?”

Determine the optimal amplitude / frequency vibration parameters allowing a good direction discrimination


• Test among: ( 4 amplitudes a ) x ( 4 frequencies f )


53

53

Procedure and plan

• “On which axis the vibration was applied ?” ● ● ●

• 15 blocks of 4 x 4 x 3 = 48 vibrations = 720 trials (35min)

●

●

●


54

54


Perceptive evaluation : results

4 amplitudes a

4 frequencies f

• Best performances:• Low frequencies

• High amplitudes

• Strategy:• Most participants relied on

intuitive perception

55

55


Subjective evaluation

• Objective: Evaluation in a real case


• Test without and with vibrations

• Subjective ratings

• Impact realism

• Impact position

• Comfort

• This evaluation provides encouraging results

*

56

56


• We proposed a method to provide impact position directly on the haptic channel information using vibrations based on a vibrating beam

• We conducted a 1DOF study and perceptive evaluation (“realistic” / “simplified” models)

- Simplified models achieved better performance

• We extended this method for 6DOF manipulation

• Perceptive study on the perception of vibration direction

- Low frequencies / High amplitudes are better

• Subjective study on a “real case”

- Better subjective perception of impact position


57

57

Conclusion




• Contributions:

• Integrated 6DOF approach for multisensory rendering techniques

• Study of rendering techniques to improve the perception of contact position

- Visual rendering technique to display multiple contact

- Haptic rendering technique to provide position information using vibrations

58

58

Perspective and future work

• Computer side:

• Higher level of information of contact (mobility)

• Rendering improvements (visual, auditory, tactile, force-feedback)

• Deeper investigation on vibratory tactile rendering

• Human side:

• Perceptive studies: multimodal perceptive effects / pseudo-haptics

• Quantitative evaluation of the rendering techniques in 6DOF manipulations

59

59

Publications

• Jean Sreng, Anatole Lécuyer, Christine Mégard, and Claude Andriot. Using Visual Cues of Contact to Improve Interactive Manipulation of Virtual Objects in Industrial Assembly/Maintenance Simulations. IEEE Transactions in Visualization and Computer Graphics, 12(5):1013–1020, 2006

• Jean Sreng, Florian Bergez, Jérémie Legarrec, Anatole Lécuyer, Claude Andriot. Using an event-based approach to improve the multimodal rendering of 6DOF virtual contact. In Proceedings of ACM Symposium on Virtual Reality Software and Technology, pages 165–173, 2007

• Jean Sreng, Anatole Lécuyer, Claude Andriot. Using Vibration Patterns to Provide Impact Position Information in Haptic Manipulation of Virtual Objects. In Proceedings of EuroHaptics, pages 589–598, 2008

• Jean Sreng, Anatole Lécuyer, Claude Andriot. Spatialized Haptic Rendering: Improving 6DOF Haptic Simulations with Virtual Impact Position Information, 2009, In Proceedings of IEEE Virtual Reality, 2009, Accepted paper

• Jean Sreng, Legarrec, Anatole Lécuyer, Claude Andriot. Approche Evénementielle pour l’Amélioration du Rendu Multimodal 6DDL de Contact Virtuel. Actes des journées de l’Association Française de Réalité Virtuelle, 97–104, 2007

• Jean Sreng, Anatole Lécuyer. Perception tactile de la localisation spatiale des contacts. Sciences et Technologies pour le Handicap, 3(1), 2009, Invited paper

60

60

Thank you. Questions ?

• Jean Sreng, Anatole Lécuyer, Christine Mégard, and Claude Andriot. Using Visual Cues of Contact to Improve Interactive Manipulation of Virtual Objects in Industrial Assembly/Maintenance Simulations. IEEE Transactions in Visualization and Computer Graphics, 12(5):1013–1020, 2006

• Jean Sreng, Florian Bergez, Jérémie Legarrec, Anatole Lécuyer, Claude Andriot. Using an event-based approach to improve the multimodal rendering of 6DOF virtual contact. In Proceedings of ACM Symposium on Virtual Reality Software and Technology, pages 165–173, 2007

• Jean Sreng, Anatole Lécuyer, Claude Andriot. Using Vibration Patterns to Provide Impact Position Information in Haptic Manipulation of Virtual Objects. In Proceedings of EuroHaptics, pages 589–598, 2008

• Jean Sreng, Anatole Lécuyer, Claude Andriot. Spatialized Haptic Rendering: Improving 6DOF Haptic Simulations with Virtual Impact Position Information, 2009, In Proceedings of IEEE Virtual Reality, 2009, Accepted paper

• Jean Sreng, Legarrec, Anatole Lécuyer, Claude Andriot. Approche Evénementielle pour l’Amélioration du Rendu Multimodal 6DDL de Contact Virtuel. Actes des journées de l’Association Française de Réalité Virtuelle, 97–104, 2007

• Jean Sreng, Anatole Lécuyer. Perception tactile de la localisation spatiale des contacts. Sciences et Technologies pour le Handicap, 3(1), 2009, Invited paper

Documents

1 1 Contribution to the study of visual, auditory and haptic rendering of information of contact in virtual environments 9/12/2008 Jean Sreng Advisors: