Haptic Interaction in Mixed Environments and Virtual Reality

Haptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality Environments

Konstantinos Moustakas, Assistant ProfessorKonstantinos Moustakas, Assistant Professor

Electrical and Computer Engineering Department, University of Electrical and Computer Engineering Department, University of PatrasPatras

•• WhatWhat isis haptichaptic interaction?interaction?•• HapticHaptic psychophysicspsychophysics outlineoutline•• HapticHaptic interactioninteraction essentialessential processingprocessing::

Outline Haptic interactionHaptic interaction

OutlineOutline

•• HapticHaptic interactioninteraction essentialessential processingprocessing::–– CollisionCollision detectiondetection–– HapticHaptic renderingrendering

•• ApplicationsApplications•• FutureFuture researchresearch directionsdirections

• Wikipedia, Haptics: Haptics refers to the sense oftouch (from Greek άπτω = "I fasten onto, I touch).

• Merriam-Webster, Haptic: relating to or based onthe sense of touch

Introduction Haptic interactionHaptic interaction

HapticsHaptics

the sense of touch• Haptics is the science of applying touch

(tactile) sensation and control to interactionwith computer applications.

HapticsHaptics

• Tactile information– The responses of

the receptors of the skin in contact with an objectan object

• Kinesthetic information– Net forces applied

to limbs

Haptic interaction Haptic interaction -- ImportanceImportance

• The sense of touch is underestimated!• Loss of the sense of touch can cause impairments that

cannot be compensated by sight– Hand dexterity– Haptic capabilities, grasping, etc.– Walking– Walking– Limb position perception– Eating– Speaking

• In virtual environments the loss of the sense of touch can impair performance and interaction efficiency

HapticHaptic interactioninteraction -- ImportanceImportance

Strike a match (with vision)

Normal case

7.5 sec

Anesthesia

29 sec

Haptic Psychophysics Haptic interactionHaptic interaction

Different types of tactile sensorsDifferent types of tactile sensors

Surface encoding Surface encoding ––deformation mattersdeformation matters

Spatial resolutionSpatial resolution

Temporal resolutionTemporal resolution

~1 kHz constraint

while in graphics~25 Hz constraint

• Tactile– Surface texture– Orientation– Shape

What do we sense?What do we sense?

• Haptic– Mass/Weight– Force– Stiffness– Shape

– Size– Pressure– Curvature– Motion

– Stiffness– Viscosity

• Thermal– Cooling– Warming

Haptic Rendering Haptic interactionHaptic interaction

HapticsHaptics & Environment& Environment

Haptic Devices Haptic interactionHaptic interaction

• Haptic devices:– Tactile devices– Force feedback devices

• Exoskeletons• One point of interaction devices

Haptic devicesHaptic devices

Haptic devices VS Human SensorsHaptic devices VS Human Sensors

Haptic devicesHuman sensors

Many sensors One sensor (x N)

Direct tactile sensation Only through motion

Deformation N/A

• Haptic devices act as a filter to the human tactile/haptic sensors.

• How would you see a visual scene with such a filter?

Haptic devices VS Human SensorsHaptic devices VS Human Sensors

Haptic vs Graphics rendering processesHaptic vs Graphics rendering processes

ViewViewTransformTransform

LightingLighting ProjectionProjection TexturingTexturingScene Scene

TraversalTraversal

GeometryGeometry RasterizerRasterizerApplicationApplication DisplayDisplay

ForceForce TactileTactileTraversalTraversal DisplayDisplay

ForceForceCalculationCalculation

Force Force SmoothingSmoothing

ForceForceMappingMapping

HapticHapticTexturingTexturing

Collision Collision DetectionDetection

Collision detection Haptic interactionHaptic interaction

• Collision is one of natures powerful tools:– Object interaction

• Walking• Machines are based on collisions

Collision detectionCollision detection

• Machines are based on collisions

– Sound source

• In the real world physical laws and matter take-over

• In the virtual world???

Collision detectionCollision detection

In no case a trivial issue!!!

Bounding volume hierarchiesBounding volume hierarchies

Bounding volume hierarchies (video)Bounding volume hierarchies (video)

Force feedbackForce feedback

A. Vogiannou, K. Moustakas, D. Tzovaras and M.G. Strintzis, “Enhancing Bounding Volumes using Support Plane Mappings for Collision Detection”, Eurographics Computer Graphics Forum, vol. 29, no. 5, pp. 1595-1604, August 2010.

Haptic rendering basicsHaptic rendering basics

Haptic rendering: Stiffness vs StabilityHaptic rendering: Stiffness vs Stability

Force Calculation Force Calculation ––Virtual wallVirtual wall

Insufficient stiffnessInsufficient stiffnessV V <<<<<<<< 00

Virtual wall Virtual wall Moving into the wallMoving into the wall

timetime

K K wallwall •• ∆∆ x + B vx + B vF =F =

SolutionsSolutionsHigh frequencyHigh frequency

Virtual couplingVirtual couplingGod object God object –– virtual proxyvirtual proxy

Force discontinuitiesForce discontinuities

Force shadingForce shading

• Height maps• Friction/Damping fields• Stochastic textures

Haptic texturingHaptic texturing

• Virtual textures/bumps

The force field haptic rendering method

Haptic rendering Haptic interactionHaptic interaction

method

K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia , vol. 14, no. 1, pp. 62-72, January 2007.

From the distance field to the force fieldFrom the distance field to the force field

-di -di

Distance field

Force field

Haptic renderingHaptic rendering• The force feedback is calculated for a given

point through:

• whereD(x) andF(x) are the values and the

∈ 3x R

• whereD(x) andF(x) are the values and the vector of the distance and force field respectively

-di -di

Distance field Force field

SQ-Map: Efficient layered collision detection and haptic renderingdetection and haptic rendering

K. Moustakas, D. Tzovaras and M.G. Strintzis, “SQ-Map: Efficient Layered Collision Detection and Haptic Rendering”, IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 1, pp. 80 - 93, January 2007.

• Motivation: New method fast as the distance fields, without high memory requirements

• SQ-Map steps– Preprocessing

SQSQ--Map overviewMap overview

– Preprocessing• Object approximate segmentation• Superquadric modeling• Distance map generation

– Run-time processing• Layered collision detection (3 levels of processing)

K. Moustakas, D. Tzovaras and M.G. Strintzis, “SQ-Map: Efficient Layered Collision Detection and Haptic Rendering”, IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 1, pp. 80 - 93, January 2007.

SQSQ--Map distance mapsMap distance maps

Collision detection: Layer 1Collision detection: Layer 1

, ,x y z

F x y za a a

εε ε

, ,x y z

F x y za a a

εε ε

If d>DSQ collision is detected

• Similar processing to layer 2 with two add-ons– Possibility to perform distance map

subdivision so as to increase accuracy

subdivision so as to increase accuracy– The Layer 3 distance map is used that

projects also mesh vertices onto the superquadric surfaces

• Lemma: – The distance map and the superquadric can

provide an exact representation of the 3D model iff function fC that maps all points of the

Accuracy analysisAccuracy analysis

model iff function fC that maps all points of the object’s surface onto the superquadric is injective.

– Proof in the text

Accuracy analysisAccuracy analysis

Concave region of typeQQ2

Concave region of type Q1

Haptic renderingHaptic rendering

Force magnitude estimation

Force direction estimation

Can be analytically calculated

Advantages:

• Rapid estimation of the reaction force

• Possibility to analytically process and handle the • Possibility to analytically process and handle the force feedback

Friction:

Haptic texture:

Gaussian noise

ComparisonsComparisons

Qualitative resultsQualitative results

OBB SQ-Map, Layer 1

SQ-Map, Layer 2 SQ-Map, Layer 3

Qualitative resultsQualitative results

Applications Haptic interactionHaptic interaction

• Surgical simulation

• Telemanipulation, robot-assisted surgery

Application examplesApplication examples

• Computer aided design

• Entertainment• Haptic Visualization• Applications for the

disabled• …

Haptic mapsHaptic maps

Haptic map with embedded semantic iformation

Haptic browserHaptic browser

Virtual reality cane simulationVirtual reality cane simulation

D. Tzovaras et.al., “Design and implementation of haptic virtual environments for the training of the visually impaired, IEEE Neural Systems and Rehab. Eng., 2004

Mixed reality cane simulationMixed reality cane simulation

Virtual environment

Virtual

Magnetic sensor 1

CyberGrasp

D. Tzovaras, K. Moustakas, G. Nikolakis and M.G. Strintzis, "Interactive Mixed Reality White Cane Simulation for the Training of the Blind and the Visually Impaired", Springer Journal on Personal and Ubiquitous Computing, vol.13, no.1, pp.51-58, January 2009.

Virtual cane

Cane replica

CyberGrasp

Magnetic sensor 2

VR game for the hearing and the visually impaired

visually impaired

K. Moustakas, L. Dybkjaer, O. Aran, D. Tzovaras and N.O. Bernsen, "Communication Between Blind and Hearing Impaired People Through a Multimodal Interactive Game", IEEE Multimedia , accepted for publication.

The ideal caseThe ideal case

ArchitectureArchitecture

Game stepsGame steps

Game videoGame video

Future work Haptic interactionHaptic interaction

• Haptic visualization– Use the haptic channel to complement vision

in information visualization applications

• Haptic interaction with videos

Future directionsFuture directions

• Haptic interaction with videos– Highly dynamic

• Content• Interaction

– Partial input of the environment, ill-posed

• Towards a theory of haptic rendering…

Haptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality Environments

Thank you for your

Konstantinos Moustakas, Assistant Professor Konstantinos Moustakas, Assistant Professor moustakas@ece.upatras.grmoustakas@ece.upatras.gr

Electrical and Computer Engineering Department, University of Electrical and Computer Engineering Department, University of PatrasPatras

Thank you for your attention

Haptic Interaction in Mixed Environments and Virtual Reality

Technology

Augmented Reality Environments in Learning ... · !! 22!! Augmented Reality Environments in Learning, Communicational and Professional Contexts in Higher Education! J.Martín Gutiérrez

HAPTIC COMMUNICATION FOR REMOTE ENVIRONMENTSusir.salford.ac.uk/2039/1/thesis.pdf · HAPTIC COMMUNICATION FOR REMOTE MOBILE AND MANIPULATOR ROBOT OPERATIONS IN HAZARDOUS ENVIRONMENTS

HARP: A Framework for Visuo-Haptic Augmented Reality · HARP: A Framework for Visuo-Haptic Augmented Reality Ulrich Eck Christian Sandor† Magic Vision Lab University of South Australia

Assessment of Virtual Reality Environments for design

AirPiano: Enhancing Music Playing Experience in Virtual ...jessekim.com/docs/AirPiano.pdf · system in virtual reality with mid-air haptic ... via ultrasonic non-contact haptic

Touching the Past: Haptic Augmented Reality for Museum ... · PDF fileTouching the Past: Haptic Augmented Reality for Museum Artefacts ... engagement in museums which seeks to

ISMAR 2014: Comprehensive Workspace Calibration for Visuo-Haptic Augmented Reality

Virtual Reality Without Vision: A Haptic and Auditory …...Virtual Reality Without Vision: A Haptic and Auditory White Cane to Navigate Complex Virtual Worlds Alexa F. Siu Stanford

Bimanual Haptic Interaction with Virtual Environments …videos.rennes.inria.fr/soutenance-AnthonyTalvas/Slides-talkAnthon... · Bimanual Haptic Interaction with Virtual Environments

EVALUATION OF HAPTIC VIRTUAL REALITY GUIDANCE IN …nearlab.polimi.it/wp-content/uploads/2019/02/berni_presentation.pdf · EVALUATION OF HAPTIC VIRTUAL REALITY GUIDANCE IN PROSTATE

Haptic Interface to Virtual Environments

Multi-modal musical environments for mixed-reality performancerob/papers/springer_jmui_hamilton.pdf · Multi-modal musical environments for mixed-reality performance Robert Hamilton

CHAPTER 4: Haptic Devices for Virtual Reality, Telepresence and Human-Assistive Robotics

Haptic Devices in Virtual reality

Augmented Reality Environments in Learning

Virtual Environments – Including Virtual Reality

Soft robotic glove for kinesthetic haptic feedback in ...web.eng.ucsd.edu/~jschulze/publications/Jadhav2017.pdfSoft robotic glove for kinesthetic haptic feedback in virtual reality

Stable Haptic Interaction with Virtual Environments Using an … · 2001-06-15 · Stable Haptic Interaction with Virtual Environments Using an Adapted Voxmap-PointShell Algorithm

Preliminary Experiment Combining Virtual Reality …haptic/pub/RN-ET-AL-EH-10.pdfPreliminary Experiment Combining Virtual Reality Haptic Shoes And Audio Synthesis Rolf Nordahl 1, Amir

ExoInterfaces: Novel Exosceleton Haptic Interfaces …files.tachilab.org/.../tsetserukou201004ACMAH.pdfExoInterfaces: Novel Exosceleton Haptic Interfaces for Virtual Reality, Augmented