2003 Stefan Seipel, IT, Uppsala University

VR based HCITechniques & Application

May 9, 2003

stefan.seipel@it.uu.se

2003 Stefan Seipel, IT, Uppsala University

What is Virtual Reality ?

Coates (1992):Virtual Reality is electronic simulations of environments experienced via headmounted eye goggles and wired clothing enabling the end user to interact in realistic three-dimensional situations.

Greenbaum (1992):Virtual Reality is an alternate world filled with computer-generated images that respondto human movements. These simulated environments are usually visited with the aid ofan expensive data suit which features stereophonic video goggles and fiber-optic gloves.

Krueger (1991):….The term (virtual worlds) typically refers to three-dimensional realities implemented with stereo viewing goggles and reality gloves.

2003 Stefan Seipel, IT, Uppsala University

Variables to define Virtual Reality

Vividness(richness of an environments representation)

• breadth (visibility, audibility, touch, smell)• depth (quality, fidelity)

Interactivity (extend to which a user can modify form and content of a mediated environment)

• speed (update rates, time lag)• mapping (text, speech, gestures, gaze, complex behavior patterns)

2003 Stefan Seipel, IT, Uppsala University

Classification of Virtual Reality and other Media

high

high

low Interactivity

Vividness

book

35mm film

3D IMAX

TV pay-TV

VR

photo-graph

tama-gotchi

Jonathan Steuer

phone

videoconferencing

Sensorama

2003 Stefan Seipel, IT, Uppsala University

History of Virtual Reality(technological milestones)

1956 Sensorama (Morton Heilig)3D visuals, vibration, stereo sound, wind, smell, little interaction

1961 Headsight System (Philco Corp.)HMD, head tracking, remote video camera, telepresence

1965 The Ultimate Display (Ivan Sutherland)Stereoscopic HMD, computer generated images, tracking, visually coupled system

1967 Grope (University of North Carolina)6 degree of freedom force feedback

1977 The Sayre Glove (Sandin, Sayre, DeFanti Univ. Illinois)Gesture recognition

1987 Virtual Cockpit (British Aerospace)head and hand tracking, eye tracking, 3d visuals, 3D audio, speech recognitionvibro tactile feedback

2003 Stefan Seipel, IT, Uppsala University

VR VR -- (visually) coupled systems(visually) coupled systems

computersimulation

multimodalinput

multimediafeedback

real world

replication

real-time

2003 Stefan Seipel, IT, Uppsala University

How Do We Perceive 3D ?

2003 Stefan Seipel, IT, Uppsala University

How Do We Perceive 3D ?

Visual depth cues:

a) monoscopic cuesrelative sizeinterposition and occlusionperspective distortionlighting and shadowstexture gradientmotion parallax

b) binocular (stereoscopic) cuesstereodisparityconvergence

2003 Stefan Seipel, IT, Uppsala University

Binocular VisionBinocular Vision

Binocular focal zonePeriphery field of view

Horizontal disparityConvergence

2003 Stefan Seipel, IT, Uppsala University

How can we recreate 3D sensation ?

Providing visual cues:

a) monoscopic cuesrealistic rendering / lighting simulation

b) stereoscopic cues -> stereodisparitypresentation of appropriate view to each eye - time multiplexing of images- multiplexing with chromatic filters (anaglyph)- multiplexing with polarizer filters- providing two views simultaneously

2003 Stefan Seipel, IT, Uppsala University

Realistic 3D rendering

http://www.agh.edu.pl/htdocs/gifs/computer_art/raytracing/final18.jpg

2003 Stefan Seipel, IT, Uppsala University

Stereoscopic rendering

2003 Stefan Seipel, IT, Uppsala University

Time Multiplexed Stereo Image Pair

V-Sync at 60 Hz

Addidional V-Sync at 120 Hz(enforced with sync. doubler)

open

clos

e

clos

e

open

Active Shutter Glasses(LCD-Shutters)

2003 Stefan Seipel, IT, Uppsala University

Dual Channel Head Mounted Display (HMD)(© nVision)

Datavisor 80

Datavisor

2003 Stefan Seipel, IT, Uppsala University

What Renders VR Applications Specific ?

1. Visual and Acoustic Realism of Objects• 3D effect• level of detail• specularity• color and texture

2. Real-time Response (approx. >15 Hz)

3. Natural Like Interaction Metaphors(many degrees of freedom input)

4. Peripheral Visual Stimuli

2003 Stefan Seipel, IT, Uppsala University

System Architecture in Virtual Environments

DisplayDevices

SensorDevices

Simulation ManagerScene GeneratorSimulation Loop

RenderEngine

SensorHandler

User

Scene DB

2003 Stefan Seipel, IT, Uppsala University

Types of Virtual Reality Environments

1. Immersive Virtual Environments• subjects are visually isolated from the real environment• virtual scene is responding to the subjects actions• subjects are unable to perform in the real environment

2. Semi-Immersive Virtual Environments• subjects can perform both in the real and virtual environment• subjects perceive a strong involvement into the VE• subjects may perform less in the real environment

3. Non-Immersive Virtual Environments• the three-dimensional scene is considered as a part of

the physical environment• subjects do fully respond in the real environment• relatively little involvement into the VE

(4. Augmented Reality Interfaces)

Deg

ree

of I

mm

ersi

onC

onsciousness in Physical E

nvironment

2003 Stefan Seipel, IT, Uppsala University

Display Devices

Visual Displays (3D imagery)• Head Mounted Displays (HMD)• Projection Displays (CAVE, Virtual Plane)

Acoustic Displays (spatial sound)• Multi-Channel Sound Systems

• Specialized Convolution Processors (e.g. Convolvotron)

Haptic Displays (force feedback)• Robot Arms (e.g. Grope, Phantom)

• Active Joystics (e.g. Microsoft Sidewinder)

• Vibrotactile Devices (e.g. Logitec Cyberman)

2003 Stefan Seipel, IT, Uppsala University

An Immersive Car Simulator Using HMD(© British Aerospace)

2003 Stefan Seipel, IT, Uppsala University

A BOOM Display Application in Aerodynamics(© NASA Ames Research Center)

2003 Stefan Seipel, IT, Uppsala University

CAVE - An Immersive VR Environment(EVL, University of Illinois at Chicago)

2003 Stefan Seipel, IT, Uppsala UniversityPlanung - Behandlung - Ausbildung

Six-sided CAVE (KTH, Stockholm)

2003 Stefan Seipel, IT, Uppsala University

The ImmersaDesk - A Semi-Immersive Device(University of Illinois at Chicago)

2003 Stefan Seipel, IT, Uppsala University

Non-Immersive Desktop VR : 3D Implant Planning(© 1995 CMD, Uppsala University)

2003 Stefan Seipel, IT, Uppsala University

The Haptic Display Grope III(© University of North Carolina)

2003 Stefan Seipel, IT, Uppsala University

Examples of Haptic Devices

Low Cost Force Feedback Device

High Fidelity Force Feedback Devices

PHANToM ©SensableDevices

2003 Stefan Seipel, IT, Uppsala University

The Virtual Workbench(© 1998 Kent Ridge Digital Labs (KRDL), Singapore)

2003 Stefan Seipel, IT, Uppsala University

Sensor Devices

1. Spatial Position/Orientation Sensors• 2DOF (Mouse)• 3DOF (Microscribe, FreeD Joystick)• 6DOF (Polhemus Fastrack)

2. Directional Force Sensors• 5 DOF (Spacemouse)• 2 DOF (Joystick)

3. Gesture Recognition• Data Gloves

4. Eye Tracking

5. Speech Recognition Systems

2003 Stefan Seipel, IT, Uppsala University

Example : Spatial Position/Orientation Sensors

Polhemus InsideTrack

FreeD Joystick(UltraSonic Tracking)

MicroScribe(Mechanical Tracking)

(Magnetic Tracking)

2003 Stefan Seipel, IT, Uppsala University

Example : Directional Force Sensors

SpaceMaster

SpaceBall 2003

SpaceBall 3003

2003 Stefan Seipel, IT, Uppsala University

Example : Gesture Recognition

Cyberglove, 5th Dimension

SUPERGLOVE, Nissho

Dextrous Hand Master, Exos

2003 Stefan Seipel, IT, Uppsala University

Example : Eye Tracking Systems

http://psych.utoronto.ca/~reingold/eyelink/eyelink.htm

2003 Stefan Seipel, IT, Uppsala University

Application Examples from the Uppsala VR Lab

2003 Stefan Seipel, IT, Uppsala University

The Virtual Plane(© 1999 CMD, Uppsala University)

Flight Mission RehearsalExamensarbete Anders Seton, VT99

Virtual Implant Planning Stefan Seipel, 1999

2003 Stefan Seipel, IT, Uppsala University

The Role of Dynamic Perspective

The alternate-ego view The primary-ego view

2003 Stefan Seipel, IT, Uppsala University

The Role of Dynamic Perspective

2003 Stefan Seipel, IT, Uppsala University

The Stereoscopic Powerwall(© 1999 CMD, Uppsala University)

2003 Stefan Seipel, IT, Uppsala University

Automatic Dental Automatic Dental Occlusion AnalysisOcclusion Analysis

2003 Stefan Seipel, IT, Uppsala University

Applied CG&VR Research

Multi-User Collaborative Shared Virtual Environments

Virtual Teaching Settings for Learning

low-bandwidth protocols for network VR

intelligent clients rather than full state replication

transformation driven state propagation

highly aggregated objects for net VR

2003 Stefan Seipel, IT, Uppsala University

Virtual Teaching in Computer Graphics Education

2003 Stefan Seipel, IT, Uppsala University

Virtual Teaching in the Dental Curriculum

2003 Stefan Seipel, IT, Uppsala University

Other Examples from the Medical Field

2003 Stefan Seipel, IT, Uppsala University

Planning and Rapid Prototyping Planning and Rapid Prototyping TheThe SurgicaseSurgicase SystemSystem

1999-2000 http://www.materialise.be/surgicase/

• Manual segmentation

• Surface Rendering

• Rapid prototyping

2003 Stefan Seipel, IT, Uppsala University

1999 S. Haßfeld et al.Mund-, Kiefer- und Gesichtschirurgie, Universität Heidelberg

Automatic SegmentationAutomatic Segmentation

• Automatic segmentationof mandibular nerve

• Hybrid rendering

2003 Stefan Seipel, IT, Uppsala University

Registration and Visualizationof Prosthetic Design

• CT Scanning of jaws

• CT Scanning of dentures

• Correlationof datasets

• Planning with regard to

prosthetic restoration

1998 K. Verstreken et al., Laboratory for Medical ImagingResearch (ESAT & Radiology), K.U.Leuven/Belgium

2003 Stefan Seipel, IT, Uppsala University

HapticHaptic Simulators for Training of Simulators for Training of Tooth BrushingTooth Brushing

Quelle: http://wwwipr.ira.uka.de/~salb

2003 Stefan Seipel, IT, Uppsala UniversityQuelle: http://wwwipr.ira.uka.de/~salb

Haptic Manipulator

Visual Simulation

HapticHaptic Simulators for Training of Simulators for Training of Tooth BrushingTooth Brushing

2003 Stefan Seipel, IT, Uppsala University

Computer manufactured mechanical templates

• Stereo-lithography• Machine milling

Planning CAM Template Surgery

http://www.materialise.be/surgicase/

2003 Stefan Seipel, IT, Uppsala University

ArtmaArtma

Interactive Consultation (ref. University of Vienna,Clinic for Oral and maxillofacial Surgery

Augmented Instrument Navigation(Artma, Vienna) www.artma.com

2003 Stefan Seipel, IT, Uppsala University

• Registration of view on real world

• Correlation with computer data

• Fusion of real world view withcomputer generated information

Augmented Vision Guidance

Department of Computer Science, University of North Carolina at Chapel Hill

Department of Computer Science, University of North Carolina at Chapel Hill

2003 Stefan Seipel, IT, Uppsala University

VoxelManVoxelMan -- 3D Anatomy Atlas3D Anatomy Atlas

• Data from “The Visible Human Project”

• Segmentation of anatomic structures

• Labeling of anatomic structures

• Semantic database

2003 Stefan Seipel, IT, Uppsala University

VoxelManVoxelMan -- 3D Anatomy3D Anatomy--AtlasAtlas

2003 Stefan Seipel, IT, Uppsala UniversityQuelle : http://cs.millersv.edu/haptics/lumbar/lumbar.html

A PrototypeA Prototype HapticHaptic LumbarLumbarPuncture SimulatorPuncture Simulator

A Prototype Haptic Lumbar Puncture Simulator, Paul Gorman1, Thomas Krummel1, Roger Webster2, Monica Smith2, David Hutchens2 1Department of Surgery, School of Medicine Stanford University2Department of Computer Science School of Science and Mathematics Millersville University of Pennsylvania

2003 Stefan Seipel, IT, Uppsala University

A PrototypeA Prototype Haptic Haptic Suturing SimulatorSuturing Simulator

A Prototype Haptic Suturing Simulator Randy Haluck, M.D.1, Roger Webster, Ph.D.2,Will Wang, M.D.1, Dean Zimmerman2, Betty Mohler2, Alan Synder, Ph.D.1, Mike Melkonian, M.D.1 1Department of Minimally Invasive Surgery Penn State University College of Medicine 2Department of Computer Science School of Science and Mathematics Millersville University of Pennsylvania

Quelle : http://cs.millersv.edu/haptics/suture/suture.html

2003 Stefan Seipel, IT, Uppsala University

Dynamic Tissue SimulationDynamic Tissue Simulation

Quelle : http://cs.millersv.edu/haptics/suture/suture.html

2003 Stefan Seipel, IT, Uppsala University

3D GUI for mobile computing

Physical client area = virtual client area : 640x480 pixels

2003 Stefan Seipel, IT, Uppsala University

Display tilt at 15 degrees

Virtual client area : 640x526 pixels

2003 Stefan Seipel, IT, Uppsala University

Display tilt at 30 degrees

Virtual client area : 640x626 pixels

2003 Stefan Seipel, IT, Uppsala University

Display tilt at 45 degrees

Virtual client area : 640x844 pixels