Raskar Graphics Interface May05

Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005

Ramesh RaskarMitsubishi Electric Research Labs (MERL)

Cambridge, MA

Special Effects in the Real WorldSpecial Effects in the Real World


Special EffectsSpecial Effects

– Photorealism •Stays on screens•Yet to affect daily life in real time

– Entertainment and Visualization– Unlike other fields of CS


Changing AppearanceChanging AppearanceVirtual light

source


Changing Virtual IlluminationChanging Virtual Illumination



Special Effects in the Real Special Effects in the Real WorldWorld

– Photorealism •Limited to display screens•Yet to affect daily life in real time

– Entertainment and Visualization– Unlike other fields of CS

– Fusion: real world with graphics•Next big challenge in CG ?•Believable co-existence


Changing AppearanceChanging Appearance

Projector Projector

Virtual light source


Changing Virtual IlluminationChanging Virtual Illumination

Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)


Virtual MotionVirtual Motion


Virtual MotionVirtual Motion

Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)


Dynamic AugmentationDynamic Augmentation

Projecting on Tracked Objects


Virtual Reflectance Virtual Illumination

InteractionVirtual Motion

ShaderLampsShaderLamps

www.ShaderLamps.com


Projector-based AugmentationProjector-based Augmentation

www.ShaderLamps.com

Virtual Reflectance Virtual Illumination

InteractionVirtual Motion


Poor Man’s PalacePoor Man’s Palace


Maya: World is an IllusionMaya: World is an Illusion

• Fake World– We all live in one

• Real-life Fakes– Not just photos and videos but surroundings

• Privacy


Augmented Reality (AR)Augmented Reality (AR)

With See-through Head-mounted Display

© Andrei State


hand-helddisplay

spatial see-throughdisplay

head-mounted display

retinal display

Head-attached Body-attached spatial

projector

projector

projector

Spatial Augmented Reality

Classification of Augmented RealityClassification of Augmented Reality


Being ThereBeing There• Recreate Large Environments

– ‘BeingThere’, walk-around– Museums, Exhibitions, Training

Kok-Lim Low, Greg Welch, Anselmo Lastra, Henry Fuchs. “Life-Sized Projector-Based Dioramas,” Proc. ACM Symposium on Virtual Reality Software and Technology 2001 (VRST 2001), November 2001.


Desired Virtual Model

© Andrei State

Projected Guidance for Placement



Head-mounted Display Projectors

© Andrei State

Augmented RealityAugmented Reality


OpportunitiesOpportunities

• Information Augmentation- Office, factory or assembly Training

• Increasing value of real products - Smart illumination to enhance appearance of object

• Entertainment- Animating objects, Museums, Special Effects, Laser Shows

• Advertising- Car showrooms, Furniture store

• Design and Ergonomics- Protyping products, visualize material changes

Projector + Camera


• Why– Special Effects in the Real World– Photorealism for everyday

• Tools for Augmentation– Projectors, Sensors, RFID

• Illusion– Beyond Visual Senses– Beyond Projectors– Beyond Gouraud

Spatially Augmented RealitySpatially Augmented Reality


Spatially Augmented Reality Spatially Augmented Reality (SAR)(SAR)

HMD-VR Spatially Immersive-VR

AR using HMDSpatially Augmented

Reality

e.g. CAVE

Video or Optical see-through

SAR, Shaderlamps


AR IssuesAR Issues

• Preprocessing:– Authoring

• Runtime:– Identification: Recognition of objects

• Using markers and visual tags– Registration: Finding relative pose of display

device• Dynamic estimate of translation and rotation• Render/Warp images

– Interaction:• Widgets, Gesture recognition, Visual feedback



Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines‘iLamps: Geometrically Aware and Self-Configurable Projectors’,

SIGGRAPH 2003


AR IssuesAR Issues

• Preprocessing:– Authoring

• Runtime:– Identification: Recognition of objects

• Using markers and visual tags– Registration: Finding relative pose of display

device• Dynamic estimate of translation and rotation• Render/Warp images

– Interaction:• Widgets, Gesture recognition, Visual feedback

– Photorealism ?

RFID

Cameras

Trackers


StepsSteps

– Preprocessing• Scan 3D object• Roughly align projectors• Compute correction

– Run time• Render images of 3D model• Correct


StepsSteps– Preprocessing

• Scan 3D object and create virtual model

Faro arm



• Scan 3D object and create model• Approximately position projector(s)• Calibrate by finding pose



• Scan 3D object and create model• Approximately position projector(s)• Calibrate by finding pose

– Run-time• Inverse Rendering of 3D Models


Warehousing

Routing

Library Baggage

handling

Currency

Livestock tracking

RFID


Micro Controller

Memory Computer

READER

Micro Controller

Memory

Conventional Passive RFID


Tagged Books in a LibraryTagged Books in a Library Id : List of books in RF range

No Precise Location Data Are books in sorted order ?

Which book is upside down ?


Handheld ProjectorHandheld Projector

iLamps 2002 RFIG Lamps 2003-04

Pocket Projector 2004-05


Mitsubishi Pocket ProjectorMitsubishi Pocket Projector



Prototype TagPrototype TagRF tag +

photosensor


READER

Micro Controller

RF Data

Memory

Conventional RFID

Computer

READER

Micro Controller

RF Data

Light

Memory

Photosensor

Computer

Conventional RF tag

Photo-sensing RF tag


READER

Projector Micro

Controller RF Data

Light

Memory

Photosensor

Computer


Find tag location using

handheld Projector

Photosensing

Wireless Tags

Many geometric ops

R F I R F I

R F I DR F I D

Interactive stabilized projection

(Radio Frequency Id & Geometry)(Radio Frequency Id & Geometry)GG

Siggraph 2004


AR with Photosensing RFID AR with Photosensing RFID and Handheld Projectorand Handheld Projector


Visual feedback of 2D positionVisual feedback of 2D position

a. Receive via RF {(x1,y1), (x2,y2), …} pixels

b. Illuminate those positions


RFID(Radio Frequency Identification)

RFIG(Radio Frequency Id and Geometry)


CurvedPlanar Non-planar

SingleProjector

MultipleProjectors

Projector

jUser : T

?

Pocket-ProjObjects

Projectors and SurfacesProjectors and Surfaces

2000

2000 2002

2001

1999

2002

20031999

1998


Pieces of the PuzzlePieces of the PuzzleActuated SurfacesActuated Surfaces

Actuated Workbench [Pangaro, Maynes-Aminzade, Ishii UIST 2002]


• Why– Special Effects in the Real World– Photorealism for everyday

• Tools for Augmentation– Projectors, Sensors, RFID

• Illusion– Beyond Visual Senses– Beyond Projectors– Beyond Gouraud



Believable Co-existenceBelievable Co-existence

• Other Senses – Haptic, Smell, Taste, Locomotion

• Beyond Gouraud Effects– Complex BRDF

• Display Choices– Polymers, e-Ink


Augmentation of “five senses”Augmentation of “five senses”

• Visual Displays• Auditory Displays• Haptic Displays• Locomotion Interfaces• Olfactory Displays• Taste Displays ?• Sixth Sense ..

Extendingsensorymodalities


Sense of BalanceSense of Balance

Galvanic Vestibular Stimulation

Masahiko Inami (UEC)


Electric Cutaneous DisplayElectric Cutaneous Display

Skin

Nerve

Electrode

Dr. H. Kajimoto, U TokyoMasahiko Inami (UEC)


Smart TouchSmart Touch

Masahiko Inami (UEC)


Scent DisplayScent Display

Tracking nose direction

Aiming atthe nose

Air cannon

Deliveringscented air byvortex rings Camera

Platform

Yasuyuki Yanagida, ATR

Spatio-Temporal Controlled Delivery of Scent


Air CannonAir Cannon

Generating a doughnut-shaped vortex ring

Aperture

Push


Air CannonAir Cannon


Prototype SystemPrototype System

• Nose tracking• Multiple scents• Scent switching• Improved delivery range

Yasuyuki Yanagida, ATR


Beyond White ObjectsBeyond White Objects

Grossberg, Nayar et alGrossberg, Nayar et al



Grossberg, Nayar et alGrossberg, Nayar et al


The Virtual ShowcaseThe Virtual Showcase

Oliver BimberOliver BimberTracking and MirrorsTracking and Mirrors


DisplaysDisplays ContendersContenders

OrganicLED

LightEmittingPolymers

E-Ink


HolodustHolodustPerlin et alPerlin et al


Fake WorldFake World


•Participating Media –simulating or in presence of smoke, fog

•Complex Geometry–Spaghetti

•Complex Reflectance –Arbitrary appearance on arbitrary BRDF surfaces

•Shape and Motion –Transformation and Animation of real surfaces

•NPR, Distortions, Perceptual factors–Great thesis topics ..

Beyond Gouraud Shading of White ObjectsBeyond Gouraud Shading of White Objects


RecapRecap• Special Effects in Real World

– Photorealism yet to impact daily life– Poor Man’s Palace

• Spatial Augmented Reality– Solution for fusion– Geometry, Photometry, Id– Sense, Control, Compensate– Needs hardware components: Projectors, RFID, Sensors

• Open Problems– Natural phenomenon, complex BRDF, other displays– All senses: haptic, olfactory, preprioception

• Next Challenge: Photorealism


Spatial Augmented RealitySpatial Augmented Reality

• Graphics– Photorealism yet to impact daily life– Special effects in real world

• Spatial Augmentation– Solution for fusion– Needs hardware components– Projectors, RFID, Sensors

• Next Challenge: Photorealism

raskar.com


MoreMore

• Networked fake world• Poor man’s palace• Intelligence moving from PC to Proj to

Surf• Electronic items eventually become

cheaper than those scavenged from nature


Special EffectsSpecial Effects

– Photorealism • Limited to display screens• Does not affect daily life in real

time– Entertainment and Visualization– Unlike other fields of CS

• Provoke not offend• The primary advantage AR has over VR is

that AR uses the parts of the body that aren't just the eyes and ears: proprioception, vestibular perception, and othe cues that old-fashioned VR just can't handle. The disjunct between vestibular information and visual information that you get in VR is the source of the motion-sickness that often accompanies it.

• VR, like a lot of early 'cyberspace' mythology, was built on an unrealistic rejection of the body, and a fantasy of "pure mind."

• Bud Fake Cup of Coffee 3000http://www.visit4info.com/details.cfm?

adid=21038


ApplicationsApplications– Static setup

• Color everyday objects• Engineering drilling locations• Stage backdrops, seasons or aging

– Laser shows on buildings

– Simulate motion• Car shows, wheels, dirt marks• Virtual moving lights


ApplicationsApplicationsIndoors, under controlled lightingIndoors, under controlled lighting

– Architectural models• Augment walk-around scaled model of buildings• Project and ‘paint’ surface colors, textures• Lighting, sunlight, seasons• Internal structure, pipes, wiring

– Assembly line• Instructional text, images and procedures

– Entertainment• Live shows, exhibits, demonstrations


Tools for AugmentationTools for Augmentation

• Environment– Sense, Compensate and Control– Id, Geometry, Photometry(reflectance/Illumination), Id

• Display

• Rendering