Raskar Graphics Interface May05

Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2005

Ramesh RaskarMitsubishi Electric Research Labs (MERL)

Cambridge, MA

Special Effects in the Real WorldSpecial Effects in the Real World

Special EffectsSpecial Effects

– Photorealism •Stays on screens•Yet to affect daily life in real time

– Entertainment and Visualization– Unlike other fields of CS

Changing AppearanceChanging AppearanceVirtual light

source

Changing Virtual IlluminationChanging Virtual Illumination

Special Effects in the Real Special Effects in the Real WorldWorld

– Photorealism •Limited to display screens•Yet to affect daily life in real time

– Entertainment and Visualization– Unlike other fields of CS

– Fusion: real world with graphics•Next big challenge in CG ?•Believable co-existence

Changing AppearanceChanging Appearance

Projector Projector

Virtual light source

Changing Virtual IlluminationChanging Virtual Illumination

Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)

Virtual MotionVirtual Motion

Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)

Dynamic AugmentationDynamic Augmentation

Projecting on Tracked Objects

Virtual Reflectance Virtual Illumination

InteractionVirtual Motion

ShaderLampsShaderLamps

www.ShaderLamps.com

Projector-based AugmentationProjector-based Augmentation

www.ShaderLamps.com

Virtual Reflectance Virtual Illumination

InteractionVirtual Motion

Poor Man’s PalacePoor Man’s Palace

Maya: World is an IllusionMaya: World is an Illusion

• Fake World– We all live in one

• Real-life Fakes– Not just photos and videos but surroundings

• Privacy

Augmented Reality (AR)Augmented Reality (AR)

With See-through Head-mounted Display

hand-helddisplay

spatial see-throughdisplay

head-mounted display

retinal display

Head-attached Body-attached spatial

projector

Spatial Augmented Reality

Classification of Augmented RealityClassification of Augmented Reality

Being ThereBeing There• Recreate Large Environments

– ‘BeingThere’, walk-around– Museums, Exhibitions, Training

Kok-Lim Low, Greg Welch, Anselmo Lastra, Henry Fuchs. “Life-Sized Projector-Based Dioramas,” Proc. ACM Symposium on Virtual Reality Software and Technology 2001 (VRST 2001), November 2001.

Desired Virtual Model

Projected Guidance for Placement

Head-mounted Display Projectors

Augmented RealityAugmented Reality

OpportunitiesOpportunities

• Information Augmentation- Office, factory or assembly Training

• Increasing value of real products - Smart illumination to enhance appearance of object

• Entertainment- Animating objects, Museums, Special Effects, Laser Shows

• Advertising- Car showrooms, Furniture store

• Design and Ergonomics- Protyping products, visualize material changes

Projector + Camera

• Why– Special Effects in the Real World– Photorealism for everyday

• Tools for Augmentation– Projectors, Sensors, RFID

• Illusion– Beyond Visual Senses– Beyond Projectors– Beyond Gouraud

Spatially Augmented RealitySpatially Augmented Reality

Spatially Augmented Reality Spatially Augmented Reality (SAR)(SAR)

HMD-VR Spatially Immersive-VR

AR using HMDSpatially Augmented

Reality

e.g. CAVE

Video or Optical see-through

SAR, Shaderlamps

AR IssuesAR Issues

• Preprocessing:– Authoring

• Runtime:– Identification: Recognition of objects

• Using markers and visual tags– Registration: Finding relative pose of display

device• Dynamic estimate of translation and rotation• Render/Warp images

– Interaction:• Widgets, Gesture recognition, Visual feedback

Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines‘iLamps: Geometrically Aware and Self-Configurable Projectors’,

SIGGRAPH 2003

AR IssuesAR Issues

• Preprocessing:– Authoring

• Runtime:– Identification: Recognition of objects

• Using markers and visual tags– Registration: Finding relative pose of display

device• Dynamic estimate of translation and rotation• Render/Warp images

– Interaction:• Widgets, Gesture recognition, Visual feedback

– Photorealism ?

Cameras

Trackers

StepsSteps

– Preprocessing• Scan 3D object• Roughly align projectors• Compute correction

– Run time• Render images of 3D model• Correct

StepsSteps– Preprocessing

• Scan 3D object and create virtual model

Faro arm

• Scan 3D object and create model• Approximately position projector(s)• Calibrate by finding pose

– Run-time• Inverse Rendering of 3D Models

Warehousing

Routing

Library Baggage

handling

Currency

Livestock tracking

Micro Controller

Memory Computer

READER

Micro Controller

Memory

Conventional Passive RFID

Tagged Books in a LibraryTagged Books in a Library Id : List of books in RF range

No Precise Location Data Are books in sorted order ?

Which book is upside down ?

Handheld ProjectorHandheld Projector

iLamps 2002 RFIG Lamps 2003-04

Pocket Projector 2004-05

Mitsubishi Pocket ProjectorMitsubishi Pocket Projector

Prototype TagPrototype TagRF tag +

photosensor

READER

Micro Controller

RF Data

Memory

Conventional RFID

Computer

READER

Micro Controller

RF Data

Memory

Photosensor

Computer

Conventional RF tag

Photo-sensing RF tag

READER

Projector Micro

Controller RF Data

Memory

Photosensor

Computer

Find tag location using

handheld Projector

Photosensing

Wireless Tags

Many geometric ops

R F I R F I

R F I DR F I D

Interactive stabilized projection

(Radio Frequency Id & Geometry)(Radio Frequency Id & Geometry)GG

Siggraph 2004

AR with Photosensing RFID AR with Photosensing RFID and Handheld Projectorand Handheld Projector

Visual feedback of 2D positionVisual feedback of 2D position

a. Receive via RF {(x1,y1), (x2,y2), …} pixels

b. Illuminate those positions

RFID(Radio Frequency Identification)

RFIG(Radio Frequency Id and Geometry)

CurvedPlanar Non-planar

SingleProjector

MultipleProjectors

Projector

jUser : T

Pocket-ProjObjects

Projectors and SurfacesProjectors and Surfaces

2000 2002

20031999

Pieces of the PuzzlePieces of the PuzzleActuated SurfacesActuated Surfaces

Actuated Workbench [Pangaro, Maynes-Aminzade, Ishii UIST 2002]

• Why– Special Effects in the Real World– Photorealism for everyday

• Tools for Augmentation– Projectors, Sensors, RFID

• Illusion– Beyond Visual Senses– Beyond Projectors– Beyond Gouraud

Believable Co-existenceBelievable Co-existence

• Other Senses – Haptic, Smell, Taste, Locomotion

• Beyond Gouraud Effects– Complex BRDF

• Display Choices– Polymers, e-Ink

Augmentation of “five senses”Augmentation of “five senses”

• Visual Displays• Auditory Displays• Haptic Displays• Locomotion Interfaces• Olfactory Displays• Taste Displays ?• Sixth Sense ..

Extendingsensorymodalities

Sense of BalanceSense of Balance

Galvanic Vestibular Stimulation

Masahiko Inami (UEC)

Electric Cutaneous DisplayElectric Cutaneous Display

Electrode

Dr. H. Kajimoto, U TokyoMasahiko Inami (UEC)

Smart TouchSmart Touch

Masahiko Inami (UEC)

Scent DisplayScent Display

Tracking nose direction

Aiming atthe nose

Air cannon

Deliveringscented air byvortex rings Camera

Platform

Yasuyuki Yanagida, ATR

Spatio-Temporal Controlled Delivery of Scent

Air CannonAir Cannon

Generating a doughnut-shaped vortex ring

Aperture

Air CannonAir Cannon

Prototype SystemPrototype System

• Nose tracking• Multiple scents• Scent switching• Improved delivery range

Yasuyuki Yanagida, ATR

Beyond White ObjectsBeyond White Objects

Grossberg, Nayar et alGrossberg, Nayar et al

The Virtual ShowcaseThe Virtual Showcase

Oliver BimberOliver BimberTracking and MirrorsTracking and Mirrors

DisplaysDisplays ContendersContenders

OrganicLED

LightEmittingPolymers

HolodustHolodustPerlin et alPerlin et al

Fake WorldFake World

•Participating Media –simulating or in presence of smoke, fog

•Complex Geometry–Spaghetti

•Complex Reflectance –Arbitrary appearance on arbitrary BRDF surfaces

•Shape and Motion –Transformation and Animation of real surfaces

•NPR, Distortions, Perceptual factors–Great thesis topics ..

Beyond Gouraud Shading of White ObjectsBeyond Gouraud Shading of White Objects

RecapRecap• Special Effects in Real World

– Photorealism yet to impact daily life– Poor Man’s Palace

• Spatial Augmented Reality– Solution for fusion– Geometry, Photometry, Id– Sense, Control, Compensate– Needs hardware components: Projectors, RFID, Sensors

• Open Problems– Natural phenomenon, complex BRDF, other displays– All senses: haptic, olfactory, preprioception

• Next Challenge: Photorealism

Spatial Augmented RealitySpatial Augmented Reality

• Graphics– Photorealism yet to impact daily life– Special effects in real world

• Spatial Augmentation– Solution for fusion– Needs hardware components– Projectors, RFID, Sensors

• Next Challenge: Photorealism

raskar.com

MoreMore

• Networked fake world• Poor man’s palace• Intelligence moving from PC to Proj to

Surf• Electronic items eventually become

cheaper than those scavenged from nature

Special EffectsSpecial Effects

– Photorealism • Limited to display screens• Does not affect daily life in real

time– Entertainment and Visualization– Unlike other fields of CS

• Provoke not offend• The primary advantage AR has over VR is

that AR uses the parts of the body that aren't just the eyes and ears: proprioception, vestibular perception, and othe cues that old-fashioned VR just can't handle. The disjunct between vestibular information and visual information that you get in VR is the source of the motion-sickness that often accompanies it.

• VR, like a lot of early 'cyberspace' mythology, was built on an unrealistic rejection of the body, and a fantasy of "pure mind."

• Bud Fake Cup of Coffee 3000http://www.visit4info.com/details.cfm?

adid=21038

ApplicationsApplications– Static setup

• Color everyday objects• Engineering drilling locations• Stage backdrops, seasons or aging

– Laser shows on buildings

– Simulate motion• Car shows, wheels, dirt marks• Virtual moving lights

ApplicationsApplicationsIndoors, under controlled lightingIndoors, under controlled lighting

– Architectural models• Augment walk-around scaled model of buildings• Project and ‘paint’ surface colors, textures• Lighting, sunlight, seasons• Internal structure, pipes, wiring

– Assembly line• Instructional text, images and procedures

– Entertainment• Live shows, exhibits, demonstrations

Tools for AugmentationTools for Augmentation

• Environment– Sense, Compensate and Control– Id, Geometry, Photometry(reflectance/Illumination), Id

• Display

• Rendering

Raskar Graphics Interface May05

Technology

Raskar Paris Nov08

Raskar Npar04final

Raskar 6Sight Keynote Talk Nov09

Fidic conditions of contract may05

Raskar Emtech2010 Mar Final

Amartya sen interview translated by_Tushar Raskar

Introduction to Camera Challenges - Ramesh Raskar

110016 Bec Vantage Rep May05

Computational Illumination - Researchgroups.csail.mit.edu/graphics/classes/CompPhoto06/... · Ramesh Raskar, Computational Illumination Image-Based Actual Re-lighting Film the background

NESDIS Data Center Users Workshop (May05)

Building Regs 1997_TGD-B_Fire Safety_[May05 Reprint]

Raskar TEDMED 2013

Raskar Graphics Interface May05 Web

Raskar, Camera Culture, MIT Media Lab Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab

May05-36: Boone Cemetery Management Software Boone Cemetery Management Software May05-36 Greg Thede, Director, Boone Parks Department Dr. Kothari Joseph

Ramesh Raskar Associate Prof, Media Lab, MIT Camera ...web.media.mit.edu/~raskar/Slides/02LectureFeb13.pdf · Ramesh Raskar Associate Prof, Media Lab, MIT ... • Beyond Visible Spectrum

School Plan Reports May05

Raskar COSI invited talk Oct 2009

Raskar ideahexagonapr2010

Raskar, Camera Culture, MIT Media Lab Camera Culture Ramesh Raskar MIT Media Lab Ramesh Raskar Associate Professor