Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Ramesh Raskar, Paul Beardsley, Jeroen van

Mitsubishi Electric Research LabsRaskar, Beardsley, vanBaar, Wang, Dietz,

Lee, Leigh, WillwacherR F I G Lamps

Ramesh Raskar, Paul Beardsley, Jeroen van Baar, Yao Wang,

Paul Dietz, Johnny Lee, Darren Leigh, Thomas Willwacher

Mitsubishi Electric Research Labs (MERL), Cambridge, MA

R F I R F I GG Lamps Lamps : : Interacting with a Self-describing World Interacting with a Self-describing World

via Photosensing Wireless Tags and via Photosensing Wireless Tags and ProjectorsProjectors



Radio Frequency Identification Tags Radio Frequency Identification Tags (RFID)(RFID)

microchip

Antenna

No batteries,

Small size,

Cost few cents



WarehousingWarehousing

RoutingRouting

Library Library

Baggage

handling

Baggage

handling

CurrencyCurrency

Livestock trackingLivestock tracking



Micro Controller

Memory Computer

READER

Micro Controller

Memory

Conventional Passive RFID



Tagged Books in a LibraryTagged Books in a Library

Id List of books in RF range

No Precise Location Data Are books in sorted order ?Which book is upside down ?



Where are boxes with Where are boxes with Products close to Expiry Date ?Products close to Expiry Date ?



READER

Micro Controller

RF Data

Memory

Conventional RFID

Computer

READER

Micro Controller

RF Data

Light

Memory

Photosensor

Computer

Conventional RF tag

Photo-sensing RF tag



READER

Projector Micro

Controller RF Data

Light

Memory

Photosensor

Computer

Photosensor ?

Compatible with RFID size and power needs

Projector ?

Directional transfer,AR with Image overlay



b. Tags respond with date and precise (x,y) pixel location, Projector beams ‘Ok’ at that location

b. Tags respond with date and precise (x,y) pixel location, Projector beams ‘Ok’ at that location

a. Photosensing RFID tagsare queried via RF

a. Photosensing RFID tagsare queried via RF

c. Multiple users can simultaneously work from a distance without RF collision

c. Multiple users can simultaneously work from a distance without RF collision





RFID

(Radio Frequency Identification)

RFIG

(Radio Frequency Id and Geometry)



Prototype TagPrototype Tag

RF tag + photosensor

RF tag + photosensor



OutlineOutline

• Photosensing RF tags– Location sensing

• Geometric Operations– Multi-view analysis– Deformation

• Interactive Projection– Mouse-like interaction– Image stabilization



OutlineOutline




• Applications



Pattern

MSB

Pattern

MSB

Pattern

MSB-1

Pattern

MSB-1

Pattern

LSB

Pattern

LSB

Projected Sequential Frames

•Handheld Projector beams binary coded stripes

•Tags decode temporal code



Pattern

MSB

Pattern

MSB

Pattern

MSB-1

Pattern

MSB-1

Pattern

LSB

Pattern

LSB






Pattern

MSB

Pattern

MSB

Pattern

MSB-1

Pattern

MSB-1

Pattern

LSB

Pattern

LSB






Pattern

MSB

Pattern

MSB

Pattern

MSB-1

Pattern

MSB-1

Pattern

LSB

Pattern

LSB






Pattern

MSB

Pattern

MSB

Pattern

MSB-1

Pattern

MSB-1

Pattern

LSB

Pattern

LSB






Pattern

MSB

Pattern

MSB

Pattern

MSB-1

Pattern

MSB-1

Pattern

LSB

Pattern

LSB

For each tag

a. From light sequence, decode x and y coordinate

b. Transmit back to RF reader (Id, x, y)

For each tag

a. From light sequence, decode x and y coordinate

b. Transmit back to RF reader (Id, x, y)

00 11 11 00 00 X=12

X=12



Visual feedback of 2D Visual feedback of 2D positionposition

a. Receive via RF {(x1,y1), (x2,y2), …} pixels

b. Illuminate those positions

a. Receive via RF {(x1,y1), (x2,y2), …} pixels

b. Illuminate those positions



LimitationsLimitations

• Line of sight~Surface patches~Multiple tags per object

• Ambient light~Modulated infrared

• Surface reflectance and shadows



OutlineOutline




• Applications



Find 3D coordinatesFind 3D coordinates

a. Observe structured patterns via Camera

b. Triangulate cam/proj view

a. Observe structured patterns via Camera

b. Triangulate cam/proj view



3D from 2 Projector Views3D from 2 Projector Views(Structure from Motion)(Structure from Motion)

• Two+ unknown projector views• Correspondence is trivial• Applications

– Detect 3D deformations– Trajectory grouping



Change Detection Change Detection without fixed camera, in any lighting conditionwithout fixed camera, in any lighting condition

Compare with new coordinates from a different

view

Record coordinates of tags from one view

Before After



Texture AdaptationTexture Adaptation



OutlineOutline



• Interactive Projection– Desktop-like interaction– Image stabilization

• Applications



Desktop-like InteractionDesktop-like Interaction

Selecting Selecting tagstags



Support for handheld Support for handheld projectionprojection



Mouse SimulationMouse Simulation

• Cursor follows handheld projector motion

• Pre-warped image remains stable



Image Quasi-StabilizationImage Quasi-Stabilization

Eliminate hand jitter using inertial Eliminate hand jitter using inertial sensors+camerasensors+camera



Absolute StabilizationAbsolute Stabilization

Image stays registered with world Image stays registered with world featuresfeatures



Image StabilizationImage Stabilization



Interactive ProjectionInteractive Projection

(Also in Emerging Technologies Booth)(Also in Emerging Technologies Booth)



Adaptive ProjectionAdaptive Projection

‘‘Paste’ : Geometric and Photometric Paste’ : Geometric and Photometric compensationcompensation



Prototype Handheld Prototype Handheld ProjectorProjector



Related WorkRelated Work• Smart objects

– Smart-its– FindIT Flashlight (Ma and Paradiso 2002)

• Location sensing – Multiple readers, large antennas– Olivetti Active Badge, Xerox PARCtab (Want 1995)

• Interaction and Display– Augmented Reality– Projector-based AR, Shaderlamps

(Raskar 1998, Underkoffler 1999, Pinhanez 2001)

– Handheld projector (Raskar 2003)



ApplicationsApplications

• Single Tags– Authoring for AR, Store augmentation: display per

object

• Multiple tags on objects– 3D Pose– Packing and placement strategy– Robot navigation

• Interaction between tagged objects– History of geometric interaction– Orientation mismatch

• Distributed tags– Deformation– Interpolated values for temperature sensors



AcknowledgementsAcknowledgements

• MERL– Cliff Forlines– Joe Marks, – Dick Waters, Kent Wittenburg– Vlad Branzoi– Rebecca Xiong, Debbi van Baar

• Mitsubishi Electric, Japan– Mamuro Kato, – Keiichi Shiotani



Interacting with a Self-Describing WorldInteracting with a Self-Describing World

• Hybrid optical and RF communication– Photosensing Wireless Tags + Projector illumination

• Geometric Analysis– Location, Selection,

Augmentation At a distance

• Interactive Projection– Desktop like interaction– Stabilized images

• RFIG Applications beyond logistics– Identity, Geometry, History, Annotation

www.MERL.com



Find tag location using

handheld Projector

Find tag location using

handheld Projector

Photosensing

Wireless Tags

Photosensing

Wireless Tags

Many geometric ops Many geometric ops

R F I R F I R F I DR F I D

Interactive stabilized projection

Interactive stabilized projection

(Radio Frequency Id & (Radio Frequency Id & GeometryGeometry))GG



(End of Presentation)(End of Presentation)



Towards Passive RFIDTowards Passive RFID• Photosensor

– Most compatible with passive batteryless RFID– Power: near zero power for sensing– Size: Each photocell is tens of micrometer (smaller than RFID

which are about 500 micrometer)– Very low cost, can be built with silicon of RFID microchip– RFID+Photosensor can be size of grain of rice– RFIG tag is ‘visible’ to projector but not to humans– Current prototype uses battery because passive tags are difficult

to program

• Other possibilities– Adding a light emitting diode (LED)

• Requires on board power• Size has to be considerably large• Size and power requirements are high



Future DirectionsFuture Directions

• Sophisticated tags• Light modulation

• Handheld projector



Comparison with Bar-codesComparison with Bar-codes

• RFID– Barcodes take up more space– Long bit sequences– Multiple tags can be simultaneously read– Read distance is high

• RFIG– Multiple barcodes per object is cumbersome– Difficult to attach sensor (e.g. temperature)



Locate Tags Precisely using a Locate Tags Precisely using a handheld scanner/projectorhandheld scanner/projector

•Handheld device

•Eliminates multiple RF readers for triangulation or large antennas

•Projector indicates location of tag

Illuminate tags with binary coded stripes with projector



Photosensor added to RFID tags

Hybrid communication via RF and via light

Solve 3 main issues with conventional RFIDs

1. Locate tags precisely

2. Select a subset of tags for operation

3. Long distance operation without collision

Applications beyond inventory management

1. Warehouse: locate objects for specific queries

2. Training and maintenance: interactive augmented reality

3. Surveillance: keep trail of object locations



2. Select subset of tags for operation2. Select subset of tags for operation

Advantages

•Only tags which are illuminated respond to queries

•No RF collision between multiple tags

•Projector also use as a display to indicate result of operation

Technique

•Illuminate subset of tags with flashlight or projector

3. Long distance operation without RF collision3. Long distance operation without RF collision



Interactive ProjectionInteractive Projection

• Desktop like interaction– Cursor follows use motion– Center pixel is the pointer– Rest of the image is stabilized– Selection

• Image stabilization– Absolute: camera and inertial sensor– Relative: world markers

• Copy-paste– Camera captures texture during ‘copy’– Projection after geometric/photometric adaptation

Documents

Mitsubishi Electric Research Labs Raskar, Beardsley, vanBaar, Wang, Dietz, Lee, Leigh, Willwacher R F I G Lamps Ramesh Raskar, Paul Beardsley, Jeroen van