A Theory for Photometric Self-A Theory for Photometric Self-Calibration of Multiple Overlapping Calibration of Multiple Overlapping Projectors and CamerasProjectors and Cameras

Peng SongPeng SongTat-Jen ChamTat-Jen Cham

Centre for Multimedia & Network Technology (CeMNet)Centre for Multimedia & Network Technology (CeMNet)School of Computer EngineeringSchool of Computer EngineeringNanyang Technological UniversityNanyang Technological University

Singapore-MIT AllianceSingapore-MIT Alliance

MotivationMotivation Projector camera systemsProjector camera systems

Oblique displaysOblique displaysUncontrolled environmentUncontrolled environmentOff-the-shelf commodity equipmentOff-the-shelf commodity equipmentQuick and easy setupQuick and easy setup

Photometric calibrationPhotometric calibration

Camera observation

Calibrated model

Projector input

Camera image

Predicted image

Related WorkRelated Work

Modeling of combined projector-to-camera traModeling of combined projector-to-camera transfer functionnsfer function Surati [99]Surati [99] Jaynes et al. [03]Jaynes et al. [03]

Photometers/spectroradiometersPhotometers/spectroradiometers Majumder et al. [00, 02]Majumder et al. [00, 02] Yang et al. [01]Yang et al. [01]

High Dynamic Range imaging approachesHigh Dynamic Range imaging approaches Debevec et al. [97]Debevec et al. [97] Mitsunaga et al. [99]Mitsunaga et al. [99] Raij et al. [04]Raij et al. [04]

Intensity Transfer Function S1

Basic FrameworkBasic Framework

))((1

AISCZN

jjj

P2 …P1 Pn

C

Color FilterIntensity Transfer Function S2

Intensity Transfer Function C

Color Filter

Projector pixel value I2

Image Pixel Intensity Z

Ambient Lighting

Projector2

Camera P’

Multiple overlapping Multiple overlapping projectors and a projectors and a cameracamera

Color FilterProjector pixel value I1

Projector1

Isointensity CurvesIsointensity Curves Two overlapping projectorsTwo overlapping projectors

Different projector input combinations affecting a single cDifferent projector input combinations affecting a single camera pixelamera pixel

Isointensity curvesIsointensity curves Different combinations with same camera observed valueDifferent combinations with same camera observed value Same radiance into the cameraSame radiance into the camera

The Staircase MethodThe Staircase Method

Any transition between two isointensity curAny transition between two isointensity curves leads to the same radiance changeves leads to the same radiance change

Consider only horizontal and vertical transiConsider only horizontal and vertical transitionstions

1I

2I

o1I

1S

o

b

a

a

b

Recursive Bisection MethodRecursive Bisection Method

Find a new isointensity curve between two prFind a new isointensity curve between two previous isointensity curvesevious isointensity curves Radiance change from previous curves to the new cRadiance change from previous curves to the new c

urve = ½ radiance change between previous curvesurve = ½ radiance change between previous curves

1I

2I

o1I

1S

o

Gridline Optimization MethodGridline Optimization Method Define vertical and horizontal gridlinesDefine vertical and horizontal gridlines

Want equal radiance change between adjacent Want equal radiance change between adjacent gridlinesgridlines

Diagonal gridline intersections should have Diagonal gridline intersections should have the same camera observed intensitythe same camera observed intensity Shift gridlines to minimize varianceShift gridlines to minimize variance

1I

2I

o1I

1S

o

Further Calibration StagesFurther Calibration Stages

Derive additional methods forDerive additional methods for Computing radiances contribution from Computing radiances contribution from

projector black offsets & indoor lightsprojector black offsets & indoor lights Camera photometric calibrationCamera photometric calibration

Extension of calibration of a pixel to all Extension of calibration of a pixel to all projector pixels across the displayprojector pixels across the display

),( 00 yx

),( yx

N

jjjjyx yxAyxISyxM

1, ),()),((),(

Experiments (I)Experiments (I)

Estimating Intensity Transfer FunctionsEstimating Intensity Transfer Functions

Experiments (I)Experiments (I) Combined projector-to-camera intensity transfer functionsCombined projector-to-camera intensity transfer functions

Computed Data

Observed Data

Observed Vs. Computed Data Difference

Mean Absolute Difference = 0.21027 intensity levels

Experiments (II)Experiments (II)

Prediction of camera output based on Prediction of camera output based on known projector inputknown projector input

Observation Prediction

PredictionObservation

Observation

Observation

Prediction

Prediction

Experiments (III)Experiments (III)

Determining projector input to create Determining projector input to create desired camera observed imagedesired camera observed image

Uncompensated Image

Compensated image

Desired Image

Uncompensated Image

Desired Image

Compensated Image

Conclusion and Future WorkConclusion and Future Work A principled framework for photometric A principled framework for photometric

self-calibration of overlapping projectors self-calibration of overlapping projectors and camerasand cameras three methods proposedthree methods proposed

Staircase methodStaircase method Recursive bisection methodRecursive bisection method Gridline optimization methodGridline optimization method

No requirement for expensive equipmentNo requirement for expensive equipment Only needs a cheap camera with minimal Only needs a cheap camera with minimal

controlscontrols Extension of grayscale photometric Extension of grayscale photometric

calibration to colorimetric calibration (in calibration to colorimetric calibration (in progress)progress)

Q & AQ & A

Thank youThank you

Time Analysis of Three MethodsTime Analysis of Three Methods

The staircase methodThe staircase methodComplexityComplexityWorst case time 10-15 minsWorst case time 10-15 mins

Recursive bisection methodRecursive bisection methodComplexityComplexityWorst case time 20-30 minsWorst case time 20-30 mins

Gridline optimization methodGridline optimization methodData collection time 1-2 hours Data collection time 1-2 hours

)log( 2 NNO

)log( 22 NNO