A Keystone-free Hand-held Mobile Projection System Li Zhaorong And KH Wong Reference: Zhaorong Li,...

A Keystone-free Hand-held Mobile Projection System

Li ZhaorongAnd

KH WongReference: Zhaorong Li, Kin-Hong Wong, Yibo Gong, and Ming-Yuen Chang, “An Effective Method for Movable Projector Keystone Correction”, IEEE Transactions on Multimedia, VOL. 13, NO. 1, Feb 2011.

keystone correction 1

Outline

• Introduction• Methodology

– Pro-cam pair calibration– Projection region detection and tracking– Automatic keystone correction

• Experimental results• Conclusion

keystone correction 2

Introduction

keystone correction 3

Aim of this work

• A mobile projector keystone correction method– project keystone free content on a general flat

surface– without adding markings or boundaries on the

surface

keystone correction 4

Aim of this work

• Configuration

keystone correction 5

Aim of this work

• Desired Results

Keystoned projection Corrected projection

keystone correction 6

Motivation

• Mobile projector becomes popular– Flexible projector-screen position– More freedom of display control such as viewing angle,

distance etc– General flat surface can be used as screen– Physical size and cost shrinks quickly– Emergence of mobile devices with embedded projector– Others……

keystone correction 7

Motivation

• Limitation: keystone distortion !– When projecting onto a screen at oblique

position, the projection region becomes a trapezoid instead of a rectangle

– Traditional static projection system also face this problem

– Stringent for mobile projection system

keystone correction 8

Motivation

• Existing keystone correction method– All for static projector system, not suitable for

mobile projection, eg.• Su [1]: requires bounded screen• Li et al[2]: requires bounded screen • Raskar [3]: high computation, not real-time• One-time correction, not continuous

keystone correction 9

Motivation

• Special need for mobile keystone correction– Screen independence

• no specially designed or position-fixed screen should be required

– Continuous processing in real time• Continuous correction instead of one-time correction is

expected

keystone correction 10

Our approach

• Propose an effective method that can continuously correct the keystone distortion on a general markless screen

• Only additional device used is a webcam attached with the projector

keystone correction 11

Our approach

• We add a green frame to the projector screen and project it to the display screen

• Track the green frame use the camera and then automatically correct the keystone correction

keystone correction 12

Contributions

• An artfully designed particle filter tracker for tracking projection region

• An efficient and accurate recovery algorithm for recovering 3d projection region

• A continuous and markless mobile projector keystone correction system

keystone correction 13

Methodology

keystone correction 14

Overview

• Three major modules– Projector-camera pair calibration– Projection region detection and tracking– Automatic keystone correction

3D projectionregion recovery

Projector-camera pair calibration

G, K

Corrected projection

Keystoned projection

Pre-warped image3D rectangle

Keystone correctionProjection region tracking

Camera frame

Flow chart

keystone correction 15

Part 1: Projector-camera pair calibration

• Projector-camera relationship– Projection matrix from 3D camera coordinate system

to projector image plane

• Benefits– Independent from projector movement– No need to estimate explicit parameters– Easy to estimate

keystone correction 16

Projector-camera pair calibration

• Projective model

p c x GXpxcX

G

: 2D projector image pixel

: 3D point in camera frame

: 3x4 projection matrix

: scale factor

keystone correction 17

Projector-camera pair calibration

• Estimation method– Project cross to an ordinary cardboard with

known size– Collect a number of projector-camera

corresponding crosses– Calculate the 3D coordinate of the cross– Estimate G matrix using SVD

keystone correction 18

Part 2: Projection region detection and tracking

• Add a green frame to the projection region• Detect the frame in the initial frame• Track the frame in the subsequent frames

keystone correction 19

Detection• Based on the Canny edge map and Hough line segments• Find a quadrangle satisfying following criteria

– each side of the quadrangle longer than a threshold– opposite sides should have similar lengths– each angle within the range from 30 to 150 degree– the overlapping ratio of the line segments to the four sides of the

formed quadrangle bigger than a threshold– the quadrangle approximately located around the center of the

camera image

keystone correction 20

Tracking

• The relationship between projector screen and the projection region in camera

p c x Hx

Hpxcx

: the homography matrix

: the projector image pixel

: the camera image pixel

keystone correction 21

Tracking

• The homography matrixT

1[ ]d

tn

H J R K

JRtn

Kd

: intrinsic parameter matrix of projector

: rotation matrix of the projector w.r.t camera

: translation vector of the projector w.r.t camera

: normal of the screen

: distance of the screen from the camera: intrinsic parameter matrix of camera

H is dominated by n and d since

J,R,t,K are all fixed

keystone correction 22

Tracking

• Tracking state vector

• Particle filter tracking– State dynamic model– Observation model– Initialization

T[ , ]ds n

keystone correction 23

Tracking

• State dynamic model

1 1 1( | ) ( , )k k k kp Uniform s s s e s e

1,k ks s

e

: the state of k-1 and k frame

: the uncertainty of the movement of the projector

keystone correction 24

Tracking

• Observation model– Re-project each particle to the camera

T1[ ]

d

JtnH JR K

3 3 3 1[ , ] [ , ] JR Jt G G

c 1 p x H x

keystone correction 25

Tracking

• Observation model– Compute likelihood by comparing the re-projected

quadrangle with the edge map• Specifically, check how many edge points on each of four sides• The likelihood of each side is the percentage of the edge points

among all side points• The likelihood of the particle is the sum of likelihoods of four sides

keystone correction 26

Tracking

• Initialization– The detected quadrangle in the detection stage is

used to initialize the particle filter– First, recover the 3D position of the quadrangle

[using the method in Part 3]– Second, compute its normal and distance from the

camera

keystone correction 27

Part 3: Automatic keystone correction

• Three steps– Recover 3D projection region– Look for inscribed rectangle– Pre-warp projection image

keystone correction 28

Recover 3D projection region

• Solve following equation set for each corner of the projection region using SVD

c c x KXp c x GX

: the corner of the projection region in camera

: the 3D corner of the projection region

: the corner of the projector screen

cxpxcX

keystone correction 29

Recover 3D projection region

• Incorporate co-planarity constraint

2 2 2 2T T c T T c4 41 i 2 i 1 i 2 iT c T c T c T c

1 13 i 3 i 3 i 3 i

2c c c c c c1 4 1 2 1 3

i i i iu v

c c

i i

k X k X g X g X

k X k X g X g X

X X X X X X

000000000000000000000000000000000000000000

T T T1 2 3, ,k k kT T T1 2 3, ,g g g

K

G: three rows of

: three rows of

: weight

Use the SVD solution as the initial value

keystone correction 30

Look for inscribed rectangle

• Simple geometric intersection

keystone correction 31

Pre-warp projection image

• Re-project inscribed rectangle into the camera image

• Find homography between the projection and projector screen

• Pre-warp the display image using the homography

keystone correction 32

Pre-warp projection image

Pre-warped projection image Display result

keystone correction 33

Experimental results

keystone correction 34

Devices

• PC with 2.16GHz CPU, 1 GB memory• Optoma mobile projector 1280x1024• Logitech Quickcam Pro 4000 webcam 320x240

keystone correction 35

Projector camera pair calibration

• Error distribution

Back-projection error corresponding to 80% inliers is 4.2 pixels

keystone correction 36

Projection region tracking

• Mean and std tracking error w.r.t different noise levels on simulation data

keystone correction 37

Projection region tracking

• Mean and std tracking error on real data are 3.4 and 3.6 pixels

Trajectory of the real projector movement

keystone correction 38

Keystone correction

• Some real correction results

keystone correction 39

Keystone correction

• The error of keystone correction against different poses of the projector

keystone correction 40

Keystone correction

• Comparison of our keystone correction module with the static projector keystone correction

keystone correction 41

Speed

• 16 fps on our platform• The per-frame processing time is about 60 ms• The pre-warping step occupies most of the

time (about 36 ms)

keystone correction 42

Conclusion

• Proposed an effective mobile keystone correction method

• Mobility and Markinglessness are the most distinguishing features

• Limitation: need to project a green frame• Future work: Use invisible IR LED to eliminate the

green frame

keystone correction 43

Reference• [1] R. Sukthankar, R. Stockton, and M. Mullin, “Smarter presentations:

Exploiting homography in camera-projector systems,” in Intl. Conf. on Computer Vision, 2001.

• [2] B. Li and I. Sezan, “Automatic keystone correction for smart projectors with embedded camera,” in Intl. Conf. on Image Processing, 2004

• [3] R. Raskar and P. Beardsley, “A self-correcting projector,” in Intl. Conf. on Computer Vision and Pattern Recognition, 2001

• [4] Zhaorong Li, Kin-Hong Wong, Yibo Gong, and Ming-Yuen Chang, “An Effective Method for Movable Projector Keystone Correction”, IEEE Transactions on Multimedia, VOL. 13, NO. 1, Feb 2011.

keystone correction 44

Thank you

keystone correction 45