Image Based Rendering And Modeling Techniques And Their Applications Jiao-ying Shi State Key...

Image Based Rendering And Modeling Image Based Rendering And Modeling Techniques And Their ApplicationsTechniques And Their Applications

Jiao-ying ShiJiao-ying ShiState Key laboratory of Computer Aided Design and State Key laboratory of Computer Aided Design and

GraphicsGraphics

Zhejiang University, Hangzhou, ChinaZhejiang University, Hangzhou, China

jyshi@cad.zju.edu.cnjyshi@cad.zju.edu.cn

http://cad.zju.edu.cn/home/jyshihttp://cad.zju.edu.cn/home/jyshi

Image Based Rendering Image Based Rendering (IBR)(IBR)

PART IPART I

Traditional Computer GraphicsTraditional Computer Graphics

• Use geometry and lighting model to simulate the imaging process and generate realistic scene

– No guarantees for the rightness of the models– A lot of computation time needed

Use of Images in Computer Use of Images in Computer GraphicsGraphics

• Texture mapping

• Environment map

• How about more images?

Image Based RenderingImage Based Rendering

IBR: To Synthesize a new scene with novel view point based on given images

ComputerVision

ComputerGraphics

Geometry

Image basedrendering

Images Images

A Framework of Image Based A Framework of Image Based RenderingRendering

Real Scene

Sampling System

Data Storage System

Data representation System

Rendering System

Synthesized view

The Key Part of IBRThe Key Part of IBR

The data representation system is the key part of IBR, It determines the other three subsystems.

-A taxonomy based on the data representation system

A Taxonomy of IBRA Taxonomy of IBR

• The Geometry based data representation• The Image based data representation• The plenoptic function based data

representation

The Geometry Based Data The Geometry Based Data RepresentationRepresentation

• Geometry elements used as data representation in IBR:– polyhedra(Debevec, et. al 1996)– layers (Baker, Szeliski and Anandan 1998)– points(Shade et al. 1998)

• Similar to traditional Computer Graphics, except the geometry model comes from images

Image Based Data Image Based Data RepresentationRepresentation

• data are treated as a series of images with correspondence relations

• “optical flow” “morphing map”

• forward/ reverse mapping;morphing

Examples: View interpolation (Chen and William,1993)

View Morphing(Seitz and Dyer 1998)

Plenoptic Function Based Data Plenoptic Function Based Data RepresentationRepresentation

• Plenoptic function (Adelson and Bergen,1991)

),,,,,,( tVVVPlenoptic zyx

Representative IBR Methods Based on Representative IBR Methods Based on Plenoptic FunctionsPlenoptic Functions

• Plenoptic Modeling: 5D

• Light field/Lumigraph: 4D

• Concentric Mosaics : 3D

• Panorama: 2D

[L. McMillan 95]

[M. Levoy 96, S. J. Gortler 96]

[H. Y. Shum 99]

[S. E. Chen 95,R. Szeliski 97]

IBR

The Geometry based data

representation

The Geometry based data

representation

The Plenoptic function based

data representation

IBR data are composed of

geometry elements

IBR data are composed of a

set of images with

correspondence relations

IBR data are composed of

a set of light rays

polyhedra

layers

points

View interpolation[CW93]

View morphing[SD96]]

Transfer mode [LF94]

Plenoptic

modeling[MB95]

Lightgield [LH96]/

Lumigraph[GGSC96]

Concentric

Mosaics[SH99]

Panorama[Chen95],[SS97]

MCOP images[RB98],

LDI[SGHS98]

Depth based [BSA98]、Motion based [LS97]、TIP [HAA97] etc.

Hybrid approach of

geometry and image

[DTM96]

5D plenoptic

function

4D plenoptic

function

3D plenoptic

function

2D plenoptic

function

ConclusionConclusion

• The progress of IBR technique is also the progress of new data representation method.

We treat an image:– as texture in geometry texture mapping– as images with correspondence relation view interpolation /morphing– as light beams light field– as slit image concentric mosaics...

Demo of IBRDemo of IBR(1) Tour in Dunhuang Art Cave(1) Tour in Dunhuang Art Cave

Demo of IBRDemo of IBR(2) Tour in Lingyin Temple in Hangzhou(2) Tour in Lingyin Temple in Hangzhou

Image Based Modeling Image Based Modeling (IBM)(IBM)

PART IIPART II

The Common Methods Used for Modeling objects

• Using Geometry Modeling Software Packages, such as

3D MAX ， Mayer ， SoftImage and AutoCAD etc. to create wireframe models, surface models or volume models.

• Using 3D Laser Scanners• Using Image-based Modeling techniques to create

geometry models or appearance models of objects.

A Taxonomy of IBM

• IBM methods using active cues

The active cues are refered to the artificially generated silhouette, which are projected onto the surface of the modeled objects.

• IBM methods using passive cues

The passive cues are refered to the implicit characteristics of the modeled objects, such as geometry features and textures of the objects.

IBM Methods Using Active cuesIBM Methods Using Active cues

IBM Methods Using Active cuesIBM Methods Using Active cues

IBM Methods Using Passive CuesIBM Methods Using Passive Cues

• Based on known geometry

• Based on visual hull

• Based on light field

• Based on stereo vision

IBM Methods Using Passive CuesIBM Methods Using Passive Cues

• Based on known geometry[Debevec96]

IBM Methods Using Passive CuesIBM Methods Using Passive Cues• Based on visual hull [Wojciech Matusik 2001]

IBM Methods Using Passive CuesIBM Methods Using Passive Cues

• Based on Light Field[Marc Pollefeys et. al.]

IBM Methods Using Passive CuesIBM Methods Using Passive Cues

• Based on stereo pairs: the goal is to automatically extract a realistic 3D model by freely moving a camera around an object.

Neither the camera motion nor the camera settings have to be known.

The obtained 3D model is a scaled version of the original object.

The surface appearance is obtained from the image sequence as well.

IBM Methods Using Passive CuesIBM Methods Using Passive Cues

How can we get 3D information (depth) from a 2D image?

It is impossible to get depth information of the object from a

2D image.

C

P

IBM Methods Using Passive CuesIBM Methods Using Passive CuesIt is possible to recover the depth information

of the object by using two or more images throughtriangulation.

C1

P

C2

p1

p2

Reconstruction of 3D point through triangulation

IBM Methods Using Passive CuesIBM Methods Using Passive Cues

But the following information are needed to know

for this purpose:• Corresponding image points• Relative pose of the camera for different views(so called

camera extrinsic parameters, i.e. the position and orientation of the camera)

• Relation between the image points and the corresponding line of sight. It is defined by the camera model, which usually is a pinhole model and cameras intrinsic parameters and extrinsic parameters.

IBM Methods Using Passive CuesIBM Methods Using Passive Cues

Flowchart of the IBM

technique based on

stereo pairs

Feature PointsExtraction & Match

Camera Calibration: f, R, T

Calculation of dencedepth map

Surface reconstruction

Rectification

Reconstruction of Architectural Reconstruction of Architectural Models Based on Image SequenceModels Based on Image Sequence

Acquesition of imagesequences

O X

Y

c

x1y1

t1t2

t

o1

xy

zc'y'

x'z'

A

BUV

u1 v1a1 b1

cameracalibration

modeling basedon single image

Modelmerging of

multiplescenes

finalmodel

modelmerging

Camera Calibration Using Corner Structure Camera Calibration Using Corner Structure and Parallel Structure in Real Sceneand Parallel Structure in Real Scene

• Consisted of at least 4 line segments.

• 3 of them are perpendicular each other.

• The forth segment is parallel to one of previous 3 segments.

(a) (b)

ox

y

z

l 4

o x

y

l 1l 2

l 3

l 1l 2

l 3

l 4

(a) (b)

ox

y

z

l 4

o x

y

l 1l 2

l 3

l 1l 2

l 3

l 4

l4

l1

l2

(a)

l3

l4

l1

l2

(b)

l3

l4

l1

l2

(a)

l3

l4

l1

l2

(b)

l3

• Corner Structure

• Parallel Structure• Consisted of at least 4 line

segments.

• 2 of them are perpendicular each other.

• Other 2 segments are parallel to previous 2 segments separately.

Calculating focal length based on single imageCalculating focal length based on single image

is the directional vector of segment caTaaca fyxd ],,[

is the normal vector of the projective planecbcaab ddn

is the directional vector of segment ABozabAB nnd

because

0 OYOX ddWe can get (*)

This is a quartic equation relative to f , from which we can get one plus real solution of the focal length.

2f

Projection of the corner structure

Then the directional vectors of OX and OY can be written as an expression relative to the focal length f

OYOX dd x

y

z

O

Y

Z

L

o

x

y

z

A

B

abl

c

X

Calculating rotation matrix and translation Calculating rotation matrix and translation vector based on single image related to vector based on single image related to SCFSCF

)( dOX tVrAxs

From the projection equation of pinhole camera model, we can get:

Rotation matrix relative to SCF is:

Z

Z

OY

OY

OX

OX

d

d

d

d

d

dr

fyxt ood

The direction of the translation vector relative to CCF is

dtt *

xy

z

O

Y

Z

L

o

x

y

z

A

B

abl

c

X

SCF

CCF

Projection of the corner structure

SCF: structure coordinate frameCCF: camera coordinate frame

Translation vector can be deduced:

Optimizing the camera’s parameters: Optimizing the camera’s parameters: f, r and tf, r and t

We choose the distance DIST between the original image point and the reprojected image point as the measurement function for error.

n

i

TT

PPP

ptPrpptPrp

ZYXtztytxtztytxrrrrffFDIST

1

22

33113311

)2))2(2('1))1(1((

),,,2,2,2,1,1,1,22,11,',(

Once the initial values of camera parameters were derived, we can optimize these parameters by using other line segments, which are parallel to the segments of the corner structure in two images.

Procedures for optimizationProcedures for optimization

Rotation matrices Translation vector

Irr

IrrT

T

2*2

1*1

022

011

22

22

tlent

tlent

Structure condition

0)( PQVPQ

),,,2,2,2,1,1,1,22,11,',( 33113311 PPP ZYXtztytxtztytxrrrrffFMin

Objective function of optimization:

To ensure the orthogonality

of rotation matrices

To ensure the length of the two transla-

tion vectors will not be changed in

optimization.

tlen1 and tlen2 are the modules of the

two translation vectors, which can be

calculated from the initial solutions.

To ensure directions of the segments will not be changed in optimization

Constrained conditions:

Experimental ResultsExperimental Results

Two pictures of kiosk at the campus of Zhejiang University taken by hand-hold camera

Camera positions and top-down views of feature points

• Error Statistics: Choose the distance between feature points to its epipolar line as the measurement of error. In the ideal situation this distance should be equal to zero.

n

iii ppDist

nErr

1

)ˆ,(1

Image Err before optimization (pixels)

Err after optimization (pixels)

Left 7.96 1.09

Right 9.08 1.25

Metric(Pixel)

Architecture Reconstruction Based on Architecture Reconstruction Based on Single ImageSingle Image

Reconstruction of planeReconstruction of plane

• Assign base point O and determine its coordinates• Assign base plane which contains the base point O as one of its

corners. Determine the normal vector of the base plane.• Determine unknown neighbor plane of the base plane: choose one

point on the common edge of the neighbor planes as the base point of the unknown plane.

• Determine normal vector of the unknown plane.

Decomposition of sweeping surface into Decomposition of sweeping surface into plane patchesplane patches

A1

v

A2

B1

B2

C1

C2

D1

D2

E1

E2

P1

P2

Wireframe modelWireframe model

Texture mapping modelTexture mapping model

Texture mapping modelTexture mapping model

Strategy of model merging for Strategy of model merging for multiple scenesmultiple scenes

w1

v2

w2

Z

OX

Y

camera1

x1y1

z1c1

camera2

x2

y2

z2

c2

camera3

x3

y3

z3

c3o1

u1

v1

u2

o2

o1u1v1w1 o2u2v2w2

u2

v2

o2

w2o1u1

v1

w1

camera3

camera2

x2

y2

z2c2

w2o1

u1

v1

w1u2

v2

o2

camera2

x2

y2

z2c2

x3

y3z3

c3u2

v2

o2w2o1

u1v1

w1

camera1x1y1

z1c1

Scene structure and 3 camera positions Model merging diagram for multiple scenes

Example of model mergingExample of model merging

Image sequence

Reconstructed model for each image

Main challenges for model merging procedureMain challenges for model merging procedure

• How to transform all the models in different local coordinates to one world coordinate frame?

• Model integration: scaling, vertex match?

• Deletion of overlapped plane?

Establishing listof commoncoordiante

frames

Coordinatetransformation

Mergingvertices

Mergingplanes

Finalmodel

Reconstructed Tea Box with Texture MappedReconstructed Tea Box with Texture Mapped

Experiment 1: image sequenceExperiment 1: image sequence

Experiment 1: reconstructed modelExperiment 1: reconstructed model

Experiment 2: image sequenceExperiment 2: image sequence

Experiment 2: reconstructed modelExperiment 2: reconstructed model

Thanks a lotThanks a lot