Image Based Rendering And Modeling Image Based Rendering And Modeling Techniques And Their ApplicationsTechniques And Their Applications
Jiao-ying ShiJiao-ying ShiState Key laboratory of Computer Aided Design and State Key laboratory of Computer Aided Design and
GraphicsGraphics
Zhejiang University, Hangzhou, ChinaZhejiang University, Hangzhou, China
[email protected]@cad.zju.edu.cn
http://cad.zju.edu.cn/home/jyshihttp://cad.zju.edu.cn/home/jyshi
Image Based Rendering Image Based Rendering (IBR)(IBR)
PART IPART I
Traditional Computer GraphicsTraditional Computer Graphics
• Use geometry and lighting model to simulate the imaging process and generate realistic scene
– No guarantees for the rightness of the models– A lot of computation time needed
Use of Images in Computer Use of Images in Computer GraphicsGraphics
• Texture mapping
• Environment map
• How about more images?
Image Based RenderingImage Based Rendering
IBR: To Synthesize a new scene with novel view point based on given images
ComputerVision
ComputerGraphics
Geometry
Image basedrendering
Images Images
A Framework of Image Based A Framework of Image Based RenderingRendering
Real Scene
Sampling System
Data Storage System
Data representation System
Rendering System
Synthesized view
The Key Part of IBRThe Key Part of IBR
The data representation system is the key part of IBR, It determines the other three subsystems.
-A taxonomy based on the data representation system
A Taxonomy of IBRA Taxonomy of IBR
• The Geometry based data representation• The Image based data representation• The plenoptic function based data
representation
The Geometry Based Data The Geometry Based Data RepresentationRepresentation
• Geometry elements used as data representation in IBR:– polyhedra(Debevec, et. al 1996)– layers (Baker, Szeliski and Anandan 1998)– points(Shade et al. 1998)
• Similar to traditional Computer Graphics, except the geometry model comes from images
Image Based Data Image Based Data RepresentationRepresentation
• data are treated as a series of images with correspondence relations
• “optical flow” “morphing map”
• forward/ reverse mapping;morphing
Examples: View interpolation (Chen and William,1993)
View Morphing(Seitz and Dyer 1998)
Plenoptic Function Based Data Plenoptic Function Based Data RepresentationRepresentation
• Plenoptic function (Adelson and Bergen,1991)
),,,,,,( tVVVPlenoptic zyx
Representative IBR Methods Based on Representative IBR Methods Based on Plenoptic FunctionsPlenoptic Functions
• Plenoptic Modeling: 5D
• Light field/Lumigraph: 4D
• Concentric Mosaics : 3D
• Panorama: 2D
[L. McMillan 95]
[M. Levoy 96, S. J. Gortler 96]
[H. Y. Shum 99]
[S. E. Chen 95,R. Szeliski 97]
IBR
The Geometry based data
representation
The Geometry based data
representation
The Plenoptic function based
data representation
IBR data are composed of
geometry elements
IBR data are composed of a
set of images with
correspondence relations
IBR data are composed of
a set of light rays
polyhedra
layers
points
View interpolation[CW93]
View morphing[SD96]]
Transfer mode [LF94]
Plenoptic
modeling[MB95]
Lightgield [LH96]/
Lumigraph[GGSC96]
Concentric
Mosaics[SH99]
Panorama[Chen95],[SS97]
MCOP images[RB98],
LDI[SGHS98]
Depth based [BSA98]、Motion based [LS97]、TIP [HAA97] etc.
Hybrid approach of
geometry and image
[DTM96]
5D plenoptic
function
4D plenoptic
function
3D plenoptic
function
2D plenoptic
function
ConclusionConclusion
• The progress of IBR technique is also the progress of new data representation method.
We treat an image:– as texture in geometry texture mapping– as images with correspondence relation view interpolation /morphing– as light beams light field– as slit image concentric mosaics...
Demo of IBRDemo of IBR(1) Tour in Dunhuang Art Cave(1) Tour in Dunhuang Art Cave
Demo of IBRDemo of IBR(2) Tour in Lingyin Temple in Hangzhou(2) Tour in Lingyin Temple in Hangzhou
Image Based Modeling Image Based Modeling (IBM)(IBM)
PART IIPART II
The Common Methods Used for Modeling objects
• Using Geometry Modeling Software Packages, such as
3D MAX , Mayer , SoftImage and AutoCAD etc. to create wireframe models, surface models or volume models.
• Using 3D Laser Scanners• Using Image-based Modeling techniques to create
geometry models or appearance models of objects.
A Taxonomy of IBM
• IBM methods using active cues
The active cues are refered to the artificially generated silhouette, which are projected onto the surface of the modeled objects.
• IBM methods using passive cues
The passive cues are refered to the implicit characteristics of the modeled objects, such as geometry features and textures of the objects.
IBM Methods Using Active cuesIBM Methods Using Active cues
IBM Methods Using Active cuesIBM Methods Using Active cues
IBM Methods Using Passive CuesIBM Methods Using Passive Cues
• Based on known geometry
• Based on visual hull
• Based on light field
• Based on stereo vision
IBM Methods Using Passive CuesIBM Methods Using Passive Cues
• Based on known geometry[Debevec96]
IBM Methods Using Passive CuesIBM Methods Using Passive Cues• Based on visual hull [Wojciech Matusik 2001]
IBM Methods Using Passive CuesIBM Methods Using Passive Cues
• Based on Light Field[Marc Pollefeys et. al.]
IBM Methods Using Passive CuesIBM Methods Using Passive Cues
• Based on stereo pairs: the goal is to automatically extract a realistic 3D model by freely moving a camera around an object.
Neither the camera motion nor the camera settings have to be known.
The obtained 3D model is a scaled version of the original object.
The surface appearance is obtained from the image sequence as well.
IBM Methods Using Passive CuesIBM Methods Using Passive Cues
How can we get 3D information (depth) from a 2D image?
It is impossible to get depth information of the object from a
2D image.
C
P
IBM Methods Using Passive CuesIBM Methods Using Passive CuesIt is possible to recover the depth information
of the object by using two or more images throughtriangulation.
C1
P
C2
p1
p2
Reconstruction of 3D point through triangulation
IBM Methods Using Passive CuesIBM Methods Using Passive Cues
But the following information are needed to know
for this purpose:• Corresponding image points• Relative pose of the camera for different views(so called
camera extrinsic parameters, i.e. the position and orientation of the camera)
• Relation between the image points and the corresponding line of sight. It is defined by the camera model, which usually is a pinhole model and cameras intrinsic parameters and extrinsic parameters.
IBM Methods Using Passive CuesIBM Methods Using Passive Cues
Flowchart of the IBM
technique based on
stereo pairs
Feature PointsExtraction & Match
Camera Calibration: f, R, T
Calculation of dencedepth map
Surface reconstruction
Rectification
Reconstruction of Architectural Reconstruction of Architectural Models Based on Image SequenceModels Based on Image Sequence
Acquesition of imagesequences
O X
Y
c
x1y1
t1t2
t
o1
xy
zc'y'
x'z'
A
BUV
u1 v1a1 b1
cameracalibration
modeling basedon single image
Modelmerging of
multiplescenes
finalmodel
modelmerging
Camera Calibration Using Corner Structure Camera Calibration Using Corner Structure and Parallel Structure in Real Sceneand Parallel Structure in Real Scene
• Consisted of at least 4 line segments.
• 3 of them are perpendicular each other.
• The forth segment is parallel to one of previous 3 segments.
(a) (b)
ox
y
z
l 4
o x
y
l 1l 2
l 3
l 1l 2
l 3
l 4
(a) (b)
ox
y
z
l 4
o x
y
l 1l 2
l 3
l 1l 2
l 3
l 4
l4
l1
l2
(a)
l3
l4
l1
l2
(b)
l3
l4
l1
l2
(a)
l3
l4
l1
l2
(b)
l3
• Corner Structure
• Parallel Structure• Consisted of at least 4 line
segments.
• 2 of them are perpendicular each other.
• Other 2 segments are parallel to previous 2 segments separately.
Calculating focal length based on single imageCalculating focal length based on single image
is the directional vector of segment caTaaca fyxd ],,[
is the normal vector of the projective planecbcaab ddn
is the directional vector of segment ABozabAB nnd
because
0 OYOX ddWe can get (*)
This is a quartic equation relative to f , from which we can get one plus real solution of the focal length.
2f
Projection of the corner structure
Then the directional vectors of OX and OY can be written as an expression relative to the focal length f
OYOX dd x
y
z
O
Y
Z
L
o
x
y
z
A
B
abl
c
X
Calculating rotation matrix and translation Calculating rotation matrix and translation vector based on single image related to vector based on single image related to SCFSCF
)( dOX tVrAxs
From the projection equation of pinhole camera model, we can get:
Rotation matrix relative to SCF is:
Z
Z
OY
OY
OX
OX
d
d
d
d
d
dr
fyxt ood
The direction of the translation vector relative to CCF is
dtt *
xy
z
O
Y
Z
L
o
x
y
z
A
B
abl
c
X
SCF
CCF
Projection of the corner structure
SCF: structure coordinate frameCCF: camera coordinate frame
Translation vector can be deduced:
Optimizing the camera’s parameters: Optimizing the camera’s parameters: f, r and tf, r and t
We choose the distance DIST between the original image point and the reprojected image point as the measurement function for error.
n
i
TT
PPP
ptPrpptPrp
ZYXtztytxtztytxrrrrffFDIST
1
22
33113311
)2))2(2('1))1(1((
),,,2,2,2,1,1,1,22,11,',(
Once the initial values of camera parameters were derived, we can optimize these parameters by using other line segments, which are parallel to the segments of the corner structure in two images.
Procedures for optimizationProcedures for optimization
Rotation matrices Translation vector
Irr
IrrT
T
2*2
1*1
022
011
22
22
tlent
tlent
Structure condition
0)( PQVPQ
),,,2,2,2,1,1,1,22,11,',( 33113311 PPP ZYXtztytxtztytxrrrrffFMin
Objective function of optimization:
To ensure the orthogonality
of rotation matrices
To ensure the length of the two transla-
tion vectors will not be changed in
optimization.
tlen1 and tlen2 are the modules of the
two translation vectors, which can be
calculated from the initial solutions.
To ensure directions of the segments will not be changed in optimization
Constrained conditions:
Experimental ResultsExperimental Results
Two pictures of kiosk at the campus of Zhejiang University taken by hand-hold camera
Camera positions and top-down views of feature points
• Error Statistics: Choose the distance between feature points to its epipolar line as the measurement of error. In the ideal situation this distance should be equal to zero.
n
iii ppDist
nErr
1
)ˆ,(1
Image Err before optimization (pixels)
Err after optimization (pixels)
Left 7.96 1.09
Right 9.08 1.25
Metric(Pixel)
Architecture Reconstruction Based on Architecture Reconstruction Based on Single ImageSingle Image
Reconstruction of planeReconstruction of plane
• Assign base point O and determine its coordinates• Assign base plane which contains the base point O as one of its
corners. Determine the normal vector of the base plane.• Determine unknown neighbor plane of the base plane: choose one
point on the common edge of the neighbor planes as the base point of the unknown plane.
• Determine normal vector of the unknown plane.
Decomposition of sweeping surface into Decomposition of sweeping surface into plane patchesplane patches
A1
v
A2
B1
B2
C1
C2
D1
D2
E1
E2
P1
P2
Wireframe modelWireframe model
Texture mapping modelTexture mapping model
Texture mapping modelTexture mapping model
Strategy of model merging for Strategy of model merging for multiple scenesmultiple scenes
w1
v2
w2
Z
OX
Y
camera1
x1y1
z1c1
camera2
x2
y2
z2
c2
camera3
x3
y3
z3
c3o1
u1
v1
u2
o2
o1u1v1w1 o2u2v2w2
u2
v2
o2
w2o1u1
v1
w1
camera3
camera2
x2
y2
z2c2
w2o1
u1
v1
w1u2
v2
o2
camera2
x2
y2
z2c2
x3
y3z3
c3u2
v2
o2w2o1
u1v1
w1
camera1x1y1
z1c1
Scene structure and 3 camera positions Model merging diagram for multiple scenes
Example of model mergingExample of model merging
Image sequence
Reconstructed model for each image
Main challenges for model merging procedureMain challenges for model merging procedure
• How to transform all the models in different local coordinates to one world coordinate frame?
• Model integration: scaling, vertex match?
• Deletion of overlapped plane?
Establishing listof commoncoordiante
frames
Coordinatetransformation
Mergingvertices
Mergingplanes
Finalmodel
Reconstructed Tea Box with Texture MappedReconstructed Tea Box with Texture Mapped
Experiment 1: image sequenceExperiment 1: image sequence
Experiment 1: reconstructed modelExperiment 1: reconstructed model
Experiment 2: image sequenceExperiment 2: image sequence
Experiment 2: reconstructed modelExperiment 2: reconstructed model
Thanks a lotThanks a lot