Upload
hadley-fowler
View
43
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Combining Photometric and Geometric Constraints. Yael Moses IDC, Herzliya. Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion. Problem 1:. Recover the 3D shape of a general smooth surface from a set of calibrated images. Problem 2:. - PowerPoint PPT Presentation
Citation preview
Y. Moses 1
Combining Photometric and Geometric
Constraints
Yael Moses
IDC, Herzliya
Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion
Y. Moses 2
Recover the 3D shape of a general smooth surface from a set of calibrated images
Problem 1:
Y. Moses 3
Problem 2:
Recover the 3D shape of a smooth bilaterally symmetric object from a single image.
Y. Moses 4
Shape Recovery
Geometry: Stereo Photometry:
Shape from shading Photometric stereo
Main problems: Calibrations and Correspondence
Y. Moses 5
3D Shape Recovery
Photometry: Shape from
shading Photometric
stereo
Geometry:Stereo
Structure from motion
Y. Moses 6
Geometric Stereo
2 different images Known camera parameters Known correspondence
+ +
Y. Moses 7
Photometric Stereo
3D shape recovery: surface normals from two or more
images taken from the same viewpoint
Y. Moses 8
Photometric Stereo
Solution:
)ˆ(
3
2
1
3
2
1
n
l
l
l
I
I
I
IT
T
T
LnI
IL
ILn
1
1
nlyxI
nlyxI
nlyxI
ˆ),(
ˆ),(
ˆ),(
33
22
11
Three images Matrix notation
IL
ILn
1
Y. Moses 9
Photometric Stereo
3D shape recovery (surface normals) Two or more images taken from the
same viewpoint
Main Limitation:
Correspondence is obtained by a fixed viewpoint
IL
ILn
1
Y. Moses 10
Overview
Combining photometric and geometric stereo: Symmetric surface, single image Non symmetric: 3 images
Mono-Geometric stereo Mono-Photometric stereo Experimental results.
Y. Moses 11
The input Smooth featureless surface Taken under different viewpoints Illuminated by different light sources
The Problem: Recover the 3D shape from a set of calibrated images
Y. Moses 12
Assumptions
Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)
*
*
ln
nlI ˆ
n
Three or more images
Perspective projection
Y. Moses 13
Our method
Combines photometric and geometric stereo
We make use of:
Given Correspondence:
Can compute a normal
Can compute the 3D point
Y. Moses 14
IL
ILn
1
1
ˆ
Basic Method
GivenCorrespondence
Y. Moses 15
First Order Surface Approximation
Y. Moses 16
First Order Surface Approximation
Y. Moses 17
First Order Surface Approximation
P() = (1 - )O1 + P,
N)O - (P
N)O - (PT
1
T1
N (P() - P) = 0
Y. Moses 18
First Order Surface Approximation
Y. Moses 19
PMp ii
New Correspondence
Y. Moses 20
IL
ILn
1
1
ˆ
New Surface Approximation
Y. Moses 21
Dense Correspondence
Y. Moses 22
Basic Propagation
Y. Moses 23
Basic Propagation
Y. Moses 24
Basic method: First Order
Given correspondence pi and L
P and n Given P and n T Given P, T and Mi
a new correspondence qi
Y. Moses 25
Extensions
Using more than three images Propagation:
Using multi-neighbours Smart propagation
Second error approximation Error correction:
Based on local continuity Other assumptions on the surface
IL
ILn
1
Y. Moses 26
Multi-neighbors Propagation
Y. Moses 27
Smart Propagation
Y. Moses 28
Second Order: a Sphere
P()
N+N
N
N
P
(P-P())(N+N)=0
Y. Moses 29
Second Order Approximation
Y. Moses 30
Second Order Approximation
Y. Moses 31
Using more than three images
Reduce noise of the photometric stereo
Avoid shadowed pixels Detect “bad pixels”
Noise Shadows Violation of assumptions on the surface
Y. Moses 32
Smart Propagation
Y. Moses 33
Error correction
The compatibility of the local 3D shape can be used to correct errors of:
Correspondence Camera parameters Illumination parameters
Y. Moses 34
Score
Continuity: Shape Normals Albedo
The consistency of 3D points locations and the computed normals: General case: full triangulation Local constraints
Y. Moses 35
Extensions
Using more than three images Propagation:
Using multi-neighbours Smart propagation
Second error approximation Error correction:
Based on local continuity Other assumptions on the surface
Y. Moses 36
Real Images
Camera calibration Light calibration
Direction Intensity Ambient
Y. Moses 37
Error correction + multi-neighbor5 Images
Y. Moses 38
5pp
3pp 3nn
5nn5pn
Y. Moses 39
Y. Moses 40
Y. Moses 41
Y. Moses 42
Y. Moses 43
Y. Moses 44
Detected Correspondence
Y. Moses 45
Error correction + multi-neighbordMulti-neighborsBasic scheme (3 images)Error correction no multi-neighbors
Y. Moses 46
New ImagesSynthetic Images
Y. Moses 47
Sec a
Ground truthBasic schemeMulti-neighbors Error correction
Y. Moses 48
Sec b
Ground truthBasic schemeMulti-neighbors Error correction
Y. Moses 49
Sec c
Ground truthBasic schemeMulti-neighbors Error correction
Y. Moses 50
Ground truthBasic schemeMulti-neighbors
approx.Error correction
Sec d
Ground truthBasic schemeMulti-neighbors Error correction
Y. Moses 51
Combining Photometry and
Geometry
Yields a dense correspondence and dense shape recovery of the object
in a single path
Y. Moses 52
Assumptions
Bilaterally Symmetric object Lambertian surface with constant
albedo Orthographic projection Neither occlusions nor
shadows Known “epipolar geometry”
Y. Moses 53
Geometric Stereo
2 different images Known camera parameters Known viewpoints Known correspondence
3D shape recovery
Y. Moses 54
Computing the Depth from Disparity
pl
pr
P
ql
qrZ
Z
Orthographic
Projection
Y. Moses 55
Symmetry and Geometric Stereo
Non frontal view of a symmetric object
Two different images of the same object
Y. Moses 56
Symmetry and Geometric Stereo
Non frontal view of a symmetric object
Two different images of the same object
Y. Moses 57
Geometry
Weak perspective projection:
)(~ tsRpprojp
Around X Around Z Around Y
zyx RRRR
xR zR yRI I
Y. Moses 58
Geometry
Projection of Ry:
010
)sin(0)cos()(' yRprojR
Around Y
is the only pose parameter
PRp '~
Image point
Object point
Y. Moses 59
objectx
z
image
),,( zyxP l Tr zyxP ),,(
)~,~(~ rrr yxp )~,~(~ lll yxp
Correspondence
Assume YxZ is the symmetry plane.
x~
Y. Moses 60
Mono-Geometric Stereo
3D reconstruction: given correspondence and ,
unknown
known
PRp '~
x~
z
image
x object
)sin()cos(~)sin()cos(~
~~
zxx
zxx
yyy
l
r
lr
xx
rx~lx~
Y. Moses 61
Viewpoint Invariant
Given the correspondence and unknown
2
1
22
11
~~
~~
x
x
pp
pplr
lr
*1
~p
*2~p
*3~p 3*
~p2*
~p1*
~p
lr pp 11~~
)sin()cos(~)sin()cos(~
~~
zxx
zxx
yyy
l
r
lr
Invariant
Y. Moses 62
Photometric Stereo
2 images Lambertian reflectance Known illuminations Known correspondence
(same viewpoint)
3D shape recovery
Y. Moses 63
Symmetry and Photometric Stereo
Non-frontal illumination of a symmetric object
Two different images of the same object
Y. Moses 64
Notation: Photometry
Corresponding object points:
Illumination:
Tyx
r zzn )1,,( Tyx
l zzn )1,,( Tzyx
r nnnn ),,(ˆ Tzyx
l nnnn ),,(ˆ
Tzyx eeeE ),,(
Tzyx
r nnnn ),,(ˆ Tzyx
l nnnn ),,(ˆ
Y. Moses 65
Mono-Photometric Stereo
3D reconstruction given correspondence and E (up to a twofold ambiguity):
unknown
known
zzyyxxr neneneI
zzyyxxl neneneI
)(2
2
zzyylr
xxlr
neneIII
neIII
Y. Moses 66
Invariance to Illumination
Given correspondence and E unknown
Invariant:x
x
n
n
I
I
2
1
2
1
)(2
2
zzyylr
xxlr
neneIII
neIII
Y. Moses 67
Mono-Photometric Stereo
3D reconstruction E unknown but correspondence is given
Frontal viewpoint with non-frontal illumination. Use image first derivatives.
Y. Moses 68
Mono-Photometric Stereo Using image derivatives
3 global unknowns: E For each pair:
5 unknowns zx zy zxx zxy zyy
6 equations 3 pairs are sufficient
Y. Moses 69
Mono-Photometric StereoUnknown Illumination
Y. Moses 70
Correspondence
No correspondence => no stereo. Hard to define correspondence in
images of smooth surfaces. Almost any correspondence is legal
when: Only geometric constraints are
considered. Only photometric constraints are
considered.
Y. Moses 71
Combining Photometry and
Geometry Yields a dense correspondence (dense shape recovery of the
object).
Enables recovering of the global parameters.
Y. Moses 72
Self-Correspondence A self-correspondence function:
lr ppC ~)~(
*~ lp
rp~*lrr xyxC ~)~,~(
Y. Moses 73
Dense Correspondence using Propagation
Assume correspondence between a pair of points, p0
l and p0r.
rr yxCydCxdpCpdpC ~~
~~)~()~~(
lp2~
rp0*~
rp1*~
rp2~
*1~ lp
*0~ lp
)~,~(~ ydxdpd
Y. Moses 74
Dense Correspondence using Propagation
* ***
**
lp2~
rp0*~
rp1*~
rp2~
*1~ lp
*0~ lp
Y. Moses 75
x
x~
z
image
objectn̂
dxdx
dz
)sin()cos(~ dzdxxd )cos(~ dxxd
xdxcxdxc ~)~()~~(
dx
dz
xCxdxCxdxC ~~)~()~~(
Y. Moses 76
Tyx zzn )1,,(
First derivatives of the Correspondence
Assume known Assume known E
)sin()cos(
)2sin(
;)sin()cos(
)sin()cos(
~
~
x
y
y
x
xx
z
zC
z
zC
r
r
Y. Moses 77
)sin()cos(~)sin()cos(~
~~
zxxzxx
yyy
l
r
lr
Computing and
ryy
ryxy
rxy
rxxx
yCxCC
yCxCC
rr
rr
~~
~~
~~
~~
rr yxCC ~~
Object coordinates:Given computing and is trivial
Moving from object to image coordinates depends on the viewing parameter
rr yxCCn̂
Y. Moses 78
Derivatives with respect to the object coordinates:
Derivatives with respect to the image coordinates:
Y. Moses 79
x
x~
z
image
object
xd~
yx cc ~~
E
)~,~(~ rrr yxp
)~,~(~ lll yxp
Y. Moses 80
Given a corresponding pair and E n=(zx,zy,-1)T Given and n
cx and cy
Given cx and cy
a new corresponding pair
General Idea
Y. Moses 81
Results on Real Images: Given global parameters
Y. Moses 82
Finding Global Parameters
Assume E and are unknown. Assume a pair of corresponding
points is given. Two possibilities:
Search for E and directly.
Compute E and from the image second derivatives.
Y. Moses 83
All roads lead to Rome …
Find and verify correct correspondence
Recover global parameters, E and
Integration Constraint:Circular Tour
Y. Moses 84
Finding Global Parameters
Consider image second derivatives Due to foreshortening effect:
and
We can relate image and object derivatives by
xll
xxrr
xIIII rr ~~
ry
r
y
ry
r
x
ry
rx
r
y
rx
r
x
rx
yIxII
yIxII
rr
rr
~~
~~
~~
~~
Y. Moses 85
For each corresponding pair:
and
Plus 4 linear equations in 3 unknown.
Where
Testing E and : Image second derivatives
),,,(34
EzzfA yxx
yy
xy
xx
l
y
r
y
l
x
r
x
zzz
A
I
III
r
r
r
r
~
~
~
~
rrll nEInEI ˆˆ
Y. Moses 86
Counting
5 unknowns for each pair: zx zy,zxx zxy zyy
4 global unknowns: E, For each pair: 6 equations. For n pairs: 5n+4 unknowns
6n equations. 4 pairs are sufficient
Y. Moses 87
Results on Simulated Data
Ground Truth Recovered Shape
Y. Moses 88
Recovering the Global Parameters
Y. Moses 89
Degenerate Case
Close to frontal view: problems with geometric-stereo.
reconstruction problem Close to frontal illumination:
problems with photometric-stereo. correspondence problem
Y. Moses 90
Future work
Perspective photometric stereo Use as a first approximation to global
optimization methods Test on other reflection models Recovering of the global parameters:
Light Cameras
Detect the first pair of correspondence
Y. Moses 91
Future Work
Extend to general 3 images under 3 viewpoints and 3 illuminations.
Extend to non-lambertian surfaces.
Y. Moses 4
Assumptions
Bilaterally Symmetric object Lambertian surface with constant
albedo Orthographic projection Neither occlusions nor shadows Known “epipolar geometry”
Y. Moses 92
Thanks
Y. Moses 93
x
x~
z
image
object
)cos(~
dx
xd
n̂
dx
dz
dx
dz
Y. Moses 94
Integration Constraint
ry
r
y
ry
r
x
ry
rx
r
y
rx
r
x
rx
yIxII
yIxII
rr
rr
~~
~~
~~
~~
l
xylx
l
y
r
xyrx
r
y
x
lxl
x
x
rxr
x
lr
rr
r
r
IznEI
IznEI
z
nEI
z
nEI
~~
~~
~
~
)sin(ˆ
)sin(ˆ
)sin()cos(
ˆ
)sin()cos(
ˆ
Y. Moses 95
Integration Constraint
)(2
2
zzyy
xx
neneI
neI
l
xylx
l
y
r
xyrx
r
y
x
lxl
x
x
rxr
x
lr
rr
r
r
IznEI
IznEI
z
nEI
z
nEI
~~
~~
~
~
)sin(ˆ
)sin(ˆ
)sin()cos(
ˆ
)sin()cos(
ˆ
Y. Moses 96
Searching for E
Illumination must satisfy:
E is further constrained by the image second derivatives.
),max( lr IIE
)(2 zzyy neneI
xxneI 2
Y. Moses 97
Image second derivatives:
xll
x
xrr
x
yrl
y
yrr
y
II
II
II
II
r
r
r
r
~
~
~
~
lx
l
x
rx
r
x
ly
l
y
ry
r
y
nEI
nEI
nEI
nEI
r
r
r
r
ˆ
ˆ
ˆ
ˆ
~
~
~
~
),,,,(ˆ yyxyxxyxrx zzzzzn
yy
xy
xx
l
y
r
y
l
x
r
x
zzz
A
I
III
r
r
r
r
~
~
~
~ll nEI ˆrr nEI ˆ
Where ),,(34
EzzfA yxx
4 linear equations in 3 unknown
Y. Moses 98
For each corresponding pair and E: 4 linear equations in 3 unknown.
Where
Image second derivatives
),,(34
EzzfA yxx
yy
xy
xx
l
y
r
y
l
x
r
x
zzz
A
I
III
r
r
r
r
~
~
~
~
zzyyxxr neneneI
zzyyxxl neneneI
Y. Moses 99
Counting
5 unknowns for each pair: zx,zy,zxx,zxy,zyy
3 global unknowns: E For each pair: 6 equations. For n pairs: 5n+3 unknowns
6n equations.3 pairs are sufficient
Y. Moses 100
Correspondence
Tr zyxP ),,( Tl zyxP ),,(
)~,~(~ rrr yxp )~,~(~ lll yxp
Y. Moses 101
Variations
Known/unknown distant light source Known/unknown viewpoint Symmetric/non-symmetric image
Frontal/non-frontal viewpoint Frontal/non-frontal illumination
Y. Moses 102
Correspondence
Epipolar geometry is the only geometric constraint on the correspondence.
Weak photometric constraint on the correspondence.
)(2
2
zzyy
xx
neneI
neI
Y. Moses 103
Lambertian Surface
Basic radiometric
)cos(ˆ),(Re EnEqpf I =
E
*
*P
EE
n2
n1
n̂
Y. Moses 104
E
Photometric Stereo
First proposed by Woodham, 1980. Assume that we have two images ..
nEInEI ˆˆ 2211