Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Selected Topics in Computer VisionSelected Topics in Computer Vision
Computational Photography
Lecture #2
Francesc Moreno‐Noguer
Today
Shape From XRadiometry and ReflectanceRadiometry and ReflectancePhotometric Stereo
Shape from Xp
Shape from X
Methods for obtaining the 3D shape fromMethods for obtaining the 3D shape from images. “X” can be many different cues:– Stereo (2 or more views)( )– Motion (moving camera or object)– ShadingShading– Changing Lighting (Photometric Stereo)– Texture variation– Texture variation– Focus/Defocus
Structured light– Structured light
Shape from XShape From Stereo and Motion
-Triangulate the same point on two or more imagesg p g-Images acquired simultaneously SF Stereo-Images from a moving camera or object SF Motion
Shape from XShape From Shading
Shading: “Variation of brightness due to changes in surfaceShading: Variation of brightness due to changes in surface orientation”
Shading reveals 3D surface geometryS f SShape-from-Shading: Just one image to recover shape Requires many constraints
Shape from XPhotometric Stereo
Single viewpoint, and multiple images under different lighting. g p , p g g g
Shape from XShape From Focus/Defocus
Camera Focus/Defocus is another cue to estimate depth.p
Shape from XShape From Texture
Texture: “Repetition of an element or the appearance of a specific template over a surface”of a specific template over a surface
Shape-from-Texture: guess the shape of a surface f h d f i f i T l (TEX EL )from the deformation of its Texels (TEXture ELements)
Shape from XStructure Light and Laser Range Finders
Active TriangulationActive Triangulation−Projector-plane / Image-plane triangulation
Shape from XTraditional Approaches
– Stereo– Motion– Shading– Photometric Stereo Lecture #2– Texture variation
Camera Focus/Defocus Lecture #3– Camera Focus/Defocus Lecture #3– Structured Light
Novel Techniques– Helmholtz Stereopsis Lecture #3
Projector Defocus Lecture #4– Projector Defocus Lecture #4– Coded and Multi Aperture Lecture #5
Radiometry and Reflectancey
Digital Image Formation
LightingCamera g g
Physical Models
Computer
Scene
We need to understand the relation between the lighting, surface reflectance and medium
and the image of the scene.
Digital Image Formation
• Light Source – Image Plane:
SurfaceRadiance
Lens ImageIrradiance
SurfaceLight S
SurfaceIrradiance Radiance IrradianceSource
surfaceL EIrradiance
surfaceE
• Image Plane – Measured Pixels:
CameraElectronics
ImageIrradiance E
Measured Pixel Values, I
Digital Image Formation
• Light Source – Image Plane:
SurfaceRadiance
Lens ImageIrradiance
SurfaceLight S
SurfaceIrradiance Radiance IrradianceSource Irradiance
surfaceE surfaceL E
• Image Plane – Measured Pixels:
CameraElectronics
ImageIrradiance E
Measured Pixel Values, I
Digital Image Formationz source
viewingincidentdirection
= normal direction
y
θviewingdirection
normal
direction
),( ii φθ ),( rr φθ
x
φsurfaceelement
),( iisurfaceE φθ
),( rrsurfaceL φθ
Irradiance at Surface in direction ),( ii φθRadiance of Surface in direction ),( rr φθ
BRDF :),(),(),;,(
iisurface
rrsurface
rrii ELf
φθφθ
=φθφθ),( ii φ
Bidirectional Reflectance Distribution Function
Digital Image Formation
• Light Source – Image Plane:
SurfaceRadiance
Lens ImageIrradiance
SurfaceLight S
SurfaceIrradiance Radiance IrradianceSource Irradiance
surfaceE surfaceL E
• Image Plane – Measured Pixels:
CameraElectronics
ImageIrradiance E
Measured Pixel Values, I
Digital Image Formation
surface patch
image plane
θdA
αα
sdA
idAimage patch
fz fz
2⎞⎛ d
It can be shown that:
4cos4
α⎟⎟⎠
⎞⎜⎜⎝
⎛π=
fdLE surface
• Image irradiance is PROPORTIONAL to Surface Radiance!Image irradiance is PROPORTIONAL to Surface Radiance!• Small field of view Effects of 4th power of cosine are small• d/f is the inverse of the f-number.
Digital Image Formation
• Light Source – Image Plane:
SurfaceRadiance
Lens ImageIrradiance
SurfaceLight S
SurfaceIrradiance Radiance IrradianceSource Irradiance
surfaceE surfaceL E
• Image Plane – Measured Pixels:
CameraElectronics
ImageIrradiance
Measured Pixel Values, I
E
Digital Image Formation
• The camera response function relates image irradiance at the image plane to the measured pixel intensity values.p y
IEg →:
Pixel values in the final image
(Grossberg and Nayar, CVPR 2003)
Usually this is a Non-linear Mapping!
Digital Image Formation
• Important preprocessing step for many vision and graphics algorithms such ash t t i t i i t d th i i d i i b d d i t
Radiometric Linearization of the Cameraphotometric stereo, invariants, de-weathering, inverse rendering, image based rendering, etc.
EIg →− :1
• Use a color (Magbeth) chart with precisely known reflectances.
Valu
es
255
?1−g
Pix
el V
0
g
Irradiance = const * Reflectance3.1%9.0%19.8%36.2%59.1%90%
• Method assumes constant lighting on all patches and works best when source is
00 1?
far away (example sunlight).
• Unique inverse exists because g is monotonic and smooth for all cameras.
Mechanisms of Reflectionsource
surfacereflection
incidentdi ti reflectiondirection
bodyreflection
surface
• Body Reflection: • Surface Reflection:
S l R fl tiDiffuse ReflectionMatte AppearanceNon-Homogeneous MediumCla paper etc
Specular ReflectionGlossy AppearanceHighlightsDominant for MetalsClay, paper, etc Dominant for Metals
Image Intensity = Body Reflection + Surface Reflection
Example Surfaces
Diffuse Reflection Specular ReflectionDiffuse + SpecularReflection
Reflectance Models
Diffuse Reflection: Lambertian ModelHow to model the mechanisms of reflectance?
Diffuse Reflection: Lambertian Model• Surface appears equally bright from ALL directions! (independent of )vr
Source intensity K
viewing
normal
incidentnr
viewingdirection
f
incidentdirection
iθvrsr
surfaceelement
lb d
πρφθφθ d
rriif =),;,(• Lambertian BRDF is simply a constant :albedo
Reflectance Models
Diffuse Reflection: Lambertian Model
• Surface Radiance : snKKL di
d rr⋅
πρ
=θπρ
= cossource intensity
• Commonly used in Vision and Graphics!
• This is the Lambert’s cosine law of reflection from matte surfaces.
Reflectance ModelsSpecular Reflection (Mirror BRDF)
• Valid for very smooth surfaces.Valid for very smooth surfaces.
• All incident light energy reflected in a SINGLE direction
source intensity K specular/mirrory
incident
rrp
direction ),( rr φθ
sr
viewingdirectionsurface
normal
c de tdirection
),( ii φθ)( φθ
nr
vrs
directionsurfaceelement
),( vv φθv
• Mirror BRDF is simply a double-delta function :Mirror BRDF is simply a double delta function :
)()(),;,( vivisvviif φπφδθθδρφθφθ −+−=specular albedo
Reflectance ModelsSpecular Reflection (Mirror BRDF)
• Surface Radiance : )()( vivisKL φ−π+φδθ−θδρ=
( ) ( )⎧ π−φ=φθ=θρ andifK ii( ) ( )
⎪⎩
⎪⎨
⎧ πφφθθρ=
otherwise
andifKL
vivis
0 z specular/mirrordirection⎩ otherwise0
θ),( ii φθ( ) ( )π+φθ=φθ iirr ,,
direction
x
yφ
• Viewer receives light only when rv rr=
x
Reflectance ModelsDiffuse + Specular Reflection
• Many real surfaces combine diffuse and glossy components ofMany real surfaces combine diffuse and glossy components of reflection
• In order to approximate these surfaces Models combining Diffuse and Specular reflection (Phong, Dichromatic, …)
Observed Image Color = α x Body Color + β x Specular Reflection Color
Diffuse and Specular reflection (Phong, Dichromatic, …)
diffuse specular diffuse+specular
Reflectance ModelsPhong Model
( ) ( )α∑Ambient Diffuse Specular
( ) ( ) sα
slights
ddaa ikikikI vrns ⋅+⋅+= ∑• ia id is: Intensities of the ambient
Models an angular falloff of the highlights
s rv
nia,id,is: Intensities of the ambient specular and diffuse components of the light sources.
v•ka,kd,ks: material components
•α : shininess how evenly the light is reflected
sn)ns(r −⋅= 2is reflected
Reflectance ModelsHomework: Implement Phong Model for a Sphere
(for a sphere is easy to compute the normals)
)Ryxy,(x,R1y)N(x, 222 −−=
Reflectance Models
Results
Phong Model Parametersia:0.19 id:0.44 is:0.59
Phong Model Parametersia:0.17 id:0.88 is:0.41
Phong Model Parametersia:0.16 id:0.43 is:0.45
ka:0.03 kd:0.47 ks:0.50alpha:8.94
ka:0.13 kd:0.47 ks:0.53alpha:17.09
ka:0.14 kd:0.49 ks:0.43alpha:8.63
Photometric Stereo
Surface Orientation
N Surface : ( )y)x,fy,x,y)s(x, (=surface normalz
Tangent vectors : xf1,0,
xy)s(x,
⎟⎠⎞
⎜⎝⎛
∂∂
=∂
∂f(x,y)
yf0,1,
yy)s(x,
⎟⎟⎠
⎞⎜⎜⎝
⎛∂∂
=∂
∂y
x
ys
xs
∂∂
×∂∂
=NNormal :
⎞⎛ ∂∂ ss
'yImage Plane
⎟⎟⎠
⎞⎜⎜⎝
⎛∂∂
−∂∂
−= 1ys
xs ,,
( ) 1qp ,,=
'x
Gradient Spacez
Unit Normal vector
1=zq
1N
S ( )1
1,,22 ++
==qpqp
NNn
yp s niθ
( )1,,== SS qpSs
Source vector
x 122 ++==
SS qpSs
1=z plane is called the Gradient Space (pq plane)• Its components p and q, are the surface slopes in the x- and y- directions• Every point on it corresponds to a particular surface orientationEvery point on it corresponds to a particular surface orientation
Reflectance Map• Relates image irradiance I(x,y) to surface orientation (p,q)
for GIVEN source direction and surface reflectance• Tool used in developing method for recovering shape from
images. Let’s consider an example:• Lambertian case: ( )yxI• Lambertian case: ( )yxI ,
: source brightness
: surface albedo (reflectance)
kρ
s vniθ
: surface albedo (reflectance)
: constant (optical system)
ρc
Image irradiance:Image irradiance:
sn ⋅== kckcI i πρθ
πρ cos
Let 1=kcπρ
then sn ⋅== iI θcos
Reflectance Map
( ) ( )qqpp 1++
Lambertian Case( ) ( )qpR
qpqpqqppI
SS
ssi ,
111cos
2222=
++++
++=⋅== snθ
R fl t MReflectance Map(Lambertian)
• Contours of constant brightness are conic sections in• Contours of constant brightness are conic sections in the pq-plane:
( ) ( )( )( ) cqpR =, ( ) ( )( )111 222222 ++++=++⇒ SSss qpqpcqqpp
Reflectance MapLambertian Case
( ) ( )( )111 222222 ++++++
Iso-brightness contour ( ) cqpR =,
( ) ( )( )111 22222 ++++=++ SSss qpqpcqqpp
p
cone of constant iθq
Reflectance MapLambertian Case
qiso-brightnesscontour
0.1
9.08.0
( ) 7.0, =qpR( )
contour
( )SS qp ,
po90=iθ
( )01=++ SS qqpp
3.0
0.0Note: is maximum when( )qpR , ( ) ( )SS qpqp ,, =
Reflectance MapDiffuse + Specular
• Reflectance map with two peaks: orientations that
q Diffuse peak
p pmaximize each of the two different types of reflection.
( )SS qp ,
p
Specular peak
p
Specular peak
( ) 5.0, =qpR
Shape from a Single Image?• Given a single image of an object with known surface
reflectance taken under a known light source, can we grecover the shape of the object?
• Given R(p,q) ( (pS,qS) and surface reflectance) can we determine (p q) uniquely for each image point?determine (p,q) uniquely for each image point?
q
All the points on the line, are solutions for I(x,y)
I(x,y)
NO
p
( y)
NO
Solution
• Take more imagesTake more images– Photometric stereo
• Add more constraints– Shape-from-shading (Too restrictive p g (
to be useful )
Photometric Stereo
q
( )11 ,SS
qp
p( )22 ,
SSqp
( )33( )33 ,SS
qp
Photometric Stereo
⎠⎞
⎜⎝⎛ =1kc
Lambertian case:
sn ⋅== ρθρikcI cos
Image irradiance:⎠⎝ π
11 sn ⋅= ρIn
2s3s
ρπ i
11 ρ1s v
22 sn ⋅= ρI33 sn ⋅= ρI
• We can write this in matrix form:
s ⎤⎡⎤⎡ TIns
s
⎥⎥⎥⎤
⎢⎢⎢⎡
=⎥⎥⎤
⎢⎢⎡
TII
22
1 1
ρs ⎥⎦⎢⎣⎥
⎥⎦⎢
⎢⎣
TI 32
Solving the Equations
s⎥⎤
⎢⎡
⎥⎤
⎢⎡ TI 1 1
nss ρ
⎥⎥⎥
⎦⎢⎢⎢
⎣
=⎥⎥⎥
⎦⎢⎢⎢
⎣T
T
II
3
2
2
2
⎦⎣⎦⎣ 32
I S n~133313 13×33×13×
ISn 1~ −= inverse
n~=ρnn ~~
Since n is a unit vector,the norm of will be ρ
ρn
nnn ~ ==
More than Three Light Sources• Get better results by using more lights
s ⎤⎡⎤⎡ TIn
s
sρ
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
TI
IMM11
s ⎥⎦⎢⎣⎥⎦⎢⎣ NNI
• Least squares solution:~nSI ~=
nSSIS ~TT =1331 ××× = n~SI NN
( ) ISSSn TT 1~ −=
M P• Solve for as beforen,ρ Moore-Penrose pseudo inverse
Color Images• The case of RGB images
– get three sets of equations, one per color channel:g q p
SnI ρ=SnI RR ρ=SnI GG ρ=SnI BB ρ=
– Simple solution: first solve for using one channel– Then substitute known into above equations to get
( )ρρρn
n
– Or combine three channels and solve for
( )BGR ρρρ ,,
nSnIIII ρ=++= 222
BGR
Trick for Handling Shadows• Weigh each equation by the pixel brightness:
( ) ( )• Gives weighted least-squares matrix equation:
( ) ( )iiii III sn ⋅= ρ
g q q
s⎥⎤
⎢⎡
⎥⎤
⎢⎡ TII
MM11
21
ns
ρ⎥⎥⎥
⎦⎢⎢⎢
⎣
=⎥⎥⎥
⎦⎢⎢⎢
⎣TNNN II
MM2
• Solve for as beforen,ρ
Depth From Normals• Photometric Stereo “just” recovers the surface normals• We need to compute the depth!!!
Easy Method: Shape By Integration (1D)• Partial Derivatives: Give the change in surface height with a small step.
f∂0p• We can get the surface by summing the changes.
ixxi x
fp=∂
∂=
0p
dxxfxf ix
i ∫ ∂∂
=0
)(Integrated
orientations
ii pxf += − )( 1
Depth From NormalsShape By Integration (2D)
y( )0000 qp ,
( )qp
y
( )1010 qp ,
x( )ijij qp ,( )ijij qp ,
∫∫∂∂ jiyx ff
Assigned a priori
∑∑∫∫ ++=∂∂
+∂∂
=j
iji
yx
ij qpfdyyfdx
xff ji
0000000
Depth From NormalsShape By Integration (2D)
Limitations:• Integration is good only if object shape is continuous• It is sensitive to noise:
− Solution: Average the results over many different paths.
X X X XX X X X
High cost and still inaccurate!!!!!!
Depth From NormalsFrankot-Chellappa Algorithm
• Best algorithm I found to compute depth from normalsBest algorithm I found to compute depth from normals.
• Very robust to noise.
• Fast and very simple implementation.
• Let be the surface to be reconstructed( )),(ˆ,, yxfyxFormulation of the problem:
• The basic idea is to project the estimated surface slopes p(x,y) and
q(x,y) onto a set of integrable surface slopes and ),(ˆ yxp ),(ˆ yxqsuch that:
1. Integratibility Constraint: xyyxf
yxyxfq
xp
y ∂∂∂
=∂∂
∂⇔
∂∂
=∂∂ ),(),(ˆˆ
22
2. The distance error is minimized:
xyyxxy ∂∂∂∂∂∂
∫∫ −+− dxdyqqpp 22 ˆˆ
Depth From NormalsFrankot-Chellappa Algorithm
Solution:• Expand by Fourier:
Solution:),(ˆ yxf
∑ ωφω= )()(ˆ)(ˆ yxCyxf This expansion ensures∑Ω∈ω
ωφ⋅ω= ),,()(),( yxCyxf
Fourier Coefficients
This expansion ensures integrability (condition 1 before), because Fourier basis functions are integrable
• We need to find the coefficients that minimize the distance error of
condition 2. The paper shows that these optimal coefficients are:
)(ˆ ωC
22yx
yyxx CjCjwC
ω+ωωω−ωω−
=)()(
)(ˆ
)()(C FFT These are just the Fourierh )()( pCx FFT=ω
)()( qCy FFT=ω
These are just the Fourier Transforms of the original estimated slopes p and q !!!!!
where:
Depth From NormalsFrankot-Chellappa Algorithm
Depth From NormalsFrankot-Chellappa Algorithm
fftshiftfft2 fftshift
ifftshift
fft2