Robot Vision SS 2009 Matthias Rüther 1
ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction
Matthias Rüther
Robot Vision SS 2009 Matthias Rüther 2
Overview
Shape from X techniques
– Structured light– Plane Sweep Stereo– Global optimization methods– Shape from (de)focus– Specularities– Shading, Photometric Stereo– Alternative methods
Robot Vision SS 2009 Matthias Rüther 3
Structured Light Methods
In principle same as multi-view/stereo
Project artificial pattern on the object
Pattern alleviates the correspondence problem
Variants:– Laser Pattern (point, line)
– Structured projector pattern (several lines, pattern sequence)
– Random projector pattern
Robot Vision SS 2009 Matthias Rüther 4
Structured Light Range Finder
1. Sender (projects plane)2. Receiver (CCD Camera)
X- directionGeometry Z- direction Sensor image
Robot Vision SS 2009 Matthias Rüther 5
1 plane -> 1 object profile
Object motion by conveyor band:=> synchronization: measure distance along conveyor=> y-accuracy determined by distance measurement
Scanning Units (e.g.: rotating mirror) are rare (accurate measurement of mirror motion is hard, small inaccuracy there -> large inaccuracy in geometry
Move the sensor: e.g. railways: sensor in wagon coupled to speed measurement
To get a 3D profile:• Move the object• Scanning Unit for projected plane• Move the Sensor
Robot Vision SS 2009 Matthias Rüther 6
Robot Vision SS 2009 Matthias Rüther 7
Commercially Available
Robot Vision SS 2009 Matthias Rüther 8
CAESARTM
Civilian American and European Surface Anthropometry Resource Project—CAESARTM
– 2400 Male/Female Americans
– 2000 Male/Female Europeans
Robot Vision SS 2009 Matthias Rüther 9
Problems
Occlusions
Sharpness and Contrast
Speckle noise
Robot Vision SS 2009 Matthias Rüther 10
Gleichzeitige Projektion mehrerer Lichtschnitte
Anstatt einer Lichtebene werden mehrere Lichtebenen auf das Objekt projeziert, umdie Anzahl der aufzunehmenden Bilder zu reduzieren.
Entfernungsberechnung:wie mit einer Lichtebene, jedochmuß jeder Lichtstreifen im Bildeindeutig identifizierbar sein.Problem: Aufgrund von Verdeckungen sind einzelne Streifen teilweise oder gar nicht im Kamerabild sichtbar -> keine eindeutige Identifikation der LichtstreifenAnwendung: Glattheitsüberprüfung bei planaren Oberflächen ohne Tiefenwertberechnung.
Robot Vision SS 2009 Matthias Rüther 11
Pattern projection
CameraCamera: IMAG
CCD, Res:750x590,
f:16 mm
ProjectorProjector: Liquid Crystal Display (LCD 640), f: 200mm, Distance to object plane: 120cm
Projected light stripes
Range Image
Robot Vision SS 2009 Matthias Rüther 12
Projector
Lamp
Lens system
LCD - Shutter
Pattern structure
Example
Focusing lens (e.g.: 150mm)
Line projector (z.b: LCD-640)
Robot Vision SS 2009 Matthias Rüther 13
Tiefenberechnung für Streifenprojektor
1) Unterschiedlich breite Lichtstreifen werden zeitlich aufeinanderfolgend in die Szene projiziert und von der Kamera aufgenommen.2) Für jede Aufnahme wird für jeden Bildpunkt festgestellt, ob dieser beleuchtet wird oder nicht.3) Diese Information wird für jeden Bildpunkt und für jede Aufnahme im sog. Bit-Plane Stack abgespeichert.
Verschiedene Lichtstreifen sind notwendig, um für jeden Bildpunkt einen zugehörigen Lichtstreifen festzustellen zu können.
Durch die zeitliche Abfolge der Aufnahmen wird es ermöglicht, daß jeder Lichtstreifen im Kamerabild identifiziert wird.
4) Findet man im Bit-Plane Stack für einen Bildpunkt die Information, daß er bei den Aufnahmen z.B. hell, dunkel, dunkel, hell war (Code 1001), dann folgt daraus, daß dieser Bildpunkt vom vierten Lichtstreifen beleuchtet wird.5) eindeutige Zuordnung „Lichtstreifen – Bildpunkt“ möglich
Robot Vision SS 2009 Matthias Rüther 14
Coded Light + Phase Shift
Binary code is limited to pixel accuracy (at most).
Increase accuracy by projecting sine wave and measuring phase shift between projected and captured pattern.
Robot Vision SS 2009 Matthias Rüther 15
Joaquim Salvi,
Pattern codification strategies in structured light systems
Robot Vision SS 2009 Matthias Rüther 16
Random Texture Projection
Robot Vision SS 2009 Matthias Rüther 17
Moiré Range Finder
Project line structure, observe line structure through a grid
1. Sender (Projektor mit Linien)2. Receiver (CCD Camera with line filter)
Problem: identification of line ordering possible but hard, unsharp lines => inaccurate results
Moiré ImageMoiré Pattern
Robot Vision SS 2009 Matthias Rüther 18
Moiré Range Finder
Y. Suenaga, 3D Measurements for Computer Animation
Robot Vision SS 2009 Matthias Rüther 19
Plane Sweep Stereo
Sweep family of planes through volume
– each plane defines an image composite homography
input imageinput image
projectiveprojective re-sampling of ( re-sampling of (X,Y,ZX,Y,Z))
Richard Szeliski, IBMR 1998
Robot Vision SS 2009 Matthias Rüther 20
Plane Sweep Stereo
For each depth plane– compute composite (mosaic) image — mean
– compute error image — variance
– convert to confidence and aggregate spatially
Select winning depth at each pixel
Richard Szeliski, IBMR 1998
Robot Vision SS 2009 Matthias Rüther 21
Voxel Coloring
Robot Vision SS 2009 Matthias Rüther 22
Voxel Coloring / Space Carving
S={} /* initial set of colored voxels is empty
for i = 1 to r do /* traverse each of r layers
foreach V in the ith layer of voxels do
project V into all images where V is visible
if sufficient correlation of the pixel colors then
add V to S
Photorealistic Scene Reconstruction by Voxel Coloring S. M. Seitz and C. R. Dyer, Proc. Computer Vision and Pattern Recognition Conf., 1997, 1067-1073
Robot Vision SS 2009 Matthias Rüther 23
Shape From Focus
Recover shape of surfaces from limitied depth of view.– Requires visibly rough surfaces– Typical application: optical microscopy
Robot Vision SS 2009 Matthias Rüther 24
Shape From Focus
Visibly Rough Surfaces– Optical roughness: the smallest spatial variations are much larger than the
wavelength of incident electromagnetic wave.
– Visible roughness: smallest spatial variations are comparable to viewing area of discrete elements (pixels).
– Magnification:• Multi-facet level: w1 >> wf, smooth texture
• Facet level: w2 ~= wf, rough texture
Robot Vision SS 2009 Matthias Rüther 25
Shape From Focus
Focused / Defocused images
– Focused:
– Defocused: object point is mapped to spot with radius
=> defocusing is equivalent to convolution with low pass kernel (pillbox function)
fio
111
i
Rr
otherwise
ryxifryxp
yxIyxpyxI fd
0
1),(
),(*),(),(
2222
Robot Vision SS 2009 Matthias Rüther 26
Shape From Focus
Changing Focus
– Displacing the sensor: changes sharp region, magnification and brightness
– Moving the lens: changes sharp region, magnification and brightness
– Moving the object: changes sharp region only
=> Object is moved in front of static camera
Robot Vision SS 2009 Matthias Rüther 27
Shape From Focus
Overview:– At facet level magnification, rough
surfaces give texture-rich images– A defocused image is equivalent to a
low-passed image
– As S moves towards focused plane, its focus increases. When S is best focused,
– Challenges:• How to measure focus?
• How to find best focus from finite number of measurements?
ddd fs
Robot Vision SS 2009 Matthias Rüther 28
Shape From Focus
Focus measure operator– Purpose:
• respond to high frequency variations in image intensity within a small image area
• produce maximum response when image area is perfectly focused
– Possible solution: • determine high frequency content using Fourier transform (slow)
– Alternative:• Laplacian operator (problem with elimination)
• Modified Laplacian
Robot Vision SS 2009 Matthias Rüther 29
Shape From Focus
• Sum Modified Laplacian
• Tenengrad Focus Measure
• Alternatives: variance of intensities, variance of gradients
101
202
101
xS
121
000
121
yS
M
m
N
n
yx
nmSITEN
nmSnmSnmS
2
22
)],([)(
)],([)],([),(
INxM … local intensity function (image window)
Robot Vision SS 2009 Matthias Rüther 30
Shape From Focus
Example
Infinite DOF DEM
Robot Vision SS 2009 Matthias Rüther 31
Shape From Focus
Sampling the focus measure function– Consider a single image point (x,y)
– Focus measure F is function of depth d: F(d)
– Goal find Fpeak from finite number of samples F1…F8
Robot Vision SS 2009 Matthias Rüther 32
Shape from focus
Sampling the focus measure function– Possibility1: find highest discrete sample
Robot Vision SS 2009 Matthias Rüther 33
Shape from focus
Sampling the focus measure function– Possibility2: Gaussian interpolation
Fit Gauss function to three strongest samples
Robot Vision SS 2009 Matthias Rüther 34
Shape from Specularity
Suitable for highly reflective Surfaces
Specular Reflection map of a single point source forms a sharp peak (Specular model, Phong model)
Robot Vision SS 2009 Matthias Rüther 35
Shape from Specularity
Principle:– If a reflection is seen by the camera
and the position of the point source is known, the surface normal can be determined.
– => use several point sources with known position: structured highlight inspection
Robot Vision SS 2009 Matthias Rüther 36
Shape From Shadow
Also: Shape from Darkness
Reconstruct Surface Topography from self-occlusion
E.g. Building reconstruction in SAR images, terrain reconstruction in remote sensing
Robot Vision SS 2009 Matthias Rüther 37
Shape From Shadow
A static camera C observes a scene.
Light source L travels over the scene x, position of L is given by angle .
L and C are an infinite distance away (orthographic projection).
Shadowgram: binary function f(x, ), stating whether scene point x was shadowed at light position .
Robot Vision SS 2009 Matthias Rüther 38
Photometric Stereo
Multiple images, static camera, different illumination directions
At least three images
Known illumination direction
Known reflection model (Lambert)
Object may be textured
Robot Vision SS 2009 Matthias Rüther 39
REFLECTANCE MODELS
albedo Diffusealbedo
Specularalbedo
PHONG MODEL
L = E (aCOS bCOS )n
a=0.3, b=0.7, n=2 a=0.7, b=0.3, n=0.5
LAMBERTIAN MODEL
L = E COS
Robot Vision SS 2009 Matthias Rüther 40
Photometric Stereo
Reflection model
3 unknowns per pixel
)],(),,([),(),( yxqyxpRyxyxI
Albedo (reflectivity)
),(
),(
),(
yx
yxq
yxp
at least three illumination directions
Robot Vision SS 2009 Matthias Rüther 41
Photometric Stereo
1
1
1
1
( , ) ( , )
( , )
|| ||
i i
K
I x y R p q
I x y L n
L
L
L
n L I
L In
L I
surface normal
albedo
light direction
Robot Vision SS 2009 Matthias Rüther 42
Photometric Stereo: Example
From Forsyth & Ponce
Inputimages
Recoveredalbedo
Recoverednormals
Robot Vision SS 2009 Matthias Rüther 43
Range Finder
Range Finder principles:
• Runtime Range Finder
• Triangulation Range Finder
- Sender and receiver with known position, triangulation similar to stereo principle
1 image
Receiver/Sender positionDepth information2 dim + geometry => 3 dim
• Optical Range Finder• Ultrasound Range Finder
e.g.: • Spot Projectors• Moiré Range Finder• Structured Light Range Finder• Pattern (stripe) projection
Robot Vision SS 2009 Matthias Rüther 44
Runtime Range Finder
Determine sensor-object distance by measuring radiation runtime:
1. Sender (coherent light)2. Scanning Unit3. Receiver (light detector)4. Phase detector
Alt. Method: send light pulses => LIDAR (Light Radar), defense industry
Problem: generating pulses, measuring runtime (both very short)
Robot Vision SS 2009 Matthias Rüther 45
Ultrasound Range Finder
Used in commercial cameras (Autofocus Spot)
AdvantagesIndependent of sorrounding light,Slow speed of ray
Applications:
• Obstacle Detection: e.g.: “Car parking radar”• Level Measurement, silos, tanks, …
• Underwater ranging (sonar), ...
Typ. specifications:
• Range: 5cm to 1-5m
• Accuracy: +-3mm
Disadvantagescoarse resolution, Bad accuracy, Pointwise, scanner necessaryMultiple Reflection/Echoes