Transcript
Page 1: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 1

ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction

Matthias Rüther

Page 2: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 2

Overview

Shape from X techniques

– Structured light– Plane Sweep Stereo– Global optimization methods– Shape from (de)focus– Specularities– Shading, Photometric Stereo– Alternative methods

Page 3: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 3

Structured Light Methods

In principle same as multi-view/stereo

Project artificial pattern on the object

Pattern alleviates the correspondence problem

Variants:– Laser Pattern (point, line)

– Structured projector pattern (several lines, pattern sequence)

– Random projector pattern

Page 4: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 4

Structured Light Range Finder

1. Sender (projects plane)2. Receiver (CCD Camera)

X- directionGeometry Z- direction Sensor image

Page 5: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 5

1 plane -> 1 object profile

Object motion by conveyor band:=> synchronization: measure distance along conveyor=> y-accuracy determined by distance measurement

Scanning Units (e.g.: rotating mirror) are rare (accurate measurement of mirror motion is hard, small inaccuracy there -> large inaccuracy in geometry

Move the sensor: e.g. railways: sensor in wagon coupled to speed measurement

To get a 3D profile:• Move the object• Scanning Unit for projected plane• Move the Sensor

Page 6: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 6

Page 7: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 7

Commercially Available

Page 8: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 8

CAESARTM

Civilian American and European Surface Anthropometry Resource Project—CAESARTM

– 2400 Male/Female Americans

– 2000 Male/Female Europeans

Page 9: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 9

Problems

Occlusions

Sharpness and Contrast

Speckle noise

Page 10: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 10

Gleichzeitige Projektion mehrerer Lichtschnitte

Anstatt einer Lichtebene werden mehrere Lichtebenen auf das Objekt projeziert, umdie Anzahl der aufzunehmenden Bilder zu reduzieren.

Entfernungsberechnung:wie mit einer Lichtebene, jedochmuß jeder Lichtstreifen im Bildeindeutig identifizierbar sein.Problem: Aufgrund von Verdeckungen sind einzelne Streifen teilweise oder gar nicht im Kamerabild sichtbar -> keine eindeutige Identifikation der LichtstreifenAnwendung: Glattheitsüberprüfung bei planaren Oberflächen ohne Tiefenwertberechnung.

Page 11: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 11

Pattern projection

CameraCamera: IMAG

CCD, Res:750x590,

f:16 mm

ProjectorProjector: Liquid Crystal Display (LCD 640), f: 200mm, Distance to object plane: 120cm

Projected light stripes

Range Image

Page 12: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 12

Projector

Lamp

Lens system

LCD - Shutter

Pattern structure

Example

Focusing lens (e.g.: 150mm)

Line projector (z.b: LCD-640)

Page 13: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 13

Tiefenberechnung für Streifenprojektor

1) Unterschiedlich breite Lichtstreifen werden zeitlich aufeinanderfolgend in die Szene projiziert und von der Kamera aufgenommen.2) Für jede Aufnahme wird für jeden Bildpunkt festgestellt, ob dieser beleuchtet wird oder nicht.3) Diese Information wird für jeden Bildpunkt und für jede Aufnahme im sog. Bit-Plane Stack abgespeichert.

Verschiedene Lichtstreifen sind notwendig, um für jeden Bildpunkt einen zugehörigen Lichtstreifen festzustellen zu können.

Durch die zeitliche Abfolge der Aufnahmen wird es ermöglicht, daß jeder Lichtstreifen im Kamerabild identifiziert wird.

4) Findet man im Bit-Plane Stack für einen Bildpunkt die Information, daß er bei den Aufnahmen z.B. hell, dunkel, dunkel, hell war (Code 1001), dann folgt daraus, daß dieser Bildpunkt vom vierten Lichtstreifen beleuchtet wird.5) eindeutige Zuordnung „Lichtstreifen – Bildpunkt“ möglich

Page 14: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 14

Coded Light + Phase Shift

Binary code is limited to pixel accuracy (at most).

Increase accuracy by projecting sine wave and measuring phase shift between projected and captured pattern.

Page 15: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 15

Joaquim Salvi,

Pattern codification strategies in structured light systems

Page 16: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 16

Random Texture Projection

Page 17: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 17

Moiré Range Finder

Project line structure, observe line structure through a grid

1. Sender (Projektor mit Linien)2. Receiver (CCD Camera with line filter)

Problem: identification of line ordering possible but hard, unsharp lines => inaccurate results

Moiré ImageMoiré Pattern

Page 18: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 18

Moiré Range Finder

Y. Suenaga, 3D Measurements for Computer Animation

Page 19: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 19

Plane Sweep Stereo

Sweep family of planes through volume

– each plane defines an image composite homography

input imageinput image

projectiveprojective re-sampling of ( re-sampling of (X,Y,ZX,Y,Z))

Richard Szeliski, IBMR 1998

Page 20: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 20

Plane Sweep Stereo

For each depth plane– compute composite (mosaic) image — mean

– compute error image — variance

– convert to confidence and aggregate spatially

Select winning depth at each pixel

Richard Szeliski, IBMR 1998

Page 21: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 21

Voxel Coloring

Page 22: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 22

Voxel Coloring / Space Carving

S={} /* initial set of colored voxels is empty

for i = 1 to r do /* traverse each of r layers

foreach V in the ith layer of voxels do

project V into all images where V is visible

if sufficient correlation of the pixel colors then

add V to S

Photorealistic Scene Reconstruction by Voxel Coloring S. M. Seitz and C. R. Dyer, Proc. Computer Vision and Pattern Recognition Conf., 1997, 1067-1073

Page 23: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 23

Shape From Focus

Recover shape of surfaces from limitied depth of view.– Requires visibly rough surfaces– Typical application: optical microscopy

Page 24: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 24

Shape From Focus

Visibly Rough Surfaces– Optical roughness: the smallest spatial variations are much larger than the

wavelength of incident electromagnetic wave.

– Visible roughness: smallest spatial variations are comparable to viewing area of discrete elements (pixels).

– Magnification:• Multi-facet level: w1 >> wf, smooth texture

• Facet level: w2 ~= wf, rough texture

Page 25: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 25

Shape From Focus

Focused / Defocused images

– Focused:

– Defocused: object point is mapped to spot with radius

=> defocusing is equivalent to convolution with low pass kernel (pillbox function)

fio

111

i

Rr

otherwise

ryxifryxp

yxIyxpyxI fd

0

1),(

),(*),(),(

2222

Page 26: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 26

Shape From Focus

Changing Focus

– Displacing the sensor: changes sharp region, magnification and brightness

– Moving the lens: changes sharp region, magnification and brightness

– Moving the object: changes sharp region only

=> Object is moved in front of static camera

Page 27: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 27

Shape From Focus

Overview:– At facet level magnification, rough

surfaces give texture-rich images– A defocused image is equivalent to a

low-passed image

– As S moves towards focused plane, its focus increases. When S is best focused,

– Challenges:• How to measure focus?

• How to find best focus from finite number of measurements?

ddd fs

Page 28: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 28

Shape From Focus

Focus measure operator– Purpose:

• respond to high frequency variations in image intensity within a small image area

• produce maximum response when image area is perfectly focused

– Possible solution: • determine high frequency content using Fourier transform (slow)

– Alternative:• Laplacian operator (problem with elimination)

• Modified Laplacian

Page 29: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 29

Shape From Focus

• Sum Modified Laplacian

• Tenengrad Focus Measure

• Alternatives: variance of intensities, variance of gradients

101

202

101

xS

121

000

121

yS

M

m

N

n

yx

nmSITEN

nmSnmSnmS

2

22

)],([)(

)],([)],([),(

INxM … local intensity function (image window)

Page 30: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 30

Shape From Focus

Example

Infinite DOF DEM

Page 31: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 31

Shape From Focus

Sampling the focus measure function– Consider a single image point (x,y)

– Focus measure F is function of depth d: F(d)

– Goal find Fpeak from finite number of samples F1…F8

Page 32: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 32

Shape from focus

Sampling the focus measure function– Possibility1: find highest discrete sample

Page 33: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 33

Shape from focus

Sampling the focus measure function– Possibility2: Gaussian interpolation

Fit Gauss function to three strongest samples

Page 34: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 34

Shape from Specularity

Suitable for highly reflective Surfaces

Specular Reflection map of a single point source forms a sharp peak (Specular model, Phong model)

Page 35: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 35

Shape from Specularity

Principle:– If a reflection is seen by the camera

and the position of the point source is known, the surface normal can be determined.

– => use several point sources with known position: structured highlight inspection

Page 36: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 36

Shape From Shadow

Also: Shape from Darkness

Reconstruct Surface Topography from self-occlusion

E.g. Building reconstruction in SAR images, terrain reconstruction in remote sensing

Page 37: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 37

Shape From Shadow

A static camera C observes a scene.

Light source L travels over the scene x, position of L is given by angle .

L and C are an infinite distance away (orthographic projection).

Shadowgram: binary function f(x, ), stating whether scene point x was shadowed at light position .

Page 38: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 38

Photometric Stereo

Multiple images, static camera, different illumination directions

At least three images

Known illumination direction

Known reflection model (Lambert)

Object may be textured

Page 39: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 39

REFLECTANCE MODELS

albedo Diffusealbedo

Specularalbedo

PHONG MODEL

L = E (aCOS bCOS )n

a=0.3, b=0.7, n=2 a=0.7, b=0.3, n=0.5

LAMBERTIAN MODEL

L = E COS

Page 40: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 40

Photometric Stereo

Reflection model

3 unknowns per pixel

)],(),,([),(),( yxqyxpRyxyxI

Albedo (reflectivity)

),(

),(

),(

yx

yxq

yxp

at least three illumination directions

Page 41: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 41

Photometric Stereo

1

1

1

1

( , ) ( , )

( , )

|| ||

i i

K

I x y R p q

I x y L n

L

L

L

n L I

L In

L I

surface normal

albedo

light direction

Page 42: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 42

Photometric Stereo: Example

From Forsyth & Ponce

Inputimages

Recoveredalbedo

Recoverednormals

Page 43: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 43

Range Finder

Range Finder principles:

• Runtime Range Finder

• Triangulation Range Finder

- Sender and receiver with known position, triangulation similar to stereo principle

1 image

Receiver/Sender positionDepth information2 dim + geometry => 3 dim

• Optical Range Finder• Ultrasound Range Finder

e.g.: • Spot Projectors• Moiré Range Finder• Structured Light Range Finder• Pattern (stripe) projection

Page 44: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 44

Runtime Range Finder

Determine sensor-object distance by measuring radiation runtime:

1. Sender (coherent light)2. Scanning Unit3. Receiver (light detector)4. Phase detector

Alt. Method: send light pulses => LIDAR (Light Radar), defense industry

Problem: generating pulses, measuring runtime (both very short)

Page 45: Robot Vision SS 2009 Matthias Rüther 1 ROBOT VISION Lesson 7: State of the Art in 3D Reconstruction Matthias Rüther

Robot Vision SS 2009 Matthias Rüther 45

Ultrasound Range Finder

Used in commercial cameras (Autofocus Spot)

AdvantagesIndependent of sorrounding light,Slow speed of ray

Applications:

• Obstacle Detection: e.g.: “Car parking radar”• Level Measurement, silos, tanks, …

• Underwater ranging (sonar), ...

Typ. specifications:

• Range: 5cm to 1-5m

• Accuracy: +-3mm

Disadvantagescoarse resolution, Bad accuracy, Pointwise, scanner necessaryMultiple Reflection/Echoes


Recommended