Range Imaging

From Calibration to Modeling

Norbert Pfeifer, Sajid Ghuffar, Willi Karel,

Camillo Ressl, Stefan Niedermayr

Institute of Photogrammetry and Remote Sensing

Vienna University of Technology

Why range cameras ?

Background 1 is Photogrammetry

• Passive brightness images, stills

• Object reconstruction by intersection of multiple rays to corresponding points

• Accuracy 1:10.000 – 1:100.000 (++)

• Resolution: diffraction limited

Background 2 is Laser Scanning

• Active ranging, hemispherical, sequential

• Object reconstruction from multiple scans for complex objects

• Accuracy 1:10.000 – 1:100.000

• Beam divergence near diffraction limit, sampling distance function of time

Range Imaging

• Active range imaging, in comparatively narrow FoV

• Accuracy 1:100 – 1:1.000

• High frame rate

Geodata collection

• Geometric parameters of urban furniture/vegetation

e.g.: tree height, DBH, height of first living branch

• Enrichment of existing GeoDB

• Object static, camera dynamic

• Advantage: Ease of use, automating different manual procedures

See also: B. Jutzi, KIT, Karlsruhe

Building interiors, 3D BIM

• „As used“ documentation of buildings: 3D models of rooms

• Ontop floorplan backbone

• Object static, camera dynamic

• Advantage: Very high automation possible, cost-effective

See also: J. Böhn, UCL, London

GEO domain applications 1/2

Fast geomorphological processes

• Landslide, mudslide, embankment failures

• Boundary values for better process understanding

• Object dynamic, camera static

• Advantage: range imaging as enabling technology

Traffic monitoring

• Cars, trains, etc.

• Objects dynamic, camera static (or on moving platform)

• Advntage: high reliability

See also: many others

GEO domain applications 2/2

From calibration to modeling – Presentation outline

Scattering compensation Laboratory

Calibration Laboratory

Orientation Real scenes under advantageous conditions

Modeling Real scenes under advantageous conditions

Overall aim (medium term goal)

Scattering compensation Lab measurements/not required(?)

Calibration model identification Lab measurements

Calibration parameters + orientation Real scenes, on-the-job

Modeling Real scenes

Scattering

Echo of the emitted optical signal is scattered to some extent over the sensor due to multiple reflections within the camera:

• i.e. between the lens, the optical filter, and the sensor

• “Lens Flare” in conventional photogrammetry

Distance distortions for far away objects can be up to 1m or greater

Ongoing instrumental developments, but effects also reported in new generation (Lichti, UCalgary)

Modelling the Point Spread Function

Experimental Setup: point like target in foreground scatters light over

darker background

Investigating the depedence of PSF on the parameters:

• Target size

• Target distance

• Target position

• Integration time

PSF from deconvolution

Spatially variant Point Spread Function

The intensity of scattering distortion changes with angle and distance

from the principal point

Shape of the scattering halo remains constant

[m]

Evaluating the PSF

Applying the correction model to real test scenes

Compensating for distance and amplitude distortion in a combine

deconvolution algorithm

Overall error reduced, but compensation partly overshootes

Calibration in Lab, handheld movies

Reference plane with control points

• Avoids scattering and object space multi path

• Very simple object model

Spatial resection only from amplitude image

Exterior and interior orienation estimated

for each frame

Target tracking fully automated

Significant intrinsic parameters

• Principal point coordinates

• Focal length

• Radial distortion of 3rd order

• Principle point depends on camera attitude

Calibration in Lab, handheld movies

Reference plane with control points

Spatial resection

Significant intrinsic parameters

Systematic error = range observation – reference range

Advantage: very large data volume (e.g. 6000 frames)

Disadvantage

• Low amplitudes for large distances

• Motion blur for large distances

Calibration in Lab – stills

Larger distances due to stills (max. range)

Experimental environment similar

• Control points cover larger volume

Target identification automated

• But computationally more intensive

Number of frames smaller (e.g. 850)

• But better SNR

amplitudes – observed range – range std.dev. – systematic range error

Calibration results

Harmonic range errors according to modulation wavelength (10cm span)

Hyperbolic-type range error for amplitudes (20cm span)

Range error as function of position in sensor plane (50cm span)

Range error increases with integration time (3cm span)

0 1 2 3 4 5 6 7 8-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

Observed dist. [m]

Der.

min

us o

bs.

dis

t. [

m]

Der. minus obs. dist. corr. 4 all but obs. dist. , corr. model

offset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs

0 1 2 3 4 5 6 7 80

2

4

6

8

10

12x 10

4

count

0 2000 4000 6000 8000 10000 12000 14000 16000 18000-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

Amplitude []

Der.

min

us o

bs.

dis

t. [

m]

Der. minus obs. dist. corr. 4 all but amplitude , corr. model

offset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs

0 2000 4000 6000 8000 10000 12000 14000 16000 180000

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5x 10

5

count

10 30 60 90 120 150 180 210 255-0.005

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

int. time []

der.

min

us o

bs.

dis

t. [

m]

Der. minus obs. dist. corr. 4 all but int. time , corr. model

offset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs

Orientation and Modeling

Example 1

• Handheld movie

• Focus on orienation

• Weak object modeling to support orientation

• Scene requirements: containing planes (for noise reduction)

Example 2

• Simulation of range cameras with improved accuracy

• Focus on modeling

• Orienatition solved as necessary task for modeling extended scenes

• Scene requirements: brightness differences in object

Orientation (and Modeling)

Scence contains large objects with simple geometric description (planes)

Objects found automatically by (video) segmentation

Subsequent scenes of movie transformed onto each other

• Scene n: described as set of planes

• Scene n+1: select „plane segment points“ close to planes of scene n

• Euclidean transformation of (n+1) to (n) [ ICP-type orientation with filters ]

Orientation (and Modeling)

Video Segmentation

Orientation (and Modeling)

Transformation of new to previous scene(s)

(Orientation and) Modeling

Video data acquisition including still scenes (tripod)

• Similarity between stills ~ 75% (rotation by ¾ FoV)

Noise reduction in still sequences by averaging

Point correspondence between images throught SIFT

Orientation

• Using direction and range observations

• Tracking points through multiple images

• Global orientation in the final stage

Modeling as subsequent step

(Orientation and) Modeling

Orientation fully automated

Amplitudes Object space in camera view

(Orientation and) Modeling

Modeling

Filter erroneous points (low intensity, very short distances, corona points)

Triangulate (Geomagic), smooth triangulation, automated hole filling

(Orientation and) Modeling

Residuals of

selected points

vs. smoothed

triangulation

up to 5cm

Conclusions 1/2

Scattering

• Instrumental developments reduce problems

• Because range observations are used directly,

complete removal remains a research topic

On the job calibration

• Feasible

• Lab environment

• Reduction of systematic errors by 50-90%

Orienation

• Feasible for hand held movies

• Scene requirements in examples (planes or markers or noise reduction)

Modeling

• Random noise reduction necessary

• Tight integration with calibration and orientation seems possible

EO device may become necessary (MEMS IMU, etc.)

Outlook 2/2

Ongoing projects

• Acclimatization (MESA and PMDTec)

• Warm-Up and other influcences (MESA, together with Lichti)

• Orientation without scence requirements

Dreams and Wishes

1. Increase device stability (weight!)

2. Increase insensitivty to background light: outdoor applications

3. Increase resolution

4. Increase maximum range

5. Increase accuracy