Trentino Research System PHOTOGRAMMETRY FOR 3D …€¦ · Trentino Research System Bruno Kessler Foundation (FBK) ... + helicopter –balloon UAV images + terrestrial images 1

7/19/2013

1

Fabio Remondino - PHOTOGRAMMETRY FOR 3D MODELING AND MAPPING

PHOTOGRAMMETRY FOR

3D MODELING AND MAPPING

- STATE OF THE ART AND OUTLOOK -

Fabio REMONDINO

3D Optical Metrology (3DOM)

Bruno Kessler Foundation (FBK)

Trento, Italy

Email: [email protected]

http://3dom.fbk.eu Contents:

- 3D surveying VS 3D modeling

- Photogrammetry

- UAV / drones

- Recent developments

- Examples Fabio Remondino - PHOTOGRAMMETRY FOR 3D MODELING AND MAPPING

Autonomous Province of Trento (PAT)

University of Trento

Edmund Mach Foundation (FEM)

Trentino Research System

Bruno Kessler Foundation

(FBK)



- Born in 1962 as public research center of the Autonomous Province of Trento (Italy)

- Research primarily in the fields of ICT, Materials & Sensors

- More than 350 researchers and developers, 7 laboratories, 26 research units, 11 spin-offs

- Research units and labs: Biosensors, Bioelectronics, Optical sensors, Radiation detectors,

Renewable energies, Intelligent interfaces, Predictive models, Technologies for Vision, 3D

Optical Metrology, etc.

- Hosting BSc, MSc and PhD students

- Master and PhD programs jointly held with international universities



3DOM

7/19/2013

2


FBK Spin-offs



3DOM unit

6 researchers (FBK), 2 Post-doc (1 FBK, 1 NSF), 1 PhD (UniSalamanca)

Hosting BSc, MSc and PhD students (2010: 4 guests; 2011: 9 guest; 2012: 7 guests, 2013: 2+4)

Exchange agreement with int. universities for hosting student (ENSG Paris, CGT Siena, etc.)

Funding from local (PAT, private sector, etc.) and international grants (EU, BMBF, private

sector, etc.)

Applied research work in the fields of Photogrammetry, Laser scanning and Remote Sensing,

3D Mapping, Geomatics

Organization of int. events and summerschools for knowledge and technology transfer

Collaborations with national and international universities and private companies


Topics:

Automated 3D reconstruction from terrestrial images

Geometric processing of aerial (satellite, airborne and UAV) images

Heating loss monitoring and PV mapping

Heritage 3D recording

Sensor integration

“Hidden buildings” identification

Characterization of passive and active sensors

3D surveying and multi-spectral analysis of paintings

Human posture analyses

Identification of critical areas along power lines

Environmental monitoring & change detection

Human body modeling

Robotics applications

R&D at 3DOM-FBK

Fabio Remondino - PHOTOGRAMMETRY FOR 3D MODELING AND MAPPING 8

Contents:

- 3D surveying VS 3D modeling

- Photogrammetry

- UAV / drones

- Recent developments

- Examples

PHOTOGRAMMETRY FOR 3D MODELING AND MAPPING

- STATE OF THE ART AND OUTLOOK-

7/19/2013

3


3D Surveying & Modeling

Surveying: reality-based / metric recording of unstructured 3D data (point clouds) of existing

structures → real measurements (“reality-based”) on the field (photogrammetry, laser

scanning, etc.)

Modeling: generation of structured 3D data (mesh, TIN) from the surveyed data

3D Modeling is a chain of lossy procedures and data transformations performed to

derive new products

“3D Modeling” is more common in terrestrial application, “Mapping” is the general term in

aerial/satellite-based applications

Data collection Data processing 3D product visualization,

analysis, interpretation, etc.

3D MODELING


• Image-based techniques (passive sensors)

• Surveying methods

• Computer graphics (3DStudioMax, Maya, Sketch-up, etc.)

- no real measurements

Realit

y-b

ased m

odelin

g

• Procedural modeling

- photogrammetry, computer vision, shape from X, etc.

→ dense or sparse 3D points (point cloud)

- laser scanners, stripe projection systems, radar, etc.

→ TOF or triangulation measurement principle

→ dense point clouds

- GNSS, Total stations, maps, etc.

→ sparse points, low density, low resolution model,

time consuming

• Range-based techniques (optical active sensors)

10

Techniques for 3D data acquisition


Image-based techniques

(Passive sensor)

Silhouette extraction

Computer Vision

Focus/de-focus

Shape from texture

Shape from edges

Shape from shading

Image-based techniques

Photogrammetry

Remote Sensing


Analogue Analytical Digital

Remote Sensing, Computer Vision

The technique to derive reliable & precise measurements by means of photographs / images

The art, science and technology of obtaining reliable information about physical

objects and the environment through the process of recording, measuring, and

interpreting photographic images and pattern of electromagnetic radiant energy

and other phenomena

The art of turning images into 3D models

Applications: from Mars to nanotechnologies, from small to large scale, from m to micron

Photogrammetry:

INPUT:

• Satellite images

• Analog aerial-photos

• Digital airborne images

• Terrestrial images (digital/analog)

• GPS data

• …

OUTPUT:

• Digital Terrain/Surface Models (DTM/DSM)

• Ortho-images

• Maps

• (Textured) 3D models

• GIS layers

• …

12

Photogrammetry

../../../video/graphitech_computergraphics1.mp4

../Users/aa/Videos/procedural_modeling_of_buildings.avi

../Users/aa/Videos/tombe_etrusche_overview.wmv

../Users/aa/Videos/3cime_modello3D.wmv

7/19/2013

4


+ Computer vision: image-based techniques with automation as first goal

+ Remote Sensing: technique mainly for image interpretation and classification and NOT 3D

geometric reconstructions

PHOTOGRAMMETRY is not…

13

+ image-based techniques aiming at accurate 3D reconstruction with reliable procedures

PHOTOGRAMMETRY is

Photogrammetry


+ satellite (optical) images

+ airborne images

+ helicopter – balloon – UAV images

+ terrestrial images

+ underwater images

600-800 km

1-10 m

alt

itu

de

14

Photogrammetry with…


- Space intersection of rays passing through homologous points in multiple images

- At least 2 images / photos

- Generation of 3D coordinates (object) from 2D coordinates (image)

- Knowledge of GCPs or distance to get metric results (not like this for 3D scanning!!)

- Example: aerial block

How does it work?

15

Photogrammetric principle


- 2D images need a mathematical model to convert

the image measurements into 3D data

- At least 2 images to derive 3D information

- Collinearity principle

- Camera calibration to correct systematic errors due

to lens distortion

X

Y

Z

OA OB

image A image B

object

Reference system

terrestrial example

16

How does it work?

Photogrammetric principle

7/19/2013

5


Image data acquisition

Image pre-processing

Calibration and orientation

Measurements and & 3D point cloud generation

Surface generation and Texture

Mapping

Visualization, GIS products, replicas, inspection, virtual

restoration, etc.

Possibility to derive, according to the project requirements and scene, sparse or dense 3D

point clouds, geometric features of textured 3D models, etc.

17

Photogrammetric pipeline


LANDSCAPE & ARCHITECTURES 3D RECONSTRUCTION

Xochicalco, Mexico Pre-hispanic site, 700-800 AD

Digitally reconstructed from archive aerial images

Automated terrain reconstruction, interactive structure 3D modeling

18

Photogrammetric examples

Use of historical images!


• Lines and figures drawn in the desert (geoglyphs)

• Unknown motivation, many hypotheses (astronomy, religion, water)

Project @ ETH Zurich:

→ ca 400 aerial images, 1:7000 scale

→ Mapping and archaeological analysis of the geoglyphs

→ Digital preservation as of 1998

So far, ca 700 geoglyphs mapped / documented

The NASCA lines, Peru

19



7/19/2013

6


Virtual flight over the geoglyphs

3D Modeling and Visualization


21



GIS Visibility Analysis

- Visibility analysis -

GREEN AREA = areas from where it’s possible to see the RED DOT (e.g. the geoglyph)

- Are geoglyphs really only visible from air?

- Is intervisibility between geoglyphs higher than between arbitrary

points?


22



Photogrammetric Products

• First complete map of the region including the

geoglyphs

• Supports the archaeological fieldwork

• Physical 3D model, shown in museum of Palpa

• Virtual 3D model, used for archaeological

analysis and prospection


23



3D city modeling, Zurich, Switzerland

Example – 3D city modeling

3D city planning, Zurich, Switzerland

24


7/19/2013

7


Human movement reconstruction and analysis

(MOCAP - MOtion CAPture)

Example – MOCAP

25



Example – Deformation analyses Crash tests

Deformation analyses

26



Passive sensors (imaging)

Type: CCD, CMOS

Platforms: Satellite, aerial, UAV, mobile mapping, terrestrial, underwater rover

Trends / Developments:

1. Increasing geometric resolution (up to 60-80 Megapixel)

2. Sensor integration (Gigapixel / Panoramic camera)

3. Stereo cameras

Photogrammetric sensors & platforms


Great amount of terrestrial cameras providing image data

Industrial - SRL – consumer grade – mobile phones

Frame sensor or linear array (industrial app)

Prices ranging from 150 Eur to 20K Eur

Visible or IR or UV part of the spectrum

Type Relative accuracy

potential

Resolution

Amateur / consumer

cameras / mobile phones

ca 1 : 20 000 Up to 8-10 MPixel

Professional cameras ca 1 : 100 000 Up to 20 MPixel

Photogrammetric / Industrial

cameras

> 1 : 200 000 Up to 50 MPixel

+ Samsung mobile phone – 8 megapixel camera

+ Hasselblad – 60 megapixel camera

28

Photogrammetric terrestrial cameras

../Users/aa/Videos/mocap_motionanalysis.mpg

../video/MOCAPzizou.mov

../Users/aa/Videos/mocap_tigerwoods.mov

../../../../Users/aa/Videos/crash-test_1.mpeg

7/19/2013

8


UAV facts

Platforms: balloons, airship, kite, model helicopter (electric or ICE), fixed/rotary

wing aircraft, solar powered, etc.

Onboard sensors: navigation units (GPS/INS), digital camera or active sensors

(laser scanner, Kinect, etc.)

Terminology: Drone, Remotely Piloted Vehicle (RPV), Remotely Operated

Aircraft (ROA), Micro Aerial Vehicles (MAV), Remote Controlled (RC)

Helicopter, Model Helicopter, etc.

Regulations under creation

Point of interest for different communities: photogrammetry, surveying,

robotics, computer vision, artificial intelligence, space domain, archaeology,

geography, forestry, etc.

Different open-source and low-cost solutions (HW and SW level)

Problems and limitations are still existing: payload, endurance, instability,

direct geo-referencing, etc.

Photogrammetric platforms: UAV


• Bamiyan, 200 km N-O di Kabul

• Area rich of Buddhist monuments, in the middle of the ‘Silk Road’

• Destruction in 2000/2001

• Included in UNESCO World Heritage List since 2003 [http://whc.unesco.org]

• ETH work (started 2001): 3D landscape modeling, reconstruction of the façade and of the Buddha

statues and their empty niches

Example - 3D Reconstruction of the Big Buddha statue in Bamiyan, Afghanistan

30



Bamiyan, Afghanistan

Niche of the Great Buddha

Niche of the Small Buddha

31 Fabio Remondino - PHOTOGRAMMETRY FOR 3D MODELING AND MAPPING

Before March 2001:

- 53 m high

- tallest representation of a standing Buddha

- niche full of frescos

After March 2001:

- empty niche

- no more frescos

- risk of collapse


32


G:/fabio/pano/Bamian_mosaic_facade.mov

G:/fabio/video/explosion_compr.avi

7/19/2013

9


“The Buddhas are false idols that must be destroyed!!

… all we are breaking are just stones …” (Mullah Omar)

“First we destroyed the small statue. It was a woman.

Then we blew up her husband.” (Mullah Omar)


33



3D Modeling:

Internet, Tourist and Metric images


34



After March 2001 Before March 2001


35




36


../Users/aa/Videos/empty_video.avi

../Users/aa/Videos/buddha_video.avi

7/19/2013

10


Before explosion - real Actual situation - real


Actual situation - virtual

Before explosion - virtual

37



1:200

1:25


Physical replica

38



- = 60%

Is ANASTYLOSIS possible?

The leftover of the statue after the explosion are ca 60% of the original

→ Anastylosis not really possible


39



The Giant Buddhas:

www.giant-buddhas.com

Film production

and

virtual reality

August 2003 Virtual Reality


40


../../video/giant-buddhas_VR.avi

7/19/2013

11


Photogrammetry - Recent developments

Automated image orientation / Structure from Motion

SIFT, SURF, etc. operators for automated tie point extraction

Structure from Motion (SfM) & SLAM algorithms for bundle adjustment (low

reliability)

Open-source solutions: SBA, Bundler, APERO, etc.

Web-based tools: MS Photosynth, ARC3D, Autodesk 123D-Catch, etc.

Different solutions if a different order of the images is used

Not possible to use Ground Control Points in SfM tools => deformations

One-click tools not suitable and reliable for metric applications requiring

accurate results


Dataset name # img resolution (pixel) Focal length

(mm) image type

dim. L x W x H

(m) ground truth

NAVONA 92 4000x3000 6 terr. 50 x 250 x 15 calib. camera





- Block of ca 120 aerial images over a flat area

- SfM algorithm derived deformed results

SfM → bending / object deformation

Photogrammetric bundle adjustment




- Block of ca 600 aerial images (UAV, GSD: 3mm) over an archaeological area

- Required product: map @ 1:20 (!!!!)

Archaeological area of Ventimiglia

(ca 50x 50 m)

DSM with ground control

DSM with SfM

…SfM algorithm derived deformed results…


7/19/2013

12


Dense image matching

Correlate automatically 2 or more images and derive dense point clouds

Algorithms: Semi-Global Matching (SGM), patch-based or optimal flow or least

squares

Multi-image vs stereo-pairs

[Remondino, F., El-Hakim, S., Gruen, A., Zhang, L., 2008: Turning images into 3D models - Development and

performance analysis of image matching for detailed surface reconstruction of heritage objects. IEEE Signal

Processing Magazine, Vol. 25(4), pp. 55-65]



Dense image matching

Open-source solutions: MicMac, PMVS, etc.

Comparable geometric results as terrestrial laser scanning

[Pierrot-Deseilligny, M., De Luca, L., Remondino, F., 2011: Automated image-based procedures for accurate

artifacts 3D modeling and orthoimage generation. Proc. CIPA Symposium, Prague, Czech Republic]



Photogrammetric processing of panoramic images

Spherical photogrammetry for 3D restitutions

[Barazzetti, L., Fangi, G., Remondino, F., Scaioni, 2010: Automation in Multi-Image Spherical Photogrammetry

for 3D Architectural Reconstructions. Proc. VAST Symposium, Paris, France]



1 - Single image (“rectification”)

2 - Two station configurations

- convergent, large/short baseline

3 - Multi station configuration (image block)

48

Photogrammetric image configuration

7/19/2013

13


• High geometric accuracy

• Complete details

• (Automation)

• Photo-realism

• Low cost

• Portability

• Flexibility

49

Photogrammetric requirements


• Digital Terrain/Surface Models (DTM/DSM)

• Orthophotos

• 3D Models

• Cartography

• City Planning

• Industrial measurements

• Animations & Visualization

• 3D documentation and digital conservation

• Digital restoration

• Architectural drawings

• Physical reconstruction

• Virtual tourism

• Educational resources

• …

50

Photogrammetric products


Example – 3D modeling of a relief in Trento, Italy Ca 2 900 000 points

5 images (12M pixel), Buonconsiglio Castle, Trento, Italy

ETH matching software for surface reconstruction

3D comparison with laser scanner data – resulted standard deviation: ± 3 mm

ca 2 meters

std

3 m

m

51



Example – 3D modeling of a architectural monuments – Lalibella church, Ethiopia

Manual (interactive) reconstruction

52


../school_Volterra/ppt/ppt_varia/skyline_everest_map/everestmap.teh

../../../video/Otranto.avi

../../../video/meda.mpg

7/19/2013

14


Example – 3D modeling of a architectural monuments – Fontaine Saint-Jean

Automatic reconstruction

Image orientation results Derived 3D point cloud via image matching

Unwrapped ortho

53



Example – 3D modeling of a architectural monuments – Portail Saint-Trophime


Derived 3D point cloud via

image matching

orthophoto

54



Example – 3D modeling of a heritage bassrelief – Copan mask, Honduras

55 Automatic reconstruction



Example – 3D reconstruction of archaeological excavation area

- Pava area, S.Giovanni Val d’Arno, Siena (Italy)

- UAV platform to acquire aerial images

- Photogrammetric processing (ca 3 hours) to deliver a dense point cloud useful for excavation

monitoring, volume computation, orthophoto creation, etc.

Image mosaic of the acquired

(40) images

3D data

Ortho

56



Multi-temporal analyses

pons/copan/3D/final.obj

7/19/2013

15


Example – Automated 3D reconstruction (DSM) from aerial / satellite images

- Trento (Italy)

- Aerial images, 15 cm GSD

- Satellite images (WorldView, 50 cm GSD)

Satellite

Airborne

57




Example – Automated 3D reconstruction (DSM) from aerial / satellite images

58


DSM at 1.5 m geometric resolution

Ortho at 1 m resolution


DSM

BUILDINGS

Example – Feature extraction (buildings, maps, etc.) from DSM



Example – DSM for photovoltaic potential of building roofs (3DSolarWeb project)


[Wh/m2/day]

7/19/2013

16


Example – photovoltaic potential of building roofs (3DSolarWeb project)


- Geometric resolution: 50 cm

- Web-GIS application for Public Administration and citizens


Example – 4D analyses of landscape (MEM3D Project)


1944 1973 2009

[Nocerino, E., Menna, F., Remondino, F., 2012: Multi-temporal analysis of landscapes and urban areas. Int.

Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. 39(4), pp. 85-90. XXII

ISPRS Congress, Melbourne, Australia]


Example – 4D analyses of landscape (MEM3D Project)


Photogrammetric processing -> DSM generation -> orthoimage production -> comparison


Example – Monitoring electric power lines

Satellite

64


7/19/2013

17


Example - Monitoring of man-made structures (bridges) – X-Bridge project

- 3D surveying & modeling

- FEM analyses / simulations

- Stress analyses integrating x-ray technology



0.1 m 1 m 10 m 100 m 1 km 10 km 100 km 1000 km

10 Mil

1 Mil

100 000

10 000

1 000

100

10

1

Ob

ject

/ S

cen

e C

om

ple

xit

y [

po

ints

/ob

ject]

Object / Scene Size

Close-range

photogrammetry

and

terrestrial laser scanners

Aerial

photogrammetry

and LiDAR

Satellite

Remote Sensing

Tactile / CMM

Hand

measurements

Total stations

GNSS

Techniques for 3D Data Recording

UAV


0.1 m 1 m 10 m 100 m 1 km 10 km 100 km 1000 km

10 Mil

1 Mil

100 000

10 000

1 000

100

10

1

Ob

ject

/ S

cen

e C

om

ple

xit

y [

po

ints

/ob

ject]

Object / Scene Size

Close-range

photogrammetry

and

terrestrial laser scanners

Aerial

photogrammetry

and LiDAR

Satellite

Remote Sensing

Tactile / CMM

Hand

measurements

Total stations

GNSS

Techniques for 3D Data Recording

UAV

IMAGING

&

RANGING


Photogrammetry = passive method (passive sensors) = image-based method

Scanning = active method (active sensors) = range-based method

Image data acquisition

Image pre-processing

Calibration and orientation

Measurements and & 3D point

cloud generation

Surface generation, feature

extraction and texture mapping

Visualization, GIS products,

replicas, inspection, virtual

restoration, etc.

Range data acquisition

(3D point cloud)

Editing and alignment

Surface generation, feature

extraction and texture mapping

Visualization, GIS products,

replicas, inspection, virtual

restoration, etc.

IMAGING VS RANGING

1:8 1:10

7/19/2013

18


– Exist many and varied geomatics techniques for 3D surveying and modeling applications

• Several fundamental positioning approaches

• Selection of appropriate technique is not trivial

– No panacea: techniques used in combination / integrated fashion generate best results

Research &

surveying

Objects to be

recorded in 3D

Available data

and sources

Regional scale

Landscape

Topography

Sites

• Middle and high resolution satellite images

• Small scale aerial imagery

• Radar & LiDAR

• GNSS

Local scale

Sites

Architectures

Excavation layers

• Large scale aerial imagery (Airborne and UAV)

• Radar, magnetometry & LiDAR

• Terrestrial range data – Time of Flight

• Terrestrial images

• Total stations

• GNSS

Object scale Excavated artefacts

Objects

• Terrestrial images

• Terrestrial range data – Triangulation

GIS

69

INTEGRATION of techniques


http://www.tapenade.gamsau.archi.fr/


http://www.map.archi.fr/aibm/


http://www.heritage3d.org/

../../../../Users/aa/Videos/pompei_short.mpg

Documents

Trentino Research System PHOTOGRAMMETRY FOR 3D …€¦ · Trentino Research System Bruno Kessler Foundation (FBK) ... + helicopter –balloon UAV images + terrestrial images 1