Research and development in photogrammetry ... - casm.ac.cncasm.ac.cn/dbfile/2019/06/21/8306/2.pdf · 60 years CASM: 1959 - 2019 International Symposium on Innovation, Integration

Institut für Photogrammetrie und GeoInformation

60 years CASM: 1959 - 2019

International Symposium on Innovation, Integration and Development of Surveying and Mapping

Research and development in

photogrammetry and remote sensing

- and the role of ISPRS

Christian Heipke

IPI - Institut für Photogrammetrie und GeoInformation,

Leibniz Universität Hannover

President, ISPRS


Greetings from ISPRS Council …

… and the whole ISPRS community


The International Society for

Photogrammetry and

Remote Sensing - ISPRS

• … an international NGO with a focus on

– science and development

• in photogrammetry, remote sensing, spatial information

– cooperation between all relevant stakeholders

• academia, private industry, government, end users

– truly global cooperation

• education, technology transfer, capacity building

• more than 100 years old

• approx. 180 institutional members


Table of contents

• Introduction

• Trends in research and development

• Deep learning

• The role of ISPRS


• Globally increasing population – in particular in emerging countries and coastal areas

– shrinking population in some developed areas, aging society

• Increasing globalisation – mobility of people, markets, capital, education, information, …

– digitisation

• Global companies entered the scene

• Changing communication – request for 24/7 availability, real-time response

– social media, a totally connected world

• Global change (only ONE earth, no plan[et] B …) – „Fridays for Future“

– 2030 UN Agenda and SDGs, Sendai Framework, Paris Agreement

Social and political frame


UN Sustainable Development Goals

… Earth Observation supports virtually all of them


Global ICT trends

Digitisation Mobility

Autonomous

driving

Internet of things,

AI

adapted from Volker Liebig, ESA 2016

Industry 4.0

SDG goals

Block chain Quantum computing


A connected world – location is key

• location uniquely ties items

together, in 2D and 3D, and

thus unlocks value in other‘s

data (N. Clifford, then CEO

Ordnance Survey, 2016)

• we are part of a more

complex, connected world,

not an isolated island

• interoperability

• crowd sourcing

BIM


Trends in research and

development


Principle of photogrammetry & remote sensing

• determination of characteristics

of the electromagnetic radiation

of a given wave length, reflected

or emitted from some surface

• energy, phase, polarisation, signal

form, travel time

• derivation of object and surface

characteristics from these

measurements

• geometry: position, size, shape

• radiometry: orthophoto mosaic

• semantics: object-ID, attributes


• new and better sensors, new platforms

– UAV, mobile mapping

– NewSpace: small COTS satellites, swarms, space video, …

• sensor and data fusion

– images + point clouds, maps, VGI, text, voice, …

• multi-temporal data analysis

– change detection, update

– process monitoring (data continuity !)

• real-time applications: traffic, environment, security, …

– robotics for mapping / mapping for robotics

Research and development trends

big data – machine / deep learning (data mining) big data, no alternative to automation


Sentinel-1A mosaic of Europe

using 400 scenes collected between May and July 2015,

contains modified Copernicus Sentinel data

© (2015) ESA/Array SC Canada Sentinel-2A cloudfree mosaic of Africa

using almost 7000 scenes collected between Dec 2015 and April 2016

contains modified Copernicus Sentinel data

© (2016) ESA/Brockmann Consult, GER/ Uni. Catholique de Louvain, BEL


Satellite constellations

• Planet: “The whole Earth in high resolution every day”

• standard cube sat’s (nano-sat’s), 10*10*30 cm3, 4 kg

• 3 – 5 m GSD, 4 bands, design constantly adapted

• flock of 28 sat’s. deployed from ISS (in 2014/2016, …)

• Feb. 2017: 88 doves (flock 3p) launched with Indian PSLV rocket

• currently 150+ sat’s in orbit (incl. 5 RE + 13 Skybox sat.)


Increased data volume © Bamler, 2018


Deep learning - the answer to automation needs ?


YOLO: You Only Look Once (version 2) – a Convolutional Neural Network (CNN)

for object detection

J. Redmon, S. Divvalay, R. Girshick, A. Farhadiy, Uni. Washington, 2016


AI / ML / DL

• AI / DL (“data science”) recently caught lots of attention in

science and society alike – big data

– „2nd best for everything“

– hardware developments (in part. GPU)

– abundant (training) data

• one of 10 breakthrough techno-

logies in MIT Tech. Review 2013

• recently pushed by Internet companies – Google, Facebook, Microsoft, Baidu, …

• said to rival (even resemble?) human cognitive ability

– victory of AlphaGo software against Go champion Lee Sedol (Oct.’17)

– „super human“ performance in computer vision challenges

Artificial

Intelligence


Deep learning - what is it, what is new?

• traditional classification:

Cleverly-Designed Features

Input Data ML model

most of the “heavy lifting” here, final performance only as good as feature set

… “learning” an input-output mapping from examples by machines (supervised classification)

• deep learning:

features and model learned together, mutually reinforcing each other

Features Input Data model Deep Learning


Development of deep learning

• Cybernetics („the perceptron“) – f: step function, linear decision boundary

– McCulloch, Pitts, 1943, Rosenblatt, 1958/62

• Connectionism, ANN, MLPs – f: sigmoid/tanh function, non-linear decision boundary

distributed representation, back propagation for training

– Rumelhart et al., 1986, Hinton et al., 1986, LeCun, 1987

• Deep learning – learn representation (features)

– learn from actions (e.g. to write SW)

– fast hardware (GPU), vast amount of training data:

efficient training of networks with „many“ layers

– Hinton et al., 2006

– Krizhevsky et al., 2012: CNN wins ILSVRC, decreasing the error by nearly 50%

.

.

.

x1

x2 x3

xn bj

S f

wj1

wjn

wj3

wj2 aj

Input Neuron j Output

wjb


CNN: Convolutional neural networks

• convolution

• (max-)pooling

• activation function e.g. ReLU(x) =

max(0,x)

7 5 9 8

5 0 3 1

0 4 7 3

8 6 6 2

9 8

7 4

• for regularly structured data

• repeated execution of those steps

based on “some“ architecture

• training of hierarchical feature

representation (filter coefficients)

by back propagation

• classification: one label per patch


AlexNet architecture (2012)

Krizhevsky, A., Sutskever, I., Hinton, G. E., 2012: ImageNet Classification with Deep

Convolutional Neural Networks, NIPS 2012.

• Goal: Classification of entire (small) images

predict one class label per image

ImageNet Large-Scale Visual Recognition Challenge (ILSVRC):

1.2 million training images, 1000 classes

5 convolutional layers, two FC layers, 60 Mio. parameters

with AlexNet, CNN really took off (16.4% top-5 error)

https://www.cs.toronto.edu/~kriz/imagenet_classification_with_deep_convolutional.pdf





Classification (CV terminology), examples


Success story of DL/CNN in computer vision

ImageNet Large Scale Visual Recognition Comp. (INLSVRC):

• 1,2 Mio test images, 1000 classes

• one class label per image

• ever increasing no. of layers


Other ILSVRC tasks solved using CNN

© Fei Fei Li et al., Stanford CS 231, 2017

good

semantic segmentation (CV) = pixel-level classification (Photo/RS)


Computer vision data discussed so far

from:

GoogLe

Net,

2014

https://ai.googleblog.com/2014/09/building-deeper-understanding-of-images.html

https://2.bp.blogspot.com/-TDP7iU5MRSo/VAjer5jrPzI/AAAAAAAAAV0/xk3lWXsO2xQ/s1600/ILSVRC2012_val_00021791.PNG

https://2.bp.blogspot.com/-qiLxaXSfcp8/VAjllBsNOhI/AAAAAAAAAWM/fIPPugiUWfQ/s1600/1XNYvfGqsv.png

https://2.bp.blogspot.com/-R7rgVNYKYM8/VAjmduy-VqI/AAAAAAAAAWU/TzKcFhfAfXc/s1600/ojbE4A2LeM.png


Typical remote sensing data


Photogrammetry, remote sensing - CNNs

• bird‘s eye view (sky is not „up“), geolocated + known GSD

• large images, complex scenes, multiple objects

• multi-modal information (spectral bands, 3D point clouds,

maps, social media data, …)

• monitoring of geo-processes, time-series processing,

spatio-temporal-spectral modelling

• prior knowledge (in part. of observed processes) essential

• geo-/bio-parameters vs. classification labels

• shortage of training data, but outdated DB available

• global challenges -> global transferability of approaches

• more stringent quality requirements (geom., semantics)

Institut für Photogrammetrie und GeoInformation 28

Example I: Road extraction / refinement

using CNNs

– first CNN application using

aerial images

– binary classification

(road / no road)

– training data: OSM

• incomplete

• partly with geometric errors

– models for errors in training data, learning of the related model

parameters

Mnih & Hinton, 2012

Mnih V., Hinton G, 2012: Learning to label aerial images from noisy data. In: McCallum, Roweis

(eds.), ICML 2012

Mnih & Hinton, 2012


Example II: land cover / land use update

DSM

CNN-based land cover

classification

CNN-based land use

classification

WO

RK

LF

LO

W

DOP DTM

Building

Sealed surface

Bare soil

Grass

Tree

Water

Rails

Car

Others

Land

cover

DTM

Settlement

Road

Railway

Water

Agriculture

Forest

Others

GIS

© Yang et al., 2018 - IPI

Yang C., Rottensteiner F., Heipke C., 2018: Classification of land cover and land use based on

convolutional neural networks. ISPRS Annals of the Ph, RS IV-3, 251-258.


Ensemble classifier: RGB, IR, nDSM, SkipNet architecture

training from scratch incl. data augmentation

Land cover classification (sem. seg.)

convolution block max-

pooling

upsample skip

connection

softmax


Land use update

Network architecture

predict the class for an input image of 256 x 256

require the generated patches scaled to 256 x 256

large objects (e.g. roads) subdivided and re-scaled

64 96 128 256 2 x 256 2 x 256

mask RGB/IR/nDSM land cover

posteriors

…

convolution block max-

pooling

average-

pooling

ROI

location

softmax


Results, land cover

Datasets Hameln (12 km2), Schleswig, 8 classes

Data CRF-Approach1 CNN-Approach

OA [%] Av. F1 [%] OA [%] Av. F1 [%]

Hameln 83.7 66.7 89.1 81.8

Schleswig 82.5 64.6 87.3 79.3

Dataset Mecklenburg-Vorpommern, 10 classes

Data CNN-Approach

OA [%] avg. F1 [%]

MV 83.5 79.7 © Yang et al., 2019 - IPI

1 Albert L., Rottensteiner F., Heipke C., 2017: A higher order conditional random field model for

simultaneous classification of land cover and land use. JPRS (130) 63-80.


Results, land cover

Input Our result Groundtruth

Building

Asphalt

Soil

Grass

Tree

Water

Car

Others


ISPRS Semantic Labelling Challenge

www2.isprs.org/commissions/comm3/wg4/tests.html

Institut für Photogrammetrie und GeoInformation 36

Example III: Dense matching using CNNs

Kang J., Chen L., Deng F., Heipke C., 2019: Encoder-Decoder network for local structure

preserving stereo matching Illumination. Proc. DGPF, Vol. 28

left image gt/right image Our baseline

Zbontar J, LeCun Y, 2015: Computing the Stereo Matching Cost with a Convolutional Neural

Network, CVPR, pp. 1592-1599, DOI: 10.1109/CVPR.2015.7298767

https://arxiv.org/ct?url=http://dx.doi.org/10.1109/CVPR.2015.7298767&v=c4cdaed3


Conclusions - evaluation

• many other examples in the last few years (see e.g.

review Zhu et al., 197 reference entries)

– more flexible wrt. different object characteristics

– strength: learning features, classifier not so important

– key to good performance: network depth, no. of training data

– has outperformed just about all classical algorithms by a

large margin, provided enough training data is available

– is about to do the same in geometric tasks

• no expert-free totally automatic processing chain, but

integration with human capabilities

Zhu X., Tuia D., Mou L., Xia G., Zhang L., Xu F., Fraundorfer F., 2017. Deep Learning in Remote

Sensing: A comprehensive review and list of resources. IEEE GRSS Magazine 5(4): 8-36.



• Open-source CNN implementations available Tensorflow (Google): https://www.tensorflow.org

CAFFE2 (Facebook) https://caffe2.ai/

…

• CNN has arrived in the commercial world

– Descartes Lab, https://www.descarteslabs.com/

– Simularity, simularity.com/solutions/satellite-anomaly-detection/

– Orbital Insight, https://orbitalinsight.com/

– spire, https://spire.com/

– NVIDIA (computer games!), https://www.nvidia.com/

– …

https://www.tensorflow.org/

https://caffe2.ai/

https://www.descarteslabs.com/

https://simularity.com/solutions/satellite-anomaly-detection/





https://orbitalinsight.com/

https://orbitalinsight.com/

https://spire.com/

https://spire.com/

https://www.nvidia.com/



• CNN is not magic - fundamentally a classifier – based on correlation between data sets

– an enormous amount of unknowns to be estimated

• lots of training data, long training times (days … weeks)

– sensitive to

• overfitting (“curse of dimensionality”)

• non-rep. training data (incorrect / biased / unbalanced / …)

• user choices, incl. hyper-parameters

– network architecture, (non-lin.) activation function

– design of loss function (function to be optimised)

– training parameters (weight initialisation, learning rate, drop

out rate, “momentum”, batch size, regularisation, …)

– generalisation and prediction capabilities unclear

• the system can’t learn what it never saw


The G

uard

ian,

Int.

Editio

n,

July

1st, 2

017


Conclusions – to do’s

• integrate prior knowledge, e.g. physical models

– don’t try to learn what we already know (e.g. laws of nature)

• CNNs for point clouds and other irregularly rep. data

• tackle training data shortage

– weakly, semi-, unsupervised learning, reinforcement learning

– data augmentation and simulation

– deal with errors in training data, e.g. when using outdated DB

– combination with crowd sourcing and social media data

– investigate limits of pre-training + fine tuning

• pay attention to geometric accuracy of object delineation

• sequential data, time series processing - RNNs

– streaming (big) data, online CNN learning


Special thanks to

Prof. Franz Rottensteiner

Chair, IPI research group

Photogrammetric image analysis

Chen Lin, Max Coenen, Philo Humberg, Uyen Nguyen, Dennis Wittich, Chun Yang, Junhua Kang

Documents

Research and development in photogrammetry ... - casm.ac.cncasm.ac.cn/dbfile/2019/06/21/8306/2.pdf · 60 years CASM: 1959 - 2019 International Symposium on Innovation, Integration