Embed Size (px)
Citation preview
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Sensor FusionMulti-Sensor Data Fusion
Felix Riegler
8. Mai 2014
Felix Riegler 1/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Felix Riegler 2/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
1 Definition
2 Domains and properties
3 Examples
4 General data fusion methods
5 Stereo vision
6 Conclusion
Felix Riegler 3/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Definition
Felix Riegler 4/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Richardson and March - 1988
Fusion of Multisensor data.
Felix Riegler 5/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Hall - 1992
Multisensor data fusion seeks to combine data from multiplesensors to perform inferences that may not be possible from asingle sensor alone.
Felix Riegler 6/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Starr and Desforges - 1998
Data fusion is a process that combines data and knowledge fromdifferent sources with the aim of maximising the useful informationcontent, for improved reliability or discriminant capability, whileminimising the quantity of data ultimately retained.
Felix Riegler 7/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Breakdown
Input: sensor data from multiple sensors
Process: combining data
Goal: to get better and/or more reliable data
Felix Riegler 8/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Breakdown
Input: sensor data from multiple sensors
Process: combining data
Goal: to get better and/or more reliable data
Felix Riegler 8/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Breakdown
Input: sensor data from multiple sensors
Process: combining data
Goal: to get better and/or more reliable data
Felix Riegler 8/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Domains and properties
Felix Riegler 9/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Domains
Military
Localization/Detection
Navigation/Pathfinding
(Air-)Traffic control
Environment prediction
Robotics
....
Felix Riegler 10/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Domains
Military
Localization/Detection
Navigation/Pathfinding
(Air-)Traffic control
Environment prediction
Robotics
....
Felix Riegler 10/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Domains
Military
Localization/Detection
Navigation/Pathfinding
(Air-)Traffic control
Environment prediction
Robotics
....
Felix Riegler 10/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Domains
Military
Localization/Detection
Navigation/Pathfinding
(Air-)Traffic control
Environment prediction
Robotics
....
Felix Riegler 10/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Domains
Military
Localization/Detection
Navigation/Pathfinding
(Air-)Traffic control
Environment prediction
Robotics
....
Felix Riegler 10/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Domains
Military
Localization/Detection
Navigation/Pathfinding
(Air-)Traffic control
Environment prediction
Robotics
....
Felix Riegler 10/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Domains
Military
Localization/Detection
Navigation/Pathfinding
(Air-)Traffic control
Environment prediction
Robotics
....
Felix Riegler 10/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3 Properties
Source:
homogeneous
heterogeneous
Goal:
reliability
new data
Representation in a system:
blackboard
module
Felix Riegler 11/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Examples
Felix Riegler 12/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Radar
emits electromagnetic waves and detects reflections
surveillance radar
secondary or active radar
+ fire control radar
heterogeneous; reliability and new data; blackboard
Felix Riegler 13/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Radar
emits electromagnetic waves and detects reflections
surveillance radar
secondary or active radar
+ fire control radar
heterogeneous; reliability and new data; blackboard
Felix Riegler 13/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Radar
emits electromagnetic waves and detects reflections
surveillance radar
secondary or active radar
+ fire control radar
heterogeneous; reliability and new data; blackboard
Felix Riegler 13/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Radar
emits electromagnetic waves and detects reflections
surveillance radar
secondary or active radar
+ fire control radar
heterogeneous; reliability and new data; blackboard
Felix Riegler 13/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Radar
emits electromagnetic waves and detects reflections
surveillance radar
secondary or active radar
+ fire control radar
heterogeneous; reliability and new data; blackboard
Felix Riegler 13/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
sound locatisation
two microphones
homogeneous
new data
module
Felix Riegler 14/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
sound locatisation
two microphones
homogeneous
new data
module
Felix Riegler 14/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
sound locatisation
two microphones
homogeneous
new data
module
Felix Riegler 14/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
sound locatisation
two microphones
homogeneous
new data
module
Felix Riegler 14/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
camera + infrared projector
e.g. Kinect
heterogeneous
new data
?
Felix Riegler 15/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
camera + infrared projector
e.g. Kinect
heterogeneous
new data
?
Felix Riegler 15/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
camera + infrared projector
e.g. Kinect
heterogeneous
new data
?
Felix Riegler 15/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
camera + infrared projector
e.g. Kinect
heterogeneous
new data
?
Felix Riegler 15/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
stereo vision
two cameras
homogeneous
new data
module
Felix Riegler 16/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
stereo vision
two cameras
homogeneous
new data
module
Felix Riegler 16/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
stereo vision
two cameras
homogeneous
new data
module
Felix Riegler 16/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
stereo vision
two cameras
homogeneous
new data
module
Felix Riegler 16/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
General data fusion methods
Felix Riegler 17/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
General data fusion methods
Bayesian network
Kalman filter
Fuzzy logic
Monte Carlo methods
Dempster–Shafer theory
Felix Riegler 18/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
General data fusion methods
Bayesian network
Kalman filter
Fuzzy logic
Monte Carlo methods
Dempster–Shafer theory
Felix Riegler 18/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
General data fusion methods
Bayesian network
Kalman filter
Fuzzy logic
Monte Carlo methods
Dempster–Shafer theory
Felix Riegler 18/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
General data fusion methods
Bayesian network
Kalman filter
Fuzzy logic
Monte Carlo methods
Dempster–Shafer theory
Felix Riegler 18/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
General data fusion methods
Bayesian network
Kalman filter
Fuzzy logic
Monte Carlo methods
Dempster–Shafer theory
Felix Riegler 18/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Bayesian network
Design on abstract or raw data?
Felix Riegler 19/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Kalman Filter
tracking, localisation, navigation
suited for many different (sensor-)inputs
recursive state estimation, thus
simple and efficient in many cases
x(t|t− 1) - state at time t with data t-1
P (t|t) - covariant - error estimate
Felix Riegler 20/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Kalman Filter
tracking, localisation, navigation
suited for many different (sensor-)inputs
recursive state estimation, thus
simple and efficient in many cases
x(t|t− 1) - state at time t with data t-1
P (t|t) - covariant - error estimate
Felix Riegler 20/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Kalman Filter
tracking, localisation, navigation
suited for many different (sensor-)inputs
recursive state estimation, thus
simple and efficient in many cases
x(t|t− 1) - state at time t with data t-1
P (t|t) - covariant - error estimate
Felix Riegler 20/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Kalman Filter
tracking, localisation, navigation
suited for many different (sensor-)inputs
recursive state estimation, thus
simple and efficient in many cases
x(t|t− 1) - state at time t with data t-1
P (t|t) - covariant - error estimate
Felix Riegler 20/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Kalman Filter
tracking, localisation, navigation
suited for many different (sensor-)inputs
recursive state estimation, thus
simple and efficient in many cases
x(t|t− 1) - state at time t with data t-1
P (t|t) - covariant - error estimate
Felix Riegler 20/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Kalman Filter
tracking, localisation, navigation
suited for many different (sensor-)inputs
recursive state estimation, thus
simple and efficient in many cases
x(t|t− 1) - state at time t with data t-1
P (t|t) - covariant - error estimate
Felix Riegler 20/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Kalman Filter in a nutshell
Felix Riegler 21/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Fuzzy logic
many-value logic
f → [0, 1]
degree of certainty
especially use in threshold controlled systems
Felix Riegler 22/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Fuzzy logic
many-value logic
f → [0, 1]
degree of certainty
especially use in threshold controlled systems
Felix Riegler 22/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Fuzzy logic
many-value logic
f → [0, 1]
degree of certainty
especially use in threshold controlled systems
Felix Riegler 22/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Fuzzy logic
many-value logic
f → [0, 1]
degree of certainty
especially use in threshold controlled systems
Felix Riegler 22/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Stereo Vision
Felix Riegler 23/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Stereo Vision
stereopsis
PR2 has 2 stereo vision systems
another way for depth recognition
obviously 2 cameras capturing (more or less) the same picture
calibration is problematic
Felix Riegler 24/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Stereo Vision
stereopsis
PR2 has 2 stereo vision systems
another way for depth recognition
obviously 2 cameras capturing (more or less) the same picture
calibration is problematic
Felix Riegler 24/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Stereo Vision
stereopsis
PR2 has 2 stereo vision systems
another way for depth recognition
obviously 2 cameras capturing (more or less) the same picture
calibration is problematic
Felix Riegler 24/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Stereo Vision
stereopsis
PR2 has 2 stereo vision systems
another way for depth recognition
obviously 2 cameras capturing (more or less) the same picture
calibration is problematic
Felix Riegler 24/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Stereo Vision
stereopsis
PR2 has 2 stereo vision systems
another way for depth recognition
obviously 2 cameras capturing (more or less) the same picture
calibration is problematic
Felix Riegler 24/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Anaglyph 3D
Felix Riegler 25/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
2 pictures
Felix Riegler 26/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
3D Reconstruction
Felix Riegler 27/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Standard Geometry
z =b ∗ focallengthxcamL − xcamR
x =xcamL ∗ z
focallength
y =ycamL ∗ zfocalwidth
Felix Riegler 28/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Triangulation
general case
cameras need to be thoroughly calibrated
both intrinsic and extrinsic matrix have to be known
(lens properties + position in relation to the global coordinatesystem)
Felix Riegler 29/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Triangulation
Felix Riegler 30/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Traingulation Calculation
z1 ∗ point1 = camMatrix1 ∗ pointreal
z2 ∗ point2 = camMatrix2 ∗ pointreal
Felix Riegler 31/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
1 image + depth information
Felix Riegler 32/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Conclusion
Felix Riegler 33/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Sensor fusion
combining (multiple) sensor data to get better and/or morereliable data
core ability for humans
used in many domains
(humanoid)robotics
Felix Riegler 34/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Sensor fusion
combining (multiple) sensor data to get better and/or morereliable data
core ability for humans
used in many domains
(humanoid)robotics
Felix Riegler 34/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Sensor fusion
combining (multiple) sensor data to get better and/or morereliable data
core ability for humans
used in many domains
(humanoid)robotics
Felix Riegler 34/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Sensor fusion
combining (multiple) sensor data to get better and/or morereliable data
core ability for humans
used in many domains
(humanoid)robotics
Felix Riegler 34/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Algorithms for sensor fusion
Bayesian networks
Kalman filter
Fuzzy logic
stereo vision - depth recognition
Felix Riegler 35/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Algorithms for sensor fusion
Bayesian networks
Kalman filter
Fuzzy logic
stereo vision - depth recognition
Felix Riegler 35/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Algorithms for sensor fusion
Bayesian networks
Kalman filter
Fuzzy logic
stereo vision - depth recognition
Felix Riegler 35/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Algorithms for sensor fusion
Bayesian networks
Kalman filter
Fuzzy logic
stereo vision - depth recognition
Felix Riegler 35/36
DefinitionDomains and properties
ExamplesGeneral data fusion methods
Stereo visionConclusion
Thank you,Questions?
Felix Riegler 36/36
![Multi-Sensor Fusion - Store & Retrieve Data Anywhere€¦ · Origin Multi-sensor fusion is also known as multi-sensor data fusion [1, 2], which is an emerging technology originally](https://img.pdfslide.us/doc/110x75/5b6da87a7f8b9aa32b8d015c/multi-sensor-fusion-store-retrieve-data-anywhere-origin-multi-sensor-fusion.jpg)
