Lesson I: Introduction

David L. Hall

Lesson Objectives

• Introduce this course and instructor• Provide an understanding of the

course logistics, requirements, grading, assignments, and ground rules

• Introduce the topic of data and information fusion

Course Objectives

• To provide an introduction to the field of data and information fusion– Models of multisensor data fusion– The JDL Data Fusion Process Model– Techniques for data fusion ranging from estimation to

pattern recognition and automated reasoning– Guide you through a team exercise involving design of a

data fusion system to address a selected application– Present a balanced view of the advantages and

limitations of fusion– Understand the role of the human in the loop

analyst/decision maker – Provide a basis for further study and specialization

Revealing my pedagogical hand• Brief presentations• In-class (on-line) exercises• Humor and stories• On-line discussion &

presentations• Planned time for group

meetings & work

It is important that you:1) Participate every week2) Focus on and complete on-line materials, lecture material, readings, assignments.

It is important that you:1) Participate every week2) Focus on and complete on-line materials, lecture material, readings, assignments.

On-Line MaterialsOn-line Lessons

• Site navigation on LHS

• Each lesson summarizes• Video pre-view • Introduction • Lesson objectives • Commentary &

discussion• Activities &

assignments

• Links to readings (via electronic library reserve)

• Electronic copy of Lecture

materials

Text and Readings

Assigned Text

• D. Hall and S. A. McMullen, Mathematical Techniques in Multisensor Data Fusion, Artech House, 2004

Selected Readings

• D. Hall and J. Llinas, editors, Handbook of Multisensor Data Fusion, CRC Press, 2001

• Excerpts from selected textbooks• Selected technical papers• One science fiction story

Course Lessons

1. Introduction2. JDL Model3. Project initiation4. Sensor processing5. Level 0 processing6. Level 1 – Correlation7. Level 1 – Estimation8. Level 1 – Target ID9. Systems Engineering

10. Project design11. Level 2 (situation refinement)12. Level 3 (Consequence

refinement)13. Level 4 – Process refinement14. Level 5 – Cognitive

refinement15. Project detailed design16. Data Fusion state of the art17. Final Presentation

Assignments & Weights

Individual Assignments and weights (70 %)

• Ten (10) low stakes quizzes (20 %)• Six (6) on-line discussion participation (15 %)• Eight (8) Individual writing assignments (24 %) • Peer evaluation (11 %)

Group project and weights (30 %)

• Final technical report (20 %)• Final presentation (10 %)

Ground Rules

• Attendance/participation & preparation• Plagiarism (cheating)• Academic integrity• Affirmative Action & Sexual Harassment• Americans with Disabilities Act

You have one week to question or dispute grades, missed assignments, or missed classes:

Note – I dislike “wheedling” for extra credit

The origin of multisensor data fusion

“I say fifty, maybe a hundred horses . . . What do you say, Red Eagle?”

Biological origins of sensor fusion

Sight

Smell

SonarTouch Chemical detection

Sound

Augmentation of single senses

A long history of single sense augmentation has included eyeglasses, hearing aids, telescopes, microscopes (and more recently electronic noses, chemical detectors and many others

Augmentation of cognition• Similar to the

augmentation of our senses, a long history of effort has sought to augment out cognition

• Data fusion seeks to support the augmentation & automation of the multi-sensing, cognition process for improved awareness and understanding of the world

JDL Data Fusion Model

• Course organized using the JDL Model “Levels”

• Hall and McMullen text organized around JDL model

• Lessons in on-line site focus on JDL levels

• Lessons systematically “walk through” the levels

• Project focus on designing a data fusion system for selected application using the JDL framework

Data Fusion Functional Model

The JDL model (1987-91) and the draft revised model (1997)

• Level 0 — Sub-Object Data Association and Estimation: pixel/signal level data association and characterization

• Level 1 — Object Refinement: observation-to-track association, continuous state estimation (e.g. kinematics) and discrete state estimation (e.g. target type and ID) and prediction

• Level 2 — Situation Refinement: object clustering and relational analysis, to include force structure and cross force relations, communications, physical context, etc.

• Level 3 — Significance Estimation [Threat Refinement]: threat intent estimation, [event prediction], consequence prediction, susceptibility and vulnerability assessment

• Level 4 — Process Refinement: adaptive search and processing (an element of resource management)

Adapted from A. Steinberg

Joint Directors of Laboratories Data Fusion Subpanel:

DEFINITION OF DATA FUSION:DEFINITION OF DATA FUSION:

A continuous process dealing with the association, correlation, and combination of data and information from multiple sources to achieve refined entity position and identity estimates, and complete and timely assessments of resulting situations and threats, and their significance.

Definitions . . .• Sensor Fusion = Data Fusion from Multiple Sensors

(same or different sensor types)• Data Fusion = Combining information to estimate or

predict the state of some aspect of the world• Data Fusion Functions:

– Data Alignment(spatio-temporal, data normalization, evidence conditioning)

– Data Association (hypothesize entities)

– State Estimation & Prediction

(etc.)

Platform

(etc.)

Reports

Situation

Cross-Force Relations

Force Structure

Unit

Traditional Focus

Representative Data Fusion Applications for Defense Systems

SPECIFICAPPLICATIONS

INFERENCESBY DF PROCESS

PRIMARYOBSERVABLE DATA

SURVEILLANCE VOLUME

SENSORPLATFORMS

Ocean Surveillance Detection, Tracking,Identification ofTargets/Events

EM Signals Acoustic Signals Nuclear Related Derived Observations

(wake)

Hundreds ofNautical Miles

Air/Surface/Sub-Surface

Ships Aircraft Submarines Ground-based Ocean-based

Air-to-Air andSurface-to-AirDefense

Detection, TrackingIdentification ofAircraft

EM Radiation Hundreds of Miles(Strategic)

Miles (Tactical)

Ground-based Aircraft

BattlefieldIntelligence,Surveillance andTarget Acquisition

Detection andIdentification ofPotential GroundTargets

EM Radiation Tens to Hundredsof Miles about aBattlefield

Ground-based Aircraft

Strategic Warningand Defense

Detection ofIndications ofImpending StrategicActions

Detection/Trackingof Ballistic Missilesand Warheads

EM Radiation Nuclear Related

Global Satellites Aircraft

Representative Data Fusion Applications (Non-DoD Systems)

SPECIFICAPPLICATIONS

INFERENCESBY DF PROCESS

PRIMARYOBSERVABLE DATA

SURVEILLANCE VOLUME

SENSORPLATFORMS

Condition-basedMaintenance

Detection, characterizationof system faults

Recommendations formaintenance/correctiveactions

EM Signals Acoustic Signals Magnetic Temperature X-rays

Microscopicinspection tohundreds of feet

Ships Aircraft Ground-

based (e.g.,factory)

Robotics Object location,recognition

Guide the locomotion ofrobot hands, feet, etc.

TV Acoustic Signals EM Signals X-rays

Microscopic totens of feetabout the robot

Robot Body

Medical Diagnosis Location, identification oftumors. abnormalities, anddisease

X-rays NMR Temperature IR Visual Inspection Chemical/Biological

Data

Human bodyvolume

Laboratory

EnvironmentalMonitoring

Identification, location ofnatural phenomena(earthquakes, weather)

SAR Seismic EM Radiation Core Samples Chemical/Biological

Data

Hundreds ofmiles

miles (sitemonitoring)

Satellites Aircraft Ground-

based Underground

Samples

What does data fusion do?

OPERATORS &SUPPORT SYSTEMS

SENSORS

SOURCESMISSION

EQUIPMENT &WEAPONRY

MISSIONENVIRONMENT

OPERATES ON:• sensor data• processed data• reference data

DATA FUSION FUNCTIONS

• ASSOCIATION• ESTIMATION• PREDICTION• INFERENCING• ANALYSIS• ASSESSMENT

• positional, identity, and attribute estimates about objects and events

• situation refinement• refinement of enemy threats,

vulnerabilities, opportunities

TO SUPPORT

Hierarchy of Inference Techniques

Type of Inference Applicable Techniques

- Threat Analysis

- Situation Assessment

- Behavior/Relationships of Entities

- Identity, Attributes and Location of an Entity

- Existence and Measurable Features of an Entity

High

Low

- Knowledge-Based Techniques

- Decision-Level Techniques

- Estimation Techniques

- Signal Processing Techniques

- Expert Systems- Scripts, Frames, Templating- Case-Based Reasoning - Genetic Algorithms

- Neural Nets- Cluster Algorithms- Fuzzy Logic

- Bayesian Nets- Maximum A Posteriori

Probability (e.g. Kalman Filters, Bayesian)

- Evidential Reasoning

INF

ER

EN

CE

LE

VE

LIN

FE

RE

NC

E L

EV

EL

Multi-Level/Multi-Perspective Inferencing

Level 1:Positional, Identity &Attribute Refinement

Level 2:SituationRefinement

Level 3:ThreatRefinement

Level 4:ProcessRefinement

where what whywhen who how

DATA FUSION PROCESSING

Level O:Signal Refinement

PHYSICAL OBJECTS

individual organizations

EVENTS

specific aggregated

TERRAIN & ENEMY TACTICS

local global

ENEMY DOCTRINE & OBJECTIVES

specific global

FRIENDLY VULNERABILITIES & MISSION

options needs

FRIENDLY ASSETS

specific & global

Benefits of Data Fusion: Marginal Gain of Added Sensors

Nahin and Pokokski, IEEE AES, 16 May 1980.

0.13

0.12

0.11

0.10

0.09

0.08

0.07

0.06

0.05

0.04

0.03

0.02

0.01

0.5 0.6 0.7 0.8 0.9 1.0

PN Single Sensor Probability of Correct Classification

P

N

(P

N +

2)

- P

N=

PN

PN

N=1(13 Sensors)

N=3(35 Sensors)

N=5(57 Sensors)

Benefits of Data Fusion: Enhanced Spatial Resolution RADAR

TARGET REPORTLOS

ABSOLUTE UNCERTAINTYREGION INTERSECTION

RADAR ABSOLUTEUNCERTAINTY REGION

FLIR ABSOLUTEUNCERTAINTYREGION

ELEVATIONUNCERTAINTY

TARGETREPORT

COMBINED

AZIMUTHUNCERTAINTY

FLIR

TARGET REPORTLOS

TARGETREPORT

SLANT RANGEUNCERTAINTY

AZIMUTHUNCERTAINTY

SLANT RANGEUNCERTAINTY

ELEVATIONUNCERTAINTY

FLIR and Radar Sensor Data CorrelationFLIR and Radar Sensor Data Correlation

Adapted from W.G. Pemberton, M.S. Dotterweich, and L.B. Hawkins, “An Overview of Fusion Techniques”, Proc. of the 1987 Tri-Service Data Fusion Symposium, vol. 1, 9-11 June 1987, pp. 115-123.

The DoD Legacy: Extensive Research Investments

• JDL Process model• Taxonomy of Algorithms• Lexicon• Engineering Guidelines

– Architecture Selection

– Algorithm Selection• Evolving Tool-kits• Extensive Legacy of

technical papers, books• Training Materials• Test-beds• Numerous prototypes

Challenges in Data Fusion . . .

• Robust sensors:– no perfect sensors available– difficult to predict sensor performance– unable to effectively task geographically distributed non-

commensurate sensors• Image and non-image fusion:

– no true fusion of imagery and non-imagery data– unable to optimally translate image in time-series data into

meaningful symbols– no requisite models for coherent fusion of non-commensurate sensor

data• Robust target identification:

– insufficient training data– unable to perform automated feature extraction– no unified method for incorporating implicit and explicit information

for identification (e.g., information learned from exemplars, model information, and cognitive-based contextual information)

Challenges in Data Fusion . . .

• Unified calculus of uncertainty (e.g., random set theory):– do not know how to effectively use these techniques – limited experience in trade-offs and use of fuzzy logic, rules

probability, Dempster-Shafer’s method, etc.– unsure how to select the best uncertainty method

• Pathetic cognitive models for Level 2 and 3:– unknown how to select the appropriate knowledge

representation techniques– argue about competing methods– do not know how to use hybrid methods– unable to perform knowledge engineering

Challenges in Data Fusion . . .

• Non-commensurate sensors:– uncertainty as how to optimize use of wildly non-commensurate

sensors– inability to know how to link decision needs to sensor management– unable to effectively use 10N sensors– no consensus on MOE/MOP

• Human computer interface (HCI):– trendy and driven by technology and not cognitive needs of user– suffer from the Gutenberg Bible syndrome– no effective tools to overcome cognitive deficiencies– unable to capitalize on built-in human pattern recognition (e.g.,

recognition of faces, concepts of harmony)

Data Fusion Issues . . .

• What algorithms or techniques are appropriate and optimal for a particular application?

• What architecture should be used (i.e., where in the processing flow should data be fused (viz. at the data, feature, or decision levels)?

• How should individual sensor data be processed to extract the maximum amount of information?

• What accuracy can realistically be achieved by a data fusion process?• How can the fusion process be optimized in a dynamic sense?• How does the data collection environment (i.e., signal propagation,

target characteristics, etc.) affect the processing?• Under what conditions does multi-sensor data fusion improve system

operation (under what conditions does it impede or degrade performance)?

Lesson 1 Assignments

• Review the on-line introduction material (lesson 1)• Read chapter 1 of Hall and McMullen• Writing assignment 1: write a brief biographical

sketch of yourself (to share with the class)• Writing assignment 2: write a paragraph describing

the occurrence of data fusion in a natural setting• Team Assignment (T-1) - Meet with your assigned

team to discuss the semester collaboration

Data Fusion Tip of the Week

“Here’s where we plan to use data fusion.”

Despite enormous amounts of funding for data fusion research – there is still no magic data fusion system or techniques!