Introduction and Overview of Data and Information Fusion

James LlinasResearch Professor, Director, Emeritus

Center for Multisource Information FusionUniversity at Buffalollinas@buffalo.edu

CMIF's Approach: A Total Fusion Systems Perspective

• Basic through Advanced R&D In:― IF Based Situation and Threat Assessment― Multiple-Sensor System Data Integration/Analysis― Multi-Intelligence Information Environments

― COMINT, ELINT, OSINT, HUMINT, IMINT…..

― Graph Analysis Methodology and Systems

• Applications:― Defense: C4ISR Support and Tactical Applications― Non-Defense: Disaster Consequence Management,

Critical Infrastructure Protection, Medical Informatics

Intelligence Analysis

Situation and Threat Assessment for Multiple Application Domains

Focus on High Level Information Fusion and Graph Analytics

Maritime Domain Awareness

Contents are CMIF Proprietary

CMIF Foundational Sciences and Methods

3

Sciences• Graph Theory• Information Fusion

• Situation Assessment• Threat Assessment• Resource Management

• Information Theory• Mathematical Optimization• Modeling and Simulation• Knowledge Representation and

Reasoning (KRR)

Business/Engineering• Systems and Software

Engineering• Product Lifecycle Management• Service Oriented Design• Rich Web Applications• Complex Event Processing• Database Design• Realtime Systems

Slide 28Slide 28

Maximizes UAV surveillance with consideration for fuel consumption

Surveillance benefit of sensor s at target j can’t exceed the demand of sensor s at target j

Total number of each sensor s can’t exceed the number available at the base

Conservation of sensor delivery

UAV sensor capacity

Route continuity

A target may not be visited by the same UAV multiple times

All UAVs must begin at base location (location 0)

Route assignments cannot exceed total adjusted flight time

Sensor delivery may not take place unless a target is visited with the appropriate sensor attached

Maintains the time that each target is visited and ensures this time falls within the desired time window

Integrated Sensor Selection and Routing Model

CMIF STEF System – Main Window

Messages

Target Graphs

Click to match single Target graph with a

selected Message graph

Click to match all Target graphs with a selected Message graph

ComplexMathematicalOptimizations

Advanced Techniques In

Linguistic Processing

ModernGraphical Methods

Contents are CMIF Proprietary

Sampling of Methods

Collaborative Decision Support:CMIF Command and Control Lab

Center for Multisource Information Fusion (CMIF)Flexibility in Research and Development

• Core Technology Research & Development– State University of New York at Buffalo, NY, USA

• Top level research university and scientific staff— focus is basic research and proof of concept experiments– In existence 20+ years (Average grant revenue ~US$8M)

• Collaborative Partner for Technology Transition:– CUBRC, a not-for-profit defense R&D organization in Buffalo, New York, USA (~135 people, ~US$35M)

• Cleared, experienced staff, facilities— focus is development and transition

UB & CUBRC

UNIVERSITY at BUFFALO

BroadBaseOfDefenseR&D

Core Technology R&D

Transition & “Hardening”

History of Information Fusion• Dates to early 80’s—fairly young in the sense of

technological history—a maturing technology/field of study

• Driven by defense and intelligence needs– Originally as a “data compression” device to digest huge

amounts of sensed data as sensors advanced in capability (a “push” requirement)

– Later as an important element for decision support (a “pull” requirement)

• Matures to very broad range of application– Robotics, medicine, imagery/remote sensing, intelligent

transportation, conditioned-based maintenance, biometrics, medicine, etc

What is (Automated) Information Fusion?

Information fusion is an Information Process (Software) comprising:• FUNCTIONS:

• Alignment• Association, correlation • Combination of data and information from

• INPUTS:• Single and multiple sensors or sources to achieve

• OUTPUTS: • Refined Estimates of :

• parameters, characteristics, events, behaviors and relations for/among observed entities in an observed field of view

It is sometimes implemented as a Fully Automatic process or as a Human-Aiding process for Analysis and/or Decision Support

Data Fusion: Definition

" Data Fusion is the process of combining data (or information) for the purpose of estimating or predicting some aspect of the world"

Steinberg, Bowman, and White, “Revisions to the JDL Data Fusion Model”, NSSDF 1998

Multiple types of data

--various types of information

--redundant

--and complementary)

“Associated” or “Correlated” to :

--the same object

or event

or behavior

So that estimation algorithms (mathematical techniques)—or—automated reasoning methods (artificial intelligence techniques) can produce better estimates (than based on any single type of data)

Multiple types of data Related to things of interest

To improve estimates about those things

These Basic Ideas are Transferable to Many Types of Problems

RealWorld

Most Simply--

Observation System

Observations(Multiple)

Associationof Observations

Estimation

Basic Role of Fusion

(Dynamic) Real

States in the World

ObservationalMeans

(Streaming)

Common ReferencingAlignment

EstimatesOf World States

Dec-MkgAnalysis

etc

• One means to satisfy user information needs for decision/analysis support, i.e., most frequently inserted to support human user

Data Association

Evaluation

Actions

Process Refinement

Requirements driven from here

Everyday Data Fusion

Pain

Taste

Images

Touch

Smell

Temperature

Sound

Balance

Body Awareness(Proprioception

Multinodal Fusion

Augmented Sensing

RoboticMultisensor Fusion

[12]

Sensor Fusion Exploits Sensor Commonalities and Differences, Knowledge of Errors

Data Association Uses Overlapping Sensor Capabilities so that State Estimation Can Exploit their Synergies

CONFIDENCE COVERTCOVERAGE

DETECTIONRANGE ANGLE

KINEMATICSCLASS TYPE

CLASSIFICATION

POOR GOOD FAIR FAIR POOR

FAIR POOR GOOD FAIR FAIR

FAIR GOOD FAIR FAIR FAIR FAIR

RADAR

EO/IR

C3I

GOOD GOOD GOOD GOOD GOOD GOOD

DataAlignment

DataAssociation

StateEstimation

SENSORFUSION

FAIR

FAIR

Unknown Moving Object

[13]

A Persistent Focus: Reduced Uncertainty

Multisource Data (Evidence) Association

STATEESTIMATION

& PREDICTION

STATEESTIMATION

& PREDICTION

STATEESTIMATION

& PREDICTION

STATEESTIMATION

& PREDICTION

DATAASSOCIATION

“Assigned” ObservationsResulting from some “Best” way to decide which Observations should be “given” to

each State Estimator

M ObservationsFrom N Sensors Tracks “T”

Need some type of Mappingthat determines a goodway to allocate ObsvnsTo Tracks

mimjmm

O TkTj TlTi

Multiple Observations &

Multiple Entities

What a “Message” looks like— a Graphical Structure

An Observational Evidence “Atom“--Not a Point Measurement--

An Observation (description—Representation of an Observation)

Multiple Relationships

Disconnected semantic fragments

Synonyms

Generally all elements have some type of

imperfection or error

Some errors Quantified

Includes judgments as well as

observations

Design of the Association Process for Linguistic (Msg) Inputs

Human Observer 1

Human Observer 2

HypothesesSelf-generatedby node/arccontent

HypothesesScored viaSemanticSimilarity Scores that accnt For uncertainty

Pick a node/arc,Search other graphFor associable elements(eg exploit ontology)

Smart GraphSearch

HypothesesEvaluationBy high-dimensionalAssignment problemsolution

Apply JVC or otherModern assignmentProblem solution

Good AssignmentSolution &

Graph Merging

Effective

Semantic

scoring

Interdependency withText, Semantic

Operations

Linguistic, Textual

(Semantic) Inputs

Fusion of Realtime Data and A Priori Data Bases

DATA INFORMATION KNOWLEDGE UNDERSTANDING

Terrain/Cultural FeaturesImagery Overlays

Logistics

IntelligenceWeather

Coalition Forces

Situation

• Intel Sources• Air

Surveillance• Surface

Surveillance• Space

Surveillance

Decision Maker

Decision-SpecificInformation

• Timely • Accurate • Consistent • Structured• Integrated

FusionTechnology

Realtime

A Priori and Realtime

“Context”Today—Includes

--Sociocultural Info--Social Media

Data Fusion Functional Model(Jt. Directors of Laboratories (JDL), 1993)

Level 0 — Sub-Object Data Association & Estimation: pixel/signal level data association and characterization

Level 1 — Object Refinement: observation-to-track association, continuous state estimation (e.g. kinematics) and discrete state estimation (e.g. target type and ID) and prediction

Level 2 — Situation Refinement: object clustering and relational analysis, to include force structure and cross force relations, communications, physical context, etc.

Level 3 — Impact Assessment: [Threat Refinement]: threat intent estimation, [event prediction], consequence prediction, susceptibility and vulnerability assessment

Level 4: Process Refinement: adaptive search and processing (an element of resource management)

Level 0ProcessingSub-object DataAssociation &

Estimation

Level 1Processing

Single-ObjectEstimation

Level 2Processing

SituationAssessment

Level 3Processing

Threat/ImpactAssessment

Level 4ProcessingAdaptive Process

Refinement

Data BaseManagement System

SupportDatabase

FusionDatabase

INFORMATION FUSION PROTOTYPEJEM

JWARN3GCCS• Point and

Standoff Sensors• Data Sources• Intel Sources• Air Surveillance• Surface Sensors• Standoff Sensors• Space

Surveillance

Methods:--Combinatorial Optimization

--Linear/NL Estimation--Statistical

--Knowledge-based--Control Theoretic

DetectionTracking

ID

AggregationBehavior

Events

LethalityIntent

Opportunity

Sensor MgmtProcess Mgmt

• Reliability• Improved Detection

• Extended Coverage(spatial and temporal)

• Improved SpatialResolution

• Robustness (Weather/visibility, Countermeasures)

• Improved Detection• Improved State Estimation

(Type, Location, Activity)

• MultipleSensors

Operational Benefits of Multiple SensorData Fusion

State Estimates of Reduced UncertaintyAnd Improved Accuracy

• DistributedSensors

• DiverseSensors

Information Fusion: The Defense Context

“Associated” or “Correlated” to the same object or event or behavior

Multiple types ofsensor data

Related to thingsof interest in theReal World

To improveestimates aboutthose things

RealWorld

Fusion (Estimation)Techniques

In the defense problem:• Non-cooperative,• Unfriendly• Deceptive

!

Today’s IF Process Design Environment:Information-space Motivation: Exploitation of all Information

Modern Fusion Process

Observational Data

Sensor Observations

Human Observations

Dynamic Real World

Language

Numbers

Semantic Label

Contextual Information

Weather

Cultural

Financial

PoliticalWeb

A Priori Dynamic World Model

World State EstimatesLearning Processes

“SOFT”“HARD” “HARD & SOFT”

ChatTwitter

L4 Knowledge Mgmt

Declarative Knowledge:Ontologies

The Soft Front-end Input

UnconstrainedVocabulary

(Possibly different languages)

Semantics

Language Processing

AutomatedText

Extraction

~ RDF Triples (++)Typical

Atomic, RawData Input

(Digitized)

Computational Linguistics,NLP

Trained Observer

Untrained Observer

Interview

Bystander

Source Characterization

Perceptual and Cognitive Errors in observation

RealWorld Truth

Error in oral expression

Error in audio capture

Error in audio -to -text

conversion Error in text extraction

Conversion

Soft Data

To Common Ref, Data Association

є4

є5

є2

є3

є1

Hard Data Calibration

(Truth)Target

Pd (Obs Params)

To Common Ref, Data Association

Data Characteristic

Hard Soft Remarks

Observation sampling rate

High Low Imputes requirements for adaptive, retrodiction-type processing (i.e. “Out-of-Sequence Measurement” type processing), as well as agile Temporal Reasoning

Semantic Content Limited to specific, usually singular Entities

Can be conceptually broader than single Entities

Imputes requirements to design an automated Semantic Labeling process, coupled to a rich Domain Ontology Requires ability to associate and infer at multiple levels of abstraction

Limited to Entity Attributes

Can include Judged Relationships

Accuracy, Precision

Relatively high, good repeatability (Precision)

Broadly low accuracy in attributes, high at the conceptual level

Imputes requirements for robust Common Referencing and Data Association

Totally distinct from Hard SensorsPhilosophy: Relations not directly

observable—require reasoning over properties of entities

Brower, J., (2001) "Relations without Polyadic Properties: Albert the Great on the Nature and Ontological Status of Relations." Archiv für Geschichte der Philosophie 83: 225–57.

This line of thought suggests that relations are the result of a process of some type of comparison, ie [Brower, 2001], “an act of reasoning”.

Humans can also judge intangibles--emotional state

Some Distinctions in Hard and Soft Observational Data

Counterinsurgency Problem Environment

Soft Sensing Hard Sensing

MURIInformation Fusion

Technology

Counter-Insurgency

Soft Operations Kinetic Operations

COIN Decision Support

Not comprehensive

UNCLASSIFIED//FOUO

Context *

* Connable, B, Culture and COIN, www.citadel.edu/.../Connable,%20Culture%20and%20Counterinsurgency%20Brief.ppt

24

Some Remarks on Ontology and Information Fusion

Dr. James LlinasResearch Professor, Director (Emeritus)

Center for Multisource Information FusionUniversity at Buffalo and CUBRC

llinas@buffalo.edu

25

Roles for Ontologies in IF Processes/Systems• Reasonably reliable a priori Declarative Knowledge

about some domain– In the face of domains for which reliable a priori Procedural

(dynamic) Knowledge is hard to specify• “Weak Knowledge” problems

• As such, they provide a framework that connects Entities and Relationships– Of fundamental concern for COIN, Ctr-Terrorism, Irregular

Warfare re social structures and militarily-significant entity relations

• The basic construct of a “Situation” or a “Threat” and thus Level 2, 3 Fusion estimation

26

Complexities in Distributed and Networked Systems

• In modern Distributed/Networked Systems there are No single points of authority: These systems are collages of Legacy systems—Joint/Multiservice systems—Coalition systems

• Nodal Ontologies for Fusion/Situational Estimation, and Communication-support Ontologies for Inter-Nodal Communications/Data-sharing (eg JC3IEDM)

• Harmonizing NLP Operations and Ontologies within and across such systems

• The issue of Uncertainty in Ontological specification:– Probabilistic and Non-Probabilistic Ontologies

• Is there an Inescapable need for Semantic Mediation?– Mediator systems well-studied and developed*

• Eg Gio Wiederhold (June 1, 1993). "Intelligent integration of information". ACM SIGMOD Record 22 (2)(This was a major DARPA program) 27

Semantic Complexity• Controlling Semantic Proliferation/Complexity:

– Ontologies– Controlled Languages

• Eg Battle Management Language– Eg Shade, U., et al, From Battle Management Language (BML) to Automatic Information

Fusion, Chapter in Information Fusion and Geographic Information Systems, Lecture Notes in Geoinformation and Cartography,Popovich, V.V.; Claramunt, C.; Devogele, Th.; Schrenk, M.; Korolenko, K. (Eds.), 2011, Springer

• Understanding complexity drivers in text• Eg McDonald, D.D., Partially Saturated Referents as a Source of Complexity in

Semantic Interpretation, Proceedings of NLP Complexity Workshop: Syntactic and Semantic Complexity in Natural Language Processing Systems, 2000

• Measuring Semantic Complexity• Eg, Pollard, S and Biermann, A.W., A Measure of Semantic Complexity for Natural

Language Systems (2000) Proceedings of NLP Complexity Workshop: Syntactic and Semantic Complexity in Natural Language Processing Systems, 2000

28

The Association Problem

• The Ontologically-specified World is controllable—the Real Data World is not• While Ontologies can help in Fusion-based estimation and inferencing

problems, the mechanics of exploitation will involve the associability of Real (uncontrolled) data to (controlled) Entities and Relations in the Ontologies– Semantic similarity, metrics, degree (“hops”), etc– Efficient algorithms—eg Cloud implementations– PhD-level research

• There is also the issue of “Coverage”—in poorly-understood/known problems, how does one specify an Ontology that has “adequate” coverage?– Issue of negative information

29

Summary

• Ontologies have a useful role in the design and development of Information Fusion systems

• Questions regarding issues of:– Authoritative control of semantics in distributed systems

• Acceptable, optimal methods for mediation– Complexity of semantics

• Understanding, measuring, controlling– Association of semantic terms and complex, high-dimensional

semantic structures • Seem to require further, continuing study to better define

best ways to employ ontological information in complex, distributed, large-scale Information Fusion systems and applications

30

Unified Research on Network-based Hard and Soft Information Fusion

• A CMIF 5-year “Multidisciplinary University Research Initiative (MURI)” Program– Funded by the Army Research Office; ~ $7M– UB/CMIF lead + Penn State + Tenn State

• Soft/NLP/Ontology Lead: Prof Stu Shapiro, CSE– Building “TRACTOR” Soft front-end

31

32

33

34