Fundamentals and Principles of Information Fusionstids.c4i.gmu.edu/papers/STIDSPresentations/STIDS16_Blasch_Keynot… · Low-level Information Fusion High-level ... Shape video and

Importance of Semantic

Ontologies in Information Fusion

Erik Blasch

AFRL

Glue for High Level

Information Fusion

https://books.google.ca/books?id=V2dBDAAAQBAJ&printsec=frontcover&source=gbs_ge_summary_r&cad=0

Erik Blasch – STIDS16 2

Abstract

The use of semantic technologies has essential implications for information fusion systems solutions.

An emerging development in high-level information fusion (HLIF) is the importance of the user for mission management, command and control, as well as process refinement. The ability of the user to be part of the systems solution supports low-level information fusion (LLIF) functions of object, situation, and impact assessment. Future technology designs will require coordinating the LLIF physics-based big data measurements with the HLIF human-derived information content.

A semantic ontology is necessary for physics-based and human-derived information fusion (PHIF). The fusion of measurements and content should augment contextual understanding, refine uncertainty estimates, and provide robust decision support. This talk will provide trends in high-level information fusion, address developments in an uncertainty ontology, and provide examples of PHIF.

Examples include unmanned aerial vehicle (UAV), multi-intelligence, and space situation awareness…. IoT


Outline

High-Level Information Fusion

1) Cyber-Physical Systems (CPS) and Internet of Things (IoT)

Methods to utilize data from IoT sensors (physics-based)

Mission needs (human-derived) requirements

2) Challenges in High Level Information Fusion

Ontology to link LLIF-HLIF problem definitions

Ontology to support uncertainty analysis

3) Uncertainty Analysis

Developments from the URREF

4) Examples

UAV traffic management

Machine Analytics

5) Summary

Erik Blasch – STIDS16

Level 0 SAW

SENSATION

Level 1 SAW

PERCEPTION

Level 2 SAW

COMPREHENSION

Level 3 SAW

PROJECTION Human

Situation

Awareness (SAW)

Machine Level 1 MIF

OBJECT

ASSESSMENT

Level 0 MIF

SUB-OBJECT

ASSESSMENT

Machine Information Fusion (MIF)

Interface

Observables

(dots on maps)

Situations

(storytelling)

Objects

(lines on maps)

Scenarios

(forecasting)

Low-level Information Fusion High-level Information Fusion

Visualization

(Evaluation)

Level 2 MIF

SITUATION

ASSESSMENT

Level 3 MIF

IMPACT

ASSESSMENT

Low-Level & High-Level Information Fusion

Decompose problem into elements of LLIF and HLIF

Determine the user (situation awareness) and machine (computation)

Discussion on evaluation/visualization and projection


Previous Panels What is HLIF,

• How to measure HLIF, and • Coordination machine/user.

5

Summary – Top Ten Trends

• Area A. Data/Knowledge Representation • Reference Model • Taxonomy of notations, symbols, and meanings

•Area B. SA/TA/IA Assessment • Semantics/ontologies • Social/Behavioral/Cultural Models

•Area C. Systems Design • User/agent coordination • Display (interactive)

•Area D. Evaluation • Common scenario, Perf. comparison • Metrics / Uncertainty analysis

•Area E. Information Management • Resource planning and information analysis • Joint theory of methods integration

Determined from 2000-2010 Panel Discussions at the International Conference on Information Fusion

Blasch, et. al.“High Level Information Fusion (HLIF) Survey of Models, Issues, and Grand Challenges,”

IEEE AES Mag., Aug 2012.

Evaluation of Techniques for Uncertainty Representation (ETUR) Working Group,

• uncertainty analysis • ontologies


Situation Uncertainty

Time

Space

Accuracy

Timeliness

Confidence

Meaning (Knowledge)

Representation SA/IA/TA

Assessment

User Coordination

System Design

Metrics (Evaluation)

Information Management

HLIF Uncertainty Issues (Systems Approach)

Blasch, et. al.“High Level Information Fusion (HLIF) Survey of Models, Issues, and Grand Challenges,”

IEEE AES Mag., Aug 2012.


Summary

Panel 2000 2001 2004 2005 2006 2007 2008 2009 2009 2010 Topic Vision L2-4 HLF KR-RM RM Agent HLIF Coalition TA/IA HLIF-GC

Reference Model

Data/Knowledge Representation O O O O O O

Semantics/Ontologies O O O O O O

SA/TA/IA Assessment

Social/Behavioral Model

User/Agent Coordination

Display (Interactive) X X X X

Common Scenario X X X X

Performance Eval/Metrics

Uncertainty Analysis █ █ █ █ █ █ █ █

Resource Planning

Joint Theory of Methods X X X X X

Current and Consistent Theme

O Key Importance

X General Importance

E. P. Blasch, D. A. Lambert, P. Valin, M. M. Kokar, J. Llinas, S. Das, C-Y. Chong, and E. Shahbazian, “High Level

Information Fusion (HLIF) Survey of Models, Issues, and Grand Challenges,” IEEE AES Mag., Aug 2012.


Outline










4) Examples


Machine Analytics

5) Summary


Cloud, CPS, IoT

NIST: CPS

NIST Cyber-Physical Systems Public Working Group; Draft Framework to Help ‘Cyber Physical Systems’ Developers

https://pages.nist.gov/cpspwg/

https://pages.nist.gov/cpspwg/


Cloud, CPS, IoT

Development of Human-Internet-Machine

http://www.sose2016.org/internetofthings.html

CyPhERS: “Cyber-Physical European Roadmap & Strategy” deliverable D5.2 CPS: Significance,

Challenges and Opportunities, from

Ref:

CyPhERS: “Cyber-Physical European Roadmap & Strategy” deliverable D5.2 CPS: Significance, Challenges and Opportunities, from

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwiNpPbNkavQAhUJziYKHd-YDXkQjRwIBw&url=http://www.sose2016.org/internetofthings.html&psig=AFQjCNG-gc1lrj8VFkwMn4a0RihNJZfbyw&ust=1479312170191189

https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwj-oIjTkqvQAhXE1CYKHRmZATsQjRwIBw&url=https://www.linkedin.com/topic/m2m&psig=AFQjCNG-gc1lrj8VFkwMn4a0RihNJZfbyw&ust=1479312170191189


User-Machine Learning

Multiple Users – OODA Loops, Functions

11

Observe

Orient

Decide

Act

SENSING

Exploitation

LEARNING

Data Mining

Analyze

Prepare

Prob

Discover

Real-time

OPERATOR

Forensic

ANALYST

Information Exploitation – both sensing and mining

- Enabled by cloud computing, analytics, and systems

E. Blasch, A. Steinberg, S. Das, J. Llinas, C.-Y. Chong, O. Kessler, E. Waltz, and F. White, "Revisiting the JDL model for information Exploitation," Int’l Conf. on Info Fusion, 2013.

Physical

Cyber

https://books.google.ca/books?id=V2dBDAAAQBAJ&printsec=frontcover&source=gbs_ge_summary_r&cad=0


Why Cross-Layer Design in IoT

• Ubiquitous communications, autonomous and self-aware device operation and handling of multiple sensed data of varying characteristics:

Physical

Data Link

Network

Transport

Application

Communication Protocol Stack

Info

rmati

on

Information

Physical Component

Info

rmati

on

Connectivity

Processing/Computing

• IoT devices need to access information from different layers of the cyber physical system and

• Can process the information in an integrated manner.

• Information needs to be integrated at a processing element for the IoT device to feature self-aware characteristics and be able to operate autonomously.

Andres Kwasinski, “Cross-Layer Framework in the Internet of Things for Cyber-Physical Systems,” SPIE 2016


Architecture for Cross-layer IoT

• All-layer cognitive agent module:

• A software module that gathers information from the different layered components of an IoT device.

• Able to develop the functions of self-awareness and autonomous operation while also bridging the separation between layers.

Based on Observe-Decide-Act cognitive cycle



What is the IoT?

• Lack of uniform agreement.

• Adopted view:

three distinct features for an instance of IoT application:

1) awareness - as a result of a sensing/data collection operation,

2) autonomy - complete operation without human intervention,

3) actionable - using the results from the data processing for decision making and operation.

• Focused IoT application: integration with infrastructure to enable a cyber-physical system called a “smart infrastructure”;

• Example: smart grid.


• Dramatic growth in Internet-connected devices- Most of this grow will come from sensing and actuation devices that act as nodes in the Internet of Things (IoT).

Source: "IMT Traffic Estimates for the Years 2020 to 2030," International Telecommunication Union (ITU)

technical report M.2370-0, July 2015.


Application Case: Powering Cellular Base Stations From the Smart Grid

• IoT has had a key role in modernizing the electric grid – the “smart grid”.

• One development from the smart grid: microgrids.

• Microgrid: electric power grids that are confined to a local area and which can operate connected to or isolated from a main grid because loads and local energy sources (generators or energy storage devices) are integrated through a controller that operates independently of the grid.

• ``Sustainable Wireless Area'' (SWA): an architecture that integrates a group of cellular base stations in a microgrid with the goal of maximizing the use of renewable energy to power the cellular infrastructure

New management dimension –

Shape video and data traffic.

Traffic shaping is reflected on cellular service quality experienced by end users.



16

Fig. 9: Big Data Management for Smart Grid

Big Data in Smart Grid Wei Yu, “On Secure and Resilient Energy-Based Critical Infrastructure ,” SPIE 2016

Smart grid must be dependable, cost-effective, secure, and operate in real-time

High volume data streams associated with smart grid operations need to be quickly

processed and analyzed (power grid to energy management system (EMS))


Outline










4) Examples


Machine Analytics

5) Summary


BOOK: High-Level Information Fusion


High-Level Information Fusion Management and Systems Design

Hu

man

- M

ac

hin

e In

terfa

ce

Systems Design Machine Human

High-Level

Information Fusion

Info

rmati

on

/Reso

urc

e

M

an

ag

em

en

t

Situ

atio

n A

naly

sis

E

valu

atio

n

Low-Level Information Fusion

Perception

Object

Observable

Situation

Scenario

Sensation

Comprehension

Projection

Assessment Awareness


HLIF Book Outline

Human System Interaction

Information Fusion Concepts and Representations

Information Fusion Evaluation

4. Interpreted Systems

2. The IFSA Model

5. Role of Information Management

7. INFORM Testbed

8. Legal Agreement Protocol 10. C-OODA

11. SBD Applications

12. Coalition Approach

13. Operation Condition Model

15. Measures of Worthiness

16. Measures of Effectiveness

9. UDOP

14. Int. Sys. Evaluation Toolbox

3. The STDF Model

6. Coalition Testbed

Scenario-Based Design

Distributed Information Fusion and Management


High-Level Information Fusion Management and Systems Design

1. Overview the HLIF problem (~ 1 hour)

Architecture, domain, algorithms, purpose (SA Approaches)

2. Methods for Situation Awareness (~ 1 hour)

Set up analysis of SAW/SA (functional)

Describe three types of approaches

Process, Interpreted, and State Transition

Develop notions of SA Prediction/Projection

3. Develop a IF Management and System Level Design (~ 1 hour)

Present System Management and Testbeds

Human Factors issues (C-OODA, UDOP)

4. Demonstrate HLIF Evaluation and Scenario Design (~ 1 hour)

Determine the design, testing, scenarios, and operability

Evaluation Methods


Motivation

1. Three years of discussion

Focused on the main issues in HLIF

See companion paper in Fusion Panel Studies

See other tutorial on Evaluation

2. Collaboration (SUM)

Sensor Management - HLIF is about different INTs

User – HLIF is about a collection of users

Mission – HLIF is about focusing on the goal (Top-Down)

3. Each Coordination brought together ideas

Technical panels – C3I, Info Mgt, User, and Testbeds

Countries and perspectives – each had end-to-end solution

4. Developments fostered from the Grand Challenges

Issues to explore in the next decade


Lesson 01: HLIF Overview


HLIF Architectures: JDL to Data Fusion Information Group (DFIG)

Grand Challenges

Paradigm , Semantic , Epistemic : HLIF Purpose

Interface, System: HLIF Management

Design, Evaluation : HLIF Design


SA Approaches

Process (DFIG) – US [Blasch, Salerno, Tangney]

Interpreted Systems (IS) /ODDA – Canada [Bosse, Jousselme/Maupin, Valin]

State Transition Data Fusion (STDF) – AUS [Lambert]

Common Issues: Metrics, Design, Future Concentrations





High Level Fusion Adapted from E. Waltz and J. Llinas, Multisensor Data Fusion, Artech House, Norwood, MA [1990])

Low-Level Processing

Sensor 1

Detection

Sensor 2

Detection

Sensor 3

Detection

Data

Association

State Estimation

Attribute Classification

Predicted States of

Targets in Track

Estimated Tracks

State Estimates

Target Identities

Low-Level Assessment

High-Level Processing

Assessment Detection of Pattern of Behavior

Association of Entities and Events

Prediction of Future Behavior

Classification of Situation

High-Level Assessment

On Situation

Behavior

Future Activities

Intent


User Fusion Model From E. Blasch and S. Plano, “DFIG Level 5 (User Refinement) issues supporting Situational Assessment

Reasoning,” Int. Conf. on Info Fusion - Fusion 05, July 2005.

Distributed

Local

Intel SIG

IM

EL

EW

Sonar

Radar HRR

SAR

MTI

Data Sources

Distributed

Human

Computer

Interface

Information

Sources

Level 4

Process

Refinement

DataBase Management System

Support

DataBase Fusion

DataBase Sensor Management

Level 2 – Situation Assessment

Level 3 – Impact Assessment

Level 1 – Object Assessment

Level 0 – Pre-processing Level 5

USER

Refinement

Interface

Design

Team

interaction

Proactive SF

Effectiveness Risk Throughput Utility

Performance Assessment/ Metrics

Context

Intent

Value

Priority

SOFT

HARD


26

L 4

DFIG - Fusion Model (Data Fusion Information Group), Fusion 2006 (from 2004)

Real

World

Platform

Ground

Station

Sensors

And

Sources

Human

Decision

Making

Resource Management

Mission Management

Explicit

Fusion

Machine

Tacit

Fusion

Human

Info Fusion

L 0

L 1 L2/3 L 5

Reasoning

Knowledge Representation

E. Blasch, I. Kadar, J. Salerno, M. M. Kokar, S. Das, G. M. Powell, D. D. Corkill, and E. H. Ruspini, “Issues and challenges of knowledge representation and reasoning methods in situation

assessment (Level 2 Fusion)”, J. of Advances in Information Fusion, Dec. 2006.

L 6


27

DFIG - Fusion Model (Data Fusion Information Group), Fusion 2006 (from 2004)

Low Level Information Fusion (LLIF)

Level 0 Data Assessment: estimation and prediction of signal/object observable states on the basis of pixel/signal level data association (e.g. information systems collections);

Level 1 Object Assessment: estimation and prediction of entity states on the basis of data association, continuous state estimation and discrete state estimation (e.g. data processing);

High Level Information Fusion (HLIF)

Level 2 Situation Assessment: estimation and prediction of relations among entities, to include force structure and force relations, communications, etc. (e.g. information processing);

Level 3 Impact Assessment: estimation and prediction of effects on situations of planned or estimated actions by the participants; to include interactions between action plans of multiple players (e.g. assessing threat actions to planned actions and mission requirements, performance evaluation);

Level 4 Process Refinement (an element of Resource Management): adaptive data acquisition and processing to support sensing objectives (e.g. sensor management and information systems dissemination, command/control).

Level 5 User Refinement (an element of Knowledge Management): adaptive determination of who queries information and who has access to information (e.g. information operations) and adaptive data retrieved and displayed to support cognitive decision making and actions (e.g. human computer interface).

Level 6 Mission Management (an element of Platform Management): adaptive determination of spatial-temporal control of assets (e.g. airspace operations) and route planning and goal determination to support team decision making and actions (e.g. theater operations) over social, economic, and political constraints.




28

Fusion Model Comparisons

Activity DFIG SAW Model OODA C-OODA

Command

Execution

Level 6 Resource

Tasking

Act Action

Implementation

Decision

Making

Level 5 User Control

User Refinement

Decide Recall

Evaluate

Sensor

Management

Level 4 Decision Making

Impact

Assessment

Level 3 Projection Orient Projection

Situation

Assessment

Level 2 Comprehension Comprehension

Object

Assessment

Level 1 Object

Assessment

Feature Matching

Signal/Info

Processing

Level 0 Signal/Feature

Processing

Observe Perception

Data

Acquisition

Sensing

Registration

Data Gathering

* DFIG (Data Fusion Information Group), SA(Situation Assessment) - J. of Adv. in Info. Fusion, Dec. 2006.

* C-OODA (Cognitive Observe, Orient, Decide, Act) – Fusion11, 2011

E. Blasch, R. Breton, P. Valin, and E. Bosse, “User Information Fusion Decision Making

Analysis with the C-OODA Model,” Int. Conf. on Info Fusion - Fusion11, 2011.





Grand Challenges





SA Approaches









High Level Information Fusion Challenges

Focus of the text:

Paradigm Challenge: How should the interdependency between the sensor

fusion and information fusion paradigms be managed?

Semantic Challenge: What symbols should be used and how do those

symbols acquire meaning?

Epistemic Challenge: What information should we represent and how

should it be represented and processed within the machine?

Interface Challenge: How do we interface people to complex symbolic

information stored within machines to provide decision support?

System Challenge: How should we manage information fusion systems

formed from combinations of people and machines?

Design Challenge: How should we design information fusion systems


Evaluation Challenge: How should we evaluate the effectiveness of

information fusion systems?

E. P. Blasch, D. A. Lambert, P. Valin, M. M. Kokar, J. Llinas, S. Das, C-Y. Chong, and E. Shahbazian, “High Level Information Fusion (HLIF) Survey

of Models, Issues, and Grand Challenges,” IEEE Aerospace and Electronic Systems Mag., Vol. 27, No. 9, Sept. 2012.


Joint HLIF Contributions HLIF Perspectives for Paradigm Challenges

AUS CAN US Paradigm

Challenge

Unifying human and machine

functional models across level 0 to 3

(STDF Ch3)

Formal models across level 0 to

level 3 fusion

(IS/ODDA – Ch 04)

Operational process models across

level 0 to level 5 fusion

(DIFG/IFSA Ch02)

Semantic

Challenge

Axiomatic semantics in First Order

Logics (FOLs) and Description Logics

(DLs) covering various metaphysical,

environmental, functional, cognitive and

social concepts

Axiomatic semantics in Modal

Logics covering various

metaphysical, environmental, and

functional concepts (Ch 4)

Operational semantics of computational

models to infer meaning over

environmental, functional, cognitive and

social concepts (Ch 2, Ch13, Ch 15)

Epistemic

Challenge

Cognitive agents with semantic,

epistemic (declarative facts and rules)

and episodic (procedural cognitive

routines) long-term memories

User (agent) with semantic,

epistemic (facts and rules), and

episodic (procedural) interactive

goals. Belief Theory (Ch 4, 7, Ch14)

Agents for workflow and service-based

semantic, epistemic (facts and rules) and

episodic (procedures) information

processing (Ch5)

Interface

Challenge

Interactive virtual news engaging

virtual advisers, battlespace, interactive

planning rooms, video, & newspapers

(web pages), Lexpresso controlled

natural language

Semantic and symbology

presentation, visualization, and

interactive sensor and mission

management (Ch 6, Ch 7, Ch 9)

Visualizations for a Common Operational

Picture (COP) with symbologies, info.

management, and collaboration tools (Ch

9). User refinement support to fusion

methods with cognitive theory (Ch10)

System

Challenge

Legal agreements between

combinations of CDIFT connected

human and machine cognitive

agents based on formal semantic

theories

OODA-based agent (Ch 7), state-

space approach, belief networks (Ch

4, Ch 7,Ch 14)

Use of ontologies and

workflow/service/human agents for the

CDIFT. Coordination of user/machine

fusion methods based on information

needs and tools (Ch10, Ch13, Ch16)

Design

Challenge

Use of synthetic development

environments containing track data,

intelligence reports, and various

domain knowledge

Track data, intelligence reports,

various domain knowledge,

simulations (Ch 6, Ch12)

Track data, intelligence reports, various

domain knowledge (Ch 6, Ch12)

Evaluation

Challenge

Probabilistic propositional set

disparity measures based on random

inference networks

OODA agents operate in a

distributed feedback loop (Ch 7)

Model checking techniques (Ch 14)

Bayes networks to measure probabilistic

variations from Operational Conditions (Ch

13) and derivation of MOEs from MOPs

(Ch 10)


HLIF Compare and Contrast (1)

Paradigm Challenge: How should the interdependency between the sensor

fusion and information fusion paradigms be managed?

Models: US IFSA framework (Ch 2); the AUS STDF framework (Ch 3); and

the Canadian IS framework (Ch 4).

COMMON:

• Promote situations as a fundamental construct of the world.

• Utilize the machine interpretation of situations and the machine

prediction of situations in the world.

• Represent situations in machines through states and time stepped

transitions between states.

CONTRAST:

• Situations : represented very formally under the IS and STDF frameworks

less formally under the IFSA framework.

• Machine processing of situations is characterized by formal logics under the IS

and by functional architecture process models under the STDF and IFSA



Semantic Challenge: What symbols should be used and how do those

symbols acquire meaning?

Meaning: US IFSA framework (Ch 2); the AUS STDF framework (Ch 3); and

the Canadian IS framework (Ch 4).

COMMON:

• States are implemented as knowledge representations within the machine.

• Knowledge representations can express sophisticated concepts well beyond

sensed characteristics.

• Transitions between states are understood as graphs.

CONTRAST:

• Semantics: IS and IFSA implement state vectors with operational semantics,

STDF: Mephisto engages propositional formulae with axiomatic semantics.

• State Transitions: IS and IFSA models use directed graphs.

STDF: graphs, expressed as regular expression cognitive routines with

procedural semantics (see Ch 12 for example), but actual state transitions

are simply expressed through knowledge base content.



Epistemic Challenge: What information should we represent and how

should it be represented and processed within the machine?

Complexity: Social Relationships

COMMON:

• Processing emphasis shifts from the world to the machine .

• LLIF processing is machine extracting content from information sensed .

• HLIF processing is machine imposing content on the sensed information.

• HLIF machines are termed agents.

• HLIF agent can only infer that a sensed airborne object poses a threat if it

imposes background knowledge about alliances, possible targets, et cetera.

CONTRAST:

• Cognition:

• STDF : ATTITUDE TOO Cognitive Model

• IS/C-OODA: Cognitive-OODA model (Ch 10)

• IFSA: User refinement composes cognitive refinement (UDOP)



Interface Challenge: How do we interface people to complex symbolic

information stored within machines to provide decision support?

Linking: Human Situation Awareness with Machines

COMMON:

• Pairing involves interfaces across the different levels of fusion

• Interface technology moves beyond the traditional “dots on maps” and “lines on

maps” technology of LLIF (UDOP in Ch 9, command and control graphical user

interface in Ch 7 and HiCOP in [4, 12, 13]).

CONTRAST:

• Modeling:

• IS/C-OODA and STDF same modal logic framework to both people and

machines.

• IFSA introduces additional fusion levels

• Role of Human : • IFAS : obtaining and utilizing human SAW;

• IS/C-OODA: directed toward decision support

• STDF: agnostic toward what is performed by humans and machines.



System Challenge: How should we manage information fusion systems


Distributed : Collections of humans and clusters of machines: CoABS (Ch

06), IS (Ch 14), and LAP (Ch08)

COMMON:

• Information management is deemed fundamental (Ch 5, TTCP C3I TP3).

• Distributed infrastructure is used to facilitate interaction between clusters of

fusion machines (CDIFT Ch 6 and INFORM Ch 7).

• CDIFT as common HLIF testbed (TP1) - support interoperable fusion products.

CONTRAST:

• Coordination : to manage multi-agent engagements

• IS/C-OODA and IFAS use a game theoretic model for agent interaction

• STDF : employs an agreement protocol for agent interaction

• Ch 6 (TP3) Agent-based systems (CoABS) framework (Ch 6) employs the

knowledge acquisition in automated specification (KAoS) system to resource

constrain distributed agents.



Design Challenge: How should we design information fusion systems


Content: Role of Agent

COMMON:

• Agent imposing content on the sensed information

• promotion of a scenario-based approach to the development of HLIF

• HLIF design system cannot occur without a rich context of the world in mind.

• Multi-national collaboration.

CONTRAST:

• Fidelity : to manage various levels of design

• IS/C-OODA and IFAS use a hierarchical model

• IFSA uses operational conditions of sensor, target, and environment

• STDF : employs a similar design across levels for design



Evaluation Challenge: How should we evaluate the effectiveness of

information fusion systems?

Metrics: IFSA (Ch15), IS/C-OODA (Ch 7, Ch14), STDF [14]

COMMON:

• Use of goals and missions

• Measures of content similarity or disparity assessments.

CONTRAST:

• IFSA and IS/C-OODA includes a number of SA measures

• MOPs: based on activities,

• Evidential reasoning to measure probabilistic relations,

• Game theory to measure action tradeoffs, and

• MOEs: Information theory for situation analysis

• The Australian offering [14] promotes probabilistic measures of the disparity

between sets of propositions.


HLIF Book Outline

Human System Interaction

Information Fusion Concepts and Representations

Information Fusion Evaluation

4. Interpreted Systems

2. The IFSA Model

5. Role of Information Management

7. INFORM Testbed

8. Legal Agreement Protocol 10. C-OODA

11. SBD Applications

12. Coalition Approach

13. Operation Condition Model

15. Measures of Worthiness

16. Measures of Effectiveness

9. UDOP

14. Int. Sys. Evaluation Toolbox

3. The STDF Model

6. Coalition Testbed


Distributed Information Fusion and Management


Information Fusion Situation Awareness Data Fusion Information Group and SA Reference Model

Platform

Ground

Station

Human

Decision

Making

Resource Management

Mission Management

Explicit Fusion

Tacit Fusion

Information Fusion

Reasoning

Knowledge Representation/ Discovery

S

O

U

R

C

E

S

D

A

T

A

Sensor mgt

Data mgt

Real World

Level 0

Object

Recognition

And

Tracking

Level 1 Level 2

Level 3 New Revised Models and Collection Requirements

Level 6

Level 4

Level 5

Situation Assessment

Knowledge of ‘Us”

Impact

(Changes)

Knowledge of ‘Them”

Knowledge

of ‘Us”

Possible Features

Plausible Features

Impact

Threat

X

issue for book, not intended to be a new model

HLIF Figure 2.8





Grand Challenges





SA Approaches









IF Quality of Service Performance Measures

COMM Human

Factors

Info Fusion ATR TRACK

Delay Reaction Time Timeliness Acquisition /Run

Time

Update Rate

Probability of

Error

Confidence Confidence Prob. (Hit)

Prob. (FA)

Probability of

Detection

Delay Variation Attention Accuracy Positional

Accuracy

Covariance

Throughput Workload Throughput # Images No. Targets

Cost Cost Cost Collection

platforms

No. Assets

Stallings 2002 Wickens, 1992 Blasch, 2003 COMPASE

Morrison

Blasch, (DDB)

Hoffman 2000

E. Blasch, M. Pribilski, B. Daughtery, B. Roscoe, and J. Gunsett, “Fusion Metrics for

Dynamic Situation Analysis”, Proc. of SPIE, Vol. 5429, April 2004.



SUMMARY 3


Multi-sensor Data

Situation Assessment

Increasing Operational Relevance and Semantic Specificity

SAR/EO ILCD

SAR OLCD

EO OLCD

HSI / IR

SAR

•

•

•

User Coordination


Verifying and Validating Assessment of Information Association SA/SAW

TESTBEDS

EO

STDF Model

OODA Model

DFIG/ IFSA Model


Evaluation

R

A

B

V

DA DP

DE

SK

EF SBD

Sensor,

User

Cueing

Data,

Info,

Management

Hu

ma

n

-


Simulated World

Information Fusion

Hu

ma

n-M

ac

hin

e

Re

so

urc

e M

gt


Simulated World

Information Fusion

Coalition - DIFT Model

proposes

accepts

Situation

Scenario

Analysis

IS Model

Awareness A,

Implicit

Knowledge

K,

Explicit Knowledge

X,


Correlation:

Very Strong Relationship

Strong Relationship

Weak Relationship

Research Selection

User

Requirements

Importance

to User

Accuracy 5

Timeliness 5

Confidence 5

Throughput 4

Cost 3

Relevance 4

Completeness 3

Kn

ow

led

ge

Rep

rese

nta

tio

n

Sit

ua

tio

n A

wa

ren

ess

Info

rma

tio

n Q

ua

lity

Co

ali

tio

n S

tan

da

rds

Inte

rop

era

ble

Sem

an

tics

Info

mra

tio

n

Ma

na

gem

ent

Th

eory

Sce

na

rio

Tes

tin

g

Hu

ma

n-M

ach

ine

Inte

rfa

ce

3 44 4 5 5Importance Weighting 3

Engineering Characteristics

Deployment

+

House of Quality


Outline










4) Examples


Machine Analytics

5) Summary


Example 1: UAV C. C. Insaurralde, E. Blasch, “Ontological Knowledge Representation for Avionics Decision-Making

Support,” IEEE/AIAA Digital Avionics Systems Conference, 2016.

NextGen

• A key example for NextGen system includes developments in the weather ontology, implemented in three operational capability phases [1]:

• Initial (2013): Significantly enhanced weather infrastructure providing modestly improved meteorological data to all users of the Nation's Air Transportation System.

• Midterm (2016): NextGen begins to implement automated decision assistance tools and algorithms for managing the air space, requiring high resolution weather forecasts and observations with a greater degree of accuracy and precision.

• Farterm (2022): NextGen weather must meet all meteorological and engineering performance requirements to support the NextGen traffic management systems.

SESAR

• One example is the European ATM Reference Model (AIRM). Specifically, they looked at Notices to Airman (NOTAM).

• NOTAMs provide weather and emergency updates to aviators in the form of text messages.


Example 1: UAV C. C. Insaurralde, E. Blasch, “Ontological Knowledge Representation for Avionics Decision-Making


• From: Semantic Driven Assurance for System Engineering in SESAR/NextGen,

Rainer Koelle, EUROCONTROL


Avionics Ontology

On-Demand Information Retrieval

Data

Acce

ss

Structured Authoring

Analytic Tools

Information

Loading

Information

Template Analyst Directory

Aviation Systems

Collaboration

• Decision Making

• Collection Management

• ATM

Data

Weather

NOTAMS

Flight Into

Airspace

No

rmalizati

on

Standardized Data

Taxonomy

Data Template

Ontology

Visualization Information

Catalog

• Mandates

• Policies

Automated Information

Retrieval

E. Blasch, “Ontologies for NextGen Avionics Systems,” IEEE/AIAA Digital Avionics Systems Conference, 2015.


Avionics Ontology

Ontology

Computer

Science

Information

Science

Knowledge Domain

Computation

Reasoning

Set of

Concepts

Concepts

Relationships

Subject of

Contains

Contains

Contains

Contains

Subject of Representation of

Description of

Makes

use of

Supports

Supports

Application of

Describes

Application of

Sructures

Uses

Affords

Enables

E. Blasch, “Ontologies for NextGen Avionics Systems,” IEEE/AIAA Digital Avionics Systems Conference, 2015.


Ex. 1: UAV – Ontology Representation C. C. Insaurralde, E. Blasch, “Ontological Knowledge Representation for Avionics Decision-Making


Ontology Semantics

• The Ontology Web Language (OWL) and the Protégé tool [10] are selected to realize the ontology for the approach proposed.

• The main OWL components to be created are the concepts (classes), properties for individuals, and instances of classes (individuals). They are set for Avionics Analytics Ontology (AAO) as follows:

• Classes (concepts) can be atomic classes (stand-alone ones) or associate classes (subclasses) along with “is-a” links, e.g., airspace, weather, vehicle (aircraft), person, location, and building.

• Properties (roles); are basically relationships between classes (or eventually individuals), e.g., hasAirpace, hasLanding, hasRoute, hastatus, hasTakeoff, etc.

ThingAirport

Airspace

Route

Vehicle

Aircraft

Weather

is-a

is-a

is-a

is-a

is-a

is-a

• Individuals; they are instances of classes (objects), e.g. a Boeing 747-800 is an individual (instance of the class “aircraft”).


Ex 1: UAV – Ontology Representation C. C. Insaurralde, E. Blasch, “Ontological Knowledge Representation for Avionics Decision-Making


Ontology Semantics

• Property restrictions along with classes and individuals are the building block to define axioms.

• Tbox: A terminological axioms (in particular, based on operators such as inclusion, equivalence, etc.)

• Abox: A set of assertional axioms (facts or assertions)

• The ABox and the TBox form the AAO knowledge base.

Set of axioms for the class “Aircraft”Set of axioms for the class “Route”Set of axioms for the class “Airport”Set of axioms for the class “Airspace”Set of axioms for the class “Weather”

Set of facts for the class “Aircraft”Set of facts for the class “Route”Set of facts for the class “Airport”Set of facts for the class “Airspace”Set of facts for the class “Weather”

TBox

ABox

Examples of terminological axioms

• Aircraft_A subclass of AircraftcannotLand and AircraftcanTakeoff

• Route_B subclass of NoLanding and Takeoff

• Airport_I subclass of LandingAirport and TakeoffAirport

• Airspace_IV subclass of FlyingAirspace

• ClearSky subclass of GoodWeather and VeryGoodWeather

Examples of assertional axioms

• AircraftcannotLand equivalent to Aircraft and (hasRoute only NoLanding)

• Takeoff equivalent to Route and (hasTakeoff only TakeoffAirport)

• NonFlyingAirspace equivalent to Weather and (hasWeather only VeryBadWeather)

• BadWeather equivalent to Weather and (Storm or ThuderStorm)


Example 1: UAV – Ontology Reasoning C. C. Insaurralde, E. Blasch, “Ontological Knowledge Representation for Avionics Decision-Making


ThingAirport

Airspace

Route

Vehicle

Aircraft

Weather

is-a

is-a

is-a

is-a

is-a

is-a

hasRoute

hasWeather

hasAirspace

hasTakeoff/Landing

Ontology Reasoning

• Reasoners are the engine for the knowledge-based reasoning queries.

• Aircraft have routes which in turn have a start point (departure or take-off from an airport) and an end point (landing in an airport).

• Airports have their own airspace that is part of a larger airspace. Airspaces have weather conditions as well as air traffic.


Example 1: UAV – Ontology Example 1 C. C. Insaurralde, E. Blasch, “Ontological Knowledge Representation for Avionics Decision-Making


Case Study I: A Weather Condition on and nearby an Airport




Application Examples

Case Study I: A Weather Condition on and nearby an Airport

• The query inference results suggest that (from left to right)

• Aircraft A and B (Flight A and B) can take off but they will not be able to land.

• Aircraft C (Flight C) can take off and it will be able to land.




Application Examples

Case Study II: Weather Condition on and nearby an Airport, and Runway Availability




Case Study II: Weather Condition on and nearby an Airport, and Runway Availability

• The query inference results suggest that (from left to right)

• Aircraft C & D (Flight C & D) will not be able to land as planned in route C and D

• Aircraft B, C, & D (Flight B, C, & D) should be advised to change routes as planned (Route B, C, & D)

• Aircraft A & B (Flight A & B) will be able to land as planned in route A and B.




Case Study III: Aircraft Proximity in Navigation based on UAV Size




Case Study III: Aircraft Proximity in Navigation based on UAV Size • The query inference results suggest that (from left to right)

• The Boeing 747 (due to its size and on-board pilots) requires 10 km.

• The UAV 1 and 2 can allow a distance of 3 km.

• The UAV 2 should keep a distance of 7 km (because its size) but it could be approached up to 3 km since it has a remote pilot who can be reached by ATM controllers.

• The Cessna 400 and UAV 3 require 7 km of minimum distance, even though the UAV 3 is small; it is autonomous (no pilot whatsoever).

• The UAV 4 is larger than the UAV 3 (no pilot) but it has a contactable remote pilot to deal with its waypoints.


World being reported World being sensed

Uncertainty Reasoning

- Computational cost - Performance - Consistency - Correctness - Scalability

Uncertainty Reasoning

- Computational cost - Performance - Consistency - Correctness - Scalability

? ? ?

? ? ?

?

InputInput

-- Relevance toRelevance to problemproblem -- Weight ofWeight of evidenceevidence -- CredibilityCredibility

OutputOutput

-- InterpretationInterpretation -- QualityQuality -- TraceabilityTraceability

Evaluation Framework Boundary

Uncertainty Representation

- Evidence handling - Knowledge handling

System Processes

Ideally, evaluating how the management of uncertainty affects the overall performance of a fusion

system would be just a matter of isolating the directly related aspects. However,

Real-life information fusion systems are usually too complex to allow for such a clear-cut separation,

Most of the aspects considered in system-wide performance are entangled with or influenced by uncertainty

representation considerations to some degree.

Isolating the impact of uncertainty handling in an IF system is thus a matter of understanding how

intertwined the choice of an uncertainty representation and reasoning approach is to the major

performance metrics used to evaluate the IF system itself.

Example 2: Machine Analytics E. Blasch, P. C. G. Costa, K. B. Laskey, H. Ling, and G. Chen, “The URREF Ontology for Semantic

Wide Area Motion Imagery Exploitation,” Proc. of the Seventh International Conference on Semantic Technologies for Intelligence, Defense, and Security (STIDS), Fairfax, VA, October, 2012, pp. 88-95


Machine and Visual Analytics

Analytics and Information Fusion.

Fusion Machine Concept

Level 0 Scientific Access to data and pedigree of information and

issues of structured/unstructured data

Level 1 Information

(Images, text)

Development of graphical methods for data

analysis

Level 2 Descriptive Uses data mining to estimate the current state

(i.e. Machine learning) over different reasoning

of trends for modeling

Level 3 Predictive Future options from current estimates

Level 4 Prescriptive Sequencing of selected actions

Level 5 Visual Sensing Making and Reasoning

Level 6 Activity-Based

Analytics

Policy instantiation of desired outcomes as to a

focused mission



Machine Analytics: User Involvement

Multiple Users – OODA Loops, Functions

61

Observe

Orient

Decide

Act

SENSING

Exploitation

LEARNING

Data Mining

Analyze

Prepare

Prob

Discover

Real-time

OPERATOR

Forensic

ANALYST

Information Exploitation – both sensing and mining

- Enabled by cloud computing, analytics, and systems



Ex: Automating Data Analysis with Semantics

EO UAV

MWIR UAV

Wide-Area Motion Imagery

Building-mounted EO

Timeliness (duration fulfilled)

WAMI Data Analysis

Veracity (truthfulness)

Known terrain

Given an image, predict the story

quality

Completeness (Area requested)

(Day-Night)

Accuracy (spot requested - CEP)

Precision (point requested)

Specificity (reduce false alarms)

E. Blasch, P. C. G. Costa, K. B. Laskey, H. Ling, and G. Chen, “The URREF Ontology for Semantic Wide Area Motion Imagery Exploitation,” Proc. of the Seventh International Conference on Semantic Technologies for Intelligence, Defense, and Security (STIDS), Fairfax, VA, October, 2012, pp. 88-95


Example 2: Machine Analytics: URREF E. Blasch, P. C. G. Costa, K. B. Laskey, H. Ling, and G. Chen, “The URREF Ontology for Semantic



Example 2: URREF


Example 2: Machine Analytics: CoT E. Blasch, P. C. G. Costa, K. B. Laskey, H. Ling, and G. Chen, “The URREF Ontology for Semantic



Example 3: Space Situational Awareness

RSO (Resident Space Object)

X. Tian, G. Chen, T. Martin, K. C. Chang, T. Nguyen, K. Pham, E. Blasch, “Multi-entity Bayesian network for the handling of uncertainties at SATCOM Links,” Proc. SPIE, Vol. 9469, 2015.




Alexander P. Cox, et al.” The Space Object Ontology,” Int’t. Conf. On Information Fusion, 2016




Alexander P. Cox, et al.” The Space Object Ontology,” Int’t. Conf. On Information Fusion, 2016


Summary

Low-level Information Fusion (Context)

Cyber-Physical Systems (CPS) and Internet of Things (IoT)


High-Level Information Fusion (Content)

Ontologies and user involvement (visualization)


Challenges in Information Fusion



Examples


Machine Analytics

Space tracking


Outline










4) Examples


Machine Analytics

5) Summary


Previous Texts

Focus of the text [1] Bosse, E., Roy, J., and Wark, S., Concepts, Models, and Tools for Information Fusion,

Artech House, Inc., Norwood, MA, 2007.

[2] Lambert, D. A., “Grand Challenges of Information Fusion,” Intl. Conf on Info. Fusion,

2003.

[3] Blasch, E. P., Llinas, J., Lambert, D. A., Valin, P., Das, S., Chong, C-Y., Kokar, M. M,

and Shahbazian, E., “High Level Information Fusion Developments, Issues, and

Grand Challenges – Fusion10 Panel Discussion,” Intl. Conf on Info. Fusion, 2010.

[4] Lambert, D. A., “A Blueprint for Higher-Level Fusion Systems,” Journal of Information

Fusion, Vol. 10, No. 1, pp. 6 – 24, 2009.

[5] Waltz, E., and Llinas, J., Multisensor and Data Fusion, Artech House, Norwood, MA,

1990.

[6] Blackman, S., and Popoli, R., Design and Analysis of Modern Tracking Systems,

Artech House, Norwood, MA, 1999.

[7] Das, S., High-Level Data Fusion, Artech House, Norwood, MA, 2008.

[8] Waltz, E., Knowledge Management in the Intelligence Enterprise, Artech House,

Norwood, MA, 2003.

[9] Hall, D. L., and Jordan, J. M., Human-Centered Information Fusion, Artech House,

Norwood, MA, 2010.

[10] Lambert, D. A., “Unification of Sensor and Higher-Level Fusion,” presentation at Intl.

Conf on Info. Fusion, 2006.

Documents

Fundamentals and Principles of Information Fusionstids.c4i.gmu.edu/papers/STIDSPresentations/STIDS16_Blasch_Keynot… · Low-level Information Fusion High-level ... Shape video and