T OPIC 9: S YSTEM E NGINEERING David L. Hall. T OPIC O BJECTIVES Discuss issues associated with the definition, design and implementation of a data

TOPIC 9: SYSTEM ENGINEERING

David L. Hall

TOPIC OBJECTIVES

Discuss issues associated with the definition, design and implementation of a data fusion system

THE IMPLEMENTATION ENVIRONMENT

Combined Hardware and Software DevelopmentCombined Hardware and Software Development Procured commercial hardware Specialized hardware Large-scale, real-time software

Complex System ArchitectureComplex System Architecture Multiprocesssing Distributed database and processing External interfaces

Stringent (and Changing) RequirementsStringent (and Changing) Requirements Throughput/response requirements Platform constraints (air/land/sea) Deployment environment Reliability/availability

Multiple Users and DevelopersMultiple Users and Developers Security RequirementsSecurity Requirements

Multi-level securityMulti-level security

Enterprise Services EnvironmentEnterprise Services Environment Service Oriented Architectures (SOA)Service Oriented Architectures (SOA)

FUSION IN THE CONTEXT OF REAL SYSTEMS

Type N Sensor

• Obs Modeling• Resource

Optimization• Goal Prorgamming

• Obs Correlation• Parameter Estimation• Pattern Recognition

Sensor Management

Single Sensor Obs Processing

Single Sensor Obs Processing

Multiple Sensor Obs Processing

Decision Aid/Planning Support

• Situation Assessment• Planning

• Multiple Sensor Correlation

• Parameter Estimation• Pattern Recognition

HCI

Analyst

Sensors

• Input Message - Event Detection - Alerts - Routing• Output Message - Routing

Me

ss

ag

e

Pro

ce

ss

ingData

Processing

• Smoothing• Preliminary Target I/D

SensorTasking

Co

mm

un

ica

tio

ns

Dat

a B

ase

Man

agem

ent

Analyst. . .

PROFILE OF DATA FUSION SOFTWARE FUNCTIONS

Dat

a F

usi

on

Dat

a P

roce

ssin

g

Dat

a M

an

agem

ent

OP

Sy

stem

Co

mm

un

icat

ion

s

Har

dw

are

Co

ntr

ol

S

pec

ial

Dis

pla

y

Use

r In

terf

ace

Pro

cess

ing

Co

ntr

ol

15

10

5

20

Per

Cen

t o

f T

ota

l S

oft

war

eP

er C

ent

of

To

tal

So

ftw

are

Categories of Software Functions

REQUIREMENTS FLOWDOWN PROCESS FOR SENSORS

REQUIREMENTS FLOWDOWN PROCESS FOR DATA FUSION

Sensor Performance Requirements

Data Fusion Systems

Requirements

Subsystem Design

Process

Processing System Design

Sensor Systems Design

• Coverage

• Detection

• Tracking

• Classification

• Algorithms

• Database

• Architecture

Display System Design

• HCI

Communications Design

• Capacity

• Architecture

Algorithm Requirements

Process Requirements

Display Requirements

Sensor Requirements Communication

Requirements

Performance Analysis

Simulations

SCENARIOS

Test/Evaluation Requirements

DISTRIBUTED FUSION ARCHITECTURES

F

C

F

F

F

C

C

F

F

F

F

C

C

F

C

Sensor Node

Fusion NodeInformation Consumer Node

Centralized Hierarchical Distributed

SS

S

S

S

S

S

S

S

SS S

S

S

S

S

S

S

S

Sensor Data Flow

Knowledge Level Info. Flow

Inference Message Flow

Courtesy, M. Liggins

INFERENCE ARCHITECTURES – CENTRALIZED

Inference Link(association or causal link)

C

S

S

S

S

S

S

-Inference NodeReceive Data from Sensors

Fusion Node

C - Output/query Node

F

CS

S

S

S

S

S

GENERIC TRACKER/CORRELATOR ARCHITECTURES: CENTRALIZED

FUSION

SensorA

SensorB

SensorN

Preprocessing

Preprocessing

Preprocessing

•••

Sensor Controls

Gating and Control Parameters

Target State

Processed Sensor

Data

• Target Classification• Probability of Successful Declaration

CorrelationData Alignment and Association

Composite Filtering

Classification

Level 1 Fusion

GENERIC TRACKER/CORRELATOR ARCHITECTURES: AUTONOMOUS (OR

DISTRIBUTED) FUSION

Sensor Controls


Target State

Tracking and Classification Parameters • Target Classification

• Probability of Successful Declaration


Composite Filtering

Classification

Detection and Estimation

Tracking & Classification



SensorA

SensorB

SensorN

Preprocessing

•••

Preprocessing

Preprocessing

GENERIC TRACKER/CORRELATOR ARCHITECTURES: HYBRID FUSION

Select and MergeMUX

Sensor Controls


Target State

Tracking and Classification Parameters • Target Classification

• Probability of Successful Declaration


Composite Filtering

Classification

Detection and Estimation




SensorA

SensorB

SensorN

Preprocessing

•••

Preprocessing

Preprocessing Detection Parameters

SUMMARY OF ALTERNATIVE FUSION ARCHITECTURES

• Minimum information loss• Requires commensurate

sensors• Difficult association problem• Much communication between

sensors/fusion system

ARCHITECTURE TYPE COMMENTSLEVEL OF

INFORMATION FUSEDAPPLICABLE TECHNIQUES

Centralized RAW DATA - direct fusion of sensor data:- Sample signal- Imagery

• Physical models• Pattern recognition• Estimation techniques

• Information loss due to feature extraction

• Large dimension composition feature vector

• Simplified association• Allows non-commensurate

sensors• Reduced communications

• Significant information loss• Local optimization may

prohibit global optimization• Allows non-commensurate

sensors• Minimum communication

Must account for statistical interdependence

• Combines features of centralized and autonomous architectures

• Complex control logic• Increased communications

requirement

Distributed/Autonomous

Hybrid

FEATURE VECTORS:- Attribute features- Locational/ kinematic parameters

DECISION-LEVEL INFORMATION:

- State vectors- Identity

declarations

COMBINATION OF:- Raw data- Feature vectors- Decision level

• Pattern recognition• Estimation techniques• Cluster analysis• Neural networks• Parametric templates

• Estimation techniques• Bayesian inference• Dempster-Shafer's

method• Logical templates• Voting methods

• All of above techniques

Fusion of raw data from sensors

Fusion of derived data (features) from sensors

Fusion of state vectors or identity declarations

Combination of centralized and autonomous architectures

DESCRIPTION

TEST AND EVALUATION ISSUES/COMPLEXITIES

Numeric/Symbolic Components - Testing of Algorithms and Knowledge Acid Tests - with/without fusion

Dominant sensor (1,2,3, … N) source effects/marginal gain Comparison between fusion techniques

Measuring Global Optimality Standards/expectations for optimality (detection level versus mission-level) Complex role of feedback in optimization Broad range of architectural choices (e.g., pixel fusion versus decision level)

Stochastic Details Validity of simplifying assumptions Often interested in rare events (e.g., situations)

No Gold Standard in Situation and Threat Assessment (What constitutes the quality of a situation assessment?)

FOUR CATEGORIES OF MEASURES OF MERIT

MEASURE DEFINITION TYPICAL EXAMPLES Measure of Force Effectiveness (MOFE) Measure of how a C3 system and the force of

which it is a part perform military mission (sensors, weapons, C3 system)

Outcome of battle Cost of system Survivability Attrition rate Exchange ratio Weapons on targets

Measures of Effectiveness (MOE) Measure of how a C3 system performs its functions within an operation environment

Target nomination rate Timeliness of information Accuracy of information Warning time Target leakage Countermeasure immunity Communication survivability

Measures of Performance (MOP) Measures closely related to dimensional parameters (both physical and structural), but attributed to behavior

Detection probability False alarm rate Location estimation accuracy Identification range Time from detection to transmission Communication time delay Sensor spatial coverage Target classification accuracy

Dimensional Parameters The properties or characteristics inherent in the physical entities whose values determine system behavior and the structure under question, even when not operating

Signal-to-noise ratio Operations per second Aperture dimensions Bit error rates Resolution Sample rates Anti-jamming margins Cost

Courtesy of J. Llinas

EXAMPLE OF A FUSION EVALUATION TEST BED

(MITRE CORPORATION)

Post Processing Data Analysis Tools

• Graphs• Statistical Tests between Algorithms

Sensor Models Module

• RADAR• IFF• ESM• Hypothetical Sensor A• Hypothetical Sensor B

Measures of Effectiveness

Output Module

• Track Kinematics• Track Maintenance• Correlation• Classification Identification

Fusion Algorithm Under Test

Live Recorded Data Module

Scenario Generation Module

• Target Files• Platform Files• Scenario Generation File• Administrative Data File

Target/Platform Motion

and Dynamics

Module

Files

Files

DATA FUSION THRUPUT PARAMETRICS

ASSUMPTIONS: • Single-level Tracking• Fixed Processing Loop per Input Report

N = ABCD

R

MOPS

D

C

B

A

H2 Hypothesis

S Searches/Tgt

J MOPS/UP

X

Data Association

Tracking

Overhead

Track File Data Base

Management

Data Combination

Sensor Management

Backward Logical

Inferences

R

H/RM Sensors

N Targets

S1 Visits/Sec

S2 Visits/Sec

•

•

S5 Visits/Sec

Revisit Model

Target Density Model

MOPS+

PARAMETRIC FACTORS IN FUSION PROCESSING

APPLICATION PARAMETERS• Number of Sensor Contributors• Sensor Spatial Resolution• Measurement Bandwidth/

Preprocessing• Search Volume• Target Density• Track/ID Response Time• Scenario Force Mix• Number of Target Classes• Feature/Cond. Probability Variance

ALGORITHM FACTORS• Correlation Track Generalization and

Cluster Approach• Degree of Multi-scan and

Recursiveness• Parametric/Non-parametric

PROCESSING PARAMETERS

• Track Storage

• Dimensionalityof decision space

• Decision Rate(target decisions/sec)

• Decision Rule Storage

(parametric/non-parametric)

• Tracking Recursion

• Track File Interaction

• Decision Rule Test Rate

COMPUTATION PARAMETERS

• Correlation/Classification Arithmetic Operation Rate (MIPS/MOPS)

• I/O Transfer Rate

• Track File Capacity

• Classifier Database Capacity/Access

OPERATIONS ANALYSIS: TRANSACTION ANALYSIS

Sensor 1

Sensor 2• • •

Human Interface

Transaction A

Transaction B

Transaction N

• • • S

yste

m R

eso

urc

e A

nal

ysis

Sys

tem

Re

sou

rce

An

alys

is

• Resource Requirements• Capacity• Response Time

• • •

Analysis Results

• MIPS/MOPS• Transmission BWs• Access Speeds

• • •

System Resources

Ra

tes

TransactionsA B N

• • •

EXPLOITATION OF PARALLELISM

EVOLUTION OF COMPUTING POWER

http://www.en.wikipedia.org/wiki/Moore’s_law

EXAMPLES OF DATA INVOLVED IN MSDF

DATA DATA DESCRIPTION INTERPRETATIONModel Parameters Sensor characteristics

Sensor locations Physical constants Time delays, biases, and model

coefficients

Characteristics modelinformation for observationprediction, state vectorpropagation, and sensor dataprocessing

Sensor Data Observations- State vectors- Attribute/feature vectors- Raw signal or imagery

Observed data from sensors

External Databases Previous observations Data from external sensors or

fusion systems

Related data from other fusionsystems or external databases

Human Input Control information Inferences/hypotheses Requests for data Annotations

Data input by humans forsystem control or data update

Environmental Data Geography/topology/hydrology Weather Environmental conditions Man-made objects (e.g., cities,

roads, and networks)

Information about the static ordynamic environment in whicha situation is observed

EXAMPLES OF DATA INVOLVED IN MSDF

DATA DATA DESCRIPTION INTERPRETATION Situation Database (Level 2)

Location, identity of entities Relationships among entities (physical,

communications, hierarchy, sequential, etc.) Interpretation of order-of-battle

Results of Level 2 processing to achieve a dynamic interpreta-tion of entity relationships, activities, or events

Threat Database (Level 3)

Location, identity of threatening conditions or entities, sensors

Characteristics of threatening conditions (threat envelopes and sensors)

Probable courses of action Interpretation of possible consequences

(intent, lethality, and opportunity)

Results of Level 3 processing aimed at determining an enemy threat to friendly or neutral forces

Performance Data (Level 4)

Assessment of accuracies of fusion products Measures of performance Measures of effectiveness Objective function Constraints

Data that characterizes and allows control of the dynamic sensor observation and fusion process

A Priori Data Knowledge bases (rules, frames, scripts, etc.) or target and event behaviors, enemy tactics, doctrine

Technical data (e.g., enemy sensor performance, weapon characteristics)

Mission data - Objective and goals - Strategies - Tactics

Data developed by a priori (usually extensive) analysis to provide a basis for knowledge-based fusion processing

PUSH/PULL FOR SERVICE ORIENTED ARCHITECTURES (SOA)

Technology Push Requirements PullSOA

- Rapid evolution of Internet - Commercial practices - Wireless, mobile communications - Smart sensor/processors - Utilization of web-based models and tools - Etc

- Heterogeneous smart sensors - Distributed team environments - New threats requiring multi-force, multi-org. participants - DoD directives

SERVICE ORIENTED ARCHITECTURE

Service-Oriented Architecture – a component model that inter-relates the different functional units of an

application, called services, through well-defined interfaces and contracts between these services

The interface is defined in a neutral manner should be independent of the hardware platform, the operating system, and

the programming language the service is implemented in

This allows services, built on a variety of such systems, to interact with each other in a uniform and universal manner

SERVICE ORIENTED ARCHITECTURE

SecurityServices

MonitoringServices

ServiceRegistries

MessagingServices

DataServices

TransformationServices

Service Enabled InfrastructurePublish

Data and applications available for use, accessible via services. Metadata added to services based on producer’s format.

Service Producer

• Describes content using metadata• Posts metadata in catalogs for discovery• Exposes data and applications as services

Discover

Invoke

Automated search of data services using metadata. Pulls data of interest. Based on producer registered format and definitions, translates into needed structure.

Service Consumer

• Searches metadata catalogs to find data services

• Analyzes metadata search results found• Pulls selected data based on metadata

understanding

SecurityServices

MonitoringServices

ServiceRegistries

MessagingServices

DataServices


Service Enabled Infrastructure

SecurityServices

MonitoringServices

ServiceRegistries

MessagingServices

MessagingServices

DataServices

DataServices


Service Enabled InfrastructurePublishPublish


Service Producer



Service Producer


DiscoverDiscover

InvokeInvoke


Service Consumer



understanding


Service Consumer



understanding

(Post) (Find)

(Bind)

From: GiG Enterprise Services Piloting; Rob Walker, DISA April 2004

LocatorServices

Measuring and DisseminatingReal-World Information

Report MgtServices

WeatherServices

Decision Making

Planning and ConductingOperations

AssociationServices

Organizing and ManagingData and Information

Entity MgtServices

WorkspaceServices

Process MgtServices

Organizing and ManagingWorkflow and C2 Processes

WorkflowServices

Resource MgtServices

OverlayServices

OceanographyServices

Sense &RespondServices

AlertServices

NGC2* SupportServices Coordinated

ActivitiesServices

VisualizationServices

FusionServices

Sense Making andBattlespace Understanding

IntelServices

ReadinessServices

ForceProjectionServices

Air/SpaceOperations

Services

Joint Fires& Maneuvers

Services

ExecutiveSummaryServices

ForceProjectionServices

SituationalAwareness

Services

. . .

UserManagement

Services

. . .

. . .

. . .

. . .

. . .

. . .

MCPs

Core C2 SupportTools & Functions

NGC2*

CES* DiscoveryServices

Hosting and enterprise accessto information

MediationServices

MessagingServices

ESM*Services

SecurityServices . . .

C2 CoI*CommonServices

* MCP – Mission Capability Packages NGC2 – Next Generation Command and Control COI – Community of Interest CES – Core Enterprise Services ESM – Enterprise Service Management

HIERARCHY OF ANTICIPATED SERVICES

From: GiG Enterprise Services Piloting; Rob Walker, DISA April 2004

ANTICIPATED ADVANTAGES OF SOA

Leverage Existing Assets ( Don’t throw away existing systems) Support all types or “styles” of integration.

User Interaction Application Connectivity Process Integration Information Integration Build to integrate

Allow for incremental integrations and migration of assets Include a development environment that has the following features

Should be built around the component framework Promote better reuse of modules and systems Allow legacy assets to be migrated to the framework Allow for timely implementation of new technologies

Allow implementation of new models: Specifically new portal-based client models, Grid computing On-demand computing

SOA TRANSITION PERSPECTIVES

Implementation realities – how do we transition from core services to an effective system?

Impact on operations – how do we really operate in an SOA environment?

Maintenance nightmares – will SOA be a maintenance nightmare?

IMPLEMENTATION ISSUES (EXAMPLES)

Development of application-level components & services – if we build it they will come

Amount & level of imposed structure - level of federation versus free market

Demands and changes for legacy systems – your problem versus my problem

Growth & evolution of core services – where do we stop? Development of fundamental standards, ontologies, templates – tower

of Babel Security – multi-level security etc Inclusion - boundaries of services & components

OPERATIONAL ISSUES (EXAMPLES)

Use of dynamic discovery/link capability – how will real users use dynamic link capabilities?

Human in the loop processing – role of human as consumers & suppliers

Hierarchical heterogeneous users – different needs, capabilities & priorities

Evolution & changes to components – introduction of new components & changes

Adjudication of resource demands – contention, control of user/suppliers

Impact on legacy systems Etc.

SYSTEM MANAGEMENT (EXAMPLES)

System configuration management Rules of engagement of component entrance Version thrashing Upgrades

System maintenance Core services Components

Level and type of support services Ownership and responsibilities Etc

THE SECURITY ENGINEERING PROCESS

Operational Objectives

Formal Security Policy and Guidance

System Concept

Formal Requirements

Engineering Analysis/Trade-Offs

Security CONOPS

System Security Policy

Secure Applications

Assurances

Software Security Model

CHECKLIST TO SUCCESSFULLY BUILD A DATA FUSION SYSTEM

Do You Understand the Application?

What is the mission? What decisions/inferences are required? What are the timeframe/rates for decisions? Who are the system users? How do decisions affect the mission? What is the typical situation/threat environment? What is the appropriate MOE/MOP? What are the platform/system constraints? What are the other data sources? Is real data available? Do we have (former) users/analysts available? How does mission environment affect system architectures?

Do You Understand the Inference Process?

What is processing chain from energy detection to inferences?

Do you understand the cognitive process for effective inferences?

Can you characterize effective versus ineffective inferences?

What are the barriers to correct inferences (human limitations, CM, etc.)?

What is the use of negative information? What is the decision environment (stress, decision styles,

doctrine, etc.)? What are the effective/applicable processing techniques?

SOFTWARE TOOLS AND SYSTEMS

http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software (Comparison of agent-based software)

http://www.lionhrtpub.com/orms/surveys/FSS/fss9.html (survey of estimation software tools) http://cqm.cs.mcgill.ca/~godfried/teaching/pr-web.html (large set of links to training materials,

tutorials, and software tools for pattern recognition) http://onlinelibrary.wiley.com/doi/10.1002/widm.24/full (extensive list and comments on software

for data mining, data warehousing, and knowledge discovery tools) http://www.cs.cofc.edu/~manaris/ai-education-repository/expert-systems-tools.html (directory of

software for expert systems and knowledge-based systems) http://www.lionhrtpub.com/orms/surveys/sa/sa-surveymain.html (survey of 125 vendors of statistical

software) C. N. Mutchler, “An investigation of tools/systems for multi-source data fusion at the national air

intelligence center,” Aerospace Technical Report TOR-20001(1758)-0396e ;2001 S. A. H. MacMullen and R. Sherry, “A survey of COTS software for multi-sensor data fusion: what’s

new since Hall and Linn?”, Proceedings of the MSS National Symposium on Sensor and Data Fusion (San Diego, CA 13-15 August, 2002)

TOPIC 9 ASSIGNMENTS

Preview the on-line topic 9 materials Read chapter 10 of Hall and McMullen (2004) Read Bowman and Steinberg (chapter 16) in

Handbook of Multisensor Data Fusion

DATA FUSION TIP OF THE WEEK

Picture: Stuart MacFarlane,London, Daily News, October 1999.

The process of designing and implementing a data fusion system may seem overwhelming and painful – indeed numerous data fusion systems don’t actually fuse anything (they develop infrastructure software and run out of time/funding before implementing the fusion software). On the positive side there are significant resources available to support a successful implementation including COTS software.

Documents

T OPIC 9: S YSTEM E NGINEERING David L. Hall. T OPIC O BJECTIVES Discuss issues associated with the definition, design and implementation of a data