View
71
Download
0
Category
Tags:
Preview:
DESCRIPTION
Introduction and Overview of Data and Information Fusion. James Llinas Research Professor, Director, Emeritus Center for Multisource Information Fusion University at Buffalo llinas@buffalo.edu. CMIF's Approach: A Total Fusion Systems Perspective. Basic through Advanced R&D In: - PowerPoint PPT Presentation
Citation preview
Introduction and Overview of Data and Information Fusion
James LlinasResearch Professor, Director, Emeritus
Center for Multisource Information FusionUniversity at Buffalollinas@buffalo.edu
CMIF's Approach: A Total Fusion Systems Perspective
• Basic through Advanced R&D In:― IF Based Situation and Threat Assessment― Multiple-Sensor System Data Integration/Analysis― Multi-Intelligence Information Environments
― COMINT, ELINT, OSINT, HUMINT, IMINT…..
― Graph Analysis Methodology and Systems
• Applications:― Defense: C4ISR Support and Tactical Applications― Non-Defense: Disaster Consequence Management,
Critical Infrastructure Protection, Medical Informatics
Intelligence Analysis
Situation and Threat Assessment for Multiple Application Domains
Focus on High Level Information Fusion and Graph Analytics
Maritime Domain Awareness
Contents are CMIF Proprietary
CMIF Foundational Sciences and Methods
3
Sciences• Graph Theory• Information Fusion
• Situation Assessment• Threat Assessment• Resource Management
• Information Theory• Mathematical Optimization• Modeling and Simulation• Knowledge Representation and
Reasoning (KRR)
Business/Engineering• Systems and Software
Engineering• Product Lifecycle Management• Service Oriented Design• Rich Web Applications• Complex Event Processing• Database Design• Realtime Systems
Slide 28Slide 28
Maximizes UAV surveillance with consideration for fuel consumption
Surveillance benefit of sensor s at target j can’t exceed the demand of sensor s at target j
Total number of each sensor s can’t exceed the number available at the base
Conservation of sensor delivery
UAV sensor capacity
Route continuity
A target may not be visited by the same UAV multiple times
All UAVs must begin at base location (location 0)
Route assignments cannot exceed total adjusted flight time
Sensor delivery may not take place unless a target is visited with the appropriate sensor attached
Maintains the time that each target is visited and ensures this time falls within the desired time window
Integrated Sensor Selection and Routing Model
CMIF STEF System – Main Window
Messages
Target Graphs
Click to match single Target graph with a
selected Message graph
Click to match all Target graphs with a selected Message graph
ComplexMathematicalOptimizations
Advanced Techniques In
Linguistic Processing
ModernGraphical Methods
Contents are CMIF Proprietary
Sampling of Methods
Collaborative Decision Support:CMIF Command and Control Lab
Center for Multisource Information Fusion (CMIF)Flexibility in Research and Development
• Core Technology Research & Development– State University of New York at Buffalo, NY, USA
• Top level research university and scientific staff— focus is basic research and proof of concept experiments– In existence 20+ years (Average grant revenue ~US$8M)
• Collaborative Partner for Technology Transition:– CUBRC, a not-for-profit defense R&D organization in Buffalo, New York, USA (~135 people, ~US$35M)
• Cleared, experienced staff, facilities— focus is development and transition
UB & CUBRC
UNIVERSITY at BUFFALO
BroadBaseOfDefenseR&D
Core Technology R&D
Transition & “Hardening”
History of Information Fusion• Dates to early 80’s—fairly young in the sense of
technological history—a maturing technology/field of study
• Driven by defense and intelligence needs– Originally as a “data compression” device to digest huge
amounts of sensed data as sensors advanced in capability (a “push” requirement)
– Later as an important element for decision support (a “pull” requirement)
• Matures to very broad range of application– Robotics, medicine, imagery/remote sensing, intelligent
transportation, conditioned-based maintenance, biometrics, medicine, etc
What is (Automated) Information Fusion?
Information fusion is an Information Process (Software) comprising:• FUNCTIONS:
• Alignment• Association, correlation • Combination of data and information from
• INPUTS:• Single and multiple sensors or sources to achieve
• OUTPUTS: • Refined Estimates of :
• parameters, characteristics, events, behaviors and relations for/among observed entities in an observed field of view
It is sometimes implemented as a Fully Automatic process or as a Human-Aiding process for Analysis and/or Decision Support
Data Fusion: Definition
" Data Fusion is the process of combining data (or information) for the purpose of estimating or predicting some aspect of the world"
Steinberg, Bowman, and White, “Revisions to the JDL Data Fusion Model”, NSSDF 1998
Multiple types of data
--various types of information
--redundant
--and complementary)
“Associated” or “Correlated” to :
--the same object
or event
or behavior
So that estimation algorithms (mathematical techniques)—or—automated reasoning methods (artificial intelligence techniques) can produce better estimates (than based on any single type of data)
Multiple types of data Related to things of interest
To improve estimates about those things
These Basic Ideas are Transferable to Many Types of Problems
RealWorld
Most Simply--
Observation System
Observations(Multiple)
Associationof Observations
Estimation
Basic Role of Fusion
(Dynamic) Real
States in the World
ObservationalMeans
(Streaming)
Common ReferencingAlignment
EstimatesOf World States
Dec-MkgAnalysis
etc
• One means to satisfy user information needs for decision/analysis support, i.e., most frequently inserted to support human user
Data Association
Evaluation
Actions
Process Refinement
Requirements driven from here
Everyday Data Fusion
Pain
Taste
Images
Touch
Smell
Temperature
Sound
Balance
Body Awareness(Proprioception
Multinodal Fusion
Augmented Sensing
RoboticMultisensor Fusion
[12]
Sensor Fusion Exploits Sensor Commonalities and Differences, Knowledge of Errors
Data Association Uses Overlapping Sensor Capabilities so that State Estimation Can Exploit their Synergies
CONFIDENCE COVERTCOVERAGE
DETECTIONRANGE ANGLE
KINEMATICSCLASS TYPE
CLASSIFICATION
POOR GOOD FAIR FAIR POOR
FAIR POOR GOOD FAIR FAIR
FAIR GOOD FAIR FAIR FAIR FAIR
RADAR
EO/IR
C3I
GOOD GOOD GOOD GOOD GOOD GOOD
DataAlignment
DataAssociation
StateEstimation
SENSORFUSION
FAIR
FAIR
Unknown Moving Object
[13]
A Persistent Focus: Reduced Uncertainty
Multisource Data (Evidence) Association
STATEESTIMATION
& PREDICTION
STATEESTIMATION
& PREDICTION
STATEESTIMATION
& PREDICTION
STATEESTIMATION
& PREDICTION
DATAASSOCIATION
“Assigned” ObservationsResulting from some “Best” way to decide which Observations should be “given” to
each State Estimator
M ObservationsFrom N Sensors Tracks “T”
Need some type of Mappingthat determines a goodway to allocate ObsvnsTo Tracks
mimjmm
O TkTj TlTi
Multiple Observations &
Multiple Entities
What a “Message” looks like— a Graphical Structure
An Observational Evidence “Atom“--Not a Point Measurement--
An Observation (description—Representation of an Observation)
Multiple Relationships
Disconnected semantic fragments
Synonyms
Generally all elements have some type of
imperfection or error
Some errors Quantified
Includes judgments as well as
observations
Design of the Association Process for Linguistic (Msg) Inputs
Human Observer 1
Human Observer 2
HypothesesSelf-generatedby node/arccontent
HypothesesScored viaSemanticSimilarity Scores that accnt For uncertainty
Pick a node/arc,Search other graphFor associable elements(eg exploit ontology)
Smart GraphSearch
HypothesesEvaluationBy high-dimensionalAssignment problemsolution
Apply JVC or otherModern assignmentProblem solution
Good AssignmentSolution &
Graph Merging
Effective
Semantic
scoring
Interdependency withText, Semantic
Operations
Linguistic, Textual
(Semantic) Inputs
Fusion of Realtime Data and A Priori Data Bases
DATA INFORMATION KNOWLEDGE UNDERSTANDING
Terrain/Cultural FeaturesImagery Overlays
Logistics
IntelligenceWeather
Coalition Forces
Situation
• Intel Sources• Air
Surveillance• Surface
Surveillance• Space
Surveillance
Decision Maker
Decision-SpecificInformation
• Timely • Accurate • Consistent • Structured• Integrated
FusionTechnology
Realtime
A Priori and Realtime
“Context”Today—Includes
--Sociocultural Info--Social Media
Data Fusion Functional Model(Jt. Directors of Laboratories (JDL), 1993)
Level 0 — Sub-Object Data Association & Estimation: pixel/signal level data association and characterization
Level 1 — Object Refinement: observation-to-track association, continuous state estimation (e.g. kinematics) and discrete state estimation (e.g. target type and ID) and prediction
Level 2 — Situation Refinement: object clustering and relational analysis, to include force structure and cross force relations, communications, physical context, etc.
Level 3 — Impact Assessment: [Threat Refinement]: threat intent estimation, [event prediction], consequence prediction, susceptibility and vulnerability assessment
Level 4: Process Refinement: adaptive search and processing (an element of resource management)
Level 0ProcessingSub-object DataAssociation &
Estimation
Level 1Processing
Single-ObjectEstimation
Level 2Processing
SituationAssessment
Level 3Processing
Threat/ImpactAssessment
Level 4ProcessingAdaptive Process
Refinement
Data BaseManagement System
SupportDatabase
FusionDatabase
INFORMATION FUSION PROTOTYPEJEM
JWARN3GCCS• Point and
Standoff Sensors• Data Sources• Intel Sources• Air Surveillance• Surface Sensors• Standoff Sensors• Space
Surveillance
Methods:--Combinatorial Optimization
--Linear/NL Estimation--Statistical
--Knowledge-based--Control Theoretic
DetectionTracking
ID
AggregationBehavior
Events
LethalityIntent
Opportunity
Sensor MgmtProcess Mgmt
• Reliability• Improved Detection
• Extended Coverage(spatial and temporal)
• Improved SpatialResolution
• Robustness (Weather/visibility, Countermeasures)
• Improved Detection• Improved State Estimation
(Type, Location, Activity)
• MultipleSensors
Operational Benefits of Multiple SensorData Fusion
State Estimates of Reduced UncertaintyAnd Improved Accuracy
• DistributedSensors
• DiverseSensors
Information Fusion: The Defense Context
“Associated” or “Correlated” to the same object or event or behavior
Multiple types ofsensor data
Related to thingsof interest in theReal World
To improveestimates aboutthose things
RealWorld
Fusion (Estimation)Techniques
In the defense problem:• Non-cooperative,• Unfriendly• Deceptive
!
Today’s IF Process Design Environment:Information-space Motivation: Exploitation of all Information
Modern Fusion Process
Observational Data
Sensor Observations
Human Observations
Dynamic Real World
Language
Numbers
Semantic Label
Contextual Information
Weather
Cultural
Financial
PoliticalWeb
A Priori Dynamic World Model
World State EstimatesLearning Processes
“SOFT”“HARD” “HARD & SOFT”
ChatTwitter
L4 Knowledge Mgmt
Declarative Knowledge:Ontologies
The Soft Front-end Input
UnconstrainedVocabulary
(Possibly different languages)
Semantics
Language Processing
AutomatedText
Extraction
~ RDF Triples (++)Typical
Atomic, RawData Input
(Digitized)
Computational Linguistics,NLP
Trained Observer
Untrained Observer
Interview
Bystander
Source Characterization
Perceptual and Cognitive Errors in observation
RealWorld Truth
Error in oral expression
Error in audio capture
Error in audio -to -text
conversion Error in text extraction
Conversion
Soft Data
To Common Ref, Data Association
є4
є5
є2
є3
є1
Hard Data Calibration
(Truth)Target
Pd (Obs Params)
To Common Ref, Data Association
Data Characteristic
Hard Soft Remarks
Observation sampling rate
High Low Imputes requirements for adaptive, retrodiction-type processing (i.e. “Out-of-Sequence Measurement” type processing), as well as agile Temporal Reasoning
Semantic Content Limited to specific, usually singular Entities
Can be conceptually broader than single Entities
Imputes requirements to design an automated Semantic Labeling process, coupled to a rich Domain Ontology Requires ability to associate and infer at multiple levels of abstraction
Limited to Entity Attributes
Can include Judged Relationships
Accuracy, Precision
Relatively high, good repeatability (Precision)
Broadly low accuracy in attributes, high at the conceptual level
Imputes requirements for robust Common Referencing and Data Association
Totally distinct from Hard SensorsPhilosophy: Relations not directly
observable—require reasoning over properties of entities
Brower, J., (2001) "Relations without Polyadic Properties: Albert the Great on the Nature and Ontological Status of Relations." Archiv für Geschichte der Philosophie 83: 225–57.
This line of thought suggests that relations are the result of a process of some type of comparison, ie [Brower, 2001], “an act of reasoning”.
Humans can also judge intangibles--emotional state
Some Distinctions in Hard and Soft Observational Data
Counterinsurgency Problem Environment
Soft Sensing Hard Sensing
MURIInformation Fusion
Technology
Counter-Insurgency
Soft Operations Kinetic Operations
COIN Decision Support
Not comprehensive
UNCLASSIFIED//FOUO
Context *
* Connable, B, Culture and COIN, www.citadel.edu/.../Connable,%20Culture%20and%20Counterinsurgency%20Brief.ppt
24
Some Remarks on Ontology and Information Fusion
Dr. James LlinasResearch Professor, Director (Emeritus)
Center for Multisource Information FusionUniversity at Buffalo and CUBRC
llinas@buffalo.edu
25
Roles for Ontologies in IF Processes/Systems• Reasonably reliable a priori Declarative Knowledge
about some domain– In the face of domains for which reliable a priori Procedural
(dynamic) Knowledge is hard to specify• “Weak Knowledge” problems
• As such, they provide a framework that connects Entities and Relationships– Of fundamental concern for COIN, Ctr-Terrorism, Irregular
Warfare re social structures and militarily-significant entity relations
• The basic construct of a “Situation” or a “Threat” and thus Level 2, 3 Fusion estimation
26
Complexities in Distributed and Networked Systems
• In modern Distributed/Networked Systems there are No single points of authority: These systems are collages of Legacy systems—Joint/Multiservice systems—Coalition systems
• Nodal Ontologies for Fusion/Situational Estimation, and Communication-support Ontologies for Inter-Nodal Communications/Data-sharing (eg JC3IEDM)
• Harmonizing NLP Operations and Ontologies within and across such systems
• The issue of Uncertainty in Ontological specification:– Probabilistic and Non-Probabilistic Ontologies
• Is there an Inescapable need for Semantic Mediation?– Mediator systems well-studied and developed*
• Eg Gio Wiederhold (June 1, 1993). "Intelligent integration of information". ACM SIGMOD Record 22 (2)(This was a major DARPA program) 27
Semantic Complexity• Controlling Semantic Proliferation/Complexity:
– Ontologies– Controlled Languages
• Eg Battle Management Language– Eg Shade, U., et al, From Battle Management Language (BML) to Automatic Information
Fusion, Chapter in Information Fusion and Geographic Information Systems, Lecture Notes in Geoinformation and Cartography,Popovich, V.V.; Claramunt, C.; Devogele, Th.; Schrenk, M.; Korolenko, K. (Eds.), 2011, Springer
• Understanding complexity drivers in text• Eg McDonald, D.D., Partially Saturated Referents as a Source of Complexity in
Semantic Interpretation, Proceedings of NLP Complexity Workshop: Syntactic and Semantic Complexity in Natural Language Processing Systems, 2000
• Measuring Semantic Complexity• Eg, Pollard, S and Biermann, A.W., A Measure of Semantic Complexity for Natural
Language Systems (2000) Proceedings of NLP Complexity Workshop: Syntactic and Semantic Complexity in Natural Language Processing Systems, 2000
28
The Association Problem
• The Ontologically-specified World is controllable—the Real Data World is not• While Ontologies can help in Fusion-based estimation and inferencing
problems, the mechanics of exploitation will involve the associability of Real (uncontrolled) data to (controlled) Entities and Relations in the Ontologies– Semantic similarity, metrics, degree (“hops”), etc– Efficient algorithms—eg Cloud implementations– PhD-level research
• There is also the issue of “Coverage”—in poorly-understood/known problems, how does one specify an Ontology that has “adequate” coverage?– Issue of negative information
29
Summary
• Ontologies have a useful role in the design and development of Information Fusion systems
• Questions regarding issues of:– Authoritative control of semantics in distributed systems
• Acceptable, optimal methods for mediation– Complexity of semantics
• Understanding, measuring, controlling– Association of semantic terms and complex, high-dimensional
semantic structures • Seem to require further, continuing study to better define
best ways to employ ontological information in complex, distributed, large-scale Information Fusion systems and applications
30
Unified Research on Network-based Hard and Soft Information Fusion
• A CMIF 5-year “Multidisciplinary University Research Initiative (MURI)” Program– Funded by the Army Research Office; ~ $7M– UB/CMIF lead + Penn State + Tenn State
• Soft/NLP/Ontology Lead: Prof Stu Shapiro, CSE– Building “TRACTOR” Soft front-end
31
32
33
34
Recommended