Situation Awareness Telcordia’s E2A Architecture and Three Case Studies

Situation AwarenessTelcordia’s E2A Architecture and Three Case Studies

Dimitrios [email protected]

EPS, SF, November, 2006

Slide 2 Nov. 2006

Awareness

• Awareness is information packages (complex event objects, their pedigree, and related evidence) that are highly relevant to the situated needs of a user/event consumer

• Contextual relevancy- Events must be cast in terms of concepts (e.g., space, time, objects) familiar

to the user

• Situational relevancy- Delivered events must help each user perform the specific activities he/she

is working on or is responsible for

• Temporal relevancy- Events must be delivered in timely fashion to permit effective response

Slide 3 Nov. 2006

Events to Awareness Concept of Operations

Continuously analyze and

(re)contextualizeevents

Task an expertto evaluate a

situation and route related evidense

Perform more specialized

analysis & extraction tasks

Generate and route alerts

providing evidence

Receive alertsand relatedevidence

Perform event extraction

& analysis tasks

Detect anomalies

from observationof normal situations

Author awareness

specifications

Subscribe to awareness

Administrators

Users/other systems

E2A-based system

E2A activities and roles

Decompose subscriptions

to event sources and events they

can detectCapture context

info

Continuously detect event patterns in awareness

specifications

Event Processing System

Slide 4 Nov. 2006

Telcordia’s Events to Awareness Architecture (E2A)

SensorInterfaceSensor

InterfaceEvent Sourcesand Interfaces

Content Routing

andCoordination

AwarenessSpecifications

EventOntology

AwarenessComputation

(AC)

Users

Routing & TaskSpecifications

PrimitiveEvents

ContextualizedEvents

AxtionableEvents

(i.e. alerts & taskrequests)

Awareness

Proactive Event analysis Tasking

EventContextualization

(EC)

ContextContextcontexts

Legend Event flow Utilization Tasking

Continuous stream processing of events for real time event detection Event Subscriptions and tasks

Event repositiry

EventExtraction &

Analysis(EA)

Slide 5 Nov. 2006

E2A Component responsibilities

• Event contextualization - Injects primitive events- Contextualizes and fuses events

• Awareness Computation- Utilizes user-specified awareness specifications to

compute complex events continuously and incrementally- Proactively seek missing events

• Coordination- Manages alert and tasking interactions with end-users- Manages tasking of event sources

• Application context(s), event ontology, awareness specifications, and task specifications- Permits application-specific customization

Achieve contextual, situational, and temporal relevancy

Slide 6 Nov. 2006

Situation Awareness Case Studies

• Complex event sensing - Surveillance- Critical Infrastructure protection- Reconnaissance- Broadcast news analysis- UAVs/UASs

• Coordination and adaptation- Intelligence gathering involving collaboration of large multi-

organizational teams- Disaster/crisis mitigation

• …at a large scale- Blue Force Tracking (DoDs Net-Centric Data Strategy)

Slide 7 Nov. 2006

The Surveillance Problem

Slide 8 Nov. 2006

Providing Situation Awareness in Video Surveillance

• Provide situation awareness by automatically delivering alerts and related evidence to the appropriate users

• Situation understanding involves determining the causes of an alert

• Supports situation understanding via event drill down

• Users can view constituent events and evidence

Slide 9 Nov. 2006

Surveillance Case Study

• Event sources- Video cameras, IR, radar, acoustic, images- RFID readers, badge scanners, biometric- People

• Surveillance case study characteristics- Video, sound, and images must be analyzed to extract events- Event extraction and analysis by far the costliest operation and this makes resource

optimization hard- Events emerge over time and space - Out of order events are typical due to analysis overhead- To provide situation awareness complex events must be mapped into the context of the

specific facility/retain under surveillance (i.e., must be re-contextualized form the context of the specific sensors to the context understood by the users)

- Windows do not make much sense - Events are often uncertain due to the complexity of the activity they report on (e.g.,

human behavior)- Events must be detected in “human” real-time to enable responce to security threats

Situational relevancy- …..

Slide 10 Nov. 2006

The Intelligence Gathering Problem

• Event-driven collaboration of large, multi-organizational teams using CT analysis tools and operating in dynamically changing situations

• Reduce information overload, and improve decision-making

• Real time enterprise adaptation as the situation evolves

Slide 11 Nov. 2006

Intelligence Gathering Case Study

• Event sources- Information/knowledge sources (e.g., open sources in the web), people - Policies, processes, resources, - Analysis algorithms (e.g., text analysis, evidential reasoning)

• Intelligence Gathering case study characteristic- Events are typically heterogeneous- Events must be mapped and evaluated into many different contexts reflecting

jurisdictions, organizations, teams, and activities- To determine compliance with a policy defined in another context- To determine whether to start or adapt a process defined in a different

context- Out of order events due to analysis overhead and human decision making- Events are often uncertain due to the complexity of the activities monitored (e.g.,

human behavior) and due to gaps in available information- Events must be detected in “human” real-time to be able to respond to threats- Event-driven process adaptation is common

Slide 12 Nov. 2006

A Context Network for Intelligence Gathering

Policy & resource flow

n Policies:1 - Federal Search Warrant2 - FBI Affidavit3 - NJ Search Warrant4 - DHS Notification5 – Information sharing

k Activities and processes:1 - CBP Admission2 - DHS Notification3 - Search Warrant4 - Database Search5 - Investigation6 - Event subscription

mEvents/Resources:1 - Person enters the US2 - Group active in the US3 - Person belongs to group4 – Person belongs to active group in the US

Event flow

RelationsFederal

Austin

FBI

DHS

CBP

BobMary Carol John Alice

1

5

TexasNJ

3

4

1

2

3 4Xavier Yanni

6

2

1

2 3

Task force

5

4

Slide 13 Nov. 2006

Providing Situation Awareness in Intelligence Gathering

• Situation awareness

• Teamwork awareness

• Ongoing policy compliance

• Dynamic adaptation to reflect changes in the events- Process

adaptation- Context net

adaptation

Slide 14 Nov. 2006

The Department of Defense Strategy

To move from privately owned and stored data in disparate networks and within legacy systems/applications to an enterprise information environment where authorized known and authorized unanticipated users can access any information and can post their contributions for enterprise-wide access.

To Consumer-centric:• Data is visible, accessible and understandable• Shared data – supports planned and unplanned consumers• Shared meaning of the data enables understanding

Ubiquitous Global Network

MetadatMetadata a

CatalogCatalogss

Enterprise Enterprise & &

Community Community ServicesServices

ApplicatioApplication Services n Services (e.g., Web)(e.g., Web)

Shared Shared Data Data

SpaceSpace

Metadata Metadata RegistriesRegistries

Security Security Services Services (e.g., (e.g.,

PKI, SAML)PKI, SAML)

Producer

DeveloperFrom Producer-centric:• Multiple calls to find data • Private data – only supports planned consumers• Data translation needed for understanding when pulled from multiple sources

Consumer

Producer and

Developer

System 1 Data

System 2 Data

System N Data

Consumer

...

Enabling Net-Centricity Data Strategy

Slide 15 Nov. 2006

End-User Consumer End-User Producer

B A R R I E R B A R R I E R B A R R I E R B A R R I E R

“What data exists?““How do I access the data?”“How do I know this data is what I need?”“How can I tell someone what data I need?”

“How do I share my data with others?” “How do I describe my data so others can understand it?”

Organization “A” Organization “B” Organization “C”

User is unaware this data exists

User knows this data existsbut cannot access it because of organizational and/or technical barriers

?User knows data exists and can access it but may not know how to make use of it due to lack of understanding of what data represents

Data Strategy Approach:

DiscoveryMetadata


Web Enabling, Web-service Enabling


Communities of Interest,Metadata Registry

Barriers to Identifying, Accessing and Understanding Data

Slide 16 Nov. 2006

DoD Discovery Catalogs

DoD Metadata Registry

DoD Service Registry

Data ProducerData Consumer

“Shared Space”

Unanticipated Authorized User of System A Data

Query Catalogs and Registry

System X

“Pull” Structural and Semantic Metadata

“Pull” Data

PublishStructural and Semantic

MetadataPublish Data and

Services

Pu

bli

sh

Dis

co

ve

ry M

eta

da

ta

System BData exchanged across engineered, well-defined

interfaces

Known User of System A Data

System A

All Data Assets are Tagged with DoD

Discovery Metadata Specification (DDMS)

Metadata

Leverages

Service Oriented Architecture

Publishing and Subscribing of Data & ServicesSupporting Both Known and Unanticipated Authorized Users

Slide 17 Nov. 2006

• Thank you for your attention!

Dimitrios Georgakopoulos (dimitris@ research.telcordia.com)

Slide 18 Nov. 2006

Backup Slides

Slide 19 Nov. 2006




Content Routing

andCoordination


EventOntology


(AC)

Users


PrimitiveEvents


AxtionableEvents


Awareness



(EC)




Event repository

EventExtraction &

Analysis(EA)

Slide 20 Nov. 2006

Event Contexts and Context Management

• A Context typically contain information about:- Entities (e.g., actors or objects or interest)- Activities and state changes of the entities- Time interval of those activities and state changes- Spatial coordinates in which the entities are situated- Relationships of entities and activities to other contexts- Contexts contain both current and historical info

• Context management- E2A permits the initial modeling of one or more application specific

contexts the relationships between them

Slide 21 Nov. 2006

A Simple Context for Surveillance

Facility context dynamically correlates and tracks events from multiple cameras

• Facility Space Hierarchy- Spaces are organized into a containment hierarchy

with the rooms interconnected by portals- Site-specific attributes: e.g., name, secure, public, etc.

• Identities- Partial information on specific people who may use the facility- Site-specific attributes: employee, security clearance,group, etc.

• Entities that move about the facility over time- Usually people, though the idea extends to portable objects, like brief cases and documents- Have a source-independent sequence of locations (supported by object tracking) of how the it

changed positions over time- Identity of the movable object may be known with some degree of certainty

• Pedigree information concerning the above

Lobby Hall

Rm1 Rm2 Rm3

Rm4 Rm5 Rm6

Tracks of people within facility

Slide 22 Nov. 2006

Event Contextualization

Steps performed upon receipt of a primitive event:

• Correlate event parameters and event source metadata with the information of the target and other related contexts

• Incrementally fuse the primitive event with the info already present in the context

• Incrementally publish the resulting contextualized events to its subscribers

Example: When a person enter a room in a facility, the location of the person is updated in the facility context and fused with the location of the camera

Slide 23 Nov. 2006




Content Routing

andCoordination


EventOntology


(AC)

Users


PrimitiveEvents


AxtionableEvents


Awareness



(EC)




Event repository

EventExtraction &

Analysis(EA)

Slide 24 Nov. 2006








Slide 25 Nov. 2006

Awareness Specification

VEAS-provided customization permits users to specify:• What types of events are of interest• How to detect them

- When- Where- Which method to use

• Who should be alerted• What/how event evidence and pedigree should be

presented to each user

Slide 26 Nov. 2006

Event Ontology

• E2A surveillance ontology defines what type of events are of interest:

- Event types are defined formally in OWL

- Existing event ontologies can be imported and used

- New event ontologies can be created and existing ones can be modified via Protégé to provide site-specific and situation-specific customizations

- Ontology provides an agreement about situation- and site-specific events of interest

• Example: ZoneVisit• Supported by: Protégé,

Awareness Computation

Slide 27 Nov. 2006

Awareness Specification (How Event Patterns are Specified)

• Specifications - Build from interconnected event operators- Example: “Gale’s desk monitor” detects if an

object has been taken from her desk during her absence

• Operators - Perform processing on events- Examples: generic filter, custom set difference

“Anybody but owner in target office”

• Interconnections define contracts- Specify the event flow between operators- Define event types of the flowing events- VEAS users author interconnections by utilizing

event types defined in the surveillance ontology- Example: ZoneVisit event type flows from

“Owner in target office” to “Anybody but owner in target office”

Slide 28 Nov. 2006

Core Awareness Operator Classes

• Contextualized event operators- Subscribe to contextualized events and can be customized to filter such events

• Alert delivery operators- submit alerts requests (by issuing actionable events) to E2A’s Coordination component

• Proactive event production operators- submit task requests (by issuing actionable events) to E2A’s Coordination component

• Stream processing operators- OR: computes a union of its input streams- Difference: computes a set of difference of input streams

• Relational algebra operators - Filtering: culling of uninteresting events- Joining: combines related events from multiple sources into a composite event- Grouping and aggregation: regrouping and aggregations of events or multiple events

• Statistical and sampling operators - Sampling operators can be added to compute changes in rate of occurrence of a specific

event type- Statistical operators can be introduce to utilize learned patterns of normal behavior to

detect statistical anomalies• Extensible pallet of operators

Slide 29 Nov. 2006




Content Routing

andCoordination


EventOntology


(AC)

Users


PrimitiveEvents


AxtionableEvents


Awareness



(EC)




Event repository

EventExtraction &

Analysis(EA)

Slide 30 Nov. 2006








Slide 31 Nov. 2006

Coordination for Alert Delivery and Proactive Event Production

• E2A’s coordination component embodies the capabilities of a workflow management system

• Rich-media dataflow type

• Accepts actionable events from Alert Delivery and Proactive Event Production operators

• Routes alerts and evidence to the user role(s) specified in the alert delivery operators

• Integrates external programs that can interact with event sources for- tasking them to produce a specific event or events or a specific type- managing them (e.g., changing their settings)

Documents

Situation Awareness Telcordia’s E2A Architecture and Three Case Studies