Rich Feeds for RESCUE and PALMS

Rich Feeds for RESCUE and

PALMSAn Integration Story

Barry DemchakCalifornia Institute for Telecommunications and Information Technology (Calit2)

July 8, 2008

RESCUE Project

• Calit2 at UC San Diego

• Gather, maintain, leverage, present emergency information

• Serve emergency response networks and general public

• Save lives and infrastructure, return to normalcy

Rich Feeds for RESCUE

• Captures, preserves, integrates, and exposes

• Unconventional and emergent data feeds

• Real time or archivally

• Serve emergency response networks and general public

Rich Feeds Objectives and Challenges• Acquisition of data feeds from (disinterested) producers

Heterogeneous data sources Possibly uncooperative producers

• Distribution of data feeds to arbitrary consumers (agencies or public) for domain integration, historical analysis, ??? Data must be purveyed as received Multiple data access paths

• Data feed intermediaries can add new feeds, determine who can add data, who can consume data, … Policy driven authorizations Authentication of all users Policy definition infrastructure

• Long term archiving Database with schemas

• Access by external systems

Research Feeds

Calit2 Traffic Incidents Calit2 Tracked Assets

User View

• Today’s Data Feeds– Traffic– Trackable Objects– UCSD Police Cameras– CalIT2 Cameras

• Today’s Visualizations– Google Maps– Google Earth (soon)

Network

Policy System

Logging System

Database

Research Feeds

Traffic

Tracked Objects

Cameras

Sensors

Network Ops

Visualizations

Internet Browsers

Yahoo Pipes

ODBC

Internet Explorer, FireFox, etc

Excel, Crystal Reports, etc

Preview• Integration Architecture and Methodology

• Visualizations

• Operating in the Real World

Messenger

Router/Interceptor

Policy

Ser

vice

/Dat

aC

onne

ctor

Messenger

Router/Interceptor

Failure Manager

...

<<Rich Service>> S

Ser

vice

/Dat

aC

onne

ctor

...<<Rich Service>> S.n

Service/DataConnector }<<

Rich Infrastructure

Services>>

EncryptionService/DataConnector

LoggingService/Data

Connector

Failure Manager

Service/DataConnector

...


S.1


S.2


}

<<Rich

Application Services

>>

S.n.2


S.n.m


}

<<Rich


>>

S.n.1



Logging


Encryption


Policy ...



<<Rich

Infrastructure Services

>>}Rich Services Virtual Network

Rich ServicesRAS4

Services

Service S1

Roles

U1

U2

U3

U4

U5

Use Case Graph

ConcernsC1 C2 C3

C4CC1

CC2CC3

Domain Model

R1 R2

R3 R4

R5 R6

R1 R2

msg

R3

CC1CC2

Role Domain Model

R1 R2

R3 R4

R5 R6

CC1 CC2 CC3

Router/Interceptor

Messenger/Communicator

RAS1 RAS2

CC1 CC4 CC5

Router/Interceptor


RAS5 RAS6RAS3

S/D

S/D

RIS:

RIS:

Serv

ice

Elic

itatio

nR

ich

Serv

ice

Arc

hite

ctur

e

RAS7

System of Systems Topology

H1 H2

H3

H5

H6

H7

H8

H9H4

RAS1 RAS2 RAS3

RAS5 RAS6 RAS7

Infrastructure Mapping

H1:RAS1 H2:RAS2

H3:CC1

H5:RAS2

H6:RAS5

H7:RAS7H8:RAS7

H9:RAS6

H4:RAS3

Opt

imiz

atio

n ImplementationRAS1 RAS2

RAS3 RAS4

RAS5 RAS6

RAS7 CC1

CC2 CC3

CC4 CC5

Ana

lysi

s

Syn

thes

is

Ana

lysi

s

Iden

tific

atio

n

Def

initi

on

Con

solid

atio

n

Refinement

Hierarchic composition

Refinement

Logical Model

Syst

em A

rchi

tect

ure

Def

initi

on

Logical Architecture Loop

Deployment Loop

Our View: Systems of Systems Integration

• Bottom up• Unintrusive to producer

Policy

Integration System

Consumer SystemsProducer SystemsDatabase

• Quick• Ripe for Services and SOA

Rich Services Architectural Pattern

Messenger

Router/Interceptor

Policy

Ser

vice

/Dat

aC

onne

ctor

Messenger

Router/Interceptor

Failure Manager

...

<<Rich Service>> S

Ser

vice

/Dat

aC

onne

ctor

...

<<Rich Service>> S.n

Service/DataConnector }<<

Rich Infrastructure

Services>>

EncryptionService/Data

Connector

LoggingService/Data

Connector

Failure Manager


...


S.1


S.2


}<<

Rich Application Services

>>

S.n.2


S.n.m


}

<<Rich


>>

S.n.1



Logging


Encryption


Policy ...



<<Rich

Infrastructure Services

>>}

From tightly to l o o s e l y coupled systems

a hierarchically decomposed structure supporting“horizontal” and “vertical” service integration

Rich Feeds Logical Architecture

• Scales to support large numbers of users• Storage that scales• Processing and DB intensive data analysis• Integration with GIS systems and databases• Appropriate visualization methods

Authorization Monitor

Authentication Monitor

Integration System

ODBC Adapter

Database

Logging System

Service / Data

Connector

Visualizer Client

Consumer Adapter

Consumer Systems

Service / Data

Connector

Producer Adapter

Experiment Server

Producer Systems

System of Systems

Deployment Architecture

• Scales to support large numbers of users• Storage that scales• Processing and DB intensive data analysis• Integration with GIS systems and databases• Appropriate visualization methods

Mule ESB with ActiveMQ

Authentication Monitor

Authorization Monitor

Integration System

ODBC Adapter

POJO Interface+

Consumer Adapter

POJO Interface+

Provider Adapter

POJO Interface+

MySQL Database

Logging System

Traffic Server

Tracked Object Server

Browser, Javascript,

Google Maps

Internet

Producers Consumers

Internet

Rich Feeds Web Visualization• UC San Diego Active Shooter Drill

– October 2007– Demonstrated Gizmo moving with embedded camera image

• San Diego Firestorms – October 2007– Demonstrated addition of Calit2 Webcams (2 hours)

• San Diego Metropolitan Medical Strike Team Drill – January 2008– Demonstrated policy exclusion of UCSD Police Webcams

Rich Services Development Process

Rich Services Virtual Network

Rich ServicesRAS4

Services

Service S1

Roles

U1

U2

U3

U4

U5

Use Case Graph

ConcernsC1 C2 C3

C4CC1

CC2CC3

Domain Model

R1 R2

R3 R4

R5 R6

R1 R2

msg

R3

CC1CC2

Role Domain Model

R1 R2

R3 R4

R5 R6

CC1 CC2 CC3

Router/Interceptor


RAS1 RAS2

CC1 CC4 CC5

Router/Interceptor


RAS5 RAS6RAS3

S/D

S/D

RIS:

RIS:

Serv

ice

Elic

itatio

nR

ich

Serv

ice

Arc

hite

ctur

e

RAS7

System of Systems Topology

H1 H2

H3

H5

H6

H7

H8

H9H4

RAS1 RAS2 RAS3

RAS5 RAS6 RAS7

Infrastructure Mapping

H1:RAS1 H2:RAS2

H3:CC1

H5:RAS2

H6:RAS5

H7:RAS7H8:RAS7

H9:RAS6

H4:RAS3

Opt

imiz

atio

n ImplementationRAS1 RAS2

RAS3 RAS4

RAS5 RAS6

RAS7 CC1

CC2 CC3

CC4 CC5

Ana

lysi

s

Syn

thes

is

Ana

lysi

s

Iden

tific

atio

n

Def

initi

on

Con

solid

atio

n

Refinement

Hierarchic composition

Refinement

Logical Model

Syst

em A

rchi

tect

ure

Def

initi

on

Logical Architecture Loop

Deployment Loop

Logging System

RESCUE

ODBC Adapter

Visualization Tool

Research Data FeedDatabase

Logical Architecture

Logical Architecture w/Policy

Policy System

RESCUE

ODBC Adapter

Data

Fe

ed

P

rod

uce

r

Au

the

ntic

atio

n

S/D Connector

Vis

ua

liza

toin

T

oo

l

Au

the

ntic

atio

n

S/D Connector

Da

taba

se

Ob

ligatio

n

Pro

cess

ing

S/D Connector

Request + Identity Certificate (X.509 or SAML)Request + Obligations

(Identity Attributes) x Policy = [Decision, Obligations]Logging

System

PERMIS Organization

PERMIS Sequencing

Subject PEPUser Subject PDP Credential Issue Svc

Attribute Authority * Target PEPIdentity SOA

Target PEP Master Target PDP Target PDP * Certificate

Authority Obligation Svc

Execute(action, target, identity)

Valiidate(identity)

Exportable Attribute List

GetAttributeList(target)

GetCredentials(attributeList, identity)

SignCredential(attribute)

SignedCredential

Execute(action, target, identity, credentialList)

CredentialList

AttributeList

Subject

Target

Valiidate(identity)

Credential Issue Svc

Attribute Authority *

Valiidate(credentialList)

GetCredentialis(attributeList, identity)SignCredential

(attribute)SignedCredential

CredentialListValidation, ValidatedAttributeList

Integrate(attributeList, subjectEnvironment)

DecideAccess(attributeList)

Credential Validation Svc


Decision, Obligations

PerformObligations(action, target, obligationList)

Target

ResultExecute(result)

Integrate(decisions, obligations)

Result

Valiidate(credemtial)Result

Integrate(validatedAattributeList, targetEnvironment)


PALMS

User View

Analysis Engine

Analysis Engine

Network

Access Policies

Event Logger

Data Repository

HIPAA Policies

Research Feeds

Blood Pressure

GeoTracker

Camera

CO2

Sensor

TextMessage

Visualizations

Internet Browsers

Geo Display

Export

Internet Explorer, FireFox, etc

Excel, Crystal Reports, etc

Analysis Engine

Subject Registry


Event Logger Access Policies

PALMS Integration System

Integration Adapter

Data Repository

HIPAA Policies

Service/ Data

Connector

ViewerViewer Adapter

Consumer Systems

Service/ Data

Connector

Sensor AdapterSensor

Producer Systems

Subject Repository

Analysis Engine

Analysis Engine

Analysis Engine

System of Systems

• Scales to support large numbers of users• Storage that scales• SMS and sensor messaging that scales• Processing and DB intensive data analysis• Integration with GIS systems and databases• Appropriate visualization methods

Deployment Architecture

• Enterprise Service Bus (ESB)• ESRI Database• MySQL Database• PERMIS Policy Engine• Analytics Engines

Mule ESB with ActiveMQ

PERMIS Policy Engine

Logging System

PALMS

Integration Adapter

VisualizersData Collectors

ESRI Database

HIPAA Policy Engine

Sensors CamerasBrowsers,

Geoviewers, Exporters

Internet

Failure Detection/Mitigation

MySQL Database

Analysis Engines

Text Messages

Use Case

Rich Services to the RESCUE

“To boldly go where no service has gone before”.

• an extension of the service notion, based on an architectural pattern• Dynamic adaptation

– new services can be introduced at runtime– no need to change or adapt the implementation of existing services

• Manage the complexity of a system-of-systems – decomposing into primary and crosscutting concerns– providing flexible encapsulation for these concerns– generating a model that can easily be leveraged into a deployment

• Workflow management– Service choreography at the infrastructure or application level

Roadmap

• Introduction to RESCUE and Rich Feeds

• Rich Feeds Objectives

• Unconventional and emergent data feeds

• SOA-based Systems of Systems Integration

• Rich Services applied to rapid integration

• Rich Feeds at Calit2/UCSD

Credits

• Funding– NSF RESCUE (#03311690)– NSF Responsphere (#0403433)– NSF ASOSA: Automotive Service-Oriented Software and Systems Engineering

(#CCF0702791)– California Institute for Telecommunications and Information Technology

(Calit2)

• Pictures– Barry Demchak (2008 MMST Drill at Coors Amphitheater)– San Diego County Firestorms After Action Report 2007 (

http://www.sdcounty.ca.gov/oes/ready/docs/2007_SanDiego_Fire_AAR_Main_Document_FINAL.pdf)

– MMST Exercise @ UCSD (http://mmstexercise.calit2.net/)– Wikipedia (http://en.wikipedia.org/wiki/Composite_pattern,

http://www.sdcounty.ca.gov/oes/ready/docs/2007_SanDiego_Fire_AAR_Main_Document_FINAL.pdf

http://www.sdcounty.ca.gov/oes/ready/docs/2007_SanDiego_Fire_AAR_Main_Document_FINAL.pdf

http://mmstexercise.calit2.net/

http://en.wikipedia.org/wiki/Composite_pattern

Backup Slides• Go back …

Logging System

RESCUE

ODBC Adapter

Visualization Tool

Research Data FeedDatabase


Logical Architecture w/Policy

Policy System

RESCUE

ODBC Adapter

Data

Fe

ed

P

rod

uce

r

Au

the

ntic

atio

n

S/D Connector

Vis

ua

liza

toin

T

oo

l

Au

the

ntic

atio

n

S/D Connector

Da

taba

se

Ob

ligatio

n

Pro

cess

ing

S/D Connector

Request + Identity Certificate (X.509 or SAML)Request + Obligations

(Identity Attributes) x Policy = [Decision, Obligations]Logging

System

PERMIS Organization

PERMIS Sequencing

Subject PEPUser Subject PDP Credential Issue Svc

Attribute Authority * Target PEPIdentity SOA

Target PEP Master Target PDP Target PDP * Certificate

Authority Obligation Svc

Execute(action, target, identity)

Valiidate(identity)

Exportable Attribute List

GetAttributeList(target)

GetCredentials(attributeList, identity)

SignCredential(attribute)

SignedCredential

Execute(action, target, identity, credentialList)

CredentialList

AttributeList

Subject

Target

Valiidate(identity)

Credential Issue Svc

Attribute Authority *

Valiidate(credentialList)

GetCredentialis(attributeList, identity)SignCredential

(attribute)SignedCredential

CredentialListValidation, ValidatedAttributeList

Integrate(attributeList, subjectEnvironment)


Credential Validation Svc



PerformObligations(action, target, obligationList)

Target

ResultExecute(result)

Integrate(decisions, obligations)

Result

Valiidate(credemtial)Result

Integrate(validatedAattributeList, targetEnvironment)


Services and SOA

• Manageability• Scalability• Dependability• Testability

Producer Database

OK

StoreData(xxx)

Tim

e

Producer Database

Message Bus

Sto

reD

ata(

xxx)

OK

Network Implementation

Single Server, Multiple Processes

Single Application, Linked Modules

Logical Deployment

• Malleability• Interoperability• Composition• Incremental

development

Patterns

Composite Pattern – Hierarchy (Vertical Integration)

Interceptor Pattern

Service 1

Service 1.2Service 1.1 Service 1.3

Service 1.3.1 Service 1.3.2

Service 2

Service 2.2Service 2.1

Interceptor Service

Message Pattern – Loose Coupling (Horizontal Integration)

Demonstrate Showing All Feeds

(Click on map)

Demonstrate Animation

(Click on map)