Rapid Systems of Systems Integration –Combining an ...cseweb.ucsd.edu/classes/wi07/cse218/Teaching_Documents/Winter… · Beyond Traditional Systems Integration ... • Enterprise

California Institute for Telecommunications and Information TechnologiesLa Jolla, CA 92093-0405, USA

Department of Computer Science & EngineeringUniversity of California, San DiegoLa Jolla, CA 92093-0114, USA

Rapid Systems of Systems Integration – Combining an Architecture-Centric Approach with Enterprise Service

Bus Infrastructure

Ingolf H. Krueger

UCSD/CSE/Calit2

© Ingolf H. Krueger 2February 26, 2007 CSE

Background and Motivation

• Dramatic increase in distribution and complexity of software systems

– Business/Enterprise Systems

– Technical/Embedded Systems

• Shift from stand-alone to networked systems

• Internet/Wireless Networks have become key enabling technologies for advanced services

• Convergence between business and technical systems:

– Telecommunication/Networking

– Web Services

– Embedded Systems

• Examples:

– Enterprise Service Bus (ESB)

– Automotive Systems Engineering

– RUNES/Sensor Networks

– Command and Control

– CAMERA


Ah, the “Good” Old Days

DB-Server

Application


Ah, the “Ok” Old Days

DB-Server

Application

DB-Server

Application

Integration Goals:

• Application Integration

• Data Integration

• Process Integration

• UI Integration

Techniques:

• Source-based integration

• File-exchange

• Shared databases

• UI “scraping”


Whoops?

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application


System Integration, Spaghetti-Style?

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

DB-Server

Application

… and address:

• Policies

• Governance

• Security

• Reliability

• Quality of Service

• (B2B) Workflows

• Agility

• …


Contribution

• Layout of requirements space for complex

systems of systems integration

• Combination of

– Agile architecture design process

– Enterprise Service Bus deployment infrastructure

• Co-iteration of ConOps/business process elicitation

and architecture design

• Requirements discharge at both the process and

infrastructure level


Fast Forward to Today: Enterprise Service Bus

From tightly to

loosely coupled

systems


Outline

• Motivation and Overview

• Systems of Systems Integration Requirements

• Process and Architecture Design

• Infrastructure

• Example Applications:

– BioNet

– RESCUE/ResponseSphere


• Enterprise Integration Patterns

• Summary and Outlook


Systems of Systems Requirements Space

SecurityPrivacy

Presentation

ProcessIntegration

DataIntegration


Beyond Traditional Systems Integration

• Application areas such as homeland security and

disaster response go beyond traditional systems

integration challenges:

– High demand for flexibility with respect to the configuration

and support of business processes to anticipate and cater to

changing threat and mitigation scenarios

– High agility demands during both development and

production to address legacy and emergent capabilities,

processes, applications and technologies

– Wide variety of trust relationships among and across

stakeholders and their organizations.


Beyond Traditional Systems Integration

• Our Approach:

– Balance challenging integration requirements while rapidly

delivering a high-quality, value added, integrated system

architecture and service-based implementation infrastructure

– Agile, yet systematic approach to architecture discovery and

design

– Enterprise Service Bus (ESB) helps discharge many of the

obligations induced by requirements


Refined DSS Requirements Grouping

Data-, Process- and Application Integration

Information Management

Infrastructure Provisioning

Presentation/UI/Visualization

Trust- and

Security-

Management

Quality- and

Non-Technical

Requirements


Data-, Process- and Application Integration Requirements

• DSSs usually integrate numerous, vastly disparate data sources and

corresponding interfaces:

• Sensor networks,

• Databases,

• Individual applications, …

• Flexibility/Agility Requirements

– Integration and configuration of legacy and emerging data and applications.

– During production many data sources (and their interfaces), as well as

business processes will change.

– Need for flexible business process configuration and adaptation.

• All but excludes traditional point-to-point integration solutions

– they lack the required flexibility.

• View data sources and applications as “operational nodes”

• Interpret the capabilities they offer as “services” (SOA)


Information Management Requirements

• Data and information extracted from it needs to be made

available to eligible parties.

• This requires

– Provisioning for information access and dissemination strategies,

integrated with trust and security management

– Provisioning for persistence mechanism, such that it is available to

all parties who need access at the appropriate time – including

building and storing historical information.

• Providing access to contextual rather than purely incidental

information

– Key element of decision making process


Infrastructure Provisioning

• Typical scenario: both fixed and mobile access needed

– Laptops,

– Personal Digital Assistants (PDAs),

– Cell phones,

– Physical/Virtual command center

• New networking capabilities need to be established in case of need

• Varying degrees of connectivity within distributed, heterogeneous

networks

• Inherent need for multi-modal interaction and its integration into a

consistent situational DSS picture

• Need to hide network complexity and the details of access methods

• Networking/communication infrastructure needs to support monitoring

and management of the infrastructure itself


Trust and Security Management

• Provide mechanisms for establishing and enforcing a wide variety of trust relationships among and across stakeholders and their organizations

• All stakeholders have their own trust relationships/standards

– How do these established relationships change in the face of devastation?

– How quickly can newly established rules for trust for the duration of a response effort be effectively implemented as part of the supporting IT infrastructure?

• Statically coded trust relationships may hinder an effective response

• Security policies

– Who can access what under what circumstances?

– When and where does data need to be encrypted to what degree? …

• Provide flexible access control and authentication ̶ including monitoring, communication and (physical) infrastructure security, as well as data privacy


Presentation/UI/Visualization

• Establish or increase situational awareness

• User interfaces need to flexibly reflect the user role and their

associated “need to know” and decision making capabilities

• Usability is a major concern

– Consider seriously and consciously,

– Build into the integrated system,

– Cater to the wide range of end-users from data gathering in the

field to decision makers,

– Think of utility and efficacy under stress.

• Information Architecture (!)


Quality Aspects

• Scalability,

• Configurability,

• Availability,

• Survivability,

• Reliability,

• Adherence to standards,

• …


Outline




• Infrastructure


– BioNet






From Requirements to Process

• Emergent rather than stable requirements

• Disparate systems lead to contradictory requirements

• Task of systems integration

– Capture business logic dealing with system interactions

– Ensure that requirements conflicts are resolved where needed

• IT is one important aspect of an overall software and systems

architecture and integration effort

– Issues such as governance and policy management extend beyond mere IT

– IT integration architecture and infrastructure development can support and

sometimes automate these aspects

– Mechanism: simplifying, streamlining, automating, where possible, the

identified business logic

– Bring out privacy and security requirements clearly in the relevant business

processes


Co-Iteration of Business Process and Architecture

• Facilitates transfer between text and models

• Increases agility due to shortened review cycles

• Supports multi-view capturing and modeling

• Provides mechanism for validation/consistency checking


FEA

Zachman

Architecture vs. Process

UML 4+1

*

*

*

*

*

1

*1..*

*

Domain Model,

Architecture

Business Processes,

Use Cases, User Stories,

Requirements, Risks

Architecture

Document Implementation

DoDAF:

AV, OV,

SV, TV


Architecture Document Development Process

• SCRUM-inspired Process:

– Quick iterations

– Managed adaptation of the goals to the changing (requirements) environment

– Enables quick reactions to frequent changes in ConOps

– Supports rapid feedback provisioning to the business process development teams

• Instituted incremental peer review strategy

– Peer review every increment to every artifact daily

– Keeps high standards throughout the development effort

– Ensures that there always is a consistent architectural basis

• Monthly milestones

– Every development group delivers current versions of their artifacts

– With each iteration, the artifacts progress and address an expanding set of requirements


Architecture Document Development Process

• Balance agility by keeping architecture in the center of the integration effort– DoDAF views can be derived from domain model

– OVs can capture service-oriented architectures effectively

– OV-7 centerpiece, in conjunction with OV-5, OV-6

– Derive “capabilities” (services) and interface needs (OV-1..3)

• Domain model anchor-point for security analysis– Use of the Common Criteria (CC)

– Asset-Analysis

– Threat-Analysis

– Impact-Analysis,

– …

• Lessons learned:– Has worked well for us within a ten to 15 person architecture development team

– Including a team of dozens to hundreds of stakeholders into an agile process requires establishing clear interfaces/responsibilities/authority

– Designate representatives of the other stakeholder groups as reviewers or authors of architecture artifacts


DB-Server

App/Business-Logic

UI

Migration to Service-Oriented Architecture

UI

Domain Model/

Data

Business

Logic

Service-Interfaces/

Contracts

Service 1

Domain Model/

Data

Business

Logic

Service-Interfaces/

Contracts

Service n

…

Service-Infrastructure

*

*

*

*

*

1

*1..*

*

Domain Model,

Architecture

extract/create

slice/configure/refactor

validate

validate

DB-Server

App/Business-Logic

UI

DB-Server

App/Business-Logic

UI


Service-Oriented Architectures

• Service-Oriented Architecture

– Multiple varying definitions exist

– Collection of (network-accessible) services, each addressing a

business function

– Enables service access/interaction

• by simple data passing

• as several services coordinating some activity

– provides means for registering of and connecting to services

• Service

– Orchestration/Coordination of domain entities to deliver (business)

function over pertinent data entities

– Described as interactions among roles

– Internal vs. external services


What’s in a Component?

Interface Classes

Domain Classes

ControlClasses

Control communication with the environment

Coordinate complex

tasks within component

Conceptual entities of the application domain


What’s in a Service?

Service Interfaces/Contracts

Domain Model/Data

Business Logic

Published in well-known location

adapted from [KBS05]


Components and Objects vs. Services



• Middlewares

– CORBA ORBs (Object Request Brokers)

– DCOM

– Web Service Middlewares

• J2EE

• .NET

• Service infrastructure

– Service registration and lookup

– Service binding

• Development-time

• Runtime

– Communication using standard protocols and formats



Definition of

• Services that can be called upon

• Syntactic interface

• Discovery/Access mechanism

• Transport protocol


Benefits of SOA

• Partial interaction specification

• Loose coupling

• Agility

• Flexibility

• Scalability

• Portability

• Encapsulation

• Abstraction

• Composability


Web Services – Base Technologies

Universal Description,Discovery, and Integration

(UDDI) Protocol

Simple Object Access Protocol

(SOAP)

Web Service Description Language

(WSDL)

Syntactic Interface(“IDL”)

Transport Protocol

Registration/Discovery



SOAP-Message

Envelope +

XML document


Web Services Architecture*

* L. F. Cabrera, C. Kurt: Web Services Architecture and its Specifications: Essentials for understanding WS-*, Microsoft Press, 2005


Service Registration and Lookup

• Motivation for a Service Registry

– Service providers register their service at the registry

– Service requesters can lookup at runtime services they need

– Service registry matches service requesters and available services, using a service descriptor

• Motivation for a “Web Service Descriptor”

– Each service provider could implement the Web Service differently

Service

Registry

Service

Requester

Service

Provider

find publish

bind


WSDL

• Describes a service, its methods and where it can be found

• How parameters are encoded (Encoded or Literal)

• How the SOAP Body element is encoded

(RPC or Document)

• Can be used to (automatically) build a proxy


Sample WSDL Document

<definitions name="BabelFishService“ lns:tns="http://www.xmethods.net/sd/BabelFishService.wsdl"

<message name="BabelFishRequest">

<part name="translationmode" type="xsd:string" /><part name="sourcedata" type="xsd:string" />

</message>

<message name="BabelFishResponse">

<part name="return" type="xsd:string" /> </message>

<portType name="BabelFishPortType">

<operation name="BabelFish">

<input message="tns:BabelFishRequest" /><output message="tns:BabelFishResponse" />

</operation></portType>

<binding name="BabelFishBinding" type="tns:BabelFishPortType">

<soap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http" />

<operation name="BabelFish">

<soap:operation soapAction="urn:xmethodsBabelFish#BabelFish" />

<input>

<soap:body use="encoded" namespace="urn:xmethodsBabelFish" encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" /> </input><output>

<soap:body use="encoded" namespace="urn:xmethodsBabelFish" encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" />

</output></operation></binding><service name="BabelFishService">

<port name="BabelFishPort" binding="tns:BabelFishBinding">

<soap:address location="http://services.xmethods.net:80/perl/soaplite.cgi" />

</port></service></definitions>


SOAP

• HTTP with an XML payload

• A SOAP Envelope contains an optional header and a body

• Format interoperability issues


Sample SOAP Message

POST /SimpleMathservice/Math.asmx HTTP/1.1User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client

Protocol 1.0.3705.0)Content-Type: text/xml; charset=utf-8SOAPAction: "http://myadvancedwebserviceURI/SetSomeProperty"Content-Length: 338Expect: 100-continueConnection: Keep-AliveHost: localhost<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><SetSomeProperty xmlns="http://myadvancedwebserviceURI/">

<str>SomePropertyValuesentfromclient</str></SetSomeProperty>

</soap:Body></soap:Envelope>


Web Services – Emerging Standards and Technologies

WS-BPEL/

BPEL4WS

WS-

Coordination

WS-

Transaction

WSDL

Web

Services

SOAP UDDI

WS-Reliable

Messaging

WS-Policy

WS-

Security


Outline




• Infrastructure


– BioNet






Message/Service Bus Architecture


ESB: Infrastructure Offerings

Xbean, Spring 1.2.8, Mule 1.3Object Brokers

ActiveMQ 4.0, JBOSS Messaging 1.0, WebsphereMQ 6.0, SonicMQ 7.0

Messaging

XA, JOTM 2.0.x, JTA 1.0.xTransactions

MySQL 5.x, Postgresql 8.1, Apache Derby 10.x

Persistence

SSL, HTTPS, …Encryption

Acegi 1.0Authentication



Xfire 1.1, Axis 1.x, Axis 2.x, ActiveSOAPWebServices

Groovy 1.0Scripting Engine

WS-BPEL 1.1 and 2.x, Drools 2.0/3.0Rules Engine

JMX (J2SE 5.x)Configuration Management

REST, TuscanyLanguage neutral access

JDBC 3.x, JDBC 4.x, XA, JMS 1.1Transaction support

WS-BPEL 1.1 and 2.x, Mule 1.8, PXE 1.0Orchestration/Workflow



Aggregate, Resequence, Forwarding consumer, Filtering outbound routing, Recipeint list, Multicasting, Chaining, Message splitting, Exception based, Response aggregator, Idempotent, Selective consumer

Routing

SEDAScalability

Peer to peer, ESB, Client/ServerTopologies

EJB 2.x, EJB 3.x, File, FTP, IMAP, Quartz 1.5.2, RMI, SOAP, SSL, Stream, TCP, UDP, VFS, WSDL, JMS, VM (embedded), JDBC 3.x, JDBC 4.x, TCP, UDP, Multicast, HTTP, Servlet, SMTP, POP3, XMPP

Provider Transports /Bindings

Spring 1.2.8, Hivemind 1.1, ServiceMix 2.x, JBI, Mule 1.3, Celtix 1.0, iBatis 2.1.7

Containers


ESB Logical Architecture


ESB Deployment Mapping


Outline




• Infrastructure


– BioNet






Example: BioNet

• Effective consequence management of Bio Event

• Cooperative program between DHS and DTRA

• Objectives:

– Develop interoperable military and civilian concepts of operation

– Integrate military and civilian capabilities to detect and characterize bio event

– Provide common situational awareness to ensure timely, effective, and

consistent response

• Large Scale Systems of Systems Integration


Example: BioNet Architecture/Process

Rapid, Iterative and Incremental Software and Systems Integration


Outline




• Infrastructure


– BioNet






Messaging Patterns

• Hohpe et. al: Enterprise Integration Patterns, Addison

Wesley, 2003:

• Messaging Systems

• Messaging Channels

• Message Construction

• Message Routing

• Message Transformation

• Messaging Endpoints

• System Management


Example: Messaging Systems

Message Channel

adapted from [HW03]


Example: Messaging Systems

Pipes and Filters

Pipe Pipe Pipe Pipe

Filter Filter FilterMessage Message

adapted from [HW03]


Example: Messaging Channels

Publish-Subscribe Channel

adapted from [HW03]


Example: Pattern Composition

From tightly to

loosely coupled

systems


The End

Thank you for your attention!

Questions?