Choosing an applicationintegration strategy

Anko DuizerClass-A

Overview

• Why integration?• What is integration?• Challenges• Common scenario’s• Which technologies does Microsoft offer?• Messaging• Design considerations• A different viewpoint: The Ring(s)• The Matrix

Why?

• Enterprises are typically comprised of hundreds applications– Custom built, acquired by a third party, legacy

systems

• Reasons– Creating a single, big application to run a

complete business is impossible– Flexibility to select “the best”– Use a collaborative partner model

Those who can not remember the past are condemned to repeat it!

The current situation

FB

E

A

D

C

What?

• Connecting– Computer systems– Companies– People

Application integration is the secure and orchestrated sharing of processes and/ or

data between applications

Integration scenarios

• Information portals

• Data replication

• Shared business functions

• Service-oriented architectures

• Distributed business processes

• Business-to-business integration

Basic challenges

• Networks are unreliable

• Networks are slow

• Any two applications are different

• Change is inevitable

• Organizational problems

Integration approaches

• Simple association

• Central hub

• The big database approach

• Message bus

• Service integration

• Hybrid

Simple association

UI

BO

BO

Data

Application

UI

BO

BO

Data

Application

Endpoint Endpoint

Central hub

UI

BO

BO

Data

Application

UI

BO

BO

Data

Application

Endpoint Endpoint

ERP System

The big database approach

UI

BO

BO

Data

Application

UI

BO

BO

Application

Message bus

UI

BO

BO

Data

Application

UI

BO

BO

Data

Application

Endpoint Endpoint

Message bus

Service integration (1)

UI

BO

BO

Data

Application

UI

BO

BO

Data

Application

Endpoint Endpoint

Service

Service integration (2)

Application

Endpoint

Service

Service Service Service

Endpoint

Application

Endpoint

Application

Endpoint

Application

Microsoft technology

• BizTalk Server 2004

• Web Services

• MSMQ

• DTS

• Indigo (not available yet)

• SQL Service Broker (not available yet)

BizTalk Server 2004

actionorchestrationdesigner

biztalk editor

biztalkmapper

messagingmanagement

development operation management

administration

tracking

messaging services

orchestration services

config

tracking

port

pipelinedesigner

applications

monitoring

Businessrules

Web Services

Host

Component

Client

Application

Host

Page

Browser

XML over HTTP

HTTP request

.NETCOMEJBPerl…

listenerproxy

proxy

Internet

MSMQ

Queue

msgmsgmsgmsg

Sender Receiver

DTS

OLE DB ODBCFixed fieldASCII delimited

Source

OLE DB ODBCFixed fieldASCII delimitedRepl. publication

Destination

DTS Data Pump

In Out

ActiveX ScriptCopyTrim String…Custom

Transforms

Indigo

Port

Service

Service

Message

Channel

Channel

Service

ServiceChannel

SQL service brokerSQL Server object DescriptionMessage Type Defines valid messages for exchange between services

Contract Specifies type of messages and their direction in a conversation, initiator or target

Queue Stores messages before sending and after receipt as result set

Service Program The part of a service broker application that reads messages from a queue and processes them.

Service Addressable endpoint for service communication

msgmsgmsgmsg

Service

Contract Serviceprogram

Queue

Message type

Messaging

• Loosely coupled

• Communicating asynchronously

• Reliable

• Makes the messaging responsible for transferring data from one application to another

Sync or async?

time

Process A

Process B

blocked

time

Process A

Process B

Basic messaging concepts

ApplicationApplication Channel

Endpoint

Endpoint

Message Translation

Routing

Messaging patternsY

X

Y

X

1)

• Fire and Forget

• No Response required

2)

• Request / Reply

• System makes request

and gets response

3)

• Needs ability to cache response from multiple requests to respond to one message

Y

XEAIEAI

EAI

B

X

4A/B)

• Multiple Applications needed to satisfy request

• 4A: XREF Look-up

• 4B: XREF Update

A C

XREF

EAI

A

X

5)

• Publish Subscribe

• A, B and C subscribe to message

B

CEAI

6A)

• Needs data from Z to be

able to update Y

Y

X EAI

Z

The message is the message!

Main design considerations

• Timing/ Latency

• Identification– Industry keys, Internal keys, Chained keys &

Matching algorithms

• Sharing entities, activities, processes and utilities

• Designing endpoints

• Message transformation

Main design considerations

• Routing

• Security

• System management

• ACID Transactions

• Service-oriented architectures

Shared data ownership

E-Commerce

Inventory

E-Commerce

Choosing

To EAI or to ETL?Data (ETL)

Data Transformation Services

Messages (EAI)

BizTalk Server 2004

Unit of modeling

“Package” – an XML definition of “workflow” and data flow

Schema that capture structure/

Contracts that capture behavior

Unit of interaction

“Synchronous” DTS pipeline for data movement

Asynchronous message passing

Instancing DTS Service controls instances Weak/opaque URI-based references

Extensibility Scripts, SQL, custom components

Intermediaries and message routing

Platform Databases Multiple

Evaluation criteriaCriterion Description

Latency How quickly is the data to be transferred?

Transformation Complexity of the transformation

Scope Application, organization or partner?

Volume Quantity of exchange by a transfer

Intrusion Degree of change to existing applications in order to effect transfer

Effort Effort required to build and maintain the solution

Organization Which solution fits best with the organization?

The Ring(s)

1. Application

2. Enterprise

3. Collaborative partners

4. Everybody else

The Matrix

BizTalk SSB Indigo MSMQ DTS

Latency Near real time

Near real time

Real time Near real time

Batch

Transformation High High N/A N/A High

Scope Enterprise & partners

App. App. App. App. & Enterprise

Volume Medium Medium Low Medium High

Intrusion Medium Low High High Low

Effort High Medium High Medium Medium

Organization tbd tbd tbd tbd tbd

The Matrix - ReloadedIndigo MSMQ SSB

Environment Any WS-* compliant

Windows SQL Server 2005 (both)

Application Any distributed application

Asynchronous Database application

Message store

In-memory or database

NT File system

SQL Server 2005

Type of message

Persistent & Non-peristent

Reliable, Express, Transaction

Transactional only

Protocol Various Various TCP only

The Matrix - Revolutions

Ring 0 1 2 3Communication Synchronous Asynchronous Asynchronous

Focus Performance Security

Integration

Security

Integration

Firewall

Indigo Yes Maybe No

BizTalk Server No Yes Yes

WS-* Maybe Yes Yes

MSMQ No Yes No

SSB Yes No No

Conclusions

• Integration is hard!

• Integration requires good thinking!

• Different technologies available, use them smartly!

• Start thinking asynchronously and message based!

References

• Books– Enterprise integration Patterns, Hohpe &

Wools– Enterprise Integration Solutions, Spackman &

Speaker

• Web– http://www.class-a.nl– http://www.objectwatch.com– http://msdn.microsoft.com/architecture

Questions ?

The Rings(s)

1. Synchronous, .NET or J2EE? , Performance, Indigo, SSB, Web Services

2. Asynchronous, MSMQ or MQSeries?, DTS, Security, WS-*

3. Asynchronous, BizTalk Server, Security, WS-*

4. Firewalls, for the rest you don’t care!