43
Global Scale ESB with Mule Andrew D Kennedy [email protected] March 2011

Global Scale ESB with Mule

Embed Size (px)

DESCRIPTION

Presentation on Mule 2 and 3 and associated scalability issues and bottlenecks

Citation preview

Page 1: Global Scale ESB with Mule

Global Scale ESB with Mule

Andrew D [email protected]

March 2011

Page 2: Global Scale ESB with Mule

Who Am I

• Enterprise Java Developer• Apache Qpid Committer• Previously Security Engineer• Over 15 Years Java Experience• Worked with various Investment Banks• Interested in Data Mining and

Visualisation

Page 3: Global Scale ESB with Mule

This Presentation

• What is an ESB• What is Mule• Mule Applications• Mule Scalability

Page 4: Global Scale ESB with Mule

What is an ESB

• Enterprise Service Bus• SOA

– Service Oriented Architecture• SEDA

– Staged Event-Driven Architecture• EIP

– Enterprise Integration Patterns

Page 5: Global Scale ESB with Mule

Service Oriented Architecture

• Services– Shared Business Functions

• Decoupling– Local and Remote Services

• Distributed Applications• Directory and Discovery

Page 6: Global Scale ESB with Mule

EIP

• Similar to Design Patterns– Gang of Four Book

• Shared Terminology• Names for Common Concepts• Allows Discussion• Simplifies Design

Page 7: Global Scale ESB with Mule

(Some) Common Patterns

• Aggregator (268)• Channel Adapter (127)• Content Based Router (230)• Dead Letter Channel (119)• Selective Consumer (515)• Message Translator (85)• Guaranteed Delivery (122)

Page 8: Global Scale ESB with Mule

What is Mule

• Open Source Project• MuleSoft

– Mule Forge– Mule Enterprise– Mule MQ– Tcat Server

• 2.x - Legacy Applications• 3.1.1 - Current Community Release• 3.2.0 - Latest Developer Build

Page 9: Global Scale ESB with Mule

Transports

• Carry Messages between Services• Connector

– Configuration– Threading– Retry

• Endpoint– Connects Services– Transform Messages– Transaction Boundary

Page 10: Global Scale ESB with Mule

Endpoint Examples

<file:inbound-endpoint path="/esb/in"comparator="org.mule.transport.file.comparator.OlderFirstComparator" reverseOrder="true” />

<jetty:endpoint name="server" host="localhost"port="60203" path="services/Lookup" />

<imaps:endpoint name=”mail" host="localhost"password=”hunter2" port="123" user=”adk"/>

<quartz:endpoint name=”batch" repeatCount="10"repeatInterval=”60000" jobName="job"/>

<jms:inbound-endpoint queue="test.queue"/>

Page 11: Global Scale ESB with Mule

VM

• Communication within VM• Synchronous or Asynchronous• Transactional<vm:connector name="async"

queueEvents="true” /><vm:inbound-endpoint path="in"

connector-ref="async"/>

Page 12: Global Scale ESB with Mule

JMS

• Queues• Topics• Request-Response<jms:connector name=”jms" specification="1.1"

connectionFactory-ref=”qpid"username=”adk" password=”hunter2” />

<jms:outbound-endpoint queue=”audit” /><jms:inbound-endpoint topic=”uk.*.gbp” />

Page 13: Global Scale ESB with Mule

Web Services

• HTTP and HTTPS– Custom Data Formats

• CXF– JAX-WS– SOAP– WSDL

• REST

Page 14: Global Scale ESB with Mule

CXF JAX-WS

<inbound-endpointaddress="http://localhost:63081/hello" />

<cxf:jaxws-serviceserviceClass="org.example.ServiceImpl” />

<cxf:jaxws-client clientClass=”com.example.Client”wsdlPort="SoapPort”wsdlLocation="classpath:/save.wsdl”operation=”output” />

<outbound-endpointaddress="http://www.example.com/services/save” />

Page 15: Global Scale ESB with Mule

File System

• Poll Directory for Input<file:connector name="input" fileAge="500"

autoDelete="true" pollingFrequency="100"moveToDirectory="/backup"moveToPattern="#[header:originalFilename].backup”/>

• Write Output File<file:connector name="output" outputAppend="true"

outputPattern="#[function:datestamp]-#[header:originalFilename]" />

Page 16: Global Scale ESB with Mule

Other Transports

• AJAX• jBPM• JDBC• Quartz• STDIO• RMI• Servlet

• TCP• UDP• FTP• POP3• IMAP• SMTP• XMPP

Page 17: Global Scale ESB with Mule

Transformers

• Change message content• XML to POJO via JAXB• Applied to endpoints• Efficient mechanism for processing• Similar to AOP

Page 18: Global Scale ESB with Mule

Transformer Types

• Encryption and Decryption• ByteArray to Object• Expression Evaluator• XML and JSON• Compression• Encoding and Decoding

Page 19: Global Scale ESB with Mule

Filtering

• Apply to endpoints• Payload Type• Expression

– XPath, OGNL, JXPath• RegEx• Wildcard• Message Property• Logical Operators

– And, Or, Not

Page 20: Global Scale ESB with Mule

Filtering Examples<and-filter>

<message-property-filterpattern="JMSCorrelationID=1234567890"/>

<message-property-filter pattern="JMSReplyTo=null"/></and-filter><or-filter>

<payload-type-filter expectedType="java.lang.String"/><payload-type-filter expectedType="java.lang.StringBuffer"/>

</or-filter><not-filter>

<wildcard-filter pattern=”com.example.*"/></not-filter>

Page 21: Global Scale ESB with Mule

Routers

• Control message flow at endpoints• Used to link services• Both inbound and outbound• Specify Filtering and Transformations

Page 22: Global Scale ESB with Mule

Selective Consumer<inbound>

<selective-consumer-router><mulexml:jxpath-filter

expression="msg/header/resultcode ='success'"/>

</selective-consumer-router><forwarding-catch-all-strategy>

<jms:endpoint topic="error.topic"/></forwarding-catch-all-strategy>

</inbound>

Page 23: Global Scale ESB with Mule

Idempotent Filter

<inbound><idempotent-receiver-router

idExpression="#[message:id]-#[header:foo]"><simple-text-file-store directory="./idempotent” />

</idempotent-receiver-router></inbound>

Page 24: Global Scale ESB with Mule

Filtering Router<outbound>

<forwarding-catch-all-strategy><jms:outbound-endpoint queue="error.queue"/>

</forwarding-catch-all-strategy><filtering-router>

<smtp:outbound-endpoint to=”[email protected]"/><payload-type-filter expectedType="java.lang.Exception"/>

</filtering-router><filtering-router>

<jms:outbound-endpoint queue="string.queue"/><and-filter>

<payload-type-filter expectedType="java.lang.String"/><regex-filter pattern="the quick brown (.*)"/>

</and-filter></filtering-router>

</outbound>

Page 25: Global Scale ESB with Mule

Services

• Combine Endpoints, Routers andComponents

• Mule 2.0 idiom• Replaced with Flows and Patterns

– Simpler to use and configure– Some features still not available

Page 26: Global Scale ESB with Mule

Flows

• Mule 3 idiom• Inbound endpoint• Message processors

– Chained execution– Components

• Outbound endpoint for one-way• Response for request-response

Page 27: Global Scale ESB with Mule

Components

• Carry out actions• Business logic in a flow• Simple components

– Logging– Passthrough– Testing

• Spring or POJO• Web services

Page 28: Global Scale ESB with Mule

Message Processors

• Aggregator and Splitter• Message Filters• Recipient List• Resequencer• Round Robin• Wire Tap

Page 29: Global Scale ESB with Mule

Example Flow<flow name=”OrderFlow">

<file:inbound-endpoint path="/incoming"><file:filename-filter name=”order-*.xml"/>

</file:inbound-endpoint><xml:xslt-transformer xsl-file=”order-transform.xsl"/><splitter expression="xpath://order"/><!-- The following message processors will be invoked for each order in the xml file --><expression-filter expression="xpath://order[@type='book']"/><component class=”com.example.BookOrderProcessor"/><smtp:outbound-endpoint subject="Order Confirmation" address="#[variable:email]"/><jdbc:outbound-endpoint ref=“saveOrder"/ ><default-exception-strategy>

<jms:outbound-endpoint queue="failedOrders"/></default-exception-strategy>

</flow>

Page 30: Global Scale ESB with Mule

JMS Flow<flow name=”JmsFlow">

<jms:inbound-endpoint queue="in"><jms:transaction action="ALWAYS_BEGIN" />

</jms:inbound-endpoint><component class="com.example.ProcessMessage" /><jms:outbound-endpoint queue="out">

<jms:transaction action="ALWAYS_JOIN" /></jms:outbound-endpoint><default-exception-strategy>

<commit-transaction exception-pattern="com.example.ExampleException” /><jms:outbound-endpoint queue="dead.letter">

<jms:transaction action="JOIN_IF_POSSIBLE" /></jms:outbound-endpoint>

</default-exception-strategy></flow>

Page 31: Global Scale ESB with Mule

Complex Flow<flow name=”TwitterFlow”>

<poll frequency="5000"><twitter:search query=”example" />

</poll><splitter evaluator="json" expression="results" /><idempotent-message-filter idExpression="#[json:id]" /><enricher target="#[variable:userDescription]" source="#[json:description]">

<twitter:user userId="#[json:from_user]" /></enricher><salesforce:create type="Opportunity">

<salesforce:sObject><salesforce:field key="Name" value="#[json:from_user]" /><salesforce:field key="Description" value="#[variable:userDescription]" />

</salesforce:sObject></salesforce:create>

</flow>

Page 32: Global Scale ESB with Mule

Patterns

• Specific integration Features• Simple Service

– Exposes Components as Web Services• Web Service Proxy• Bridge

– Message Adapter and Transformer• Validator

– Validates inbound messages synchronously– Processes asynchronously

Page 33: Global Scale ESB with Mule

Questions?

Page 34: Global Scale ESB with Mule

Mule Application

Mule 2 Application Design

Page 35: Global Scale ESB with Mule

Mule Scalability

• Global Enterprises• Performance• Multiple Data Locations, Sources,

Destinations and Owners• Services

– Software as a Service– Infrastructure as a Service– Platform as a Service

Page 36: Global Scale ESB with Mule

Problems

• Performance– Computation– Latency– Throughput– Infrastructure

• Integration– Adding Functionality– Collecting Data

• Security and Identity• Transactions

Page 37: Global Scale ESB with Mule

Transactions

• ACID– Atomicity, Consistency, Isolation, Durability

• Easy on one system• Hard with distributed systems

– Pick your boundaries• Robust failure handling

Page 38: Global Scale ESB with Mule

Computation

• Parallel Processing• Asynchronous Tasks• Orchestration

– Workflow• Pooling and Threading• More and faster is better

Page 39: Global Scale ESB with Mule

Bottlenecks

• Bandwidth– Networking– Latency and Throughput

• External service providers• Data sources

– Database• Messaging

Page 40: Global Scale ESB with Mule

Messaging Backbone

• Sets scalability limits• Message flow through system• Infrastructure choices

– JMS– Web based– AMQP– XMPP– TIBCO

• Hardware solutions available

Page 41: Global Scale ESB with Mule

Clustering

• Various external components– Database– Messaging– Web Services

• Business logic components– Terracotta– GigaSpaces XAP

Page 42: Global Scale ESB with Mule

Load Balancing

• Web Services• Software• Hardware

– XML Transformation– SSL

• Round Robin

Page 43: Global Scale ESB with Mule

Questions?

Thanks for ListeningAndrew D [email protected]