Enterprise Integration - Whiteship's Eprilwhiteship.me/wp-content/uploads/2011/11/SpringOne_2011___John... · Enterprise Integration The seriously ... day you still need to understand

John DaviesCTO

Enterprise IntegrationThe seriously nasty stuff

SpringOneChicago

26th October 2011

1Wednesday, 26 October 11

Please welcome my son...

• Pierre• It’s school

holidays in the UK


Agenda

• Integration is more than it seems

•Where does Spring Integration fit?

• Some examples of how our customers are using C24 & S.I.

• Demo and Code review (time permitting)


Financial Services Integration

• Having a first rate SOA or ESB is great but at the end of the day you still need to understand the messages you’re passing around• From Binary (ISO-8583, ASN.1) through CSV and proprietary

standards like SWIFT and FIX to more structured ones such as ISO-20022, SEPA & FpML

• Spring Integration is the same•We can read a file, listen to a socket or read from a WS end-point but

what is it we’re seeing?

• How do we read in something as simple as a comma-delimited file and route it internally based on content?


Integration - High Volume

• Front Office• Very high volume ( 100-100,000 / sec),

usually simple messages

• Latency is critical (< 10ms)

• FIX, FAST, ASN.1, IIOP are most common payloads and protocols

• Light-weight XML only (if any)

• Credit card processing• ISO-8583, Binary, NVP, Batch

• 10,000 / sec or 180m / day


Numbers

• Large Telco• 20 million per second (want 100 million per sec.)

•@ 512 bytes per message it’s 10GB/sec, 36TB/hr, just under 1 PB/day

• Trading volumes• Average 100,000 per second (recent peek @350k/sec) - Per channel!

•@ 512 bytes per message, 8 hours a day

• That’s up to 180MB /sec and averaging ~1.5 TB/day, ~400TB/year

• Now do that across 20 exchanges (28 with a recent client)


Integration - Complex

• Middle Office• Volumes are medium to high (1-1000 / sec), very complex messages

• Calculations are complex and grid/HPC is usually required

• Derivative contracts on ISDA’s FpML

• Corporate Actions (SWIFT, FpML & XBRL)

• SEPA on ISO-20022, Murex, SwapsWire, CSVs are also common

• XML widely used but usually over MQ & JMS

• Tax processing•Wealth records, inheritance, PAYE etc.


Integration - High Value

• Back Office• Low volume (10-1000 / hour), sometimes just a few per day

• Very high value messages, strict compliance and validation

• Often over $1T per day

• Proprietary networks, mostly SWIFT

• Back to numbers...• At 1% interest per year that’s 0.0027% per day.

• 0.0027% of $1bn is $27,261.55

• That’s $1m interest per day for each $37bn

• USD/GBP has gone from 1.592 to 1.605 that’s 0.8% (not unusual)

• The loss (or gain) from 0.8% in one day is $8m per $billion!


SWIFT - A seriously reliable network

• SWIFT is 3 things, a secure network, a standards body and a connectivity provider• It is used by over 9,000 banks (>80,000 branches), in 209 countries

handling over 15 million messages a day (>2 billion/year)

• Mostly payments and securities, Europe is >65% of the volume

• SWIFT is over 30 years old, has a systems availability of 99.986% ( <1½ minutes/week) , they’ve NEVER lost a message

• The figures are impressive but the messages are a real bastard!• Around 330 types of message

• >400 complex types, >1000 complex validation rules


CSV & Text files

• One of the simplest but also one of the most commonName,Card Number,Expiry Date,Amount,Currency,Transaction Date,CommissionOliver Twist,4325-6486-3757-2678,10/06,100,GBP,26-09-2006,8,14988603,UKUriah Heep,4724-7345-4725-7835,11/04,258,USD,21-12-2006,5,15688632,UKMr Scrooge,4924-7264-1264-8536,04/09,1250.6,USD,13-09-2006,15,66846035,USCharles Dickens,4457-436-0087-0104,05/08,350,EUR,13-11-2006,10,93440252,DERowCount=4

•While any programmer can parse this in minutes it’s not obvious how to...• Validate semantically & syntactically

• Route based on values

• Transform into another format

• Manage multiple similar formats in a consistent way

• Document the format / contract


We’ve just started

In most organisations XML is usually the desired format but it’s rarely more than the minority


FIX 4.4 - Post-Trade Confirmation

• This is a FIX 4.4 Post-Trade Conformation• There’s no time for the “<“ and “>”, this is a low-latency domain

8=FIX.4.4 9=1 35=AK 49=STRING 56=STRING 90=1 91=D 34=1 50=STRING 142=STRING 57=STRING 143=STRING 144=STRING 145=STRING 52=20020101-00:00:00.000 122=20020101-00:00:00.000 212=1 213=D 347=ISO-2022-JP 369=1 627=1 628=STRING 629=20020101-00:00:00.000 630=1 664=STRING 772=STRING 859=STRING 666=0 773=2 797=N 650=Y 665=4 453=1 448=STRING 447=B 452=1 802=1 523=STRING 803=1 60=20061122-00:00:00.000 75=20061122 55=STRING 65=STRING 48=STRING 22=1 454=1 455=STRING 456=1 460=1 461=STRING 167=FAC 762=STRING 200=200201 541=20020101 224=20020101 225=20020101 239=RP 226=1 227=1.0 228=1.0 255=STRING 543=STRING 470=AF 471=GB 472=STRING 240=20020101 202=1.0 947=USD 206=0 231=1.0 223=1.0 207=XLON 106=STRING 348=1 349=D 107=STRING 350=1 351=D 691=STRING 667=200611 875=99 876=STRING 864=1 865=99 866=20061117 867=4.3 868=STRING 873=20061117 874=20061117 80=400 54=2 862=1 528=A 529=12 863=200 79=STRING 6=1.5 381=123.45 118=115.78 93=6 89=STRING 10=000


FIX isn’t complex

• FIX is “very” simple, it’s basically tag/value pairs

8=FIX.4.1 9=154 35=6 49=BRKR 56=INVMGR 34=236 52=19980604-07:58:48 23=115685 28=N 55=SPMI.MI 54=22 7=200000 44=10100.000000 25=H 10=159

• So, simple, the tag represents the field...• 44 refers to Price

• 52 is sending Date/Time

• 55 refers to the symbol

• Basic but it’s still better than XML when latency comes into play


SWIFT - MT564 Corporate Action Notification

• Plenty of time for the “<“ and “>” but it’s 30 years old and 9,000 banks already use it

{1:F01INTRUS33AXXX9999999999}{2:O5640947040127FRNYUS33AXXX42181834250401270947N}{3:{108:MT564}}{4::16R:GENL:20C::SEME//2003041800000042:20C::CORP//12345:23G:NEWM/CODU:22F::CAEV//XMET:22F::CAMV//VOLU:98A::PREP//20010901:25D::PROC//PREC


Even XML isn’t simple

Even XML isn’t just XML, most standards have business rules associated with the Schema


• FpML - Complex• 15 levels

• >3000 elements

• But well defined

<paymentDates id="EquityPaymentDate"> <paymentDatesInterim id="InterimEquityPaymentDate"> <relativeDates> <periodMultiplier>3</periodMultiplier> <period>D</period> <dayType>CurrencyBusiness</dayType> <businessDayConvention>FOLLOWING</businessDayConvention> <businessCenters id="PrimaryBusinessCenter"> <businessCenter>USNY</businessCenter> </businessCenters> <dateRelativeTo href="InterimValuationDate"/> </relativeDates> </paymentDatesInterim> <paymentDateFinal id="FinalEquityPaymentDate"> <relativeDate> <periodMultiplier>3</periodMultiplier> <period>D</period> <dayType>CurrencyBusiness</dayType> <businessDayConvention>FOLLOWING</businessDayConvention> <businessCentersReference href="PrimaryBusinessCenter"/> <dateRelativeTo href="FinalValuationDate"/> </relativeDate> </paymentDateFinal> </paymentDates>

FpML


ISO-20022 & SEPA

• The new standard for standards, ISO-20022• The foundation for SEPA (Single European Payments Area)


Implement this!

• From SEPA• ChequeMaturityDateRule: If ChequeType is present and is DRFT or

ELDR, then ChequeMaturityDate is optional. If ChequeType is not present or is different from DRFT or ELDR, then ChequeMaturityDate is not allowed.

• From FpML• ird-57 (Mandatory): If rollConvention is neither NONE nor SFE, nor a

day of the week (MON, TUE,WED, THU, FRI, SAT or SUN) then the period must be M or Y.


OK, so it’s difficult

The business guys don’t care how complex the integration is, they just see the pretty diagrams and assume it’s all easy

And of course, why are you charing so much???


Getting on with Business• The business requirements define workflow, not how to

implement the integration

• Should we really care what the message type is or it’s format?

• The workflow should be the same regardless of the messages we’re processing


Enter C24

C24 is a company (www.c24.biz)

Integration Objects (iO) is the product

Adam Skogman - VMWare

Oliver Gierke - VMWare

Oleg Zhurakousky - VMWare

Esteban Serrano - VMWare


http://www.c24.biz

http://www.c24.biz

C24‘s Integration Object

• Integration Objects is a Java-Binding tool that generates code based on built-in, imported or manually defined models

• Built-in includes standard Financial Services standards• SWIFT, FpML, ISO-20022, FIX, DTCC, ISO-8583, SwapsWire etc.

• Imports include• XML Schema, CSV, Binary, DTD, WSDL, Java, Databases

• Generated code has getters/setters, implements XPath 2.0 can execute XSLT and generates Spring friendly configs


So what’s the trick?

• Three parts to Spring Integration...• The core - built on Spring

• Communication protocol adapters (bottom)

• Data transformation adapters

• The latter is limited to Java & XML

Spring Integration

WS REST File JDBC Gemfire 3rd Party Plug-In

AMQP JMS

XML XML Java C24


• C24 does three basic things in Spring Integration

• Parsing input data• unmarshalling-transformer

• Transforming it

• Marshalling it back out

• To simplify configuration, a custom namespace is available for C24 under spring, which we'll see in the following slides

C24 UnmarshallingTransformer

String orbyte[]

Channel Channel

ComplexDataObject

C24 Transformer

ComplexDataObject

Channel Channel

ComplexDataObject

C24 MarshallingTransformer

ComplexDataObject

Channel Channel

String orbyte[]


Parsing (unmarshalling)

• Parsing is done by using an "unmarshalling-transformer"• “Transformer” because that's the thing it performs in spring

Integration lingo

• Just refer to a model (generated by C24) and the input byte array or Sting is converted to a C24 Complex Data Object

• The parsing will be done from the default format of the C24 model (such as XML, CSV or something like FIX)

<int-c24:unmarshalling-transformer model-ref="fpmlModel" input-channel="…" output-channel="…" />

<c24:model id="fpmlModel" base-element="org.fpml.x2010.fpml49.FpMLElement" />


Parsing

• The parsing can be further customized by specifying a sourceFactory• Useful for setting things like encoding

<int-c24:unmarshalling-transformer model-ref="fixModel" source-factory-ref="sourceFactory" input-channel="…" output-channel="…" />

<bean id="sourceFactory" class="biz.c24...source.FixSourceFactory"> <property name="encoding" value="UTF-8" /></bean>

C24 UnmarshallingTransformer

String orbyte[]

Channel Channel

ComplexDataObject


Transformation

• Real X-to-Y transforms are modelled in C24

• The generated transform slots into SI by just specifying the C24 generated class

<int-c24:transformer transform-class="biz.c24...basic.ExampleTransform" input-channel="…" output-channel="…" />

C24 Transformer

ComplexDataObject

Channel Channel

ComplexDataObject


Message Transformation

• Can be 100% code with getters and setters, XSLT, XPath or graphically...


Marshalling

• Marshalling is done using another transformer• The output can be customised by configuring a sink-factory

• The only mandatory part is the output type• Specify String or Byte Array

<int-c24:marshalling-transformer sink-factory="xmlSinkFactory" output-type="BYTE_ARRAY" input-channel="…" output-channel="…" />

<bean id="xmlSinkFactory" class="biz.c24.io.spring.sink.XmlSinkFactory" />

C24 MarshallingTransformer

ComplexDataObject

Channel Channel

String orbyte[]


Validation• C24‘s Integration Objects include syntactic and semantic rule

validation• This is included in the namespace

• One way of invoking C24 validation - by validating and putting the result in headers• The message is kept, so that downstream, you can put a filter or

router that acts on the header (valid/invalid) and does something with the message

<int-c24:validating-header-enricher fail-events="true" pass-events="false" statistics="false" />

<int:logging-channel-adapter channel="…" level="WARN" expression="headers[c24_failEvents]" />


Validation<int-c24:validating-header-enricher fail-events="true" pass-events="false" statistics="false" />

<int:logging-channel-adapter channel="…" level="WARN" expression="headers[c24_failEvents]" />

ComplexDataObject

Channel Channel

ComplexDataObject

Header Enricher


Now the clever stuff...

•We can parse, validate, marshal, un-marshal but we can also filter, route and transform

• C24 Integration Objects implement an XPath 2.0 navigator

• ANY message bound to an Integration Object implements XPath 2.0• This extends to XSLT, i.e. An XSLT transform can be applied directly

to the bound Integration Object

• If we could treat the Integration Object as if it were XML we could execute XPath on anything parsable


SWIFT - MT564 Corporate Action Notification

{1:F01INTRUS33AXXX9999999999}{2:O5640947040127FRNYUS33AXXX42181834250401270947N}{3:{108:MT564}}{4:

:16R:GENL

:20C::SEME//2003041800000042

:20C::CORP//12345

:23G:NEWM/CODU

:22F::CAEV//XMET

:22F::CAMV//VOLU

:98A::PREP//20010901

:25D::PROC//PREC

:16R:LINK

:22F::LINK//INFO

:13A::LINK//992

:20C::RELA//ABC

:16S:LINK

:16S:GENL

:16R:USECU

:35B:/ISIN/IDENTIFIER12

:16R:FIA

:12C::CLAS//ESVUFR

:11A::DENO//AUD


XPath on any message

• Take the earlier SWIFT message...

{1:F01INTRUS33AXXX9999999999}{2:O5640947040127FRNYUS33AXXX42181834250401270947N}{3:{108:MT564}}{4:

:16R:GENL

:20C::SEME//2003041800000042

:20C::CORP//12345

:23G:NEWM/CODU

:22F::CAEV//XMET

:22F::CAMV//VOLU

:98A::PREP//20010901

:25D::PROC//PREC

:16R:LINK

• As examples of XPath• To read the whole line with the date in it... /Block4/SeqA/Field98a2

• To read just the date... /Block4/SeqA/Field98a2/A/DateYYYYMMDD

• To read all the dates in Block 4... /Block4//DateYYYYMMDD

• Count the number of dates in Block 4... count(/Block4//DateYYYYMMDD)


XPath Routing

• This is the logical architecture of a large European clearing house

• Routing can be performed by applying XPath queries on the incoming Integration Objects


XPath Transformer

• Options: Object, String, Number, Boolean, List

<int-c24:xpath-transformer xpath-statement="//trade/tradeHeader" evaluation-type="OBJECT_RESULT" input-channel="…" output-channel="…"/>

ComplexDataObject

Channel Channel

ComplexDataObject

C24 XPath Transformer


XPath Router<int-c24:xpath-router input-channel="fpmlChannel" xpath-statement="name(//tradeHeader/following-sibling::*)" default-output-channel="wrongChannel"> <int-c24:mapping value="fxSwap" channel="aChannel" /> <int-c24:mapping value="fxSimpleOption" channel="bChannel" /></int-c24:xpath-router>

ComplexDataObject

Channel C24 XPathRouter

Channel

Channel

ComplexDataObject


Routing by Object Type

<int:payload-type-router input-channel=fpmlChannel"> <int:mapping type="org.fpml.x2010.fpml49.RequestMessage" channel="requestChannel"/> <int:mapping type="org.fpml.x2010.fpml49.ResponseMessage" channel="responseChannel"/> <int:mapping type="org.fpml.x2010.fpml49.NotificationMessage" channel="notificationChannel"/></int:payload-type-router>

• Since the Integration Objects are Java (not just DOM Nodes) we can route on type


Flexibility

<int:chain input-channel="…" output-channel="…"> <int:filter ref="boolSelector" /> <int:filter ref="stringSelector" /></int:chain>

<int-c24:xpath-selector id="boolSelector" evaluation-result-type="boolean" xpath-statement="//Employee/FirstName/text()='Andy'" />

<int-c24:xpath-selector id="stringSelector" evaluation-result-type="string" xpath-statement="//Employee/FirstName" string-test-value="Andy" />

• Two different ways of doing the same thing...


Header Enrichment

<int-c24:xpath-header-enricher input-channel="fpmlChannel" output-channel="richerChannel"> <int-c24:header name="fpml_sentBy" xpath-statement="//header/sentBy"/> <int-c24:header name="fpml_sendTo" xpath-statement="//header/sendTo"/></int-c24:xpath-header-enricher>

• Header Enrichment is a common way of pulling out information from complex messages and enriching the header for later filtering or processing


Some Use-Cases• NewEdge• Spring Integration & C24 Integration Object to connect to up to 28

different exchanges - into GemFire

• HSBC• Global SWIFT payments, Spring Integration & C24

• Norwegian Tax Office• Full replacement of the Norwegian Tax system, all records (people and

accounts) through C24 into GemFire

• Payments integration (somewhere on planet Earth)• Integration to Issuing and Acquiring banks for payments processing

• Many others• Citi, JPMC, Federal Reserve, RBC, BlackRock, M&G, Mizuho, BNP

Paribas, Indeval, ABN Amro, Dexia etc. etc.


It’s question time...

www.C24.biz


http://www.C24.biz

http://www.C24.biz

We need people

•We are working on multiple projects with VMWare in London, New York & San Francisco

• [email protected]

• Twitter: @jtdavies


mailto:[email protected]

mailto:[email protected]

Documents

Enterprise Integration - Whiteship's Eprilwhiteship.me/wp-content/uploads/2011/11/SpringOne_2011___John... · Enterprise Integration The seriously ... day you still need to understand