Lessons Learnt Implementing High-Performance Integration using SAP PI

Premier Partners: Supported by: Produced by:

Sascha Wenninger - Australia Post

Recipes For the Perfect PI - Simple Ingredients for Complex Requirements

Mastering SAP Technologies 2011

Bottom-Up Approach

Photo by Gidzy

http://www.flickr.com/photos/gidzy/3118871380/

http://www.flickr.com/photos/gidzy/3118871380/


You’ll Hear About:

Basis

Adapter Framework

Interface Design

Questions

Key Points to Take Home


30,000 Foot View

SAP POS DM

SAP ERP

SAP SCEM

…JMS Queue

XMLXMLXMLXML

SAP PI

Routi

ng

Mapping

Mapping

Mapping

Mapping

Mapping…


Some Figures…

Retail Transactions from 8,000+ terminals in 3,400 stores:

500,000 – 750,000 messages per dayPeaks of 45/second into PI, then split by receiver6 receiving systems, 1-3 receivers per messageMessage size <10kB

Roll-out: January – June 2011


Objectives

Near-real time: <15 minutes end to endScalable to 2,000,000 messages/day and 14 receiver systemsFuture peaks of 90/second into PINo impact on other critical PI interfaces

e.g. Parcel Tracking interface moving 2m+ events per dayAs simple as possibleFuture-proof (no ccBPM, no ABAP mapping, etc.)


The Solution?

Many small steps enabled us to get there.


Key Topic

Basis


Basis

• System Memory• Database Stuff• Java Server Nodes• The JVM• Java Heap Space• Wily Introscope


System Memory

One of the main foundations of the system

Needs to accommodate all aspects of the system:Java + ABAP + ICM + DBMS + OS + caching/buffers + other apps

Need to avoid swapping to disk at all times – causes huge GC delays!


The Database

DB StatisticsWithout DB stats, our throughput dropped to 1/6th of the normal! Run daily in production, but be careful in non-prod systems

DB ReorgsPI DB tables churn very frequently and fragment after a few weeksTest your system/DBMS before and after a reorg to evaluate value


Java Server Nodes

More Java Server Nodes (Java processes) provide:• Better system memory utilisation• Increased throughput via parallelisation

…but also incur costs in cluster synchronisation.

This can be significant for high volumes of EOIO interfaces with more than a few Java Server Nodes


The JVM

• PI 7.1 and later use SAP’s own JVM– SAPJVM 4.1 for PI 7.0 coming soon (Notes 1495160, 1522198)

• Bug fixes and performance improvements delivered continually

Several JVM Upgrades over the past 12 months significantly reduced OOM crashes and Garbage Collection times for our systems.

This enables the use of larger heap spaces!

Patching via JSPM takes 30 minutes and 1 restart, plus some testing


Java Heap Space

More Heap = process larger messages, fewer full GCs, fewer OOMs

• SAP’s recommendation has historically been 2GB per Java process. – Recently changed to 4GB (for SAP JVM)

• AusPost has been using 4GB for 9 months now. Main criteria: Full GCs should take less than 10 seconds.


Java Memory Analysis

This is in SolMan!


Wily Introscope

• Monitoring for the Java Stack• Basic version is free with NetWeaver Java• Provides functionality normally accessed via ABAP transactions


Wily Dashboards


More Dashboards


And More…


Tons of Dashboards!

…and this is only the free version!


Our PI System

PI 7.11 SP5JVM 5.1 Patch level 59 (64 in test systems)3 Java server nodes, 4GB heap space each.AIX 6.1 on IBM POWER6, 60GB RAM


Basis

System Memory Database Stuff Java Server Nodes The JVM Java Heap Space Wily Introscope

Many aspects which require special PI-Basis skills!


Key Topic

Adapter Framework


Adapter Framework

• Thread Behaviour• Messaging System


Threads

Messages are processed using a number of adapter-specific queues.Each queue has its own thread pool.

e.g. for JMS:

Queue Purpose Default Threads

JMS Send Async Sender 5

JMS Recv Async Receiver 5

JMS Call Sync Sender 5

JMS Rqst Sync Receiver 5


Threads

Threads are assigned to process messages from their respective poolsUse Wily Introscope to monitor utilisation and backlogs


Threads

In ‘traditional’ scenarios, both Sender and Receiver queues and thread pools are used.

For Integrated Scenarios in the AAE, only the Sender-side queues and thread pools are used!

Threads from the Sender pools do all the work!


Caveat

Some Adapters process messages serially!

e.g. JMS: 1 Adapter Thread per Communication Channel reads messages from the JMS queue

1 JMS Comm Channel3 Java Server Nodes{


Threads in the AAE

Adapter Thread picks up a message from the JMS queue, then:• Executes Adapter Modules• Performs Receiver Determination• Persists 1 message per Receiver in Messaging System queue.• Confirms message and removes from the JMS queue

The longer this takes, the lower throughput will be.


What does this mean?


Sender-Side Performance is Really Important


Implications

Throughput depends on Sender Adapter performance

Try to avoid adapter modules in Sender adapters. We achieved a 15% improvement by moving an XSLT mapping from

the JMS adapter to the Operation Mapping.


Implications

Throughput depends on Receiver Determination

The initial “Adapter” thread evaluates conditions and copies the message for each Receiver.

Optimise XPath conditionsDB performance: more Receivers = more I/O


Implications

Throughput depends on parallelisation

Increase the Sender-side Thread Pools and Application Threads. We use 15 JMS Sender Threads and 350 Application Threads

(also see Note 937159)


2 Options

In our system, each thread can process 10-15 messages/sec

In order to consistently process 45 messages/sec from JMS, we need 4-5 concurrent JMS Sender Adapter threads.

Option 1: 5 Java server nodes

Option 2: Multiple JMS Communication Channels.


Multiple Comm. Channels

We cloned the Integrated Config Object and Sender Comm Channel2 JMS Sender Adapter Threads per Java Server node

Better thread pool utilisation No increased memory requirements More scalable

This is not needed for SOAP since it’s truly parallel.Image Source: http://t.co/ZOBdlH0

http://t.co/ZOBdlH0


Adapter Framework

Thread Behaviour Messaging System

Increased importance for Integrated Configuration scenarios!


Key Topic

Interface Design


Integrated Configuration

Receiver Determination+ Interface Determination+ Receiver AgreementsIntegrated Configuration Object

Causes an interface to be executed in the Advanced Adapter Engine


Integrated Configuration

• Message Processing entirely in Java (AAE)• Introduced in PI 7.1; more features in 7.11 and 7.3• Improved performance by:

– Reducing database I/O– Eliminating ‘stack jumping’ between ABAP and Java

• 7-10 times throughput possible• Not available for all adapters and scenarios.


Example


Example


Constraints

There are a few things you can’t do with Integrated Configuration:

IDocs*

Multi-Mappings*

ccBPMs

But this is an Opportunity to be Creative!

*features added in PI 7.3


Mapping Programs

XSLT• Easier to debug and tune• Better performance from PI 7.1 onwards• Easier to support

Consolidate mapping stepsSame performance, but less GC!• Reuse still possible via XSLT imports.


Communication Channels

ABAP Proxies/Enterprise Services Use SOAP Adapter in XI 3.0 mode

JMS Senders Clone comm. Channels for greater parallelisation

Non-SAP Systems Ensure inbound queuing is available to avoid backlog in PI


Design-Time Governance

Don’t Overdo It!

Find a balance which facilitates reuse but doesn’t impose too much up-front work.

Image Source: http://geekandpoke.typepad.com

http://geekandpoke.typepad.com/geekandpoke/2008/06/the-benefits-of.html


Questions


Key Points to Take Home

Wily IntroscopeEssential for understanding performance of the Java stack

Use Integrated Configuration Objects20 minutes to migrate an existing interface

Keep your JVM CurrentContinuous improvements without much fuss

Most of the other points can be quick wins too if you don’t do them all at once.


Sascha WenningerAustralia Post

[email protected]

@sufw

+61 403 933 472


Further Reading

• PI Performance Check• SOA MiddlewareArticles on SDN

– Incl. the Performance section

http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/f0b96675-87cf-2c10-b489-dee0ac03f782

http://www.sdn.sap.com/irj/sdn/nw-soa

http://www.sdn.sap.com/irj/sdn/soa-servicebus?rid=/webcontent/uuid/70ba8e92-214f-2a10-1087-8fa768d407f8%23section2

Premier Partners: Supported by: Produced by:

Technology

Lessons Learnt Implementing High-Performance Integration using SAP PI