Dave Gorman gormand@uk.ibm - WebSphere Integration … - ESB03 - … · · 2014-06-11Dave Gorman...

IBM Integration Bus

How to analyse your system to optimise performance and throughput in IIBv9

Dave Gormangormand@uk.ibm.com

Overview

� The purpose of this presentation is to demonstrate how to find the cause of poor performance for an IBM Integration Bus node (broker) for two different types of problem.

� The examples are obtained on a Windows system but the principles of investigation and problem determination apply equally on all platforms. The system level tools will differ though.

Agenda

� Introduction

� Tools

� Techniques

� Demonstration

What are the main performance costs in message flow s?

ZYX…CBA

Parsing Tree Navigation

Root.Body.Level1.Level2.Level3.Description.Line[1];

Tree Copying

Set OutputRoot = InputRoot;

Resource Access Processing Logic

Integration Bus Processes

Execution group – provides message flow runtime environmentDataFlowEngine [n]

HTTP Listener (broker wide)biphttplistener

Administrative agentbipbroker

Administration agent availabilitybipservice

FunctionTask

� Important to understand:– Topology of broker– SOAP and HTTP listeners

Which resources and how much� In busy times expect to use what is needed (!)

– Exactly what will depend on the configuration and the applications– Typical to use CPU and memory plus I/O to some level

� In quiet times Message Broker and MQ processes – should use very little CPU– Should use very little I/O capacity– Will retain memory unless memory utilisation is very high

� The amount of CPU and memory used will depend on the situation– A complex configuration many MQ channels, hundreds of message flows will use much more

memory and CPU than a single message flow

� Some memory sizes– bipservice 4.6 MB– bipbroker 162 MB– biphttplistener 48 MB– DataFlowEngine 191 MB for JavaTransform samples

• can use from ~100MB to GigaBytes depending on• The number of flows, the complexity of the message flow, the size of the messages

� MQ processes– Expect it to be less than Message Broker (192 MB for simple queue manager on xLinux)– Will depend on number of open queues, channels, queue buffer sizes etc.

Tools That are Needed

� Monitoring Tools– At the operating system level to observe

• System resource usage – CPU, memory, I/O activity• Heaviest resource users

– At the component level to observe• Behaviour within the particular component (MQ/Integration Bus)

– Both types of tools of needed• They have different views of the world• They are complimentary

� Driving Tools– Needed to generate a continuous workload

• Important to assess performance after warm-up during sustained activity

UNIX Tools

� Vmstat

� Iostat

� Nmon

� filemon

System Configuration: lcpu=64 mem=8192MB

kthr memory page faults cpu----- ----------- ------------------------ ------------ -----------r b avm fre re pi po fr sr cy in sy cs us sy id wa1 0 1977672 25823 0 0 0 0 0 0 3 958 696 4 0 96 01 0 1977838 25719 0 2 0 98 100 0 29 2941 2250 4 0 96 01 0 1977685 25872 0 0 0 0 0 0 2 636 483 4 0 96 0

System configuration: lcpu=64 drives=5 paths=6 vdisks=2

tty: tin tout avg-cpu: % user % sys % idle % iowait0.0 29.5 3.6 0.1 96.2 0.0

Disks: % tm_act Kbps tps Kb_read Kb_wrtnhdisk3 0.0 0.0 0.0 0 0hdisk2 0.0 0.0 0.0 0 0hdisk0 0.0 4.0 1.0 8 0hdisk1 0.0 0.0 0.0 0 0cd0 0.0 0.0 0.0 0 0

Process Explorer on Windows

� Watch system activity in detail on Windows

� Watch– CPU Usage– Commit Charge– I/O Activity– Physical Memory History– Summary Information– Individual Processes

Download from http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Process Explorer

� DataFlowEngine.exe is the Execution Group� amqzlaa0.exe is the MQ agent for the broker� Can quickly see system is busy� Customise by selecting columns of choice

z/OS Tools

� SDSF

� RMFMON II

Key Tools at the Component Level

� Integration Bus

– User Trace– Trace Nodes– WebUI

• Compare flow statistics at the node (broker), server (execution group), container (application or library) or at an individual message flow level

– IIB Explorer• View resource use at the execution group level• View Activity Log

� MQ Explorer

� Java Healthcenter

Statistics Scope

Node (broker)

Server (execution group)

Message Flow

Thread

Message ModelTerminals

ResourceStatistics

Accounting &Statistics

� Using the WebUI in Integration Bus v9:

– Control statistics at all levels– Easily view and compare

flows, helping to understand which are processing the most messages or have the highest elapsed time

– Easily view and compare nodes, helping to understand which have the highest CPU or elapsed times.

– View all statistics metrics available for each flow

– View historical flow data

WebUI Statistics

Integration Bus Explorer & Resource Statistics

� View resource statistics for resource managers in IIB such as JVM, ODBC, JDBC etc

Integration Bus Explorer & Activity Log

� View activity as it happens using explorer

� Filter by resource managers

MQ Explorer

– MQ Explorer– Trace– Statistics

IBM Support Assistant and Java Health Centre

� Java Health Centre is provided as part of the IBM Support Assistant– Offers very low overhead monitoring tool– Runs along side an IBM Java application

� Get visibility, monitoring and profiling in the following application areas: – Performance

• Java method profiling• Lock analysis• Garbage collection

– Memory – System – Java Class– File input and – Object

� Enable the application JVM prior to useIBM_JAVA_OPTIONS=-Xhealthcenter

Demonstration of Analysing Performance Issues� Identify problems in two message flows using

– Process Explorer– WebUI Statistics– MQ Explorer– Java Healthcenter

� Message Flows

JavaComputeTransformNoXpathCoordinated Request Reply

Demo 1

Analysing a performance problem in the Coordinated Request Reply Sample

Coordinated Request Reply flows� Consists of three message flows

– Request• Converts incoming message from XML to CWF• Saves the incoming message in a queue for subsequent reply processing• Writes a message for the back end reply message flow

– Backend Application• Sets the completion time in the message• Writes a reply message

– Reply• Reads the message from the back end message flow• Retreives the original message saved by the request message flow• Writes an output message

Coordinated Request Reply queues� The queues

– Request

– BackendReplyApp

– Reply

GET_REQREP_IN GET_BACKEND_REQ

GET_REPTO_STORE

GET_BACKEND_REQ GET_BACKEND_REP

GET_BACKEND_REP GET_REQREP_OUT

GET_REPTO_STORE

Run and Investigate

� Steps1. Ensure all components are started and the application works as expected

• Message flows, databases, external applications etc2. Start a load generator [JMSPerfharness in this case]3. Look at activity

• Is processing happening at the expected rate• Is CPU usage as expected• Is memory usage as expected

4. If things do not seem as expected• Look for a build up of messages• Poor service times

5. Enable and view statistics6. Analyse statistics7. Examine message flows

Step 1 – Check flows are running using the WebUI

� Check the server is running� Check the flows are

running� Check the event/sys log for

any errors� Processing messages and

no errors

Step 2 – Start a load generator� Run Perfharness

– Use 10 threads

� All threads start successfully– Each thread PUTs a

message then GETs a message so should be no messages on queues for any period of time

� Check event/sys log for any error messages

Step 3 – Look at CPU activity

� Messages being processed but– Rate is low, much lower than expected– Very little CPU being used

• Execution group does not register any CPU activity

� Key queues are GET_REQREP_INGET_REPTO_STORE RequestGET_BACKEND_REQ

GET_BACKEND_REQ BackendReplyAppGET_BACKEND_REP

GET_BACKEND_REPGET_REPTO_STORE ReplyGET_REQREP_OUT

Step 4 – Look for a build up of messages

• Build up of msgs on queues:GET_REPTO_STOREGET BACKEND_REQ

• What does this mean?

Step 4 – Look for a build up of messages…

� Looking at the flows– Queue GET_REPTO_STORE is used by Request and Reply message flows– Queue GET_BACKEND_REQ is used by BackendReplyApp message flow

– GET_REPTO_STORE is used mid-flow (so flows using this are less likely to be problem)– GET_BACKEND_REQ is the input queue for the BackendReplyApp,

• Indicates flow is not running fast enough or not enough instances allocated

� Need to investigate what is happening with BackendReplyApp– For this use WebUI flow statistics

Step 5 – Enable flow statistics

� Start and stop statistics using the WebUI for:

– All flows in a server– All flows in a container– Individual flows

Step 5 – View statisticsFlow comparison views

� Select the statistics view� Drill down to the problem flow� Start by comparing flows� Flow Analysis view for most detail

Step 6 – Compare flows

� Compare flows to determine which one might be causing the problem� We can see that the BackenReplyApp flow has an average elapsed time of

1010.1 millseconds. It only has 1 active thread, and has processed 20 messages in the 20 second statistical snapshot period.

� This matches the rate we see from PerfHarness!

Step 5 – Analyse the flow

� Display historical flow details such as message rate, CPU and elapsed time

� View all nodes within the flow to determine and sort by average elapsed and CPU times

� The compute node Modify_CompletionTimeseems to be a problem!

� What does high elapsed time and low CPU time suggest the problem might be?

5 – Review the code

� Having worked out which node is causing the problem� We can quickly see which why the node is taking 1 sec elapsed time but little CPU

Problem Found!

� 1 second sleep in the compute node within the message flow is causing slow processing times and no CPU usage

– Matches the observations at the start• Low CPU and low rate

� Unlikely to be so easy in future but slow service times, like slow synchronous web service invocation would have the same effect

� If it was slow web service response then allocate more instances to improve processing rate

Summary of Steps for this Investigation

� Use a systematic approach– Key steps used were

1. Ensure all components are started and the application works as expectedMessage flows, databases, external applications etc

2. Start a load generator 3. Look at activity

Is processing happening at the expected rate

Is CPU usage as expectedIs memory usage as expected

4. If things do not seem as expectedLook for build up of messages

5. Analyse accounting and statistics6. Examine message flows

� It is very important to – Use tools

• System level and component level– Start at a high level – system level and then close-in on the problem

Demo 2

Analysing a performance problem in the JavaNoXPath Sample

Demonstration – JavaComputeTransformNoXpath� Consists of one message flow

– JavaComputeTransformNoXpath• Reads an XML message• Transforms to a different format using a Java Compute node

JAVACOMPUTE.TRANSFORMNOXPATH.IN JAVACOMPUTE.TRANSFORMNOXPATH.OUT

What is the problem we need to solve?

� The problem is characterised by– Low message rate– High CPU usage at both system and Execution Group level– Sufficient messages on the input queue

� Likely issue is one of high CPU usage in a message flow– But which flow and which node?

Compare the flows

All of the Elapsed and CPU time is in JavaComputeTransformNoXpathFlow so continue investigation of this flow

Finding the Processing Node for Investigation

The majority of the elapsed and CPU time within in the flow is spent in the JavaComputeTransformNoXpath node

What might cause this?

As this is a Java Compute Node continueinvestigation using the Java Healthcenter

Find the Execution Group JVM Health Port for Java H ealth Center

Environment variable :

IBM_JAVA_OPTIONS=-Xhealthcenter

Open ports starting 1974, the JavaComputeTransformNoXPathFlowDataFlowEngine is 1976

Alternate Method for Finding the port number

Find the Process ID of the execution group with mqsilist and netstat

Invoking the Java Health Center

Attaching to the Execution Group JVM

Connect to a port

Connection Complete and Ready to Analyse

Analysis and Recommendations - Classes

Analysis and Recommendations - Environment

Analysis and Recommendations – Garbage Collection

Analysis and Recommendations – Native Memory

Analysis and Recommendations - Profiling

The cause

� Having worked out which node is causing the problem

� We can quickly see why the node is consuming a lot of CPU

� A call to the method bubble_sort() just before propagating out of the node is sorting the entire output message

If you suspect there is a product problem

� Identify the problem as best you can

� Find the simplest test that recreates the problem

� Collect the data identified in the Must Gather list– For IIB http://www.ibm.com/support/docview.wss?rs=849&uid=swg21209857

– For MQ http://www.ibm.com/support/docview.wss?uid=swg21229861#MG6

Summary

� Wide range of tools available covering operating system and component performance– Expect to use multiple tools– After all it is important to understand what is happening at different levels– Demonstration has shown how to use the key tools for MQ and IIB to debug a problem

� Practice before hand– Being familiar with the tools is a great help in a crisis– Learning a new tool and solving a crisis is not a good combination

� Know your applications and systems– What is normal in terms of processing rate, CPU usage etc.– This information allows to know whether there is a problem and to what extent

Additional Information

� WebSphere Message Broker: Designing for Performance � http://www.ibm.com/support/docview.wss?rs=849&uid=swg24006518

� WebSphere Message Broker V7-Message display, test & performance utilities (IH03) http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg24000637

� IBM Monitoring and Diagnostic Tools for Java - Getting started with Health Centerhttp://www.ibm.com/developerworks/java/jdk/tools/healthcenter/getting_started.html

� IBM Monitoring and Diagnostic Tools for Java - Health Center Versionhttp://www.ibm.com/developerworks/java/jdk/tools/healthcenter/

� IBM Monitoring and Diagnostic Tools for Java™ - Health Centerhttp://publib.boulder.ibm.com/infocenter/hctool/v1r0/index.jsp

Dave Gorman gormand@uk.ibm - WebSphere Integration … - ESB03 - … · · 2014-06-11Dave Gorman...

Documents

Siebe-Gorman Diving Suits - Hydroglove · Siebe-Gorman underwater swimsuit accessories Siebe-Gorman also supplied a range of woollen underclothing to be worn under dry suits. The

Gorman California Hotels

Final Project Paola gorman

Gorman paving

The Do’s and Don’ts of IBM Integration Bus Performance · PDF fileThe Do’s and Don’ts of IBM Integration Bus Performance Dave Gorman : IBM Integration Bus Performance Session

Gorman Last Hours Jesus

Gorman Family Reunion 2002

Ralph Bateman ralph@uk.ibm - SHARE...Ralph Bateman ralph@uk.ibm.com STSM Connectivity and Integration Customer Support. IBM Hursley Please Note IBM’s statements regarding its plans,

ONLINE SHOPPING! SHANI GORMAN (HTTP://

California Regional Water Quality Control Board Los ... · 3 Mr. Richard Bechguenturiant, McDonald’s Restaurant in Gorman, 49714 Gorman School Road, Gorman 7/31/18 4 Mole-Richardson

Gorman-Rupp Company

Art app gorman

Polemica O Gorman

SLUDGEKATTM - Gorman-Rupp Company

Gorman Steven1331scanned

Online Shopping! Shani gorman

Alan Powell Alan powell@uk.ibm - WebSphere User …...– Message Nodes which encapsulate required integration logic which operate on… – Message Tree which describes the data in

Hayward and Gorman 2004

TCA August 2014 - Gorman Company

BRAD GORMAN MEMORIAL BIKE TRAILHEAD - SCVBACscvbac.org/docs/GormanMemorial.pdf · from the Brad P. Gorman Memorial Bikeway located ... Please support the Brad Gorman Memorial Bike