Service Integration Bus Best Practices in WebSphere ... - IBM

IBM Software Group

®

WebSphere® Support Technical Exchange

Service Integration Bus Best Practices in WebSphere Application Server

06th September, 2016

Jhansi Kolla

[email protected]

IBM® Support Technical Exchange

mailto:[email protected]

IBM Support Technical Exchange

WebSphere® Support Technical Exchange 2

Agenda

We will discuss the most common Service Integration

Bus (SIB) best practices in WebSphere Application

Server (WAS).

How to tune the SIB component to get the maximum

outcome. (SIB performance tuning)

How to improve the performance by selecting proper SIB

policies according to your architecture and environment.

Other WAS runtime components that effect SIB

performance and how to tune them.



Intoduction

SIB is the Default Messaging Provider in WAS.

Many WAS stacked products use SIB as their base

component. So this WSTE also applies to all those

stacked products.

We will also cover a few things which are not available in

the info center.



SIB Tuning Levels

There are different ways we could configure SIB, in order

to improve SIB performance and best outcome of it:

Configuration

Runtime

JMS™ Producer side

JMS™ Consumer side



SIB Policy Options: HA + Scalability If you are using messaging cluster, you can avoid a single point of failure by

creating and running more than one Message Engine (ME) (at least 2). You can

always manually add messaging engines (in case of just HA policy is setup).

It‟s always good to have Scalability + High Availability if your architecture allows.

Best practice: To improve the throughput and scalability of the messaging functionality:

Add MEs to handle very large numbers of connections and rely on remote put to forward

messages to their destination.



Preferred Servers Only Option In the case of default HA policy if you choose to run your ME on preferred

servers only, then make sure that you have at least one spare JVM™

available to host ME (in case if ME hosting server has any issues and ME

needs to failover).

Add more

preferred

servers

Best Practice: Configure custom One of N policy with different preferred servers (if

you configure to run the ME on preferred servers option) for each ME and with

failback option.



JVM Heap Settings for Performance Each message residing inside a Messaging Engine(ME) needs around 400 bytes of Meta data (header) of heap memory, and in the case in

Pub-Sub scenario if a durable subscriber is not active then Messaging Engine has to hold a message reference apart from actual Message.

Each and every message references consumes around 100 bytes of Meta data (header) of heap memory.

So while designing the JVM memory requirements, message load (i.e the number of messages present in Messaging Engine) has to be

considered including in failure cases. In general it is better to set the minimum and maximum heap sizes to the same value for maximum

performance. This setting avoids costly JVM Garbage Collector (GC) compaction operations.

Message Retrieval Time: Messages have to be read from the message Store to run search/selector algorithms on them. The best way to

reduce message retrieval time is to increase JVM heap sizes so that more messages can be cached in memory.

Best Practice: Configuring sufficient JVM heap size to avoid the hangs or OOM during

message processing and improve performance. We suggest minimum of 2 GB or above in

customers production ME JVMs. In general setting same Max and Min heap size avoid costly

JVM garbage collector (GC) compaction operations.



Balancing Message Workload• If the producing application is connected directly to an ME in the cluster, all its

messages will go to the local SIB partition destination (by default). To spread the

messages across the ME‟s in the messaging cluster choose below configuration.

• This also increases the throughput of the message producers.

(Note: in this case message order can no longer be guaranteed due to the workload balancing of messages across queue partitions).

Best Practice: To increase the messages throughput and Work Load Management (WLM)

across the messaging cluster JVMs, configure “don‟t prefer a local queue point over the other

queue points” and “Messages may be sent to different queue points” in the queue

configuration panel.



Balancing message workload

Best practice: To increase the throughput and WLM choose above options on the

Queue object that points to the JMS Destination.

Producer Example



remotePut (Producer side) The producing application, connected to ME 1, produces messages to a queue

'Raleigh' localized on ME 2

ME 1 creates a remote queue point (RQP) to forward the messages. Once message

reached RQP then ME1 sends that message to MQ2. This eats system resources.

Best practice: If your architecture allows, always connect the application locally to

avoid the remotePut for better performance.



remoteGet (consumer side)When consuming from a Queue Point that is not located on the same ME that the application is

connected to, a process similar to store and forward is occurring, this is generally referred to as

remote get. The connected ME (ME2) requests a message from the ME with the Queue Point B (ME1)

The ME1 selects a message, persistently locks it for the ME2 and sends a copy of the message to ME2 (store and forward)

The consuming application receives the message, processes it and commits the receive.

The ME2 persists the act of committing the message and asynchronously informs ME1 of the commit.

Asynchronously, ME1 deletes the message and ME2 removes the commit state.

Best Practice: remoteGet eats lot of system resources. For better performance ALWAYS

avoid remoteGet. Make sure your consumer applications are connected to the local ME

which host the SIB destination.



Pros and cons of remotePut & remoteget

Best Practice: Configure your system to avoid remote get (always)



Avoid remoteGet (Continued)• By default, applications are able to connect to any available ME and prefer an ME in the same server, if one is

available. Connection Factory and Activation Specification definitions provide three Target properties which allows

you to target specific MEs or groups of MEs when establishing connections.

Target

• The name of an ME or a group of MEs to connect

Target Type

• Bus member: Targets MEs in a specific server or cluster

• Custom messaging engine group: Allows you to define a custom group of engines to satisfy the connection

• Messaging engine: Targets a specific ME by name

Target Significance

Required: No connection will be established unless an ME satisfying the target criteria is available

Preferred: If none of the target MEs are available it connect to any ME in that BUS.

Best Practice: Only let messages on a single queue point be visible in order to avoid remote gets and improve

scalability. The above setting avoids the remoteGet.



Always minimize the messages in the Message Store

• Minimize the number of messages currently stored in the message store and in the

cache.

• This is both messages on queue points (and subscriptions) and messages stored for

transmission to other MEs (remote queue points) in this or other buses (SIBLink and

MQLink).

• In a healthy environment as soon as a message arrives on the SIB destination, a

consumer should pick it up. Messages on the SIB destinations should be zero or near

to zero.

Best practice: Do not use queues for long term storage of messages. Keep the queues

depth minimum all the times. SIB is not a database.



Data Store Locking Issues

• If ME is abnormally shutdown or the connection between ME hosting server

and data store hosting machine is broken, the lock on the data store is not

removed automatically. When the ME starts next time, the existing stale lock

on the data store won‟t allow the new ME instance to take ownership and the

ME won‟t start.

• To resolve this enable the KeepAlive at the DB server to lower value like 4 or

5 minutes (default is 2 hours)

Best practice: Configure the KeepAlive at DB server to a lower value like 4 or 5

minutes to release the stale lock on the SIBOWNER table. Keep in mind of

connectionTimeOut and transactionTimeOut when you set this keepAlive.



Enable Restrict Long Running Locks• In WAS version 8.5 a new locking mechanism is implemented that solves this problem without the

need for setting the KeepAlive timeout. Just enable the “Restrict long running locks” feature in the

messaging engines Message Store configuration and restart the messaging engine.

• A new algorithm that ensures ME no longer holds long running locks on the SIBOWNER table.

• Active ME now holds only short locks on the SIBOWNER table while revalidating its ownership at

regular intervals

• The right to own the SIB tables is always performed during ME start-up



Detect JVM hang and failover• The messaging engine holds ownership of the database as long as it is active and processing. The

moment the messaging engine goes down or loses its connection to the database, the ownership is

lost to another instance as decided by the HA manager. The standby instance has no trouble

coming up as there are no database locks held by the failed messaging engine.

• Whenever active ME server does not crash, but hangs then this MELastUpdate don‟t update the

time after heartbeats are lapsed it removes server1 ME lock on data store, and server2 ME is

started with out any issues.

Best practice: Enable “Restrict long running locks” from WAS 8.5 onwards.

Note: if you migrated from

WAS 7.* or 8.* to 8.5 you

need to run the

“sibDBUpgrade” command

in order to modify the SIB

tables to use this new

function in WAS 8.5.



File Store Locking Issues

• NFS3 has some file locking issues. Don‟t use NFS3 for a file store.

• In case of clusters make sure file store is accessible from all ME servers in that cluster.

• Run the IBM File System Locking Protocol Test for WebSphere Application Server:

http://www.ibm.com/support/docview.wss?uid=swg24010222

• Network File System (NFS) Recommendations for WebSphere Application Server:


Best practice: Use NFS4 or approved file system (as per above test) for the file store.


http://www-01.ibm.com/support/docview.wss?uid=swg2401022




Data store performance settings

The workload that ME imposes on DB is slightly different than normal DB workloads.

ME performs mainly SQL INSERT / DELETE. This has to be considered while DB

tuning.

Each ME can request a large number of concurrent conns to the DB. By design ME

used many threads to perform DB updates concurrently.

Best practice: Making available sufficient number of DB connections to ME is the key

to have better performance during the peak loads.



Thread pool settings Default: Each ME can request a large number of concurrent connections to the DB. By design ME

uses many threads to perform DB updates concurrently.

SIBFapThreadPool is the service integration bus format API protocol (SIBFap) FAP outbound

channel thread pool, and it is utilized by all JMS application sending or consuming messages

to/from the server. The optimum value for its maximum size should be around 50.

Best practice: By default these SIB threadPools are enough, but if you have

many Message Driven Beans (MDB)‟s then you need to revise these SIB

threadPools settings in addition to the default threadPools.



Consume All Messages Before ME Stop

If you are planning to stop the ME, consume all messages from SIB destinations before

stopping. It helps to start the ME quickly when you restart the ME.

If the ME has thousands of messages on queues and the ME is stopped (precisely or

abnormal shutdown), ME restart takes longer because it must reconstitute (recreate) all

message on the queues. To avoid this try to consume all the messages from the SIB

destination as much as possible.

Best Practice: Consume all messages before ME stop. For maintenance

purposes stop the ME first and then JVM, this makes proper house keeping of

messages and ME shutdown. Never shutdown ME abnormally.



Message Reliability Vs Performance• Plan your configuration carefully to achieve maximum reliability while minimizing complexity

• Default message reliability (Quality of Service) is Assured Persistent. This guarantees that messages will never be lost but

it also has a high impact on performance.

• The fastest messaging can be achieved with Quality of Service is set to Best Effort Nonpersistent. These

messages are not stored in the message store and so messaging performance is much faster... but it is also much easier to

lose messages. This setting is best for highly transient data that is valid for a short period of time and will soon change.

• There are several other settings in between Assured Persistent and Best effort Nonpersistent. A table in the WAS Info

Center shows each and explains the conditions under which each type of message would be discarded by a messaging

engine.

• http://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/cjj9000_.html

Best effort non-persistent

Express non-persistent

Reliable non-persistent

Reliable persistent

Assured persistent.

Best Practice: Leave messages at the default Assured Persistent level. It is slower but it's the safest option and in most

cases performance using this QoS isn't an issue.

• In case if you have to change the default reliability level, plan your configuration carefully to achieve maximum reliability

while minimizing the complexity. Note: As the reliability level decreases, the faster the messages can be processed.

http://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/cjj9000_.html



ME not starting:• There are many reasons for ME is not starting.

• Check whether ME is not deactivated. If so enable the ME through the core group settings.

• Check for any locking issues of the message store.

• In case of clusters check all ME servers logs if ME started in the other servers.

• Resolve any file store or data store issues.

• Make sure HMGR is enabled in case of ME clusters.

• If you configured to run ME on preferred servers only and check whether any preferred server is running

with out any issues.

• Split brain or any communication issues between the ME server in clustered environment.

Other useful links:SIB- Messaging Engine startup problems

SIB Messaging Engine Data Store Connectivity Problems and Solutions

SIB Messaging Engine startup problems and their solutions

SIB message store problems

SIB Message flow problems






https://www-01.ibm.com/support/docview.wss?uid=swg27036148






Scaling producers• In case of HA SIB policy add MEs to handle very large numbers of

connections and rely on remote put to forward messages to their

destination

• Adding more MEs when using messaging clusters will automatically create

additional queue partitions. Each partition has the capacity of a normal

queue point. The queue capacity scales linearly with the cluster. (note: at

the same time don‟t use the queues for long term storage or don‟t use like

database)

Best Practice: Add MEs and configure preferred servers to handle very large numbers of

connections and throughput.



Don‟t use the Message listener ports. These are very old

fashioned and low efficiency consumers.

Use Activation Specifications which are more productive

and more flexible.

Best Practice: Use activation specification at consumer

side for better performance.

Consumer/MDB Side Tuning



Why are messages piled up on SIB destinations??

Producer is producing the message faster than consumer consuming

the messages.

Consumer is hung or having its internal issues. (Java™ cores will

show any hangs during that time)

Consumer is not running.

Check if any of above applies to your messages accumulation.

Best Practice:

1) In the Activation Specifications configuration panel: Increase the

„Maximum concurrent MDB invocations per endpoint‟ if you want to

increase maximum throughGet.

2) Activate MDB‟s in all servers

3) In queue configuration panel: configure „Messages on all queue

points are visible‟.



Consumer/MDB Side Tuning Message consumers in WebSphere are almost always Message Driven Beans (MDB). MDBs use Activation

Specifications for message consuming. Important performance related options include:

Always activate MDBs in all servers.

Maximum concurrent MDB invocations per endpoint (10 is the default). Increasing this value can improve

throughput, especially if MDBs are slow to complete.

Even if the MDB is deployed in the cluster, if you don‟t configure “Always activate MDBs in all servers” only the local MDB

where the ME is running is able to consume messages. If you enable this, all MDB‟s on all servers are able to

consume messages from the remote MEs. This greatly increases message consumption rates.

Best practice: Tune your MDB thread pool to ensure it can run the maximum number of MDBs without waiting. Configure

“Always activate MDB‟s in all servers” and set Maximum concurrent MDB invocations as per your architecture allows.



Configuring dedicated threadpools for MDB‟s

ThreadPool I WSVR0629I: The request buffer for thread pool "SIBJMSRAThreadPool"

has reached its capacity.

SIBJMSRAThreadPool is the thread pool used by consumer applications (MDBs) to

connect to messaging engines.

In general default threadPool settings are enough in normal conditions. Lets say, you 8

MDB’s and each MDB has 10 default “Maximum concurrent MDB invocations per

endpoint. You may need to increase the SIBJMSRAThreadPool setting to 80. If you do

not modify this, MDBs many need to wait for the threadPool to make threads available.

In this situation, MDB’s message processing may get hung.

Best practice: Make sure you have configured enough SIBJMSRAThreadPool threads

available for the MDB’s. Tune your SIBJMSRAThreadPool to ensure it can run the

maximum number of MDB’s (configured) with out waiting. If you need to increase this

setting you may need to review the over all server wise threadPools too.



Scaling consumers

•By default a JMS consumer is only able to consume messages stored on the queue

partition that it is connected to. Even if there are messages on the other queue points of

the same cluster queue, by default consumer is not able to see them. If MDB is not able to

see the messages on the remote queue points, this can be fixed by enabling the

“Message visibility” option

Best practice: To increase the messages consumption rate, set “Messages on all

queue points are visible”. This makes, let consumers consume the messages

from all queue partitions.



Best practices for using JMS connections• Once the application is successfully created the connection by getConnection()

method, your application must ensure that connections are closed whenever they

have completed their work. If a connection exception is thrown, WAS does not

automatically call connection.close() on behalf of your application connection.

• Application should call the connection.close() method in the finally {} block.

• Use Shared connections vs unshared connections to suit your applications design.

Default is Shared connections. Sessions should never be used across multiple

threads.

• Shared connections are transaction scope. Until transaction is closed, connection

don‟t go back to the Free Pool even though your application called the

connection.close() method.

Best Practices:

• Call connection.close() in finally {} block so that connection is properly closed.

• For long running transaction use unshread connections. It close the connection once

application call connection.close().



Best practices for Work Load Management• If WLM is important, set the server weights accordingly. The SIB TRM (Topology

Routing Manager) is the component that does the proper routing of the SIB messages.

SIB TRM works based on the WLM.

Best practice: In case of SIB clusters, if you need work load spread across all servers

set the servers weight equally. In addition to this in the queue configuration choose

“Do not prefer a local queue point over other queue points” and “Messages may be

sent to different queue points”.



Best practices for HA in SIB

• If HA is not enabled in the messaging cluster servers, ME failover won‟t happen.

• In high availability policy messaging clusters, it generate only one ME in the entire

messaging cluster. Always add one or two messaging engines extra to avoid single point

of failure in the environment. This also helps in performance improvements and

increased throughput.

Best practice: Make sure that HA is enabled in all servers in the SIB clustered servers. Make

sure at least 2 or more ME‟s are configured even in the HA policy messaging clusters.



sib.msgstore.cachedDataBufferSizeIn SIB, message is first stored in memory. This cache has a limited sized and the default

setting could possibly be insufficient for intensive loads. The purpose of the cached data

buffer is to optimize the performance of the messaging engine by caching in memory the data

that the messaging engine might otherwise have to read from the data store.

cachedDataBufferSize contain data for which the quality of service (QoS) attribute is better

than best effort nonpersistent. These messages are spilled from the heap to the datastore

when the in memory cache is exhausted. This cache can be controlled by the following 2

variables:

sib.msgstore.cachedDataBufferSize (default 320000 ~320KB) - the size in bytes

of the cache

sib.msgstore.cachedDataItemMaximumSize (100000) - the maximum size an

object allowed into the cache

When a message is spilled from memory (cache filled up) it is persisted to the database. He

Best practice: To improve the performance, you may need to increase these values, at the

same time be aware that these cache sizes are taken from heap. So you may also need to

increase the heap too depends up on your requirements.



sib.msgstore.DiscardableDataBufferSize

discardableDataBufferSize refers to the data for which the quality of service

attribute is best effort nonpersistent.

This data comprises both data that is involved in active transactions, and any other best effort

nonpersistent data that the messaging engine has neither discarded nor consumed. The

messaging engine holds this data entirely within this memory buffer and never writes the data

to the data store. When the messaging engine adds data to the discardable data buffer, for

example when the messaging engine receives a best effort nonpersistent message from a

client, the messaging engine might discard data already in the buffer to make space. The

messaging engine can discard only data that is not involved in active transactions. This

behavior enables the messaging engine to discard best effort nonpersistent messages.. The

messaging engine holds this data entirely within this memory buffer and never writes the data

to the data store.

Default value is: 320000 (approx. 320K)

Best practice: Increasing the size of the discardable data buffer allows

more best effort nonpersistent data to be handled before the messaging

engine begins to discard messages.

cachedDataBufferSize defines the depth in bytes of messages with nonpersistent qualities of services.



SIB custom properties

• Most commonly used SIB custom properties:

https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/commonly_used_ser

vice_integration_bus_custom_properties_in_websphere_application_server

• Complete SIB custom properties:

https://www.ibm.com/support/knowledgecenter/SSEQTP_8.0.0/com.ibm.websphere.nd.doc/inf

o/ae/ae/rjj_custom_props.html

https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/commonly_used_service_integration_bus_custom_properties_in_websphere_application_server




https://www.ibm.com/support/knowledgecenter/SSEQTP_8.0.0/com.ibm.websphere.nd.doc/info/ae/ae/rjj_custom_props.html

https://www.ibm.com/support/knowledgecenter/SSEQTP_8.0.0/com.ibm.websphere.nd.doc/info/ae/ae/rjj_custom_props.html



CWSIT0103E - No messaging Engine found troubleshooting

• We do see many PMRs quite often with this issue (Even though ME is running).

• I created a blog how to debug this issue with step by step walk through instructions.

• Blog:

https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/common_reasons_fo

r_CWSIT0103E_no_messaging_engine_was_found_exceptions_in_service_Integration_bus

Troubleshooting CWSIT0103E – No ME found(even though ME is running)

https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/common_reasons_for_CWSIT0103E_no_messaging_engine_was_found_exceptions_in_service_Integration_bus








Connect with us!

1. Get notified on upcoming webcastsSend an e-mail to [email protected] with subject line “wste

subscribe” to get a list of mailing lists and to subscribe

2. Tell us what you want to learnSend us suggestions for future topics or improvements about our

webcasts to [email protected]

mailto:[email protected]?subject=wste%20subscribe





Questions and Answers



Additional WebSphere Product Resources Learn about upcoming WebSphere Support Technical Exchange webcasts,

and access previously recorded presentations at:http://www.ibm.com/software/websphere/support/supp_tech.html

Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at:http://www.ibm.com/developerworks/websphere/community/

Join the Global WebSphere Community: http://www.websphereusergroup.org

Access key product show-me demos and tutorials by visiting IBM Education Assistant:http://www.ibm.com/software/info/education/assistant

View a webcast replay with step-by-step instructions for using the Service Request (SR) tool for submitting problems electronically:http://www.ibm.com/software/websphere/support/d2w.html

Sign up to receive weekly technical My Notifications emails:http://www.ibm.com/software/support/einfo.html

http://www.ibm.com/software/websphere/support/supp_tech.html

http://www.ibm.com/developerworks/websphere/community/

http://www.websphereusergroup.org/

http://www.ibm.com/software/info/education/assistant

http://www.ibm.com/software/websphere/support/d2w.html

http://www.ibm.com/software/support/einfo.html

http://www.ibm.com/software/support/einfo.html

Documents

Service Integration Bus Best Practices in WebSphere ... - IBM