An InformationWeek Webcast

Sponsored by

Part 3 - Performance:

How to Fine-tune Your ODM Solution

Webcast Logistics

Today’s Presenters

David Granshaw

WODM Performance Architect (Events)

Pierre-André Paumelle

WODM Performance Architect (Rules)

Demo Series: Best Practices for Implementing

Operational Decision Management Solutions

Part 3: Performance: How to Fine-tune Your ODM

Solution

April 3rd, 2012

WebSphere Decision Center

WebSphere Decision Server

WebSphere Operational Decision Management

Event Execution Runtime

Event Designer

Decision Center Console

Decision Center Repository

Decision Center for Business Space

Rule Execution Server

Rule Designer

Relations between components

Rule Solutions for Office

5

Ruledocs

Deploy Deploy

Deploy Deploy

Synchronize Synchronize

Session 2 focus

Session 1 focus

Session 3 focus

►WebSphere Operational Decision Management

► 1. Tuning Decision Server Events

► 2. Tuning Decision Server Rules

► 3. Tuning Decision Center

►4. Performance Improvements in V7.5

Agenda

Note the performance figures given in this presentation are just examples based on a

typical workload. The performance impact of specific design and runtime changes will vary

depending upon the details of the workload.

© IBM 2012

Decision Server Events – Runtime Overview

Actions

Events

Repository

Database

DB Server JMS (durable)

Event Destination

JMS (durable)

Action Topic

Events

Events

Runtime

WODM Server

Technology

Connectors

Technology

Connectors

All events and actions ultimately end up on the JMS event and action

destinations (except events delivered via Extreme Scale).

Technology Connectors can be used to send actions and receive events from a

range of sources including JMS, File, HTTP, JDBC etc

The Events Repository can be used to store information about the occurrence of

events, history and any data that may be shared across events (CSBOs).

© IBM 2012

Event Designer : Durability of Events and Actions

Non Durable processing can be 2.5 times faster than durable

processing so if actions and events do not need to survive a

server restart use non durable processing

For further information on durability see the WODM Infocenter : “Configuring

durable and nondurable events and actions”

© IBM 2012

Event Designer : Simple and Complex Event Rules

• Simple event rules – By definition, Simple event rules can be evaluated without reference to

previous events or actions and are therefore significantly faster • e.g. “if the deposit amount is > $500 then …”

• Complex Event Rules – By definition, Complex event rules have a dependency on previous events

or actions with the same context ID (e.g. customer reference number) • e.g. “If this customer has withdrawn money 3 times in the last week then …”

• For optimal performance do not add a Context Id to a Simple event rule unless required. Simple rules should only have a context ID defined if:

• Information about the occurrence of the events and action processed by this

event rule are actually required by other, Complex event rules.

• Context Scoped Business Objects are being used

© IBM 2012

Event Designer : Simple Rules versus Complex Rules

1.87

1.41

1

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Rela

tive T

hro

ug

hp

ut

(evts

/s)

Non Durable

Simple Rule

Simple Rule with Ctx ID

Complex Rule

Simple event rules are faster than Complex rules by:

Non Durable Processing: 87% Durable Processing 22%

1.22

1.111

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Rela

tive T

hro

ug

hp

ut

(evts

/s)

Durable

Simple Rule

Simple Rule with Ctx ID

Complex Rule

Don’t define a context ID on a Simple rule unless required

Consider using Simple rules for filtering

© IBM 2012

Event Designer : Ordering Multiple Expressions/Filters in an Event Rule

• Rules will evaluate expressions intelligently from the top down, so

place Simple expressions first

– For example, the second part of an ‘and’ expression will only be

evaluated if the first part is true.

Complex expression first :

More costly Complex expression is always evaluated

Simple expression first :

More costly Complex expression is only evaluated

when the Simple expression is true

© IBM 2012

Event Designer : Ordering multiple filters in a rule

1.51

1

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Rela

tive T

hro

ug

hp

ut

(evts

/s)

Non Durable

Simple Filter First

Complex Filter First

1.11

1

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Rela

tive T

hro

ug

hp

ut

(evts

/s)

Durable

Simple Filter First

Complex Filter First

Placing Simple filters first may be faster by:

Non Durable Processing: 51% Durable Processing 11%

Note: The performance difference will depend how often the first filter evaluates to

true. In the graphs above the first filter evaluated to true 1 in every 100 events.

© IBM 2012

Event Designer : Ordering multiple filters in a rule

• For optimal performance:

Always place Simple filters ahead of Complex filters.

– The same principle applies to any filters that require more processing, for

example, simplistic filters should come before filters that involve database

fetches or Context Scoped Business Objects

Also, if an event triggers multiple Complex rules to be evaluated, group

the rules together into one event rule group.

© IBM 2012

Event Designer : Technology Connectors

These examples show an example of the performance impact of using the new JMS

and HTTP connectors shipped in V7.5.

1

0.7 0.7

0

0.2

0.4

0.6

0.8

1

1.2

Rel

ativ

e T

hro

ug

hp

ut

(evt

s/s)

Event Connector

No Connector

HTTP Connector

JMS Connector 1

0.8

0.9

0

0.2

0.4

0.6

0.8

1

1.2

Rel

ativ

e T

hro

ug

hp

ut

(evt

s/s)

Action Connector

No Connector

HTTP Connector

JMS Connector

Event Connector Action Connector

For optimal performance, send events in the correct format directly to Decision

Server JMS event or durable event destinations. Also receive actions directly from

the action or durable action topic

© IBM 2012

Event Runtime Performance : Top Ten Recommendations

1. Monitor system resources (cpu, disk, network, memory)

• This will ensure you target your performance tuning effectively

2. Check you are not writing large amounts of data to log, error log, ffdc or

trace files

• It will significantly impact performance and may indicate an underlying

problem in the project or configuration, which will degrade throughput.

3. Ensure you are not recording history unless you intend to.

• History is needed for user defined charts or reports of events, actions, filters

and data. It is not needed for Complex rule evaluation or Context Scoped

Business Objects.

• History records large amounts of data. It can impact performance by 50 to

80%.

© IBM 2012

4. Tune the JVM

• A heap size of 1280MB is a good starting point for 32 bit systems; 4096MB

for 64bit.

• Consider using generational garbage collection

5. Ensure you have a sufficient number of rule processing threads

• You should consider increasing the number of rule processing threads if

events are building up on the event destination

• If you increase the number of rule processing threads you also need to

consider increasing the connection pools for database connections, JMS

Connections and data connection objects

6. Periodically (either manually or automatically) prune unnecessary

context data from the Decision Server Event Repository to improve the

database performance

Event Runtime Performance : Top Ten Recommendations

© IBM 2012

7. Ensure the repository database is tuned (or auto tuned) for the workload

– use separate, fast disk subsystems (SAN/RAID) for all logs and tables.

8. When using persistent Context Scoped Business Objects, the following

database tuning may improve performance up to a factor of 3:

– For db2 consider using the statement concentrator:

• UPDATE DB CFG USING STMT_CONC LITERALS IMMEDIATE;

– For Oracle consider using cursor sharing:

• ALTER SYSTEM SET CURSOR_SHARING=FORCE SCOPE=BOTH;

9. Tune the JMS provider and monitor the queue/topic message depths

10. Use the Performance Monitoring Infrastructure to monitor Decision

Server Events

• This will show statistics about events, actions, rules and other aspects of the

runtime. See WODM Infocenter for more details

Event Runtime Performance : Top Ten Recommendations

© IBM 2012

Note the next 4 slides were not presented at the webinar but provide

additional detail on some of the top ten tuning recommendations.

© IBM 2012

Runtime : Minimising Disk Access - logging

WODM has a number of log and error files. If these are growing significantly

over time it can have a large impact on performance and may indicate an

underlying problem.

Check that the following files and ensure they are not growing

significantly over time. If they are, read the logs to find the cause and

rectify the problem:

• <was_install_dir>\profiles\ <profileName> \logs\SystemOut.log

• <was_install_dir>\profiles\ <profileName> \logs\SystemErr.log

• <was_install_dir>\profiles\ <profileName> \ffdc\ (all files)

Also confirm that large trace files are not being generated in

• <was_install_dir>\profiles\<profileName>\logs\trace.out

• <DecisionServer_install_dir>\director\logs\connectors.log (standalone

connectors only)

For further information on trace see the WODM Infocenter : “enabling trace” and “Configuring

the technology connectors log file”.

© IBM 2012

Runtime : JVM Tuning

• The tuning of the JVM and the garbage collection policy can have a significant impact on performance.

• Settings will depend upon the WODM project and the OS but here are some recommended starting configurations

For further information on JVM tuning see the WAS Infocenter : “JVM Tuning”

32 bit

WODM installation

64 bit

WODM installation

Minimum heap size 1280 (MB) 4096 (MB)

Maximum heap

size

1280 (MB) 4096 (MB)

Generic JVM

arguments

-Xgcpolicy:gencon

–Xmn1024M

-Xgcpolicy:gencon

–Xmn2048M

© IBM 2012

Runtime : Decision Server Events Tuning

• If there is spare cpu on the WODM server and events are building up

on the event destination this may indicate an insufficient number of

rule processing threads. The number of threads can be increased in

the WAS administrative console:

– Resources > Resource Environment > Resource environment entries >

WbeSrv01 > Custom properties

– In a cluster it is recommended to use a prime number for the number of

threads. This will provide the greatest concurrency.

© IBM 2012

Runtime : Decision Server Events Tuning

• If you do increase the number of rule processor threads:

– As each rule processor may use a database connection, ensure there are

a sufficient number of JDBC threads

• Resources > JDBC > Data sources > Event Runtime Datasource > Connection

pools

– Set “Maximum connections” to “rule processor threads + history MDB concurrency +

10”.

– As each rule processor thread may use a JMS connection, ensure there

are a sufficient number of JMS connections

• Resources > JMS > Topic connection factories > WbeTopicConnectionFactory

> Connection pools

– Set “Maximum connections” to “rule processor threads + 10”

– When using database enrichment ensure there is a connection available

for each rule processor thread. This is set in event designer data

connections page. – Set the connection pool limit to at least the number of rule processor threads

© IBM 2012

HA, Scalability and Performance

• When using WebSphere Default Messaging for JMS

– There are two key clustering architectures mentioned in the WODM

Infocenter, the “Silver Topology” and the “Golden Topology”.

– The Golden topology has separate messaging and WODM clusters

– The Silver topology contains a messaging engine and a Decision Server

Runtime in each cluster member

– There is also a choice of messaging engine configurations

– The scalability options will offer the opportunity to have multiple

messaging engines and will provide the greatest throughput.

For optimal performance use the Silver Topology with a “Scalability” or

“High availability with Scalability” messaging engine configuration.

© IBM 2012

HA, Scalability and Performance – Golden versus Silver

Topology

1.8

1

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Rela

tive T

hro

ug

hp

ut

(evts

/s)

Non Durable

Silver Topology

Golden Topology

1.4

1

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Rela

tive T

hro

ug

hp

ut

(evts

/s)

Durable

Silver Topology

Golden Topology

The Silver topology can out perform the Golden topology by 40 – 80% due

to the co-location of the messaging engine and events runtimes

Non Durable Processing Durable Processing

© IBM 2012

►WebSphere Operational Decision Management

► 1. Tuning Decision Server Events

► 2. Tuning Decision Server Rules

► 3. Tuning Decision Center

►4. Performance Improvements in V7.5

Agenda

© IBM 2012 26

Decision Service architecture

Rule Engine Execution

IN IN

OUT

Rule

flow

BOM to XOM

XOM

Rules

• BOM: Business Object Model

• XOM: eXecutable Object Model

© IBM 2012 27

Decision Service performance best practices

• The performance cost for a Decision Service may look

something like Execution Time

XOM

Ruleflow

Functions

B2X

Misc. Rule engine

JVM (GC)

© IBM 2012

Decision Service performance best practices

• XOM type choices:

– JAVA XOM better performance

– XML XOM

• Dynamicity

• Useful in case of XML model

• Ruleflow: – Limit the size and the complexity of the ruleflow, it is

interpreted.

– Always use the same engine algorithm to save memory

© IBM 2012

Decision Service performance best practices

• Choose the correct engine algorithm depending on your Decision Service. – RetePlus (The default mode)

• Stateful application

• Rule chaining application

• May be useful in the case of many objects

– Sequential

• Application with many rules and few objects

• Most of the customer cases.

• Really efficient in multi-thread environment.

– Fastpath

• Application with rules implementing a decision structure and many objects

• May have longer compilation but faster at run time.

• Really efficient in multi-thread environment.

© IBM 2012 30

Decision Service performance best practices

– Smallest memory consumption

• Engine alone

– No concurrent execution

• Engine alone

• JSE Rule Execution Server

– Concurrent executions

• Rule Execution Server JSE

• Rule Execution Server JEE

– Integration with JEE animals

(EJB, MDB, POJO)

• Rule Execution Server

JEE

– Call as a Web Service

• HTDS

• MTDS

– Call from COBOL application

• zRES

Choose your architecture depending on your Decision Service characteristics:

© IBM 2012

Decision Server Rules tuning

• The log level in the Rule Execution Server should be set to level

Severe or Warning in the production environment to increase

performance.

– This property (TraceLevel) is accessible in the resource

adaptor of the Rule Execution Server or in the ra.xml.

• Tune the GC and memory size.

– Starting configuration 64bits

– -Xgcpolicy:gencon –Xmn2048M –Xmx4096M –Xms4096M

• Tune the RES pool size. A sizing methodology is available at :

http://www-01.ibm.com/support/docview.wss?uid=swg21400803

© IBM 2012

Decision Server Rules tuning

• Tune the usage of the execution trace and the

Decision Warehouse by filtering.

• A ruleset on an XML XOM should be configured to

run in multiple simultaneous.

– This is configured using the ruleset property ruleset.xmlDocumentDriverPool.maxSize.

• Use the ruleset caching information (XU dump) from

the Rule Execution Server console to get the Rule

Execution Server status.

© IBM 2012

Decision Server Rules Performance T

PS

Impact of the execution trace

© IBM 2012

Decision Server Rules Performance T

PS

Impact of the XOM type

© IBM 2012

Decision Server Rules Performance T

PS

Remote Web Service call vs. Local call

© IBM 2012

Decision Server Rules Performance T

PS

Fastpath Algorithm vs. Sequential Algorithm

© IBM 2012

►WebSphere Operational Decision Management

► 1. Tuning Decision Server Events

► 2. Tuning Decision Server Rules

► 3. Tuning Decision Center

►4. Performance Improvements in V7.5

Agenda

© IBM 2012

Decision Center tuning

• Limit memory consumption

– Parsing a ruleset consumes memory so you can disable the

“Archive parsing flag” option (Project > Edit project options >

Check the ruleset archive).

– Use Automatic build to avoid a huge ruleset generation cost

at “first ruleset generation”.

– Set in the installation manager the configuration parameter

teamserver.build.archive.storage to file instead of memory.

• Build performance

– Disable “Archive parsing flag”.

– Disable “Rule analysis checks”.

© IBM 2012

Decision Center tuning

• Memory consumption estimation on a repository of

10000 business rules

– Memory foot print of a session is ~10 MB

– Ruleset generation : ~150 MB of short-lived objects created

• If you believe the number of users connected

simultaneously to a single server will not fit into the

memory allocated for a single VM, you should deploy

Decision Center to a cluster of servers and use load-

balancing, so that HTTP sessions are dispatched

according to the server’s available memory.

© IBM 2012

►WebSphere Operational Decision Management

► 1. Tuning Decision Server Events

► 2. Tuning Decision Server Rules

► 3. Tuning Decision Center

►4. Performance Improvements in V7.5

Agenda

© IBM 2012 41

Decision Server Rule Benchmark Scenarios (relative to 7.1.1.1 GA)

• Rule Execution Server execution improvement (TPS)

Between 1.01x to 3.97x faster than 7.1.1.1

Average improvement factor = 25%

• The new Web Service stack (HTDS) for rulesets based

on java XOM offers performance improvements

Improvement factor 2x to 4x versus 7.1.1.1

© IBM 2012 42

Decision Center (relative to 7.1.1.1 GA)

• Decision Center delivers a new ruleset build chain.

Performance gains from 50% to 150% versus

Rule Team Server V7.1.1.1.

This optimization offers a better scalability of

Decision Center as performance gains increase with

rule project size.

© IBM 2012

Customer Based Scenario 1 : Credit Limit Check

Warn

Customer

Action

Withdrawal

Events Enrich Event:

Get current

balance and

credit limit from

DB

Complex rule:

Has balance

been exceeded

before?

Simple rule:

Check if limit

exceeded

Performance in V7.5 improved by (relative to V7.0.1.1): 18% for non durable processing

10% for durable processing

• This scenario processes withdrawals from a bank and looks for

customers who repeatedly exceed their credit limits. The majority of

processing is with Simple event rules.

Synthetic event

Generated 1 in

every 100 events

© IBM 2012

Customer Based Scenario 2 : Missed Payments

Credit card,

overdraft, loan

missed

payment

events

Performance in V7.5 improved by: 31% for non durable processing

14% for durable processing

• This scenario looks for customers who have repeatedly missed payments

on multiple accounts : credit card, overdraft and loan. It also uses CSBOs to

keep track of the total of all missed payments. All event rules are Complex.

Warn

Customer

Action Enrich Event:

Get current

customer details

from DB

Complex rule:

Has this occurred

more than once and

is the total debt >

$5000

Complex rule:

Have payments

been missed on all

accounts

Synthetic event

Generated 1 in

every 100 events

© IBM 2012

Customer Based Scenario 3 : Rules and Events

Offer Customer

Discount Action

Quote

Requested

Event Complex rule:

Has this

customer applied

before

Simple rule:

determine

discount from

Decision Server

Rules

• Performance in V7.5 improved by: • 27% for non durable processing

• 16% for durable processing

• Scenario 3 is a customer based scenario, which looks for customers who

are eligible for a discount as part of a new marketing campaign. The

scenario involves DSE and DSR.

Decision Server Rules (DSR)

Decision Server Events (DSE)

SOAP Technology Connector

Action

Event

© IBM 2012

Where to go for further information

• WebSphere Operational Decision Management

– www.ibm.com/operational-decision-management

• Redbook on BRMS Tuning

– Proven Practices for Enhancing Performance: A Q & A for IBM WebSphere ILOG

BRMS 7.1

• Whitepaper on WebSphere Business Events Tuning

– Getting the best performance from your complex event processing system with

WebSphere Business Events

• DeveloperWorks tool (System Integration Bus Performance) that

automatically engages and displays PMI statistics relevant to Decision Server

Event runtime performance

– WODM Performance Monitoring Tool

Q&A Session

David Granshaw

WODM Performance Architect (Events)

Pierre-André Paumelle

WODM Performance Architect (Rules)

Resources

To view this or other events on-demand please visit:

http://www.informationweek.com/events/past

For more information please visit:

WebSphere Operational Decision Management

www.ibm.com/operational-decision-management