View
198
Download
0
Category
Tags:
Preview:
Citation preview
An InformationWeek Webcast
Sponsored by
Part 3 - Performance:
How to Fine-tune Your ODM Solution
Webcast Logistics
Today’s Presenters
David Granshaw
WODM Performance Architect (Events)
Pierre-André Paumelle
WODM Performance Architect (Rules)
Demo Series: Best Practices for Implementing
Operational Decision Management Solutions
Part 3: Performance: How to Fine-tune Your ODM
Solution
April 3rd, 2012
WebSphere Decision Center
WebSphere Decision Server
WebSphere Operational Decision Management
Event Execution Runtime
Event Designer
Decision Center Console
Decision Center Repository
Decision Center for Business Space
Rule Execution Server
Rule Designer
Relations between components
Rule Solutions for Office
5
Ruledocs
Deploy Deploy
Deploy Deploy
Synchronize Synchronize
Session 2 focus
Session 1 focus
Session 3 focus
►WebSphere Operational Decision Management
► 1. Tuning Decision Server Events
► 2. Tuning Decision Server Rules
► 3. Tuning Decision Center
►4. Performance Improvements in V7.5
Agenda
Note the performance figures given in this presentation are just examples based on a
typical workload. The performance impact of specific design and runtime changes will vary
depending upon the details of the workload.
© IBM 2012
Decision Server Events – Runtime Overview
Actions
Events
Repository
Database
DB Server JMS (durable)
Event Destination
JMS (durable)
Action Topic
Events
Events
Runtime
WODM Server
Technology
Connectors
Technology
Connectors
All events and actions ultimately end up on the JMS event and action
destinations (except events delivered via Extreme Scale).
Technology Connectors can be used to send actions and receive events from a
range of sources including JMS, File, HTTP, JDBC etc
The Events Repository can be used to store information about the occurrence of
events, history and any data that may be shared across events (CSBOs).
© IBM 2012
Event Designer : Durability of Events and Actions
Non Durable processing can be 2.5 times faster than durable
processing so if actions and events do not need to survive a
server restart use non durable processing
For further information on durability see the WODM Infocenter : “Configuring
durable and nondurable events and actions”
© IBM 2012
Event Designer : Simple and Complex Event Rules
• Simple event rules – By definition, Simple event rules can be evaluated without reference to
previous events or actions and are therefore significantly faster • e.g. “if the deposit amount is > $500 then …”
• Complex Event Rules – By definition, Complex event rules have a dependency on previous events
or actions with the same context ID (e.g. customer reference number) • e.g. “If this customer has withdrawn money 3 times in the last week then …”
• For optimal performance do not add a Context Id to a Simple event rule unless required. Simple rules should only have a context ID defined if:
• Information about the occurrence of the events and action processed by this
event rule are actually required by other, Complex event rules.
• Context Scoped Business Objects are being used
© IBM 2012
Event Designer : Simple Rules versus Complex Rules
1.87
1.41
1
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Rela
tive T
hro
ug
hp
ut
(evts
/s)
Non Durable
Simple Rule
Simple Rule with Ctx ID
Complex Rule
Simple event rules are faster than Complex rules by:
Non Durable Processing: 87% Durable Processing 22%
1.22
1.111
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Rela
tive T
hro
ug
hp
ut
(evts
/s)
Durable
Simple Rule
Simple Rule with Ctx ID
Complex Rule
Don’t define a context ID on a Simple rule unless required
Consider using Simple rules for filtering
© IBM 2012
Event Designer : Ordering Multiple Expressions/Filters in an Event Rule
• Rules will evaluate expressions intelligently from the top down, so
place Simple expressions first
– For example, the second part of an ‘and’ expression will only be
evaluated if the first part is true.
Complex expression first :
More costly Complex expression is always evaluated
Simple expression first :
More costly Complex expression is only evaluated
when the Simple expression is true
© IBM 2012
Event Designer : Ordering multiple filters in a rule
1.51
1
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
Rela
tive T
hro
ug
hp
ut
(evts
/s)
Non Durable
Simple Filter First
Complex Filter First
1.11
1
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
Rela
tive T
hro
ug
hp
ut
(evts
/s)
Durable
Simple Filter First
Complex Filter First
Placing Simple filters first may be faster by:
Non Durable Processing: 51% Durable Processing 11%
Note: The performance difference will depend how often the first filter evaluates to
true. In the graphs above the first filter evaluated to true 1 in every 100 events.
© IBM 2012
Event Designer : Ordering multiple filters in a rule
• For optimal performance:
Always place Simple filters ahead of Complex filters.
– The same principle applies to any filters that require more processing, for
example, simplistic filters should come before filters that involve database
fetches or Context Scoped Business Objects
Also, if an event triggers multiple Complex rules to be evaluated, group
the rules together into one event rule group.
© IBM 2012
Event Designer : Technology Connectors
These examples show an example of the performance impact of using the new JMS
and HTTP connectors shipped in V7.5.
1
0.7 0.7
0
0.2
0.4
0.6
0.8
1
1.2
Rel
ativ
e T
hro
ug
hp
ut
(evt
s/s)
Event Connector
No Connector
HTTP Connector
JMS Connector 1
0.8
0.9
0
0.2
0.4
0.6
0.8
1
1.2
Rel
ativ
e T
hro
ug
hp
ut
(evt
s/s)
Action Connector
No Connector
HTTP Connector
JMS Connector
Event Connector Action Connector
For optimal performance, send events in the correct format directly to Decision
Server JMS event or durable event destinations. Also receive actions directly from
the action or durable action topic
© IBM 2012
Event Runtime Performance : Top Ten Recommendations
1. Monitor system resources (cpu, disk, network, memory)
• This will ensure you target your performance tuning effectively
2. Check you are not writing large amounts of data to log, error log, ffdc or
trace files
• It will significantly impact performance and may indicate an underlying
problem in the project or configuration, which will degrade throughput.
3. Ensure you are not recording history unless you intend to.
• History is needed for user defined charts or reports of events, actions, filters
and data. It is not needed for Complex rule evaluation or Context Scoped
Business Objects.
• History records large amounts of data. It can impact performance by 50 to
80%.
© IBM 2012
4. Tune the JVM
• A heap size of 1280MB is a good starting point for 32 bit systems; 4096MB
for 64bit.
• Consider using generational garbage collection
5. Ensure you have a sufficient number of rule processing threads
• You should consider increasing the number of rule processing threads if
events are building up on the event destination
• If you increase the number of rule processing threads you also need to
consider increasing the connection pools for database connections, JMS
Connections and data connection objects
6. Periodically (either manually or automatically) prune unnecessary
context data from the Decision Server Event Repository to improve the
database performance
Event Runtime Performance : Top Ten Recommendations
© IBM 2012
7. Ensure the repository database is tuned (or auto tuned) for the workload
– use separate, fast disk subsystems (SAN/RAID) for all logs and tables.
8. When using persistent Context Scoped Business Objects, the following
database tuning may improve performance up to a factor of 3:
– For db2 consider using the statement concentrator:
• UPDATE DB CFG USING STMT_CONC LITERALS IMMEDIATE;
– For Oracle consider using cursor sharing:
• ALTER SYSTEM SET CURSOR_SHARING=FORCE SCOPE=BOTH;
9. Tune the JMS provider and monitor the queue/topic message depths
10. Use the Performance Monitoring Infrastructure to monitor Decision
Server Events
• This will show statistics about events, actions, rules and other aspects of the
runtime. See WODM Infocenter for more details
Event Runtime Performance : Top Ten Recommendations
© IBM 2012
Note the next 4 slides were not presented at the webinar but provide
additional detail on some of the top ten tuning recommendations.
© IBM 2012
Runtime : Minimising Disk Access - logging
WODM has a number of log and error files. If these are growing significantly
over time it can have a large impact on performance and may indicate an
underlying problem.
Check that the following files and ensure they are not growing
significantly over time. If they are, read the logs to find the cause and
rectify the problem:
• <was_install_dir>\profiles\ <profileName> \logs\SystemOut.log
• <was_install_dir>\profiles\ <profileName> \logs\SystemErr.log
• <was_install_dir>\profiles\ <profileName> \ffdc\ (all files)
Also confirm that large trace files are not being generated in
• <was_install_dir>\profiles\<profileName>\logs\trace.out
• <DecisionServer_install_dir>\director\logs\connectors.log (standalone
connectors only)
For further information on trace see the WODM Infocenter : “enabling trace” and “Configuring
the technology connectors log file”.
© IBM 2012
Runtime : JVM Tuning
• The tuning of the JVM and the garbage collection policy can have a significant impact on performance.
• Settings will depend upon the WODM project and the OS but here are some recommended starting configurations
For further information on JVM tuning see the WAS Infocenter : “JVM Tuning”
32 bit
WODM installation
64 bit
WODM installation
Minimum heap size 1280 (MB) 4096 (MB)
Maximum heap
size
1280 (MB) 4096 (MB)
Generic JVM
arguments
-Xgcpolicy:gencon
–Xmn1024M
-Xgcpolicy:gencon
–Xmn2048M
© IBM 2012
Runtime : Decision Server Events Tuning
• If there is spare cpu on the WODM server and events are building up
on the event destination this may indicate an insufficient number of
rule processing threads. The number of threads can be increased in
the WAS administrative console:
– Resources > Resource Environment > Resource environment entries >
WbeSrv01 > Custom properties
– In a cluster it is recommended to use a prime number for the number of
threads. This will provide the greatest concurrency.
© IBM 2012
Runtime : Decision Server Events Tuning
• If you do increase the number of rule processor threads:
– As each rule processor may use a database connection, ensure there are
a sufficient number of JDBC threads
• Resources > JDBC > Data sources > Event Runtime Datasource > Connection
pools
– Set “Maximum connections” to “rule processor threads + history MDB concurrency +
10”.
– As each rule processor thread may use a JMS connection, ensure there
are a sufficient number of JMS connections
• Resources > JMS > Topic connection factories > WbeTopicConnectionFactory
> Connection pools
– Set “Maximum connections” to “rule processor threads + 10”
– When using database enrichment ensure there is a connection available
for each rule processor thread. This is set in event designer data
connections page. – Set the connection pool limit to at least the number of rule processor threads
© IBM 2012
HA, Scalability and Performance
• When using WebSphere Default Messaging for JMS
– There are two key clustering architectures mentioned in the WODM
Infocenter, the “Silver Topology” and the “Golden Topology”.
– The Golden topology has separate messaging and WODM clusters
– The Silver topology contains a messaging engine and a Decision Server
Runtime in each cluster member
– There is also a choice of messaging engine configurations
– The scalability options will offer the opportunity to have multiple
messaging engines and will provide the greatest throughput.
For optimal performance use the Silver Topology with a “Scalability” or
“High availability with Scalability” messaging engine configuration.
© IBM 2012
HA, Scalability and Performance – Golden versus Silver
Topology
1.8
1
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Rela
tive T
hro
ug
hp
ut
(evts
/s)
Non Durable
Silver Topology
Golden Topology
1.4
1
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Rela
tive T
hro
ug
hp
ut
(evts
/s)
Durable
Silver Topology
Golden Topology
The Silver topology can out perform the Golden topology by 40 – 80% due
to the co-location of the messaging engine and events runtimes
Non Durable Processing Durable Processing
© IBM 2012
►WebSphere Operational Decision Management
► 1. Tuning Decision Server Events
► 2. Tuning Decision Server Rules
► 3. Tuning Decision Center
►4. Performance Improvements in V7.5
Agenda
© IBM 2012 26
Decision Service architecture
Rule Engine Execution
IN IN
OUT
Rule
flow
BOM to XOM
XOM
Rules
• BOM: Business Object Model
• XOM: eXecutable Object Model
© IBM 2012 27
Decision Service performance best practices
• The performance cost for a Decision Service may look
something like Execution Time
XOM
Ruleflow
Functions
B2X
Misc. Rule engine
JVM (GC)
© IBM 2012
Decision Service performance best practices
• XOM type choices:
– JAVA XOM better performance
– XML XOM
• Dynamicity
• Useful in case of XML model
• Ruleflow: – Limit the size and the complexity of the ruleflow, it is
interpreted.
– Always use the same engine algorithm to save memory
© IBM 2012
Decision Service performance best practices
• Choose the correct engine algorithm depending on your Decision Service. – RetePlus (The default mode)
• Stateful application
• Rule chaining application
• May be useful in the case of many objects
– Sequential
• Application with many rules and few objects
• Most of the customer cases.
• Really efficient in multi-thread environment.
– Fastpath
• Application with rules implementing a decision structure and many objects
• May have longer compilation but faster at run time.
• Really efficient in multi-thread environment.
© IBM 2012 30
Decision Service performance best practices
– Smallest memory consumption
• Engine alone
– No concurrent execution
• Engine alone
• JSE Rule Execution Server
– Concurrent executions
• Rule Execution Server JSE
• Rule Execution Server JEE
– Integration with JEE animals
(EJB, MDB, POJO)
• Rule Execution Server
JEE
– Call as a Web Service
• HTDS
• MTDS
– Call from COBOL application
• zRES
Choose your architecture depending on your Decision Service characteristics:
© IBM 2012
Decision Server Rules tuning
• The log level in the Rule Execution Server should be set to level
Severe or Warning in the production environment to increase
performance.
– This property (TraceLevel) is accessible in the resource
adaptor of the Rule Execution Server or in the ra.xml.
• Tune the GC and memory size.
– Starting configuration 64bits
– -Xgcpolicy:gencon –Xmn2048M –Xmx4096M –Xms4096M
• Tune the RES pool size. A sizing methodology is available at :
http://www-01.ibm.com/support/docview.wss?uid=swg21400803
© IBM 2012
Decision Server Rules tuning
• Tune the usage of the execution trace and the
Decision Warehouse by filtering.
• A ruleset on an XML XOM should be configured to
run in multiple simultaneous.
– This is configured using the ruleset property ruleset.xmlDocumentDriverPool.maxSize.
• Use the ruleset caching information (XU dump) from
the Rule Execution Server console to get the Rule
Execution Server status.
© IBM 2012
Decision Server Rules Performance T
PS
Impact of the execution trace
© IBM 2012
Decision Server Rules Performance T
PS
Impact of the XOM type
© IBM 2012
Decision Server Rules Performance T
PS
Remote Web Service call vs. Local call
© IBM 2012
Decision Server Rules Performance T
PS
Fastpath Algorithm vs. Sequential Algorithm
© IBM 2012
►WebSphere Operational Decision Management
► 1. Tuning Decision Server Events
► 2. Tuning Decision Server Rules
► 3. Tuning Decision Center
►4. Performance Improvements in V7.5
Agenda
© IBM 2012
Decision Center tuning
• Limit memory consumption
– Parsing a ruleset consumes memory so you can disable the
“Archive parsing flag” option (Project > Edit project options >
Check the ruleset archive).
– Use Automatic build to avoid a huge ruleset generation cost
at “first ruleset generation”.
– Set in the installation manager the configuration parameter
teamserver.build.archive.storage to file instead of memory.
• Build performance
– Disable “Archive parsing flag”.
– Disable “Rule analysis checks”.
© IBM 2012
Decision Center tuning
• Memory consumption estimation on a repository of
10000 business rules
– Memory foot print of a session is ~10 MB
– Ruleset generation : ~150 MB of short-lived objects created
• If you believe the number of users connected
simultaneously to a single server will not fit into the
memory allocated for a single VM, you should deploy
Decision Center to a cluster of servers and use load-
balancing, so that HTTP sessions are dispatched
according to the server’s available memory.
© IBM 2012
►WebSphere Operational Decision Management
► 1. Tuning Decision Server Events
► 2. Tuning Decision Server Rules
► 3. Tuning Decision Center
►4. Performance Improvements in V7.5
Agenda
© IBM 2012 41
Decision Server Rule Benchmark Scenarios (relative to 7.1.1.1 GA)
• Rule Execution Server execution improvement (TPS)
Between 1.01x to 3.97x faster than 7.1.1.1
Average improvement factor = 25%
• The new Web Service stack (HTDS) for rulesets based
on java XOM offers performance improvements
Improvement factor 2x to 4x versus 7.1.1.1
© IBM 2012 42
Decision Center (relative to 7.1.1.1 GA)
• Decision Center delivers a new ruleset build chain.
Performance gains from 50% to 150% versus
Rule Team Server V7.1.1.1.
This optimization offers a better scalability of
Decision Center as performance gains increase with
rule project size.
© IBM 2012
Customer Based Scenario 1 : Credit Limit Check
Warn
Customer
Action
Withdrawal
Events Enrich Event:
Get current
balance and
credit limit from
DB
Complex rule:
Has balance
been exceeded
before?
Simple rule:
Check if limit
exceeded
Performance in V7.5 improved by (relative to V7.0.1.1): 18% for non durable processing
10% for durable processing
• This scenario processes withdrawals from a bank and looks for
customers who repeatedly exceed their credit limits. The majority of
processing is with Simple event rules.
Synthetic event
Generated 1 in
every 100 events
© IBM 2012
Customer Based Scenario 2 : Missed Payments
Credit card,
overdraft, loan
missed
payment
events
Performance in V7.5 improved by: 31% for non durable processing
14% for durable processing
• This scenario looks for customers who have repeatedly missed payments
on multiple accounts : credit card, overdraft and loan. It also uses CSBOs to
keep track of the total of all missed payments. All event rules are Complex.
Warn
Customer
Action Enrich Event:
Get current
customer details
from DB
Complex rule:
Has this occurred
more than once and
is the total debt >
$5000
Complex rule:
Have payments
been missed on all
accounts
Synthetic event
Generated 1 in
every 100 events
© IBM 2012
Customer Based Scenario 3 : Rules and Events
Offer Customer
Discount Action
Quote
Requested
Event Complex rule:
Has this
customer applied
before
Simple rule:
determine
discount from
Decision Server
Rules
• Performance in V7.5 improved by: • 27% for non durable processing
• 16% for durable processing
• Scenario 3 is a customer based scenario, which looks for customers who
are eligible for a discount as part of a new marketing campaign. The
scenario involves DSE and DSR.
Decision Server Rules (DSR)
Decision Server Events (DSE)
SOAP Technology Connector
Action
Event
© IBM 2012
Where to go for further information
• WebSphere Operational Decision Management
– www.ibm.com/operational-decision-management
• Redbook on BRMS Tuning
– Proven Practices for Enhancing Performance: A Q & A for IBM WebSphere ILOG
BRMS 7.1
• Whitepaper on WebSphere Business Events Tuning
– Getting the best performance from your complex event processing system with
WebSphere Business Events
• DeveloperWorks tool (System Integration Bus Performance) that
automatically engages and displays PMI statistics relevant to Decision Server
Event runtime performance
– WODM Performance Monitoring Tool
Q&A Session
David Granshaw
WODM Performance Architect (Events)
Pierre-André Paumelle
WODM Performance Architect (Rules)
Resources
To view this or other events on-demand please visit:
http://www.informationweek.com/events/past
For more information please visit:
WebSphere Operational Decision Management
www.ibm.com/operational-decision-management
Recommended