Performance Engineering Sterling MCS-OM - An Accenture Capability (3)

Copyright © 2010 Accenture All Rights Reserved. Accenture, its logo, and High Performance Delivered are trademarks of Accenture.

Performance Engineering Sterling MCS-OM

- An Accenture Capability

Copyright © 2010 Accenture All Rights Reserved.

Agenda

• Order Lifecycle Overview

• Order Management System Architecture Overview

• Performance Test Strategy

– Component Tests

– Concurrent Runs – Endurance Tests, Load/Stress Tests

– Test Results

• Performance Engineering

– Detection of Performance Bottlenecks

– Correction of Performance Bottlenecks

• NFR, Agent Server Consolidation, On-Demand Provisioning

• Questions

Copyright © 2010 Accenture All Rights Reserved. 3

Order Lifecycle

Order Lifecycle

Capture Orders

ATP Check & Confirm Order Create

Fraud Checks

Payment Checks

Schedule Orders

Release Orders

Shipment Confirmations

Invoice

Delivery Notifications





– Component Tests


– Test Results





• Questions

Agenda


Sterling MCS-OM Architecture

Overview

Integration TierESB

Application TierSterling Agents

WAS

Sterling 9.0

Business Workflows

Infr

astr

uctu

re T

ier

Database TierOracle 11G


Sterling MCS-OM Application

Architecture

RedHat Linux

WebSphere

ESB

RedHat Linux

Oracle 11G

USERS

TMS

WMS

FlagshipStores

Sterling 9.0

Business Workflows

Database Tier

ApplicationTier

User Interface

Integration Tier (NAIP)

TIBCO

CSR

Sterling Agents

Agents Tier





– Component Tests


– Test Results





• Questions

Agenda


Component

Test Strategy

8

Order Fulfillment

Component Test

Component

Test Load

Actual

Throughput

Extrapolated

ThroughputTest Status

Order Capture (.com) 5000

Order Capture (CSR) 5000

Order Create in OMS 5000

Schedule Orders 5000

NFRExpected Peak Throughput – 6000 Orders/ Hr for 2012

Expected Average Throughput – 500 Orders/ Hr for 2012


Component Test

Order Capture

Order CreateSchedule Orders

Capture 5000.COM Orders

Create 5000Orders in OMS

Schedule 5000Orders in OMS

Measure Throughput

Check for Memory Leaks

Watch for Exceptions

Track JVM Resource Utilization

Analyze DB Perf


Component

Test Results

10

Order

Fulfillment

Component Test

Component

Test Load

Actual

Throughput

Extrapolated

ThroughputTest Status

Order Capture (.com) 5000 5 mins 60000/hr Passed

Order Capture (CSR) 5000 5 mins 60000/hr Passed

Order Create in OMS 5000 12 mins 25000/hr Passed

Schedule Orders 5000 65 mins 4615/hr FAILED

NFRExpected Peak Throughput – 6000 Orders/ Hr for 2012

Expected Average Throughput – 500 Orders/ Hr for 2012


Endurance & Load

Test Strategy/Results

11

Test Duration Server Name

Expected

Throughput

(Peak)

Throughput Actual

ThroughputInstance 1 Instance 2

5Hrs 30mins Order Capture (.com) 6000/Hr 22,747 22,644 8,252/Hr

5Hrs 30mins Order Capture (CSR) 6000/Hr 17839 17,788 6,427/Hr

5Hrs 30mins Order Create in OMS 6000/Hr 25,000 25,000 9,090/Hr

5Hrs 30mins Schedule Orders 6000/Hr 30,454 25,009 10,084/Hr

5Hrs 30mins Release Orders 6000/Hr 30,500 30,000 11,000/Hr

2 Hrs InventoryLoad500,000 in 2 Hrs

200,000 279900 479,900

2 Hrs InventorySync500,000 in 2 Hrs

200,000 279,900 479,900





– Component Tests


– Test Results





• Questions

Agenda


Detection of

Performance Bottlenecks

13

Tier Tools Employed

Application &

Agents Tier

GC Logs, Heap Dumps, Thread Dumps, Core

Dumps, Patrol Perceive

DB Tier OEM, AWR Reports

Integration TierMonitoring Inbound/Outbound rates, Patrol

Perceive

UI TierLoad Runner ( Response Time, Concurrent

Users)


Detection of

Performance Bottlenecks

• Monitoring resource (CPU, RAM, NETWORK, DISK, I/O) usage

• Getting Garbage Collection (GC) logs, Heap Dumps, Thread Dumps and Java Core Dumps for the App Tier

• Analyzing the following parameters

– Frequency of garbage collection

– Frequency of allocation failures

– Continuous increase in Heap Occupancy

– Time spent in Mark, Sweep and Compaction Phases during GC

• Oracle Enterprise Manager

– Monitor the blocking locks

– Disk Read time

– AWR reports for costly queries


Time

CP

U U

tili

zati

on

Syste

m L

oa

d

100%

CPU Utilization vs. System Load

-Normal Day


CPU Utilization vs. System Load

- Problematic Day

Time

100%

CP

U U

tili

zati

on

Syste

m L

oa

d


<af type="tenured" id="26" timestamp="May 24 03:44:36 2011" intervalms="206808.702"><minimum requested_bytes="8208" /><time exclusiveaccessms="0.023" /><tenured freebytes="0" totalbytes="536870912" percent="0" >

<soa freebytes="0" totalbytes="536870912" percent="0" /><loa freebytes="0" totalbytes="0" percent="0" />

</tenured>

<gc type="global" id="40" totalid="40" intervalms="206808.932"><refs_cleared soft="0" threshold="32" weak="293" phantom="0" /><finalization objectsqueued="359" /><timesms mark="33.011" sweep="2.951" compact=“567.000" total="36.047" /><tenured freebytes="454482000" totalbytes="536870912" percent="84" ><soa freebytes="454482000" totalbytes="536870912" percent="84" /><loa freebytes="0" totalbytes="0" percent="0" />

</tenured></gc>

<tenured freebytes="454473792" totalbytes="536870912" percent="84" ><soa freebytes="454473792" totalbytes="536870912" percent="84" /><loa freebytes="0" totalbytes="0" percent="0" />

</tenured>

<time totalms=“603.263" /></af>

Garbage Collection Logs


Leaky JVM

Heap Consumption

Heap

Co

ns

um

pti

on

Time

100%


Thread Dumps

– Blocking Threads


Correction Process

• Fixing the piece of code causing Memory Leaks – Dev Team

• Reducing the size of objects/allocation requests – Dev Team

• Tuning the heap size and frequency of GC – Perf Team

• Incorporating incremental GC – Perf Team

• Repeating Component and Endurance runs to show the difference in performance – Perf Team


Leak Free JVM

Heap Consumption

Time

Heap

Co

ns

um

pti

on

100%





– Component Tests


– Test Results





• Questions

Agenda


Agent Server Consolidation

23

Process Name (JVM)

# of

instances for

2012

Xmx

(MB)

JVM Memory

Footprint (MB)

Order Create in OMS 2 512 800

Schedule Orders 2 512 800

Release Orders 2 512 800

InventoryLoad 10 512 8000

InventorySync 10 512 8000

Total RAM Required on Agents Box 18.4 GB


On-Demand Provisioning

24

Time

CP

U U

tili

izati

on

Nov

4 C

ore

s,

48 G

B R

AM

10 C

ore

s,

80 G

B R

AM

Feb


THANK YOU!!

25

Documents

Performance Engineering Sterling MCS-OM - An Accenture Capability (3)