Performance of SAP J2EE Engine

��

��

� SAP AG 2006, SAP TechEd ’06 / CD103 / 2

��

Xiaoqing ChengPerformance, Data Management & Scalability, SAP AG

Tami MorrisonPerformance, Data Management & Scalability, SAP Labs, LLC. (Houston)

Best Practices

Performance and Scalability of SAP J2EE Engine

KPIs for Performance Measurement and Analysis


��

!� ��"��#��$��#��%&��$�""�� "��'!� ��"��#��$��#��%&��$�""�� "��'!� ��"��#��$��#��%&��$�""�� "��'!� ��"��#��$��#��%&��$�""�� "��'

� Understand the SAP J2EE architecture for CPU and memory scalability

� Use introduced KPIs for performance measurements and analysis

� Analyze “Out of memory” problems of Java applications

� Analyze “Serialization” problems of Java applications


(�"� �"� �� !��

�Verification of scalability

�SAPS definitionfor Sizing

�Ability to predictsystem behavior

� Predict platform independent HW resource requirements� Ensure performance of customer production systems

Performance

Scal

abili

tyBenchm

arking

Sizing


�� " ��"�&

Size of data

Number of processed

objects

��!��

��

Concurrentusers

Number ofservers

Number ofclients

Scalability

Local/Wide Area Network

�

��"��

��)� ��#�

��

Number of objects in the DB

*"�� "�

��

��

+ ��$ ��

� ��&


!��#��!��,---��'�� "��. ��/�� &

D S

S

Central Instance

E

Loadbalancer

JVMs:D - dispatcher nodeS - server node

DB: databaseE : Enqueue serverM : Message server

…

J2EE Cluster

DB

D S

S

Dialog Instance

…D S

S

Dialog Instance

…

…M


!��#��!��,---��'�� "��. ��/�� &0��12

-3�� "�� " ��"�&

� Scaling over server machines

� Any HTTP load balancer can be used to distribute the load

4�� "�� " ��"�&

� Scaling in CPU and memory within one server machine

� A dispatcher node distributes the load to the server nodes

� CPU scalability– Multi-threading of each Java VM utilizes the CPU resource– For better CPU utilization, configure at least 2 server nodes on a physical

machine, due to the “stop the world pause” of the Java Garbage Collection

� Memory scalability– Memory utilization is controlled by the heap configuration of the Java VM– Use more server nodes to utilize more physical memory on a server machine– For detailed Java VM Heap configuration, please refer SAP Note 723909


5�4�� /� �� !� "&��

*�� 5�4��#��"�

� Reflect the real performance requirements

� Be accurate and reproducible

� Give indications to possible optimizations

6�� $

� Response time of a User Interaction Step (UIS)– The elapsed clock time from user input till the next screen is displayed

6�� &�� $

� Throughput– Number of transactions per time unit

� CPU consumption– Most important system resource shared by many concurrent users

� Memory consumption– Measurement and analysis are complicated by the automatic garbage collection of Java

� GC activities– Impact to the performance, can be tuned by optimal configurations


5�4��.��

7 # � �� 8

� CPU utilization of a physical server (in load tests)

� CPU time of a business scenario

� CPU time of an user interaction step ��

+�$�� #��.�� 8

� Use Single Activity Trace (SAT) of SAP J2EE Engine

� Use Performance Trace of SAP J2EE Engine

� Use JARM instrumentation API

� Use operating system utilities


5�4��/�� &��

/�� &��"�) �� 9/��""��&#�#� �

��

� Make sure that available physical memory really utilized

� Avoid paging for Java applications (the max. heap size + permanent space + stack must completely fit into the available physical memory)

6�� "��$��#�� &��

��# �� "�� %$�� &

5�4�

� The Framework Space– Memory footprint without user activities

� The User Session Space– Contribution of logged on users, the static part

� The Processing Space– Contribution of user activities, the dynamic part


/�� &5�4'6� � �$��

�#�6� � �$��

� Represents the memory footprint of a JVM after the J2EE engine has been started and has warmed up, without user activities

� Depends on the set of used Java applications on a JVM– Deployed, started, and used

� Indicates, for a given heap configuration, how much heap memory is available to run applications

/� ��

� Start the J2EE engine, execute representative business scenarios with several users, and then log them off

� Logon to the J2EE engine via telnet interface, jump to the server node, and trigger full GCs several times

� Look into the GC log file, read the “Bytes after GC” ‘M1’


/�� &5�4'.��

.��

� Memory occupied by a logged on, but inactive user

� Represents the user session context and cached content which is not shared by other users

� Indicates the static memory requirement of a logged on, but inactive user

/� ��

� After start up and warm up, measure the framework space ‘M1’ as described on the previous slide

� log on ‘N’ users, execute representative business scenarios, and then keep them logged on

� Log on to the J2EE engine via telnet interface, jump to the JVM, and trigger full GC several times

� Look into the GC log file, read the “Bytes after GC” ‘M2’

� User session space = (M2 – M1) / N


/�� &5�4'��

��

� The average garbage collected bytes of memory per user interaction step

� Represents the dynamic memory consumption of an active user, when processing specific user interaction steps

� Is application-specific

/� ��

� Execute a large number ‘K’ of user interaction steps

� Measure the total garbage collected bytes ‘B’ during this time period

� Processing space = B / K


4� � �� * �� ""��

�#� �� ""��

� Productivity enhancement of Java SW development

� Java memory behavior can be observed by GC behavior

� Several collectors with different algorithms (for young/old generation, concurrent/parallel, …) are available

- �#*��&�"��#��#��:�� "�# ��'/ ��%

�$��%��

� Resource consumption of GC

� “Stop the world pause”, despite parallel and concurrent algorithms

�#�� *��

� Poor response time

� Poor CPU utilization

� No locality of memory accesses, paging becomes “dangerous”

� Java resonance effect in worst cases


*�5�4�'*�� *�4�� ""

*��

� The average elapsed time for completion of a GC cycle

*�� "

� The average time between 2 successive GC occurrences

�#�*�5�4�'

� Depend on the heap configuration, the GC algorithms used, and the system load

� GC duration and interval are dependant KPIs (relative GC time = GC duration / GC interval)

/� ��

� The 2 GC KPIs can be calculated as average counters from a GC log


4�� *�5�4�

*�� *�� "

� To determine whether the impact of GC activities is acceptable

� Recommendation regarding minor GCs– GC duration < 0.2 s and GC interval > 1.0 s

� Recommendation regarding full GCs– GC duration < 10 s, GC interval > several minutes

(�" ��*��

� GC duration / GC interval– Be careful, high GC duration could cause Java resonance

� Total GC time / Test run time– Is system load / throughput dependant

� GC time per UIS / avg. response time– = total GC time / total #UIS / avg. response time– Reflects the impact of GC activities, is throughput-independent– Could be used to compare different Java heap configurations

� GC time per UIS / avg. CPU time per UIS– More reproducible than GC time per UIS / avg. response time


!�-3 � �"� ��4�� 5�4�

A certified EP-ESS Benchmark with SAP NetWeaver Portal 6.0, SP13, running on a 2 processors / 4 cores / 4 threads machine(4920 SAPS)

0.200 sGC duration

1.876 sGC interval

3.250 MBProcessing space

0.735 MBUser session space

208 MBFramework space

0.033 sCPU time per user interaction step

1.940 sAverage response time

98%CPU utilization

1440Number of concurrent users with 10 seconds think time


+�$ �!� "&)� ��/�� & 0��/2��"�� 8

��/�� &

� The JVM doesn‘t have enough heap memory to continue processing

/�� &��

� Observations: steady increase of memory

� Java objects no longer used, but still have references so that they can not be garbage collected

��"�� /

� Memory leak

� Memory shortage– Wrong (physical) memory sizing– Wrong configuration of the J2EE engine– Wrong configuration of the JVM


4�� /�� &��

(��#��/��"��

� Observe Java heap usage after full GCs

� With few users, run tests over long time period

� Distinguish between different business scenarios / user interaction steps

4��&#�� &"� ��

� “High-volume” memory leaks– Failed session termination �� check session handling– Big objects �� code review– …

� “Low-volume” memory leaks– Using Java memory profiling– …


��/��"�� /�� &�#��

Utilize available physical memory by proper Java memory configuration

For java performance, the more memory the better

OOM (Out of memory)

Increased GC activities,more frequent full GCs

Reduced GC activities

Java heap + PermSpace + Stack

Java heap + PermSpace + Stack

Size of available physical memory0


��/��"�� /�� &�#�� 0��12

��)��#&�� "� �� &

� Couple memory sizing to CPU sizing

� Configure 1 GB physical memory for every 300 SAPS CPU power

��,--��"&��"�)� � �" �"��#&�� "

� �� &

� (At least) one J2EE instance on one physical machine

� Proper number of server nodes within an instance:– Configure JVM heap according to Note 723909– #ServerNodes = availableMemory / (JavaHeap+PermSpace+Stack)

��9/��

� PermSpace– Use –XX:PrintGCDetails to determine whether OOM occurs in the PermSpace– Enlarge the PermSpace accordingly (-XX:PermSize, -XX:MaxPermSize)

� Stack overflow– Stack size depends on the number of threads– Adjust the number of application threads, if necessary (-Xss<n>M)


��/��"�� /�� &�#�� 0��12

!�#�� &

� Except in the case of memory leak, OOM and high GC activities are caused by memory shortage

� Memory shortage problems can be solved by– By using more memory resources– By using optimal memory configurations

�#��#��#��'&�� &��"&�� )�#�

� �� &�� &�� "�� 8

� Use the introduced memory KPIs to determine the memory consumption

� The general programming guidelines for memory optimization

Well designed and balanced caches in Java

Volume of long time objects

Release not needed objects as early as possible

Life time of intermediate objects

Reuse objects as long as possibleVolume of temporary objects

Programming GuidelineThe influencing factor


+�$ �!� "&)� �� "�) �� "�� 8

�&� ��

� Poor response times even with few concurrent users

� Rapid increase of response times by increased number of concurrent users

� Adding more HW machines doesn‘t serve the problem

��"��

� Unnecessary mutual exclusions in Java code

� Serialization not only within an instance, but also cluster-wide

(��

� Identify business scenarios / user interaction steps with poor response times

� Prepare users and data for load tests

� Execute load tests with increasing number of users, and measure CPU utilization, response time and CPU time per user interaction step


��.�� " ��"�&��.��

The case of “scaling well”

0

0.5

1

1.5

2

2.5

0 100 200 300 400 500

Number of Concurrent Users with 10 s Think Time

Ave

rage

resp

onse

time

/C

PU

tim

e pe

r di

alog

step

in s

econ

ds

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

RespTime CPU/UIS CPU_Util

Response time behavior as expected fromqueueing models

CPU utilizationincreases linearly

with number of users

CPU time per userinteraction step

remains constant

CP

U u

tiliz

atio

nin

per

cent

age


��.�� " ��"�&��.��0��12

The case of “busy waiting”

0

0.5

1

1.5

2

2.5

0 100 200 300 400 500

Ave

rage

resp

onse

time

/C

PU

tim

e pe

r di

alog

step

in s

econ

ds

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


CP

U u

tiliz

atio

nin

per

cent

age

significant increaseof response time already at low CPU utilization

CPU utilizationincreases

disproportionateto number of users

Number of Concurrent Users with 10 s Think Time

CPU time per userinteraction step

increases


��.�� " ��"�&��.��0��12

The case of “serialization”

0

0.5

1

1.5

2

2.5

0 100 200 300 400 500

Number of Concurrent Users

Ave

rage

resp

onse

time

/C

PU

tim

e pe

r di

alog

step

in s

econ

ds

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


CP

U u

tiliz

atio

nin

per

cent

age

rapid increase of response time already at lowCPU utilization

CPU utilizationdoesn‘t increasewith number of

users


� ��.�'+� � �� +��9/


�� &

�#� ��#��!��,---�� "��#�� " ��"�&��

��. �� &

4�� "�� $�""��5�4��

� � �� "&��

� Java memory KPIs

� Java garbage collection KPIs

� � ;�� &< ��"��

� Not only by memory leakages

� But also by memory shortages– Physical memory sizing– SAP J2EE Engine configuration– Java VM configuration

+�$��"� ��&�� "�) ��"�� ""

��$�


�� "��7 ��'

www.sap.comSAP Developer Network: www.sdn.sap.comSAP Customer Services Network: www.sap.com/services/

�� (�" �� 7 ��#��=�� !��#-�,��>

UPE 215, SAP NetWeaver Portal Benchmark and Load Test – LessonsLearned, Lecture

�� (�" �� !�-��

http://www.sap.com/education/

6��#��4��


�+!?5@�.6�(@�.(

!��-?�4�?A

B.-��4�?�C �.**-��4�?�C �4��.��4�?


Please complete your session evaluation.

Session Code: CD103

Be courteous — deposit your trash, and do not take the handouts for the following session.

Feedback

�# ��@��A�# ��@��A�# ��@��A�# ��@��A


� No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

� Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

� Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. � IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries, zSeries, z/OS, AFP,

Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or registered trademarks of IBM Corporation in the United States and/or other countries.

� Oracle is a registered trademark of Oracle Corporation.

� UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.� Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.

� HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

� Java is a registered trademark of Sun Microsystems, Inc.� JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape.

� MaxDB is a trademark of MySQL AB, Sweden.

� SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

� The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.

� This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please note that this document is subject to change and may be changed by SAP at any time without notice.

� SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.

� SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence.

� The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages.

��&��#,��>�!�!*1!""(��#�(��

Documents

Performance of SAP J2EE Engine