IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java … · 2018-05-30 · SAP NetWeaver 2004 and 7.0x Java stacks run with the IBM Technology for Java VM on IBM platforms

IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java

Environments

This document can be found on the web, www.ibm.com/support/techdocs

Version Date: March 25, 2010

Andreas Steiner Christoph Langer

Dr. Thomas Obermeier

IBM Boeblingen Lab, Germany SAP on IBM Systems Development @ SAP © Copyright 2010, IBM Corporation

http://www.ibm.com/support/techdocs�

© Copyright 2010, IBM Corporation Version 3/25/2010 http://www.ibm.com/support/techdocs IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java Environments Page 2

Table of contents Table of contents .................................................................................................................... 2 Technology Used.................................................................................................................... 5

GC Policy used for SAP NetWeaver Application Server Java................................................................. 5 Important Java VM Parameters and their defaults .................................................................................. 6 Verbose GC output .................................................................................................................................. 8

GC Problem Determination .................................................................................................. 10 Types and root causes of an Out Of Memory (OOM) ............................................................................ 10 Long GC Pause Times........................................................................................................................... 14

Class unloading times ..................................................................................................... 14 Paging on the system ...................................................................................................... 15

Frequent GC .......................................................................................................................................... 17 Insufficient heap size for the nursery area ...................................................................... 17 Insufficient heap size for the tenure area ........................................................................ 18

Tuning Recommendations based on Known Application Specific Workload .................. 19 Stability versus Performance ................................................................................................................. 19 Overall Heap Size .................................................................................................................................. 19 Application Footprint – Tenure Size ....................................................................................................... 19 Short Life Objects - Nursery Size .......................................................................................................... 24 Load balancing – Number of Server Nodes versus Larger Heap .......................................................... 25 JVM Memory Usage – Outside the Java Heap ..................................................................................... 25

Tools...................................................................................................................................... 27 Memory Problem Analysis with Eclipse Memory Analyzer .................................................................... 27

Preconditions and Installation ......................................................................................... 27 Configuration ................................................................................................................... 27 Prepare Dump Files for MAT ........................................................................................... 29 What to do with the Dump Files ...................................................................................... 29

Verbose GC Analysis ............................................................................................................................. 30 Thread Dump Analysis........................................................................................................................... 31

IBM Thread and Monitor Dump Analyzer for Java .......................................................... 31 Thread Dump Viewer for SAP NetWeaver Application Server Java 6.40/7.0 ................. 32

SAP Solution Manager - End-to-End Root Cause Analysis .................................................................. 34 Summary ............................................................................................................................... 35 Appendix ............................................................................................................................... 36

AIX ......................................................................................................................................................... 36 Native memory – svmon .................................................................................................. 36

IBM i ....................................................................................................................................................... 37 Memory pools .................................................................................................................. 37 WRKJVMJOB .................................................................................................................. 37

Linux on System z .................................................................................................................................. 38 Native memory – ps......................................................................................................... 38



Linux on AMD64/EM64T ........................................................................................................................ 39

Native memory – ps......................................................................................................... 39 Linux on Power Systems ....................................................................................................................... 39

Native memory – ps......................................................................................................... 39 Resources ............................................................................................................................. 40 About the authors ................................................................................................................. 41 Acknowledgements .............................................................................................................. 41 Trademarks and special notices ......................................................................................... 42



Abstract

Across the IBM® platforms J9 (IT4J: IBM Technology for Java) has become the preferred technology to run Java® applications in the SAP NetWeaver® stack of releases 2004 and 7.0x.

On the one hand SAP® workload may be specific in some respect. On the other hand a huge quantity of general information is available. Therefore, based on our experience in support and development, this paper aims to provide a compact and relevant selection regarding terminology, description of typical problem situations, tuning options and tool usage.

We present selected options and Java parameters and describe some basics of garbage collection. In addition we describe the basic ideas of SAP's default parameterization.

Out Of Memory situations and garbage collection issues are the main focus of a section dedicated to problem determination.

The documented default parameterization is able to support the whole range of different workloads in the SAP context with few exceptions. If it comes to potential improvements it is key to understand the ideas behind it as well as the characteristic workload of your system. We point to the most important aspects like heap layout and workload profiles to discuss our guidelines.

Lastly, we give hands on information regarding tool usage. We describe state of the art tools for analysis of garbage collection and thread dumps. In particular we explain how to prepare your system for use of the Eclipse Memory Analyzer, which stands for a major improvement in memory problem analysis.



Technology Used

SAP NetWeaver 2004 and 7.0x Java stacks run with the IBM Technology for Java VM on IBM platforms. The term Hybrid JVM (J9) is often used, as well as IT4J in the context of IBM i.

In the following sections you can find a rather compact overview of technical details of the JVM that are important when it comes to running SAP NetWeaver Java stacks upon IBM Technology for Java.

To understand this document a basic understanding of Java technology and garbage collection is required. If you are interested in more technical detail regarding the JVM please consider reading:

• The IBM Developer Kit and Runtime Environment Diagnostics Guide1

• The IBM Developer Works articles “

Java technology, IBM style”

GC Policy used for SAP NetWeaver Application Server Java

In Java, developers should not need to consider freeing (memory-) resources. Instead there’s a mechanism called “Garbage Collection” in place to take care of cleaning up and freeing resources which are no longer required. The way garbage collection (GC) is done can have a big impact on performance and resource requirements of Java applications.

Different options and algorithms exists in IBM Technology for accomplishing Java garbage collection, the so called garbage collection policies. The following figure shows the GC policies available:

Policy Option Description

Optimize for throughput

-Xgcpolicy:optthruput (optional)

The default policy. It is typically used for applications where raw throughput is more important than short GC pauses. The application is stopped each time that garbage is collected.

Optimize for pause time

-Xgcpolicy:optavgpause Trades high throughput for shorter GC pauses by performing some of the garbage collection concurrently. The application is paused for shorter periods.

Generational concurrent

-Xgcpolicy:gencon Handles short-lived objects differently than objects that are long-lived. Applications that have many short-lived objects can see shorter pause times with this policy while still producing good throughput.

Subpooling -Xgcpolicy:subpool Uses an algorithm similar to that of the default policy but employs an allocation strategy that is more suitable for multiprocessor machines. We recommend this policy for SMP machines with 16 or more processors. This policy is only available on IBM System p and System z platforms. Applications that need to scale on large machines can benefit from this policy.

Because of the runtime characteristics of an SAP NetWeaver Application Server Java which tends to create lots of short-lived objects, the generational GC policy should be used. This also offers the best tradeoff between achieving high throughput rates and having short pause times at GC invocations.

1 Although the diagnostics guide is headlined “Java Technology Edition, Version 5.0” it is valid for the IBM (J9, Hybrid) JVM 1.4 as well since this is based on 5.0 technology


http://download.boulder.ibm.com/ibmdl/pub/software/dw/jdk/diagnosis/diag50.pdf�

http://www.ibm.com/developerworks/views/java/libraryview.jsp?search_by=java+technology+ibm+style:�


Important Java VM Parameters and their defaults

This section gives a short introduction to the most important Java VM parameters, the SAP default values and a short explanation why SAP has chosen the given value. Note that in most cases the recommended parameter settings do not need to be changed. Exceptions will be described in the following sections.

The official SAP parameter recommendations can be found in the following SAP Notes

• 716927 for AIX® • 1234382 for IBM i • 1319038 for Linux on System z® • 861215 for Linux on AMD64/EM64T • 1406548 for Linux on Power Systems

Here is a list of the most important Java VM parameters

• -Xgcpolicy:gencon This parameter sets the garbage collection policy. SAP has chosen “Generational Concurrent” for the reasons given in section “Used GC policy for SAP NetWeaver Application Server Java”

• -Xmx This parameter defines the maximum size of the heap storage area which is available for Java objects. 2GB is a good base value with which to start the SAP Java Application Server. However, since application workload can grow, it might become necessary to reserve more memory. Another important point to consider here is to ensure that the complete Java heap is able to reside in main memory.

• -Xms This parameter defines the initial size of the heap storage area which is available for Java objects. SAP sets this to the same size as the maximum heap size (-Xmx) since this ensures that all memory resources are allocated to the application right at the start and precludes unnecessary operations to re-allocate memory which might be expensive.

• -Xmn This parameter defines the size of the “nursery” heap area which is used for allocation of new Java objects. Objects which survive a certain number of garbage collections in this area (tenure age) will be promoted to the tenured area. The tenured area is the remaining Java heap (-Xmx - -Xmn). SAP on AIX and IBM i use a size of 1000 Megabytes, since this has been shown to be a good trade-off in various scenarios e.g. EP or PI .We will refer to this setting in the rest of this paper. For historical reasons other IBM platforms may have different values according to the notes above.

• -Xverbose:gc This parameter writes the JVM trace information about the garbage collection to the standard output.



• -Xsoftrefthreshold

This parameter sets the number of GCs after which a soft reference will be cleared if its referent has not been marked. The default is 32, meaning that the soft reference will be cleared on the 33rd consecutive GC in which the referent has not been marked. However, this threshold only applies to the number of global collections which occur rather seldom. Since this has caused problems with some cache implementations in the SAP application, e.g. the Portal Content Directory (PCD) cache, the recommended value for SAP was the aggressive setting of 0 which means soft references are cleared in each global collection but recently changed this to 1 to improve the class/classloader unloading times and to reduce the native memory footprint with SR12.

• -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=exclusive+ prepwalk This parameter causes the JVM to write a system dump in case of an OutOfMemory condition. This can be used for the analysis with the Memory Analyzer (MAT).

• -Xdisableexplicitgc This parameter disables the user/application triggered garbage collections. Invoking such collections could result in unexpected behaviour and degraded performance of the system.

• -Xj9 (AIX, Linux on System z and Linux on Power Systems) On AIX, Linux on Power Systems and Linux on System z you have to set this option in order to activate the J9 technology.

• -Xjvm:j9vm23 (Linux on AMD64/EM64T) To activate the 2.3 build of the J9 technology as it is used on AIX and Linux on System z.

• -Djava.net.preferIPv4Stack=true (AIX only) This parameter helps to avoid startup problems in environments where IPv6 is not configured.

• -Xthr:minimizeUserCPU (Linux on AMD64/EM64T and Linux on System z) The effect of this setting is minimized user-mode CPU usage in thread synchronization, where possible. The reduction in CPU usage may be a trade-off in exchange for lower performance.



Verbose GC output

If the option -Xverbose:gc is set, the JVM will trace information about the garbage collection to standard output. For SAP systems, this can be found in the files std_<node name>.out or dev_<node name> (e.g. std_dispatcher.out, dev_server0) in the work directory of each instance. The output is XML-formatted.

Below you can find sample printouts of the garbage collection traces for the 2 typical types of collections which occur when using generational garbage collection.

The collection cycle which occurs most often is the so called “scavenger” collection, which only collects the nursery area. A typical trace reads:

<af type="nursery" id="412" timestamp="Apr 07 11:46:45 2009" intervalms="3166973.450"> <minimum requested_bytes="240" /> <time exclusiveaccessms="0.234" /> <nursery freebytes="0" totalbytes="362151936" percent="0" /> <tenured freebytes="107362792" totalbytes="402653184" percent="26" > <soa freebytes="107362792" totalbytes="402653184" percent="26" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <gc type="scavenger" id="411" totalid="414" intervalms="3166981.373"> <flipped objectcount="231212" bytes="18608616" /> <tenured objectcount="3871" bytes="343576" /> <refs_cleared soft="266" weak="16498" phantom="0" /> <finalization objectsqueued="2038" /> <scavenger tiltratio="89" /> <nursery freebytes="343468400" totalbytes="362387456" percent="94" tenureage="14" /> <tenured freebytes="106992568" totalbytes="402653184" percent="26" > <soa freebytes="106992568" totalbytes="402653184" percent="26" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="51.178" /> </gc> <nursery freebytes="343466352" totalbytes="362387456" percent="94" /> <tenured freebytes="106992568" totalbytes="402653184" percent="26" > <soa freebytes="106992568" totalbytes="402653184" percent="26" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="59.318" /> </af>

The tag “af” stands for allocation failure. This means the collection was triggered because a request for a new object allocation could not be fulfilled. One property of “af” is the timestamp which shows when the allocation failure occurred. The tag “minimum requested_bytes” contains the size of the allocation which was attempted.

This followed by information about the size of and the available space in the nursery and the tenured area before the collection. It is expected that the nursery size will be less than the size you specified with the parameter –Xmn. This is because the nursery area is split up into 2 sub-areas: one used for new allocations and the other is used during garbage collections to copy objects which need to be kept but are not old enough to be tenured (the so called flipping). When the JVM starts, the areas are allocated



half of the nursery size though this ratio might shift. The current ratio is traced in the tag “scavenger tiltratio” which gives the percentage of the area for object allocations (the part that is accounted for in tag “nursery”).

The tenured area also has 2 sub areas, the small object area (soa) and the large object area (loa) which is usually a lot smaller. Within the tag “gc” you can find statistics about the collection cycle, including information about the occupancy of the memory areas after collection and about the time it took to perform that collection cycle. After the “gc” section, again information about the heap occupancy is presented together with the overall time for the gc cycle. For a global collection cycle, the information looks similar:

<con event="collection" id="2" timestamp="Apr 07 11:16:37 2009" intervalms="314604867.541"> … <gc type="global" id="3" totalid="413" intervalms="157455948.701"> … </gc> … </con>

You may find the tag “af” instead of the tag “con” and, the “gc” cycle is not of type “scavenger” but of type “global”. You will also find some different statistics information.

Since it is difficult to analyze the garbage collection history by reading huge XML files, the tool GCMV is available for parsing and visualizing this data. It is described in section “Verbose GC analysis”.



GC Problem Determination

Types and root causes of an Out Of Memory (OOM)

Two different types of OOMs can happen. One is the so called “real” OOM, where a new allocation request can not be satisfied by the JVM. In this case, either the Java heap is completely full or there is not enough continuous memory available to satisfy the allocation request. This is, in most cases, an allocation request for a large object (several MB).

<af type="tenured" id="22" timestamp="Feb 27 06:52:32 2009" intervalms="337.224"> <minimum requested_bytes="588886488" /> <time exclusiveaccessms="0.453" /> <nursery freebytes="209321984" totalbytes="209715200" percent="99" /> <tenured freebytes="431560456" totalbytes="1728053248" percent="24" > <soa freebytes="422116712" totalbytes="1710773248" percent="24" /> <loa freebytes="9443744" totalbytes="17280000" percent="54" /> </tenured> <gc type="global" id="42" totalid="628" intervalms="338.214"> <compaction movecount="8757598" movebytes="770949264" reason="compact to meet allocation" /> <refs_cleared soft="0" threshold="0" weak="0" phantom="0" /> <finalization objectsqueued="0" /> <timesms mark="977.783" sweep="11.171" compact="974.564" total="1963.755" /> <nursery freebytes="209660120" totalbytes="209715200" percent="99" /> <tenured freebytes="432168800" totalbytes="1728053248" percent="25" > <soa freebytes="432168800" totalbytes="1728053248" percent="25" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> </gc> <gc type="global" id="43" totalid="629" intervalms="0.592"> <compaction movecount="11316940" movebytes="1283166184" reason="compact to meet allocation" /> <classloadersunloaded count="0" timetakenms="4.004" /> <refs_cleared soft="12" threshold="0" weak="0" phantom="0" /> <finalization objectsqueued="0" /> <timesms mark="1033.621" sweep="6.505" compact="4541.778" total="5586.172" /> <nursery freebytes="209660120" totalbytes="209715200" percent="99" /> <tenured freebytes="432304304" totalbytes="1728053248" percent="25" > <soa freebytes="432304304" totalbytes="1728053248" percent="25" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> </gc> <nursery freebytes="209660120" totalbytes="209715200" percent="99" /> <tenured freebytes="432304304" totalbytes="1728053248" percent="25" > <soa freebytes="432304304" totalbytes="1728053248" percent="25" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="7551.773" /> </af>

JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait. JVMDUMP007I JVM Requesting Snap Dump using '/usr/sap/PRD/D00/j2ee/cluster/server0/Snap.20090227.065239.753836.0010.trc' JVMDUMP012E Error in Snap Dump: {nothing to snap} JVMDUMP007I JVM Requesting Heap Dump using



'/usr/sap/PRD/D00/j2ee/cluster/server0/heapdump.20090227.065239.753836.0011.phd' JVMDUMP010I Heap Dump written to /usr/sap/PRD/D00/j2ee/cluster/server0/heapdump.20090227.065239.753836.0011.phd JVMDUMP007I JVM Requesting Java Dump using '/usr/sap/PRD/D00/j2ee/cluster/server0/javacore.20090227.065239.753836.0012.txt' JVMDUMP010I Java Dump written to /usr/sap/PRD/D00/j2ee/cluster/server0/javacore.20090227.065239.753836.0012.txt JVMDUMP013I Processed Dump Event "systhrow", detail "java/lang/OutOfMemoryError".

In such a case the JVM will write a thread dump (javacore), a heapdump and a snap trace. If you are running on SR12 (SR13 for Linux on System z and Linux on Power Systems) or later and the recommended JVM (-Xdump) settings are in place, a system dump will be produced in addition. The failing object cannot be found in the heapdump or system dump, so you have to check within the application, why it is requesting such a large object and if this is the normal application behaviour. To get a first indication you can check the stack trace of the current thread in the javacore. If the application behaviour is normal, then you will have to increase the Java heap, so the JVM can handle these large allocation requests. Some more hints on how to investigate this can be found in the chapter “Tuning Recommendations based on Known Application Specific Workload”.

The IBM JVM also runs into an OOM, if small allocation requests could be handled but in order to satisfy this request the GC would have to run very frequently. As an indication for this case you will find the following warning ("excessive gc activity detected") in the verbose GC output. It will occur if more than 95% of the time is spent in the GC and the GC is not able to free up more than 3% of the heap.

<af type="nursery" id="322" timestamp="Fri May 23 12:47:49 2008" intervalms="12.459"> <minimum requested_bytes="458768" /> <time exclusiveaccessms="0.479" /> <nursery freebytes="402779848" totalbytes="1179147264" percent="34" /> <tenured freebytes="179642776" totalbytes="2197815296" percent="8" > <soa freebytes="179642776" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <percolating_collect reason="insufficient remaining tenure space" /> <gc type="global" id="133" totalid="375" intervalms="30.736"> <refs_cleared soft="0" threshold="0" weak="0" phantom="0" /> <finalization objectsqueued="0" /> <timesms mark="776.538" sweep="42.234" compact="0.000" total="819.292" /> <nursery freebytes="403068104" totalbytes="1179147264" percent="34" /> <tenured freebytes="179642776" totalbytes="2197815296" percent="8" > <soa freebytes="179642776" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> </gc> <nursery freebytes="403068104" totalbytes="1179147264" percent="34" /> <tenured freebytes="179642776" totalbytes="2197815296" percent="8" > <soa freebytes="179642776" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="837.579" /> </af>

<af type="nursery" id="323" timestamp="Fri May 23 12:47:50 2008" intervalms="12.442">



<minimum requested_bytes="458768" /> <time exclusiveaccessms="0.554" /> <nursery freebytes="403019968" totalbytes="1179147264" percent="34" /> <tenured freebytes="179184008" totalbytes="2197815296" percent="8" > <soa freebytes="179184008" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <percolating_collect reason="insufficient remaining tenure space" /> <gc type="global" id="134" totalid="376" intervalms="30.228"> <refs_cleared soft="1" threshold="0" weak="1" phantom="0" /> <finalization objectsqueued="0" /> <timesms mark="552.629" sweep="41.490" compact="0.000" total="594.645" /> <nursery freebytes="403298680" totalbytes="1179147264" percent="34" /> <tenured freebytes="179184008" totalbytes="2197815296" percent="8" > <soa freebytes="179184008" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> </gc> <warning details="excessive gc activity detected" /> <nursery freebytes="402839912" totalbytes="1179147264" percent="34" /> <tenured freebytes="179184008" totalbytes="2197815296" percent="8" > <soa freebytes="179184008" totalbytes="2197815296" percent="8" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="605.693" /> </af>

JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait. JVMDUMP007I JVM Requesting Heap Dump using '/usr/sap/QX0/DVEBMGS74/j2ee/cluster/server0/heapdump.20080523.124750.2986226.phd' JVMDUMP010I Heap Dump written to /usr/sap/QX0/DVEBMGS74/j2ee/cluster/server0/heapdump.20080523.124750.2986226.phd JVMDUMP007I JVM Requesting Java Dump using '/usr/sap/QX0/DVEBMGS74/j2ee/cluster/server0/javacore.20080523.124750.2986226.txt' JVMDUMP010I Java Dump written to /usr/sap/QX0/DVEBMGS74/j2ee/cluster/server0/javacore.20080523.124750.2986226.txt

In this case the JVM will write a javacore, a heapdump and a snap trace. If you are running on SR12 or later and the recommended JVM (-Xdump) settings are in place, a system dump will be produced in addition. The heapdump or system dump can help to analyze why the GC is not able to free up memory to satisfy further allocation requests. In this case the application is holding all the memory, so you have to check if this is the normal application behaviour (i.e. large application footprint2

2 The definition of application footprint in this document is the constantly used memory in the Java heap which survives after a global GC. It is made up of J2EE Framework space and Session space.

), or if this could be a memory leak in the application. A memory leak will normally show up over a longer period of time, so you will see increasing heap usage over time.



This picture shows a reduction of free tenured heap over time even though the GC is running several times the day. The JVM will end up with an OOM due to the memory leak at the end.

Higher application footprints may require to add additional server nodes or to add more Java heap depending on the signature of the workload. Some more hints on how to investigate this can be found in the chapter “Tuning Recommendations based on Known Application Workload”.



Long GC Pause Times

The JVM has to stop active applications to acquire the heap exclusively for some specific steps during the GC. If the application is user interactive, like the SAP Portal, then this will influence the end user response time indirectly. There are a number of possible reasons for long GC pause times.

Class unloading times

Class loader and Class unloading will happen during a global GC only. Depending on the load, the tenure area does not fill up so quickly (this can take more than 10 hours), so a global GC will not happen very often. As a result, a lot of Class loaders and Classes may be in the heap (up to 60000 and more is not exceptional). As Class loader and Class unloading is an expensive operation, so this can take some time.

The GC will do this in a so-called “stop the world” phase, so the application is stopped while this is going on. A user of an interactive application will notice it immediately, due to the increased response times of the requests he sends to the J2EE® server.

The output in the verbose GC will look something like this:

<con event="kickoff" timestamp="Sep 20 22:30:09 2008"> <stats tenurefreebytes="33301488" nurseryfreebytes="46933488" tracetarget="371874464" kickoff="47042132" tracerate="8.01" /> </con>

<con event="collection" id="5" timestamp="Sep 20 22:30:13 2008" intervalms="104810343.075"> <time exclusiveaccessms="0.329" /> <nursery freebytes="671520" totalbytes="943718400" percent="0" /> <tenured freebytes="33283856" totalbytes="1098907648" percent="3" > <soa freebytes="317200" totalbytes="1065940992" percent="0" /> <loa freebytes="32966656" totalbytes="32966656" percent="100" /> </tenured> <stats tracetarget="371874464"> <traced total="365788784" mutators="131023320" helpers="234765464" percent="98" /> <cards cleaned="4554" kickoff="15680711" /> </stats> <con event="remembered set scan"> <stats objectsfound="25441" traced="19696984" timetakenms="60.827" /> </con> <con event="final card cleaning"> <stats cardscleaned="1369" traced="2497232" durationms="6.992" /> </con> <gc type="global" id="7" totalid="2991" intervalms="104810416.229"> <classloadersunloaded count="37221" timetakenms="10085.271" /> <refs_cleared soft="2114" threshold="0" weak="199843" phantom="0" /> <finalization objectsqueued="9609" /> <timesms mark="356.495" sweep="44.545" compact="0.000" total="10493.572" /> <nursery freebytes="838404520" totalbytes="943718400" percent="88" /> <tenured freebytes="552148376" totalbytes="1098907648" percent="50" > <soa freebytes="530170264" totalbytes="1076929536" percent="49" /> <loa freebytes="21978112" totalbytes="21978112" percent="100" /> </tenured>



</gc> <nursery freebytes="838404520" totalbytes="943718400" percent="88" /> <tenured freebytes="552148376" totalbytes="1098907648" percent="50" > <soa freebytes="530170264" totalbytes="1076929536" percent="49" /> <loa freebytes="21978112" totalbytes="21978112" percent="100" /> </tenured> <time totalms="10567.322" /> </con>

It took more than 10 seconds to unload the 37221 class loaders in this example. The largest proportion of the GC time was spent in classloader unloading. This also depends on the system resources (CPU) available for the GC at this time. If the tenure area isn’t overused, the interval between the global GC’s can be long. The longer the JVM runs without a global GC, the more class loaders will be created.

In case of small test/development systems, where the 2GB Java heap is not totally used, you can reduce the Java heap to get a global GC more often. With changing the parameter –Xsoftrefthreshold0 to –Xsoftrefthreshold1 some caches are working better. As a result less classes and classloaders are reloaded, because they can be reused from the cache. This can reduce the GC pause times too.

Paging on the system

If a global GC takes a very long time (more than 60 seconds) the potential for system paging is very high. Since all objects in the heap have to be checked if they are still in use or not, all of them have to be touched by the GC. So if a part of the Java heap is paged out by the OS, it has to be paged in again first to let the GC do the job. This is an expensive operation and will take some time. The GC has to wait for this and so the GC pause time will increase extremely.

The output in the verbose GC will look something like this:

<con event="collection" id="18" timestamp="Feb 15 08:52:05 2009" intervalms="220781166.438"> <time exclusiveaccessms="55.381" /> <nursery freebytes="279076808" totalbytes="835537920" percent="33" /> <tenured freebytes="8452320" totalbytes="1098907648" percent="0" > <soa freebytes="8452320" totalbytes="1098907648" percent="0" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <stats tracetarget="605053568"> <traced total="516943032" mutators="505496920" helpers="11446112" percent="85" /> <cards cleaned="625" kickoff="25396506" /> </stats> <con event="remembered set scan"> <stats objectsfound="1658" traced="4699448" timetakenms="17.520" /> </con> <con event="final card cleaning"> <stats cardscleaned="11079" traced="124056016" durationms="2157.260" /> </con> <gc type="global" id="24" totalid="2110" intervalms="220783345.355"> <classloadersunloaded count="22034" timetakenms="417573.054" /> <refs_cleared soft="2784" threshold="0" weak="14622" phantom="0" /> <finalization objectsqueued="4953" /> <timesms mark="82455.598" sweep="5061.014" compact="0.000"



total="505127.392" /> <nursery freebytes="646404712" totalbytes="835537920" percent="77" /> <tenured freebytes="222339176" totalbytes="1098907648" percent="20" > <soa freebytes="222339176" totalbytes="1098907648" percent="20" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> </gc> <nursery freebytes="646404712" totalbytes="835537920" percent="77" /> <tenured freebytes="222339176" totalbytes="1098907648" percent="20" > <soa freebytes="222339176" totalbytes="1098907648" percent="20" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="507362.851" /> </con> [Framework -> criticalShutdown] Missed broadcast due to network problems. Restart is required (advanced Reconnect) [Framework -> criticalShutdown] Exiting Listener Loop. This requires a restart of the node. Possible reason is an interrupted reconnect session to the message server. Feb 15, 2009 9:00:39 AM com.sap.engine.core.Framework [SAP J2EE Engine|MS Socket Listener] Fatal: Critical shutdown was invoked. Reason is: Exiting Listener Loop. This requires a restart of the node. Possible reason is an interrupted reconnect session to the message server. Feb 15, 2009 9:00:39 AM com.sap.engine.core.Framework [Thread[Thread-4373,5,SAPEngine_System_Thread[impl:5]_Group]] Fatal: Critical shutdown was invoked. Reason is: Missed broadcast due to network problems. Restart is required (advanced Reconnect) </verbosegc>

The J2EE server node will be restarted automatically after a specific time without a response to the message server (as defined in ms.notification.timeout of the ClusterManager).

Make sure that you have sufficient available physical memory in your system. All the memory used by the JVM must be available in the physical memory to guarantee short pause times in the GC.



Frequent GC

Insufficient heap size for the nursery area

If a lot of new objects are allocated in a very short period of time, the objects will be allocated in the nursery area first. If these objects have a short lifetime they will not be moved to the tenure area, so they will die in the nursery area. If this area is too small to handle all these new object allocations the scavenger GC will run with a very short interval.

<af type="nursery" id="177271" timestamp="Feb 15 16:46:31 2009" intervalms="979.280"> <minimum requested_bytes="64" /> <time exclusiveaccessms="0.370" /> <nursery freebytes="0" totalbytes="377487360" percent="0" /> <tenured freebytes="239816480" totalbytes="1728053248" percent="13" > <soa freebytes="239816480" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <gc type="scavenger" id="177262" totalid="177391" intervalms="983.879"> <flipped objectcount="61579" bytes="7634232" /> <tenured objectcount="3086" bytes="225280" /> <refs_cleared soft="5" weak="163" phantom="0" /> <finalization objectsqueued="41" /> <scavenger tiltratio="90" /> <nursery freebytes="369360896" totalbytes="377487360" percent="97" tenureage="14" /> <tenured freebytes="239407144" totalbytes="1728053248" percent="13" > <soa freebytes="239407144" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="84.446" /> </gc> <nursery freebytes="369358848" totalbytes="377487360" percent="97" /> <tenured freebytes="239407144" totalbytes="1728053248" percent="13" > <soa freebytes="239407144" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="89.102" /> </af> <af type="nursery" id="177272" timestamp="Feb 15 16:46:32 2009" intervalms="981.544"> <minimum requested_bytes="64" /> <time exclusiveaccessms="0.477" /> <nursery freebytes="0" totalbytes="377487360" percent="0" /> <tenured freebytes="239407144" totalbytes="1728053248" percent="13" > <soa freebytes="239407144" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <gc type="scavenger" id="177263" totalid="177392" intervalms="986.109"> <flipped objectcount="82457" bytes="11473952" /> <tenured objectcount="2760" bytes="243208" /> <refs_cleared soft="0" weak="305" phantom="0" /> <finalization objectsqueued="12" /> <scavenger tiltratio="90" /> <nursery freebytes="365535200" totalbytes="377487360" percent="96"



tenureage="14" /> <tenured freebytes="239076584" totalbytes="1728053248" percent="13" > <soa freebytes="239076584" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="105.921" /> </gc> <nursery freebytes="365533152" totalbytes="377487360" percent="96" /> <tenured freebytes="239076584" totalbytes="1728053248" percent="13" > <soa freebytes="239076584" totalbytes="1728053248" percent="13" /> <loa freebytes="0" totalbytes="0" percent="0" /> </tenured> <time totalms="111.238" /> </af>

In this case the nursery area (approximately 360MB for new allocations, the rest - up to 400MB - is reserved for the survivor space, depending on the scavenger tiltratio of 90) is filled in less than 1 second (986.109 ms). Only 11473952 bytes are flipped to the survivor space and only 243208 bytes are tenured only, so the most of the new objects are temporary used and die immediately.

Another scenario is with large/huge new object allocations as can happen in an SAP PI system. If a large object does not die fast enough, or there are too many new large objects created, the scavenger GC will have to move them to the tenure area. Moving these large objects is an expensive operation. You will also see frequent scavenger GC’s but there are more bytes flipped and moved.

Verify that this behaviour and workload is expected by the application. If so increase the nursery size to increase the scavenger GC interval correspondingly.

Insufficient heap size for the tenure area

The application footprint approaches the size of the available heap for the tenure area or is larger than the available heap. In this case the GC (global) cannot free up enough memory to satisfy new allocation requests (objects have to be moved from the nursery to the tenure area). This will lead to a frequent global GC (“con event”) and if the JVM is spending more time in the GC than in the application this will result in a JVM restart, as already described in chapter “Types and root causes of an Out Of Memory (OOM)”

Verify the size of the application footprint and increase the overall heap size (without increasing the nursery size) to free more memory for the tenure area, if the workload is already distributed to enough server nodes. Check the “Tuning Recommendations based on Known Application Specific Workload” section about this.



Tuning Recommendations based on Known Application Specific Workload This section describes which heap size adaptation, in which areas, would make sense depending on the application footprint, the amount of short life objects, huge object allocations and application characteristics.

We strongly discourage from tuning activities beyond the ones described in the following chapters and from setting/modifying other Java VM parameters that are not mentioned here.

Stability versus Performance

Don’t tune the setting only for the current specific workload: there should be room for some workload peaks. An OOM and restart of the JVM and application downtime have a much greater impact than a minimally higher response time from the application. The SAP standard parameter settings were developed with this in mind.

Overall Heap Size

There is a standard recommendation to run SAP NetWeaver on an IBM 1.4.2 64-bit JVM with 2 GB maximum heap size. This is documented by platform in the previously mentioned SAP Notes.

This 2GB is a good starting point and will help to run most of the SAP NetWeaver systems without any problems. There are several workload conditions which may require a larger heap, however e.g. huge message transfer in SAP NetWeaver PI, SAP NetWeaver EP with WebDynpro or generally in SAP NetWeaver BI systems. In addition, the internal split of the heap internal in the JVM (nursery/tenure area) may sometimes require modifications.

Application Footprint – Tenure Size

The application footprint can be determined from the verbose GC output. To check this, monitor the available free memory in the tenure area after a global GC. Check this value over a longer period of time (several global GC’s) over a week. It should be nearly constant. To calculate the application footprint, subtract the free available memory from the total tenure area. The calculated value is the application footprint. This can be done by analyzing the verbose GC output manually or more simply with the GC Memory Visualizer, which gives a graphical overview of the tenure area usage.

<con event="collection" id="1" timestamp="Apr 03 12:21:54 2009" intervalms="0.000"> <time exclusiveaccessms="0.273" /> <nursery freebytes="19162016" totalbytes="907292672" percent="2" /> <tenured freebytes="59333840" totalbytes="1098907648" percent="5" > <soa freebytes="4389072" totalbytes="1043962880" percent="0" /> <loa freebytes="54944768" totalbytes="54944768" percent="100" /> </tenured> <stats tracetarget="355386752">



<traced total="121205584" mutators="121010096" helpers="195488" percent="34" /> <cards cleaned="498" kickoff="14993723" /> </stats> <con event="remembered set scan"> <stats objectsfound="7258" traced="16554728" timetakenms="72.545" /> </con> <con event="final card cleaning"> <stats cardscleaned="57" traced="28147448" durationms="89.847" /> </con> <gc type="global" id="1" totalid="411" intervalms="0.000"> <classloadersunloaded count="7548" timetakenms="1048.516" /> <refs_cleared soft="7291" threshold="0" weak="32395" phantom="0" /> <finalization objectsqueued="9237" /> <timesms mark="101.126" sweep="35.331" compact="0.000" total="1213.780" /> <nursery freebytes="795388248" totalbytes="907292672" percent="87" /> <tenured freebytes="896263904" totalbytes="1098907648" percent="81" > <soa freebytes="841319136" totalbytes="1043962880" percent="80" /> <loa freebytes="54944768" totalbytes="54944768" percent="100" /> </tenured> </gc> <nursery freebytes="795388248" totalbytes="907292672" percent="87" /> <tenured freebytes="896263904" totalbytes="1098907648" percent="81" > <soa freebytes="841319136" totalbytes="1043962880" percent="80" /> <loa freebytes="54944768" totalbytes="54944768" percent="100" /> </tenured> <time totalms="1378.762" /> </con>

<con event="collection" id="9" timestamp="Apr 04 17:18:02 2009" intervalms="25030586.890"> <time exclusiveaccessms="0.288" /> <nursery freebytes="807144" totalbytes="905299968" percent="0" /> <tenured freebytes="13254424" totalbytes="1098907648" percent="1" > <soa freebytes="658104" totalbytes="1079128064" percent="0" /> <loa freebytes="12596320" totalbytes="19779584" percent="63" /> </tenured> <stats tracetarget="190257376"> <traced total="123853808" mutators="86309416" helpers="37544392" percent="65" /> <cards cleaned="556" kickoff="8113332" /> </stats> <con event="remembered set scan"> <stats objectsfound="7229" traced="15863280" timetakenms="51.182" /> </con> <con event="final card cleaning"> <stats cardscleaned="152" traced="21589528" durationms="57.660" /> </con> <gc type="global" id="14" totalid="6567" intervalms="8594942.868"> <classloadersunloaded count="8860" timetakenms="1122.505" /> <refs_cleared soft="9138" threshold="0" weak="35746" phantom="0" /> <finalization objectsqueued="11663" /> <timesms mark="144.068" sweep="66.509" compact="0.000" total="1336.675" /> <nursery freebytes="802748896" totalbytes="905299968" percent="88" /> <tenured freebytes="881640008" totalbytes="1098907648" percent="80" > <soa freebytes="864702200" totalbytes="1079128064" percent="80" /> <loa freebytes="16937808" totalbytes="19779584" percent="85" /> </tenured> </gc> <nursery freebytes="802748896" totalbytes="905299968" percent="88" />



<tenured freebytes="881640008" totalbytes="1098907648" percent="80" > <soa freebytes="864702200" totalbytes="1079128064" percent="80" /> <loa freebytes="16937808" totalbytes="19779584" percent="85" /> </tenured> <time totalms="1447.785" /> </con>

In this example the application footprint is nearly constant. The tenure area is filled several times and after a global GC there is around 80% free (about 840MB of 1048MB tenure area).

The next graph shows this from another perspective, where you can easily see the used tenure memory after a global GC over time:



To let the GC work optimally some extra memory in addition to the application footprint is needed; otherwise a global GC will happen too often. If the majority of short-lived objects can be handled in the nursery, a small extra amount of memory is necessary for the tenure area, in addition to the application footprint for the tenure area. Experience has shown that the tenure area should be at least 20-30% larger than the application footprint. With the standard SAP parameter recommendation the system can accommodate an application footprint of around 700-800MB without any problems. In this example the footprint is much smaller, around 340MB, so there is a big reserve for some peak workloads.

There should be enough space to handle occasional large object allocations, otherwise the JVM will run into an OOM and the application will be restarted.



With very small application footprints, e.g. in test or development systems you can also think about reducing the tenure area and so the total heap. This will fill up the tenure area faster, so a global GC will happen more often. As the heap is smaller the GC pause time will be shorter. In addition the number of class loaders and classes are lower. As less of them are created in a shorter period of time, the class loader unloading will be done faster. User interactive environments will benefit from these shorter GC pause times with a more constant response time and the overall memory usage in the system will be reduced. Be careful about reducing the heap to less than 2GB, because some reserve for peak workload is needed, otherwise you risk an OOM and JVM restart.



Short Life Objects - Nursery Size

To satisfy different workload characteristics the standard recommendation for the nursery size is 1000M on AIX and IBM i. This also allows the SAP Portal to run as a PI system with short response times and deal with a “normal” amount of throughput.

In some specific cases (like huge message transfer in a SAP NetWeaver PI system) it makes sense to increase the nursery heap, to handle these huge objects without moving them to the tenure area. The move operation is an expensive task and the tenure area fill up more slowly, if this is done. To increase the nursery, you must also increase the total heap size. Here you can find an example of huge (300MB files) message transfers in a SAP PI system:



The nursery size was set to 7GB and the tiltratio is at 50% (3.5GB allocation and 3.5GB survivor), so these huge allocation requests (objects of 500MB size) can be handled in the nursery area. There is no need to move them to the tenure area, as they are all dying in the nursery area. As a result there was no global GC for 48 hours, even though the tenure area was only 1GB in size. This is a very specific case, however, in with such a large nursery the tenure area should be increased as well. Experience has shown that an increase of the nursery should be normally be accompanied by increase of the tenure area. To handle large objects it would make sense to increase the nursery area to 2GB for example. For peak workloads which can not be handled in the nursery area, the tenure area should be increased to 2GB too, so the total heap size will be 4GB. The larger nursery leads to longer scavenger GC pause times. This is therefore only an option for non user-interactive applications, like SAP PI, where we are looking for throughput and not for short response times.

To get shorter scavenger GC’s (nursery GC), the nursery size may be reduced if the number of short-lived objects isn’t too high. Check the tilt ratio, to get a feeling for how many objects survive and are copied from the allocation to the survivor area in the nursery. The GC adapts the tilt ratio from 50% up to 90%, depending on the amount of new allocation and surviving objects. User interactive applications like SAP Enterprise Portal will benefit from short GC pause times; provide short and constant user response times. The freed up memory can be used for the tenure area if the application footprint is larger then 1GB without using more memory in total.

Load balancing – Number of Server Nodes versus Larger Heap

There is no strict recommendation on this: it has to be decided on a case-by-case basis. In general it is better to increase the number of server nodes instead of increasing the heap size per server node. As a larger heap will increase the GC pause times. There are exceptions, however, where a large application footprint requires a larger heap. In such cases more server nodes won’t help. However, after the server nodes have been sized to hold the application footprint, the number of server nodes may be increased to handle more workload.

In cases of large object allocation requests (as in SAP PI systems) the server heap size must be increased to deal with such large requests. To add additional server nodes won’t help in this case, as the objects will be handled by one server node.

JVM Memory Usage – Outside the Java Heap

Beside the Java heap, the JVM uses additional memory in the native heap. In the native heap the JVM itself holds memory for classes and classloaders, JIT, JNI, NIO and threads.

For system sizing, the native heap must be added to the Java heap. The native heap is not bound in IBM 64Bit systems, so the size of the native heap depends on the application workload too. In particular, the number of classes and classloaders is a big part of the native heap.

With IBM JVM 1.4.2 64Bit SR13 the native footprint of the classes and class loaders has been reduced. The JVM internal soft reference cache will also work better with the previous SAP standard parameter set, especially with –Xsoftrefthreshold0, which was introduced because of the SAP PCD soft reference based implementation.



With the standard SAP parameter recommendations you should see up to 1GB of native heap under real heavy workload on the system. The total (Java heap and native heap) must, therefore, be considered for the system sizing; otherwise the system might end up paging with resulting bad overall performance.

You can find very detailed articles at developerWorks about native memory on AIX: “Thanks for the Memory- Understanding how the JVM uses native memory on AIX” and for Windows and Linux: “Thanks for the Memory- Understanding how the JVM uses native memory on Windows and Linux”

For monitoring of the native memory usage; have a look to the “Appendix”.


http://www.ibm.com/developerworks/java/library/j-nativememory-aix�


http://www.ibm.com/developerworks/java/library/j-nativememory-linux�



Tools

Memory Problem Analysis with Eclipse Memory Analyzer

Memory analysis of a SAP NetWeaver Java stack has in general to deal with very large heaps and a huge amount of objects. Many of the existing tools were unable to handle dumps created in such environments efficiently. With the Eclipse Memory Analyzer, aka Memory Analyzer Tool (MAT), there is a tool at hand that can address this issue. Making use of the so-called dominator approach [cf. SDN Blog Dominators in the SAP Memory Analyzer] to parse and sort objects it can help to investigate productive heaps in a reasonable amount of time and on a relatively small PC.

There is no point in reproducing the excellent MAT documentation here, as it is available in several places. Entry points for generic information, necessary downloads and detailed installation instructions, are available on to the SDN Wiki page Java Memory Analysis and the MAT Eclipse page. This paper points out the specifics regarding J9. It provided information about installation packages, configuration of your system and gives a description of how to provide dumps.

Preconditions and Installation

MAT itself will in most cases be installed on a PC and needs the following parts

• The standalone version of MAT (RCP version) or installation into existing Eclipse IDE using the update manager, both available at the MAT download site:

• DTFJ adapter, available at developerWorks

• SAP NetWeaver Extensions for specific analysis of SAP NetWeaver processes which you can install following the instructions at https://wiki.sdn.sap.com/wiki/display/Java/Download+the+SAP+Netweaver+enhancements+for+the+Eclipse+Memory+Analyzer

At the server side your JVM will need to be at minimum service release level SR12 (SR13 for Linux on System z) in order to trigger dumps with J9 which can be parsed by MAT.

Configuration

Note that information regarding configuration and preparation of dump files for MAT might depend on the platform. If you are not sure, you should always be able to find the required information in the relevant SAP Notes

• 1259465 for AIX • 1267126 for IBM i • 1263258 for Linux on AMD64/EM64T • 1336952 for Linux on System z • 1265455 for Linux on Power Systems

With Java parameter –Xdump you can register agents in the VM which will generate dumps conditionally. Those agents are cumulative and do not mutually exclude each other. From our experience, they do not impact system performance significantly.


https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/6993�

https://www.sdn.sap.com/irj/scn/wiki?path=/display/Java/Java+Memory+Analysis�

http://www.eclipse.org/mat�

http://www.eclipse.org/mat/downloads.php�

http://www.ibm.com/developerworks/java/jdk/tools/mat.html�

https://wiki.sdn.sap.com/wiki/display/Java/Download+the+SAP+Netweaver+enhancements+for+the+Eclipse+Memory+Analyzer�



You should certainly add an agent for OutOfMemoryError conditions in the server processes, because this is the most critical case for which you would want be able to analyse the memory usage of your applications:

-Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError,request=exclusive+prepwalk

In some cases you may want to trigger dumps on request. This has to be handled with care, though: Generation of every single dump will require significant resources in terms of performance and disk space:

-Xdump:system:events=user,request=exclusive+prepwalk+compact

“User event” in this context means a signal SIGQUIT, which is sent to the respective process. This can be done in several ways, e.g.:

• kill -3 <pid>

• call jcmon and choose options 20 (Local Administration) 10 (Trigger dump) <node>

• ‘Dump Stack’ option in the process menu of sapmmc (http://<host>:5<inst>13)

Naming conventions and location of the dumps may differ from platform to platform. Look up the active defaults by calling

java –Xdump:system:defaults

Defaults can be influenced in various ways. If you want to choose your own format or need to route the dumps into a specific file system you can for example specify the new name pattern as an additional parameter e.g.

-Xdump:system:defaults,file=/my_file_system/ core.%Y%m%d.%H%M%S.%pid.%seq.dmp

Or as extension to the options above e.g.

-Xdump:system:events=user,request=exclusive+prepwalk+compact,file=/my_file_system/ core.%Y%m%d.%H%M%S.%pid.%seq.dmp

If the given path is not an absolute path the working directory of the process will be the dump location. This can be overruled by setting environment variable IBM_COREDIR accordingly.

The same methods hold for thread dumps and heap dumps, i.g.:

-Xdump:java:defaults,file=javacore.%Y%m%d.%H%M%S.%pid.%seq.txt

-Xdump:heap:defaults,file=heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd

The relevant environment variables would then be with IBM_JAVACOREDIR and IBM_HEAPDUMPDIR, respectively.



Prepare Dump Files for MAT

Once the dump has been triggered, a system core <corefile> using the name pattern given above, say core.YYYYMMDD.HHMMSS.PID.NNNN.dmp, will be written by the JVM. By default the location is the working directory of the respective process, /usr/sap/<SID>/<inst>/j2ee/cluster/server0 for the server0 process.

This file has to be transformed to a zipped file <corefile>.zip containing DTFJ information by the following calls:

AIX: <JAVA_HOME>/jre/bin/j9jextract <corefile>

IBM i (CALL Qp2term): /QOpenSys/QIBM/ProdData/JavaVM/jdk14/64bit/jre/jextract <corefile>

Linux on System z: <JAVA_HOME>/jre/bin/j9jextract <corefile>

Linux on AMD64/EM64T: <JAVA_HOME>/jre/bin/jextract <corefile>

Linux on Power Systems: <JAVA_HOME>/jre/bin/j9jextract <corefile>

The resulting file <corefile>.zip can then be read directly by MAT.

What to do with the Dump Files

If you are not a developer hunting for memory leaks in your own application, this will be the right moment to get SAP support involved for further analysis.

In order to perform your own analysis, you have to provide file system access to <corefile>.zip from the host running MAT or transfer the file there. After you start MAT, the tool provides multiple forms of guidance and instruction, as well as references to relevant sources of information.



Verbose GC Analysis

Analysis of the heap with MAT tends to be relevant for exceptional and problematic cases or for application development. Complementary, though of a more administrative nature, is the task of monitoring how heap usage develops over time and checking for indications of any pathological behaviour in the VM. This can be addressed by observing garbage collection, and therefore, Java parameter -verbose:gc is set by default.

As garbage collection occurs very often during lifetime of a server process, it is exhausting, if not impossible, to get a clear picture by looking at the pure text. Also the amount of information available for a single collection can also be quite complex and, therefore, a reasonable analysis will require some tool for assistance.

The current tool of choice is the “IBM Monitoring and Diagnostic Tools for Java™ - Garbage Collection and Memory Visualizer” (GCMV). The snapshots in the discussions above were made with GCMV.

GCMV is a plugin to the “IBM Support Assistant” (ISA). Note that you will have to install at least one product like “IBM Developer Kit for Java 1.4” within ISA, to enable GCMV within the tool. Entry point for downloads is http://www.ibm.com/software/support/isa/.

Tool handling is menu-based and quite intuitive. Besides the line plot, a summary report with observations and recommendations is given, reflecting the chosen observables. Be aware, though, that the recommendations are based on very generic assumptions and are, in many cases, irrelevant for the SAP Java environment with its very specific work load.

The “IBM Pattern Modelling and Analysis Tool for Java Garbage Collector” (PMAT) is easier to install but has less sophisticated functionality. It is available with invocation instructions at http://alphaworks.ibm.com/tech/pmat/download.


http://www.ibm.com/software/support/isa/�

http://alphaworks.ibm.com/tech/pmat/download�


Thread Dump Analysis

Sometimes you will need to get a quick overview about how threads of your Java processes are behaving and if e.g. there exist (dead) lock situations that keep your applications from running as expected. Here, thread dumps can help. They are triggered with every signal SIGQUIT the Java process receives and are stored in text files javacore<date>.txt with the same naming conventions as <corefile> above. Content is a snapshot of the process, with information on the wait or blocking status of each thread as well as call stacks of the threads.

Although it is not hard to read and interpret the content from the text file directly it is often convenient to display, highlight and filter the information. Even more important is the comparison of series of dumps from the same process. Two tools that slightly differ in their capabilities and put emphasis on different details are therefore mentioned here. Both of them are quite intuitive to use and easy to install:

IBM Thread and Monitor Dump Analyzer for Java

The IBM Thread and Monitor Dump Analyzer for Java is shipped as a jar file jca<version>.jar. It can be downloaded from http://www.alphaworks.ibm.com/tech/jca. Java 2 Runtime Environment 5.0 or higher is prerequisite for this tool. Invocation is as simple as

java -Xmx1024m –jar jca<version>.jar


http://www.alphaworks.ibm.com/tech/jca�


Thread Dump Viewer for SAP NetWeaver Application Server Java 6.40/7.0

The Thread Dump Viewer for SAP J2EE Engine 6.40/7.0 which is attached to SAP note 1020246 as jar file tdv640.JAR, may be used as an alternative to the tool described above. It can be invoked the same way:

java -jar tdv640.JAR





SAP Solution Manager - End-to-End Root Cause Analysis

SAP Solution Manager standardizes, aggregates, and correlates the following functions:

• Performance and resource metrics • Access to technical configuration • Exceptions (critical logs and dumps) • Traces • Transparency on changes to software (code), configuration, or content

More information is available at the SAP Support Portal, http://service.sap.com/diagnostics


http://service.sap.com/diagnostics�


Summary The documented default parameterization is able to support the majority of different workloads in the SAP context. Some exceptions may require specific changes. This document will assist in detection of the root cause of problems and in finding the right tuning for the system workload. The following points must always be considered:

1. Do not tune the settings only for the current specific workload: there should be room for some workload peaks.

2. System stability is more important than system performance.

For user interactive applications, like the SAP Portal, it is better to increase the number of server nodes instead of increasing the heap size per server node for load balancing and availability reasons. In cases of very large message transfer (huge object allocations), e.g. in SAP PI, this won’t help to solve an OOM situation. The message has to fit in the Java heap of one J2EE server node. In this case, the Java heap size must be increased as described in the tuning recommendations.

For the overall system sizing, the native heap must be added to the Java heap: otherwise the system can run into paging and as a result the Java performance will drop dramatically.



Appendix

Platform specifics

AIX

Native memory – svmon

To monitor the native memory of the JVM (jlaunch) you can use the AIX command svmon. For a first overview use the script which is provided with the “IBM Monitoring and Diagnostic Tools for Java™ - Garbage Collection and Memory Visualizer” (GCMV). The script and a description of use can be found via Help -> Help Contents -> IBM Monitoring and Diagnostics Tools for Java™ – Garbage Collection and Memory Visualizer -> Supported data types -> Native memory -> AIX native memory. Run the script against the PID of the SAP J2EE node (search for the jlaunch named processes with the ps –ef command) over a longer period of time (several days). The collected data can be loaded into the GCMV and will show you the overall memory usage (Java heap and native heap) of the specific process graphically.



IBM i

Memory pools

As mentioned earlier it is key to avoid paging in Java processes. On IBM i it is possible to route certain processes into a dedicated memory pool. This way sufficient memory may be assigned to Java processes and memory usage of different workload as e.g. ABAP® and Java may be separated. Utilizing memory pools has provenvery valuable in this regard. SAP note 1023092 describes in detail how this can be setup.

WRKJVMJOB

Whenever a quick snapshot of a Java process is required the OS command WRKJVMJOB can help. It yields quick access to all processes that loaded an IT4J JVM. For every process GC, thread, environment and other job information can be accessed. Within GC information the native memory consumption can be observed as ‘Other memory’.



Linux on System z

Native memory – ps

To monitor the native memory of the JVM (jlaunch) you can use the Linux command ps. The procedure for Linux on System z is very similar to AIX.

For a first overview use the script which is provided with the “IBM Monitoring and Diagnostic Tools for Java™ - Garbage Collection and Memory Visualizer” (GCMV). The script and a description of use can be found via Help -> Help Contents -> IBM Monitoring and Diagnostics Tools for Java™ – Garbage Collection and Memory Visualizer -> Supported data types -> Native memory -> Linux native memory. Run the script against the PID of the SAP J2EE node (search for the jlaunch named processes with the ps –ef command) over a longer period of time (several days).

In some circumstances this script doesn’t work correctly. In this case please use the following script:

#!/bin/sh

# The process id to monitor is the first and only argument.

PID=$1

# The interval between ps invocations, in seconds.

INTERVAL=3

# Echo the date line so we know when monitoring began.

echo timestamp = `date +%s`

# Echo the interval frequency.

echo "ps interval = $INTERVAL"

while ([ -d /proc/$PID ]) do

ps -p $PID -o pid,vsz,rss

sleep $INTERVAL

done;

The collected data can be loaded in to the GCMV and will show you the overall memory usage (Java heap and native heap) of the specific process graphically.



Linux on AMD64/EM64T


To monitor the native memory you can use the same approach as for Linux on System z.

Linux on Power Systems


To monitor the native memory you can use the same approach as for Linux on System z.



Resources

These Web sites provide useful references to supplement the information contained in this document:

• Java technology, IBM style: Garbage collection policies, Part 1 http://www.ibm.com/developerworks/java/library/j-ibmjava2/index.html

• Java technology, IBM style: Garbage collection policies, Part 2 http://www.ibm.com/developerworks/java/library/j-ibmjava3/index.html

• Understanding how the JVM uses native memory on AIX http://www.ibm.com/developerworks/java/library/j-nativememory-aix

• Understanding how the JVM uses native memory on Windows and Linux http://www.ibm.com/developerworks/java/library/j-nativememory-linux

• DTFJ adaptor available at developerWorks http://www.ibm.com/developerworks/java/jdk/tools/mat.html

• ISA download site http://www.ibm.com/software/support/isa/ • Java diagnostics, IBM style, Part 2: Garbage collection with the IBM Monitoring and Diagnostic

Tools for Java - Garbage Collection and Memory Visualizer http://www.ibm.com/developerworks/java/library/j-ibmtools2/index.html

• PMAT download site http://alphaworks.ibm.com/tech/pmat/download • IBM Thread and Monitor Dump Analyzer for Java http://www.alphaworks.ibm.com/tech/jca • SDN Blog Dominators in the SAP Memory Analyzer

https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/6993 • SDN Wiki page Java Memory Analysis

https://www.sdn.sap.com/irj/scn/wiki?path=/display/Java/Java+Memory+Analysis • MAT Eclipse page http://www.eclipse.org/mat • MAT download site http://www.eclipse.org/mat/downloads.php: • SAP NetWeaver Extensions for MAT

https://wiki.sdn.sap.com/wiki/display/Java/Download+the+SAP+Netweaver+enhancements+for+the+Eclipse+Memory+Analyzer

• SAP Service Marketplace (SAP Notes) https://service.sap.com/notes • SAP End-to-End Root Cause Analysis http://service.sap.com/diagnostics


http://www.ibm.com/developerworks/java/library/j-ibmjava2/index.html�

http://www.ibm.com/developerworks/java/library/j-ibmjava3/index.html�



http://www.ibm.com/developerworks/java/jdk/tools/mat.html�

http://www.ibm.com/software/support/isa/�

http://www.ibm.com/developerworks/java/library/j-ibmtools2/index.html�

http://alphaworks.ibm.com/tech/pmat/download�

http://www.alphaworks.ibm.com/tech/jca�

https://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/6993�

https://www.sdn.sap.com/irj/scn/wiki?path=/display/Java/Java+Memory+Analysis�

http://www.eclipse.org/mat�

http://www.eclipse.org/mat/downloads.php�



https://service.sap.com/notes�

http://service.sap.com/diagnostics�


About the authors Andreas Steiner is a Software Engineer and has a degree in computer sciences from the University of Applied Sciences of Worms in Germany. He is a member of the SAP on Power (AIX) Development team at SAP in Germany since 2003 and joined IBM in 2005. In his current role, Andreas is the technical focal point for Java™ on IBM platforms at SAP. His latest projects were focusing on AIX 6.1 pre validation in SAP environments and evaluating Workload Partitions (WPARs) for SAP environments.

Christoph Langer is a member of the joint development team for SAP on IBM i. He has 8 years of experience in that area. In the last few years he was mainly working on porting/enabling SAP’s NetWeaver Java stack on IBM i and he was acting as focal point for Java™ related issues between IBM and SAP. He holds a bachelor's degree in Information Technology Management from the BA Stuttgart, Germany.

Dr. Thomas Obermeier is a member of the joint development team for SAP on IBM i. He takes the SAP part of Java related responsibilities in this team and was recently deeply involved in the transition project from Classic VM to IT4J with SAP on this platform.

Acknowledgements Reviewer:

Enrico Mayer

Harald Duvenbeck

Bernd Franz

Sally Power

Kolby Hoelzle

Oliver Stabel



Trademarks and special notices © Copyright IBM Corporation 2009. All rights Reserved.

References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions

worldwide. A current list of IBM trademarks is available on the Web at www.ibm.com/legal/copytrade.shtml.

Java, and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

ABAP, SAP NetWeaver, SAP, and SAP logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have

achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly

available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are

taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products

and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM

products should be addressed to the supplier of those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific

levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product

announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help

with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or

performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream,

the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will

achieve throughput or performance improvements equivalent to the ratios stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of

those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.


http://www.ibm.com/legal/copytrade.shtml�

Documents

IBM JVM 1.4.2 64-bit in SAP NetWeaver Application Server Java … · 2018-05-30 · SAP NetWeaver 2004 and 7.0x Java stacks run with the IBM Technology for Java VM on IBM platforms