Introduction to the IBM Java Tools

© 2013 IBM Corporation© 2013 IBM Corporation

Introduction to the IBM Monitoring and Diagnostic Tools for JavaChris Bailey, Peter Whitehead, and Guests!IBM

TAW-1560

© 2013 IBM Corporation

Please Note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.


Introduction to the Speakers

Chris BaileyJava Serviceability and Cloud Integration Architect• 13 years experience developing and deploying Java SDKs

• Recent work focus:‒ Java integration into the cloud‒ Java monitoring, diagnostics and troubleshooting‒ Requirements gathering‒ Highly resilient and scalable deployments

• Contact Details:‒ [email protected]‒ http://www.linkedin.com/in/chrisbaileyibm‒ http://www.slideshare.net/cnbailey/

mailto:[email protected]

http://www.linkedin.com/in/chrisbaileyibm

http://www.slideshare.net/cnbailey/



Peter WhiteheadProgram Manager, Java Technology Center and ASPN Development• 23 years experience developing storage subsystems, program

management and deploying IBM's Java SDKs

• Recent work focus:‒ Improving Critical Patch Update concurrency‒ Delivery of Java runtimes for IBM platforms and products‒ Creation of serviceability tooling suite for IBM Java runtimes‒ Delivery of distributed cloud operating environment for mobile networks

• Contact Details:‒ [email protected]




Plus Guests!

• Dan Julin: STSM, AIM Serviceability, eSupport and Tools• Jason Edmeades: Messaging Service Architect• Andrew Smithson: CICS Transaction Gateway Technical Lead• Chuck Bridgham: Software Architect: Rational Analysis, Design and

Construction


Goals of the talk

• Introduce the IBM tools for monitoring, debugging and diagnosing problems in the deployment life cycle

• Understand the major problems scenarios, and how to select the best tools to diagnose the problem

• Learn how to extend the tools with knowledge of your application or to build custom tools

• See how IBM is building on top of the Java tools platform


Agenda

• Tools for Java‒ Introduction to the IBM Monitoring and Diagnostic Tools for Java

• Extending the Tools‒ Overview of the APIs and Use Cases

• IBM Extensions for the Java Tools‒ IBM Extensions for Memory Analyzer (IEMA)‒ Rational Application Developer v9.0


Tools for Java


IBM Monitoring and Diagnostic Tools for Java

• A free suite of tools for understanding different aspects of Java applications

− Garbage Collection and Memory Visualizer (GCMV)− Health Center− Memory Analyzer (and IBM Extensions for Memory Analyzer)− Interactive Diagnostic Data Explorer (IDDE)

• Provide visualisation, analysis and recommendations• Fully supported

• Available in the IBM Support Assistant− Provides the “toolbox” in which analysis tools reside− Assist with opening PMRs and working with IBM Support− Search feature to query IBM and non-IBM knowledge banks


Garbage Collection and Memory Visualizer (GCMV)

• Tool to analyze Java and “Native” Heap memory− Using verbose GC logs and OS level process memory logs

• Graphs to show garbage collection and Java heap statistics over time

• Not only for memory errors, very good for performance tuning

• Recommendations use heuristics to guide you towards issues that may be limiting performance


GCMV Highlights• Graphical Display of Data

− Allows graphing of all available data: pause times, heap size etc

− Allows zoom, cropping and change of axes value and units

− Allows comparison of multiple files

• Analysis and Recommendations− Provides tuning

recommendations based on data and flags errors.

− Analysis can be limited using cropping.

− Values and units used in analysis can be changed by changing axes values and units


Heath Center

• Live monitoring tool with very low overhead• Pre-installed in JDK and WebSphere Application Server

− Enable at startup: -Xhealthcenter− Enable at runtime:java -jar healthcenter.jar ID=<pid>

• Understand how your application is behaving− Monitor Class loading, File I/O, Environment settings, Garbage

Collection, Method Profiling, Locking, Native memory use

• Diagnose potential problems, with recommendations• Works at the JVM level – no domain-specific (e.g. J2EE) information• Suitable for all Java applications


Health Center HighlightsEnvironment reporting• Detects invalid Java options• Detects options which may hurt performance or

serviceability• Useful for remote diagnosis of configuration-

related problemsClass loading visualization• Shows all loaded classes• Shows load time• Identifies shared classes

File I/O• Monitor application file open/close events as

they occur• Lists currently open files


Health Center HighlightsGarbage Collection visualization• Visualizes heap usage and gc pause times over

time• Identifies memory leaks• Suggests command-line and tuning parameters• Same recommendation logic as GCMV

Native Memory• Detect native memory leaks in application• Determine if external forces are using more

memory• Memory counters showing which parts of the

JVM are using the most native memoryThreads view• List of current threads and states• Number of threads over time• See contended monitors


Health Center HighlightsMethod Profiling• Always-on profiling shows application activity• Identifies the hottest methods in an application• Full call stacks to identify where methods are

being called from and what methods they call• No byte code instrumentation, no recompiling

Java Lock Profiling• Always-on lock monitoring• Quickly allows the usage of all locks to be

profiled• Helps to identify points of contention in the

application that are preventing scalingLive control of application• Trigger dumps• Enable verbosegc collection


Memory Analyzer

• Tool for analyzing heap dumps and identifying memory leaks from JVMs• Works with IBM system dumps, heapdumps and Sun HPROF binary

dumps• Provides memory leak detection and footprint analysis

− Objects by Class, Dominator Tree Analysis, Path to GC Roots, Dominator Tree by Class Loader

• Shows areas of memory wastage− Collections, duplicate strings, substring/char arrays, constant value

primitives• Displays Stack trace with object references• Provides SQL like object query language (OQL)• Provides extension points to write analysis plugins


Memory Analyzer Highlights

Overview:• Overview of the dump including Java heap size

and total number of objects.• Provides links to continued analysis

Path to GC Roots:• Provides the reference chain that prevents an

object being garbage collected.

Dominator Tree grouped by Class Loader:• Lists the biggest objects using a “keep alive

tree”. Grouping by Class• Loader limits the analysis to a single application

in a JEE environment


Interactive Diagnostic Data Explorer

• Provides a visual representation of your set of dump files • An investigation log editor in which you run commands. This editor

provides the following features:− Command assistance − Auto-completion of text − The ability to save commands and output, which you can then

send to other people − Highlighted text and flagging of issues − The ability to add your own comments

• An outline view to help you navigate the commands and output in the log editor

• The ability to create your own commands, as plug-ins, and import plug-ins created by others


IDDE Highlights

• Interactive Editor:− Command assistance − Auto-completion of text − Ability to add your own comments

• Multiple format support:− All operating system− Multiple file formats− Multiple address spaces on z/OS

• Multiple plug-in support:− DTFJ− Dump Analyzer− Extensions for Memory Analyzer


Problem Scenarios and Tools

GCMV Health Center Memory Analyzer IDDE

Performance Garbage Collection performance only

Method Profiling

Lock Analysis Garbage

Collection

Garbage analysis

Collection efficiency

Memory Garbage Collection memory monitoring

Native (process) memory monitoring

Garbage Collection memory monitoring

Native (process) memory monitoring

Large object allocations

Java heap memory analysis

Runtime Process settings

Class Loading

Thread execution analysis

Application state reports

Thread execution analysis

Application state reports


Extending the Tools


Extending the Tools

• Each of the tools has extensions and/or integrations points

• Memory Analyzer and IDDE can be easily extended

• Health Center data can be integrated into other tools


Extending Health Center

• Health Center provides an API to access the data and recommendations shown in the Health Center Client

• Allows custom tools to receive, analyze and interpret the data

• Health Center is a monitoring platform


Extending Health Centerimport org.eclipse.equinox.app.IApplication;import org.eclipse.equinox.app.IApplicationContext;import org.eclipse.swt.widgets.Display;import org.eclipse.swt.widgets.MessageBox;import org.eclipse.swt.widgets.Shell;

import com.ibm.java.diagnostics.healthcenter.api.ConnectionProperties;import com.ibm.java.diagnostics.healthcenter.api.HealthCenter;import com.ibm.java.diagnostics.healthcenter.api.factory.HealthCenterFactory;import com.ibm.java.diagnostics.healthcenter.api.threads.ThreadsData;/** * This class controls all aspects of the application's execution */public class Application implements IApplication {

HealthCenter hcMon;public Object start(IApplicationContext context) throws Exception {

ConnectionProperties hcConn = new ConnectionProperties();hcMon = HealthCenterFactory.connect(hcConn, true);try {

System.out.println("hcMonWaiting for 10 seconds to allow initial data to be parsed from the connection");

Thread.sleep(10000);} catch (InterruptedException e) {

e.printStackTrace();}checkForDeadlock();return IApplication.EXIT_OK;

}

Setup connection properties

Create a Health Center connection


Extending Health Centerpublic void checkForDeadlock() {

while (!detectDeadlock()) {try {

Thread.sleep(5000);} catch (InterruptedException e) {

e.printStackTrace();}

}}private boolean detectDeadlock() {

ThreadsData hcthreadsData = hcMon.getThreadsData();if (hcthreadsData == null) {

System.out.println("No threads yet");} else {

if (hcthreadsData.deadlockDetected()) {Display display = new Display();Shell shell = new Shell(display);MessageBox mb = new MessageBox(shell);String deadlockMessage = new String();String[] hcThreadsRec = hcthreadsData

.getCriticalRecommendations();for (String rec : hcThreadsRec) {

deadlockMessage = deadlockMessage + rec + "\n";}mb.setMessage(deadlockMessage);mb.setText("Deadlock detected");mb.open();display.dispose();return true;

}}return false;

}

Request Threads data

Check for deadlock in threads

Access the threads recommendations

Display deadlock detected message


Extending Memory Analyzer and IDDE

• Memory Analyzer provides public extension points to build in additional “expert” knowledge

• org.eclipse.mat.api.nameResolver− Provides readable description of an object in some of MAT's view

(like toString())− Annotates object types with detail about an object− eg. hostname and port for a Socket object

• org.eclipse.mat.report.query− Adds a menu item that executes your plug-in code− Creates report style output in any form you want: Tree, Text,

HTML, Pie Charts, etc.


Extending Memory Analyzer and IDDEimport org.eclipse.mat.SnapshotException;

import org.eclipse.mat.snapshot.extension.IClassSpecificNameResolver;import org.eclipse.mat.snapshot.extension.Subject;import org.eclipse.mat.snapshot.model.IObject;

@Subject("com.ibm.ws.webcontainer.webapp.WebAppImpl")public class WebApp implements IClassSpecificNameResolver {

public String resolve(IObject object){String result = null;try {

IObject appName = (IObject)object.resolveValue("applicationName");IObject contextPath = (IObject)object.resolveValue("contextPath");if (appName != null && contextPath != null) {

result = appName.getClassSpecificName() + " at " +contextPath.getClassSpecificName();

}} catch (SnapshotException e) {

e.printStackTrace();}return result;

}}

NameResolvers “annotate” objects with a line of text:

Select object type to “annotate”

Return String with text to annotate

Lookup field value

Get the String name of the object field


Extending Memory Analyzer and IDDEimport org.eclipse.mat.SnapshotException;

import org.eclipse.mat.snapshot.extension.IClassSpecificNameResolver;import org.eclipse.mat.snapshot.extension.Subject;import org.eclipse.mat.snapshot.model.IObject;

@Name("Hung Thread Stacks")@Category(WASHelper.WAS_CATEGORY + WASHelper.HUNGTHREADS_SUBCATEGORY)@Help("List any threads that may be hung.\n\n")public class HungThreadsList extends BasePlugin {

public IResult execute(IProgressListener listener) throws Exception {int[] hungThreadIds = HungThreads.getHungThreadIds(snapshot);return new HungThreadsListOutgoingReferencesTree(

hungThreadIds, listener);}

Set name of query in menu

Set category in menu

Return “IResult” report

Find the Hung Threads


Extending Memory Analyzer and IDDEpublic static int[] getHungThreadIds(ISnapshot snapshot)

throws SnapshotException {int[] threadIds = findObjects(snapshot, "java.lang.Thread");List<Integer> hungThreads = new ArrayList<Integer>();for (int thr : threadIds) {

IObject thrObj = snapshot.getObject(thr);if (isThreadHung(thrObj)) {

hungThreads.add(thr);}

}int[] hungThreadObjectIds = getArrayIntegers(hungThreads);return hungThreadObjectIds;

}

Find all Threads

Check if the Thread is hung

Return the hung threads

public static boolean isThreadHung(IObject thread) throws SnapshotException {boolean result = false;boolean isHung = MATHelper.resolveValueBool(thread, "isHung");if (isHung != null && isHung.booleanValue()) {

result = true;}return result;

}

Get isHung field

Check value of field

Return field value


Extending Memory Analyzer and IDDEpublic class HungThreadsListOutgoingReferencesTree extends OutgoingReferencesTree {

private static final int sortIndex = COLUMN_INDEX_MAX_BUILTIN + 2;

public HungThreadsListOutgoingReferencesTree(ISnapshot snapshot, int[] roots, IProgressListener listener) throws SnapshotException {

super(snapshot, roots, listener);}

Customized OutgoingReferencesTree

• Hung Threads and references:


IBM Produced Extensions


IBM Extensions for Memory Analyzer

• Brings IBM product domain knowledge to Memory Analyzer and IDDE− WebSphere Application Server− WebSphere eXtreme Scale− WebSphere ESB− WebSphere Portal Server− CICS Transaction Gateway

...plus others to follow


WebSphere AppServer Extensions for Memory Analyzer

• Enable visualization and analysis of the Application Server and deployed applications

• Shows configuration, status and detailed content information:• Software install levels and startup time• Installed applications and status• Thread pool configuration and utilization• HTTP session cache size and session content• Application Class Loader leak detection


WebSphere AppServer Extensions for Memory AnalyzerApp Server Overview• Installed Levels• Server, Node and Cell names• Installed applications and status• Breakdown of memory usage by component

Thread Pool Analysis• Thread Pool configuration• Current size of threads in pools• Hung Thread detection

Web Application Analysis• Application Status• Deployed Context Root and URL• Number of user sessionsPlus many more:• HTTP Session Contents, EJB Container, Class Loader leak

analysis etc...


WebSphere ESB Extensions for Memory Analyzer

• Extends Memory Analyzer with product specifics for WebSphere ESB− Augments existing Memory Analyzer data structures− Provides detailed product specific reports

• Breakdown of installed ESB applications for improved understanding of the environment

− Improved efficiency and effectiveness of problem determination and resolution

− WebSphere ESB L3 able to improve time to resolution of memory related PMRs

− Customers have greater understanding of memory usage and run time performance

• Design recommendations report identifies inefficient mediation flow logic patterns and recommends alternatives for improved performance


WebSphere ESB Extensions for Memory Analyzer

Mediation Flows• List of Mediation Flow Components

− Number of instances and memory usage• Breakdown of Mediation Flow Component

− Mediation flow logic, display names, etc.

Application Overview• Breakdown of each WebSphere ESB

application− Name, parsing mode, dependencies,

endpoints− Binding and component information, etc.

Plus a number of other reports:• WebSphere ESB Overview, Mediation

Primitives, Design Recommendations and Statistics


CICS TG Extensions for Memory Analyzer

Improving First Failure Data Capture (FFDC)• CICS customers comfortable with taking dumps• Aim to provide similar function to IPCS• Gain useful information about state of the product


CICS TG Extensions for Memory Analyzer

Available reports• Current Workload• Statistics• Configuration• Connection details

Aims:• Get as much information as possible• Avoid recreates and trace• Allow support to fully investigate


Rational Application Developer 9.0 and Health Center

• In addition to supporting trace-based profiling, Rational Application Developer v9.0 Beta now supports sample-based profiling for diagnosing execution time problems.

− Provided by Health Center Agent, the same as in IBM Support Assistant.

− Using the Health Center API, introduced in Health Center v2.1− Very low overhead compare with trace-based profiling. − Available on all supported profiling launch types, including profiling

on WAS servers.









Fully qualified method name

Number of samples taken while a particular method was being run at the top of the stack

The % of samples taken while a particular method was being run at the top of the stack

The % of samples taken while a particular method was anywhere in the call stack


Summary

• IBM is building a set of integrated monitoring and diagnostic tools

• Taking a ground up approach

• Working across teams, products and brands

• Providing a consistent user experience


Questions?


Summary

• IBM is building a set of integrated monitoring and diagnostic tools

• Taking a ground up approach

• Working across teams, products and brands

• Providing a consistent user experience


Other Sessions

• TAW-2713: Roundtable: WAS Serviceability and Problem Determination Tools− Wednesday 13:00 and Thursday 13:00, Zeno 4703

• TAD-2709: Roundtable: IBM Support Assistant: Future Strategies for PD− Wednesday 10:15 and Thursday 14:15, Zeno 4702

• TAW-1217: Lab: Diagnostic and Performance Tools for WAS− Thursday 13:00, Murano 3301B

• AVP-3225: Lab: Java Troubleshooting with ISA 5 and TMDA− Wednesday 10:15, Zeno 4601

• ICF-4622: Roadmap for WebSphere Liberty Performance Monitoring− Thursday 15:45, Titian 2303


References

• IBM Monitoring and Diagnostic Tools for Java™− http://www.ibm.com/developerworks/java/jdk/tools/

• IBM Extensions for Memory Analyzer− http://www.alphaworks.ibm.com/tech/iema

• Debugging WebSphere Application Server from Dumps (MA)− http://www.ibm.com/developerworks/websphere/techjournal/1103_

supauth/1103_supauth.html• IBM Support Assistant (ISA)

− http://www.ibm.com/software/support/isa• Email: [email protected]

http://www.ibm.com/developerworks/java/jdk/tools/

http://www.alphaworks.ibm.com/tech/iema

http://www.ibm.com/developerworks/websphere/techjournal/1103_supauth/1103_supauth.html

http://www.ibm.com/developerworks/websphere/techjournal/1103_supauth/1103_supauth.html

http://www.ibm.com/software/support/isa



We love your Feedback!

Don’t forget to submit your Impact session and speaker feedback! • Your feedback is very important to us – we use it to improve next year’s

conference• Go to the Impact 2013 SmartSite (http://impactsmartsite/com):

‒ Use the session ID number to locate the session‒ Click the “Take Survey” link‒ Submit your feedback

http://impactsmartsite/com

http://impactsmartsite/com



Legal Disclaimer

• © IBM Corporation 2013. All Rights Reserved.• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained

in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server). Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:UNIX is a registered trademark of The Open Group in the United States and other countries.

• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.

• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.

http://www.ibm.com/legal/copytrade.shtml

http://www.ibm.com/legal/copytrade.shtml