JavaOne2013: Build Your Own Runtime Monitoring for the IBM JDK with the Health Center API

© 2013 IBM Corporation

Chris Bailey – IBM Java Service Architect

26th September 2013

Build your own MonitoringWith the Java Remote Monitoring API

Document number

© 2013 IBM Corporation

Important Disclaimers

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.

WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES.

ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.

IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.

IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS

2

© 2013 IBM Corporation3

Introduction to the speaker 13 years experience developing and deploying Java SDKs

Recent work focus:– Java applications in the cloud– Java usability and quality– Debugging tools and capabilities– Requirements gathering– Highly resilient and scalable deployments

My contact information:– [email protected]– http://www.linkedin.com/in/chrisbaileyibm– http://www.slideshare.net/cnbailey/

mailto:[email protected]

http://www.linkedin.com/in/chrisbaileyibm

http://www.slideshare.net/cnbailey/


Goals of this talk

Introduce the IBM tools for monitoring, debugging and diagnosing problems in the deployment life cycle

Understand the capabilities of Health Center

Learn how to build your own tools using the Health Center data feed

See how IBM is building on top of the Java tools platform


IBM Tools for Java


IBM Monitoring and Diagnostic Tools for Java

A free suite of tools for understanding different aspects of Java applications– Garbage Collection and Memory Visualizer (GCMV)– Memory Analyzer (and IBM Extensions for Memory Analyzer)– Interactive Diagnostic Data Explorer (IDDE)– Health Center

Provide visualization, analysis and recommendations

Available in the IBM Support Assistant– Provides the “toolbox” in which analysis tools reside– Assist with opening PMRs and working with IBM Support– Search feature to query IBM and non-IBM knowledge banks


Garbage Collection and Memory Visualizer Graphical Display of Data

– Allows graphing of all available data: pause times, heap size etc

– Allows zoom, cropping and change of axes value and units

– Allows comparison of multiple files

Analysis and Recommendations– Provides tuning recommendations based on

data and flags errors.– Analysis can be limited using cropping.– Values and units used in analysis can be

changed by changing axes values and units


Memory Analyzer Overview:

– Overview of the dump including Java heap size and total number of objects.

– Provides links to continued analysis

Path to GC Roots:– Provides the reference chain that prevents

an object being garbage collected.

Dominator Tree grouped by Class Loader:– Lists the biggest objects using a “keep alive

tree”. Grouping by Class– Loader limits the analysis to a single

application in a JEE environment

Provides extension API– Used to provide product based extensions


Interactive Diagnostic Data Explorer

Multi-platform system dump analyzer– AIX, Linux, i/OS, Windows, z/OS

Provides dump portability– Analyse dump on any platform

Has knowledge of data structures in dump:– Java, JVM, and some OS structures

Provides collaborative analysis– Store progress and notes with the dump

Provides extension API– Used to provide automated analysis


Health Center

Live monitoring tool with very low overhead– Pre-installed in IBM JDKs since Java 5– Enable at startup: -Xhealthcenter– Enable at runtime: java -jar healthcenter.jar ID=<pid>

Understand how your application is behaving– Monitor Class loading, File I/O, Environment settings, Garbage Collection, Allocations,

Method Profiling, Locking, Native memory use

Diagnose potential problems, with recommendations

Suitable for all Java applications


Health Center Environment reporting

– Detects invalid Java options– Detects options which may hurt performance or

serviceability– Useful for remote diagnosis of configuration-related

problems

Class loading visualization– Shows all loaded classes– Shows load time– Identifies shared classes

File I/O– Monitor application file open/close events as they occur– Lists currently open files


Health Center Garbage Collection visualization

– Visualizes heap usage and gc pause times over time– Identifies memory leaks– Suggests command-line and tuning parameters– Java Object allocation profiling by allocating code or type

Native Memory– Detect native memory leaks in application– Determine if external forces are using more memory– Memory counters showing which parts of the JVM are

using the most native memory

Threads view– List of current threads and states– Number of threads over time– See contended monitors


Health Center Method Profiling

– Always-on profiling shows application activity– Identifies the hottest methods in an application– Full call stacks to identify where methods are being called

from and what methods they call– No byte code instrumentation, no recompiling

Java Lock Profiling– Always-on lock monitoring– Quickly allows the usage of all locks to be profiled– Helps to identify points of contention in the application

that are preventing scaling

Live control of application– Trigger dumps– Enable verbosegc collection


Health Center Demo


Health Center: Overhead

Measured using WebSphere App Server and the DayTrader benchmark with 50 clients– Running WAS 8.5.5, IBM Java 7 SR5, AIX 7.1, POWER7– Throughput determined by number of completed transactions on 4 saturated CPUs

94

95

96

97

98

99

100

101

102

103

100

Application Throughput(percentage of baseline)

BaselineDormant (level=off)Network ClientHeadless client (level=headless)Headless client with ZIP (level=headless)




94

95

96

97

98

99

100

101

102

103

100

101.76






94

95

96

97

98

99

100

101

102

103

100

101.76

100.4






94

95

96

97

98

99

100

101

102

103

100

101.76

100.4

99.01






94

95

96

97

98

99

100

101

102

103

100

101.76

100.4

99.01

96.99




Health Center

Low overhead is achieved by surfacing existing data used by the JVM internally– Performance overhead is therefore cost of storage or IO

Garbage Collection Data:– GC collects metrics to determine how to self tune

• Heap occupancy: expansion or contraction of the heap• GC durations: expansion/contraction, concurrent kick-off

Method Profiling:– JIT compiler collects metrics to determine:

• When to compile: high use methods compiled first• How to compile: optimize for most common flow through code• When to recompile: assumptions are now incorrect


Java Monitoring API


Health Center and the Java Monitoring API

Health Center obtains its data directly from the Java runtime and produces analysis and recommendations shown in the Health Center client

The Java runtime's data is now provided to any consumer via the Java Monitoring API– Allows any tool to receive, analyze and interpret the data– Allows those tools to additionally obtain the analysis and recommendations

IBM Java Runtime provides a monitoring platform!


Using the API

The API can be used in three modes:– Remote: connecting over a socket connection (with security/authentication etc)– Local: running inside the monitored application (using a local loop socket)– Offline: loading a pre-saved binary “.hcd” file

Performance cost to monitored application is the same as for Health Center– Running in “local” mode adds has additional impact due to the the cost of parsing data


Using the API

Three step process for obtaining data from the API:

Obtain a Health Center object– Using either a File or a Connection

Obtain a data type object from the Health Center object– getCpuData() / getGCData() / getThreadsData() / getIOData() / etc

Request data points from the data type object:– getCpuData().getProcessUse() / getCpuData.getSystemUse()


Obtain a Health Center Object: Fileimport com.ibm.java.diagnostics.healthcenter.api.ConnectionProperties;import com.ibm.java.diagnostics.healthcenter.api.HealthCenter;import com.ibm.java.diagnostics.healthcenter.api.HealthCenterException;import com.ibm.java.diagnostics.healthcenter.api.factory.HealthCenterFactory;public class GCGraphPrototype {

static HealthCenter hcAPI = null;public static void main(String[] args) {

new GCGraphPrototype();}GCGraphPrototype() {

try {hcAPI = HealthCenterFactory.connect(new File("C:\\healthcenter.hcd"));

} catch (HealthCenterException | FileNotFoundException e) {e.printStackTrace();

}testGCAPI();

}...

Use factory to create Health Center Object

Connect to a File provider


Obtain a Health Center Object: Connectionimport com.ibm.java.diagnostics.healthcenter.api.ConnectionProperties;import com.ibm.java.diagnostics.healthcenter.api.HealthCenter;import com.ibm.java.diagnostics.healthcenter.api.HealthCenterException;import com.ibm.java.diagnostics.healthcenter.api.factory.HealthCenterFactory;public class GCGraphPrototype {



ConnectionProperties props = new ConnectionProperties();

try {hcAPI = HealthCenterFactory.connect(props, true);

} catch (HealthCenterException e) {e.printStackTrace();

}testGCAPI();

}...

Describe your connection target: localhost:1972

Connect to your target


Obtain a Health Center Object: Connectionimport com.ibm.java.diagnostics.healthcenter.api.ConnectionProperties;import com.ibm.java.diagnostics.healthcenter.api.HealthCenter;import com.ibm.java.diagnostics.healthcenter.api.HealthCenterException;import com.ibm.java.diagnostics.healthcenter.api.factory.HealthCenterFactory;public class GCGraphPrototype {



ConnectionProperties props = new ConnectionProperties("192.168.1.105", 35535));

try {hcAPI = HealthCenterFactory.connect(props, true);

} catch (HealthCenterException e) {e.printStackTrace();

}testGCAPI();

}...

Describe a target of 192.168.1.105:35535


Obtain a Health Center Object: Connection Securityimport com.ibm.java.diagnostics.healthcenter.api.ConnectionProperties;import com.ibm.java.diagnostics.healthcenter.api.HealthCenter;import com.ibm.java.diagnostics.healthcenter.api.HealthCenterException;import com.ibm.java.diagnostics.healthcenter.api.factory.HealthCenterFactory;public class GCGraphPrototype {



ConnectionProperties props = new ConnectionProperties("192.168.1.105", 35535));props.setJmxUsername("Username");props.setJmxPassword("Password");try {

hcAPI = HealthCenterFactory.connect(props, true);} catch (HealthCenterException e) {

e.printStackTrace();}testGCAPI();

}...

Add JMX authentication


Obtain a Health Center Object: Connection Securityimport com.ibm.java.diagnostics.healthcenter.api.ConnectionProperties;import com.ibm.java.diagnostics.healthcenter.api.HealthCenter;import com.ibm.java.diagnostics.healthcenter.api.HealthCenterException;import com.ibm.java.diagnostics.healthcenter.api.factory.HealthCenterFactory;public class GCGraphPrototype {



ConnectionProperties props = new ConnectionProperties("192.168.1.105", 35535));props.setSSLTruststoreLocation("Location");props.setSSLTruststorePassword("Password");try {

hcAPI = HealthCenterFactory.connect(props, true);} catch (HealthCenterException e) {

e.printStackTrace();}testGCAPI();

}...

Add SSL encryption


testGCAPI():

private static void testGCAPI() {GCGraph demo = new GCGraph();demo.updateDataSet(hcAPI.getGCData().getGCPauseTimeData());demo.pack();demo.setVisible(true);while (true) {

demo.updateDataSet(hcAPI.getGCData().getGCPauseTimeData());demo.repaint();try {

Thread.sleep(2 * 1000);} catch (InterruptedException e) {

e.printStackTrace();}

}}

Get the GC Data provider

Get information on Pause Times


testGCAPI():

private static void testGCAPI() {GCGraph demo = new GCGraph();demo.updateDataSet(hcAPI.getGCData().getUsedHeapData());demo.pack();demo.setVisible(true);while (true) {

demo.updateDataSet(hcAPI.getGCData().getUsedHeapData());demo.repaint();try {



}}

Get Java Heap usage information


testGCAPI():

private static void testGCAPI() {GCGraph demo = new GCGraph();demo.updateDataSet(hcAPI.getGCData().getHeapSizeData());demo.pack();demo.setVisible(true);while (true) {

demo.updateDataSet(hcAPI.getGCData().getHeapSizeData());demo.repaint();try {



}}

Get Java Heap size information


testCPUAPI():

private static void testCPUAPI() {GCGraph demo = new GCGraph();demo.updateDataSet(hcAPI.getCpuData().getProcessUse());demo.pack();demo.setVisible(true);while (true) {

demo.updateDataSet(hcAPI.getCpuData().getProcessUse());demo.repaint();try {



}}

Get the CPU Data provider

Get information on JVM CPU usage


testCPUAPI():

private static void testCPUAPI() {GCGraph demo = new GCGraph();demo.updateDataSet(hcAPI.getCpuData().getSystemUse());demo.pack();demo.setVisible(true);while (true) {

demo.updateDataSet(hcAPI.getCpuData().getSystemUse());demo.repaint();try {



}}

Get information on overall CPU usage


GCGraph Class:

public class GCPauseGraph extends JFrame {private XYSeriesCollection gcDataSet;private static final long serialVersionUID = 1L;public GCGraph() {

super();gcDataSet = new XYSeriesCollection();JFreeChart lineChartObject = ChartFactory.createXYLineChart(

“gc pauses”, "time (seconds)", "pause time (ms)",getGCDataSet(), PlotOrientation.VERTICAL, true, true, false);

ChartPanel chartPanel = new ChartPanel(lineChartObject);chartPanel.setPreferredSize(new java.awt.Dimension(500, 270));setContentPane(chartPanel);

}

Extends Swing JFrame

Store data as X,Y values

Create a JFreeChart line plot

Add the JFreeChart to the JFrame


GCGraph Class:

public void updateDataSet(PauseData[] pD) {final XYSeries series1 = new XYSeries("Pause Times");if (pD != null) {

for (int i = 0; i < pD.length; i++) {series1.add(pD[i].getTime() / 1000, pD[i].getPauseTime());

}getGCDataSet().removeAllSeries();getGCDataSet().addSeries(series1);

}}

Create a JFree XYSeries

Populate with data

Remove old data

Add new data


Monitoring API Demo


Usage by IBM


Rational Application Developer 9.0

In addition to supporting trace-based profiling, Rational Application Developer v9.0 Beta now supports sample-based profiling for diagnosing execution time problems.

Using the Monitoring API, introduced in Health Center v2.1– Very low overhead compare with trace-based profiling. – Available on all supported profiling launch types, including profiling on WAS servers.










Summary

IBM is providing a very low overhead mechanism for accessing monitoring and profiling data

Allows building of your own tools

Same mechanism is being used by IBMs tools


http://ibm.co/JavaOne2013

http://ibm.co/JavaOne2013


References IBM Monitoring and Diagnostic Tools for Java™

– http://www.ibm.com/developerworks/java/jdk/tools/

IBM Extensions for Memory Analyzer– http://www.alphaworks.ibm.com/tech/iema

Debugging WebSphere Application Server from Dumps (MA)– http://www.ibm.com/developerworks/websphere/techjournal/1103_supauth/1103_supaut

h.html

IBM Support Assistant (ISA)– http://www.ibm.com/software/support/isa

Email: [email protected]

http://www.ibm.com/developerworks/java/jdk/tools/

http://www.alphaworks.ibm.com/tech/iema

http://www.ibm.com/developerworks/websphere/techjournal/1103_supauth/1103_supauth.html

http://www.ibm.com/developerworks/websphere/techjournal/1103_supauth/1103_supauth.html

http://www.ibm.com/software/support/isa

mailto:[email protected]


Copyright and Trademarks

© IBM Corporation 2013. All Rights Reserved.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., and registered in many jurisdictions worldwide.

Other product and service names might be trademarks of IBM or other companies.

A current list of IBM trademarks is available on the Web – see the IBM “Copyright and trademark information” page at URL: www.ibm.com/legal/copytrade.shtml

http://www.ibm.com/legal/copytrade.shtml

Technology

JavaOne2013: Build Your Own Runtime Monitoring for the IBM JDK with the Health Center API