WebSphere Technical University: Introduction to the Java Diagnostic Tools

A5: Introduction to the IBM Monitoring and Diagnostic Tools

Chris BaileySTSM, IBM Runtime Monitoring and Diagnostics

© 2014 IBM Corporation

Please Note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

2

3

Introduction to the Speaker

Chris BaileyIBM Runtime Monitoring and Diagnostics Architect 13 years experience developing and deploying Java SDKs 1 years experience developing and deploying Node.js SDKs

Recent work focus:‒ Java and Node.js monitoring and diagnostics‒ Java and Node.js integration into the cloud‒ Highly resilient and scalable deployments‒ Requirements gathering

Contact Details:‒ [email protected]‒ http://www.linkedin.com/in/chrisbaileyibm‒ http://www.slideshare.net/cnbailey/‒ @Chris__Bailey

mailto:[email protected]

http://www.linkedin.com/in/chrisbaileyibm

http://www.slideshare.net/cnbailey/

4

What I hope to achieve in the next 60 mins

Introduce the IBM tools for monitoring, debugging and diagnosing problems

Understand the capabilities of each tool

See how the tools can be extended tools with knowledge of applications or to build custom tools

See how IBM is building on top of the Java tools platform

Java Tools Overview

6

● A free unified suite of tools to understand different aspects of applications

● Provides more than visualizations – also provides recommendations

● IBM supported and always current to latest IBM runtime levels

● Provides APIs to allow you to extend or create your own custom tools

IBM Monitoring and Diagnostic Tools for Java

7

Where to use our tools

The IBM Monitoring and Diagnostic Tools fall into two groups

Reactive (Post-Mortem):

Information created at runtime and analyzed offline:– Garbage Collection Memory Visualiser (GCMV)

– Memory Analyzer (MA)

– Interactive Diagnostic Dump Explorer (IDDE)

Proactive (Runtime):

Information created at runtime and analyzed immediately:– Health Center

8

How to get the tools: IBM Support Assistant (ISA)

A workbench for use by your technical personnel to:

Expedite resolution of problems

Automate collection of symptom based “MustGather” data

View and analyze collected data

Leverage powerful tools to analyze logs, JVM core and heap dumps, config files, etc.

9

How to get the tools: Eclipse Marketplace

Reactive Tools (Post-Mortem)

11

Analyses and visualizes:● Process and system memory usage● Java heap memory usage and performance

Compares multiple logs in the same plotsand reports

Many data views Reports Graphs Tables

Garbage Collector and Memory Visualizer (GCMV)

12

Process and system memory usage:


#!/bin/shPID=$1INTERVAL=3

# Echo time at start of monitoring.echo timestamp = `date +%s`

# Echo the interval frequency.echo "ps interval = $INTERVAL"

# Run the system command at intervals.while ([ -d /proc/$PID ]) do ps -p $PID -o pid,vsz,rss sleep $INTERVALdone

13







14


Java heap memory usage and performance -verbose:gc

-Xverbosegclog:[DIR_PATH][FILE_NAME] -Xverbosegclog:[DIR_PATH][FILE_NAME],X,Y

where: X is the number of files to Y is the number of GC cycles a file should contain






15

Benchmark: DayTrader with 50 clients:WebSphere 8.5.5 and IBM Java 7 SR7 on POWER 7

4 CPU machine, running at 100% utilization

Performance Overhead of Verbose:GC

Throughput Runs Min Max Stddev

Without verbose:gc 3743.953 8 3637.019 3855.971 71.6304964862

0

500

1000

1500

2000

2500

3000

3500

4000

16






With verbose:gc 3748.779 8 3654.9945 3877.9495 83.5641336842

0

500

1000

1500

2000

2500

3000

3500

4000

17






With verbose:gc 3748.779 8 3654.9945 3877.9495 83.5641336842

0

500

1000

1500

2000

2500

3000

3500

4000

0.1289% ?

18

Provides deep insights into Java Heaps

Powerful analytic capabilities to explore heap sections, including “deep” and “shallow” cost

Path to GC Roots

Provides the reference chain that prevents an object being garbage collected

Eclipse Equinox bundle explorer (works with OSGI bundles for WebSphere)

Memory Analyzer (based on Eclipse MAT)


0 2000 4000 6000 8000 10000 12000 14000 16000

0

50

100

150

200

250

300

350

Linux 64 bit PPC

Same Disk

RAM Disk

GB Netw ork

Core Size (MB)

Tim

e to

Ge

ne

rate

(s

)

Processes PHD Heapdumps, HPROF Heapdumps and OS System Dumps

System dump writing speed is relatively quick:~10s per GB providing physical memory is availableOtherwise dictated by disk write speedCan be minimized by good configuration of system

System dumps compress very well:Usually to 10% of original size

Memory Analyzer Loads Data from Dumps

20

Memory Analyzer overview cont…

Overview:Overview of the heapdump including size and total number of objects.Provides links to continued analysis

Path to GC Roots:Provides the reference chain that prevents an object being garbage collected.

Dominator Tree grouped by Class Loader:Lists the biggest objects using a “keep alive tree”.

Grouping by ClassLoader limits the analysis to a single application in a

JEE environment

21

Additional capabilities for debugging generic Java Applications

Extensions currently available for IBM Java SE runtime WebSphere Application Server CICS Transaction Gateway

Using these extensions will enable Visualization of application and IBM product code Confirm configuration of IBM Products Inspect size and contents of IBM products

Extensions are made available through ISA or installed directly into the Memory Analyser application

Extensions for Memory Analyzer

22

Interactive Diagnostic Data Explorer (IDDE)

The IBM JRE Dump artefacts provide insight into the health and integrity of your application by providing a graphical view of a system dump file

Provides access to detailed information contained dumps

Allows collaboration and user defined extensions

Community of extensions and add-ons

Ability to run Memory Analyzer Extensions

Also supports Node.js

Proactive Tools (Runtime)

24

Answers vital questions: What is my JVM doing? Is everything ok? Why is my application running slowly? Why

is it not scaling? Am I using the right options?

Live monitoring with very low overhead

Diagnose potential problems, with recommendations

Works at the JVM level, no domain-specific (e.g. J2EE) information

Suitable for all Java applications

Special view for WebSphere Real Time Helps identify outlier events (class load,

compilation, GC, application method)

IBM Monitoring and Diagnostic Tools for Java™ Health Center

25

Health Center enabled using the command line:

Health Center enabled at runtime using “Late Attach” to a running process: java Heartbeatjava -jar healthcenter.jar

Enabling Health Center

java -Xhealthcenter Heartbeat

java -jar healthcenter.jar -PID=<pid>

26

Health Center Data: Operating System and Machine

Native Memory• Detect native memory leaks in application• Determine if external forces are using more memory• View components using the most native memory

Environment reporting• Detects invalid Java options• Detects options which may hurt performance or serviceability• Useful for remote diagnosis of configuration-related problems

Machine and Process CPU Usage • Visualizes heap usage and gc pause times over time• Identifies memory leaks• Suggests command-line and tuning parameters• Same recommendation logic as GCMV

27

Health Center Data: Java Runtime

Garbage Collection visualization Visualizes heap usage and gc pause times over time Identifies memory leaks Suggests command-line and tuning parameters Same recommendation logic as GCMV

Object Allocation Profiling Understand what types of data is being allocated and

at what rate Determine which code is allocating data

Threads view List of current threads and states Number of threads over time See contended monitors

28

Health Center Data: Java Runtime

Class loading visualization Shows all loaded classes Shows load time Identifies shared classes Live class histogram information

I/O Monitor application file open/close events as they

occur Lists currently open files

29

Health Center Data: Application

Method Profiling Always-on profiling shows application activity Identifies the hottest methods in an application Full call stacks to identify where methods are being

called from and what methods they call No byte code instrumentation, no recompiling

Java Lock Profiling Always-on lock monitoring Quickly allows the usage of all locks to be profiled Helps to identify points of contention in the

application that are preventing scaling

Live control of application Trigger dumps Enable verbosegc collection

30

Measured using WebSphere App Server and the DayTrader benchmark with 50 clients

Running WAS 8.5.5, IBM Java 7 SR5, AIX 7.1, POWER7

Throughput determined by number of completed transactions on 4 saturated CPUs

Health Center overhead

80

85

90

95

100

105

100

Baseline

Network Client

Headless

Headless with ZIP

31





80

85

90

95

100

105

100.4

Baseline

Network Client

Headless

Headless with ZIP

32





80

85

90

95

100

105

99.01

Baseline

Network Client

Headless

Headless with ZIP

33





80

85

90

95

100

105

96.99

Baseline

Network Client

Headless

Headless with ZIP

The 2.2 release of Health Center provides an API ● Allows developers to access Health Center data in their applications

// Create the connection object:

ConnectionProperties conn1 = new ConnectionProperties("localhost", 1973);

// Connect to the Health Center agent, using the previous connection

// settings:

HealthCenter hcObject = HealthCenterFactory.connect(conn1, true);

// Get garbage collection data and print:

GCData gcData = hcObject.getGCData();

System.out.println("GC Mode is " + gcData.getGCMode().toString());

Access Health Center data with the API

RAD v9 Usage of Health Center Data

Enable “Health Center Agent”

Select “Sample based profiling”

Method CPU usage collected

Displayed in real time

Accessed using API

IBM Support Assistant 5

37

ISA 5.0 Team Server

Server-based model

Install once - shared by many team members via browser

Web 2.0 browser interface

Remote execution of PD tools

Off-load analysis processing

Collaboration on PD

Case Management

Tool Management

Single-user option available

Available at www.ibm.com/software/support/isa

BrowserBrowser

BrowserBrowser

ISATeam Server

IBM Support Assistant (ISA) 5.0

© 2014 IBM Corporation 38

Automation of Upload of Data to ISA 5.0

■ ISA Provide a REST interface for some of its functions:

■ Case creation:curl --user user1:user1password -X POST -H Content-Type:application/x-www-form-urlencoded -v -o /home/dumps/curl.log --url http://myISA5TeamServer.mydomain.net/rest/1/tickets -d "summary={case summary}&description={case description}"

■ File upload:curl --user user1:user1password -X POST -v -k -o /home/dumps/curl.log -F file=@'{}' --url http://myISA5TeamServer.mydomain.net/rest/1/files/${caseno}/file_upload?path=/${caseno}/

■ Can be combined with IBM JDKs “-Xdump” options to run automatically:-Xdump:tool:events=systhrow,filter=java/lang/OutOfMemoryError,exec="dumpUpload.sh %pid”

● Where dumpUpload.sh contains:

find . -name '*$1*' -exec curl -X POST -v -k -o curl.log -F file=@'{}' --url https://myISAServer.mydomain.net:10943/rest/1/files/0001/file_upload?path=/0001/ ;

Demo Time

40

Summary

IBM is building a set of integrated monitoring and diagnostic tools

Taking a ground up approach

Working across teams, products and brands

Providing a consistent user experience

What Next??

41

Questions?

Your feedback is valuable - please complete your session or lab evaluation!

Session number[A5]

Provide your evaluations by:

Evaluation forms:

Fill out a form at the end of each session

Paper forms are located in each of the session or lab rooms

Place the completed form in the tray as you exit the room- Or –

Complete the session survey on Event Connect Portal: ibmeventconnect.eu/euxdx

Select Sessions, then Session Finder, and complete the survey

43

Where to find more information

IBM Monitoring and Diagnostic Tools for Java™ on developerWorkshttp://www.ibm.com/developerworks/java/jdk/tools/

IBM Support Assistant (ISA)

http://www.ibm.com/software/support/isa

Email [email protected]

http://www.ibm.com/developerworks/java/jdk/tools/



http://www.ibm.com/software/support/isa

Visit WASdev.net for downloads and other resources

WASdev.net

https://wasdev.net/


For Additional Information

IBM Traininghttp://www.ibm.com/training

IBM WebSpherehttp://www-01.ibm.com/software/be/websphere/

IBM developerWorkswww.ibm.com/developerworks/websphere/websphere2.html

WebSphere forums and communitywww.ibm.com/developerworks/websphere/community/

45

http://www.ibm.com/training

http://www-01.ibm.com/software/be/websphere/



http://www.ibm.com/developerworks/websphere/websphere2.html

http://www.ibm.com/developerworks/websphere/community/

http://www.ibm.com/developerworks/websphere/community/

Technology

WebSphere Technical University: Introduction to the Java Diagnostic Tools