Upload
damien-dallimore
View
113
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Presentation section from Splunk Live content
Citation preview
Splunking the JVM
Damien DallimoreDeveloper Evangelist
Copyright © 2013, Splunk Inc.
What is this JVM thing ?
2
• Circa 1991, Dr. James Gosling at Sun started developing a technology for next generation smart devices/appliances
• “Green” became “Oak” which became “Java”• Java 1.0 first appeared in January 1996.• The JVM is a virtual machine that runs programs that are compiled into
Java bytecode• Available for many hardware and software platforms• 17 years later , the JVM has evolved from a consumer device
technology, to a browser oriented technology with the explosion of the web , to now becoming deeply rooted in the enterprise software landscape on the server side and in the cloud
Copyright © 2013, Splunk Inc.
17 years later
3
• Oracle took ownership of Java from Sun in January 2010
• The Java Community Process(JCP) is the forum where members develop specifications for Java technology
• Java Specification Requests(JSR) get submitted for new features, are reviewed and then voted on by the JCP Executive committee.
• Editions• Embedded Java, Java ME , Java SE , Java EE
• Current Version is Java 7 (Dolphin)• Java 8 scheduled for 2013
Application Servers Enterprise Service Buses Databases
NoSQL Distributed Big Data Web Servers
Directory Servers Search Engines Build Systems
Gaming Platforms Trading Systems Reservation Systems
Core Banking Messaging Infrastructure Proprietary Systems
Copyright © 2013, Splunk Inc.
JVM Variants
4
• Oracle Hotspot (formerly SUN)– the primary reference JVM implementation
• Oracle JRockit (formerly BEA)– free since May 2011– code base currently being merged with Hotspot, ETA ~JDK 8
• Open JDK– SUN open sourced Hotspot and the Java class library in 2006– Slight differences with Oracle Java still– OpenJDK is the official Java SE7 Reference Implementation
• J9– IBM’s JVM for AIX, Linux, MVS, OS/400, Pocket PC, z/OS
• Azul Systems Zing– based on HotSpot – supports memory heaps up to 512 GB without GC pauses and is able to grow and shrink the heap based on load
Copyright © 2013, Splunk Inc.
The JVM has a healthy future
5
• Hotspot / JRockit code merge creating a best of breed JVM, Oracle to contribute this to OpenJDK
• OpenJDK is thriving, Oracle are contributing and being good stewards of Java (despite initial skepticism)
• Proliferation of alternative JVM languages that can all co-habitate in the JVM and new features in Java 8 to further enhance this multi language platform– Scala– Groovy– Clojure
• The JVM is evolving organically with the shifting tides of Enterprise software, it isn’t about the “J” anymore.
• From the clustered Application Server domination of the 00’s we now see an explosion of Big Data products running in massively distributed environments on commodity hardware or in the cloud– Apache Hadoop family (MapReduce, Hive, Hbase, Cassandra, HDFS)
Copyright © 2013, Splunk Inc.
What is running in JVMs ?
6
Copyright © 2013, Splunk Inc.
JVM “Fanboi”
7
Dr. GoslingFanboiSpeaking of Java as a language as opposed to the JVM platform, James Gosling, the Father of Java, said "Most people talk about Java the language, and this may sound odd coming from me, but I could hardly care less." He went on to explain, "What I really care about is the Java Virtual Machine as a concept, because that is the thing that ties it all together."
Copyright © 2013, Splunk Inc.
JVM Machine Data
8
• The JVM footprint cross cuts the data centre and represents a massive source of valuable machine data• Large scale Application/Web Server clusters• Hadoop & Cassandra Node topologies in the 10’000s !!!
Custom Developed Code
WAR file
Application CodeTomcat
JVMHotspot
Operating SystemLinux
JMX, SNMP, HPROF,GC Logs, Custom Agents(AppDynamics/SplunkJavaAgent)
JMX, Application Logs
JMX, Developer Logs, Splunk Java SDK, SplunkJavaLogging
JVM process OS resource metrics
CORR
ELAT
E
Copyright © 2013, Splunk Inc.
Application & Developer Logs
9
• Application logs• default logs that are part of the product
• Developer logs• any custom code created and deployed
to the application that has it’s own logging
• Written to local disk or a mounted network volume
• Monitor with a Splunk UF
Copyright © 2013, Splunk Inc.
Splunk Java SDK / SplunkJavaLogging
10
• Alternative to writing to log file or needing to deploy a Splunk Universal Forwarder
• Use the Splunk Java SDK to input events directly to Splunk via HTTP Rest.
• Use SplunkJavaLogging to input events directly to Splunk using custom logging appenders.
Copyright © 2013, Splunk Inc.
JVM Process OS Metrics
11
• By JVM Process ID : Process State, Memory, CPU, Disk Usage, Disk I/O, Network I/O, File Descriptor Usage.
• Some OS metrics also exposed via JMX• Splunk for Unix and Linux• Splunk for Windows• Correlate this OS data across your JVM and
Application events ie: your JVM may have hung because of CPU starvation caused by some other process thrashing
Copyright © 2013, Splunk Inc.
Garbage Collection logs
12
• Extended Hotspot JVM options
-verbose:gc-Xloggc:/home/damien/jvm_logs/gc.log -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
• The log is written to at Garbage Collection time• Useful for tracing full GC cycles• Need to perform field extractions in Splunk• Many GC metrics also available via JMX
54.736: [Full GC 54.737: [Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K), [Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs]
Copyright © 2013, Splunk Inc.
Custom Instrumentation Agents (Advanced)
13
• JVM BCI (byte code instrumentation)• Write custom agents that get injected into
the running JVM• Dynamically inspect the state of
applications running in the JVM• Profiling, debugging, monitoring,
thread/memory analysis• As you write the agent code , the data
output can be file based or over the network
• Check out my SplunkJavaAgent on github• Also AppDynamics have some pretty cool
kung fu in this area, we integrate !
Copyright © 2013, Splunk Inc.
HPROF Profiling Dumps
14
• Binary JVM dumps that allow for deeper JVM resource inspection
• Typical use case is diagnosing memory issues after JVM crashes with java.lang.OutOfMemoryError
• Binary file is usually batch loaded into a third party memory analysis tool like Eclipse MAT
• SplunkJavaAgent can dynamically dump and decode hprof output and send to Splunk
• Awesome source of information for dev/test
Warning : heap dumping is an expensive operation as a full GC gets performed
Copyright © 2013, Splunk Inc.
SNMP
15
• The JVM SNMP Agent provides a single MIB that exposes the JVM’s Management and Monitoring API
http://docs.oracle.com/javase/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib
• Setup the JVM (just the basic settings shown)
Open a UDP Port : -Dcom.sun.management.snmp.port=9004 Configure the ACL : $JAVA_HOME/jre/lib/management/snmp.acl
• Traps can be caught locally to file and monitored• Splunk SNMP Modular Input can poll the JVM SNMP Objects
(coming soon to a theatre near you)
Copyright © 2013, Splunk Inc.
JMX (Java Management Extensions)
16
• Manage and Monitor the JVM and Application via exposed MBeans• JVM MBeans (java.lang domain)
• Vendor MBeans (most vendors ship their products with extensive MBean coverage)
• Custom Coded MBeans (whatever your devs wish to code)
• MBeans expose attributes, operations and notifications to give you a powerfully dynamic insight into the runtime state of the JVM and your application.
• Add Splunk to the mix for historical and realtime operational visibility, pro-active issue detection etc..
• Splunk for JMX app on SplunkBase
Copyright © 2013, Splunk Inc.
JMX vs SNMP
17
JMX• Open and easily extensible• Developers can simply create new MBeans• Vendor products(JBoss, Cassandra, Hadoop etc..) ship with thorough MBean coverage, not MIBs
SNMP• The built-in SNMP agent of the JVM is not extensible. • You will not be able to use it in order to expose your own custom MIB• If you do want to expose your own MIB, you’d have to create a custom agent
Copyright © 2013, Splunk Inc.
Putting it all together, JVM Splunking Nirvana
18
Copyright © 2013, Splunk Inc.19
Contact me
Email : [email protected] : @damiendallimoreSkype : damien.dallimoreGithub : damiendallimoreSplunkbase : damiendSlideshare : http://www.slideshare.net/damiendallimore Blogs : http://blogs.splunk.com/devWeb : http://dev.splunk.com