Upload
damien-dallimore
View
236
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Presentation from Splunk Conf 14
Citation preview
Copyright © 2014 Splunk Inc.
Damien DallimoreDev Evangelist , CSO Office @ Splunk
Splunking the JVM
2
DisclaimerDuring the course of this presentation, we may make forward looking statements regarding future events or the
expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important
factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other
commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
3
From Middle Earth
Make Splunk Apps & Add-ons
JVM background
4
5
apps.splunk.com
github.com/damiendallimore
What is this JVM thing ?
7
JVM = Java Virtual Machine
First appeared in the early 90’s
Now the dominant runtime for enterprise applications
Application Servers Enterprise Service Buses Databases
NoSQL Distributed Big Data Web Servers
Directory Servers Search Engines Build Systems
Gaming Platforms Trading Systems Reservation Systems
Core Banking Messaging Infrastructure Proprietary Systems
8
9
Many JVM Variants
Oracle Hotspot
OpenJDK
Oracle JRockit
IBM J9
Azul Zing
More or less the same. OpenJDK is reference impl for Java 7/8
Getting merged into Hotspot
Commercial alternatives
10
JVMs are here to stay
It’s not just about the “J”
Big Data frameworks
You should care about getting insights into your JVM data
Getting at the data
12
Data Sources
LogsDevelopers
JMX (Java Management Extensions)Instrumentation Agents
Operating SystemDistributed Communications
Log Data
14
Standard Log Files
Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina loadINFO: Initialization processed in 1153 msOct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternalINFO: Starting service Catalina
Application logs that are part of the product
Developer logs for any code that was deployed
Written to local disk or network storage
15
Garbage Collector Logs
54.736: [Full GC 54.737: [Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K), [Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs]
Generated by way of arguments passed to the JVM at startup
-verbose:gc-Xloggc:/home/damien/jvm_logs/gc.log -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
16
SplunkJavaLogging
Sometimes you can’t write to file or deploy a UF
Appenders for Java Util Logging , Log4J , Logback
Simply add a logging appender to your logging configuration file
17
LogBack Appender Example
18
Code
19
Best practice semantic format
2012-08-07 15:54:06:644+1200 name="Failed Login" event_id=”12345" app="myapp" user="jane”
20
Better Exception Logging
21
Easier to work with in Splunk
22
Splunk SDK for Java
Use the SDK from any JVM Language , Java / Groovy / Scala etc….
Send log events via REST , UDP or TCP directly to Splunk from your code
Spring Integration Adaptors available on Github
SDK available from dev.splunk.com
23
Using Java SDK
24
Using Spring
JMX
26
JMX = Java Management Extensions
Monitor JVM via MBean attributes , operations and notifications
JVM MBeans
Vendor MBeans
Custom Coded MBeans
27
Browse MBeans with JConsole
28
Getting this data into Splunk
Runs on all supported Splunk platforms
Works with all main JVM variants
100% Free and Open Source
29
Simple to Configure
30
Many Connectivity Options
Let’s look at this Splunk App in action
31
This is great for monitoring a single JVM……
32
But what if we have dozens , hundreds , even thousands ?
33
Multiple jmxserver elements per configuration file
34
Multiple configuration files per Splunk Instance
35
But stacking vertically will only get you so far
36
Go horizontal to achieve real scale
Indexer Cluster
UFs with JMX Mod Input
Monitored JVMs
37
In my experience , the data sources shown so far will be thorough enough for most of your JVM monitoring use cases.
But what if you want to go deeper ?
Instrumentation Agents
39
SplunkJavaAgent
Dynamic Byte Code Injection Agent for JVMs
No need to change any code to get APM level insights
This is all you pass to the JVM at startup :
-javaagent:splunkagent.jar
40
What raw metrics can you get ?Bytecode Injection
• class loading• method execution• method timings • method call stack • caught & uncaught exceptions
JMX • running in local in-memory mode
Binary HPROF Memory dumps decoded in text• no more static hprof dumps and then loading into another Mem Analysis tool
41
Simple Configuration File
42
43
Raw events streamed into Splunk
44
Search with Splunk
45
Reveal APM insights
Operating System
47
The JVM executes as a process
Let’s see what the host Operating System can tell us : top
External programs that provide additional JVM insights : jstat
Index this data in Splunk and correlate
48
top
49
50
jstat
51
Distributed Communications
53
Valuable data flows in & out of JVMs
Messaging
Capture packets off the wire
Build a Splunk App !
55
Opportunity
Reuse some of the data collection approaches in this presentation and build out a specific JVM app
Publish it on apps.splunk.com
Make some moneyGenerate publicity for yourself / your companyGenerate sales leadsSell more Splunk licenses if you are also a partnerDo it because you love community collaborationBecome my beer buddy for life
56
Some JVM App Ideas
Splunk for JBossSplunk for Tomcat
Splunk for SolrSplunk for Cassandra
Splunk for “Apache Project”
I am here to help you
THANK YOU@[email protected]
Questions ?