Upload
angelo-rajadurai
View
1.006
Download
4
Embed Size (px)
DESCRIPTION
Citation preview
1
Thursday, September 23, 2010
<Insert Picture Here>
Oracle Performance Measurement and Tuning with Solaris DTraceJeff Savit Principal Sales Consultant
Angelo RajaduraiPrincipal Software Engineer
Thursday, September 23, 2010
3
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Thursday, September 23, 2010
4
<Insert Picture Here>
Agenda
• Basic Concepts of Performance Tuning• Performance Tuning Features in Solaris• DTrace - Dynamically Observing the Entire
Application Stack• Dynamic Tracing Oracle WebLogic Server with
DTrace• Further Reading
Thursday, September 23, 2010
5
Basic Concepts for Performance Management
Define Goals
Measure Performance
Analyze Model &Deduce
Take Action
PerformanceModel
Thursday, September 23, 2010
Start with the Best PlatformSolaris - Designed for Performance & Scale
6
Dynamic, multithreaded, preemptive kernel and process modelTightly integrated file system and virtual memory64-bit kernel, supporting both 32-bit and 64-bit applicationsResource managementOptimized, SMP-hot network stackCPU and memory scalability from low to high. Today, up to:
256 CPUs on T5440, 512 on M9000; 4TB RAM on M9000Optimized for modern multicore, multithread, NUMA systemsPremier x86 support optimized for Intel Nehalem and AMD OpteronSolaris Containers (Zones): negligible-overhead, built in virtualization
Thursday, September 23, 2010
• High system time is not a problem• You may be asking the system to do stuff
• Slow I/O is not a problem• You may be streaming bulk I/O
• 100% utilization is not a problem• You are using all the box you paid for :-)• Modern CPUs, NICs, disks, can do more than 1 thing at a
time without adding delay• Fine even if saturated unless some work unable to get
enough cycles to meet SLA
• None of these are problems unless they relate to business metrics
Define the Problem
7
Thursday, September 23, 2010
8
Native Apps
ruby
C C++
Database Oracle 11g
Operating System Solaris/OEL/Windows
Container WebLogic Server
Frameworks
Monitor: Right Location & Tools
PresentationJSP/Javascript/Ajax/JavaFX
python
php
perl
shell
Apps run in VM
Thursday, September 23, 2010
Solaris Tool chest
9
Proc tools
cputrack - per-processor hw counters pargs – process arguments pflags – process flags pcred – process credentialspldd – process's library dependencies plockstat – process lock statistics psig – process signal disposition pstack – process stack dumppmap – process memory map pfiles – open files and names prstat – process statistics ptree – process treeptime – process microstate times pwdx – process working directorypgrep – grep for processes pkill – kill processes list pstop – stop processes prun – start processes prctl – view/set process resources pwait – wait for processpreap – reap a zombie process
Tracing & Debugging
abitrace - trace ABI interfaces mdb – debug process/kernel
truss – trace function and system calls
System Statistics
acctcom – process accounting busstat – Bus hardware counters
cpustat – CPU hardware counters iostat – IO & NFS statistics
kstat – display kernel statistics mpstat – processor statistics
netstat – network statistics nfsstat – nfs server stats
sar – kitchen sink utility vmstat – virtual memory stats
Toolkits
DTraceToolkit – DTrace Tools K9Toolkit – perl perf tools
nicstat – network stats
Thursday, September 23, 2010
10
perfbarsee the big picture from across the room
Don't be shy about starting with a simple graphical tool!
http://blogs.sun.com/partnertech/resource/tools/perfbar.{sparc,i386}
See:LDOM Booth for use of perfbar
Thursday, September 23, 2010
11
FenxiReading a 1000 lines of text does not show you the big picture
Open source tool written at Sun for aggregating and displaying vmstat, mpstat, iostat data
https://fenxi.dev.java.net/
Thursday, September 23, 2010
12
So why do we need another tool?
Thursday, September 23, 2010
13
Native Apps
ruby
C C++
Database Oracle 11g
Operating System Solaris/OEL/Windows
Container WebLogic Server
Frameworks
Observing the Stack is hard
PresentationJSP/Javascript/Ajax/JavaFX
python
php
perl
shell
Apps run in VM
Thursday, September 23, 2010
The Blame Game
14
Devel
oper:
“I’d as
k the
datab
ase ad
min” Database Admin:
“May be the middleware guys
will know”
Sysadmin
“I don’t know! Ask the
Developer” Middlew
are Ex
pert:
“Lets
ask th
e Sysa
dmin”
Thursday, September 23, 2010
Static instrumentation is time consuming
15
Instrument Validate
Run in production
Collect infoProcess
Add Code Compile
Custom Instruments
Stop AppRestart w/ flags
Prebuilt Instruments
Thursday, September 23, 2010
Debugging Transient Problems In production. Impossible!
16
Please let me observe a live system.
Thursday, September 23, 2010
• turn instrumentation ON and OFF dynamically
• probe any arbitrary location• not just pre-defined probe points
• collect arbitrary data • have low probing overhead• have no overhead for disabled probes
If Only We Can ...
17
Thursday, September 23, 2010
Dynamic Tracing
18
Native
cc+ph
perl pyth
oru
b
shelDatabase
Operating System Solaris/Linux/OS X/
Container Glassfish/
Frameworks
Application Logic
PresentationJSP/Javascript/Ajax
12::SELECT id, name, symbol, conversion_rate FROM currencies WHEREstatus = 'Active' and deleted = 012::SELECT category, name, value FROM config12::SELECT id FROM outbound_email WHERE type = 'system'
-> foo()-> bar()<- bar()
<- foo()
Thursday, September 23, 2010
19
/ /predicate when
{
}
actions do what
D - Scripts
: : :provider module probe name whereprobe
Thursday, September 23, 2010
DTrace examples
20
nthreads{ printf("new thread created by %s(%d)\n", execname,pid);}
pid$1::malloc:entry{
@=quantize(arg0);}
malloc size observer
Thread monitor
pid$1:::entry{
@[probefunc]=count();}
hot methodfinder
Thursday, September 23, 2010
More DTrace examples
21
profile-101/execname==”java”/{
@[tid]=count();}
pid$1::sleep:entry{
@[ustack()]=count();}
Who went to sleep?
How many Java threads are active?
io:::done{
@=quantize(args[0]->b_bcount);}
Size distribution of IO
Thursday, September 23, 2010
DTrace and Java
• DTrace can observe the following in Java• When VM starts and ends• When thread starts and ends• When class gets loaded and unloaded• When object allocated and freed• When GC starts and ends• JNI calls• When a method is called and method returns
22
Thursday, September 23, 2010
DTrace and Java - Example
23
hotspot$1:::class-loaded{ printf("%s loaded\n", copyinstr(arg0,arg1));}
hotspot$1:::method-entry{
@[copyinstr(arg1,arg2), copyinstr(arg3,arg4)]=count();}
JVM needs to be started with -XX:+ExtendedDTraceProbes
Class Loading
Java Methods
Thursday, September 23, 2010
DTrace and Java - More Example
24
GC
#!/usr/sbin/dtrace -qshotspot$1:::gc-begin{ printf("GC started at %Y\n",walltimestamp); printf("%20s | %-10s\n", "pool", "time(ms)"); printf(" ------------------------------------\n"); self->ts = timestamp;}
hotspot$1:::mem-pool-gc-begin{ self->pool_ts[copyinstr(arg2,arg3)] = timestamp;}
hotspot$1:::mem-pool-gc-end{ self->pool = copyinstr(arg2,arg3); printf("%20s | %-10d\n", self->pool, (timestamp - self->pool_ts[self->pool])/1000);}
hotspot$1:::gc-end/self->ts/{ printf(" ------------------------------------\n"); printf("%20s | %-10d\n\n","Total GC time", (timestamp - self->ts)/1000);}
Thursday, September 23, 2010
Other Providers
25
DTrace providers
javaperlruby
javascript mysqlpostgres
cpciptcpudp
plockstatnfsv3
nfsv4
kerberos
sh
iSCSI
USDT
FC
DTracecore provider
DTracecore
IOsyscall
profilesysinfo
vminfo
PID
mib
fbtproc
lockstat
python
Thursday, September 23, 2010
DTrace Eco-Systems
26
• DTraceToolkit• Collection of useful D-scripts
• Chime• DTrace Data Visualization
• D-Light• Sun Studio based tool to observe application using DTrace
• DTrace analytics• DTrace based front end for Sun ZFS storage
Thursday, September 23, 2010
DTrace & Oracle WebLogic Server
27
Thursday, September 23, 2010
Instrumenting JVM vs Byte Code
28
JVM
Garbage Collector
Class Loader
JIT Compiler
Execution Engine
Byte Code
gc-begingc-end
class-loadedclass-unloaded
method-compile-beginmethod-compile-end
method-entrymethod-return
Thursday, September 23, 2010
Instrumenting JVM vs Byte Code
29
JVM
Garbage Collector
Class Loader
JIT Compiler
Execution Engine
Byte Code @OnMethod{ clazz=”myclass”, method=”mymeth”}
BTrace Supports other annotations@OnTimer, @OnError, @OnEvent & @OnLowMemory
Thursday, September 23, 2010
30
{
}
action do what
BTrace Code Structure
@OnMethod ( clazz = “java.lang.thread”,method = “start” where
import com.sun.btrace.annotations.*;import statuc com.sun.btrace.BTraceUtil.*;@BTracepublic class BTraceSample {
)
public static void func()
Thursday, September 23, 2010
Sample B-script
31
import com.sun.btrace.annotations.*;import static com.sun.btrace.BTraceUtils.*;
@BTrace public class ThreadStart { @OnMethod( clazz="java.lang.Thread", method="start" ) public static void onnewThread(@Self Thread t) { println(strcat("starting ", name(t))); }}
New Java Thread
Tracer
Thursday, September 23, 2010
More B-scripts
32
@BTrace public class AllMethods { private static Aggregation count = newAggregation(AggregationFunction.COUNT);
@OnMethod( clazz="/javax\\.swing\\..*/", method="/.*/" ) public static void m(@ProbeClassName String probeClass, @ProbeMethodName String probeMethod) { AggregationKey key = newAggregationKey(strcat(probeClass,strcat(".",probeMethod))); addToAggregation(count,key,1); }
@OnEvent public static void onEvent() { truncateAggregation(count,10); printAggregation("Count", count); }}
Count all javax.swing methods
Thursday, September 23, 2010
DTrace Vs BTrace
33
DTrace for JVM BTrace
Dynamic tracing entire system Only within a Java Process
Instrument JVM Instrument Java Byte Code
D-script Java like instrumentation Lang
Link Java to entire stack Just within Java
High overhead for method tracing
Low overhead for method tracing
Thursday, September 23, 2010
Combining DTrace & BTrace
34
DTrace BTraceBest
ofBoth
Worlds
DFr
ame
Thursday, September 23, 2010
Dynamic Tracing Framework for Oracle WebLogic Server
35
Thursday, September 23, 2010
The DFrame config file• The Framework is controlled using the config file• Two types of probes
• mapped• config file format
class:method:entry/exit:logical name• probe format
weblogic*:::event(logical name, class.method)• unmapped
• config file formatclass:method
• probe formatweblogic*:::method-entry(class, method)weblogic*:::method-return(class,method)
36
Thursday, September 23, 2010
How to use the Framework• Only one command to remember
embedProbes• Usage
embedProbes <pid>embedProbes -f configFile <pid>embedProbes -d [debug on]embedProbes -B [save B-script]
• The command reads config file, creates the b-script and embedded dprobes into weblogic
37
Thursday, September 23, 2010
38
Demonstration
Thursday, September 23, 2010
Reducing the Observer Effect
39
• Use low overhead tools✓dframe overhead for most probes very low (<3%)
• Ability to dynamically enable and disable probes✓dframe can be turned on and off dynamically
• Measure the probe effect before running in production✓ex: for medrec application on a Sun T5140
for processing 2016 transactions,this is 487,240 events the overhead us just %3.8
Thursday, September 23, 2010
Learn more
• White paper on DFrame– http://bit.ly/DFrame
• Need pre-release access?– email [email protected]
40
Thursday, September 23, 2010
DTrace Resources
41
• DTrace Community Page• http://hub.opensolaris.org/bin/view/Community+Group+dtrace/
• DTrace Documentation• http://wikis.sun.com/display/DTrace/Documentation
• DTraceToolkit• http://hub.opensolaris.org/bin/view/Community+Group+dtrace/dtracetoolkit
• DTrace Chime• http://hub.opensolaris.org/bin/view/Project+dtrace-chime/
• DTrace Hands on Lab• http://dtracehol.com/ - AMI ami-22fd164b (dtrace-hol)
Thursday, September 23, 2010
42
Thursday, September 23, 2010