Learning Dtrace

LearningDtrace2012.11.20Outsider at

DTrace is a comprehensive dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time.

http://dtrace.org/blogs/about/dtracepony/



















http://www.flickr.com/photos/ghost_of_kuji/370072145

Bryan Cantrill Mike Shapiro Adam Leventhal

Originallydevelopedfor

2003

Open-sourcedin 2005

http://www.flickr.com/photos/tinou/171803338



http://www.flickr.com/photos/epu/4299657320

Ported to Unix-like systems

MITTechnology Review2005

“Innovators Under 35”http://www2.technologyreview.com/tr35/profile.aspx?trid=91

http://www.dowjones.com/innovation/ei_winners_2006.html


TechnologyInnovationAwards 2006

Gold Winnerhttp://www.dowjones.com/innovation/ei_winners_2006.html



STUG award 2008

https://www.usenix.org/about/stug



Dive into Dtrace

코드가 실행되는 지점동적으로 생성프로덕션에서 사용가능

Probe

Probe List:

sudo dtrace -l

Probe의 내부 ID

Probe를 제공하는 곳(ex: syscall, profile)

Unix 모듈이나Probe의 어플리케이션 라이브러리의 이름

Probe가 존재하는함수의 이름

Probe의 이름

sudo dtrace -l -n PROBE이름

sudo dtrace -l -f FUNCTION

sudo dtrace -l -P PROVIDER

sudo dtrace -l -m MODULE

DScriptD Programming Language

.d 확장자컴파일해서 커널레벨에서 실행안전성을 위한 유효성 검사프로덕션에서 안전하게 실행

Features

DScript

KernelUser

DProgram

Output

Dtrace

probe-description/optional predicate/{ action statements;}

0102030405


0102030405

provider:module:function:name(ex: syscall:::entry)


0102030405

시스템의 상태를 수집하는 명령문


0102030405

0이 아니거나 true면 action 수행

begin-end.dBEGIN { trace("begin the beguine"); exit(0); }END{ trace("that's all...");}

010203040506070809

begin-end.dBEGIN { trace("begin the beguine"); exit(0); }END{ trace("that's all...");}

010203040506070809

sudo dtrace -s begin-end.d

timer.dprofile:::tick-5sec{ trace("5sec timer");}profile:::tick-800msec{ trace("800msec timer");}

0102030405060708

beer.dint bottles;BEGIN{ bottles = 5; }profile:::tick-1sec/bottles >= 0/{ printf("%d bottles on the wall\n", bottles); printf("%d bottles.\n", bottles); printf("take one down, pass it around\n"); printf("%d bottles on the wall\n\n", bottles); bottles--; }profile:::tick-1sec/bottles < 0/{ exit(0); }END { printf("that's all..."); }

0102030405060708091011121314151617

no if-elseno loop?: operator is exist

No Flow Control

process.dproc:::exec-success{ printf("%s", execname);}

01020304

@: aggregation’s prefixname: aggregation’s namekey: D expression list (comma-separated)aggfunc: aggregation function

Aggregate

@name[key] = aggfunc(args)

count() : 호출횟수sum(expr) : 표현식의 전체 값avg(expr) : 표현식의 평균min(expr) : 표현식 중 가장 작은 값max(expr) : 표현식 중 가장 큰 값quantize(expr) : 2제곱의 빈도분포lquantize(expr,lower-bound, upper-bound, step-value) : 선형 빈도분포

Aggregate Function

aggr-count.dsyscall::read:entry{ @counts[execname] = count();}profile:::tick-5s{ exit(0);}

0102030405060708

aggr-quant.dsyscall::read:entry{ self->ts = timestamp;}syscall::read:return/self->ts/{ delta = timestamp - self->ts; @quanttime[execname] = quantize(delta);}profile:::tick-5s{ exit(0);}

0102030405060708091011121314

Dtracewith Node.js

http://www.flickr.com/photos/abandonedhero/3404826467

OS Requirements-Dtrace-ustack helper



USDTUser-Level Statically Defined Tracing

landed in node v0.6.7

http://www.flickr.com/photos/jepoirrier/2043728206

Full featureswork on OpenSolaris



Mac OS

Dtraceustack helper

Linux

Dtraceustack helper

WindowsDtrace

ustack helper

http://www.flickr.com/photos/vectorlyme/206472613

node.js is compiled with

--with-dtrace



“SunOS에서 활성화되지만 다른 시스템에선 동작하지 않는다”

Dtrace with node.json SmartOS of Joyent

Demo

server.jsvar http = require('http');http.createServer(function(req, res) { res.writeHead(200, { 'Content-Type': 'text/plain' }); res.end('Hello World\n');}).listen(8124);

console.log('Server running!');

010203040506070809

http-server.dBEGIN{ printf("%7s %2s %5s %20s (%5s) %8s %s (%s)\n", "WHO", "FD", "RPORT", "REMOTE", "BUFFR", "METHOD", "URL", "FWDFOR");}

node*:::http-server-request{ printf("+SERVER %2d %5d %20s (%5d) %8s %s (%s)\n", args[1]->fd, args[1]->remotePort, args[1]->remoteAddress, args[1]->bufferSize, args[0]->method, args[0]->url, args[0]->forwardedFor);}

node*:::http-server-response{ printf("-SERVER %2d %5d %20s (%5d)\n", args[0]->fd, args[0]->remotePort, args[0]->remoteAddress, args[0]->bufferSize);}

010203040506070809101112131415161718192021

request-count.dhttp-server-request{ @[args[0]->url]=count()}

01020304

loop.jsnew Error().stack;

function main() { func1(); }function func1() { func2(); }

function func2() { (function () { for (;;); })();}main();

0102030405060708091011

profile.dprofile-97/execname == "node" && arg1/{ @[jstack(150, 8000)] = count();}tick-10sec{ exit(0);}

010203040506070809

FlameGraphstack trace visualizer

http://dtrace.org/blogs/brendan/files/2011/12/mysql-flame.png



node-stackvis

$ npm install -g stackvis

$ sudo dtrace -s DSCRIPT > INPUT$ stackvis dtrace flamegraph-svg < INPUT > OUTPUT

Make FrameGraph

Alternativeon non-OpenSolarises

node-dtrace-providerhttps://github.com/chrisa/node-dtrace-provider

$ npm install dtrace-provider

var d = require('dtrace-provider');var dtp = d.createDTraceProvider('nodeapp');var p1 = dtp.addProbe('probe1', 'int', 'int');var p2 = dtp.addProbe('probe2', 'char *');dtp.enable();

0102030405

dtp.fire("probe1", function(p) { return [1, 2];});dtp.fire("probe2", function(p) { return ["hello, dtrace via provider", "foo"];});

010203040506

data type : int, charmaximum argument : 32

Limitations

interval.jsfunction interval(msg) { console.log(msg);}

setInterval(function() { interval('Hello Dtrace');}, 1000);

01020304050607

interval.jsinterval.jsvar d = require('dtrace-provider');

var dtp = d.createDTraceProvider('nodeapp');var p2 = dtp.addProbe('echo', 'char *');

function interval(msg) { dtp.fire('echo', function () { return [msg]; }); console.log(msg);} setInterval(function() { interval('Hello Dtrace');}, 1000);dtp.enable();

010203040506070809101112131415

interval.dnodeapp*:::echo{ trace(copyinstr(arg0));}

01020304

Thank you~

http://[email protected]@outsideris

Documents

Learning Dtrace