Debugging RubyAman Gupta

troubleshooting the VM and your app

About Aman GuptaSan Francisco, CA

Ruby Performance Consulting(Twitter, Github, Heroku, ZenDesk)

Ruby Hero 2009

EventMachine, amqp, REE, sinbook, perftools.rb, gdb.rb

github.com/tmm1

@tmm1

The Ruby VMMRI- Ruby 1.8.{6,7}

REE- Ruby 1.8.7 + patches

Tested primarily on:

Linux operating system

Intel CPUs (i686 and x86_64)

Tools for C code

list open fileslsof

lsof -nPp <pid>

lsof -nPp <pid>-nInhibits the conversion of network numbers to host names for network files.

-PInhibits the conversion of port numbers to port names for network files

FD TYPE SIZE NODE NAMEcwd DIR 4096 10437557 /var/www/myapprtd DIR 4096 2 /txt REG 1925210 1089684 /usr/bin/rubymem REG 48188 7061523 /json-1.1.9/ext/json/ext/generator.somem REG 46970 7061524 /json-1.1.9/ext/json/ext/parser.somem REG 3428025 1131339 /memcached-0.17.4/lib/rlibmemcached.somem REG 152972 5303443 /mysql-2.8.1/lib/mysql_api.somem REG 154013 1089708 /usr/lib/libtcmalloc_minimal.so.0.0.0mem REG 119288 11616294 /lib/ld-2.7.so 0u CHR 303 /dev/null 1w REG 4746405529 1106245 /usr/local/nginx/logs/error.log 2w REG 4746405529 1106245 /usr/local/nginx/logs/error.log 3u IPv4 TCP 10.8.85.66:33326->10.8.85.68:3306 (ESTABLISHED) 10u IPv4 TCP 10.8.85.66:33327->10.8.85.68:3306 (ESTABLISHED) 11u IPv4 TCP 127.0.0.1:58273->127.0.0.1:11211 (ESTABLISHED) 12u REG 20182 12463107 /tmp/RackMultipart.28957.0 33u IPv4 TCP 174.36.83.42:37466->69.63.180.21:80 (ESTABLISHED)

jsonmemcached

mysqlhttp

trace system calls and signalsstrace

strace -ttTp <pid> -o <file>

strace -cp <pid>

strace -cp <pid>-cCount time, calls, and errors for each system call and report a summary on program exit.

-p pidAttach to the process with the process ID pid and begin tracing.

% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ---------------- 50.39 0.000064 0 1197 592 read 34.65 0.000044 0 609 writev 14.96 0.000019 0 1226 epoll_ctl 0.00 0.000000 0 4 close 0.00 0.000000 0 1 select 0.00 0.000000 0 4 socket 0.00 0.000000 0 4 4 connect 0.00 0.000000 0 1057 epoll_wait------ ----------- ----------- --------- --------- ----------------100.00 0.000127 4134 596 total

strace -ttTp <pid> -o <file>-tPrefix each line of the trace with the time of day.

-ttIf given twice, the time printed will include the microseconds.

-TShow the time spent in system calls. This records the time difference between the beginning and the end of each system call.

-o filenameWrite the trace output to the file filename rather than to stderr.

01:09:11.266949 epoll_wait(9, {{EPOLLIN, {u32=68841296, u64=68841296}}}, 4096, 50) = 1 <0.033109>01:09:11.300102 accept(10, {sa_family=AF_INET, sin_port=38313, sin_addr="127.0.0.1"}, [1226]) = 22 <0.000014>01:09:11.300190 fcntl(22, F_GETFL) = 0x2 (flags O_RDWR) <0.000007>01:09:11.300237 fcntl(22, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000008>01:09:11.300277 setsockopt(22, SOL_TCP, TCP_NODELAY, [1], 4) = 0 <0.000008>01:09:11.300489 accept(10, 0x7fff5d9c07d0, [1226]) = -1 EAGAIN <0.000014>01:09:11.300547 epoll_ctl(9, EPOLL_CTL_ADD, 22, {EPOLLIN, {u32=108750368, u64=108750368}}) = 0 <0.000009>01:09:11.300593 epoll_wait(9, {{EPOLLIN, {u32=108750368, u64=108750368}}}, 4096, 50) = 1 <0.000007>01:09:11.300633 read(22, "GET / HTTP/1.1\r"..., 16384) = 772 <0.000012>01:09:11.301727 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 <0.000007>01:09:11.302095 poll([{fd=5, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout) <0.000008>01:09:11.302144 write(5, "1\0\0\0\0\0\0-\0\0\0\3SELECT * FROM `table`"..., 56) = 56 <0.000023>01:09:11.302221 read(5, "\25\1\0\1,\2\0x\234m"..., 16384) = 284 <1.300897>

stracing ruby: SIGVTALRM15:45:51.658164 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---15:45:51.658244 rt_sigreturn(0x1a) = 2207807 <0.000009>15:45:51.678208 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---15:45:51.678271 rt_sigreturn(0x1a) = 0 <0.000009>15:45:51.698161 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---15:45:51.698216 rt_sigreturn(0x1a) = 140734552062624 <0.000009>15:45:51.718154 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---15:45:51.718192 rt_sigreturn(0x1a) = 140734552066688 <0.000009>15:45:51.738185 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---15:45:51.738221 rt_sigreturn(0x1a) = 11333952 <0.000008>15:45:51.758162 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---15:45:51.758216 rt_sigreturn(0x1a) = 0 <0.000009>15:45:51.818168 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---15:45:51.819817 rt_sigreturn(0x1a) = 1 <0.000010>15:45:51.838196 --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---

ruby 1.8 uses signals to schedule its green threads

process receives a SIGVTALRM signal every 10ms

stracing ruby: sigprocmask

debian/redhat compile ruby with --enable-pthread

uses a native thread timer for SIGVTALRM

causes excessive calls to sigprocmask: 30% slowdown!

% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ----------------100.00 0.326334 0 3568567 rt_sigprocmask 0.00 0.000000 0 9 read 0.00 0.000000 0 10 open 0.00 0.000000 0 10 close 0.00 0.000000 0 9 fstat 0.00 0.000000 0 25 mmap------ ----------- ----------- --------- --------- ----------------100.00 0.326334 3568685 0 total

dump traffic on a networktcpdump

tcpdump -i eth1 -s 1500 -nqA tcp dst port 80

tcpdump -i eth0 -s 1500 -nqA tcp dst port 3306

tcpdump -i eth1 -s 1500 -w <file> tcp dst port 80

tcpdump -i <eth> -s <len> -nqA <expr>

-i <eth>Network interface.

-s <len>Snarf len bytes of data from each packet.

-nDon't convert addresses (host addresses, port numbers) to names.

-qQuiet output. Print less protocol information.

-APrint each packet (minus its link level header) in ASCII.

-w <file>Write the raw packets to file rather than printing them out.

<expr>libpcap expression, for example: tcp src port 80 tcp dst port 3306

tcpdump -i <eth> -w <file> <expr>

tcp dst port 330619:51:06.501632 IP 10.8.85.66.50443 > 10.8.85.68.3306: tcp 98E..."K@.@.Yy.UB.UD.....z....L............GZ.y3b..[......W....SELECT * FROM `votes` WHERE (`poll_id` = 72621) LIMIT 1

tcp dst port 8019:52:20.216294 IP 24.203.197.27.40105 > 174.37.48.236.80: tcp 438E...*.@.l.%&.....%0....POx..%s.oP.......GET /poll_images/cld99erh0/logo.png HTTP/1.1Accept: */*Referer: http://apps.facebook.com/realpolls/?_fb_q=1

tcpdump -w <file>

Google’s CPU profilergoogle-perftools

CPUPROFILE=/tmp/myprof ./myapp

export LD_PRELOAD=libprofiler.so

export DYLD_INSERT_LIBRARIES=libprofiler.dylib

pprof ./myapp /tmp/myprof

wget http://google-perftools.googlecode.com/files/google-perftools-1.4.tar.gztar zxvf google-perftools-1.4.tar.gzcd google-perftools-1.4

./configure --prefix=/optmakesudo make install

# for linuxexport LD_PRELOAD=/opt/lib/libprofiler.so

# for osxexport DYLD_INSERT_LIBRARIES=/opt/lib/libprofiler.dylib

CPUPROFILE=/tmp/ruby.prof ruby -e' 5_000_000.times{ "hello world" }'

pprof `which ruby` --text /tmp/ruby.prof

download

compile

profile

report

setup

Total: 103 samples 20 19.4% 19.4% 95 92.2% rb_yield_0 11 10.7% 30.1% 103 100.0% rb_eval 8 7.8% 37.9% 12 11.7% gc_sweep 3 2.9% 68.9% 52 50.5% rb_str_new3 3 2.9% 74.8% 3 2.9% obj_free 3 2.9% 77.7% 103 100.0% int_dotimes 3 2.9% 80.6% 12 11.7% gc_mark

pprof ruby ruby.prof --text

pprof ruby ruby.prof --gif

Profiling MRI10% of production VM time spent in rb_str_sub_bang

String#sub!

called from Time.parse

return unless str.sub!(/\A(\d{1,2})/, '')return unless str.sub!(/\A( \d|\d{1,2})/, '')return unless str.sub!(/\A( \d|\d{1,2})/, '')return unless str.sub!(/\A(\d{1,3})/, '')return unless str.sub!(/\A(\d{1,2})/, '')return unless str.sub!(/\A(\d{1,2})/, '')

switch to third_base gem

ThirdBase: Fast and Easy Date/DateTime class for Ruby

Profiling EM + threads

known issue: EM+threads = slow

memcpy??

thread context switches copy the stack w/ memcpy

EM allocates huge buffer on the stack

solution: move buffer to the heap

Total: 3763 samples 2764 73.5% catch_timer 989 26.3% memcpy 3 0.1% st_lookup 2 0.1% rb_thread_schedule 1 0.0% rb_eval 1 0.0% rb_newobj 1 0.0% rb_gc_force_recycle

trace library callsltrace

ltrace -ttTp <pid> -o <file>

ltrace -cp <pid>

% time seconds usecs/call calls function------ ----------- ----------- --------- -------------------- 48.65 11.741295 617 19009 memcpy 30.16 7.279634 831 8751 longjmp 9.78 2.359889 135 17357 _setjmp 8.91 2.150565 285 7540 malloc 1.10 0.265946 20 13021 memset 0.81 0.195272 19 10105 __ctype_b_loc 0.35 0.084575 19 4361 strcmp 0.19 0.046163 19 2377 strlen 0.03 0.006272 23 265 realloc------ ----------- ----------- --------- --------------------100.00 24.134999 82999 total

ltrace -c ruby threaded_em.rb

01:24:48.769408 --- SIGVTALRM (Virtual timer expired) ---01:24:48.769616 memcpy(0x1216000, "", 1086328) = 0x1216000 <0.000578>01:24:48.770555 memcpy(0x6e32670, "\240&\343v", 1086328) = 0x6e32670 <0.000418>

01:24:49.899414 --- SIGVTALRM (Virtual timer expired) ---01:24:49.899490 memcpy(0x1320000, "", 1082584) = 0x1320000 <0.000628>01:24:49.900474 memcpy(0x6e32670, "", 1086328) = 0x6e32670 <0.000479>

ltrace -ttT -e memcpy ruby threaded_em.rb

trace dlopen’d library callsltrace/libdl

http://github.com/ice799/ltrace/tree/libdl

ltrace -F <conf> -bg -x <symbol> -p <pid>

ltrace -F <conf> -b -g -x <sym>-bIgnore signals.

-gIgnore libraries linked at compile time.

-F <conf>Read prototypes from config file.

-x <sym>Trace calls to the function sym.

-s <num>Show first num bytes of string args.

-F ltrace.confint mysql_real_query(addr,string,ulong); void garbage_collect(void);int memcached_set(addr,string,ulong,string,ulong);

ltrace -x garbage_collect19:08:06.436926 garbage_collect() = <void> <0.221679>19:08:15.329311 garbage_collect() = <void> <0.187546>19:08:17.662149 garbage_collect() = <void> <0.199200>19:08:20.486655 garbage_collect() = <void> <0.205864>19:08:25.102302 garbage_collect() = <void> <0.214295>19:08:35.552337 garbage_collect() = <void> <0.189172>

ltrace -x mysql_real_query19:09:11.493395 mysql_real_query(0x19c7a500, "SELECT * FROM `users`", 21) = 0 <1.206506>19:09:16.630981 mysql_real_query(0x1c9e0500, "SET NAMES 'UTF8'", 16) = 0 <0.000324>19:09:16.631446 mysql_real_query(0x1c9e0500, "SET SQL_AUTO_IS_NULL=0", 22) = 0 <0.000322>19:09:16.654231 mysql_real_query(0x1c9e0500, "COMMIT", 6) = 0 <0.000181>

ltrace -x memcached_setmemcached_set(0x15d46b80, "Status:5456561633", 21, "\004\bo:\01{", 366) = 0 <0.001116>memcached_set(0x15d46b80, "Status:5453277696", 21, "\004\bo:\01{", 333) = 0 <0.000224>memcached_set(0x15d46b80, "Status:5435377757", 21, "\004\bo:\01{", 298) = 0 <0.001850>memcached_set(0x15d46b80, "Status:5435122010", 21, "\004\bo:\01{", 302) = 0 <0.000530>memcached_set(0x15d46b80, "Status:5407037167", 21, "\004\bo:\01{", 318) = 0 <0.000291>memcached_set(0x15d46b80, "Status:5405690802", 21, "\004\bo:\01{", 299) = 0 <0.000658>memcached_set(0x15d46b80, "Status:5343957534", 21, "\004\bo:\01{", 264) = 0 <0.000243>

the GNU debuggergdb

gdb <program> <pid>gdb <program>

Be sure to build with:-ggdb-O0

gdb walkthrough% gdb ./test-it (gdb) b averageBreakpoint 1 at 0x1f8e: file test-it.c, line 3.

(gdb) runStarting program: /Users/joe/test-it Reading symbols for shared libraries ++. doneBreakpoint 1, average (x=5, y=6) at test-it.c:33 int sum = x + y;

(gdb) bt#0 average (x=5, y=6) at test-it.c:3#1 0x00001fec in main () at test-it.c:12

(gdb) s4 double avg = sum / 2.0;(gdb) s5 return avg;

(gdb) p avg$1 = 5.5

(gdb) p sum$2 = 11

start gdbset breakpoint on function named average

run program

hit breakpoint!

show backtrace

function stack

single step

print variables

(gdb) where#0 0x0002a55e in rb_call (klass=1386800, recv=5056455, mid=42, argc=1, argv=0xbfffe5c0, scope=0, self=1403220) at eval.c:6125#1 0x000226ef in rb_eval (self=1403220, n=0x1461e4) at eval.c:3493#2 0x00026d01 in rb_yield_0 (val=5056455, self=1403220, klass=0, flags=0, avalue=0) at eval.c:5083#3 0x000270e8 in rb_yield (val=5056455) at eval.c:5168#4 0x0005c30c in int_dotimes (num=1000000001) at numeric.c:2946#5 0x00029be3 in call_cfunc (func=0x5c2a0 <int_dotimes>, recv=1000000001, len=0, argc=0, argv=0x0) at eval.c:5759#6 0x00028fd4 in rb_call0 (klass=1387580, recv=1000000001, id=5785, oid=5785, argc=0, argv=0x0, body=0x152b24, flags=0) at eval.c:5911#7 0x0002a7a7 in rb_call (klass=1387580, recv=1000000001, mid=5785, argc=0, argv=0x0, scope=0, self=1403220) at eval.c:6158#8 0x000226ef in rb_eval (self=1403220, n=0x146284) at eval.c:3493#9 0x000213e3 in rb_eval (self=1403220, n=0x1461a8) at eval.c:3223#10 0x0001ceea in eval_node (self=1403220, node=0x1461a8) at eval.c:1437#11 0x0001d60f in ruby_exec_internal () at eval.c:1642#12 0x0001d660 in ruby_exec () at eval.c:1662#13 0x0001d68e in ruby_run () at eval.c:1672#14 0x000023dc in main (argc=2, argv=0xbffff7c4, envp=0xbffff7d0) at main.c:48

Ruby VM stack traces

rb_eval recursively executes ruby code in 1.8

#include "ruby.h"

VALUEsegv(){ VALUE array[1]; array[1000000] = NULL; return Qnil;}

voidInit_segv(){ rb_define_method(rb_cObject, "segv", segv, 0);}

Debugging Ruby Segfaultstest_segv.rb:4: [BUG] Segmentation faultruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin9.7.0]

def test require 'segv' 4.times do Dir.chdir '/tmp' do Hash.new{ segv }[0] end endend

sleep 10test()

$ sudo gdb ruby 23611Attaching to program: ruby, process 236110x00007fa5113c0c93 in nanosleep () from /lib/libc.so.6(gdb) cContinuing.

Program received signal SIGBUS, Bus error.segv () at segv.c:77 array[1000000] = NULL;

1. Attach to running process$ ps aux | grep rubyjoe 23611 0.0 0.1 25424 7540 S Dec01 0:00 ruby test_segv.rb

2. Use a coredump

$ sudo mkdir /cores$ sudo chmod 777 /cores$ sudo sysctl kernel.core_pattern=/cores/%e.core.%s.%p.%t

Process.setrlimit Process::RLIMIT_CORE, 300*1024*1024

$ sudo gdb ruby /cores/ruby.core.6.23611.1259781224

def test require 'segv' 4.times do Dir.chdir '/tmp' do Hash.new{ segv }[0] end endend

test()

(gdb) where#0 segv () at segv.c:7#1 0x000000000041f2be in call_cfunc () at eval.c:5727...#13 0x000000000043ba8c in rb_hash_default () at hash.c:521...#19 0x000000000043b92a in rb_hash_aref () at hash.c:429...#26 0x00000000004bb7bc in chdir_yield () at dir.c:728#27 0x000000000041d8d7 in rb_ensure () at eval.c:5528#28 0x00000000004bb93a in dir_s_chdir () at dir.c:816...#35 0x000000000041c444 in rb_yield () at eval.c:5142#36 0x0000000000450690 in int_dotimes () at numeric.c:2834...#48 0x0000000000412a90 in ruby_run () at eval.c:1678#49 0x000000000041014e in main () at main.c:48

Enough C!

What about Ruby?

google-perftools for rubyperftools.rb

CPUPROFILE=/tmp/myprof ruby myapp.rb

export RUBYOPT=”-r`gem which perftools | tail -1`”

pprof.rb /tmp/myprof

gem install perftools.rb

http://github.com/tmm1/perftools.rb

require 'sinatra'

get '/sleep' do sleep 0.25 'done'end

get '/compute' do proc{ |n| a,b=0,1 n.times{ a,b = b,a+b } b }.call(10_000) 'done'end

$ ab -c 1 -n 50 http://127.0.0.1:4567/compute$ ab -c 1 -n 50 http://127.0.0.1:4567/sleep

== Sinatra has ended his set (crowd applauds)PROFILE: interrupts/evictions/bytes = 232/0/2152

Total: 232 samples 83 35.8% 35.8% 118 50.9% Sinatra::Application#GET /compute 56 24.1% 59.9% 56 24.1% garbage_collector 35 15.1% 75.0% 113 48.7% Integer#times

Sampling profiler:

232 samples total

83 samples were in /compute

118 samples had /compute on the stack but were in another function

/compute accounts for 50% of process, but only 35% of time was in /compute itself

CPUPROFILE=app.profCPUPROFILE_REALTIME=1CPUPROFILE=app-rt.prof

redis-rb bottleneck

why is rubygems slow?

gdb with MRI hooksgdb.rb

gdb.rb <pid>

http://github.com/tmm1/gdb.rb

def test require 'segv' 4.times do Dir.chdir '/tmp' do Hash.new{ segv }[0] end endend

test()(gdb) ruby threads

0xa3e000 main curr thread THREAD_RUNNABLE WAIT_NONE node_vcall segv in test_segv.rb:5 node_call test in test_segv.rb:5 node_call call in test_segv.rb:5 node_call default in test_segv.rb:5 node_call [] in test_segv.rb:5 node_call test in test_segv.rb:4 node_call chdir in test_segv.rb:4 node_call test in test_segv.rb:3 node_call times in test_segv.rb:3 node_vcall test in test_segv.rb:9

(gdb) ruby threads list0x15890 main thread THREAD_STOPPED WAIT_JOIN(0x19ef400) 4417 bytes0x19ef4 thread THREAD_STOPPED WAIT_TIME(57.10) 6267 bytes0x19e34 thread THREAD_STOPPED WAIT_FD(5) 10405 bytes0x19dc4 thread THREAD_STOPPED WAIT_NONE 14237 bytes0x19dc8 thread THREAD_STOPPED WAIT_NONE 14237 bytes0x19dcc thread THREAD_STOPPED WAIT_NONE 14237 bytes0x22668 thread THREAD_STOPPED WAIT_NONE 14237 bytes0x1d630 curr thread THREAD_RUNNABLE WAIT_NONE

(gdb) ruby eval 1+23(gdb) ruby eval Thread.current#<Thread:0x1d630 run>(gdb) ruby eval Thread.list.size8

(gdb) ruby objects HEAPS 8 SLOTS 1686252 LIVE 893327 (52.98%) FREE 792925 (47.02%)

scope 1641 (0.18%) regexp 2255 (0.25%) data 3539 (0.40%) class 3680 (0.41%) hash 6196 (0.69%) object 8785 (0.98%) array 13850 (1.55%) string 105350 (11.79%) node 742346 (83.10%)

(gdb) ruby objects strings 140 u'lib' 158 u'0' 294 u'\n' 619 u''

30503 unique strings 3187435 bytes

rails_warden leak(gdb) ruby objects classes 1197 MIME::Type 2657 NewRelic::MetricSpec 2719 TZInfo::TimezoneTransitionInfo 4124 Warden::Manager 4124 MethodOverrideForAll 4124 AccountMiddleware 4124 Rack::Cookies 4125 ActiveRecord::ConnectionAdapters::ConnectionManagement 4125 ActionController::Session::CookieStore 4125 ActionController::Failsafe 4125 ActionController::ParamsParser 4125 Rack::Lock 4125 ActionController::Dispatcher 4125 ActiveRecord::QueryCache 4125 ActiveSupport::MessageVerifier 4125 Rack::Head

middleware chain leaking per request

mongrel sleeper thread 0x16814c00 thread THREAD_STOPPED WAIT_TIME(0.47) 1522 bytes node_fcall sleep in lib/mongrel/configurator.rb:285 node_fcall run in lib/mongrel/configurator.rb:285 node_fcall loop in lib/mongrel/configurator.rb:285 node_call run in lib/mongrel/configurator.rb:285 node_call initialize in lib/mongrel/configurator.rb:285 node_call new in lib/mongrel/configurator.rb:285 node_call run in bin/mongrel_rails:128 node_call run in lib/mongrel/command.rb:212 node_call run in bin/mongrel_rails:281 node_fcall (unknown) in bin/mongrel_rails:19

def run @listeners.each {|name,s| s.run }

$mongrel_sleeper_thread = Thread.new { loop { sleep 1 } }end

god memory leaks(gdb) ruby objects arrays elements instances 94310 3 94311 3 94314 2 94316 1 5369 arrays 2863364 member elements

many arrays with 90k+ elements!

5 separate god leaks fixed by Eric Lindvall with the help of gdb.rb!

43 God::Process 43 God::Watch 43 God::Driver 43 God::DriverEventQueue 43 God::Conditions::MemoryUsage 43 God::Conditions::ProcessRunning 43 God::Behaviors::CleanPidFile 45 Process::Status 86 God::Metric327 God::System::SlashProcPoller327 God::System::Process406 God::DriverEvent

ruby method cache

(gdb) b rb_clear_cacheBreakpoint 1 at 0x41067b: file eval.c, line 351.(gdb) cContinuing.

Breakpoint 1, rb_clear_cache () at eval.c:351351 if (!ruby_running) return;(gdb) ruby threads

0x1623000 main curr thread THREAD_RUNNABLE WAIT_NONE node_call extend_object in sin.rb:23 node_call extend in sin.rb:23 node_call GET /other in lib/sinatra/base.rb:779 node_call GET /other in lib/sinatra/base.rb:779 node_call call in lib/sinatra/base.rb:779 node_fcall route in lib/sinatra/base.rb:474

(gdb) ruby methodcache UnboundMethod#arity Hash#[]= Class#private Array#freeze Module#Integer Fixnum#< Class#is_a? Fixnum#+ Class#protected Class#>=

2028 empty slots (99.02%)

Module#extend wipes the entire method cache!

ruby memory leak detectorbleak_house

ruby-bleak-house myapp.rb

export RUBYOPT=”-r`gem which bleak_house | tail -1`”

bleak /tmp/bleak.5979.000.dump

gem install bleak_house

http://github.com/fauna/bleak_house

191691 total objectsFinal heap size 191691 filled, 220961 freeDisplaying top 20 most common line/class pairs89513 __null__:__null__:__node__41438 __null__:__null__:String2348 lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:Array1508 lib/ruby/gems/1.8/specifications/gettext-1.9.gemspec:14:String1021 lib/ruby/gems/1.8/specifications/heel-0.2.0.gemspec:14:String 951 lib/ruby/site_ruby/1.8/rubygems/version.rb:111:String 935 lib/ruby/site_ruby/1.8/rubygems/specification.rb:557:String 834 lib/ruby/site_ruby/1.8/rubygems/version.rb:146:Array

BleakHouse

installs a patched version of ruby: ruby-bleak-house

unlike gdb.rb, see where objects were created (file:line)

create multiple dumps over time with `kill -USR2 <pid>` and compare to find leaks

Coming soon:bleak_house++

no patches or need to recompile ruby“assembly metaprogramming” to setup a trampoline on rb_new_objhttp://timetobleed.com/rewrite-your-ruby-vm-at-runtime-to-hot-patch-useful-features/

memory profilercoredump a production ruby processload the core to generate profiles of memory usage and leaks

Questions?

@joedamato

timetobleed.com

@tmm1

github.com/tmm1

Thanks for listening!