Web Performance Culture and Tools at Etsy

Mike Brittain@mikebrittain

Director of Eng., InfrastructureEtsy

All of this started in the last year.

Make PerformanceMatter

$0.20 per item3.5% transaction fee

Focus on the marketplace

ecommerce

ecommercead supported

ecommercead supportedsubscription

ecommercead supportedsubscription“free”

$$$

“Prove it.”

“Our own years of testing have conclusively shown that

when speed of a feature or product improves, usage,

quite simply, goes up.”

http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html

100+ web, api, jobs40+ databasesMany more doing lots of other stuff...

$$$

Performance is fun

I want you to tell mehow you feel...

mill

isec

onds

95th percentile

Average

mill

isec

onds This is when we

made it go faster!95th percentile

Average

I am better than you.

Make PerformanceMatter

PerformanceCulture

Our perf team is two people...

Measure

MeasureBuild tools

MeasureBuild toolsTrain & Coordinate

MeasureBuild toolsTrain & CoordinateMotivate

MeasureBuild toolsTrain & CoordinateMotivateDedicated

Our perf team is two people...

But every engineer thinksabout performance

Everything is slow. Please fix.

page generationtime

2011 Goal800 ms on five pages

http://www.etsy.com/blog/news/2011/november-2011-site-performance-report/

http://www.etsy.com/blog/news/2011/november-2011-site-performance-report/

Performance degradationis a P2 bug

Performance degradationis a P2 bug

WARNING: This part is hard

2012 Goal800 ms for all pages

Denormalize dataOptimize queriesRewrite from scratchCaching (memcache, app code)

Engineering

Network switchesOperating systemsNewer hardwareAdditional hardware capacity

Operations

Identify concerns earlyKill lagging features

Product

PerformanceCulture

Tools

How do you make the graphs?

Perf-related data belongs in your server logs

LogFormat %l %t \"%r\" %>s %b \"%{Referer}i\"

\"%{User-Agent}i\" %{custom_field}n

apache_note(“custom_field”, $whatever);

LogFormat "%{True-Client-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %{etsy_user_id}n %{php_bytes}n %{php_microsec}n %D" combined

$timer_start = microtime(true); ...$timer_diff = microtime(true) - $timer_start;

register_shutdown_function()

apache_note('php_microsec', $timer_diff);

web0060 66.249.71.110 - - [24/Aug/2011:04:16:52 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html)" - 13399576 505780 554876

Using something else?time, http method, request uri, response code, referer, user-agent, response time, response memory, custom segmentation fields...

Quick averagesgrep "GET /listing/" access.log | \

awk '{sum=sum+$(NF-1)} END {print sum/NR}'

Quick graphsgrep "GET /listing/" access.log | \

perl -pe "s/.*\[.*\d{4}:(\d{2}):(\d{2}):\d{2}.*\]/\1:\2/" | \

awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat

gives you...

Quick graphs# /tmp/pagetimes.dat

18:37 251.018:38 252.118:39 253.518:40 251.018:45 250.0 and then...

Quick graphs# GNUPLOT

set terminal png

set output 'listings.png'

set yrange [0:2000]

set xdata time

set timefmt "%d/%B/%Y:%H:%M:%S"

set format x "%H:%M"

plot '/tmp/pagetimes.dat' using 1:2 with points

Quick graphs

LogsterRun by cronMaintains a cursor on log filesSimple parsing & aggregationOutput to Ganglia or Graphite

github.com/etsy

Logster & Graphite

Signed-in vs. Signed-out

Response timesby bucket

time (ms)

resp

onse

s (pa

ges)

“Look at your Data”John Rauser, Velocity 2011

http://www.youtube.com/watch?v=coNDCIMH8bk

Idea lifted fromhttp://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html

Don’t forget the API

Measure performancedata in Production

Profiling

Lightweight, inline profiling

Start and end times wrapped around service calls databases, memcache, apc, 3rd party services, etc.

Enable with query string

“I/O Profiler”

“I/O Profiler”

“I/O Profiler”

XHProfgithub.com/facebook/xhprof

Deployed on pre-prod machineDirect access to profilingEnable with query string

Profile againstProduction

Make performance matter

Start small

Make graphs and reports

Create simple tools

Celebrate performance wins

Thank you

Mike Brittainmike@etsy.com@mikebrittain

codeascraft.etsy.com

Jobs at EtsyWe are currently hiring for a number of positions in engineering and operations.

http://www.etsy.com/careers