Scaling 101Chris Finne

CTO of Venrock's Quarry

Venrock and the Quarry are looking for• Summer Interns:

o Ruby on Rails Engineers o Community Managero Digital Media Analyst

• Digital Media Associate

• Full Time Professional Web Engineerso Mobile/Media/Socialo MVC / Full stack

OS->DB->App->Code->Web Server->HTML/JS/CSS/Ajax->Flash

www.venrock.com

Scaling 101 - Assumptions / Misc

• Target Audienceo Engineers (but not professional web infrastructure)

• Give a "lay of the land" rather than heavy specifics

• < 20M database rows

• Will distribute the preso

• Links to various topics are provided as appendix

• Please interrupt with questions, but might table them for later

• About 30 slides

Scaling can be rocket science...

Doesn't have to be rocket science...

• Scaling 101 isn't hard

• Understand the principles

• If Scaling 101 isn't enough to handle your traffic...o you've probably have enough traffic to get Series A

funding

• Sometimes there is a quick-n-easy solution

• If not, follow basic problem solving cycle...o analyzeo researcho trial-n-error (hopefully with testing ;-)

Scaling 101 Principles - Slow to Fast

Infrastructure Speed1. External network accessed2. Internal network accessed

o DB on another server:1.DB Delete2.DB Update3.DB Select (goes to disk)4.DB Select (table in

memory)5.DB Select cached

o Memcached3. local filesystem4. local DB5. local memcached6. local app server memory

Specific Examples• Roundtrip query to

Facebook• Database table scans on

large tables (>100 rows)• Dynamically Rendering

high volume generic pages• Too many database calls

per page load (>5-10)

Typical Infrastructure Evolution

1. Single Server - get the app out!2. Optimize to try to stay on a single server3. Dedicated Database Server4. Multiple Web/App Servers - Load Balancing5. Add More Database Servers6. More Separation - Web and App servers

Specific Guidelines for when to level up...

Specific Guidelines - Nada

There aren't any. Why?

• heavy static files vs. very dynamic• application code is different• database activity is different

o lots of selects vs. inserts vs. updates vs. deleteso lots of complex JOINs vs. simple selects

• traffic patternso 8, 12 or 24 hour day?o Occasional Spikes (Digg, Slashdot)

• hardware is different from hosting vendor to hosting vendor

Optimize - Profiling your app

Questions to ask:o Which pages are getting hit the most

Google Analytics Web server logs

o Which pages take the longest to render Firebug - Net tab Yahoo's YSlow Add-on Apache Benchmark tool JMeter external monitoring site, e.g. Site24x7

Optimize - Profiling your app

Next question:o What pieces of those long, popular pages are taking the

longest?

Facebook queries application code blocks database queries static file downloads (images,CSS, JS)

Optimize - Profiling your app

How to actually find the offenders...o Application Code:

put debug statements to see where the most time is being spent

o Page Loading / Static file downloads Firefox Extensions:

Yahoo YSlow for Firebug: see which pieces take the longest to download / finish rendering

Live HTTP Headers: Are your static files being cached locally?

o Database MySQL slow query log MySQL Query log MySQL "EXPLAIN" Queries

Optimize - Profiling your app

Now what?o Focus on the bottlenecks

Optimize - Remedies

• Facebook querieso Reduce Roundtrips to FB with more complex FQL

(e.g. subqueries)o Cache Results

• Use FBML where possible - make FB do the worko fb:user fb:name fb:profile-pic

• Fix inefficient application codeo any examples?

Optimize - Remedies

• Optimize SQL Querieso Database Indexeso Only select what you need to selecto Views, Stored Procedures

• Confirm static files are being cached locally in browser (Images, JS and CSS)

• Apache Config...

• Caching...

Optimize - Caching

What? - Expensive pieces

o Facebook User Information (24hrs)o complex calculationso generic pages (or fragments)o complex, big or long DB queries

Optimize - Caching

Where to?o Filesystem

HTML pages served directly from Apache w/ no PHP expensive HTML fragments loaded via PHP

o User's Session Facebook User Info Application state

o Memcached - (or BerkeleyDB) HTML Session Facebook data (24hrs max) expensive DB query results

o Database Query Cache repeated queries

Optimize - Apache

• Are you using all your resources?• If you have 10 Apache processes (MaxClients) and 15 users

hit your app at the same time, 10 will get served, 5 will get an error

• Do you need more Apache processes?o No if your box's CPU and/or RAM are maxing out (use

top) need top optimize the app or add more servers

o Yes if the requests involve waiting for a long time for Facebook to answer a query (i.e. Apache is just waiting)

• Add more processes (MaxClients)

Optimize - Apache

# prefork MPM# StartServers: number of server processes to start# MinSpareServers: minimum number of server processes which are kept spare# MaxSpareServers: maximum number of server processes which are kept spare# MaxClients: maximum number of server processes allowed to start# MaxRequestsPerChild: maximum number of requests a server process serves<IfModule mpm_prefork_module>    StartServers          5    MinSpareServers       5    MaxSpareServers      10    MaxClients          150    MaxRequestsPerChild   0</IfModule>

Multi-Server - Dedicated DB

• Needs to be sitting next to the web/app server with a fast link

• Port 3306 open from Web/App to DB servers

• MySQL User account has to allow connections from the web/app servers

Multi-Server - Dedicated DB

If standard hardware/slices:o keep your DB server where it is

o setup a new slice as a web/app server

o get your new slice working test load-test

o cut-over your DNS address

Multi-Server - Dedicated DB

If DB performance is a bottleneck and you are going to a larger server for your DB...

o Configure / Test Configure the new DB mysqldump of your existing DB (do you have enough disk? dump over the wire) new DB: mysql < dump configure your new DB server as an Apache and PHP

server as well for testing functional test / load test

Multi-Server - Dedicated DB

If DB performance is a bottleneck and you are going to a larger server for your DB...

o Cut-over shutdown the production webserver mysqldump old import mysql to new configure your existing web/app server to point to new

db

Multiple Web/App Servers

• Bring up new app server

• Test it by itself

• Add to load-balancing...

Load Balancing

Goals:o Load and/or Fault Tolerance

Technologieso DNS Round Robin

Potential Windows / IE issues due to cachingo Open Source Software

Apache reverse Proxy (Apache 2.2) HAProxy Pound

o Hardware 10x or more performance than software some hosting vendors provide it as an add-on or part

of a package

Multiple DB Servers

Getting more involved...• Master/Slave

o reads go to slaveso transactional reads or guaranteed updated data reads go

to mastero writes go to master

Getting close to Rock Science...• Master/Master (possibly with slaves)• clusters

Other Easy Tricks

Put static files on Amazon's S3o Images, JS, CSS, Videos

Optimize Page Loads (not really scalability, but...)• Put external "stuff" at the bottom of the page outside

TABLESo Ad Tagso Google Analyticso Digg buttons

• DIV's instead of tables if possibleo tables wait for all content to be loaded before renderingo div's typically render piece by piece

Finally - The End

• Feedback to Yee or me: cfinne at venrock . com

• Follow-on questions or consults: cfinne at venrock . com

• Next Talk?o Interaction Design - David Cortrighto Venture Capital - Some VC (Brian, Ilya, Dev...)o Code / Web App Design - me (again???)

Links

MySQL Slow Query Log: http://dev.mysql.com/doc/refman/5.0/en/slow-query-log.html MySQL General Query Log: http://dev.mysql.com/doc/refman/5.0/en/query-log.html MySQL Query Cache: • http://jayant7k.blogspot.com/2007/07/mysql-query-cache.ht

ml• http://dev.mysql.com/doc/refman/5.0/en/query-cache.html

MySQL Optimize Queries and DB Indexes• http://www.databasejournal.com/features/mysql/article.php/1

382791

Memcached: • http://us3.php.net/memcache• http://www.danga.com/memcached/ 

Links

Firebug:• Firebug: https://addons.mozilla.org/en-US/firefox/addon/1843 • YSlow: http://developer.yahoo.com/yslow/ 

PHP Facebook Sessions: http://wiki.developers.facebook.com/index.php/PHP_Sessions Live HTTP Headers: https://addons.mozilla.org/en-US/firefox/addon/3829 Apache Prefork Config: http://httpd.apache.org/docs/2.0/mod/prefork.html Load balancing:• Apache Reverse Proxy:

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html • HAProxy:http://haproxy.1wt.eu/ • Pound: http://www.apsis.ch/pound