Scaling PHP to 40 Million Uniques

Northeast PHP 2013Jonathan Klein@jonathanklein

Saturday, August 17, 13

Who Am I?

• Performance Engineer at Etsy

Who Am I?

• Performance Engineer at Etsy• Boston Web Perf Meetup Organizer

Who Am I?

• Performance Engineer at Etsy• Boston Web Perf Meetup Organizer• Previously at Wayfair

Who Am I?

• Performance Engineer at Etsy• Boston Web Perf Meetup Organizer• Previously at Wayfair• Led team that converted to PHP

Slides, Links:jkle.in/nephp

codeascraft.com

Some Etsy Stats

Some Etsy Stats• 1.5 billion page views/month

Some Etsy Stats• 1.5 billion page views/month• Almost $1B in sales last year

Some Etsy Stats• 1.5 billion page views/month• Almost $1B in sales last year• Over 1M lines of PHP

Some Etsy Stats• 1.5 billion page views/month• Almost $1B in sales last year• Over 1M lines of PHP• 60M+ unique visitors/month

2012 Traffic

August

2012 Traffic

August

2012 Traffic

50% Higher

Holidays:~90M Uniques

A few others...

• Search (Solr)

A few others...

• Search (Solr)• Gearman

A few others...

• Search (Solr)• Gearman• Redis

A few others...

• Search (Solr)• Gearman• Redis• Postgres (legacy)

Hardware (Supermicro)

Hardware (Supermicro)2x 8-core Intel E5-2960 CPUs

24GB of RAM

160GB SSD

Apache

• Apache 2.2

Apache

• Apache 2.2• Prefork MPM

Apache

• Apache 2.2• Prefork MPM• mod_php5

Apache

• Apache 2.2• Prefork MPM• mod_php5

StartServers 30MinSpareServers 30MaxSpareServers 60ServerLimit 60MaxClients 60MaxRequestsPerChild 0

• PHP 5.4

• PHP 5.4• Zend OPCache (3GB Memory Segment)

• PHP 5.4• Zend OPCache (3GB Memory Segment)• memory_limit: 128M

• PHP 5.4• Zend OPCache (3GB Memory Segment)• memory_limit: 128M• max_execution_time: 30

Optimizing PHP

http://www.slideshare.net/jnklein/northeast-php-high-performance-php

Optimizing PHP

• See last year’s talk

Optimizing PHP

• See last year’s talk• Use an Opcode Cache

Optimizing PHP

• xhprof

Optimizing PHP

• xhprof

• StatsD/Graphite

Optimizing PHP

• xhprof

• StatsD/Graphite

• Find hotspots

Optimizing PHP

• xhprof

• StatsD/Graphite

• Find hotspots

• Upgrade it

Static Arrays

Translationsclass language_de {

static $translations = array( "<hash>" => array("content" => 'Accessoires', "file" => "TAXONOMY"), "<hash>" => array("content" => 'Keramik', "file" => "CATEGORY"), "<hash>" => array("content" => 'Dekorieren', "file" => "CATEGORY"), "<hash>" => array("content" => 'Getaggt %s', "file" => "Foo.php"), "<hash>" => array("content" => 'Badezusatz', "file" => "CATEGORY"), "<hash>" => array("content" => 'Datum:{% $date %}', "file" => "bar.tpl")

... snip ...

Atomic Deploys

Deploying PHP

http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/

Deploying PHP

• mod_realdoc - Apache module

Deploying PHP

• mod_realdoc - Apache module• incpath - PHP Extension

Deploying PHP

• mod_realdoc - Apache module• incpath - PHP Extension• A/B symlink swap

Deploying PHP

• mod_realdoc - Apache module• incpath - PHP Extension• A/B symlink swap• Avoid recompilation to opcodes

Upgrading PHP

PHP 5.4 vs. 5.3 - CPU

PHP 5.3

PHP 5.4

PHP 5.4 vs. 5.3 - Memory

PHP 5.3

PHP 5.4

Understand Framework

Overhead

http://systemsarchitect.net/performance-benchmark-of-popular-php-frameworks/

Frameworks are Harder to Scale

Memcached

• Roughly same hardware as webs

Memcached

• Roughly same hardware as webs• More memory (48GB)

Memcached

• Roughly same hardware as webs• More memory (48GB)• Shard keys across servers

Memcached

• Roughly same hardware as webs• More memory (48GB)• Shard keys across servers• mctop - https://github.com/etsy/mctop

• MySQL 5.5 (http://jkle.in/nephp)

• MySQL 5.5 (http://jkle.in/nephp)• ~20 master/master pairs

• MySQL 5.5 (http://jkle.in/nephp)• ~20 master/master pairs• innodb

• MySQL 5.5 (http://jkle.in/nephp)• ~20 master/master pairs• innodb• thread_cache=800

• MySQL 5.5 (http://jkle.in/nephp)• ~20 master/master pairs• innodb• thread_cache=800• max_connections=2500

• MySQL 5.5 (http://jkle.in/nephp)• ~20 master/master pairs• innodb• thread_cache=800• max_connections=2500• [Almost] no joins

Theoretically we can serve ~9000 req/sec

Static Content

~500 Million Images

Offloading Requests

• Multiple CDNs

Offloading Requests

• Multiple CDNs• Img hit rate > 90%

Offloading Requests

• Multiple CDNs• Img hit rate > 90%• CSS/JS hit rate ~100%

Offloading Requests

• Multiple CDNs• Img hit rate > 90%• CSS/JS hit rate ~100%• Edge serves ~7000 reqs/sec

What about misses?

Change Management

Feature Flags

// Owner: Username$server_config['foo']['bar'] = array( 'users' => array('jsmith', 'jdoe'),);

// Owner: Username$server_config['baz'] = [ 'enabled' => [ 'variant1' => 0, 'variant2' => 0, 'variant3' => 50 ],];

// Owner: Username$server_config['qux'] = array( 'enabled' => 100, 'data' => array( 'somedata' ));

if (Feature::isEnabled('foo.bar')) { // Code}

Ramp-up:1%, 5%, 25%, 100%

Deployinator

Schemanator

Recent Improvements

Sandy Bridge

Baseline Performance (median)

Small code change

PHP 5.4

Small code change

PHP 5.4

Disabled Hyperthreading

Challenges

Options

• Scale Horizontally (more shards)

Options

• Scale Horizontally (more shards)• Scale Vertically (SSD, bigger drives)

Options

• Scale Horizontally (more shards)• Scale Vertically (SSD, bigger drives)• Change architecture

Architecture Changes

• Ongoing conversation

• Ongoing conversation• Separate logical/physical shards

• Ongoing conversation• Separate logical/physical shards• Complex

• Ongoing conversation• Separate logical/physical shards• Complex• Server failure

• Migrating data

• Consistency

Monitoring

Graphite

Stacked Search Timers

Ganglia

Nagios

Takeaways

K.I.S.S.

Use Proven Technologies

Understand Your Stack

Measure Everything

Work at Etsy ;-)

Connecthttp://www.meetup.com/Web-Performance-Boston/

www.etsy.com/careers

jonathan@etsy.com

@jonathanklein

Scaling PHP to 40 Million Uniques

Technology

22-Aug-15 | 1 |1 | Help! I need more servers! What do I do? Scaling a PHP application

InterConPHP 2014 - Scaling PHP

Craig Kerstiens - Scalable Uniques in Postgres @ Postgres Open

Platform - URL Crowdfunding · 2013. 12. 5. · Keep What You Raise Keep What You Raise Monthly Uniques Monthly Uniques Number of Campaigns Number of Campaigns 1 7.9% + 2 8 563,787

Hack%for%HipHop%cufp.org/2013/slides/verlaguet.pdf · Scaling%PHP%runAme:%HipHop% Compiler%C++ (Producon) Interpreter% (Sandboxes)% PHP%

Scaling to 200K Transactions per Second with Open Source - MySQL, Java, curl, PHP

Scaling PHP - Free Chapter

Scaling PHP in the real world!

Gate Uniques Violet Collection Look Book

High Performance Php My Sql Scaling Techniques

Lecture 14: Scaling a Web Site - Carnegie Mellon University15418.courses.cs.cmu.edu/spring2015content/lectures/14_webscalin… · Two popular content management systems (PHP) -Wordpress

Takipi's Iris Shoor on Content Marketing: Zero to 70K uniques per month

PHP on Heroku: Deploying and Scaling Apps in the Cloud

High performance PHP: Scaling and getting the most out of your infrastructure

Scaling PHP/MySQL Presentation from Flickr

Wow einzigartige bilder_3_Photos uniques

Scaling PHP in The Real World

New hasgeek Scaling PHP in the Cloud · 2012. 9. 13. · About Scaling PHP in the Cloud is a participant-driven conference on scaling web applications PHP is today the world’s most

Violet Collection by Gate Uniques

Tek12: php://memory and streams for scaling