How Badoo Saved $1M Switching to PHP7

Nikolay Krapivnyy

phpday.it, 2016

About me✦ Nikolay Krapivnyy

✦ 10+ years PHP experience

✦ 5 years with Badoo, now backend team lead

✦ This is my first talk in English :)

46languages

data

300Musers190

countries

“ BADOO IS THE LARGEST DATING NETWORK IN THE WORLD— Forbes

Badoo tech:

✦ 4 datacenters, ~3,000 servers overall

✦ up to 80,000 RPS for php backend

✦ ~200 engineers (~70 php devs)

✦ >2M PHP LoC

✦ and growing….

Today I will:✦ Share our practical experience

✦ Talk about some of our tools

✦ Tell you a million dollar story :)

1. Problem 2. Transition 3. Results

1. Problem

DBs

CLOUD SERVICES

LTM

How it works:APP

HTTPHTTP

API

API

9 %

15 %

22 %22 %

32 %phpmysqlotherphotosc/go

Hardware

32% or 1000of our servers are running php

PHP Cluster Growth

0

125

250

375

500

Jan, 2014 Jul, 2014 Jan, 2015 Jul, 2015 Jan, 2016

servers in 1DC users

CPU is the main bottleneck for us

How do we measure php performance?

Pinba!✦ Php is not a bottleneck anymore

✦ RT performance monitoring tool

✦ Extension for php + plugin for mysql

✦ Open source! http://pinba.org/

Request time

✦ Some API call: GET_USER

✦ avg request time = 0.12 sec

✦ request time = cpu time + i/o time

✦ i/o time = mysql + services + FS i/o +…

Pinba

Particular requests time

Pinba report

= +

Request time

✦ 0.12 sec = 0.07 (cpu) + 0.05 (i/o)

✦ 0.05 (i/o) = 0.012 + 0.05 + 0.03 + …

✦ 0.012 (memcache) = ~26*0.0005

✦ Realtime!

CPU usage takes ~50% of our request time

PHP7 alternatives?✦ HHVM (Facebook, http://hhvm.com/)

✦ kPHP (VK.com, https://github.com/vk-com/kphp-kdb)

+ JIT, PHP5 compatible— C++, extensions API

+ faster then HHVM

— no OOP support at all

2. Transition

Platform team:✦ C developers

✦ Php developers

✦ Badoo infrastructure support

Transition steps:2.1. Extensions

2.2. Codebase

2.3. Deploy

PHP7: extensions

xdebug

apcu

lua

imagick

✦ ~30 of them

✦ Already supported:

PHP7: extensions

blitz

xhprof

handlersocketi

pinba

leptonica

tarantool

sphinx

memtrack

judy

&& 15 more…

PHP7: extensions

https://github.com/tony2001

All php7 versions available:

Check for php7 branch

We thought we were done, but…

Runkit

What is runkit?

✦ runkit_method_add/redefine

✦ runkit_constant_add/redefine

✦ in runtime

✦ but why?

Unit testing!but why?

Unit testing:✦ 60K Tests now

✦ Coverage >50%

✦ Introduced only a few years ago

✦ Runkit: to isolate old code

Runkit: summary✦ essential for us

✦ no support for php7

Runkit for php7?✦ It`s different

✦ Alternatives? • UOPZ (https://github.com/krakjoe/uopz)

✦ Lots of changes anyway

Let`s remove this dependency

Soft mocks!✦ Soft means no extension

✦ Plain PHP, compatible with all PHP

✦ Replaces Runkit for us

✦ Open source! (https://github.com/badoo/soft-mocks)

How it works:✦ Rewrites code on the fly

✦ Inserts "interceptors" in methods

✦ Wraps all constant usages

✦ Uses PHP-Parser

1 require_once(__DIR__ . '/path.inc');2 ini_set(‘pinba.enabled', 1);3 echo RRD_BASE_PATH;

1 require_once SoftMocks::rewrite( ROOT_PATH . '/path' . '/path.inc' );2 SoftMocks::call( 'ini_set', array('pinba.enabled', 1’) );3 echo SoftMocks::getConst(‘RRD_BASE_PATH');

Original

After SoftMocks::rewrite()

1 class A2 {3 public function getValue()4 {5 return 10;6 }7 }

1 class A2 {3 public function getValue()4 { if (SoftMocks::isMocked(A::class, __FUNCTION__)) { return SoftMocks::getResult(…); }

5 return 10;6 }7 }

Original

After SoftMocks::rewrite()

So…

✦ No extension dependency

✦ 1.5 weeks for development

✦ A bit longer to adapt tests

✦ More stable tests in general

https://github.com/YuriyNasretdinov

CreditsMore cool stuff:

Soft mocks:

https://github.com/badoo/soft-mocks

2.2. Codebase

php7mar

✦ https://github.com/Alexia/php7mar

✦ Migration Assistant Report

php7mar 469 * foreachByReference 230 * oldClassConstructors 156 * funcGetArg 22 * hexadecimalString 19 * variableInterpolation 14 * yield 7 * deprecatedFunctions 4 * reservedNames 3 * arrayValueByReference

2.3. Deploy

Deploy✦ Devel: separate sandbox

✦ Staging

✦ Production

Deploy: production ✦ Several hosts in each cluster

✦ Several weeks of quarantine

✦ CLI: OK, WEB: ERROR

✦ WEB: fix for opcache reset in FPM

3. Results

CPU usage

cluster CPU usage

memory usage

We got: 2x CPU usage 9x memory usage

✦ 700 servers running FPM

✦ 1/2 required

✦ ~$4K each

✦ 350*4K = $1,4M

Let's calculate

Links✦ Pinba: http://pinba.org/

✦ Soft mocks: https://github.com/badoo/soft-mocks

✦ php7mar: https://github.com/Alexia/php7mar

✦ php7 exts: https://github.com/tony2001

Summary:✦ PHP7 rocks!

✦ Know your bottleneck

✦ Find your way

✦ Share!

Grazie!

Questions?

Twitter: @BadooTech

Blog: https://techblog.badoo.com

LinkedIn: https://linkedin.com/in/nkrapivnyy

Email: cyberklin@gmail.com

Joind.in: https://joind.in/talk/948d4