Distribute the workload, PHP Barcelona 2011

Distribute the workload

Helgi Þormar ÞorbjörnssonPHP Barcelona, 29th of October 2011

Saturday, 29 October 11

Who am I?


Co-founded Orchestra.io

Developer at PEAR

From Iceland

@h on Twitter

Helgi


Why Distribute?

Budget

Efficiency

Perception


Efficiency

10 small servers > 1 big


Budget

Spend wisely

Commodity servers

Cloud Computing (EC2)


Perception

Defer intensive processes

Give instant feedback

Users keep on browsing



Ant Colonies


Teamwork

When faced with a problem they will solve the problem as one.




Architect for Distribution


Characteristics

Decoupling

Elasticity

High Availability

Concurrency


Decoupling


Application

DB API

Cache FE


Application

DB API

Cache FE

Cache

API

API


Elasticity


Cloud Computing


Load Balancing


HA Proxy

Nginx

My Favourite


Monitoring


When do I need more servers?


Needs to be around from the start!


Keep records


Spot trends


Different types

Hardware Performance

Software Performance

Availability

Resourcing


ApplicationsNew Relic

CloudKick

ScoutApp

Nagios

Cacti

Circonus


Automation


Plug into your monitoring


Bringing together Monitoring and Elastic behaviour into one

beautiful whole!


Add some intelligence to add / remove servers as needed based

on current information.


Just make sure it doesn’t turn into...


Skynet!!Saturday, 29 October 11

High Availability


Get a highly available and resilient setup by following a few

of those recommendations


Remember, even Google has outages


What to avoid


Local Sessions


Store sessions in DB / Memcache

Solution


Local Memory


Networked Memcache

Solution


Local Files


Local Uploads


Writing to /tmp


Store on S3 or a networked FS

Solution


Serve up static files from CDNs

Solution


Servers can vanish at any given time


Internal APIs


Application

S3GFS FS

Internal Storage API


Application

MySQLMongo Cache

Internal DB API


SOA


Service Oriented Architecture


Sort of :-)


Eventually Consistent


CAP Therom


Consistency

Availability

Partition Tolerance


Consistency

All nodes see the same data at the same time


Availability

Node failures do not prevent survivors from continuing to

operate


Partition Tolerance

The system continues to operate despite arbitrary message loss


Consistency

Availability

Partition Tolerance


Queue Systems


Good forImage Processing

Distributed Logs

Data Mining

Mass Emails

Intensive transformation

Search


Common Tools

Gearman

Hadoop

ZeroMQ

RabbitMQ

And many others!


New York Times

4TB of TIFF files

Needed to get 11 million PDF versions

Used Hadoop and EC2

100 machines took 24 hours


Map/Reduce


Map

Master gets a problem to solve

Breaks into multiple sub-problems

Distributed to multiple workers

A worker can take the same steps

Answer passed back to Master


Reduce

Takes in answers from the map workers

Combines together to get an answer

There can be multiple reducers


process petabytes of data in few hours on commodity server farm


CouchDB


CouchDB

Highly Concurrent

Schema free, document based

RESTful API

Map/Reduce Views

Easy Replication


Gearman


Your Client Code

Gearman Client API(C, PHP, Perl, MySQL UDF, ...)

Gearman Job Servergearmand

Gearman Worker API(C, PHP, Perl, Python, ...)

Your Worker Code

Your App Gearman


pear.php.net/net_gearman


A Story!


Financial Software


3000+ Clients


Each one has 5 external data sources


Each data source is a web service


Ran every 6 hours every day


Cron

Gearman

Job 11

2

3

4

5

Web Services

1

43

2

5

Processing


But! That wasn’t enough


Job kicked off on login


Supervisord


j.mp/supervisord


http://j.mp/supervisord

http://j.mp/supervisord

Another Story!


CloudSplit


Near Real Time Cloud Analytics


Clients install logging agent locally


syslogd


Public API


Multiple Persistent Gearman Servers


Internal DB API


Agent syslogd

API

Gearman

Gearman

CouchDB

Worker

Worker

Worker

Internal API

Load Balanced

Load Balanced

PersistentSaturday, 29 October 11

CouchDB Setup


Write vs Read


Writes

Multi Master setup

Replicated

Deals with writes only


Reads

Multi Master setup

Replicated from write cluster

Slaves handle website requests


Heavy Map/Reduce usage for data


Questions?

@[email protected]

Joind.in: http://joind.in/4326


mailto:[email protected]

mailto:[email protected]

http://joind.in/4326

http://joind.in/4326

Technology

Distribute the workload, PHP Barcelona 2011