Embed Size (px)
Citation preview
A tale of scalabilityFrom one node to multiples DC's
Who?
2014 FIFA World Cup Brazil- 450K simultaneous users (ARG vs SWE)- 580Gbps (ARG vs SWE)- 1659 years watched (all games)- 7 x 1 (GER vs BRA)bbb, sportv, off, pfc, combate, gnews ...
Agenda
1. What this presentation is not about! 2. Basic glossary3. The story4. Questions
PS1: this presentation was not made by someone good with graphs/drawing/diagrams.
PS2: this reflects only my own unique personal individual stupid opinion not my employer.
What this presentation is not about!
Byzantine fault, 2PC, Paxos, RAFT, Threading, Locks, Leader election, ZAB, Consensus
problem, CRDTs, CALM, CAP theorem, to sum up this is not a deep distributed systems
presentation.
Basic glossary
Scalability: ability to enlarge to accommodate growth. (why?)
Basic glossary
Availability: the proportion of time a system is in a functioning condition. (why?)
Basic glossaryFault tolerance: ability to continue operating properly in the event of the failure. (why?)
Failover systems: software with automatic fault tolerance.
The story :: BananaApp
1st Solution :: 1 Server
databaseappserver
BRA
US
CAN Lost World
1st Solution :: problems
More users hit the app, it becomes slower :(CPU load was 1.3 (NOK) [h/top, w]I/O utilization was high (NOK) [iostat, vmstat]RAM usage 45% (OK) [free -m]Disk space 5% (OK) [df -h]
database
2nd Solution :: 2 Servers
appserver
BRA
USCAN
Lost World
database
2nd Solution :: good parts
Distributed the loadWe can fine tune each server separately
2nd Solution :: problems
More users hit the app, it becomes slower :(APP's CPU load was 1.3 (NOK)DB's CPU load was 0.3 (OK)I/O utilization was normal (OK)Introduce of network latency / point of failure
appserver
2nd Solution :: new conceptsPoint of Failure (or single point of failure [SPoF]) is a part of a system that, if it fails, will stop the entire system from working.Examples: our database and app server
When a solution is free from SPoF we can say it's a failover system.
3rd Solution :: 4 Servers
loadbalancerPPL
database
app1
app2
3rd Solution :: new concepts"Load balancer (LB) is a device/software that distributes network or application traffic across a number of servers (reals)." (F5)
1. How does it chooses which server to send? round-robin, least conn, weighted... 2. How does it knows about a dead node? health check /page, tcp:80...
3rd Solution :: LB exampleshttp {
upstream myapp1 {
server srv1.example.com;
server srv2.example.com;
}
server {
listen 80;
location / {
proxy_pass http://myapp1;
health_check uri=/health;
}
}}
listen appname 0.0.0.0:80 mode http stats enable balance leastconn option httpclose option forwardfor option httpchk HEAD /health HTTP/1.1 server srv1 srv1.example.com:80 check server srv2 srv2.example.com:80 check
NGINX HAProxy
3rd Solution :: problems
Users are getting signed out "randomly". This problem is also known as: session persistence, session stickiness. Nginx: sticky cookie srv_id expires=1h domain=.example.com path=/;
HAProxy: cookie srv_id insert indirect nocache
4th Solution :: 5 Servers
loadbalancerPPL
database
app1
app2
memcachedredis ...
4th Solution :: +problems
We now have 3 SPoFs: LB, memcache and database.
5th Solution :: LB (float/virtual ip)
/etc/sysctl.conf net.ip_nonlocal_bind=1/etc/ha.d/haresources lb1 192.168.0.10
5th Solution :: Database
Partition and Replication
ABCD
C B
D A
B (1,2) C (2,3)
A (3,0) D (0,1)
5th Solution :: mongo (master/bkp)
5th Solution :: cassandra (cluster)
5th Solution :: you got the idea
lb1PPL lb2
app1
app2
app3
appn
DB
Session
db1 db2
db3 db4
s1 s2
5th Solution :: + caching
lb1PPL lb2
app1
app2
app3
appn
DB
Session
db1 db2
db3 db4
s1 s2
Caching
c1
c2
c3
5th Solution :: Cachingproxy_cache_path /data/nginx/cache keys_zone=one:10m;
http {
upstream myapp1 {
server srv1.example.com;
server srv2.example.com;
}
server {
listen 80;
proxy_cache one;
location / {
proxy_cache_valid any 1m;
proxy_pass http://myapp1;
}
}}
NGINX
5th :: + Microservice Architecture
Application API's
Core
n1
c1 c1
n1
Search
n1
c1 c1
n1
Recommendation
n1
c1
n1
Social
n1
c1 c1
n1
mongodb elasticsearch spark/hadoop neo4j
n1
c1
5th :: Single datacenter (yet SPoF)
6th solution :: multihoming
6th solution :: models to replication
master / backup
master / master
2PC
Paxos
6th solution :: database
Cassandra can help you
6th :: DNS round robin
$ dig a www.youtube.com
6th solution :: anycastBorder Gateway Protocol (BGP) makes routing decisions based on paths, network policies or rule-sets configured by a network administrator, and is involved in making core routing decisions.
DNS solves www.example.com to 1.1.1.1
Clients from Colorado mostly will be routed to Colorado's DC.
Clients from California mostly will be routed to California's DC.
1.1.1.1
1.1.1.1
6th solution :: sub domain per cli
6th :: GSLB (Global Server Load Balancing)
6th :: GSLB multiples A records (BR)
$ dig a www.youtube.com
6th :: GSLB multiples A records (DE)
7th day: you shall rest
BR-DKC101
Summarizing
lb1
lb2
app1app2app3
appn
DB
Session
db1
db2d
b3
db4
s2
Cachingc
1
c2
c3
s1
US-DKC102
lb1
lb2
app1app2app3
appn
DB
Session
db1
db2d
b3
db4
s2
Cachingc
1
c2
c3
s1
JP-DKC103
lb1
lb2
app1app2app3
appn
DB
Session
db1
db2d
b3
db4
s2
Cachingc
1
c2
c3
s1
Bonus - Vagrant
Bonus - Docker (docker-compose)
Bonus - don’t blindly trust vendors
Link to this presentation
slideshare.net/leandro_moreira
References● https://f5.com/glossary/load-balancer● http://leandromoreira.com.br/2014/11/20/how-to-start-to-learn-high-scalability/● http://nginx.org/en/docs/http/load_balancing.html● https://www.digitalocean.com/community/tutorials/how-to-use-haproxy-to-set-up-http-load-balancing-on-an-ubuntu-vps● https://academy.datastax.com/courses/● http://en.wikipedia.org/wiki/Single_point_of_failure● http://book.mixu.net/distsys/single-page.html● https://www.howtoforge.com/high-availability-load-balancer-haproxy-heartbeat-debian-etch-p2● http://docs.mongodb.org/manual/core/sharding-introduction/● http://docs.mongodb.org/manual/core/replication-introduction/● http://nginx.com/resources/admin-guide/caching/● https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching?hl=en● http://martinfowler.com/articles/microservices.html● http://www.netflix.com/WiMovie/70140358?trkid=12244757● http://highscalability.com/blog/2009/8/24/how-google-serves-data-from-multiple-datacenters.html● http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html● http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/● http://tech.3scale.net/2014/06/18/redis-sentinel-failover-no-downtime-the-hard-way/● http://www.slideshare.net/gear6memcached/implementing-high-availability-services-for-memcached-1911077● http://docs.couchbase.com/moxi-manual-1.8/● http://highscalability.com/blog/2009/8/24/how-google-serves-data-from-multiple-datacenters.html● http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35590.pdf● http://static.googleusercontent.com/media/research.google.com/en//archive/chubby-osdi06.pdf● http://the-paper-trail.org/blog/distributed-systems-theory-for-the-distributed-systems-engineer/● http://nil.csail.mit.edu/6.824/2015/papers/paxos-simple.pdf● http://the-paper-trail.org/blog/consensus-protocols-paxos/● donkeykong.com● http://backreference.org/2010/02/01/geolocation-aware-dns-with-bind/● http://www.tenereillo.com/GSLBPageOfShame.htm● http://backreference.org/2010/02/01/geolocation-aware-dns-with-bind/
References● https://aphyr.com/tags/Distributed-Systems● http://pbs.cs.berkeley.edu/pbs-vldb2012.pdf● http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36737.pdf● http://blog.empathybox.com/post/19574936361/getting-real-about-distributed-system-reliability● https://www.usenix.org/legacy/events/nsdi06/tech/full_papers/freedman/freedman.pdf● http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35590.pdf
