BUILDING DISTRIBUTED SYSTEMS

Juozas Kaziukėnas // juokaz.com // @juokaz

300’000’000 products/ 24 hours = 12’500’00 products/ 3600 seconds = 3’472 products

/ 3000 nodes = 1.1 sec. per product

BUICNOSCNLKSANCLAS

Juozas Kaziukėnas, Lithuanian

You can call me Joe

More info http://juokaz.com

ONE SERVER IS NOT ENOUGH

SCALING TO MULTIPLE SERVERS

INDEPENDENT NODES

1 TO 2 SERVERS = HARD2 TO 100 SERVERS = EASY .

TWO GENERALS PROBLEM

LOCKING... LOL

MYSQL TRANSACTIONS

STATELESS

USE TIMEOUTS

ASYNCHRONOUS PROCESSING

MESSAGING AND QUEUES

COMMUNICATION BETWEEN PROCESSES

MESSAGE SIZE

GEARMAN

NO RUNNING ORDER OR RELIABILITY

DATA

LOCATION

DE-NORMALIZATION

CAP THEOREM

EVENTUAL CONSISTENCY

ATOMIC OPERATIONS

CACHE

SETTING SERVERS UP

CONFIGURATION

PUPPET FOR SERVERS MANAGEMENT

DYNAMIC SCALING

SELF DISCOVERY

AMAZON EC2 SPOT INSTANCES

10x cheaper than normal on-demand instances

24’000 CORE CLUSTER = $300/H

PROTOTYPING

START SMALL

VAGRANT FOR VMS MANAGEMENT

WHAT IS GOING ON?

HARD

DEBUGGING DISTRIBUTED SYSTEMS IS HARD

ALL STACK LEVELS FAIL

MONITORING

COUNT EVERYTHING

UPTIME, VMSTAT, NETSTAT, IOSTAT, MPSTAT, TCPDUMP, GDB, VALGRIND, ULIMIT,

SYSCTL ETC...

THANKS!Juozas Kaziukėnas

@juokaz