8/9/2019 Highload++ - myYearbook.com Architecture
1/30
myYearbook.comArchitecture
LessonsLearnedfromtheTrialsof
ScalingaHighTrafficWebsite
8/9/2019 Highload++ - myYearbook.com Architecture
2/30
Foundedin2005 3rdLargestSocial
NetworkinUnited
States
TeenageDemographic 60+Employees
8/9/2019 Highload++ - myYearbook.com Architecture
3/30
January2007
100MPageviews 1DatabaseServer 1WebApplicaOonServer Dailyissueswithloadandsiteavailability
8/9/2019 Highload++ - myYearbook.com Architecture
4/30
September2008
2.5BPageviews 30DatabaseServers 120WebApplicaOonServers 99.94%UpOmeasmeasuredbypingdom.com
8/9/2019 Highload++ - myYearbook.com Architecture
5/30
KeyArchitectureComponents PHP5,APC ApachehYpd PostgreSQL Memcached ApacheAcOveMQ
LighYpd IsilonIQClusteredNAS MessageSystems
eCelerity
Subversion
8/9/2019 Highload++ - myYearbook.com Architecture
6/30
WebApplicaOonArchitecture 2005-2007:MonolithicCodeBase 2008:MigraOngtoaServicesrientedArchitecture
ApplicaOonsgetownresourcesLooselyCoupledarchitecture
MVCApplicaOonusingXSLT
8/9/2019 Highload++ - myYearbook.com Architecture
7/30
WebApplicaOonArchitecture WhySA?
Monolithicappwasteshardware
CrossData-CenterperaOons
SelecOveMaintenance
8/9/2019 Highload++ - myYearbook.com Architecture
8/30
ScalingPostgres
RulesforScaling1.PlanforGrowth2.Knowtheinternals3.BiggerHardwareis
BeYer
8/9/2019 Highload++ - myYearbook.com Architecture
9/30
urPostgresScalingHistory
Quarter1,2007Monolithicdatabasewithoneschema,manycomplexjoinsandpooropOmizaOon
NoplanforgrowthNoDBA
8/9/2019 Highload++ - myYearbook.com Architecture
10/30
urPostgresScalingHistory
Quarter3,2008HorizontalShardedDataVerOcalParOOoning5000ConnecOons/secAvg
8/9/2019 Highload++ - myYearbook.com Architecture
11/30
ScalingPostgres:LessonsLearned
ScalingwebserversmeansmanydatabaseconnecOons,neededpooling
StartedwithpgPoolmovedtopgBouncer StartedwithSlonyreplicaOngread-onlyslaves
HighI/CPUverhead
8/9/2019 Highload++ - myYearbook.com Architecture
12/30
ScalingPostgres:LessonsLearned
BeganscalingverOcallybyseparaOngapplicaOondatabydatabaseserversand
removedreadonlyslaves
Neededfewsmalltablesreplicatedthatcouldbeslightlyinaccurateandeventuallyconsistent(BASE)
8/9/2019 Highload++ - myYearbook.com Architecture
13/30
ScalingPostgres:LessonsLearned
EnterplProxyDatabaseparOOoninglanguagebySkypeuOlizingPostgreSQLfuncOons
TriggerbasedplProxyfuncOonsreplicateneededtableswithouttheQueueoverhead
NTTRANSACTINSAFE
8/9/2019 Highload++ - myYearbook.com Architecture
14/30
ScalingPostgres:LessonsLearned
StandardUseofplProxyHorizontalparOOoningofdatabyIDacrossmulOpleservers
Example:MessagingSystem 8ServersstoreactualparOOonedmessagedata Rule#1PlanforGrowth
8/9/2019 Highload++ - myYearbook.com Architecture
15/30
ScalingPostgres:LessonsLearned
Knowinginternalspg_catalog
pg_stat_user_tables pg_stat_user_indexes
8/9/2019 Highload++ - myYearbook.com Architecture
16/30
ScalingPostgres:KnowingInternals
8/9/2019 Highload++ - myYearbook.com Architecture
17/30
ScalingPostgres:LessonsLearned DatabaseEcosystem
PerformanceFactors Indexbloat Usagechanges
Abuse CacheuOlizaOoncontenOon
8/9/2019 Highload++ - myYearbook.com Architecture
18/30
ScalingPostgres:LessonsLearned BiggerisBeYer
MoreRAM MoreDisks FasterandMoreCPU
8/9/2019 Highload++ - myYearbook.com Architecture
19/30
ScalingPostgres:LessonsLearnedScalingAcrossCPUCores
PostgreSQLScalesto32Cores
ExtensiveBenchmarking@MB
BeforeandA=erUpgade
8/9/2019 Highload++ - myYearbook.com Architecture
20/30
ScalingPostgres:FuturePlans MoreParOOoning SADataDistribuOon
Golconde PythonBased ApacheAcOveMQ
8/9/2019 Highload++ - myYearbook.com Architecture
21/30
ApacheAcOveMQ JavabasedMessage
Brokersoqware
Clientlanguageneutral ImplementsJMS1.1,
Stomp,XMPP,RESTand
thers
8/9/2019 Highload++ - myYearbook.com Architecture
22/30
Uploadedcontentprocessing ImageResize Contentanalysis(R&D) AnO-VirusScans
CommentandMessageprocessing SpamProcessing
EmailspoolingfromwebapplicaOon
Anywherewecanthatmakessense
TargetedWorkload
MessageQueuesallowfortherightserverforthejob
BeYerdistribuOonofCPUintensivetaskswithoutnegaOvelyimpacOngtheuserexperience
Clusterable,Scalable
8/9/2019 Highload++ - myYearbook.com Architecture
23/30
Memcached:KeyforSuccess ValuableScalingTool
ver250kgetreuestssecondduringpeakver750GBofcacheddataEasytoDeployThemoredistributedthecachebecomestheless
impacOngcachefailuresbecome-moreboxesarebeYerthanfewer
8/9/2019 Highload++ - myYearbook.com Architecture
24/30
Memcached:PotenOalProblems LargescaleimplementaOonscanhavesomehidden
problems Lotsofnetworktraffic Non-parOOonorevenlydistributeddata
Whattodofordatathatisnotevenlydistributed?
Implementedaround-robinclusterofmemcacheserversthatcontainthesamedata
8/9/2019 Highload++ - myYearbook.com Architecture
25/30
ResearchandDevelopment Copyr
Copy-on-WriteFilesystemReplicaOon Framewerk
PHP5DevelopmentFramework Golconde
QueueBasedDataDistribuOonforPostgreSQL Lightr
PHP5XMPPClassLibrary mod_xsltd
LighYpdXSLTransformaOonmodule Playr
PostgreSQLLogReplay Staplr
STAOsicalPackageLogicallyengineeredRight
8/9/2019 Highload++ - myYearbook.com Architecture
26/30
ToolsforSuccess
peraOonsPortalExecuOveLevelverviewofperaOonalStatusandProducOonChangeLog
StaplrTrending&AnalyOsSystem
8/9/2019 Highload++ - myYearbook.com Architecture
27/30
peraOonsPortal
8/9/2019 Highload++ - myYearbook.com Architecture
28/30
TrendingandAnalysis:Staplr Version0.6
PHPBased
Processforking ShelledRRDCommands
Version2.0 PythonBased Threaded Pythonwrapperstolibrrd
8/9/2019 Highload++ - myYearbook.com Architecture
29/30
TrendingandAnalysis:Staplr Pollsfor:
ApachehYpd ApacheAcOveMQ lighYpd memcached MySQL pgBouncer PostgreSQL SNMPData
APC,Isilon,F5,Xiotech,thers SysStat
8/9/2019 Highload++ - myYearbook.com Architecture
30/30
QuesOons?