EngineeringEngineeringv

Adam CataldoTuesday, January 24, 2012

Quick DeployA distributed systems approach to developer productivity

Engineering 2

Outline

LinkedIn architecture Quick Deploy Cool technical details

Engineering 3

LinkedIn Largest professional networking site

135M+ members 14 languages 200+ countries and

territories Estimated 4B+ people

searches per year

Engineering 4

LinkedIn stack at 20,000 feet

Online system for serving up data to clients– Profile pages, company pages, LinkedIn groups, etc.– Key metric: Latency

Offline system for creating insight from data– People you may know, contact degree of separation,

search indexing, etc.– Key metric: Throughput

Engineering 5

Offline data processingHadoopVoldemort

Map

Map

Map

Map

Reduce

Reduce

Reduce

Engineering 6

Online web applicationClient Presentation Business Logic Data Service

Datastore

Engineering 7

Growing Pains for the Online SystemAs time has gone by:

– We’ve added new services to support new features

– We’ve added new services to make our site faster and more available

– Our engineering productivity dropped

Engineering 8

Quick Deploy

Developers only locally deploy the services they actually modify

Every service your service depends on is available in some data center

This makes development move much faster!

AngelikaClayton

Engineering 9

Early Integration DatacenterDev Box

QD Proxy

Browser

Local Mid Tier

Shared Frontend

Shared Data Service

DB

Engineering 10

In DetailBrowser Proxy

EIFrontend

Dev-localMid Tier

EIData Service

QD CookieQD Cookie

QD IC

QD IC

QD IC

QD IC

Engineering 11

Proxy Behavior

Proxy receives request

Has QD cookie?

Route to developer’s local service

Route to service in EI

Read identifier

from cookie

Read identifier from IC

Is service local?

no

yes

yes

no

Engineering 12

QD Cookie and Invocation Context

A developer creates an identifier key, and registers his local services with the proxy

He identifies himself by adding his key to a cookie from his browser

When a front end gets a request with a QD cookie, it puts the key in the invocation context

Services pass the invocation context with every outbound request

Engineering 13

Shared data

OracleMySQL

VoldemortMemcached

Change Alfred’slanguage to

Japaneseきんぎょ

Huh?

Engineering 14

ActiveMQEarly Integration DatacenterDev Box

Producer

Consumer

ConsumerProducesmessage Consumes

message

QueueNever seesmessage

Engineering 15

TimersEarly Integration DatacenterDev Box

Service withTimer

Service

RPC Target

RPC TargetSchedulestime-triggered

RPC call Gets RPC call aftertimer triggers

Never gets RPC call

Engineering 16

Proxy in detail

Admin ConsolePython/Flask Web App

ProxyApache Traffic Server

> 10,000 QPS!

Routing ProfilesApache Zookeeper

ProxyApache Traffic Server

ProxyApache Traffic Server

Engineering 17

Why Apache Zookeeper

Distributed file system for small files Basically, a key-value data store, where the keys

are file names Clients can listen for file change and directory

change events

Great for storing data Also, great publish/subscribe system

Engineering 18

Dynamic DiscoveryEarly Integration DatacenterDev Box

QD ProxyBrowser

Local Mid Tier

Shared Frontend

Shared Data Service DB

Engineering 19

Conclusion

The LinkedIn architecture scales well in production

Quick deploy makes the architecture scale for developers

20% productivity gain across the engineering organization

Engineering 20

Thanks

RobLoh

SweeLim

SteveIhde

JimDumont