LinkedIn Tech FoundationsTechnical Learning and Development Team

Learning Outcomes

How we architect apps and services

How we build insanely great products

How code gets from development to production

What systems power LinkedIn and keep them stable

Where to get the information and data you need

LinkedIn’s Core Values

Our members come first�

Relationships matter�

Be open, honest, and constructive�

Demand excellence�

Take intelligent risks�

Act like an owner�

go/values

The leadership team

Reid Hoffman Executive Chairman & Co-Founder

Jeff Weiner CEO

Nick Besbeas VP

Marketing

Mike Gamson SVP

Global Solutions

Jeff Weiner Interim Leader Product & UX

Kevin Scott SVP

Engineering

Steve Sordello SVP & CFO

Shannon Stubo VP

Corporate Comm

Pat Wadors VP

Talent Org

Mike Callahan General Counsel

& Secretary

The R&D Organization at LinkedIn

Product

OperationsProduct Development

Product Development

ProductOperations

Product Development

Engineering

Operations

Product Management

User Experience

Web Dev

Data Science

go/product-teams

Product

OperationsProduct Development

Operations

Product

Operations

Product Development

NOC

SRE

DBA

SysOps

NetOps

Security

Product Development

LinkedIn’s new experience methodology designed to be simple, unified, delightful, and actionable.

Product Vision

“Build insanely brilliant and simple products that change people's lives” - Deep Nishar

Product Vision

“Build insanely brilliant and simple products that change people's lives” - Deep Nishar

LinkedIn’s experience methodology is designed to be simple, unified, delightful, and actionable.

go/katy

DaVinci

Design consistency !Re-usable styles and widgets !Branding !Voice and Tone

go/davinci

Voice & Tone

Enabling Inviting Supportive

go/voiceandtone

Performance

Faster sites make for a better user experience.

go/rumdashboard

Our mobile moment is coming

Mobile

50%2014

go/mobile-team

Increase engagement through content translation, localization and geo-customization.

go/i18n

US

94MMNON-US

186MM

Internationalization and Localizationi18n and L10n

Members first

Product Development Process

Feature scoped

1

Features designed

and reviewed

2

Sprint planning and resources

allocated

3

Specification received from

product manager /or tech lead

Initiate contact with SRE, and stay in touch!

Feature Development Flow

Feature Development Flow

Sprint planning and resources

allocated

Specification received from

Product Manager /or Tech Lead

4

Test Engineer begins test

planning and strategy

5

Developers (back-end/ front-end) write code

6

Code committed and pushed to

integration environment

Initiate contact with SRE, and stay in touch!

Remember to coordinate ramp plan with SRE

Remember to coordinate ramp plan with SRE

Developers (back-end/ front-end) write code

Code committed and pushed to

integration environment

7

Test Engineer begins manual and automated testing

8

Sign-off from Service Owner, then

devise release and ramp plan

9

Ramp until 100% to all members in production

Remember to coordinate ramp plan with SRE

Feature Development Flow

Code committed and pushed to

integration environment

Test Engineer begins manual and automated testing

8

Sign-off from Service Owner, then

devise release and ramp plan

9

Ramp until 100% to all members in production

10

Remember to coordinate ramp plan with SRE

Feature Development Flow

Product Design

Product design sign-offOnly User Experience

Design (UED) leads/managers

Product sign-off

Executives give final approval

BAM office hours

Brand and Marketing (BAM) team and Steve Johnson

go/productreview

Using the vision from Product, the visuals from Design and the infrastructure from Engineering, Web Developers bring the

User Experience to life.

Web Development

go/webdev

Everyone can help! Use our products daily to find bugs and suggest enhancements!

Test Engineers

Test Engineers live by the motto “proud to ship with high quality”.

!Features only make it into production if our

Testing Teams are proud to release it.

How we work: Agile/Scrum

10-20 days

Daily Scrum meeting

IterationIteration backlog

Product backlog as prioritized by Product owner

Backlog tasks expanded by team

Potentially shippable product

increment

go/agile

Jira

Our global Bug and Task tracker

BugsProjects Enhancements

go/jira

Systems Architecture

Aspirational Architecture

“Provide abstractions that make it easier to reason about the LinkedIn architecture.”- Swee Lim, Distinguished Software Engineer

go/superblock

Front end

Business logic layer

Super BlockData access layer

Infrastructure

Mid Tier

Framework and language agnostic way to handle requests for data from internal services.

Rest.li

Profile

go/restli

Fizzy

Combine UI components from different applications into a single page. !Server side (i18n) and client-side (embeds) rendering logic.

go/fizzy

Template Dust

LinkedIn’s template engine for server and client side rendering

Dust

=

Output HTMLData

go/howtodust

QuickDeploy against EI (QEI)

Selectively deploy and run services during development. !Requests for other services handled by our Early Integration environment.

Profile

go/qei

From request to front-end

*PoP may be in a data center that is not running

the full app stack

Point of Presence*

NetScaler External Load Balancers (SSL)

L0 Proxy Apache Traffic Server

L1 Proxy Apache Traffic Server

HAProxy

Context path resolution E.g. /api/

Move all traffic to HTTPS !Enhance security and protection !Redirect logic (mobile devices) !Map url to applications !Multi-colo routing logic

LinkedIn Data Center

Application Containers

go/multicolo-routing

Release Process

How code gets to production

ReviewBoard approved

!Commit/Push

Run unit tests on CI server

!(1-5 mins)

PCS (integration tests)

PCL Tests the health

of trunk

If unit tests pass, code committed pushed

!Artifact built

PRE PCLPCS

SRE’s notified prod deploy

is about to occur

How code gets to production

Publish artifact (certified good)

Service Owner certifies build is ready to go to

production

Automated service certification.

Sends the status back to CRT

Ramp plan should already have been identified

Artifact is pushed to EI/EI2

EI EI2

How code gets to production

Service Owner certifies build is ready to go to

production

Automated service certification.

Sends the status back to CRT

SRE’s notified prod deploy

is about to occur

Ramp plan should already have been identified

Publish artifact (certified good)

Artifact is pushed to EI/EI2

EI EI2

Artifact deployed to production canary

machines.

If canary looks good, deploy to all

production nodes

LIX experiment used to control ramp to members

LID - LinkedIn Deployment Tool

Deployment

Next generation deployment tool for fine grained control of distributed systems at LinkedIn.

Artifactory

go/lid

CRT

Check the status of your checkin from commit to deployment.

go/crt

How do you know where your code is?

Code

Trunk Development

Trunk development - no branches

If unit test fails in PCS. Auto-revert. If failed in PCL, lock trunk and investigate.

Once issue confirmed resolved, unlock trunk.

“Ship it” from code owners and reviewers. Pass local unit tests and commit to trunk.

Issue identified through CRT. Fix committed.

go/mp

PCS

PCL

Network and Multiproduct

Network Trunk Multiproducts

Network trunk: original svn repository !Multiproduct: Code management containers for testing, building, QA and deployment outside of network trunk. !Mint: Tool for managing multiproducts.

FlexibleModular Dependency ManagementStandardized

go/mp

Code reviews

Small atomic commits. !How well are you solving the problem? !Accompanying unit tests and documentation a must.

��

go/codereview

Ship it!

Shipping code

Application Code Automated Tests Documentation

Why documentation matters

"Instead of imagining that our main task is to instruct a computer what to do, let us

concentrate rather on explaining to human beings what we want a computer to do."

- Donald Knuth

Site Operations

Operations group

Ensure LinkedIn has the infrastructure and resources

needed to be stable at a huge scale.

go/siteops

Autometrics

“Self-service” model for collectively adding metrics into our graphing system.

go/autometrics

Ingraphs

Dynamic visualization for the data collected by Autometrics and other sources.

Week-over-week comparisons

Real-time browsing

Personal dashboards

go/ingraphs

LiX (LinkedIn Experimentation Platform)

Allows for A/B test framework to allow new, modified, or experimental features to be deployed to a small segment of the LI user base. !Also used to carefully control the ramp of new features in production.

20%80%

go/lix

The NOC

Centralized Team for site monitoring and status reporting of GCNs. ! #noc

Network Operations Center

go/noc

Data Centers

ELA4 - Los Angeles LVA1 - Virginia TexasPlanned 2014

go/datacenters

Akamai, Limelight and Level 3

Content Delivery Networks (CDNs)

Delivers content to our members with high availability, performance and reduced latency.

Critical site issue escalation

If you see something that doesn’t look right,

report it!

go/issuecritical

1-855-832-LINK

site-issue-critical@linkedin.com

go/issuecritical

Non Critical Email: site-issue@linkedin.com

#noc

Security

Security is everyone’s concern. !

Members must feel that there data is safe with us.

go/learnsecurity

Being Data Driven

UX Research

Research talks to the field to inspire and inform our teams on what to do next

Using data to drive User Centered Design

Product Data Bootcamp

Short course geared toward new Product Managers on the basics of using our massive amounts of data as a PM.

Taught by data scientists and other data experts once every quarter.

go/productdatabootcamp

Data Science

The Data Science Team delivers valuable data to help us understand the

member and their site usage.

A/B testing

Exposing demographic trends

Product analytics and data and business insights

XLNT

XLNT is an end-to-end A/B testing solution, providing data for decisions on product changes.

go/XLNT

Informed

go/informed

A stream of micro-updates about deployments in production and staging.

Web UI, IRC bot, CLI, REST API for reading and writing updates

Life at LinkedIn

Tech Life at LinkedIn

Hackdays�

Weekly Tech Talks�

Horizontal Initiatives�

Incubator Program�

Functional Leadership Groups�

Knowledge Sharing�

go/values

Hack, Collaborate, Share and Learn

Learning More

Peoplego/cinco

InfoWiki

go/searchin

Code

go/codesearch-preview go/codingconventions

ResourcesLearnIn

Tech Talks

©2013 LinkedIn Corporation. All Rights Reserved.

Appendix A Systems

Data storage and distribution

Data Systems

Espresso

go/espresso

Also known as Pegasus, Rest.li is a REST+JSON framework for building robust, scalable service architectures using dynamic discovery

and simple asynchronous APIs.

Voldemort

go/voldemort

Voldemort is an open source distributed key-value storage system developed at LinkedIn.

Databus

go/databus

Source-agnostic distributed change data capture system, which is an integral part of

LinkedIn's data processing pipeline. !

Also used for offline data replication.

Tech

System Foundation

Fizzy

go/fizzy

Fizzy is a piece of LinkedIn infrastructure that is responsible for aggregating UI components served off of different web applications into a

single page.

Rest.li (Pegasus)

go/restli

Rest.li is a REST+JSON framework for building robust, scalable service

architectures using dynamic discovery and simple asynchronous APIs

Apache Traffic Server

go/ats

Fast, scalable and extensible reverse proxy and forward proxy HTTP server.

Tech

System Foundation

EKG

go/whatisekg

With canary validation, exception monitoring, GC monitoring and auto-tuning, EKG is our

tool for monitoring the health of our systems in production.

Hadoop

go/hadoop

Enables the distributed processing of large data sets across clusters of commodity

servers.

Avro

go/espresso

Apache Avro™ is a data serialization system, providing a serialization format for persistent

data and a compact binary format for communication between services.

Frameworks

System Foundation

LISpring

go/lispring

LI Spring is an extended version of Spring, the enterprise Java application development

framework with support dependency injection (wiring of components) and much more.

Play

go/play

Play is an open source, lightweight, fully reactive, stateless framework that

supports hot reload for Java and Scala.

Craig Sebenik Staff Site Reliability Engineer

!

Derek Brown Lead, Web Development

Subject Matter Experts

go/tech-foundations-slides !

Survey - search tech onboarding