SF Data Science: Developing Data Products

©2012 LinkedIn Corporation. All Rights Reserved.

Developing Data ProductsSF Data Science MeetupPete Skomoroch @peteskomorochSeptember 19 2013

Developing Data ProductsExamples, Techniques, & Lessons Learned

Our MissionConnect the world’s professionals to make them

more productive and successful.

Our VisionCreate economic opportunity for every

professional in the world.

Members First!

©2012 LinkedIn Corporation. All Rights Reserved. 4

LinkedIn is the leading professional network site

Worldwide Workforce

3,300M+2

Worldwide Professionals

640M+2

LinkedIn Members238M+

1


LinkedIn profiles represent our professional identity

238MMembers 238M MemberProfiles

1 2


We have a lot of data.


We have a lot of data.And (like everyone else), we store it in Hadoop.


We have a lot of data.And (like everyone else), we store it in Hadoop.And people build awesome things with that data.

What do we mean by data products?


Building products from data at LinkedIn

A few examples:

People You May Know Skills and Endorsements Year in Review Network Updates Digest InMaps Who’s viewed my profile Collaborative Filtering Groups You May Like and more…


Collaborative Filtering: LinkedIn Skill Pages


Classification: giving structure to unstructured data

Extract


Clustering & Disambiguation


De-duplication and Normalization


Network Algorithms: Relevance & Ranking


Prediction: Personalized Skill Recommendations


Skill Endorsements: Over 2 Billion and Growing


Social Proof and the Skill Endorsement Graph


The Economic Graph: Skills, Jobs, People, Locations…

TimeLocation

Lessons learned developing data products

Collect the right data at the right time


Large amounts of data can reveal new patternsP

rob

ab

ilit

y of

Job

Tit

le

Time since graduation


Be wary of “black-box” approaches


Look at your data


Aggregate statistics can be misleading


Build a viewer app, “micro-listen”


Algorithmic intuition: include data geeks in design


OODA: Think like a jet fighter


OODA: Observe, Orient, Decide, Act


OODA: The speed you can move determines victory


Red teaming: what can go wrong likely will


Error data is valuable, analyze it and adapt


Conclusion: tips for developing data products

Collect the right data at the right time Large amounts of data can reveal new patterns Be wary of “black box” approaches Look at your raw data Aggregate statistics can be misleading Build and use viewer apps Include data geeks in design process OODA: Think like a jet fighter Red-teaming: anticipate edge cases Find opportunity in your error data


Questions?

@peteskomoroch

Technology

SF Data Science: Developing Data Products