Embed Size (px)
DESCRIPTION
Chris Conrad (Senior Engineering Manager) and Igor Perisic (Senior Director Engineering) from LinkedIn gave this talk to UC Santa Barbara in 2012.
Citation preview
The Evolution of the Professional Graph at LinkedIn
Chris Conrad Senior Engineering Manager,
Social Graph
Igor Perisic Sr. Director of Engineering, SNA
LinkedIn • The site officially launched on May 5, 2003. At the end of the first
month in operation, LinkedIn had a total of 4,500 members in the network.
• As of January 9, 2013, LinkedIn operates the world’s largest professional network on the Internet with more than 200 million members in over 200 countries and territories.
• As of September 30, 2012, LinkedIn counts executives from all 2012 Fortune 500 companies as members; its corporate talent solutions are used by 85 of the Fortune 100 companies.
• As of the school year ending May 2012, there are over 20 million students and recent college graduates on LinkedIn. They are LinkedIn's fastest-growing demographic.
In the beginning…
The Cloud • Cloud is the original name of our graph engine
• Responsible for read scaling graph queries (and it used to do search, too)
• Stored 4 primary sets of data:
Member Data
Group Membership
Network Cache
Connections
Cloud
What was wrong? • Large memory footprint
– Network cache used simple but inefficient data structures
– The size and density of the graph was increasing
• Garbage Collector woes – Large JVM heap caused long GC pauses
– Long GC pauses reduces availability resulting in site outages
C++ Graph • First project: migrate the network cache to a new data structure to
reduce memory usage
• Second project: implement a C++ JNI library to move the graph data off heap
• Result: Drastic reduction in JVM heap utilization
Member Data
Group Membership
Network Cache
Java Heap libGraphJNI.so
Connections
Cloud
Several million users later
New Problems • Growth
– The size and density of the graph was increasing
– We were running out of memory
– We were running out of CPU cycles
– Proliferation of services increased the overhead of maintaining client side software load balancer
– As of September 30, 2012, LinkedIn has 3,177 full-time employees located around the world. LinkedIn started off 2012 with about 2,100 full-time employees worldwide, up from around 1,000 at the beginning of 2011 and about 500 at the beginning of 2010.
• C++ code had a much higher maintenance cost – Coredumps are much less friendly than a NullPointerException
– LinkedIn didn’thave the expertise or infrastructure to support C++ development
Split cloud • cloud-session: Move the load balancing logic into a service we
control
• rgraph: Extract the C++ graph into its own service
Member Data
Group Membership
Network Cache
Java Heap
Cloud
libGraphJNI.so
Connections
rgraph
cloud-session
New problems, same as the old • rgraph instances still had a large memory footprint
– The density of the graph was increasing
– We were running out of memory
– We were running out of CPU cycles
• cloud-session’s software load balancer implementation was essentially a single point of failure
Distribute the Graph • Introduce Norbert a new cluster management system
• Partition the graph data
• Partition the network cache service
Member Data
Java Heap
Cloud
cloud-session
Connections
Group Membership
dgraph
Network Cache Service
Mission Accomplished
So now what?
My Connections
Common Connections
My Network
How am I connected?
What is the professional graph? • LinkedIn connections
• Current and past co-workers
• University colleagues and alumni
• Group members
• And what about geography, industry and skill overlap?
New requirements • Members aren’t the only type of node in the professional graph
• LinkedIn connections aren’t the only type of edge in the profession graph
• We already supported groups and group membership
Making changes was hard • Code was rigid
– Data was stored using class hierarchies, introducing data types was prohibitively slow
– Queries were built by combining object instances
• BDBJE
• Everything was back in the heap
– Garbage collection time was starting to go up
– GC pauses no longer caused outages, but flapping introduced high developer and operational overhead
Graph as a Service • Custom persistence engine
– Log structured
– Memory mapped files keeps data out of the Java heap
– Data described using DDL like schema
• Custom SQL like query language – Query language understands DDL
– Text based language reduces code changes
Graph Queries • Company(:id)[CompanyFollowers] • Member(:id)[MemberToMember{CreatedAt > :t}]
• Member(:id)[topN(MemberToMember, Score, 10)]
What do we have in common?
How am I connected?
What’s next? • Online schema migration
• Automated repartitioning and data migration
• Automated provisioning
• Hierarchical data partitioning
• Monitoring and statistics
• Query optimization
• Query fragment caching
• Result set caching
• Query parallelization
• Very large data set handling
• …
2 4 8
17
32
55
90
2004 2005 2006 2007 2008 2009 2010 2011 LinkedIn Members (Millions)
200M+
25th Most visit website worldwide (Comscore 6-12)
Company pages
>2.6M
63% non U.S.
2/sec
85% Fortune 100 Companies use LinkedIn to hire
And we’re still growing
Q&A
