Intro to Neo4j or why insurances should love graphs

Introduction to Graph Databases

1

@peterneubauer #neo4j

Thursday, April 19, 12

What’s the plan?

2


What’s the plan?

๏Why a graph?

2


What’s the plan?

๏Why a graph?

๏Graph Database 101

2


What’s the plan?

๏Why a graph?


๏a look at Neo4j

2


What’s the plan?

๏Why a graph?


๏a look at Neo4j

๏the Real World

2


Why a graph?

3


Q: What are graphs good for?

4



4

๏Recommendations

๏Business intelligence

๏Social computing

๏Geospatial

๏MDM

๏Systems management

๏Genealogy

A: highly connected data



4

๏Recommendations


๏Social computing

๏Geospatial

๏MDM


๏Genealogy


• Real Use Cases:

• [A] ACL from Hell

• [B] Timely recommendations

• [C] Global collaboration


Trends in BigData & NOSQL

5



5

๏1. increasing data size (big data)



5


• “Every 2 days we create as much information as we did up to 2003” - Eric Schmidt



5



๏2. increasingly connected data (graph data)



5




• for example, text documents to html



5





๏3. semi-structured data



5






• individualization of data, with common sub-set



5







๏4. architecture - a facade over multiple services



5







๏4. architecture - a facade over multiple services

• from monolithic to modular, distributed applications


4 Categories of NOSQL

6


Key-Value Category๏“Dynamo: Amazon’s Highly Available Key-Value Store” (2007)

๏Data model:

•Global key-value mapping

•Big scalable HashMap

•Highly fault tolerant (typically)

๏Examples:

•Riak, Redis, Voldemort

7


Key-Value: Pros & Cons๏Strengths

• Simple data model

•Great at scaling out horizontally

• Scalable

•Available

๏Weaknesses:

• Simplistic data model

• Poor for complex data

8


Column-Family Category๏Google’s “Bigtable: A Distributed Storage System for Structured

Data” (2006)

•Column-Family are essentially Big Table clones

๏Data model:

•A big table, with column families

•Map-reduce for querying/processing

๏Examples:

•HBase, HyperTable, Cassandra

9


Column-Family: Pros & Cons๏Strengths

•Data model supports semi-structured data

•Naturally indexed (columns)

•Good at scaling out horizontally

๏Weaknesses:

•Unsuited for interconnected data

10


Document Database Category๏Data model

•Collections of documents

•A document is a key-value collection

• Index-centric, lots of map-reduce

๏Examples

•CouchDB, MongoDB

11


Document Database: Pros & Cons๏Strengths

• Simple, powerful data model (just like SVN!)

•Good scaling (especially if sharding supported)

๏Weaknesses:

•Unsuited for interconnected data

•Query model limited to keys (and indexes)

•Map reduce for larger queries

12


Graph Database Category๏Data model:

•Nodes & Relationships

•Hypergraph, sometimes (edges with multiple endpoints)

๏Examples:

•Neo4j (of course), OrientDB, InfiniteGraph, AllegroGraph

13


14

Living in a NOSQL WorldCo

mpl

exity

Size


RDBMS

14


mpl

exity

Size


RDBMS

14


mpl

exity

Size

Key-ValueStore


RDBMS

14


mpl

exity

ColumnFamily

Size

Key-ValueStore


RDBMS

14


mpl

exity

ColumnFamily

Size

Key-ValueStore

DocumentDatabases


RDBMS

14


mpl

exity

ColumnFamily

Size

Key-ValueStore

DocumentDatabases

GraphDatabases


RDBMS

14


mpl

exity

ColumnFamily

Size

Key-ValueStore

DocumentDatabases

GraphDatabases

90%of

usecases


Graph Database: Pros & Cons

15


Graph Database: Pros & Cons๏Strengths

15



• Powerful data model, as general as RDBMS

15




• Fast, for connected data

15





• Easy to query

15





• Easy to query

๏Weaknesses:

15





• Easy to query

๏Weaknesses:

• Sharding (though they can scale reasonably well)

15





• Easy to query

๏Weaknesses:


‣also, stay tuned for developments here

15





• Easy to query

๏Weaknesses:



•Requires conceptual shift

15





• Easy to query

๏Weaknesses:



•Requires conceptual shift

‣though graph-like thinking becomes addictive

15


Graph DB 101

16


Some well-known named graphs

17see http://en.wikipedia.org/wiki/Gallery_of_named_graphsThursday, April 19, 12

http://en.wikipedia.org/wiki/Gallery_of_named_graphs



17see http://en.wikipedia.org/wiki/Gallery_of_named_graphs

diamond






diamond butterfly






diamond butterfly star






diamond butterfly bullstar







franklin







franklin robertson







franklin hortonrobertson







franklin horton hall-jankorobertson




We’re talking about a Property Graph

18



๏Nodes

18



๏Nodes

18



๏Nodes

๏Relationships

18



๏Nodes

๏Relationships

18



๏Nodes

๏Relationships

18



๏Nodes

๏Relationships

๏Properties

18



๏Nodes

๏Relationships

๏Properties

18

name:Andreasjob: talking

name: Tobiasjob: coding

knowssince: 2008

knowssince: 2006

name: Peterjob: building

name: Emiljob: plumber

knowssince: 1992

name: Stephenjob: DJ

knowssince: 2002

knowssince: 2006

name: Deliajob: barking

knowssince: 2002

knowssince: 1998

name: Tiberiusjob: dancer

knowssince: 2000

name: Allisonjob: plumberknows

since: 2002

knowssince: 1998

knowssince: 1996



๏Nodes

๏Relationships

๏Properties

18



knowssince: 2008

knowssince: 2006



knowssince: 1992


knowssince: 2002

knowssince: 2006


knowssince: 2002

knowssince: 1998


knowssince: 2000


since: 2002

knowssince: 1998

knowssince: 1996



๏Nodes

๏Relationships

๏Properties

18

+ Indexes



knowssince: 2008

knowssince: 2006



knowssince: 1992


knowssince: 2002

knowssince: 2006


knowssince: 2002

knowssince: 1998


knowssince: 2000


since: 2002

knowssince: 1998

knowssince: 1996


Compared to RDBMS

19

becomes


A look at Graph Queries

20


Query a graph with a traversal

21




knowssince: 2008

knowssince: 2006



knowssince: 1992


knowssince: 2002

knowssince: 2006


knowssince: 2002

knowssince: 1998


knowssince: 2000


since: 2002

knowssince: 1998

knowssince: 1996


21


// lookup starting point in an indexstart n=node:node_auto_index(name = ‘Andreas’)



knowssince: 2008

knowssince: 2006



knowssince: 1992


knowssince: 2002

knowssince: 2006


knowssince: 2002

knowssince: 1998


knowssince: 2000


since: 2002

knowssince: 1998

knowssince: 1996


21

n


// lookup starting point in an indexstart n=node:node_auto_index(name = ‘Andreas’)// then traverse to find resultsstart n=node:People(name = ‘Andreas’)match (n)--()--(foaf) return foaf



knowssince: 2008

knowssince: 2006



knowssince: 1992


knowssince: 2002

knowssince: 2006


knowssince: 2002

knowssince: 1998


knowssince: 2000


since: 2002

knowssince: 1998

knowssince: 1996


21

n


22

Cypher


22

Cypher๏a pattern-matching query language

๏declarative grammar with clauses (like SQL)

๏aggregation, ordering, limits

๏tabular results


22

Cypher๏a pattern-matching query language

๏declarative grammar with clauses (like SQL)

๏aggregation, ordering, limits

๏tabular results

// get node with id 0start a=node(0) return a// traverse from node 1start a=node(1) match (a)-->(b) return b// return friends of friendsstart a=node(1) match (a)--()--(c) return c


Neo4j - the Graph Database

23


Background of Neo4j๏ 2001 - Windh Technologies, a media asset management company

• CTO Peter with Emil, Johan prototyped a proper graph interface

• first SQL-backed, then revised as a full-stack implementation

• (just like Amazon-Dynamo, Facebook-Cassandra)

๏ 2003 Neo4j went into 24/7 production

๏ 2006-2007 - Neo4j was spun off as an open source project

๏ 2009 seed funding for the company

๏ 2010 Neo4j Server was created (previously only an embedded DB)

๏ 2011 Fully funded silicon valley start-up - Neo Technology

24


Neo4j is a Graph Database

25


Neo4j is a Graph Database๏A Graph Database:

25



• a Property Graph with Nodes, Relationships

and Properties on both

25





• perfect for complex, highly connected data

25






๏A Graph Database:

25







• reliable with real ACID Transactions

25








• scalable: 32 Billion Nodes, 32 Billion Relationships, 64 Billion Properties

25









• Server with REST API, or Embeddable on the JVM

25









• Server with REST API, or Embeddable on the JVM

•high-performance with High-Availability (read scaling)25


the Real World

26



27



27

๏Recommendations


๏Social computing

๏Geospatial

๏MDM


๏Genealogy




27

๏Recommendations


๏Social computing

๏Geospatial

๏MDM


๏Genealogy


• Real Use Cases:

• [A] ACL from Hell

• [B] Timely recommendations

• [C] Global collaboration


[A] ACL from Hell

28


[A] ACL from Hell๏ Customer: leading consumer utility company with tons

and tons of users

๏ Goal: comprehensive access control administration for customers

๏ Benefits:

• Flexible and dynamic architecture

• Exceptional performance

• Extensible data model supports new applications and features

• Low cost

28



and tons of users


๏ Benefits:




• Low cost

28

• A Reliable access control administration system for

5 million customers, subscriptions and agreements

• Complex dependencies between groups, companies, individuals, accounts, products, subscriptions, services and agreements

• Broad and deep graphs (master customers with 1000s of customers, subscriptions & agreements)



and tons of users


๏ Benefits:




• Low cost

28

• A Reliable access control administration system for

5 million customers, subscriptions and agreements

• Complex dependencies between groups, companies, individuals, accounts, products, subscriptions, services and agreements

• Broad and deep graphs (master customers with 1000s of customers, subscriptions & agreements)

name: Andreas

subscription: sports

service: NFL

account: 9758352794

agreement: ultimate

owns

subscribes to

has plan

includes

provides group: graphistas

promotion: fall

member of

offered

discounts

company: Neo Technologyworks with

gets discount on

subscription: local

subscribes to

provides service: Ravens

includes


[A] ACL from Hell

29


[B] Timely Recommendations

30


[B] Timely Recommendations๏ Customer: a professional social network

• 35 millions users, adding 30,000+ each day

๏ Goal: up-to-date recommendations

• Scalable solution with real-time end-user experience

• Low maintenance and reliable architecture

• 8-week implementation

30








30

๏ Problem:

• Real-time recommendation imperative to attract new users and maintain positive user retention

• Clustered MySQL solution not scalable or fast enough to support real-time requirements

๏ Upgrade from running a batch job

• initial hour-long batch job

• but then success happened, and it became a day

• then two days

๏ With Neo4j, real time recommendations








30

๏ Problem:

• Real-time recommendation imperative to attract new users and maintain positive user retention

• Clustered MySQL solution not scalable or fast enough to support real-time requirements

๏ Upgrade from running a batch job

• initial hour-long batch job

• but then success happened, and it became a day

• then two days

๏ With Neo4j, real time recommendationsname:Andreasjob: talking

name: Allisonjob: plumber


knows

knows



knows


knows

knows


knows

knows


knows

knows

knows

knows


[C] Collaboration on Global Scale

31


[C] Collaboration on Global Scale๏ Customer: a worldwide software leader

• highly collaborative end-users

๏ Goal: offer an online platform for global collaboration

• Highly flexible data analysis

• Sub-second results for large, densely-connected data

• User experience - competitive advantage

31








31

• Massive amounts of data tied to members, user groups, member content, etc. all interconnected

• Infer collaborative relationships through user-generated content

• Worldwide Availability








31




Asia North America Europe








31







Insurance <3 Graphs?

32


Q: Why should you care?

33



33

A: because you have connected data.



33

๏CRM, BI, social graphs




33


๏GeoSpatial analytics




33



๏Fraud detection




33



๏Fraud detection

๏Network management



A sample insurance domain setup

34


A sample insurance domain setup

34

Home

Building A

sub_product

Coverage: Super

covered_by

Coverage: Fire

sub_cover

Size: 120m2

attribute

Risk: Building small

risk_has_attrcovers_risk

Quote: C12

includes

Customer: C12

is_offered

Policy: C12

is_offered

Agreement: C12

signed

User: U34

owns

based_on

contains

made

Questionaire: Q1

concerns

contains_question

owns

fills_in

includes

made

T&C: X

has


Recommendations, BI, Social Computing

35



35

๏enrich your CRM with data from Facebook, Google, Twitter etc



35


๏Recommender systems for products



35


๏Recommender systems for products

๏Find influencers in your customer base for special treatment


This is what your CRM sees

36http://inmaps.linkedinlabs.com/network

Customer1

Peter Neubauer


http://inmaps.linkedinlabs.com/network


This is what your CRM doesn’t see.

37http://inmaps.linkedinlabs.com/networkThursday, April 19, 12



This is what your CRM doesn’t see.

37http://inmaps.linkedinlabs.com/networkThursday, April 19, 12



Geospatial features

38


Geospatial features

38

๏Dynamic layers from different sources


Geospatial features

38


• domain data -> flood area layer + crime index + firestation + living standard index


Geospatial features

38


• domain data -> flood area layer + crime index + firestation + living standard index

๏routes of low insurance risks


Geospatial features

39


Geospatial features

40


Geospatial features

41


Configuration/Network Management

42



42

๏Model physical and logical networks



42


• impact analysis



42


• impact analysis

• configuration management



42


• impact analysis

• configuration management

• network inventory



43


44

Questions!


45

and, Thanks :)
