Download pdf - PowerOfRelationshipsInBigData_SVNoSQL

The Database

© Objectivity, Inc. 2014

The Power of Relationships in Big Data

Leon Guzenda - Objectivity, Inc. Silicon Valley NoSQL Meetup - 1/23/14


Overview

• The Problem

• Current Big Data Analytics

• Relationship Analytics

• Leveraging NoSQL

• Big Data Connection Platform

• Solution Use Case Demo!2


Objectivity, Inc.

• Headquartered in San Jose, CA • Over two decades of NoSQL and Big Data experience • Enables complex data virtualization and Big Data

solutions for the enterprise • Software products:

• Objectivity/DB • InfiniteGraph • InfiniteGraph Social App

• Embedded in hundreds of enterprises, government organizations and products, with millions of deployments.

!3


A Typical Deployment

!4


Current Big Data Analytics

!5


The ProblemInformation Overload!

• Making sense of it all takes time and $$$

• Which lead to a rush to Big Data Analytics

!6


Current Big Data Analytics

!7


Leveraging NoSQL

!8


Not Only SQL - Four Main Technologies

!9

SimpleHighly Interconnected


Hadoop?

!10

Objectivity/DB & InfiniteGraph

• Distributed processing with multithreading client processes and simple servers*

• Distributed, segmented Federated Database with a Single Logical View down to fine grain objects

• Tuned for random access and powerful parallel queries

• Excel at handling very large graph structures with built-in relationship analytics

Hadoop:

• Parallel processing using a divide and conquer or split and merge paradigm

• Sharded, distributed file system

• Tuned for sequential scans and simple queries

• Not suitable for highly interconnected data sets (graphs)

* Process workflow could be driven using MapReduce


Incremental Analytic Improvements Aren’t Enough

• All current solutions use the same basic architectural model.

• None of the current solutions has an efficient way to store connections between entities in different silos.

• Most analytic technology focuses on the content of the data nodes, rather than the many kinds of connections between the nodes and in those connections.

• Why? Because most DBMSs are bad at handling relationships.

• Object and Graph Databases can efficiently store, manage and query the many kinds of relationships hidden in the data.

!11


Relationship Analytics…A SQL Shortcoming

!12

Table_A Table_B Table_C Table_D Table_E Table_F Table_G

There are some kinds of complex relationship handling problems that SQL wasn't designed for.


…Relationship AnalyticsA SQL Shortcoming

!13

Table_A Table_B Table_C Table_D Table_E Table_F Table_G

InfiniteGraph - The solution can be found with a few lines of code

A3 G4


Graph Terminology

!14

● VERTEX: A single node in a graph data structure

● EDGE: A connection between a pair of VERTICES

● PROPERTIES: Data items that belong to a particular Vertex or Edge

● WEIGHT: A quantity associated with a particular Edge

● GRAPH: A network of linked Vertex and Edge objects

Vertex 1 Vertex 2Edge 1

City: San Francisco Pop: 812,826

City: San Jose Pop: 967,487

Road: I-101Miles: 47.8


Example 1 - Relationship Analytics

!15

LOGISTICS HEALTHCARE INFORMATICS

MARKET ANALYSIS SOCIAL NETWORK ANALYSIS


Finding The Links…

!16

Combatant A

Civilian Q

Situation Y

Civilian P

Bank X

Civilian S

Civilian R

Events/Places People/Orgs Facts

Situation X

Target T

Cafe C S Seen Near TA Banks at X

A Called P

A Seen At Y

A Seen Near X P Emailed S

P Called Q Q Seen Near T

P Called R R Seen Near T

X Paid S

A Eats At


…Finding The Links…

!17

Combatant A

Civilian Q

Situation Y

Civilian P

Civilian S

Civilian R

Events/Places People/Orgs Facts

Situation X

Target T

VERTICES EDGES

S Seen Near TA Banks at X

A Called P

A Seen At Y

A Seen Near X P Emailed S

P Called Q Q Seen Near T

P Called R R Seen Near T

X Paid SBank X

Cafe C

A Eats At



!18

Situation X Combatant ASeen Near

Civilian P

Called

Called

Seen At Situation Y

Civilian Q

Target T

Seen Near

Emailed

Banks At

Bank X

Civilian S

Seen Near

Called

Civilian R

Seen Near

Paid

Eats At

Cafe C



!19

Situation X Combatant ASeen Near

Civilian P

Called

Called

Seen At Situation Y

Civilian Q

Target T

Seen Near

Emailed

Banks At

Bank X

Civilian S

Seen Near

Called

Civilian R

Seen Near

Paid

SUSPECTS

NEEDS PROTECTION


…Finding The Links

!20

OTHER DATABASE(S)

GRAPH DATABASE


Example 2 - Finding Patterns in Open Source Data

!21

● Data Volumes

● Fast-Changing Data

● Sensitivity of Data

● Significance of Data

The Challenges


Example 3 - Cybersecurity

!22


Big Data Connection Platform

!23


Objectivity’s Disruptive Big Data Architecture

!24

Uses Data Virtualization to hide the nodes and focus on the connections


InfiniteGraph

!25

Distributed Parallel Load and Queries

Distributed Parallel Link Finding

Start

Start

Powerful Graph Queries

X

XStart

Finish

Computational and Visualization Plugins

Start

Latency Exceeded

Custom Visualizer


Solution Use Case Demo…

Let’s see InfiniteGraph coupled with Oracle’s NoSQL Solution…

!26