Dex Technical Seminar (April 2011)

DEX

Gra

ph

Data

base

http://www.sparsity-technologies.com

Technical Seminar

Sergio Gómez

April, 2011

DEX

Gra

ph

Data

base


Introduction

Basic Concepts

Database construction

Validate construction

Query database

Graph algorithms

Script loaders

Tips & tricks

Index

DEX

Gra

ph

Data

base


Index

Introduction

Basic Concepts



Query database

Graph algorithms

Script loaders

Tips & tricks

DEX

Gra

ph

Data

base


Introduction

Graph database

Graph databases focus on the structure of the model. Nodes and edges instead of tables. Implicit relation in the model.

DEX is a programming librarywhich allows to manage agraph database.

Very large datasets. High performance

query processing.

DEX

Gra

ph

Data

base


IntroductionDEX Definition

Persistent and temporary graph management programming library.

Data model: Typed and attributed directed multigraph.

Typed: Node and edge instances belong to a type (label). Attributed: Node and edge instances may have attribute

values. Directed: Edge can be directed or undirected. Multigraph: Multiple edges between two nodes.

Type of edges: Materialized: directed and undirected. Virtual: constrained by the values of two attributes

(foreign keys) Just for navigation

DEX

Gra

ph

Data

base


IntroductionGraph Model

DEX

Gra

ph

Data

base


Introduction

Basic Concepts



Query database

Graph algorithms

Script loaders

Tips & tricks

Index

DEX

Gra

ph

Data

base


Basic Concepts

Java library: jdex.jar public API Native library

Linux: libjdex.so Windows: jdex.dll

System requirements: Java Runtime Environment, v1.5 or higher. Operative system:

Windows – 32 bits (64 bits to be supported in new release)

Linux – 32 and 64 bits Soon to announce Mac OS

DEX

Gra

ph

Data

base


Basic Concepts

DEX Session1

N

DbGraph11

RGraph

1

N

Graph

Graph factory Persistent DB

TemporaryObjects

1

N

GraphPool

1

N

Set of OIDs

Class Diagram

DEX

Gra

ph

Data

base


Basic ConceptsMain methods

DEX

open(filename) GraphPoolcreate(filename) GraphPoolclose()

GraphPoolnewSession() Session

Session

getDbGraph() DbGraphnewGraph() Rgraphclose()

Graph

newNodeType(name) intnewEdgeType(name) intnewNode(type) longnewEdge(type) longnewAttribute(type, name) longsetAttribute(oid, attr, value)getAttribute(oid, attr) value

select(type) Objectsselect(attr, op, value) Objectsexplode(oid, type) Objects

Objects.Iterator

hasNext() booleannext() long

Objects

add(long)exists(long)

copy(objs)union(objs)Intersection(objs)difference(objs)

DEX

Gra

ph

Data

base


Index

Introduction

Basic Concepts



Query database

Graph algorithms

Script loaders

Tips & tricks

DEX

Gra

ph

Data

base


Database constructionGraph

DEX: Loads library and manages graph db instances. Graph Pool: Manages a graph db instance. Session: Manages a set of queries and temporary data.

Nodes & Edges Type:

DEX identifier (integer) Public identifier (string)

Instance: DEX identifier (long) – OID belongs to a type

Attributes Attribute:

DEX identifier (long) public identifier (string) Scope: type or global

DEX

Gra

ph

Data

base



GraphPool DEX#create(String name)Creates a new graph database instance.Returns the GraphPool instance to manage the new graph database instance.

GraphPool DEX#open(String name)Opens an existing graph database instance.Returns the GraphPool instance to manage the existing graph database instance.

Session GraphPool#newSession()Initiates a new user Session.

DbGraph Session#getDbGraph()Gets the Graph instance representing the persistent graph

database.

Create a graph database

DEX

Gra

ph

Data

base



import edu.upc.dama.dex.core.*;…DEX dex = new DEX();GraphPool gpool = dex.create(“C:/image.dex”);Session s = gpool.newSession();……s.close();gpool.close();dex.close();

Create a graph database example

DEX

Gra

ph

Data

base



int Graph#newNodeType(String name)Creates a new node type with the given unique name.Returns the DEX node type identifier.

long Graph#newNode(int nodeType)Creates a new node belonging to the given node type. Returns the DEX object identifier.

Add nodes

DEX

Gra

ph

Data

base



int Graph#newEdgeType(String name, bool directed)Creates a new edge type with the given unique name. Directed

or undirected edge type. Returns the DEX edge type identifier.

int Graph#newRestrictedEdgeType(String name, int srcNodeType, int dstNodeType)

Creates a new directed edge type with the given unique name.(Integrity restriction) Source and destionation of the edge

instances are restricted to the given node types.Returns the DEX edge type identifier.

long Graph#newEdge(long tail, long head, int edgeType)

Creates a new edge belonging to the given edge type. Tail is the source and head is the target (iff directed).Returns the DEX edge identifier.

Add edges

DEX

Gra

ph

Data

base


…DbGraph dbg = s.getDbGraph();

int person = dbg.newNodeType(“PERSON”);long p1 = dbg.newNode(person);long p2 = dbg.newNode(person);long p3 = dbg.newNode (person);

int friend = dbg.newUndirectedEdgeType(“FRIEND”);long e1 = dbg.newEdge(p1, p2, friend);long e2 = dbg.newEdge(p2, p3, friend);int loves = dbg.newEdgeType(“LOVES”);long e3 = dbg.newEdge(p1, p3, loves);…

Add nodes and edges example


p1

p2

p3

p1

p2

p3

DEX

Gra

ph

Data

base



class ValueEncapsulates a value and its domain (data type): String, Integer, Long, Double, Boolean, Timestamp. Use them to set and get attribute values for the objects.

long Graph#newAttribute(int type, String name, short dataType, short kind)Creates a new attribute with the given unique name for the given node or edge type. Returns the DEX attribute identifier.

“dataType” can be: Value#STRING, Value#INT, Value#LONG, Value#DOUBLE, Value#BOOL, Value#TIMESTAMP.“kind” can be:

Graph#ATTR_KIND_BASIC. Basic attribute (just set and get values).Grahp#ATTR_KIND_INDEXED. Indexed attribute (set and get values as well as select operations)Graph#ATTR_KINDUNIQUE. Indexed attribute. Unique (PK).

Manage attributes

DEX

Gra

ph

Data

base



Graph#setAttribute(long oid, long attr, Value v)Sets the given Value for the given attribute to the given object identifier. Given attribute must be defined for the object’s type. Value ‘s data type must match attribute’s data type or NULL.

Graph#getAttribute(long oid, long attr, Value v)Gets the Value for the given attribute and for the given object identifier. Given attribute must be defined for the object’s type.

Manage attributes

DEX

Gra

ph

Data

base



…long name = dbg.newAttribute(person, “NAME”, Value.STRING);long age= dbg.newAttribute(person, “AGE”, Value.INT);Value v = new Value();

dbg.setAttribute(p1, age, v.setInt(18));dbg.setAttribute(p2, name, v.setString(“KELLY"));dbg.setAttribute(p3, name, v.setString(“MARY"));…

long since = dbg.newAttribute(friend, “SINCE”, Value.INT);

dbg.setAttribute(e1, since, v.setInt(2000));dbg.setAttribute(e2, since, v.setInt(1995));…

JOHN18

KELLY

MARY

1995

2000JOHN18

KELLY

MARY

Manage attributes example

DEX

Gra

ph

Data

base



…int phones = dbg.newEdgeType("phones");long when = dbg.newAttribute(phones, "when", Value.STRING);

long e4 = dbg.newEdge(p1, p3, phones);dbg.setAttribute(e4, when, v.setString("4pm")));

long e5 = dbg.newEdge(p1, p3, phones);dbg.setAttribute(e5, when, v.setString("5pm"));

long e6 = dbg.newEdge(p3, p2, phones);dbg.setAttribute(e6, when, v.setString("6pm"));…

System.out.println(dbg.getAttribute(p1, name));System.out.println(dbg.getAttribute(e4, when));System.out.println(dbg.getAttribute(e5, when));

1995

2000JOHN18

KELLY

MARY

4pm5pm

6pm

Manage attributes example

DEX

Gra

ph

Data

base


Index

Introduction

Basic Concepts



Query database

Graph algorithms

Script loaders

Tips & tricks

DEX

Gra

ph

Data

base



GraphPool#dumpData(File f)Dumps a summary of the logical content of the graph database.

GraphPool#dumpStorage(File f)Dumps internal information about storage content of the graph database.

Graph#export(PrintWriter pw, short kind, Export e)Exports the graph to an external format.“kind” can be: GRAPHVIZ or YGRAPHML.Export defines the visualization (if null, default export).

DEX

Gra

ph

Data

base



import java.io.PrintWriter;import java.io.FileWriter;…

//Default Export is used in the example by stating null. //Personalized export can be created implementing class Export PrintWriter stdOut = new PrintWriter(

new FileWriter("out.graphml"));dbg.export(stdOut, Type.YGRAPHML, null);pw.close();

//Dump data filegpool.dumpData(“out.data”);

//Dump internal storage filegpool.dumpStorage(“out.storage”);

Validate construction example

DEX

Gra

ph

Data

base


Validate constructionData file example

-- 1:2 = Friend-- 2:3 = Loves-- 3:4 = phonesedge types = 3-- 4:1 = Person------- #00000400 [Person] ATTR=1 //Person//age{18} TO [Friend] #00000401 [Person] ATTR=1 //Person//name{kelly} TO [Loves] #00000402 [Person] ATTR=1 //Person//name{mary} TO [phones] #00000402 [Person] ATTR=1 //Person//name{mary} #00000402 [Person] ATTR=1 //Person//name{mary}…------- 3 nodes.node types = 4

Edge types

# of edge types

Node of type person

Value for “age” attribute

# of node types

“Phones” out-going edges

DEX

Gra

ph

Data

base


Validate constructionInternal storage file example

2011-02-09 12:12:45.330 ...SIZE: 262KB / 2688KB NODES: 3 EDGES: 6 1. ... - NODES Person : N=3... ATTR DATA: 4 ... - //Friend//since[INDEXED|INDEXED LINK] 2/2... [1995] - [2000] - //Person//age[INDEXED|INDEXED LINK] 1/1 [18] - [18]...

Number of edgesNumber of nodes

Number of “Person”s

Number of attributesName and kind of the attribute

Minimum and maximum value

# different values / # values

DEX

Gra

ph

Data

base



Export file example, yEd Graph Editor<?xml version="1.0" encoding="ISO-8859-1"?><graphml xmlns="http://graphml.graphdrawing.org/xmlns/graphml"

xmlns:y="http://www.yworks.com/xml/graphml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns/graphml http://www.yworks.com/xml/schema/graphml/1.0/ygraphml.xsd"><key id="d0" for="node" yfiles.type="nodegraphics"/><key id="d1" for="edge" yfiles.type="edgegraphics"/><graph id="DB" edgedefault="undirected">

<node id="0"><data key="d0" ><y:ShapeNode><y:Geometry height="19.0" width="20"/><y:Fill color="#a5c3f6"/><y:BorderStyle color="#a5c3f6"/><y:NodeLabel fontSize="10" textColor="#000000">1024</y:NodeLabel><y:Shape type="rectangle"/> </y:ShapeNode></data></node>…</graph>

</graphml>

DEX

Gra

ph

Data

base


Index

Introduction

Basic Concepts



Query database

Graph algorithms

Script loaders

Tips & tricks

DEX

Gra

ph

Data

base


Query Database

Objects

class ObjectsUnordered set of OIDs for large collections.Implements Set<Long>, Iterable<Long>, Closeable.

boolean Objects#add(long oid)Adds the given OID to the collection.Returns true if added, false if the OID was already into the collection.

boolean Objects#exists(long oid)Returns true if the given OID exists into the collection, false otherwise.

boolean Objects#remove(long oid)Removes the given OID from the collection.Returns true if removed or false if the OID was not into the collection.

DEX

Gra

ph

Data

base


Query Database

Objects

long Objects#union(Objects objs)this = this UNION objsReturns the new size of the collection.

long Objects#intersection(Objects objs)this = this INTERSECTION objsReturns the new size of the collection.

long Objects#difference(Objects objs)this = this DIFFERENCE objsReturns the new size of the collection.

DEX

Gra

ph

Data

base


Query Database

Objects Graph#select(int t)Retrieves object identifiers belonging to the given node or edge type.

Objects Graph#select(long attr, short op, Value v)

Retrieves object identifiers which satisfay the query.“op” can be: Graph#OPERATION_{EQ|NE|GT|GE|LT|LE|LIKE|ERE}

long Graph#findObj(long attr, Value v)Randomly retrieves an object identifier which has the given value for the given attribute (or INVALID_OID if not found).Useful for unique atributes.

Retrieve data

DEX

Gra

ph

Data

base


Query Database

Objects Graph#explode(long oid, int edgeType, short direction)

Retrieves out-going or in-going edges (or both) from or to the given node identifier and for the given edge type.“direction” can be: Graph#EDGES_IN, Graph#EDGES_OUT, Graph#EDGES_BOTH.

Objects Graph#neighbors(long oid, int edgeType, short direction)

Retrieves neighbor nodes to the given node identifier which can be reached through the given edge type and direction.“direction” can be: Graph#EDGES_IN, Graph#EDGES_OUT, Graph#EDGES_BOTH.

Navigation

DEX

Gra

ph

Data

base


Query Database

…DbGraph dbg = s.getDbGraph();Objects persons = dbg.select(person);Objects.Iterator it = persons.iterator();while (it.hasNext()) {

long p = it.next();String name = dbg.getAttribute(p, name).toString();

}it.close();persons.close();…

JOHN18

KELLY

MARY

JOHN18

KELLY

MARY

1995

4pm

5pm

6pm

2000

Retrieve data example

DEX

Gra

ph

Data

base


Query Database

…Objects objs1 = dbg.select(when, >=, 5pm);// objs1 = { e5, e6 }Objects objs2 = dbg.explode(p1, phones, OUT);// objs2 = { e4, e5 }Objects objs = objs1.intersection(objs2);// objs = { e5, e6 } ∩ { e4, e5 } = { e5 }…objs.close();objs1.close();objs2.close();…

JOHN18

KELLY

MARY

JOHN18

KELLY

MARY

1995

4pm

5pm

6pm

2000

Navigation & Objects operations example

DEX

Gra

ph

Data

base


Index

Introduction

Basic Concepts



Query database

Graph algorithms

Script loaders

Tips & tricks

DEX

Gra

ph

Data

base


Graph Algorithms

Traversals

class TraversalImplements Iterator<Long>: hasNext(), next().Traverses the graph from a given node identifier.Node and edge types can be filtered out.

Class TraversalBFSUses a BFS (breadth-first search ) algorithm.

Class TraversalDFSUses a DFS (depth-first search ) algorithm.

DEX

Gra

ph

Data

base


Graph Algorithms

Traversals example

Graph graph = … // graph instancelong idsource = … // source nodeTraversalBFS bfs = new TraversalBFS(graph, idsource);

// Adding the allowed edge types for traversing the graph fs.addEdge(graph.findType("street"), Algorithm.NAVIGATION_FORWARD);bfs.addEdge(graph.findType("road"), Algorithm.NAVIGATION_FORWARD);

// Adding the allowed node types for traversing the graphbfs.addAllNodes();

// Running the algorithm long idnode;while (bfs.hasNext()) {

idnode = bfs.next();}

// Closing the traversalbfs.close();

DEX

Gra

ph

Data

base


Graph Algorithms

Shortest paths

class SinglePairShortestPathFinds the shortest path between two given node identifiers.Node and edge types can be filtered out.

Class SinglePairShortestPathBFSUses a BFS traversal to find the shortest path.For un-weighted edges.

Class SinglePairShortestPathDijkstraUses a Dijkstra algorithm to find the shortest path.For weighted edges.

DEX

Gra

ph

Data

base


Graph Algorithms

ShortestPath example

Graph graph = … // graph instancelong idsource = … // source nodelong iddestination = … // destination nodeSinglePairShortestPathBFS sp =

new SinglePairShortestPathBFS(graph, idsource, iddestination);

// Adding the allowed edge types for traversing the graphsp.addEdge(graph.findType("street"), Algorithm.NAVIGATION_FORWARD);sp.addEdge(graph.findType("road"), Algorithm.NAVIGATION_UNDIRECTED);

// Setting the maximum hopssp.setMaximumHops(7);

// Running the algorithmsp.run();

// Retrieving the generated resultsif (sp.existsShortestPath()) {

long[] spAsNodes = sp.getPathAsNodes();long[] spAsEdges = sp.getPathAsEdges();int hopsDone = sp.getCost();

}// Closing the instancesp.close();

DEX

Gra

ph

Data

base


Graph Algorithms

Connected components

class ConnectivityRetrieves connected components of the given graph.Node and edge types can be filtered out.Connected components can be materialized into an attribute.

class StrongConnectivityGabowRetrieves strong connected components in a directed graph.Uses the Gabow algorithm.

class WeakConnectivityDFSRetrieves weakly connected components in a undirected graph or in a directed graph (ignoring directions).Uses a DFS algorithm.

DEX

Gra

ph

Data

base


Index

Introduction

Basic Concepts



Query database

Graph algorithms

Script loaders

Tips & tricks

DEX

Gra

ph

Data

base


Script loaders

CREATE DBGRAPH alias INTO filename

CREATE NODE node_type_name "(“[attribute_name

(INT|LONG|DOUBLE|STRING|BOOLEAN|TIMESTAMP|TEXT)[INDEXED|UNIQUE|BASIC]

, ...]")“

CREATE [UNDIRECTED|VIRTUAL] EDGE edge_type_name[FROM node_type_name[.attribute_name] TO node_type_name[.attribute_name]] "(“

[attribute_name(INT|LONG|DOUBLE|STRING|BOOLEAN|TIMESTAMP|TEXT) [INDEXED|UNIQUE|BASIC]

, ...]") [MATERIALIZE NEIGHBORS]"

Schema definition

DEX

Gra

ph

Data

base


Script loaders

LOAD NODES file_nameCOLUMNS attribute_name [alias_name], …INTO node_type_name[IGNORE (attribute_name|alias_name), …][FIELDS

[TERMINATED char][ENCLOSED char][ALLOW_MULTILINE]]

[FROM num][MAX num][MODE (ROWS|COLUMNS [SPLIT [PARTITIONS num]])]

Load nodes

DEX

Gra

ph

Data

base


Script loaders

LOAD EDGES file_nameCOLUMNS attribute_name [alias_name], …INTO node_type_name[IGNORE (attribute_name|alias_name), …]WHERETAIL (attribute_name|alias_name) = node_type_name.attribute_nameHEAD (attribute_name|alias_name) = node_type_name.attribute_name[FIELDS

[TERMINATED char][ENCLOSED char][ALLOW_MULTILINE]]

[FROM num][MAX num][MODE (ROWS|COLUMNS [SPLIT [PARTITIONS num]])]

Load edges

DEX

Gra

ph

Data

base


Index

Introduction

Basic Concepts



Query database

Graph algorithms

Script loaders

Tips & tricks

DEX

Gra

ph

Data

base


Tips & tricks

Shell java –cp jdex.jar edu.upc.dama.dex.shell.Shell

Explore database: Schema Instances

Execute basic queries: Select Explode Neighbors

DEX

Gra

ph

Data

base


Tips & tricks

Index or not? Attributes:

Attributes used at select operations must be indexed.

Optionally, index once the attribute has been created/loaded.

Neighbors: Edge types used at neighbors

operations it is recommended to be indexed.

DEX

Gra

ph

Data

base


Tips & tricks

String attributes String

Maximum length = 2048. Indexed or not.

Select [==, !=, >, >=, <, <=, ERE] Text (Character large object)

Unlimited length. Not Indexed.

Just get and set. Streaming read and write operations.

DEX

Gra

ph

Data

base


Tips & tricks

Others: DB cross-platform format.

32 – 64 bits, OS independent. Just considere platform endianess.

Read only mode. Configuration:

edu.upc.dama.dex.utils.DEXConfig Set the maximum memory usage.

0 means unlimited. License.

No license means evaluation version.

DEX

Gra

ph

Data

base


Thanks for yourattention

Any questions?

SPARSITY-TECHNOLOGIESJordi Girona, 1-3, Edifici K2M

08034 [email protected]


Technology

Dex Technical Seminar (April 2011)