Sharded Joins for Scalable Incremental Graph Queries

Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems

Budapest University of Technology and EconomicsFault Tolerant Systems Research Group

Sharded Joins for Scalable Incremental Graph Queries

János Maginecz, Gábor Szárnyas

Agile Model-Driven Development

Modeling


Modeling

Early validationsTransformations


Modeling

Codegeneration



Modeling

Codegeneration

Testing



Modeling

Codegeneration

Testing



Modeling

Codegeneration

Testing


Scalabilitychallenges

Performance issues


Modeling

Codegeneration

Testing



Performance issues


Modeling

Codegeneration

Testing



Scalability

Incrementality

Scalability

Incrementality

Storing partialresults

Scalability

Incrementality


Trackingchanges

MDD

Scalability

Incrementality


Trackingchanges

MDD

Scalability

Incrementality

Incremental queries


Trackingchanges

MDD

Scalability

Incrementality

Incremental queries

Incremental transformation


Trackingchanges

MDD

Scalability

Incrementality

Incremental queries


Trackingchanges

Motivating Example

Error pattern for an AUTOSAR validation constraint

Communicationchannel

Logical signal Mapping Physical signal

Motivating Example




Validation

Motivating Example




Invalid submodel

Validation

Motivating Example




Invalid submodel

Validation

Motivating Example




Invalid submodel

Validation

Valid submodel

Antijoin

Join

Join

Rete Algorithm

Communication channel


Antijoin

Join

Join

Rete Algorithm



Antijoin

Join

Join

Rete Algorithm



Antijoin

Join

Join

Fill indexer nodes

Rete Algorithm



Antijoin

Join

Join

Fill indexer nodes

Rete Algorithm



Antijoin

Join

Join

Fill indexer nodes

Rete Algorithm



Antijoin

Join

Join

Fill indexer nodes

Rete Algorithm



Antijoin

Join

Join

Fill indexer nodes

Rete Algorithm



Antijoin

Join

Join

Fill indexer nodesStore interim results

Rete Algorithm



Antijoin

Join

Join


Rete Algorithm



Antijoin

Join

Join


Rete Algorithm



Antijoin

Join

Join


Rete Algorithm



Antijoin

Join

Join


Rete Algorithm



Antijoin

Join

Join


Rete Algorithm



Antijoin

Join

Join


Rete Algorithm



Antijoin

Join

Join

Fill indexer nodesStore interim resultsRead result set

Rete Algorithm



Result set

Antijoin

Join

Join

Fill indexer nodesStore interim resultsRead result set

Rete Algorithm



Antijoin

Join

Join

Fill indexer nodesStore interim resultsRead result setEdit model

Rete Algorithm



Antijoin

Join

Join

Fill indexer nodesStore interim resultsRead result setEdit modelPropagating changes

Rete Algorithm



Antijoin

Join

Join

Fill indexer nodesStore interim resultsRead result setEdit modelPropagating changes

Rete Algorithm



Antijoin

Join

Join

Fill indexer nodesStore interim resultsRead result setEdit modelPropagating changesRead result set

Rete Algorithm



Result set

Single Workstation Rete Implementation

Rete-based incremental graph query engine

Open-source Eclipse project

Java Virtual Machine limitations

o Cannot handle 15+ GB heap memory efficiently

Proposed solution

o Horizontal scaling: distributed system

IncQuery-D Architecture

Transaction

In-memory EMF model

Rete net

Indexer layer

EMF-INCQUERY


Transaction

In-memory EMF model

Rete net

Indexer layer

EMF-INCQUERY

In-memory storage


Transaction

In-memory EMF model

Rete net

Indexer layer

EMF-INCQUERY

Indexing

In-memory storage


Transaction

In-memory EMF model

Rete net

Indexer layer

EMF-INCQUERY

Indexing

In-memory storage

Production network• Stores intermediate query results• Propagates changes


Transaction

In-memory EMF model

Rete net

Indexer layer

EMF-INCQUERY


Server 1

Databaseshard 1

Server 2

Databaseshard 2

Server 3

Databaseshard 3

Transaction

Databaseshard 0

Server 0

Rete net

Indexer layer

INCQUERY-D


Server 1

Databaseshard 1

Server 2

Databaseshard 2

Server 3

Databaseshard 3

Transaction

Databaseshard 0

Server 0

Rete net

INCQUERY-D


Server 1

Databaseshard 1

Server 2

Databaseshard 2

Server 3

Databaseshard 3

Transaction

Databaseshard 0

Server 0

Rete net

INCQUERY-D

Distributed indexer Model access adapter


Server 1

Databaseshard 1

Server 2

Databaseshard 2

Server 3

Databaseshard 3

Transaction

Databaseshard 0

Server 0

INCQUERY-D

Distributed query evaluation network



Server 1

Databaseshard 1

Server 2

Databaseshard 2

Server 3

Databaseshard 3

Transaction

Databaseshard 0

Server 0

INCQUERY-D



Distributed persistent storage


Server 1

Databaseshard 1

Server 2

Databaseshard 2

Server 3

Databaseshard 3

Transaction

Databaseshard 0

Server 0

INCQUERY-D



Distributed indexing, notification



Server 1

Databaseshard 1

Server 2

Databaseshard 2

Server 3

Databaseshard 3

Transaction

Databaseshard 0

Server 0

INCQUERY-D



Distributed indexing, notification


Distributed production network• Each intermediate node can be allocated

to a different host• Remote internode communication


Server 1

Databaseshard 1

Server 2

Databaseshard 2

Server 3

Databaseshard 3

Transaction

Databaseshard 0

Server 0

INCQUERY-D




Server 1

Databaseshard 1

Server 2

Databaseshard 2

Server 3

Databaseshard 3

Transaction

Databaseshard 0

Server 0

INCQUERY-D


Indexer Indexer Indexer Indexer

Join

Join

Antijoin

Working around Memory Limits

Host-2

Host-1

Input

Node A

Node B

Distributed

Output

Host-1

Input

Node A

Node B

Local

Output

Solution 1

Simple and efficientMemory of the machine is an upper bound for the network

Nodes run on different computersThe memory of each node is limited to the assigned machine

+

−

+

−

Working around Memory Limits

Host-2

Host-1

Input

Node A

Node B

Distributed

Output

Host-1

Input

Node A

Node B

Local

Output

Solution 1

EMF-IncQuery IncQuery-D

Simple and efficientMemory of the machine is an upper bound for the network


+

−

+

−

Host-3Host-1

Host-2

Working around Memory LimitsDistributed

+Sharded

Input

Node A

Node B

Output

Solution 2

Host-2

Host-1

Input

Node A

Node B

Distributed

Output

Nodes may be allocated on more than 1 computerNetwork overhead

+

−


+

−

Host-3Host-1

Host-2

Working around Memory LimitsDistributed

+Sharded

Input

Node A

Node B

Output

Solution 2

IncQuery-DS

Host-2

Host-1

Input

Node A

Node B

Distributed

Output

IncQuery-D

Nodes may be allocated on more than 1 computerNetwork overhead

+

−


+

−

Í

Join

Antijoin

Join / Shard 2Join / Shard 1

Sharded Rete Algorithm



Í

Join

Antijoin





Í

Join

Antijoin





Í

Join

Antijoin





Í

Join

Antijoin





Í

Join

Antijoin





IncQuery-DSDistributed and Sharded

Validation of Critical Systems

Model validation for large models

Well-formedness contraints with complex graph patterns

Train Benchmark

o Open-source performance measurement framework

o Presented yesterday

Train Benchmark

Phases

o Initial read and validation

o Small changes and revalidation

• Simulating modifications from a user

Goal: Measure response times

Execution timeExecution time

Read Transformation RevalidationValidation

× 10× 3

Sharding Results

Sharding Results

Sharding Results

Join Optimization

Hash join

o Using hash maps

Sort merge join

o Using red-black trees

Collection frameworks

o Standard library in Scala

o Goldman Sachs Collections

Join Optimization Results – Execution Time

Join Optimization Results – Execution Time

Does not scalewell for updates

Summary Designed a sharded Rete engine

Evaluated its scalability

Analysis of join algorithms and collection frameworks

Future work

o Domains with similar challenges

Ω

Engineering

Sharded Joins for Scalable Incremental Graph Queries