Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems

Optimization of Incremental Queries in the Cloud

József Makai, Gábor Szárnyas, Ákos Horváth, István Ráth, Dániel Varró

Budapest University of Technology and EconomicsFault Tolerant Systems Research Group

INCQUERY-D: DISTRIBUTED INCREMENTAL MODEL QUERIES

Incremental Query Evaluation by RETE AUTOSAR well-formedness validation rule

Communication channel

Logical signal Mapping Physical signal

Invalid model fragment

Instance model

Valid model fragment

Fill the input nodesFill the worker nodesRead the result setModify the modelPropagate the changesRead the changes in the result set (deltas)

Incremental Query Evaluation by RETE

join

join

antijoin

Result set

input nodes

Communication channel

Logical signal Mapping Physical signal

worker nodes

Goals of IncQuery-D Objectives

o Distributed incremental pattern matchingo Adaptation of IncQuery tooling to graph DBso Executed over cloud infrastructure (COTS hardware)

Achieve scalability by avoiding memory bottlenecko Sharding separately• Data• Indexers• Query network

o In memory: • Index + Query

Assumptions• All Rete nodes fit on a server node• Indexers can be filled efficiently• Modification size model size≪• The application requires the complete result

set of the query (opposed to just one match)

Database shard 0

INCQUERY-D Architecture

Server 1

Database shard 1

Server 2

Database shard 2

Server 3

Database shard 3

Transaction

Server 0

Rete net

Indexer layer

INCQUERY-D

Distributed query evaluation network

Distributed indexer Model access adapter

Distributed indexing, notification

Distributed persistent storage

Distributed production network• Each intermediate node can be allocated

to a different host• Remote internode communication

INCQUERY-D Architecture

Server 1

Database shard 1

Server 2

Database shard 2

Server 3

Database shard 3

Transaction

In-memory EMF modelDatabase shard 0

Server 0

Indexer layer

INCQUERY-D

Indexer Indexer Indexer Indexer

JoinJoin

AntijoinAkka

Triple store (4store),Document DB (Mongo),RDF over Column family

(Cumulus)

RETE Deployment ProcessQuery

Language

Query Predicates

RETE Structure

Platform Description

Allocation / Mapping

Deployment Descriptor

pattern routeSensor(sensor: Sensor) = { TrackElement.sensor(switch,sensor); Switch(switch); SwitchPosition. switch(sp, switch); SwitchPosition(sp); Route.switchPosition(route, sp); Route(route); neg find head(route, sensor); }pattern head(R, Sen) = { Route.routeDefinition(R, Sen);}

route: Route sp: SwitchPosition

Switch: Switchsensor: Sensor

switchPosition

switchsensor

routeDefinition

RETE Deployment Process Construct language-

independent constraints Resolution of

o syntactic sugar o type information

Query Language

Query Predicates

RETE Structure

Platform Description

Allocation / Mapping

Deployment Descriptor

Variables route sp switchParameter sensor

Constraints

Edge: SwitchPosition.switch Edge: TrackElement.sensor Edge: Route.switchPosition Negation: head

RETE Deployment Process Construct RETE structure

(platform independently) Optimizations:

o Model statisticso Expected usage profile

Query Language

Query Predicates

RETE Structure

Platform Description

Allocation / Mapping

Deployment Descriptor

join

join

join

RETE Deployment Process Architecture model

(Cloud infrastructure)o Virtual Machines

• Memory limits• CPU speed• Storage capacity

o Communication Channels• Bandwidth

Specified by a textual DSL (Xtext)

Query Language

Query Predicates

RETE Structure

Platform Description

Allocation / Mapping

Deployment Descriptor

1 2

3 4

RETE Deployment ProcessMachine Allocated Nodes

1 In1, In2, Join2

2 In3

3 In4

4 Join1, Join3

Query Language

Query Predicates

RETE Structure

Platform Description

Allocation / Mapping

Deployment Descriptor

1 2

3 4

Join1

Join3

Join2

In1 In2 In3 In4

Allocation can be optimized for query performance and other

beneficial system characteristics!

RETE Deployment Process Configuration scripts for

o Deploymento Communication

middleware Derived by automated

code generationo Using Eclipse technology:

EMF-IncQuery + Xtend

Query Language

Query Predicates

RETE Structure

Platform Description

Allocation / Mapping

Deployment Descriptor

ALLOCATION OPTIMIZATION IN INCQUERY-D

Motivation for Allocation Optimization Considering data-intensive

systemso Over usage of resourceso Cost of the systemo Overhead of network

communication

Job Job

tLocal job

execution time

t’Data transmission time is significant component in

global execution time

~

Job

Job

Job

Network links can have different capacities

4000 MBProcess2000 MB

Process500 MB

Process2400 MB

$$$

Poor utilization leads to expensive system

The Allocation Problem Inputs Allocation constraints Output: Valid allocation Optimization targets

500 MB

3200 MB

2400 MB600 MB

Worker node

Input nodeInput node

Production node

1 2

3

4

5000 MB6000 MB

1 2

• Rete network for the query organized to processes

• Resource consumption

Available infrastructure with important resource parameters

Opt. Target: Communication Minimization

1 × 1,000,000

3 × 200,000 3 × 200,000

� Communication = 2,200,000

6000 MB

5000 MB

1

2500 MB

3200 MB

2400 MB600 MB

Worker node

Input nodeInput node

Production node

1,000,000200,000

200,000

1 2

3

4

3 × 1,000,000

1 × 200,000

1 × 200,000

� Communication = 3,400,000

5000 MB

6000 MB

1

2

Largest volume of data is sent through faster local link

Opt. Target: Cost Minimization

500 MB

3200 MB

2400 MB600 MB

Worker node

Input nodeInput node

Production node

1 2

3

4

4000 MB$5

4000 MB$5

6500 MB$7

1

2

3

� Cost = 10

4000 MB$5

4000 MB$5

6500 MB$7

1

2

3

� Cost = 12

Heuristics in Optimization

Worker node

Production node

Input node

Worker node

Input nodeInput node

Worker node

Production node

Production node

Worker node

Model database

Number of model elements

?? MBInput node

Memory consumption of Rete nodes and processes

1 1 11 1 1

1

Memory usage of Input nodes can be estimated

Communication intensity of network

communication channels2 2

2

2

2 2

3 3

3

3 3

4 4

Performance Impact of Optimization

61K 213K 867K 3M 13MModel size (number of elements)

Tim

e (s

ec)

First evaluation time of a complex query

28

45

72

114

182

290

463

739

Max. memory

Naiveoptimization

Communicationoptimization

739

616

194

144

2 minutes gain!

This approach doesn’t work for larger models!

Network Traffic Statistics

vm0 vm1 vm2 total vm0 vm1 vm2 total0

200

400

600

800

1000

1200

300 349 371

1020

248 280347

875

142

74

90

24 20

190

234

Network Traffic in Megabytes

Remote Local

Unoptimized Optimized

Unoptimized:o Remote Traffic:

1020o Local Traffic: 90o Total Traffic: 1110

Optimized:o Remote Traffic:

875o Local Traffic: 234o Total Traffic: 1109

Conclusion and Future Work Results

o Novel approach for application-specific resource allocation optimization for distributed Rete

o CPLEX-based implementation for IncQuery-Do Preliminary evaluation results

• Significant improvements for local resource management• Performance gains especially over slow / inhomogeneous networks• Efficient optimization execution (supported by runtime cutoff in CPLEX)

Future worko Hadoop / YARN support (new IncQuery-D developments)

• Support configuration optimization for other Hadoop-based cloud apps

o Static allocation Dynamic reallocation• Take existing configuration as a starting constraint set• Optimize for changed workload conditions

New INCQUERY-D Architecture

Docker container 1

Database shard 1

Docker container 2

Database shard 2

Docker container 3

Database shard 3

Transaction

In-memory EMF modelDatabase shard 0

Docker container 0

Indexer layer

New INCQUERY-D: “Hadoop over Docker”

Indexer Indexer Indexer Indexer

JoinJoin

Antijoin

• YARN resource management

• ZooKeeper monitoring

Akka actors embedded into long-running Hadoop jobs