Budapest University of Technology and EconomicsDepartment of Measurement and Information SystemsDepartment of Telecommunications and Media Informatics
Budapest University of Technology and EconomicsMcGill University, Montréal
Incremental Graph Queries for Cypher
Gábor Szárnyas, József Marton
Live railway model
Live railway model
Live railway model
Live railway model
Live railway model
Proximity detection
Live railway model
Proximity detection
Live railway model
Proximity detection
Live railway model
Trailing the switch
Proximity detection
Live railway model
Live railway model
Live railway model
c d e
g
fdiv
2
a b
1
Live railway model
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
ON
a b
1
NEXT
ON
NEXT
Proximity detection
Proximity detection
≤ 𝟐 segments
Proximity detection
seg1
NEXT: 1..2
t1
ON
Proximity detection
seg2
t2
ON≤ 𝟐 segments
Proximity detection
seg1
NEXT: 1..2
t1
ON
MATCH
(t1:Train)-[:ON]->(seg1:Segment)
-[:NEXT*1..2]->(seg2:Segment)
<-[:ON]-(t2:Train)
RETURN t1, t2, seg1, seg2
Proximity detection
seg2
t2
ON≤ 𝟐 segments
Proximity detection
seg1
NEXT: 1..2
t1
ON
MATCH
(t1:Train)-[:ON]->(seg1:Segment)
-[:NEXT*1..2]->(seg2:Segment)
<-[:ON]-(t2:Train)
RETURN t1, t2, seg1, seg2
Proximity detection
seg2
t2
ON≤ 𝟐 segments
Trailing the switch
Trailing the switch
seg div
t
STRAIGHT
ON
Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Evaluate continuously
Incremental queries
Incremental queries
Register a set of standing queries
Continuously evaluate queries on changes
Incremental queries
Register a set of standing queries
Continuously evaluate queries on changes
The Rete algorithm (1974)
o Originally for rule-based expert systems
o Indexes the graph and caches interim query results
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
divSTRAIGHT
Trailing the switchO
N
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
a
1
ON
e
2
ON
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2ON
a1ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
a
1
ON
e
2
ON
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2ON
a1ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2ON
a1ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
e divSTRAIGHT
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2ON
a1ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
e divSTRAIGHT
e divSTRAIGHT
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
e2ON
a1ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
e2ON
a1ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
e divSTRAIGHT
e2ON
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
e2ON
a1ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
e divSTRAIGHT
e2ON
e div
2
STRAIGHT
ON
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
e2ON
a1ON
e divSTRAIGHT
2ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
e divSTRAIGHT
e2ON
e div
2
STRAIGHT
ON
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
e2ON
a1ON
divSTRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
e2 divSTRAIGHTON
e2
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
e2ON
a1ON
divSTRAIGHTON
e divSTRAIGHT
2ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
e2 divSTRAIGHTON
e2
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
e2ON
a1ON
e divSTRAIGHT
2ON
e divSTRAIGHT
2ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
div
2
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
e2ON
a1ON
e divSTRAIGHT
2ON
e divSTRAIGHT
2ON
div2
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
div
2
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
e2ON
a1ON
e divSTRAIGHT
2ON
e divSTRAIGHT
2ON
div2
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXTO
N
ON
divSTRAIGHT
Trailing the switchO
N
div
2
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
e2ON
a1ON
e divSTRAIGHT
2ON
e divSTRAIGHT
2ON
div2
c e
g
fdivNEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXT
ON
divSTRAIGHT
Trailing the switchO
N
div
ON
2
d
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
d2ON
a1ON
e divSTRAIGHT
2ON
e divSTRAIGHT
2ON
div2
c e
g
fdivNEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXT
ON
divSTRAIGHT
Trailing the switchO
N
div
ON
2
d
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
d2ON
a1ON
e divSTRAIGHT
2ON
div2
c e
g
fdivNEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXT
ON
divSTRAIGHT
Trailing the switchO
N
div
ON
2
d
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
d2ON
a1ON
div2
c e
g
fdivNEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXT
ON
divSTRAIGHT
Trailing the switchO
N
div
ON
2
d
πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e divSTRAIGHT
d2ON
a1ON
c e
g
fdivNEXT NEXT
STRAIGHT TOPa b
1
NEXT NEXT
ON
divSTRAIGHT
Trailing the switchO
N
div
ON
2
d
Batch vs. incremental queries
Batch queries(pull / request-driven):
1. Client selects a query
2. Results are calculated
Query resultsobtained on demand
Batch vs. incremental queries
Batch queries(pull / request-driven):
1. Client selects a query
2. Results are calculated
Query resultsobtained on demand
Incremental queries(push / event-driven):
1. Client registers queries
2. Graph is changed
3. Results are maintained
4. Goto 2
Query results arealways available
Incremental query engines
CLIPS C structures NASA
Drools POJO Red Hat
VIATRA EMF BME / IncQuery Labs.
Incremental query engines
CLIPS C structures NASA
Drools POJO Red Hat
VIATRA EMF BME / IncQuery Labs.
INSTANS RDF Aalto University
i3QL POJO TU Darmstadt
IncQuery-D RDF BME
Incremental query engines
CLIPS C structures NASA
Drools POJO Red Hat
VIATRA EMF BME / IncQuery Labs.
INSTANS RDF Aalto University
i3QL POJO TU Darmstadt
IncQuery-D RDF BME
No implementations for property graphs yet
ingraph
An incremental, in-memory graph query engine
ingraph
An incremental, in-memory graph query engine
ingraphclient
ingraph
An incremental, in-memory graph query engine
ingraphclient
register queries
ingraph
An incremental, in-memory graph query engine
ingraphclient
register queries
update graph
ingraph
An incremental, in-memory graph query engine
ingraphclient
register queries
query results
update graph
ingraph
An incremental, in-memory graph query engine
ingraphclient
register queries
query results
change notifications
update graph
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
openCypherquery
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
openCypherquery
Querysyntax tree
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Queryparser
openCypherquery
Querysyntax tree
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Queryparser
openCypherquery
Relationalalgebramodel
Querysyntax tree
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Relationalalgebra builder
Queryparser
openCypherquery
Relationalalgebramodel
Querysyntax tree
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Relationalalgebra builder
Queryparser
openCypherquery
Relationalalgebramodel
Querysyntax tree
Relationalalgebra model
Rete network
Rete network model
Relationalalgebra model
Rete network
Rete network model
Transformerand optimizer
VIATRA
Relationalalgebra model
Rete network
Rete network model
Transformerand optimizer
VIATRA
Relationalalgebra model
Rete network
Rete network model
Transformerand optimizer
Querydeployer
VIATRA
FORMALIZATION OF OPENCYPHER
Relational algebra
Standard relational algebra
o 𝜋, 𝜎
o∪,∩,∖
o×,⋈
Common extensions
o 𝛿 – duplicate elimination
o 𝛾 – grouping
o 𝜏 – sorting
Graph-specific operators
Jürgen Hölsch, Michael Grossniklaus:An Algebra and Equivalences to Transform Graph Patterns in Neo4j, GraphQ 2016, EDBT,http://ceur-ws.org/Vol-1558/paper24.pdf
GetVertices: returns a graph relation containing all vertices of the underlying graph G
Expand: return the neighbors of a given node
Additional extensions
GetEdges: returns a graph relation containing all edges of the underlying graph G
Gábor Szárnyas, József Marton:openCypher specification, Technical reporthttp://docs.inf.mit.bme.hu/ingraph/pub/opencypher-report.pdf
While pattern matching, Neo4j makes sure to not include matches where the same graph relationship is found multiple times in a single pattern.
Uniqueness of edges
All-different operator
EXAMPLE QUERIES
Uniqueness of edges
Get neighbours
Filter out based on node prop name
Use multiple MATCH clauses to do a Cartesian product
Two subgraphs
SwitchMonitored
INCREMENTAL GRAPH QUERIESWITH OPENCYPHER
openCypher constructs
Standard constructs
o pattern matching
o filtering
o lists, maps
o data manipulation
o variable length paths
openCypher constructs
Standard constructs
o pattern matching
o filtering
o lists, maps
o data manipulation
o variable length paths
Legacy constructs
o indexing, constraints
o regular expressions
o some list functions, including reduce
omost predicate functions
o shortest path functions
o CASE expressions
o id()
openCypher constructs
Standard constructs
o pattern matching
o filtering
o lists, maps
o data manipulation
o variable length paths
Legacy constructs
o indexing, constraints
o regular expressions
o some list functions, including reduce
omost predicate functions
o shortest path functions
o CASE expressions
o id()Difficult to handle
incrementally
Challenges for incremental openCypher
Lists
o ['a', 1, 2, true]
o ['a', [1, [2]], true]
o UNWIND
Efficient aggregation
o min(), max()
o collect()
Bag semantics, ORDER BY, SKIP and LIMIT
o Idea: collect(x ORDER BY x.name)
Incremental queries – use cases
Standing queries on large & quickly changing graph
Runtime monitoring (train example)
Model validation: The Train Benchmark
Static analysis of JavaScript source code
Fraud detection
IT infrastructure monitoring
Future work
Path operatorGiacomo Bergami, Matteo Magnani, and Danilo Montesi, A join operator for property graphs, GraphQ 2016, EDBT, http://jackbergus.alwaysdata.net/paper_graph_graphq2017.pdf
Unwind operator: 𝜔Elena Botoeva et al.:A Formal Presentation of MongoDB,https://arxiv.org/abs/1603.09291
Open-Source Projects
Incremental Graph Engine:https://github.com/ftsrg/ingraph
Train Benchmark:https://github.com/ftsrg/trainbenchmark
BME-MODES3:https://github.com/ftsrg/bme-modes3
Available under EPL v1.0.