GraphConnect Europe 2016 - Tuning Your Cypher - Petra Selmer, Mark Needham

Tuning CypherMark Needham @markhneedham

Petra Selmer@Aethelraed

Why do we need to tune?

‣ No query planner is ever perfect‣ You know your domain better than the

database

The Cost planner

‣ Introduced in 2.2.0‣ It uses the statistics service in Neo4j to

assign costs to various query execution plans, picking the cheapest one

‣ All queries use this by default

Cypher query execution

‣ http://neo4j.com/docs/snapshot/execution-plans.html‣ http://neo4j.com/blog/introducing-new-cypher-query-optimizer

How do I view a query plan?

‣ EXPLAIN• shows the execution plan without actually

executing it or returning any results.

‣ PROFILE• executes the statement and returns the results

along with profiling information.

Neo4j’s longest plan (so far…)

Neo4j’s longest plan (so far…)

Neo4j’s longest plan (so far…)

What is our goal?

At a high level, the goal is simple: get the number of db hits down.

an abstract unit of storage engine work.

What is a database hit?

“”

‣ Operators to look out for• All nodes scan expensive

• Label scan cheaper

• Node index seek cheapest

• Node index scan used for range queries

‣ http://neo4j.com/docs/3.0.0-RC1/execution-plans.html

Execution plan operators

Our data set

Finding The Matrix

MATCH (movie {title: "The Matrix"})

RETURN movie

Finding The Matrix

MATCH (movie

{title: "The Matrix"})

RETURN movie

Tip: Use labels

MATCH (movie:Movie

{title: "The Matrix"})

RETURN movie

Tip: Use labels

MATCH (movie:Movie

{title: "The Matrix"})

RETURN movie

Finding The Matrix MATCH (movie

{title: "The Matrix"})

RETURN movie

MATCH (movie:Movie

{title: "The Matrix"})

RETURN movie

Tip: Use indexes and constraints

‣ Indexes for non unique values‣ Constraints for unique values

CREATE INDEX ON :Movie(title)

CREATE INDEX ON :Person(name)

CREATE CONSTRAINT ON (g:Genre)

ASSERT g.name IS UNIQUE

How does Neo4j use indexes?

‣ Indexes are only used to find the starting point for queries.

Use index scans to look up rows in tables and join them with rows from other tables

Use indexes to find the starting points for a query.

Relational

Graph

Tip: Use indexes and constraints

MATCH (movie:Movie

{title: "The Matrix"})

RETURN movie

Finding The Matrix (no index)MATCH (movie:Movie

{title: "The Matrix"})

RETURN movie

(index)MATCH (movie:Movie

{title: "The Matrix"})

RETURN movie

Actors who appeared together

MATCH (a:Person {name:"Tom Hanks"})

-[:ACTS_IN]->()<-[:ACTS_IN]-

(b:Person {name:"Meg Ryan"})

RETURN COUNT(*)

Actors who appeared together

MATCH (a:Person {name:"Tom Hanks"})

-[:ACTS_IN]->()<-[:ACTS_IN]-

(b:Person {name:"Meg Ryan"})

RETURN COUNT(*)

Tip: Enforce index usage

MATCH (a:Person {name:"Tom Hanks"})

-[:ACTS_IN]->()<-[:ACTS_IN]-

(b:Person {name:"Meg Ryan"})

USING INDEX a:Person(name)

USING INDEX b:Person(name)

RETURN COUNT(*)

Tip: Enforce index usage

MATCH (a:Person {name:"Tom Hanks"})

-[:ACTS_IN]->()<-[:ACTS_IN]-

(b:Person {name:"Meg Ryan"})

USING INDEX a:Person(name)

USING INDEX b:Person(name)

RETURN COUNT(*)

Actors who appeared togetherMATCH (a:Person {name:"Tom Hanks"})

-[:ACTS_IN]->()<-[:ACTS_IN]-

(b:Person {name:"Meg Ryan"})

RETURN COUNT(*)

MATCH (a:Person {name:"Tom Hanks"})

-[:ACTS_IN]->()<-[:ACTS_IN]-

(b:Person {name:"Meg Ryan"})

USING INDEX a:Person(name)

USING INDEX b:Person(name)

RETURN COUNT(*)

Tom Hanks’ colleagues’ movies

MATCH (p:Person {name:"Tom Hanks"})

-[:ACTS_IN]->(m1)<-[:ACTS_IN]-

(coActor)-[:ACTS_IN]->(m2)

RETURN distinct m2.title

Tom Hanks’ colleagues’ movies

MATCH (p:Person {name:"Tom Hanks"})

-[:ACTS_IN]->(m1)<-[:ACTS_IN]-

(coActor)-[:ACTS_IN]->(m2)

RETURN distinct m2.title

Tip: Reduce cardinality of WIP

MATCH (p:Person {name:"Tom Hanks"})

-[:ACTS_IN]->(m1)<-[:ACTS_IN]-

(coActor)

WITH DISTINCT coActor

MATCH (coActor)-[:ACTS_IN]->(m2)

RETURN distinct m2.title

Tip: Reduce cardinality of WIP

MATCH (p:Person {name:"Tom Hanks"})

-[:ACTS_IN]->(m1)<-[:ACTS_IN]-

(coActor)

WITH DISTINCT coActor

MATCH (coActor)-[:ACTS_IN]->(m2)

RETURN distinct m2.title

MATCH (p:Person {name:"Tom Hanks"})

-[:ACTS_IN]->(m1)<-[:ACTS_IN]-(coActor)

WITH DISTINCT coActor

MATCH (coActor)-[:ACTS_IN]->(m2)

RETURN distinct m2.title

Tom Hanks’ colleagues’ moviesMATCH (p:Person {name:"Tom Hanks"})

-[:ACTS_IN]->(m1)<-[:ACTS_IN]-

(coActor)-[:ACTS_IN]->(m2)

RETURN distinct m2.title;

Hints

USING INDEX Force the use of a specific index

MATCH (a:Person {name:"TomHanks"})-[:ACTS_IN]->()

USING INDEX a:Person(name)

RETURN count(*)

Hints

USING SCAN Forces a label scan on lower cardinality labels

MATCH (a:Actor)-->(m:Movie:Comedy)

USING SCAN m:Comedy

RETURN count(distinct a)

Even more tips...

Use parameters

MATCH (p:Person {name: {name}})

-[:ACTS_IN]->(m)

RETURN m.title

MATCH (p:Person {name:"Tom Hanks"})

-[:ACTS_IN]->(m)

RETURN m.title

Avoid Cartesian products

‣ Easy to do this inadvertently:

MATCH (a:Actor), (m:Movie)

RETURN count(a), count(m)

‣ This is correct, and performs betterMATCH (a:Actor)

WITH count(a) as a_count

MATCH (m:Movie)

RETURN a_count, count(m)

Watch out for those warnings!

Cardinalities

Watch those rows!

Only RETURN what you need

‣ This is not recommended:MATCH (a:Actor)

RETURN a

‣ Use this instead:MATCH (a:Actor)

RETURN a.name, a.birthdate, a.height

tl;dr

‣ View query plans with EXPLAIN and PROFILE‣ Use labels‣ Index your starting points‣ Reduce work in progress‣ Remember the hints

Thanks for coming

‣ And don’t forget, if the tips aren’t working ask us for help on Stack Overflow!

Mark Needham @markhneedham Petra Selmer @Aethelraed