MongoDB Europe 2016 - Advanced MongoDB Aggregation Pipelines

Advanced MongoDB Aggregation Pipelines

Tom Schreiber

{ "About me" : {

"Name" : "Tom Schreiber",

"Title" : "Senior Consulting Engineer",

"Email" : "tom.schreiber@mongodb.com",

"Twitter" : "@SchreiberTom1",

"Location" : "London, UK"

"Data should be easy to work with."

"The data should serve you. You should not serve the data."

"The same (data-) model in your head, in your code, and in your data - NO impedance mismatch." Eliot Horowitz

CTO & Co-Founder of MongoDB MongoDB World’16

MongoDB Aggregation Framework

"The same (data-) model in your head, in your code, and in your data - NO impedance mismatch."

A Highly Composable Compute Pipeline

$sum$match

Notions of computation and monads

Eugenio Moggi∗

Abstract

The λ-calculus is considered an useful mathematical tool in the study of programminglanguages, since programs can be identified with λ-terms. However, if one goes further anduses βη-conversion to prove equivalence of programs, then a gross simplification is introduced(programs are identified with total functions from values to values), that may jeopardise theapplicability of theoretical results. In this paper we introduce calculi based on a categoricalsemantics for computations, that provide a correct basis for proving equivalence of programs,for a wide range of notions of computation.

IntroductionThis paper is about logics for reasoning about programs, in particular for proving equivalence ofprograms. Following a consolidated tradition in theoretical computer science we identify programswith the closed λ-terms, possibly containing extra constants, corresponding to some features ofthe programming language under consideration. There are three semantic-based approaches toproving equivalence of programs:

• The operational approach starts from an operational semantics, e.g. a partial functionmapping every program (i.e. closed term) to its resulting value (if any), which induces acongruence relation on open terms called operational equivalence (see e.g. [Plo75]). Thenthe problem is to prove that two terms are operationally equivalent.

• The denotational approach gives an interpretation of the (programming) language in amathematical structure, the intended model. Then the problem is to prove that two termsdenote the same object in the intended model.

• The logical approach gives a class of possible models for the (programming) language.Then the problem is to prove that two terms denotes the same object in all possible models.

The operational and denotational approaches give only a theory: the operational equivalence ≈or the set Th of formulas valid in the intended model respectively. On the other hand, the logicalapproach gives a consequence relation ⊢, namely Ax ⊢ A iff the formula A is true in all modelsof the set of formulas Ax, which can deal with different programming languages (e.g. functional,imperative, non-deterministic) in a rather uniform way, by simply changing the set of axiomsAx, and possibly extending the language with new constants. Moreover, the relation ⊢ is oftensemidecidable, so it is possible to give a sound and complete formal system for it, while Th and ≈are semidecidable only in oversimplified cases.

We do not take as a starting point for proving equivalence of programs the theory of βη-conversion, which identifies the denotation of a program (procedure) of type A → B with atotal function from A to B, since this identification wipes out completely behaviours like non-termination, non-determinism or side-effects, that can be exhibited by real programs. Instead, weproceed as follows:

1. We take category theory as a general theory of functions and develop on top a categoricalsemantics of computations based on monads.

∗Research partially supported by EEC Joint Collaboration Contract # ST2J-0374-C(EDB).

$sum$match

Eugenio Moggi∗

Abstract

A monad defines what it means to chain operations together.

This allows the programmer to build pipelines that process data in a series of steps […]

Monads allow a programming style where programs are written by putting together highly composable parts […]

$sum$match

Eugenio Moggi∗

Abstract

A monad defines what it means to chain operations together.

This allows the programmer to build pipelines that process data in a series of steps […]

Monads allow a programming style where programs are written by putting together highly composable parts […]

RubyModule Enumerable:collect, select, group_by, …

C# / LINQIEnumerable<T>:Select, Where, GroupBy, …

JavaStream<T>:collect, filter, groupingBy, …

MongoDBAggregation Framework:$project, $match, $group, …

map, filter, group, …

HaskellModule Data.List:

$sum$match

Aggregation Pipelines

(1) are one of the easiest ways to work with your data.

(2) are highly scalable and optimised for performance.

(3) are (much) more capable and powerful than you think!

4$sum$match

1 Alex DE 3002 Bert DE 1003 Cora DE 2004 Drew US 2005 Erik US 4006 Fred US 3007 Gina US 2008 Herb NL 6009 Ivan NL 40010 Jill UK 500

ID NAME DPT SAL

The two top paid employees per departement

1 Alex DE 3003 Cora DE 200

US5 Erik US 4006 Fred US 300

NL8 Herb NL 6009 Ivan NL 400

10 Jill UK 500

DB Land

ID NAME DPT SAL

10 Jill UK 500

DB Land

ID NAME DPT SAL

select id, name, dpt, sal from emps where sal =

(select max(sal) from emps as e where e.dpt = emps.dpt)

or sal = (select max(sal) from emps as e where e.dpt = emps.dpt and sal < (select max(sal) from emps as e2 where e2.dpt = emps.dpt)); DB Land

ID NAME DPT SAL

select r.id, r.name, r.dpt, r.sal from (select id, name, dept, sal, rank() OVER (PARTITION BY dpt ORDER BY sal DESC) as rank from emps) AS r where r.rank <= 2 order by r.dpt, r.rank

1 Alex DE 3003 Cora DE 2005 Erik US 4006 Fred US 3008 Herb NL 6009 Ivan NL 40010 Jill UK 500

ID NAME DPT SAL

DB Land

ID NAME DPT SAL

ID NAME DPT SAL1 Alex DE 3003 Cora DE 200

10 Jill UK 500

DB Land

ID NAME DPT SAL

ID NAME DPT SAL1 Alex DE 3003 Cora DE 200

10 Jill UK 500

DB Land

ID NAME DPT SAL

DB LandPL Land

PL Land

ID NAME DPT SAL

DB Land

PL Land

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]] {"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},, {"id"=> 7, "name"=>"Gina", "dpt"=>"US", "sal"=>200},, {"id"=> 6, "name"=>"Fred", "dpt"=>"US", "sal"=>300},, {"id"=> 5, "name"=>"Erik", "dpt"=>"US", "sal"=>400},, {"id"=> 4, "name"=>"Drew", "dpt"=>"US", "sal"=>200},, {"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},, {"id"=> 2, "name"=>"Bert", "dpt"=>"DE", "sal"=>100},,[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

DB Land

PL Land

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]] {"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},, {"id"=> 7, "name"=>"Gina", "dpt"=>"US", "sal"=>200},, {"id"=> 6, "name"=>"Fred", "dpt"=>"US", "sal"=>300},, {"id"=> 5, "name"=>"Erik", "dpt"=>"US", "sal"=>400},, {"id"=> 4, "name"=>"Drew", "dpt"=>"US", "sal"=>200},, {"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},, {"id"=> 2, "name"=>"Bert", "dpt"=>"DE", "sal"=>100},,[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

emps. {|e| [e["dpt"], -e["sal"]]} . {|e| e["dpt"]} . {|k,v| [k => v. (2)]}collect

group_bysort_by

DB Land

PL Land

group_bysort_by

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]] {"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},, {"id"=> 7, "name"=>"Gina", "dpt"=>"US", "sal"=>200},,

{"id"=> 6, "name"=>"Fred", "dpt"=>"US", "sal"=>300},, {"id"=> 5, "name"=>"Erik", "dpt"=>"US", "sal"=>400},,

{"id"=> 4, "name"=>"Drew", "dpt"=>"US", "sal"=>200},,

{"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},, {"id"=> 2, "name"=>"Bert", "dpt"=>"DE", "sal"=>100},,

[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

DB Land

PL Land

[{"DE"=>

[{"US"=>

[{"NL"=>

[{"UK"=>

group_bysort_by

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]]

{"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},,

{"id"=> 7, "name"=>"Gina", "dpt"=>"US", "sal"=>200},,

{"id"=> 4, "name"=>"Drew", "dpt"=>"US", "sal"=>200},,

{"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},, {"id"=> 2, "name"=>"Bert", "dpt"=>"DE", "sal"=>100},,

[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

DB Land

PL Land

}, {"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},,[[{"NL"=>

{"id"=> 6, "name"=>"Fred", "dpt"=>"US", "sal"=>300},, {"id"=> 5, "name"=>"Erik", "dpt"=>"US", "sal"=>400},,[[{"US"=>

[{"DE"=> {"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},,[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]][[{"UK"=> ]

group_bysort_by

DB Land

PL Land

emps. {|e| [e["dpt"], -e["sal"]]} . {|e| e["dpt"]} . {|k,v| [k => v. (2)]}takecollect

group_bysort_by

}, {"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},,

][[{"NL"=>

][[{"US"=>

[{"DE"=> {"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},,[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]][[{"UK"=> ]

10 Jill UK 500

DB Land

group_bysort_by

PL Land

emps.sorted(Comparator.comparing(Employee::getDpt) .thenComparing(Employee::getSal) .reversed()).collect(Collectors .groupingBy(Employee::getDpt, Collectors .collectingAndThen(Collectors.toList(),l->l.stream() .limit(2).collect( Collectors.toList()))));

emps.OrderBy(e => e.Dpt).ThenByDescending(e => e.Sal) .GroupBy(e => e.Dpt) .Select(g => new {Key = g.Key, Emps = g.Take(2)})

C# / LINQ

Java Streams

DB Land

PL Land

group_by$sort

DB Land

PL Land

$sort$group

DB Land

PL Land

emps. {|e| [e["dpt"], -e["sal"]]} . {|e| e["dpt"]} . {|k,v| [k => v. (2)]}take

$sort$group$project

DB Land

PL Land

emps. {|e| [e["dpt"], -e["sal"]]} . {|e| e["dpt"]} . {|k,v| [k => v. (2)]}

$sort$group$project $slice

DB Land

PL Land

db.emps.aggregate([ { : {dpt:1, sal:-1}}, { : {_id:"$dpt", emps:{$push: "$$CURRENT"}}}, { : {dpt:"$_id", emps:{ : ["$emps",2]}}} ])

Aggregation Pipeline

DB Land

PL Land

DB Land

$project

$slice$group$sort

MongoDB

PL Land

PL Land DB Land

$project

$slice$group$sort

MongoDB

PL Land

{"_id":10, "name":"Jill", "dpt":"UK", "sal":500}] {"_id": 9, "name":"Ivan", "dpt":"NL", "sal":400}, {"_id": 8, "name":"Herb", "dpt":"NL", "sal":600}, {"_id": 7, "name":"Gina", "dpt":"US", "sal":200}, {"_id": 6, "name":"Fred", "dpt":"US", "sal":300}, {"_id": 5, "name":"Erik", "dpt":"US", "sal":400}, {"_id": 4, "name":"Drew", "dpt":"US", "sal":200}, {"_id": 3, "name":"Cora", "dpt":"DE", "sal":200}, {"_id": 2, "name":"Bert", "dpt":"DE", "sal":100},[{"_id": 1, "name":"Alex", "dpt":"DE", "sal":300},

DB Land

MongoDB

PL Land

{"_id":10, "name":"Jill", "dpt":"UK", "sal":500}] {"_id": 9, "name":"Ivan", "dpt":"NL", "sal":400}, {"_id": 8, "name":"Herb", "dpt":"NL", "sal":600}, {"_id": 7, "name":"Gina", "dpt":"US", "sal":200},

{"_id": 6, "name":"Fred", "dpt":"US", "sal":300}, {"_id": 5, "name":"Erik", "dpt":"US", "sal":400},

{"_id": 4, "name":"Drew", "dpt":"US", "sal":200},

{"_id": 3, "name":"Cora", "dpt":"DE", "sal":200}, {"_id": 2, "name":"Bert", "dpt":"DE", "sal":100},

[{"_id": 1, "name":"Alex", "dpt":"DE", "sal":300},

DB Land

MongoDB

PL Land

{"_id":10, "name":"Jill", "dpt":"UK", "sal":500}]

{"_id": 9, "name":"Ivan", "dpt":"NL", "sal":400}, {"_id": 8, "name":"Herb", "dpt":"NL", "sal":600},

{"_id": 7, "name":"Gina", "dpt":"US", "sal":200},

{"_id": 4, "name":"Drew", "dpt":"US", "sal":200},

{"_id": 3, "name":"Cora", "dpt":"DE", "sal":200}, {"_id": 2, "name":"Bert", "dpt":"DE", "sal":100},

[{"_id": 1, "name":"Alex", "dpt":"DE", "sal":300},[{"_id" :"DE", "emps":

}, {"_id" :"US", "emps":

}, {"_id" :"NL", "emps":

}, {"_id" :"UK", "emps":

DB Land

MongoDB

PL Land

{"_id": 3, "name":"Cora", "dpt":"DE", "sal":200},[{"_id": 1, "name":"Alex", "dpt":"DE", "sal":300},

[{"dpt" :"DE", "emps":

{"dpt" :"US", "emps":

{"dpt" :"NL", "emps":

{"_id":10, "name":"Jill", "dpt":"UK", "sal":500}] {"dpt" :"UK", "emps": }]

DB Land

MongoDB

PL Land

10 Jill UK 500

DB Land

MongoDB

PL Land

10 Jill UK 500

DB Land

MongoDB

emps.OrderBy(e => e.Dpt).ThenByDescending(e => e.Sal) .GroupBy(e => e.Dpt) .Select(g => new {Key = g.Key, Emps = g.Take(2)})

emps.sorted(Comparator.comparing(Employee::getDpt) .thenComparing(Employee::getSal) .reversed()).collect(Collectors .groupingBy(Employee::getDpt, Collectors .collectingAndThen(Collectors.toList(),l->l.stream() .limit(2).collect( Collectors.toList()))));

group_bysort_by Ruby

DB LandPL Land

Benchmarking PL Land vs DB Land

2,2 GHz Intel Core i7 CPU 16 GB 1600 MHz DDR3 RAM SSD-Storage

MongoDB Enterprise 3.4.0-rc0 WiredTiger (12 GB Cache Size,

snappy compressor)

PL Land DB Land

# Employees Data Size Index Size Runtime

1M 50 MB 20 MB 6 s + 7 s

10M 500 MB 200 MB 133 s + 99 s

100M 5 GB 2 GB 2 h - OOM

PL Land DB Land

# Employees Data Size Index Size Runtime Runtime

1M 50 MB 20 MB 6 s + 7 s 100 ms

10M 500 MB 200 MB 133 s + 99 s 1 s

100M 5 GB 2 GB 2 h - OOM 5 m

PL Land DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

db.emps.aggregate([ { : {dpt:1, sal:-1}}, { : {_id:"$dpt", emps:{$push: "$$CURRENT"}}}, { : {dpt:"$_id", emps:{ : ["$emps",2]}}} ]

$slice)

$project$group$sort

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$slice)

$project$group$sort

5 GB"errmsg":"Exceededmemorylimitfor$group,butdidn'tallowexternalsort.PassallowDiskUse:truetooptin."

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$slice)

$project$group$sort

5 GB,{allowDiskUse:true}

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$slice)

$project$group$sort

5 GB,{allowDiskUse:true}

!CPU Load

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

mongod --dbpath ~/data/shard1 --port 40001 mongod --dbpath ~/data/shard2 --port 40002 … mongod --dbpath ~/data/shard10 --port 40010

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

!sh.addShard("Toms-MacBook-Pro.local:40001") sh.addShard("Toms-MacBook-Pro.local:40002") … sh.addShard("Toms-MacBook-Pro.local:40010")

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

sh.shardCollection("DB.emps", {dpt:1})

!DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

sh.shardCollection("DB.emps", {dpt:1})

"Micro-Sharding"0.5 GB0.5 GB0.5 GB0.5 GB0.5 GB

0.5 GB 0.5 GB 0.5 GB 0.5 GB 0.5 GB

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

0.5 GB0.5 GB 0.5 GB0.5 GB

0.5 GB0.5 GB0.5 GB0.5 GB0.5 GB

0.5 GB$project$group$sort

$project$group$sort

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

0.5 GB0.5 GB 0.5 GB0.5 GB

0.5 GB0.5 GB0.5 GB0.5 GB0.5 GB

0.5 GB$project$group$sort

$project$group$sort

!CPU Load

DB Land

{"name":"Cora", "dpt":"DE", "sal":200},{"name":"Alex", "dpt":"DE", "sal":300},

{"name":"Fred", "dpt":"US", "sal":300},{"name":"Erik", "dpt":"US", "sal":400},

{"name":"Ivan", "dpt":"NL", "sal":400},{"name":"Herb", "dpt":"NL", "sal":600},

{"name":"Jill", "dpt":"UK", "sal":500}] {"dpt" :"UK", "emps" }]

db.emps.aggregate([

{$sort: {dpt:1,sal:-1}}, {$group: {_id:"$dpt",emps:{$push:{name:"$name",sal:"$sal"}}}}, {$project: {dpt:"$_id",emps:{$slice :["$emps",2]}}}])

DB Land

db.emps.aggregate([

{$sort: {dpt:1,sal:-1}}, {$group: {_id:"$dpt",emps:{$push:{name:"$name",sal:"$sal"}}}}, {$project: {dpt:"$_id",emps:{$slice :["$emps",2]}}} {$match: {dpt:"DE"}},])

DB Land

db.emps.aggregate([

{$sort: {dpt:1,sal:-1}}, {$group: {_id:"$dpt",emps:{$push:{name:"$name",sal:"$sal"}}}},

{$project: {dpt:"$_id",emps:{$slice :["$emps",2]}}} {$match: {dpt:"DE"}},

DB Land

db.emps.aggregate([

{$sort: {dpt:1,sal:-1}},

{$group: {_id:"$dpt",emps:{$push:{name:"$name",sal:"$sal"}}}}, {$project: {dpt:"$_id",emps:{$slice :["$emps",2]}}}

{$match: {dpt:"DE"}},

DB Land

db.emps.aggregate([

,{explain:true})

db.emps.aggregate([

,{explain:true})

DB Land

db.emps.aggregate([

,{explain:true})

DB Land

{ "stages": [ { "$cursor": { "query": { "dpt": "DE" }, "sort": { "dpt": 1, "sal": -1 }, "fields": { "dpt": 1, "name": 1, "sal": 1,

db.emps.aggregate([

,{explain:true})

DB Land

{ "stages": [ { "$cursor": { "query": { "dpt": "DE" }, "sort": { "dpt": 1, "sal": -1 }, "fields": { "dpt": 1, "name": 1, "sal": 1, "_id": 0 }, "queryPlanner": {

db.emps.aggregate([

,{explain:true})

DB Land

{ "stages": [ { "$cursor": { "query": { "dpt": "DE" }, "sort": { "dpt": 1, "sal": -1 }, "fields": { "dpt": 1, "name": 1, "sal": 1, "_id": 0 }, "queryPlanner": {

db.emps.aggregate([

,{explain:true})

DB Land

$cursor

], "name": [ "[MinKey, MaxKey]" ] } } }, "rejectedPlans": [] } } }, { "$group": { "_id": "$dpt", "emps": { "$push": { "name": "$name", "sal": "$sal" } } } }, { "$project": {

db.emps.aggregate([

,{explain:true})

DB Land

$cursor

], "name": [ "[MinKey, MaxKey]" ] } } }, "rejectedPlans": [] } } }, { "$group": { "_id": "$dpt", "emps": { "$push": { "name": "$name", "sal": "$sal" } } } }, { "$project": {

db.emps.aggregate([

,{explain:true})

DB Land

$group

$cursor

"_id": "$dpt", "emps": { "$push": { "name": "$name", "sal": "$sal" } } } }, { "$project": { "_id": false, "dpt": "$_id", "emps": { "$slice": [ "$emps", { "$const": 2 } ] } } } ],

db.emps.aggregate([

,{explain:true})

DB Land

$group

$cursor

"_id": "$dpt", "emps": { "$push": { "name": "$name", "sal": "$sal" } } } }, { "$project": { "_id": false, "dpt": "$_id", "emps": { "$slice": [ "$emps", { "$const": 2 } ] } } } ],

db.emps.aggregate([

,{explain:true})

DB Land

$group

$project

$cursor

"queryPlanner": { "plannerVersion": 1, "namespace": "DB.emps", "indexFilterSet": false, "parsedQuery": { "dpt": { "$eq": "DE" } }, "winningPlan": { "stage": "PROJECTION", "transformBy": { "dpt": 1, "name": 1, "sal": 1, "_id": 0 }, "inputStage": { "stage": "IXSCAN", "keyPattern": { "dpt": 1, "sal": -1, "name": 1 },

db.emps.aggregate([

,{explain:true})

DB Land

$group

$project

$cursor

db.emps.aggregate([

,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN

Index on "dpt":1, "sal":-1, "name":1

PROJECTION

"multiKeyPaths": { "dpt": [], "sal": [], "name": [] }, "isUnique": false, "isSparse": false, "isPartial": false, "indexVersion": 2, "direction": "forward", "indexBounds": { "dpt": [ "[\"DE\", \"DE\"]" ], "sal": [ "[MaxKey, MinKey]" ], "name": [ "[MinKey, MaxKey]" ] } } }, "rejectedPlans": []

db.emps.aggregate([

,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN

PROJECTION

db.emps.aggregate([

,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN

PROJECTION

Index used for sorting

db.emps.aggregate([

,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN

PROJECTION

Pipeline Reordering:

Index used for matching

db.emps.aggregate([

,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN

PROJECTION

db.emps.aggregate([

,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN

PROJECTION

Projection Optimization:

Query turned to Covered Query

db.emps.aggregate([

,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN

PROJECTION

Query turned to Covered Query

Index used

Query turned to

DB Land

_id name dpt sal

DB Land

_id name dpt sal103161055108

10 Jill UK 500_id name dpt sal bid

5 Erik US 400 10_id name dpt sal bid

7 Gina US 200 5_id name dpt sal bid

6 Fred US 300 5_id name dpt sal bid

9 Ivan NL 400 8_id name dpt sal bid

8 Herb NL 600 10_id name dpt sal bid

1 Alex DE 300 10_id name dpt sal bid

3 Cora DE 200 1_id name dpt sal bid

2 Bert DE 100 3_id name dpt sal bid

4 Drew US 200 6_id name dpt sal bid

db.emps.aggregate([

{$match: {name:"Bert"}},db.emps.aggregate([

Bert’s bosses

{$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}}

"bid" "_id""$bid""bosses"

$match $graphLookup

Bert’s bosses

{ }2 Bert DE 100 3_id name dpt sal bid

"bosses":[, ]

Bert’s bosses

"bosses":[, ]

Bert’s bosses

"bosses":[, ]

Bert’s bosses

"bosses":[, ]

Bert’s bosses

"bosses":[, ]3 Cora DE 200 1_id name dpt sal bid

"lvl":0,{ }

Bert’s bosses

"lvl":0,{ }

Bert’s bosses

"lvl":0,{ }

Bert’s bosses

"bosses":[,

"lvl":0,{ }

"lvl":1,{ }

db.emps.aggregate([

Bert’s bosses

{$match: {name:"Bert"}}, {$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}}

"bosses":[,

"lvl":0,{ }

"lvl":1,{ }

Bert’s bosses

"bosses":[,

"lvl":0,{ }

"lvl":1,{ }

Bert’s bosses

"bosses":[,

"lvl":0,{ }

"lvl":1,{ }

"lvl":2,{ }

Bert’s bosses

"bosses":[,

"lvl":0,{ }

"lvl":1,{ }

"lvl":2,{ }

Bert’s bosses

"bosses":[,

"lvl":0,{ }

"lvl":1,{ }

"lvl":2,{ }

Bert’s bosses

The two employees bosseswith most

db.emps.aggregate([

, {$sort:{"bosses.leve:-1}}, {$limit:2} "bosses.lvl"

$limit$graphLookup $sort

8 Herb NL 600 10

10 Jill UK 500

5 Erik US 400 105 Erik US 400 10

10 Jill UK 500

6 Fred US 300 5

1 Alex DE 300 10

3 Cora DE 200 1

10 Jill UK 50010 Jill UK 500_id name dpt sal bid

1 Alex DE 300 102 Bert DE 100 33 Cora DE 200 14 Drew US 200 65 Erik US 400 106 Fred US 300 57 Gina US 200 58 Herb NL 600 109 Ivan NL 400 810 Jill UK 500

{ ,"bosses":[[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[]}]

]},]},]},]},]},]},]},]},]},

db.emps.aggregate([ {$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}}

"bid" "_id""$bid""bosses" ,

{$sort:{"bosses.leve:-1}}, {$limit:2} "bosses.lvl"

8 Herb NL 600 10

10 Jill UK 500

5 Erik US 400 105 Erik US 400 10

10 Jill UK 500

6 Fred US 300 5

1 Alex DE 300 10

3 Cora DE 200 1

]},]},]},]},]},]},]},]},]},

8 Herb NL 600 10

10 Jill UK 500

5 Erik US 400 105 Erik US 400 10

10 Jill UK 500

6 Fred US 300 5

1 Alex DE 300 10

3 Cora DE 200 1

]},]},]},]},]},]},]},]},]},

{ ,"bosses":[{ ,"lvl":0}[{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}

10 Jill UK 500

{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[]}]

3 Cora DE 200 11 Alex DE 300 106 Fred US 300 510 Jill UK 5005 Erik US 400 105 Erik US 400 1010 Jill UK 5008 Herb NL 600 10

]},]},]},]},]},]},]},]},]},

10 Jill UK 50010 Jill UK 50010 Jill UK 500

5 Erik US 400 10

10 Jill UK 500

1 Alex DE 300 10

{ ,"bosses":[{ ,"lvl":0}[{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}

10 Jill UK 500

{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[]}]

3 Cora DE 200 11 Alex DE 300 106 Fred US 300 510 Jill UK 5005 Erik US 400 105 Erik US 400 1010 Jill UK 5008 Herb NL 600 10

]},]},]},

]},]},

{ ,"bosses":[{ ,"lvl":0}[{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},

10 Jill UK 500

{ ,"bosses":[{ ,"lvl":0}{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},

{ ,"bosses":[{ ,"lvl":0}{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},

{ ,"bosses":[]}]

3 Cora DE 200 1 1 Alex DE 300 101 Alex DE 300 10 10 Jill UK 5006 Fred US 300 5 5 Erik US 400 1010 Jill UK 5005 Erik US 400 10 10 Jill UK 5005 Erik US 400 10 10 Jill UK 50010 Jill UK 5008 Herb NL 600 10 10 Jill UK 500

]},]},

10 Jill UK 50010 Jill UK 50010 Jill UK 500_id name dpt sal bid

{ ,"bosses":[{ ,"lvl":0}[{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},

10 Jill UK 500

{ ,"bosses":[]}]

3 Cora DE 200 1 1 Alex DE 300 101 Alex DE 300 10 10 Jill UK 5006 Fred US 300 5 5 Erik US 400 1010 Jill UK 5005 Erik US 400 10 10 Jill UK 5005 Erik US 400 10 10 Jill UK 50010 Jill UK 5008 Herb NL 600 10 10 Jill UK 500

]},]},

{ ,"bosses":[{ ,"lvl":0} }] ,[{ ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,

10 Jill UK 500

{ ,"bosses":[]}]

3 Cora DE 200 1 1 Alex DE 300 10 10 Jill UK 5001 Alex DE 300 10 10 Jill UK 5006 Fred US 300 5 5 Erik US 400 10 10 Jill UK 50010 Jill UK 5005 Erik US 400 10 10 Jill UK 5005 Erik US 400 10 10 Jill UK 50010 Jill UK 5008 Herb NL 600 10 10 Jill UK 500

}] ,}] ,

]},]},

{ ,"bosses":[{ ,"lvl":0} }] ,[{ ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,

10 Jill UK 500

{ ,"bosses":[]}]

3 Cora DE 200 1 1 Alex DE 300 10 10 Jill UK 5001 Alex DE 300 10 10 Jill UK 5006 Fred US 300 5 5 Erik US 400 10 10 Jill UK 50010 Jill UK 5005 Erik US 400 10 10 Jill UK 5005 Erik US 400 10 10 Jill UK 50010 Jill UK 5008 Herb NL 600 10 10 Jill UK 500

}] ,}] ,

]},]},

1 Alex DE 300 10{ ,"bosses":[{ ,"lvl":0} }] ,10 Jill UK 500

10 Jill UK 500{ ,"bosses":[]}]

5 Erik US 400 10{ ,"bosses":[{ ,"lvl":0}10 Jill UK 500 }] ,8 Herb NL 600 10{ ,"bosses":[{ ,"lvl":0}10 Jill UK 500 }] ,}] ,

2 Bert DE 100 3 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,3 Cora DE 200 1 1 Alex DE 300 10 10 Jill UK 500 ]},4 Drew US 200 6 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,6 Fred US 300 5 5 Erik US 400 10 10 Jill UK 500 ]},

6 Fred US 300 5 { ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},5 Erik US 400 10 10 Jill UK 500 ]},7 Gina US 200 5 { ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},5 Erik US 400 10 10 Jill UK 500 ]},

3 Cora DE 200 1 { ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},1 Alex DE 300 10 10 Jill UK 500 ]},

9 Ivan NL 400 8 { ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},8 Herb NL 600 10 10 Jill UK 500 ]},

[ 2 Bert DE 100 3 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,3 Cora DE 200 1 1 Alex DE 300 10 10 Jill UK 500 ]},4 Drew US 200 6 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,6 Fred US 300 5 5 Erik US 400 10 10 Jill UK 500 ]}]

db.emps.aggregate([

, {$sort:{"bosses.leve:-1}}, {$limit:2} "bosses.lvl"

{$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}} , {$sort:{"bosses.leve:-1}}, {$limit:2} "bosses.lvl"

db.emps.aggregate([

[ 2 Bert DE 100 3 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,3 Cora DE 200 1 1 Alex DE 300 10 10 Jill UK 500 ]},4 Drew US 200 6 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,6 Fred US 300 5 5 Erik US 400 10 10 Jill UK 500 ]}]

{$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}} , {$sort:{"bosses.leve:-1}}, {$limit:2} "bosses.lvl"

db.emps.aggregate([

{$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}} , {$sort:{"bosses.leve:-1}}, {$limit:2}

db.emps.aggregate([

The two employeesbosses with most

"bosses.lvl"

db.emps.aggregate([

"bid""_id""$bid""bosses"

"bosses.lvl"

, {$sort:{"bosses.leve:-1}}, {$limit:2}

db.emps.aggregate([

"emps""bosses.lvl"

"$_id" "_id" "bid"

db.emps.aggregate([

"bosses.lvl"

"$_id" "_id" "bid""emps" ,

db.emps.aggregate([

"hdcnt"

db.emps.aggregate([

"hdcnt"{$addFields: {hdcnt:{$size:"$emps"}}},

{$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}} , {$sort:{"bosses:-1}}, {$limit:2}

db.emps.aggregate([

$limit$graphLookup $sort$addFields

db.emps.aggregate([

[{ ,"emps":[], "hdcnt":0}]

9 Ivan NL 400 8

[{ ,"emps":[], "hdcnt":0}

7 Gina US 200 5

[{ ,"emps":[], "hdcnt":0}

4 Drew US 200 6

[{ ,"emps":[], "hdcnt":0}

2 Bert DE 100 3

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

8 Herb NL 600 10 9 Ivan NL 400 8

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

6 Fred US 300 5 4 Drew US 200 610 Jill UK 500_id name dpt sal bid

db.emps.aggregate([

[ 5 Erik US 400 10 { ,"lvl":0} { ,"lvl":1}{ ,"emps":[{ ,"lvl":0}, ,7 Gina US 200 5 6 Fred US 300 5 4 Drew US 200 6 ], "hdcnt":3}

[{ ,"emps":[], "hdcnt":0}

7 Gina US 200 5

[{ ,"emps":[], "hdcnt":0}

4 Drew US 200 6

[{ ,"emps":[], "hdcnt":0}

2 Bert DE 100 3

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

3 Cora DE 200 1 2 Bert DE 100 3

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

8 Herb NL 600 10 9 Ivan NL 400 8

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

6 Fred US 300 5 4 Drew US 200 6

[ { ,"lvl":1}],{ ,"emps":[{ ,"lvl":0}, "hdcnt":2},

1 Alex DE 300 10 3 Cora DE 200 1 2 Bert DE 100 3

, 10 Jill UK 500_id name dpt sal bid

db.emps.aggregate([

[ 10 Jill UK 500{ ,"emps":[ { ,"lvl":0} { ,"lvl":0}{ ,"lvl":0}, , ,{ ,"lvl":1} { ,"lvl":1}{ ,"lvl":1}, , ,{ ,"lvl":2} { ,"lvl":2}{ ,"lvl":1}, , 4 Drew US 200 6 ],2 Bert DE 100 3

1 Alex DE 300 10 5 Erik US 400 10 8 Herb NL 600 103 Cora DE 200 1 7 Gina US 200 5 6 Fred US 300 59 Ivan NL 400 8

"hdcnt":9},

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

3 Cora DE 200 1 2 Bert DE 100 3

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

8 Herb NL 600 10 9 Ivan NL 400 8

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

6 Fred US 300 5 4 Drew US 200 6

db.emps.aggregate([

"hdcnt":9},

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

3 Cora DE 200 1 2 Bert DE 100 3

db.emps.aggregate([

db.emps.aggregate(["$_id" "_id" "bid""emps" ,

"hdcnt" ,{$addFields: {hdcnt:{$size:"$emps"}}},

"hdcnt":9},

{$addFields: {tsal:{$reduce: { input: "$emps", initialValue: 0, in: {$add: ["$$value", "$$this.sal"]}}}}}])

"hdcnt":9},

$limit$graphLookup $sort$addFields

"hdcnt":9},

$addFields

$reduce$limit$graphLookup $sort$addFields

"hdcnt":9},

, "tsal":2700

, "tsal":700

MongoDB

_id name dpt sal bid

MongoDB

_id name dpt sal bidMunichHamburgEssen

ChicagoBostonOrlandoAustinUtrechtArnheimLondon

MongoDB

MunichHamburgEssen

_id name dpt sal bid loc

MunichHamburgEssen

Chicago Boston

Austin

Orlando

MunichHamburgEssen

Chicago Boston

Austin

Orlando

MunichHamburgEssen

Chicago Boston

Austin

Orlando

MunichHamburgEssen

Chicago Boston

Austin

Orlando

$1 Alex DE 300 102 Bert DE 100 33 Cora DE 200 14 Drew US 200 65 Erik US 400 106 Fred US 300 57 Gina US 200 58 Herb NL 600 109 Ivan NL 400 810 Jill UK 500

MunichHamburgEssen

Chicago Boston

Austin

Orlando

MunichHamburgEssen

Chicago Boston

Austin

Orlando

MunichHamburgEssen

Chicago Boston

Austin

Orlando

MunichHamburgEssen

Chicago Boston

Austin

Orlando

MunichHamburgEssen

Chicago Boston

Austin

Orlando

MunichHamburgEssen

Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

Boston 980 ChicagoBoston 2000 AustinBoston 1300 OrlandoChicago 980 BostonChicago 1150 AustinChicago 1200 OrlandoAustin 2000 BostonAustin 1150 ChicagoAustin 1100 OrlandoOrlando 1300 BostonOrlando 1200 ChicagoOrlando 1100 Austin

start enddist

Orlando 1300 BostonBoston 980 ChicagoChicago 1200 OrlandoOrlando 1100 AustinAustin 2000 BostonBoston 980 ChicagoChicago 1150 AustinAustin 1100 OrlandoOrlando 1300 BostonBoston 1300 OrlandoOrlando 1100 AustinAustin 1150 ChicagoChicago 980 BostonBoston 1300 OrlandoOrlando 1200 ChicagoChicago 1150 AustinAustin 2000 BostonBoston 2000 AustinAustin 1100 OrlandoOrlando 1200 ChicagoChicago 980 BostonBoston 2000 AustinAustin 1150 ChicagoChicago 1200 Orlando

…,{$graphLookup:{from:"dists",startWith:"$end",connectFromField:"end",connectToField:"start", dpthField:"lvl", as:"routes"}}, …, {$sort:{"tdist":1}}, {$limit:1})]

start enddist

Boston 980 ChicagoBoston 2000 AustinBoston 1300 OrlandoChicago 980 BostonChicago 1150 AustinChicago 1200 OrlandoAustin 2000 BostonAustin 1150 ChicagoAustin 1100 OrlandoOrlando 1300 BostonOrlando 1200 ChicagoOrlando 1100 Austin

Chicago Boston

Austin

Orlando$limit$graphLookup $sort...

5280Boston 980 Chicago Chicago 1200 Orlando Orlando 1100 Austin Austin 2000 Boston

4530Boston 980 Chicago Chicago 1150 Austin Austin 1100 Orlando Orlando 1300 Boston

4530Boston 1300 Orlando Orlando 1100 Austin Austin 1150 Chicago Chicago 980 Boston

5650Boston 1300 Orlando Orlando 1200 Chicago Chicago 1150 Austin Austin 2000 Boston

5280Boston 2000 Austin Austin 1100 Orlando Orlando 1200 Chicago Chicago 980 Boston

5650Boston 2000 Austin Austin 1150 Chicago Chicago 1200 Orlando Orlando 1300 Boston

start enddist start enddist start enddist start enddist tdist

Chicago Boston

Austin

Orlando

5280Boston 980 Chicago Chicago 1200 Orlando Orlando 1100 Austin Austin 2000 Boston

4530Boston 1300 Orlando Orlando 1100 Austin Austin 1150 Chicago Chicago 980 Boston

5650Boston 1300 Orlando Orlando 1200 Chicago Chicago 1150 Austin Austin 2000 Boston

5280Boston 2000 Austin Austin 1100 Orlando Orlando 1200 Chicago Chicago 980 Boston

5650Boston 2000 Austin Austin 1150 Chicago Chicago 1200 Orlando Orlando 1300 Boston

Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

4530Boston 980 Chicago Chicago 1150 Austin Austin 1100 Orlando Orlando 1300 Boston✈

Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

4530Boston 980 Chicago Chicago 1150 Austin Austin 1100 Orlando Orlando 1300 Boston ✈

Facets{

MongoDB

_id name dpt sal

MongoDB

_id name dpt sal

MongoDB

_id name dpt sal expertise

MongoDB

_id name dpt sal expertise5 results for mongodb

Alexlocation: DE salary: 300 expertise: MongoDB, Ruby

Bertlocation: DE salary: 100 expertise: MongoDB, Java

Coralocation: DE salary: 200 expertise: MongoDB, Java

Eriklocation: US salary: 400 expertise: MongoDB, Ruby

Herblocation: NL salary: 600 expertise: MongoDB, Scala

Location

AllDENL

SalaryAll100 - 200200 - 400400 - 600

ExpertiseAllMongoDBJava

https://www.awesomeempview.com/search/f?type=all&key…

Search | EmpView

mongodb

(2)(2)(1)

Ruby (2)

US (1)

3 results for oracle

Fredlocation: US salary: 300 expertise: Oracle, Cobol

Ginalocation: US salary: 200 expertise: Oracle, Pascal

Ivanlocation: NL salary: 400 expertise: Oracle, Fortran

Search | EmpView

Location

AllUSNL

ExpertiseAllOracleCobolPascalFortran

SalaryAll100 - 200200 - 400

oracle

(2)(1)

(3)(1)(1)(1)

(1)(2)

https://www.awesomeempview.com/search/f?type=all&key… MongoDB

MongoDB

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

5 results for mongodb

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

db.emps.aggregate([ { $match: { $text : { $search : "mongodb" }}}, { $sortByCount: "$dpt"}])

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

{"_id":"DE", "count":3},

{"_id":"NL", "count":1},

{"_id":"US", "count":1}

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

db.emps.aggregate( [ {$match: { $text : {$search : "mongodb" }}}, {$unwind: "$expertise"}, {$sortByCount: "$expertise"}])

{"_id":"MongoDB","count":5},

{"_id":"Java","count":2},

{"_id":"Ruby","count":2},

{"_id":"Scala","count":1}

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

db.emps.aggregate([ {$match: { $text : {$search : "mongodb" }}}, {$bucket: { groupBy: "$sal", boundaries: [100, 201, 401, 601, Infinity]}} ])

{"_id":100,"count":2},

{"_id":201,"count":2},

{"_id":401,"count":1}

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

db.emps.aggregate([ {$match: { $text : {$search : "mongodb" }}}, {$bucket: { groupBy: "$sal", boundaries: [100, 201, 401, 601, Infinity], output: { count: {$sum: 1}, matches: {$push: "$$CURRENT"} }}} ])

{"_id":100,"count":2,"matches":[

]}, {"_id":201,"count":2,"matches":[

]}, {"_id":401,"count":1,"matches":[

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

db.emps.aggregate([ {$match: { $text : {$search : "mongodb" }}}, {$bucketAuto: { groupBy: "$sal", buckets: 4}} ])

{"_id":{"min":100,"max":200}, "count":1},

{"_id":{"min":200,"max":300}, "count":1},

{"_id":{"min":300,"max":400}, “count":1},

{"_id":{"min":400,"max":600}, "count":2}

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

db.emps.aggregate([ {$match: { $text : {$search : {$facet: { "Location": [ {$sortByCount: "$dpt"} ], "Expertise":[ {$unwind: "$expertise"}, {$sortByCount: "$expertise"} ], "Salary":[ {$bucket: { groupBy: "$sal", boundaries: [100, 201, 401, 601, Infinity]}} ]}}])

{"Location":[ {"_id":"DE","count":3}, {"_id":"NL","count":1}, {"_id":"US","count":1}], "Expertise":[ {"_id":"MongoDB","count":5}, {"_id":"Java","count":2}, {"_id":"Ruby","count":2}, {"_id":"Scala","count":1}], "Salary":[ {"_id":100,"count":2}, {"_id":201,"count":2}, {"_id":401,"count":1}]}

}}},"mongodb"

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

}}},"mongodb"

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

}}},"mongodb"

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

}}},"\"mongodb\" \"ruby\""

LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

(2)(2)(1)

Ruby (2)

US (1)

Search | EmpView

mongodb

{"Location":[ {"_id":"DE","count":1}, {"_id":"US","count":1}], "Expertise":[ {"_id":"MongoDB","count":2}, {"_id":"Ruby","count":2}], "Salary":[ {"_id":201,"count":2}]}

MongoDB

LocationAllDE

Salary

All200 - 400

Expertise

AllMongoDB

( )2Ruby (2)

US (1)

Search | EmpView

mongodb

{"Location":[ {"_id":"DE","count":1}, {"_id":"US","count":1}], "Expertise":[ {"_id":"MongoDB","count":2}, {"_id":"Ruby","count":2}], "Salary":[ {"_id":201,"count":2}]})

MongoDB

One more thing…

PL Land

nats = nats. (nats. {|n| nats. (n-1) . (1) . {|p| n % p != 0} . })

droptake

mapall?

mapzip[1,2,3,4,5,6,7,8,9,10]

PL Land

[[1, true ], [2, true ], [3, true ], [4, false], [5, true ], [6, false], [7, true ], [8, false], [9, false], [10,false]]

nats = nats. (nats. {|n| nats. (n-1) . (1) . {|p| n % p != 0} . })

droptake

mapall?

mapzip[1,2,3,4,5,6,7,8,9,10]

PL Land

droptake

mapall?

mapzip[1,2,3,4,5,6,7,8,9,10]

PL Land

droptake

mapall?

mapzip$range: [1, 11, 1]

PL Land

droptake

mapall?

map$zip$range: [1, 11, 1]

PL Land

droptake

mapall?

["A", "B", "C"][10, 20, 30]

PL Land

droptake

mapall?

["A", "B", "C"][10, 20, 30]

[["A",10], ["B",20], ["C",30]]

PL Land

droptake

mapall?

$map$zip$range: [1, 11, 1]

PL Land

dropmapall?

$slice$map$zip$range: [1, 11, 1]

PL Land

mapall?

$slice$slice

PL Land

$slice$slice$map

PL Land

$slice$slice$map

$allElementsTrue

PL Land

$slice$slice

$map$allElementsTrue

$map$zip

$range: [1, 11, 1]Ruby

$let: {vars: {nats: { }}, in: { : {inputs: ["$$nats", { : {input: "$$nats", as: "n", in: { : [{ : {input: { : [{ : ["$$nats", {$subtract: ["$$n", 1]}]}, 1, {$size: "$$nats"}]}, as: "p", in: { $ne: [{$mod: ["$$n", "$$p"]}, 0]}}}]}}}]}}}

PL Land

$slice$slice

$map$zip

: [1, 11, 1]$rangeAggregation Pipeline

PL Land

$slice$slice

$map$zip

$allElementsTrue$map$slice$slice

$range $zip

MongoDB

PL Land

$slice$slice

$map$zip

$range $zip

MongoDB

PL Land

[[1, true ],[2, true ],[3, true ],[4, false],[5, true ],[6, false],[7, true ],[8, false],[9, false],[10,false]]

$slice$slice

$map$zip

$range $zip

$slice$slice

$map$zip

$range

$slice

$allElementsTrue

$range

$indexStats

$stdDevSamp

$stdDevPop

$log10$ln $pow

$trunc

$floor$arrayElemAt $concatArrays

$isArray$filter

$sum$stdDevPop

$stdDevSamp

$split

$indexOf

$reduce

$reverseArray

$indexOfArray

$isoWeekYear$isoDayOfWeek

$isoWeek

$switch$count

$replaceRoot

$addFields

$graphLookup $facet

$bucket $bucketAuto $sortByCount$out$redact

$setEquals

$setIntersection$setUnion$setDifference

$setIsSubset $anyElementTrue$literal

$geoNear $millisecond $concat

$project $match $limit$skip

$unwind $group $sort

$and $or

$cmp$eq $gt

$lt$lte

$divide

$multiply

$subtract

$toLower

$toUpper

$strcasecmp$meta

$let$dayOfYear

$dayOfMonth

$dayOfWeek

$year$month

$minute

$second

$millisecond

$dateToString

$ifNull

$first$last

$addToSet

$indexOfBytes

$indexOfCP

$strLenBytes$strLenCP

$substrBytes$substrCP

$slice

$allElementsTrue

$range

$sample$lookup

$dateToString

$collStats

$indexStats

$stdDevSamp

$stdDevPop

$log10$ln $pow

$trunc

$isArray$filter

$sum$stdDevPop

$stdDevSamp

$out$redact

$setEquals

$geoNear $millisecond $concat

$and $or

$cmp$eq $gt

$lt$lte

$divide

$multiply

$subtract

$toLower

$toUpper

$strcasecmp$meta

$let$dayOfYear

$dayOfMonth

$dayOfWeek

$year$month

$minute

$second

$millisecond

$dateToString

$ifNull

$first$last

$addToSet

$slice

$allElementsTrue

$sample$lookup

$dateToString

$split

$indexOf

$reduce

$reverseArray

$indexOfArray

$isoWeekYear$isoDayOfWeek

$isoWeek

$switch$count

$replaceRoot

$addFields

$graphLookup $facet

$bucket $bucketAuto $sortByCount$indexOfBytes

$indexOfCP

$strLenBytes$strLenCP

$substrBytes$substrCP

$range

$collStats

2.4 $geoNear $millisecond $concat

$sum$stdDevPop

$stdDevSamp

$and $or

$cmp$eq $gt

$lt$lte

$divide

$multiply

$subtract

$toLower

$toUpper

$strcasecmp$meta

$let$dayOfYear

$dayOfMonth

$dayOfWeek

$year$month

$minute

$second

$millisecond

$dateToString

$ifNull

$first$last

$addToSet

2.6 $out$redact

$setEquals

$allElementsTrue

3.2$indexStats

$stdDevSamp

$stdDevPop

$log10$ln $pow

$trunc

$isArray$filter

$slice

$sample$lookup

3.0$dateToString

$indexOf $indexOfArray

$isoWeekYear$in

$switch$count

$replaceRoot

$addFields

$graphLookup $facet

$indexOfCP$substrBytes$substrCP

$range

$split

$reduce

$reverseArray $isoWeekYear$isoDayOfWeek

$isoWeek

$strLenBytes$strLenCP $collStats

Data-Parallel Pipeline Execution

Shard 1 Shard 2 Shard 3 Shard N

mongos

PipelineOptimisations

Pipeline Projection Optimization

Pipeline Coalescence Optimization

Pipeline Reordering

Pipeline Combining

2.4 $geoNear $millisecond $concat

$sum$stdDevPop

$stdDevSamp

$and $or

$cmp$eq $gt

$lt$lte

$divide

$multiply

$subtract

$toLower

$toUpper

$strcasecmp$meta

$let$dayOfYear

$dayOfMonth

$dayOfWeek

$year$month

$minute

$second

$millisecond

$dateToString

$ifNull

$first$last

$addToSet

2.6 $out$redact

$setEquals

$allElementsTrue

3.2$indexStats

$stdDevSamp

$stdDevPop

$log10$ln $pow

$trunc

$isArray$filter

$slice

$sample$lookup

3.0$dateToString

$indexOf $indexOfArray

$isoWeekYear$in

$switch$count

$replaceRoot

$addFields

$graphLookup $facet

$indexOfCP$substrBytes$substrCP

$range

$split

$reduce

$reverseArray $isoWeekYear$isoDayOfWeek

$isoWeek

$strLenBytes$strLenCP $collStats

Use the Force Aggregation Pipelines!

Use the Force Aggregation Pipelines! And take control over your Data.

ASK THE EXPERTSGet your technical questions answered

By appointment only – register in one of two ways:

In person Online

Come to the MongoDB Team Stand on the ground floor

calendly.com/mongodb

MongoDB Europe 2016 - Advanced MongoDB Aggregation Pipelines

Data & Analytics

MongoDB Aggregation and Data Processingdb.prof.ninja/class10/monagg.pdf · Map-Reduce (page 10) Map-reduce is a generic multi-phase data aggregation modality for processing quanti-ties

Realtime Analytics with MongoDB - MongoDB Meetup NYC

MongoDB Europe 2016 - Distributed Ledgers, Blockchain + MongoDB

MongoDB Aggregation Framework inside Countly

MongoDB - Aggregation Pipeline · What is the Aggregation Pipeline? 5 A framework for data visualization and or manipulation using one ore multiple stages in

MongoDB 3.0 migration - MongoDB Days Munich

Time Series Data in MongoDB Senior Solutions Architect, MongoDB Inc. Massimo Brignoli #mongodb

MongoDB Internals: From Storage Engine to Aggregation Framework

MongoDB Evenings Minneapolis: Medtronic's MongoDB Journey

MongoDB World 2016: MongoDB & IBM

MongoDB Days Silicon Valley: Introducing MongoDB 3.2

MongoDB: Queries and Aggregation Framework with NBA Game Data

MongoDB for Time Series Data Part 2: Analyzing Time Series Data Using the Aggregation Framework and Hadoop

MongoDB Europe 2016 - Debugging MongoDB Performance

Advanced MongoDB Aggregation Pipelines

MongoDB Europe 2016 - Graph Operations with MongoDB

Webinar: Applikationsentwicklung mit MongoDB: Teil 5: Reporting & Aggregation

Joins and Other MongoDB 3.2 Aggregation Enhancements

Aggregation Framework MongoDB Days Munich

MongoDB Aggregation and Indexing · Aggregation •MongoD’s aggregation framework is modeled on the concept of data processing pipelines. •Documents enter a multi-stage pipeline