MongoDB Europe 2016 - Advanced MongoDB Aggregation Pipelines

Advanced MongoDB Aggregation Pipelines

Tom Schreiber

{ "About me" : {

"Name" : "Tom Schreiber",

"Title" : "Senior Consulting Engineer",

"Email" : "[email protected]",

"Twitter" : "@SchreiberTom1",

"Location" : "London, UK"

}

}

"Data should be easy to work with."

"The data should serve you. You should not serve the data."

"The same (data-) model in your head, in your code, and in your data - NO impedance mismatch." Eliot Horowitz

CTO & Co-Founder of MongoDB MongoDB World’16

MongoDB Aggregation Framework



"The same (data-) model in your head, in your code, and in your data - NO impedance mismatch."

A Highly Composable Compute Pipeline


$sum$match


34

3 13

646

36

4


$sum$match


34

3 13

646

36

4


$sum$match


34

3 13

646

36

4


$sum$match

Notions of computation and monads

Eugenio Moggi∗

Abstract

The λ-calculus is considered an useful mathematical tool in the study of programminglanguages, since programs can be identified with λ-terms. However, if one goes further anduses βη-conversion to prove equivalence of programs, then a gross simplification is introduced(programs are identified with total functions from values to values), that may jeopardise theapplicability of theoretical results. In this paper we introduce calculi based on a categoricalsemantics for computations, that provide a correct basis for proving equivalence of programs,for a wide range of notions of computation.

IntroductionThis paper is about logics for reasoning about programs, in particular for proving equivalence ofprograms. Following a consolidated tradition in theoretical computer science we identify programswith the closed λ-terms, possibly containing extra constants, corresponding to some features ofthe programming language under consideration. There are three semantic-based approaches toproving equivalence of programs:

• The operational approach starts from an operational semantics, e.g. a partial functionmapping every program (i.e. closed term) to its resulting value (if any), which induces acongruence relation on open terms called operational equivalence (see e.g. [Plo75]). Thenthe problem is to prove that two terms are operationally equivalent.

• The denotational approach gives an interpretation of the (programming) language in amathematical structure, the intended model. Then the problem is to prove that two termsdenote the same object in the intended model.

• The logical approach gives a class of possible models for the (programming) language.Then the problem is to prove that two terms denotes the same object in all possible models.

The operational and denotational approaches give only a theory: the operational equivalence ≈or the set Th of formulas valid in the intended model respectively. On the other hand, the logicalapproach gives a consequence relation ⊢, namely Ax ⊢ A iff the formula A is true in all modelsof the set of formulas Ax, which can deal with different programming languages (e.g. functional,imperative, non-deterministic) in a rather uniform way, by simply changing the set of axiomsAx, and possibly extending the language with new constants. Moreover, the relation ⊢ is oftensemidecidable, so it is possible to give a sound and complete formal system for it, while Th and ≈are semidecidable only in oversimplified cases.

We do not take as a starting point for proving equivalence of programs the theory of βη-conversion, which identifies the denotation of a program (procedure) of type A → B with atotal function from A to B, since this identification wipes out completely behaviours like non-termination, non-determinism or side-effects, that can be exhibited by real programs. Instead, weproceed as follows:

1. We take category theory as a general theory of functions and develop on top a categoricalsemantics of computations based on monads.

∗Research partially supported by EEC Joint Collaboration Contract # ST2J-0374-C(EDB).

1


34

3 13

646

36

4


$sum$match


Eugenio Moggi∗

Abstract










1

A monad defines what it means to chain operations together.

This allows the programmer to build pipelines that process data in a series of steps […]

Monads allow a programming style where programs are written by putting together highly composable parts […]


34

3 13

646

36

4


$sum$match


Eugenio Moggi∗

Abstract










1

A monad defines what it means to chain operations together.

This allows the programmer to build pipelines that process data in a series of steps […]

Monads allow a programming style where programs are written by putting together highly composable parts […]

RubyModule Enumerable:collect, select, group_by, …

C# / LINQIEnumerable<T>:Select, Where, GroupBy, …

JavaStream<T>:collect, filter, groupingBy, …

MongoDBAggregation Framework:$project, $match, $group, …

map, filter, group, …

HaskellModule Data.List:


34

3 13

646

36

4


$sum$match

Aggregation Pipelines

(1) are one of the easiest ways to work with your data.

(2) are highly scalable and optimised for performance.

(3) are (much) more capable and powerful than you think!


34

3 13

646

36

4$sum$match





1 Alex DE 3002 Bert DE 1003 Cora DE 2004 Drew US 2005 Erik US 4006 Fred US 3007 Gina US 2008 Herb NL 6009 Ivan NL 40010 Jill UK 500

ID NAME DPT SAL

The two top paid employees per departement

1 Alex DE 3003 Cora DE 200

DE

US5 Erik US 4006 Fred US 300

NL8 Herb NL 6009 Ivan NL 400

UK

10 Jill UK 500

DB Land


ID NAME DPT SAL



DE



UK

10 Jill UK 500

DB Land


ID NAME DPT SAL


select id, name, dpt, sal from emps where sal =

(select max(sal) from emps as e where e.dpt = emps.dpt)

or sal = (select max(sal) from emps as e where e.dpt = emps.dpt and sal < (select max(sal) from emps as e2 where e2.dpt = emps.dpt)); DB Land


ID NAME DPT SAL


select r.id, r.name, r.dpt, r.sal from (select id, name, dept, sal, rank() OVER (PARTITION BY dpt ORDER BY sal DESC) as rank from emps) AS r where r.rank <= 2 order by r.dpt, r.rank

1 Alex DE 3003 Cora DE 2005 Erik US 4006 Fred US 3008 Herb NL 6009 Ivan NL 40010 Jill UK 500

ID NAME DPT SAL

DB Land


ID NAME DPT SAL



ID NAME DPT SAL1 Alex DE 3003 Cora DE 200

DE



UK

10 Jill UK 500

?

DB Land


ID NAME DPT SAL



ID NAME DPT SAL1 Alex DE 3003 Cora DE 200

DE



UK

10 Jill UK 500

?




DB Land


ID NAME DPT SAL

DB LandPL Land

PL Land


ID NAME DPT SAL

DB Land

PL Land

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]] {"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},, {"id"=> 7, "name"=>"Gina", "dpt"=>"US", "sal"=>200},, {"id"=> 6, "name"=>"Fred", "dpt"=>"US", "sal"=>300},, {"id"=> 5, "name"=>"Erik", "dpt"=>"US", "sal"=>400},, {"id"=> 4, "name"=>"Drew", "dpt"=>"US", "sal"=>200},, {"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},, {"id"=> 2, "name"=>"Bert", "dpt"=>"DE", "sal"=>100},,[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

DB Land

PL Land

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]] {"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},, {"id"=> 7, "name"=>"Gina", "dpt"=>"US", "sal"=>200},, {"id"=> 6, "name"=>"Fred", "dpt"=>"US", "sal"=>300},, {"id"=> 5, "name"=>"Erik", "dpt"=>"US", "sal"=>400},, {"id"=> 4, "name"=>"Drew", "dpt"=>"US", "sal"=>200},, {"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},, {"id"=> 2, "name"=>"Bert", "dpt"=>"DE", "sal"=>100},,[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

emps. {|e| [e["dpt"], -e["sal"]]} . {|e| e["dpt"]} . {|k,v| [k => v. (2)]}collect

group_bysort_by

take

Ruby

DB Land

PL Land


group_bysort_by

take

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]] {"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},, {"id"=> 7, "name"=>"Gina", "dpt"=>"US", "sal"=>200},,

{"id"=> 6, "name"=>"Fred", "dpt"=>"US", "sal"=>300},, {"id"=> 5, "name"=>"Erik", "dpt"=>"US", "sal"=>400},,

{"id"=> 4, "name"=>"Drew", "dpt"=>"US", "sal"=>200},,

{"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},, {"id"=> 2, "name"=>"Bert", "dpt"=>"DE", "sal"=>100},,

[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

Ruby

DB Land

PL Land

},

[{"DE"=>

]

]

]

[

[

[

[{"US"=>

[{"NL"=>

[{"UK"=>

},

},

]


group_bysort_by

take

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]]

{"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},,

{"id"=> 7, "name"=>"Gina", "dpt"=>"US", "sal"=>200},,


{"id"=> 4, "name"=>"Drew", "dpt"=>"US", "sal"=>200},,

{"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},, {"id"=> 2, "name"=>"Bert", "dpt"=>"DE", "sal"=>100},,

[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

Ruby

DB Land

PL Land

}, {"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},,[[{"NL"=>

{"id"=> 6, "name"=>"Fred", "dpt"=>"US", "sal"=>300},, {"id"=> 5, "name"=>"Erik", "dpt"=>"US", "sal"=>400},,[[{"US"=>

},

[{"DE"=> {"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},,[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

},

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]][[{"UK"=> ]


group_bysort_by

take

]

]]

]]

Ruby

DB Land

PL Land

emps. {|e| [e["dpt"], -e["sal"]]} . {|e| e["dpt"]} . {|k,v| [k => v. (2)]}takecollect

group_bysort_by

}, {"id"=> 9, "name"=>"Ivan", "dpt"=>"NL", "sal"=>400},, {"id"=> 8, "name"=>"Herb", "dpt"=>"NL", "sal"=>600},,

][[{"NL"=>


][[{"US"=>

},

[{"DE"=> {"id"=> 3, "name"=>"Cora", "dpt"=>"DE", "sal"=>200},,[{"id"=> 1, "name"=>"Alex", "dpt"=>"DE", "sal"=>300},,[

] },

{"id"=>10, "name"=>"Jill", "dpt"=>"UK", "sal"=>500}]][[{"UK"=> ]


DE



UK

10 Jill UK 500

Ruby

DB Land


group_bysort_by

Ruby

PL Land

emps.sorted(Comparator.comparing(Employee::getDpt) .thenComparing(Employee::getSal) .reversed()).collect(Collectors .groupingBy(Employee::getDpt, Collectors .collectingAndThen(Collectors.toList(),l->l.stream() .limit(2).collect( Collectors.toList()))));

emps.OrderBy(e => e.Dpt).ThenByDescending(e => e.Sal) .GroupBy(e => e.Dpt) .Select(g => new {Key = g.Key, Emps = g.Take(2)})

C# / LINQ

Java Streams

take

DB Land

PL Land


group_by$sort

Ruby

DB Land

PL Land


$sort$group

Ruby

DB Land

PL Land

emps. {|e| [e["dpt"], -e["sal"]]} . {|e| e["dpt"]} . {|k,v| [k => v. (2)]}take

$sort$group$project

Ruby

DB Land

PL Land

emps. {|e| [e["dpt"], -e["sal"]]} . {|e| e["dpt"]} . {|k,v| [k => v. (2)]}

$sort$group$project $slice

Ruby

DB Land

PL Land

db.emps.aggregate([ { : {dpt:1, sal:-1}}, { : {_id:"$dpt", emps:{$push: "$$CURRENT"}}}, { : {dpt:"$_id", emps:{ : ["$emps",2]}}} ])


Aggregation Pipeline

DB Land

PL Land




DB Land

$project

$slice$group$sort

MongoDB

PL Land




PL Land DB Land

$project

$slice$group$sort

MongoDB

PL Land

{"_id":10, "name":"Jill", "dpt":"UK", "sal":500}] {"_id": 9, "name":"Ivan", "dpt":"NL", "sal":400}, {"_id": 8, "name":"Herb", "dpt":"NL", "sal":600}, {"_id": 7, "name":"Gina", "dpt":"US", "sal":200}, {"_id": 6, "name":"Fred", "dpt":"US", "sal":300}, {"_id": 5, "name":"Erik", "dpt":"US", "sal":400}, {"_id": 4, "name":"Drew", "dpt":"US", "sal":200}, {"_id": 3, "name":"Cora", "dpt":"DE", "sal":200}, {"_id": 2, "name":"Bert", "dpt":"DE", "sal":100},[{"_id": 1, "name":"Alex", "dpt":"DE", "sal":300},




DB Land

MongoDB

PL Land

{"_id":10, "name":"Jill", "dpt":"UK", "sal":500}] {"_id": 9, "name":"Ivan", "dpt":"NL", "sal":400}, {"_id": 8, "name":"Herb", "dpt":"NL", "sal":600}, {"_id": 7, "name":"Gina", "dpt":"US", "sal":200},

{"_id": 6, "name":"Fred", "dpt":"US", "sal":300}, {"_id": 5, "name":"Erik", "dpt":"US", "sal":400},

{"_id": 4, "name":"Drew", "dpt":"US", "sal":200},

{"_id": 3, "name":"Cora", "dpt":"DE", "sal":200}, {"_id": 2, "name":"Bert", "dpt":"DE", "sal":100},

[{"_id": 1, "name":"Alex", "dpt":"DE", "sal":300},




DB Land

MongoDB

PL Land

{"_id":10, "name":"Jill", "dpt":"UK", "sal":500}]

{"_id": 9, "name":"Ivan", "dpt":"NL", "sal":400}, {"_id": 8, "name":"Herb", "dpt":"NL", "sal":600},

{"_id": 7, "name":"Gina", "dpt":"US", "sal":200},


{"_id": 4, "name":"Drew", "dpt":"US", "sal":200},

{"_id": 3, "name":"Cora", "dpt":"DE", "sal":200}, {"_id": 2, "name":"Bert", "dpt":"DE", "sal":100},

[{"_id": 1, "name":"Alex", "dpt":"DE", "sal":300},[{"_id" :"DE", "emps":

}, {"_id" :"US", "emps":

}, {"_id" :"NL", "emps":

}, {"_id" :"UK", "emps":

]

]

]

}]

[

[

[




DB Land

MongoDB

PL Land

{"_id": 3, "name":"Cora", "dpt":"DE", "sal":200},[{"_id": 1, "name":"Alex", "dpt":"DE", "sal":300},

[{"dpt" :"DE", "emps":

},]


{"dpt" :"US", "emps":

},]


{"dpt" :"NL", "emps":

},]

{"_id":10, "name":"Jill", "dpt":"UK", "sal":500}] {"dpt" :"UK", "emps": }]

[

[

[




DB Land

MongoDB

PL Land



},]



},]



},]


[

[

[




DE



UK

10 Jill UK 500


DB Land

MongoDB

PL Land



},]



},]



},]


[

[

[




DE



UK

10 Jill UK 500





DB Land

MongoDB




emps.OrderBy(e => e.Dpt).ThenByDescending(e => e.Sal) .GroupBy(e => e.Dpt) .Select(g => new {Key = g.Key, Emps = g.Take(2)})

C#

emps.sorted(Comparator.comparing(Employee::getDpt) .thenComparing(Employee::getSal) .reversed()).collect(Collectors .groupingBy(Employee::getDpt, Collectors .collectingAndThen(Collectors.toList(),l->l.stream() .limit(2).collect( Collectors.toList()))));

Java


group_bysort_by Ruby

take

DB LandPL Land





Benchmarking PL Land vs DB Land

2,2 GHz Intel Core i7 CPU 16 GB 1600 MHz DDR3 RAM SSD-Storage

MongoDB Enterprise 3.4.0-rc0 WiredTiger (12 GB Cache Size,

snappy compressor)

!

λ

PL Land DB Land

λ

PL Land DB Land

# Employees Data Size Index Size Runtime

1M 50 MB 20 MB 6 s + 7 s

10M 500 MB 200 MB 133 s + 99 s

100M 5 GB 2 GB 2 h - OOM

PL Land DB Land

Q

PL Land DB Land

Q

# Employees Data Size Index Size Runtime Runtime

1M 50 MB 20 MB 6 s + 7 s 100 ms

10M 500 MB 200 MB 133 s + 99 s 1 s

100M 5 GB 2 GB 2 h - OOM 5 m

PL Land DB Land

Q

100M 5 GB 2 GB 2 h - OOM 5 m

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

db.emps.aggregate([ { : {dpt:1, sal:-1}}, { : {_id:"$dpt", emps:{$push: "$$CURRENT"}}}, { : {dpt:"$_id", emps:{ : ["$emps",2]}}} ]

$slice)

$project$group$sort

$project$group$sort

5 GB

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m


$slice)

$project$group$sort

$project$group$sort

5 GB"errmsg":"Exceededmemorylimitfor$group,butdidn'tallowexternalsort.PassallowDiskUse:truetooptin."

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m


$slice)

$project$group$sort

$project$group$sort

5 GB,{allowDiskUse:true}

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m


$slice)

$project$group$sort

$project$group$sort

5 GB,{allowDiskUse:true}

!CPU Load

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

5 GB

mongod --dbpath ~/data/shard1 --port 40001 mongod --dbpath ~/data/shard2 --port 40002 … mongod --dbpath ~/data/shard10 --port 40010

!

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

5 GB


!

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

5 GB


!sh.addShard("Toms-MacBook-Pro.local:40001") sh.addShard("Toms-MacBook-Pro.local:40002") … sh.addShard("Toms-MacBook-Pro.local:40010")

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

5 GB



sh.shardCollection("DB.emps", {dpt:1})

!DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

5 GB



sh.shardCollection("DB.emps", {dpt:1})

"Micro-Sharding"0.5 GB0.5 GB0.5 GB0.5 GB0.5 GB

0.5 GB 0.5 GB 0.5 GB 0.5 GB 0.5 GB

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

5 GB

!

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

0.5 GB0.5 GB 0.5 GB0.5 GB

0.5 GB0.5 GB0.5 GB0.5 GB0.5 GB

0.5 GB$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

!

DB Land

100M 5 GB 2 GB 2 h - OOM 5 m

$project$group$sort

0.5 GB0.5 GB 0.5 GB0.5 GB

0.5 GB0.5 GB0.5 GB0.5 GB0.5 GB

0.5 GB$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

$project$group$sort

6 s

!CPU Load

!

DB Land

{"name":"Cora", "dpt":"DE", "sal":200},{"name":"Alex", "dpt":"DE", "sal":300},


},]

{"name":"Fred", "dpt":"US", "sal":300},{"name":"Erik", "dpt":"US", "sal":400},


},]

{"name":"Ivan", "dpt":"NL", "sal":400},{"name":"Herb", "dpt":"NL", "sal":600},


},]

{"name":"Jill", "dpt":"UK", "sal":500}] {"dpt" :"UK", "emps" }]

[

[

[

db.emps.aggregate([

[

{$sort: {dpt:1,sal:-1}}, {$group: {_id:"$dpt",emps:{$push:{name:"$name",sal:"$sal"}}}}, {$project: {dpt:"$_id",emps:{$slice :["$emps",2]}}}])

DB Land



},]



},]



},]


[

[

[

db.emps.aggregate([

[

{$sort: {dpt:1,sal:-1}}, {$group: {_id:"$dpt",emps:{$push:{name:"$name",sal:"$sal"}}}}, {$project: {dpt:"$_id",emps:{$slice :["$emps",2]}}} {$match: {dpt:"DE"}},])

DB Land



},]



},]



},]


[

[

[

db.emps.aggregate([

[

{$sort: {dpt:1,sal:-1}}, {$group: {_id:"$dpt",emps:{$push:{name:"$name",sal:"$sal"}}}},

{$project: {dpt:"$_id",emps:{$slice :["$emps",2]}}} {$match: {dpt:"DE"}},

])

DB Land



},]



},]



},]


[

[

[

db.emps.aggregate([

[

{$sort: {dpt:1,sal:-1}},

{$group: {_id:"$dpt",emps:{$push:{name:"$name",sal:"$sal"}}}}, {$project: {dpt:"$_id",emps:{$slice :["$emps",2]}}}

{$match: {dpt:"DE"}},

])

DB Land



},]



},]



},]


[

[

[

db.emps.aggregate([

]

[




,{explain:true})

db.emps.aggregate([

]




,{explain:true})

DB Land

db.emps.aggregate([

]




,{explain:true})

DB Land

{ "stages": [ { "$cursor": { "query": { "dpt": "DE" }, "sort": { "dpt": 1, "sal": -1 }, "fields": { "dpt": 1, "name": 1, "sal": 1,

db.emps.aggregate([

]




,{explain:true})

DB Land

{ "stages": [ { "$cursor": { "query": { "dpt": "DE" }, "sort": { "dpt": 1, "sal": -1 }, "fields": { "dpt": 1, "name": 1, "sal": 1, "_id": 0 }, "queryPlanner": {

db.emps.aggregate([

]




,{explain:true})

DB Land

{ "stages": [ { "$cursor": { "query": { "dpt": "DE" }, "sort": { "dpt": 1, "sal": -1 }, "fields": { "dpt": 1, "name": 1, "sal": 1, "_id": 0 }, "queryPlanner": {

db.emps.aggregate([

]




,{explain:true})

DB Land

$cursor

], "name": [ "[MinKey, MaxKey]" ] } } }, "rejectedPlans": [] } } }, { "$group": { "_id": "$dpt", "emps": { "$push": { "name": "$name", "sal": "$sal" } } } }, { "$project": {

db.emps.aggregate([

]




,{explain:true})

DB Land

$cursor

], "name": [ "[MinKey, MaxKey]" ] } } }, "rejectedPlans": [] } } }, { "$group": { "_id": "$dpt", "emps": { "$push": { "name": "$name", "sal": "$sal" } } } }, { "$project": {

db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$cursor

"_id": "$dpt", "emps": { "$push": { "name": "$name", "sal": "$sal" } } } }, { "$project": { "_id": false, "dpt": "$_id", "emps": { "$slice": [ "$emps", { "$const": 2 } ] } } } ],

db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$cursor

"_id": "$dpt", "emps": { "$push": { "name": "$name", "sal": "$sal" } } } }, { "$project": { "_id": false, "dpt": "$_id", "emps": { "$slice": [ "$emps", { "$const": 2 } ] } } } ],

db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$project

$cursor

"queryPlanner": { "plannerVersion": 1, "namespace": "DB.emps", "indexFilterSet": false, "parsedQuery": { "dpt": { "$eq": "DE" } }, "winningPlan": { "stage": "PROJECTION", "transformBy": { "dpt": 1, "name": 1, "sal": 1, "_id": 0 }, "inputStage": { "stage": "IXSCAN", "keyPattern": { "dpt": 1, "sal": -1, "name": 1 },

db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$project

$cursor


db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN

Index on "dpt":1, "sal":-1, "name":1

PROJECTION

"multiKeyPaths": { "dpt": [], "sal": [], "name": [] }, "isUnique": false, "isSparse": false, "isPartial": false, "indexVersion": 2, "direction": "forward", "indexBounds": { "dpt": [ "[\"DE\", \"DE\"]" ], "sal": [ "[MaxKey, MinKey]" ], "name": [ "[MinKey, MaxKey]" ] } } }, "rejectedPlans": []

db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN


PROJECTION


db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN


PROJECTION

Index used for sorting


db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN


PROJECTION


Pipeline Reordering:

Index used for matching


db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN


PROJECTION





db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN


PROJECTION




Projection Optimization:

Query turned to Covered Query


db.emps.aggregate([

]




,{explain:true})

DB Land

$group

$project

$cursor

IXSCAN


PROJECTION





Query turned to Covered Query


Index used


Query turned to

DB Land






DB Land

_id name dpt sal


DB Land

_id name dpt sal103161055108

bid

10 Jill UK 500_id name dpt sal bid

5 Erik US 400 10_id name dpt sal bid

7 Gina US 200 5_id name dpt sal bid

6 Fred US 300 5_id name dpt sal bid

9 Ivan NL 400 8_id name dpt sal bid

8 Herb NL 600 10_id name dpt sal bid

1 Alex DE 300 10_id name dpt sal bid

3 Cora DE 200 1_id name dpt sal bid

2 Bert DE 100 3_id name dpt sal bid

4 Drew US 200 6_id name dpt sal bid

db.emps.aggregate([












{$match: {name:"Bert"}},db.emps.aggregate([

])

Bert’s bosses

{$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}}

"bid" "_id""$bid""bosses"

$match $graphLookup













])

Bert’s bosses













{ }2 Bert DE 100 3_id name dpt sal bid

"bosses":[, ]



])

Bert’s bosses














"bosses":[, ]



])

Bert’s bosses














"bosses":[, ]



])

Bert’s bosses














"bosses":[, ]


"


])

Bert’s bosses














"bosses":[, ]3 Cora DE 200 1_id name dpt sal bid

"lvl":0,{ }


])

Bert’s bosses















"lvl":0,{ }



])

Bert’s bosses















"lvl":0,{ }


"


])

Bert’s bosses













{

}


"bosses":[,

]


"lvl":0,{ }


"lvl":1,{ }

,

db.emps.aggregate([

])

Bert’s bosses

{$match: {name:"Bert"}}, {$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}}












{

}


"bosses":[,

]


"lvl":0,{ }


"lvl":1,{ }

,



])

Bert’s bosses













{

}


"bosses":[,

]


"lvl":0,{ }


"lvl":1,{ }

,


"


])

Bert’s bosses













{

}


"bosses":[,

]


"lvl":0,{ }


"lvl":1,{ }

,


"lvl":2,{ }

,


])

Bert’s bosses













Bert’s bosses

{

}


"bosses":[,

]


"lvl":0,{ }


"lvl":1,{ }

,


"lvl":2,{ }

,


])













Bert’s bosses

{

}


"bosses":[,

]


"lvl":0,{ }


"lvl":1,{ }

,


"lvl":2,{ }

,


])













Bert’s bosses


])













The two employees bosseswith most



])

db.emps.aggregate([

, {$sort:{"bosses.leve:-1}}, {$limit:2} "bosses.lvl"

$limit$graphLookup $sort

8 Herb NL 600 10

10 Jill UK 500

5 Erik US 400 105 Erik US 400 10

10 Jill UK 500

6 Fred US 300 5

1 Alex DE 300 10

3 Cora DE 200 1

10 Jill UK 50010 Jill UK 500_id name dpt sal bid










1 Alex DE 300 102 Bert DE 100 33 Cora DE 200 14 Drew US 200 65 Erik US 400 106 Fred US 300 57 Gina US 200 58 Herb NL 600 109 Ivan NL 400 810 Jill UK 500

{ ,"bosses":[[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[{ ,"bosses":[]}]

]},]},]},]},]},]},]},]},]},


])

db.emps.aggregate([ {$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}}

"bid" "_id""$bid""bosses" ,

{$sort:{"bosses.leve:-1}}, {$limit:2} "bosses.lvl"

8 Herb NL 600 10

10 Jill UK 500

5 Erik US 400 105 Erik US 400 10

10 Jill UK 500

6 Fred US 300 5

1 Alex DE 300 10

3 Cora DE 200 1













]},]},]},]},]},]},]},]},]},


])




8 Herb NL 600 10

10 Jill UK 500

5 Erik US 400 105 Erik US 400 10

10 Jill UK 500

6 Fred US 300 5

1 Alex DE 300 10

3 Cora DE 200 1













]},]},]},]},]},]},]},]},]},


])















{ ,"bosses":[{ ,"lvl":0}[{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}

10 Jill UK 500

{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[]}]

3 Cora DE 200 11 Alex DE 300 106 Fred US 300 510 Jill UK 5005 Erik US 400 105 Erik US 400 1010 Jill UK 5008 Herb NL 600 10

]},]},]},]},]},]},]},]},]},


])




10 Jill UK 50010 Jill UK 50010 Jill UK 500

5 Erik US 400 10

10 Jill UK 500

1 Alex DE 300 10












{ ,"bosses":[{ ,"lvl":0}[{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}

10 Jill UK 500

{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[{ ,"lvl":0}{ ,"bosses":[]}]

3 Cora DE 200 11 Alex DE 300 106 Fred US 300 510 Jill UK 5005 Erik US 400 105 Erik US 400 1010 Jill UK 5008 Herb NL 600 10

]},]},]},

]},]},

]},]},

]},

]},


])















{ ,"bosses":[{ ,"lvl":0}[{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},

10 Jill UK 500

{ ,"bosses":[{ ,"lvl":0}{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},

{ ,"bosses":[{ ,"lvl":0}{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},

{ ,"bosses":[]}]

3 Cora DE 200 1 1 Alex DE 300 101 Alex DE 300 10 10 Jill UK 5006 Fred US 300 5 5 Erik US 400 1010 Jill UK 5005 Erik US 400 10 10 Jill UK 5005 Erik US 400 10 10 Jill UK 50010 Jill UK 5008 Herb NL 600 10 10 Jill UK 500

]},

]},

]},]},

]},]},

]},]},

]},


])




10 Jill UK 50010 Jill UK 50010 Jill UK 500_id name dpt sal bid











{ ,"bosses":[{ ,"lvl":0}[{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},

10 Jill UK 500



{ ,"bosses":[]}]

3 Cora DE 200 1 1 Alex DE 300 101 Alex DE 300 10 10 Jill UK 5006 Fred US 300 5 5 Erik US 400 1010 Jill UK 5005 Erik US 400 10 10 Jill UK 5005 Erik US 400 10 10 Jill UK 50010 Jill UK 5008 Herb NL 600 10 10 Jill UK 500

]},

]},

]},]},

]},]},

]},]},

]},


])















{ ,"bosses":[{ ,"lvl":0} }] ,[{ ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,

10 Jill UK 500



{ ,"bosses":[]}]

3 Cora DE 200 1 1 Alex DE 300 10 10 Jill UK 5001 Alex DE 300 10 10 Jill UK 5006 Fred US 300 5 5 Erik US 400 10 10 Jill UK 50010 Jill UK 5005 Erik US 400 10 10 Jill UK 5005 Erik US 400 10 10 Jill UK 50010 Jill UK 5008 Herb NL 600 10 10 Jill UK 500

}] ,

}] ,}] ,

]},

]},]},

]},]},

]},


])















{ ,"bosses":[{ ,"lvl":0} }] ,[{ ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,{ ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},{ ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,

10 Jill UK 500



{ ,"bosses":[]}]

3 Cora DE 200 1 1 Alex DE 300 10 10 Jill UK 5001 Alex DE 300 10 10 Jill UK 5006 Fred US 300 5 5 Erik US 400 10 10 Jill UK 50010 Jill UK 5005 Erik US 400 10 10 Jill UK 5005 Erik US 400 10 10 Jill UK 50010 Jill UK 5008 Herb NL 600 10 10 Jill UK 500

}] ,

}] ,}] ,

]},

]},]},

]},]},

]},


])














[

1 Alex DE 300 10{ ,"bosses":[{ ,"lvl":0} }] ,10 Jill UK 500

10 Jill UK 500{ ,"bosses":[]}]

5 Erik US 400 10{ ,"bosses":[{ ,"lvl":0}10 Jill UK 500 }] ,8 Herb NL 600 10{ ,"bosses":[{ ,"lvl":0}10 Jill UK 500 }] ,}] ,

2 Bert DE 100 3 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,3 Cora DE 200 1 1 Alex DE 300 10 10 Jill UK 500 ]},4 Drew US 200 6 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,6 Fred US 300 5 5 Erik US 400 10 10 Jill UK 500 ]},

6 Fred US 300 5 { ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},5 Erik US 400 10 10 Jill UK 500 ]},7 Gina US 200 5 { ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},5 Erik US 400 10 10 Jill UK 500 ]},

3 Cora DE 200 1 { ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},1 Alex DE 300 10 10 Jill UK 500 ]},

9 Ivan NL 400 8 { ,"lvl":1}{ ,"bosses":[{ ,"lvl":0},8 Herb NL 600 10 10 Jill UK 500 ]},


])
















[ 2 Bert DE 100 3 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,3 Cora DE 200 1 1 Alex DE 300 10 10 Jill UK 500 ]},4 Drew US 200 6 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,6 Fred US 300 5 5 Erik US 400 10 10 Jill UK 500 ]}]


])

db.emps.aggregate([

, {$sort:{"bosses.leve:-1}}, {$limit:2} "bosses.lvl"

{$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}} , {$sort:{"bosses.leve:-1}}, {$limit:2} "bosses.lvl"











db.emps.aggregate([

])

[ 2 Bert DE 100 3 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,3 Cora DE 200 1 1 Alex DE 300 10 10 Jill UK 500 ]},4 Drew US 200 6 { ,"lvl":1} { ,"lvl":2}{ ,"bosses":[{ ,"lvl":0}, ,6 Fred US 300 5 5 Erik US 400 10 10 Jill UK 500 ]}]



{$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}} , {$sort:{"bosses.leve:-1}}, {$limit:2} "bosses.lvl"











db.emps.aggregate([

])



{$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}} , {$sort:{"bosses.leve:-1}}, {$limit:2}











db.emps.aggregate([

])


The two employeesbosses with most

"bosses.lvl"












db.emps.aggregate([

])

"bid""_id""$bid""bosses"


"bosses.lvl"

, {$sort:{"bosses.leve:-1}}, {$limit:2}












db.emps.aggregate([

])


"emps""bosses.lvl"

"$_id" "_id" "bid"












db.emps.aggregate([

])


"bosses.lvl"

"$_id" "_id" "bid""emps" ,












db.emps.aggregate([

])



"hdcnt"












db.emps.aggregate([

])



"hdcnt"{$addFields: {hdcnt:{$size:"$emps"}}},

{$graphLookup: {from:"emps", startWith:"$bid", connectFromField:"bid", connectToField:"_id", dpthField:"lvl", as:"bosses"}} , {$sort:{"bosses:-1}}, {$limit:2}











db.emps.aggregate([

])















db.emps.aggregate([

])




$limit$graphLookup $sort$addFields












db.emps.aggregate([

])















db.emps.aggregate([

])




[{ ,"emps":[], "hdcnt":0}]

9 Ivan NL 400 8

[{ ,"emps":[], "hdcnt":0}

7 Gina US 200 5

[{ ,"emps":[], "hdcnt":0}

4 Drew US 200 6

[{ ,"emps":[], "hdcnt":0}

2 Bert DE 100 3

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

8 Herb NL 600 10 9 Ivan NL 400 8

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

6 Fred US 300 5 4 Drew US 200 610 Jill UK 500_id name dpt sal bid











db.emps.aggregate([

])




[ 5 Erik US 400 10 { ,"lvl":0} { ,"lvl":1}{ ,"emps":[{ ,"lvl":0}, ,7 Gina US 200 5 6 Fred US 300 5 4 Drew US 200 6 ], "hdcnt":3}

[{ ,"emps":[], "hdcnt":0}

7 Gina US 200 5

[{ ,"emps":[], "hdcnt":0}

4 Drew US 200 6

[{ ,"emps":[], "hdcnt":0}

2 Bert DE 100 3

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

3 Cora DE 200 1 2 Bert DE 100 3

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

8 Herb NL 600 10 9 Ivan NL 400 8

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

6 Fred US 300 5 4 Drew US 200 6

[ { ,"lvl":1}],{ ,"emps":[{ ,"lvl":0}, "hdcnt":2},

1 Alex DE 300 10 3 Cora DE 200 1 2 Bert DE 100 3

, 10 Jill UK 500_id name dpt sal bid











db.emps.aggregate([

])




[ 10 Jill UK 500{ ,"emps":[ { ,"lvl":0} { ,"lvl":0}{ ,"lvl":0}, , ,{ ,"lvl":1} { ,"lvl":1}{ ,"lvl":1}, , ,{ ,"lvl":2} { ,"lvl":2}{ ,"lvl":1}, , 4 Drew US 200 6 ],2 Bert DE 100 3

1 Alex DE 300 10 5 Erik US 400 10 8 Herb NL 600 103 Cora DE 200 1 7 Gina US 200 5 6 Fred US 300 59 Ivan NL 400 8

"hdcnt":9},


[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

3 Cora DE 200 1 2 Bert DE 100 3

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

8 Herb NL 600 10 9 Ivan NL 400 8

[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

6 Fred US 300 5 4 Drew US 200 6



,












db.emps.aggregate([

])






"hdcnt":9},


[{ ,"emps":[{ ,"lvl":0}], "hdcnt":1},

3 Cora DE 200 1 2 Bert DE 100 3



,












db.emps.aggregate([

])





db.emps.aggregate(["$_id" "_id" "bid""emps" ,

"hdcnt" ,{$addFields: {hdcnt:{$size:"$emps"}}},

























"hdcnt":9},


},

]}

{$addFields: {tsal:{$reduce: { input: "$emps", initialValue: 0, in: {$add: ["$$value", "$$this.sal"]}}}}}])




























"hdcnt":9},


},

]}





























"hdcnt":9},


},

]}

$limit$graphLookup $sort$addFields





























"hdcnt":9},


},

]}

$addFields

$reduce$limit$graphLookup $sort$addFields





























"hdcnt":9},


},

]}





























"hdcnt":9},


},

]}





























"hdcnt":9},


},

]}

, "tsal":2700

, "tsal":700





















MongoDB


_id name dpt sal bid

MongoDB


_id name dpt sal bidMunichHamburgEssen

ChicagoBostonOrlandoAustinUtrechtArnheimLondon

loc

MongoDB


MunichHamburgEssen


_id name dpt sal bid loc


MunichHamburgEssen



Chicago Boston

Austin

Orlando


MunichHamburgEssen



Chicago Boston

Austin

Orlando

✈


MunichHamburgEssen



Chicago Boston

Austin

Orlando

$


MunichHamburgEssen



Chicago Boston

Austin

Orlando

$1 Alex DE 300 102 Bert DE 100 33 Cora DE 200 14 Drew US 200 65 Erik US 400 106 Fred US 300 57 Gina US 200 58 Herb NL 600 109 Ivan NL 400 810 Jill UK 500

MunichHamburgEssen



Chicago Boston

Austin

Orlando

$


MunichHamburgEssen



Chicago Boston

Austin

Orlando

$


MunichHamburgEssen



Chicago Boston

Austin

Orlando


MunichHamburgEssen



$

Chicago Boston

Austin

Orlando

✈


MunichHamburgEssen



Chicago Boston

Austin

Orlando

✈


MunichHamburgEssen



Chicago Boston

Austin

Orlando

Chicago Boston

Austin

Orlando

Boston 980 ChicagoBoston 2000 AustinBoston 1300 OrlandoChicago 980 BostonChicago 1150 AustinChicago 1200 OrlandoAustin 2000 BostonAustin 1150 ChicagoAustin 1100 OrlandoOrlando 1300 BostonOrlando 1200 ChicagoOrlando 1100 Austin

start enddist

Orlando 1300 BostonBoston 980 ChicagoChicago 1200 OrlandoOrlando 1100 AustinAustin 2000 BostonBoston 980 ChicagoChicago 1150 AustinAustin 1100 OrlandoOrlando 1300 BostonBoston 1300 OrlandoOrlando 1100 AustinAustin 1150 ChicagoChicago 980 BostonBoston 1300 OrlandoOrlando 1200 ChicagoChicago 1150 AustinAustin 2000 BostonBoston 2000 AustinAustin 1100 OrlandoOrlando 1200 ChicagoChicago 980 BostonBoston 2000 AustinAustin 1150 ChicagoChicago 1200 Orlando

…,{$graphLookup:{from:"dists",startWith:"$end",connectFromField:"end",connectToField:"start", dpthField:"lvl", as:"routes"}}, …, {$sort:{"tdist":1}}, {$limit:1})]

start enddist

Boston 980 ChicagoBoston 2000 AustinBoston 1300 OrlandoChicago 980 BostonChicago 1150 AustinChicago 1200 OrlandoAustin 2000 BostonAustin 1150 ChicagoAustin 1100 OrlandoOrlando 1300 BostonOrlando 1200 ChicagoOrlando 1100 Austin

Chicago Boston

Austin

Orlando$limit$graphLookup $sort...


5280Boston 980 Chicago Chicago 1200 Orlando Orlando 1100 Austin Austin 2000 Boston

4530Boston 980 Chicago Chicago 1150 Austin Austin 1100 Orlando Orlando 1300 Boston

4530Boston 1300 Orlando Orlando 1100 Austin Austin 1150 Chicago Chicago 980 Boston

5650Boston 1300 Orlando Orlando 1200 Chicago Chicago 1150 Austin Austin 2000 Boston

5280Boston 2000 Austin Austin 1100 Orlando Orlando 1200 Chicago Chicago 980 Boston

5650Boston 2000 Austin Austin 1150 Chicago Chicago 1200 Orlando Orlando 1300 Boston

start enddist start enddist start enddist start enddist tdist

Chicago Boston

Austin

Orlando


5280Boston 980 Chicago Chicago 1200 Orlando Orlando 1100 Austin Austin 2000 Boston

4530Boston 1300 Orlando Orlando 1100 Austin Austin 1150 Chicago Chicago 980 Boston

5650Boston 1300 Orlando Orlando 1200 Chicago Chicago 1150 Austin Austin 2000 Boston

5280Boston 2000 Austin Austin 1100 Orlando Orlando 1200 Chicago Chicago 980 Boston

5650Boston 2000 Austin Austin 1150 Chicago Chicago 1200 Orlando Orlando 1300 Boston


Chicago Boston

Austin

Orlando




Chicago Boston

Austin

Orlando



Chicago Boston

Austin

Orlando

4530Boston 980 Chicago Chicago 1150 Austin Austin 1100 Orlando Orlando 1300 Boston✈


Chicago Boston

Austin

Orlando


✈


Chicago Boston

Austin

Orlando


$


Chicago Boston

Austin

Orlando


$


Chicago Boston

Austin

Orlando


$


Chicago Boston

Austin

Orlando


$


Chicago Boston

Austin

Orlando


$


Chicago Boston

Austin

Orlando


✈


Chicago Boston

Austin

Orlando

4530Boston 980 Chicago Chicago 1150 Austin Austin 1100 Orlando Orlando 1300 Boston ✈


4530Boston 980 Chicago Chicago 1150 Austin Austin 1100 Orlando Orlando 1300 Boston ✈

Facets{

MongoDB

MongoDB

_id name dpt sal

MongoDB

_id name dpt sal

MongoDB

_id name dpt sal expertise

MongoDB

_id name dpt sal expertise5 results for mongodb

Alexlocation: DE salary: 300 expertise: MongoDB, Ruby

Bertlocation: DE salary: 100 expertise: MongoDB, Java

Coralocation: DE salary: 200 expertise: MongoDB, Java

Eriklocation: US salary: 400 expertise: MongoDB, Ruby

Herblocation: NL salary: 600 expertise: MongoDB, Scala

Location

AllDENL

SalaryAll100 - 200200 - 400400 - 600

ExpertiseAllMongoDBJava

Scala

https://www.awesomeempview.com/search/f?type=all&key…

Search | EmpView

mongodb

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔

3 results for oracle

Fredlocation: US salary: 300 expertise: Oracle, Cobol

Ginalocation: US salary: 200 expertise: Oracle, Pascal

Ivanlocation: NL salary: 400 expertise: Oracle, Fortran

Search | EmpView

Location

AllUSNL

✔

ExpertiseAllOracleCobolPascalFortran

✔

SalaryAll100 - 200200 - 400

✔

oracle

(2)(1)

(3)(1)(1)(1)

(1)(2)

https://www.awesomeempview.com/search/f?type=all&key… MongoDB


MongoDB







LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb

5 results for mongodb






LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb








LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb



db.emps.aggregate([ { $match: { $text : { $search : "mongodb" }}}, { $sortByCount: "$dpt"}])






LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb









LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb


{"_id":"DE", "count":3},

{"_id":"NL", "count":1},

{"_id":"US", "count":1}








LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb


db.emps.aggregate( [ {$match: { $text : {$search : "mongodb" }}}, {$unwind: "$expertise"}, {$sortByCount: "$expertise"}])


{"_id":"MongoDB","count":5},

{"_id":"Java","count":2},

{"_id":"Ruby","count":2},

{"_id":"Scala","count":1}






LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb


db.emps.aggregate([ {$match: { $text : {$search : "mongodb" }}}, {$bucket: { groupBy: "$sal", boundaries: [100, 201, 401, 601, Infinity]}} ])

{"_id":100,"count":2},

{"_id":201,"count":2},

{"_id":401,"count":1}







LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb


db.emps.aggregate([ {$match: { $text : {$search : "mongodb" }}}, {$bucket: { groupBy: "$sal", boundaries: [100, 201, 401, 601, Infinity], output: { count: {$sum: 1}, matches: {$push: "$$CURRENT"} }}} ])

{"_id":100,"count":2,"matches":[

]}, {"_id":201,"count":2,"matches":[

]}, {"_id":401,"count":1,"matches":[

]}







LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb


db.emps.aggregate([ {$match: { $text : {$search : "mongodb" }}}, {$bucketAuto: { groupBy: "$sal", buckets: 4}} ])

{"_id":{"min":100,"max":200}, "count":1},

{"_id":{"min":200,"max":300}, "count":1},

{"_id":{"min":300,"max":400}, “count":1},

{"_id":{"min":400,"max":600}, "count":2}







LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb


db.emps.aggregate([ {$match: { $text : {$search : {$facet: { "Location": [ {$sortByCount: "$dpt"} ], "Expertise":[ {$unwind: "$expertise"}, {$sortByCount: "$expertise"} ], "Salary":[ {$bucket: { groupBy: "$sal", boundaries: [100, 201, 401, 601, Infinity]}} ]}}])

{"Location":[ {"_id":"DE","count":3}, {"_id":"NL","count":1}, {"_id":"US","count":1}], "Expertise":[ {"_id":"MongoDB","count":5}, {"_id":"Java","count":2}, {"_id":"Ruby","count":2}, {"_id":"Scala","count":1}], "Salary":[ {"_id":100,"count":2}, {"_id":201,"count":2}, {"_id":401,"count":1}]}


}}},"mongodb"






LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb





}}},"mongodb"






LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb





}}},"mongodb"






LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb





}}},"\"mongodb\" \"ruby\""






LocationAllDENL

Salary

All100 - 200200 - 400400 - 600

Expertise

AllMongoDBJava

Scala

✔

(1)

(2)

(1)

(2)(2)(1)

✔

3( )

( )5

Ruby (2)

US (1)

✔


Search | EmpView

mongodb





{"Location":[ {"_id":"DE","count":1}, {"_id":"US","count":1}], "Expertise":[ {"_id":"MongoDB","count":2}, {"_id":"Ruby","count":2}], "Salary":[ {"_id":201,"count":2}]}

MongoDB



LocationAllDE

Salary

All200 - 400

Expertise

AllMongoDB

✔

(2)

✔

1(

( )2Ruby (2)

US (1)

✔


Search | EmpView

mongodb





{"Location":[ {"_id":"DE","count":1}, {"_id":"US","count":1}], "Expertise":[ {"_id":"MongoDB","count":2}, {"_id":"Ruby","count":2}], "Salary":[ {"_id":201,"count":2}]})

MongoDB

MongoDB

One more thing…

PL Land

PL Land

nats = nats. (nats. {|n| nats. (n-1) . (1) . {|p| n % p != 0} . })

droptake

mapall?

mapzip[1,2,3,4,5,6,7,8,9,10]

Ruby

PL Land

[[1, true ], [2, true ], [3, true ], [4, false], [5, true ], [6, false], [7, true ], [8, false], [9, false], [10,false]]

nats = nats. (nats. {|n| nats. (n-1) . (1) . {|p| n % p != 0} . })

droptake

mapall?

mapzip[1,2,3,4,5,6,7,8,9,10]

Ruby

PL Land

droptake

mapall?

mapzip[1,2,3,4,5,6,7,8,9,10]

Ruby

PL Land

droptake

mapall?

mapzip$range: [1, 11, 1]

Ruby

PL Land

droptake

mapall?

map$zip$range: [1, 11, 1]

Ruby

PL Land

droptake

mapall?


["A", "B", "C"][10, 20, 30]

Ruby

PL Land

droptake

mapall?


["A", "B", "C"][10, 20, 30]

[["A",10], ["B",20], ["C",30]]

Ruby

PL Land

droptake

mapall?

$map$zip$range: [1, 11, 1]

Ruby

PL Land

dropmapall?

$slice$map$zip$range: [1, 11, 1]

Ruby

PL Land

mapall?

$slice$slice


Ruby

PL Land

all?

$slice$slice$map


Ruby

PL Land

$slice$slice$map

$allElementsTrue


Ruby

PL Land

$slice$slice

$map$allElementsTrue

$map$zip

$range: [1, 11, 1]Ruby

$let: {vars: {nats: { }}, in: { : {inputs: ["$$nats", { : {input: "$$nats", as: "n", in: { : [{ : {input: { : [{ : ["$$nats", {$subtract: ["$$n", 1]}]}, 1, {$size: "$$nats"}]}, as: "p", in: { $ne: [{$mod: ["$$n", "$$p"]}, 0]}}}]}}}]}}}

PL Land

$slice$slice


$map$zip

: [1, 11, 1]$rangeAggregation Pipeline


PL Land

$slice$slice


$map$zip


$allElementsTrue$map$slice$slice

$map

$range $zip

MongoDB

PL Land


$slice$slice


$map$zip



$map

$range $zip

MongoDB

PL Land

[[1, true ],[2, true ],[3, true ],[4, false],[5, true ],[6, false],[7, true ],[8, false],[9, false],[10,false]]


$slice$slice


$map$zip



$map

$range $zip

$slice$slice


$map$zip

$range

$slice

$map

$allElementsTrue

$zip

$range

$indexStats

$stdDevSamp

$stdDevPop

$sqrt

$abs

$log

$log10$ln $pow

$exp

$trunc

$ceil

$floor$arrayElemAt $concatArrays

$isArray$filter

$avg

$min

$max

$sum$stdDevPop

$stdDevSamp

$split

$indexOf

$reduce

$reverseArray

$indexOfArray

$type

$isoWeekYear$isoDayOfWeek

$isoWeek

$in

$switch$count

$replaceRoot

$addFields

$graphLookup $facet

$bucket $bucketAuto $sortByCount$out$redact

$setEquals

$setIntersection$setUnion$setDifference

$setIsSubset $anyElementTrue$literal

$size

$geoNear $millisecond $concat

$project $match $limit$skip

$unwind $group $sort

$and $or

$not

$cmp$eq $gt

$gte

$lt$lte

$ne

$abs

$add

$ceil

$divide

$mod

$multiply

$subtract

$toLower

$toUpper

$strcasecmp$meta

$let$dayOfYear

$dayOfMonth

$dayOfWeek

$year$month

$week

$hour

$minute

$second

$millisecond

$dateToString

$cond

$ifNull

$first$last

$push

$addToSet

$indexOfBytes

$indexOfCP

$strLenBytes$strLenCP

$substrBytes$substrCP

$slice

$map

$allElementsTrue

$zip

$range

$sample$lookup

$dateToString

$collStats

3.0

3.4

3.2

2.6

2.4

2.2

$indexStats

$stdDevSamp

$stdDevPop

$sqrt

$abs

$log

$log10$ln $pow

$exp

$trunc

$ceil


$isArray$filter

$avg

$min

$max

$sum$stdDevPop

$stdDevSamp

$out$redact

$setEquals



$size

$geoNear $millisecond $concat



$and $or

$not

$cmp$eq $gt

$gte

$lt$lte

$ne

$abs

$add

$ceil

$divide

$mod

$multiply

$subtract

$toLower

$toUpper

$strcasecmp$meta

$let$dayOfYear

$dayOfMonth

$dayOfWeek

$year$month

$week

$hour

$minute

$second

$millisecond

$dateToString

$cond

$ifNull

$first$last

$push

$addToSet

$slice

$map

$allElementsTrue

$sample$lookup

$dateToString

$split

$indexOf

$reduce

$reverseArray

$indexOfArray

$type

$isoWeekYear$isoDayOfWeek

$isoWeek

$in

$switch$count

$replaceRoot

$addFields

$graphLookup $facet

$bucket $bucketAuto $sortByCount$indexOfBytes

$indexOfCP

$strLenBytes$strLenCP

$substrBytes$substrCP

$zip

$range

$collStats

2.4 $geoNear $millisecond $concat

2.2

$avg

$min

$max

$sum$stdDevPop

$stdDevSamp



$and $or

$not

$cmp$eq $gt

$gte

$lt$lte

$ne

$abs

$add

$ceil

$divide

$mod

$multiply

$subtract

$toLower

$toUpper

$strcasecmp$meta

$let$dayOfYear

$dayOfMonth

$dayOfWeek

$year$month

$week

$hour

$minute

$second

$millisecond

$dateToString

$cond

$ifNull

$first$last

$push

$addToSet

$map

2.6 $out$redact

$setEquals



$size

$allElementsTrue

3.2$indexStats

$stdDevSamp

$stdDevPop

$sqrt

$abs

$log

$log10$ln $pow

$exp

$trunc

$ceil


$isArray$filter

$slice

$sample$lookup

3.0$dateToString

3.4

$indexOf $indexOfArray

$type

$isoWeekYear$in

$switch$count

$replaceRoot

$addFields

$graphLookup $facet


$indexOfCP$substrBytes$substrCP

$zip

$range

$split

$reduce

$reverseArray $isoWeekYear$isoDayOfWeek

$isoWeek

$strLenBytes$strLenCP $collStats

Data-Parallel Pipeline Execution

Shard 1 Shard 2 Shard 3 Shard N

mongos

Q

PipelineOptimisations

Pipeline Projection Optimization

Pipeline Coalescence Optimization

. . .

Pipeline Reordering

Pipeline Combining

2.4 $geoNear $millisecond $concat

2.2

$avg

$min

$max

$sum$stdDevPop

$stdDevSamp



$and $or

$not

$cmp$eq $gt

$gte

$lt$lte

$ne

$abs

$add

$ceil

$divide

$mod

$multiply

$subtract

$toLower

$toUpper

$strcasecmp$meta

$let$dayOfYear

$dayOfMonth

$dayOfWeek

$year$month

$week

$hour

$minute

$second

$millisecond

$dateToString

$cond

$ifNull

$first$last

$push

$addToSet

$map

2.6 $out$redact

$setEquals



$size

$allElementsTrue

3.2$indexStats

$stdDevSamp

$stdDevPop

$sqrt

$abs

$log

$log10$ln $pow

$exp

$trunc

$ceil


$isArray$filter

$slice

$sample$lookup

3.0$dateToString

3.4

$indexOf $indexOfArray

$type

$isoWeekYear$in

$switch$count

$replaceRoot

$addFields

$graphLookup $facet


$indexOfCP$substrBytes$substrCP

$zip

$range

$split

$reduce

$reverseArray $isoWeekYear$isoDayOfWeek

$isoWeek

$strLenBytes$strLenCP $collStats





Use the Force Aggregation Pipelines!

Use the Force Aggregation Pipelines! And take control over your Data.

ASK THE EXPERTSGet your technical questions answered

By appointment only – register in one of two ways:

In person Online

Come to the MongoDB Team Stand on the ground floor

calendly.com/mongodb

Data & Analytics

MongoDB Europe 2016 - Advanced MongoDB Aggregation Pipelines