Upload
nicolas-motte
View
409
Download
1
Embed Size (px)
Citation preview
Confi
dent
ial
RESTRICTED
© 2
014
Amad
eus I
T Gr
oup
SA
Confi
dent
ial
RESTRICTED
MongoDB Monitoring & Operability
Training for App Mgmt and Operations
Nicolas Motte, 18/11/2014
Confi
dent
ial
RESTRICTED
Page 2
© 2
014
Amad
eus I
T Gr
oup
SA
Link to this presentation
_ http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Mongos_and_MongoDB_OSAG• Section 10. Training
_ http://gmstst.muc.amadeus.net/trends/• NoSQL section• Select “Mongos and MongoDB OSAG”• Section 10. Training
Confi
dent
ial
RESTRICTED
Page 3
© 2
014
Amad
eus I
T Gr
oup
SA
Agenda_ Overview of MongoDB_ Amadeus Architecture around NoSQL_ MongoDB concepts_ Workflows_ Operability & Monitoring
414213741
© 2
014
Amad
eus I
T Gr
oup
SA
Confi
dent
ial
RESTRICTED
Overview of MongoDB1
Confi
dent
ial
RESTRICTED
Page 5
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Document store
High-level
Confi
dent
ial
RESTRICTED
Page 6
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Document store• Stores JSON-style documents
High-level
Confi
dent
ial
RESTRICTED
Page 7
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Document store• Stores JSON-style documents• Uses BSON format for storage
High-level
Confi
dent
ial
RESTRICTED
Page 8
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Document store• Stores JSON-style documents• Uses BSON format for storage
_ Several binaries written in C++
High-level
Confi
dent
ial
RESTRICTED
Page 9
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Document store• Stores JSON-style documents• Uses BSON format for storage
_ Several binaries written in C++• Mongod : The actual database process, owning the data
High-level
Confi
dent
ial
RESTRICTED
Page 10
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Document store• Stores JSON-style documents• Uses BSON format for storage
_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer
High-level
Confi
dent
ial
RESTRICTED
Page 11
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Document store• Stores JSON-style documents• Uses BSON format for storage
_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology
High-level
Confi
dent
ial
RESTRICTED
Page 12
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Document store• Stores JSON-style documents• Uses BSON format for storage
_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology• Agent : Monitoring process
High-level
Confi
dent
ial
RESTRICTED
Page 13
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Document store• Stores JSON-style documents• Uses BSON format for storage
_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology• Agent : Monitoring process• MMS : Monitoring UI, connected to the Agents
High-level
Confi
dent
ial
RESTRICTED
Page 14
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Document store• Stores JSON-style documents• Uses BSON format for storage
_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology• Agent : Monitoring process• MMS : Monitoring UI, connected to the Agents
_ Opensource and Enterprise versions
High-level
Confi
dent
ial
RESTRICTED
Page 15
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Mongod
Mongod
Mongod
Architecture
Confi
dent
ial
RESTRICTED
Page 16
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Mongod
Mongod
Mongod
Mongos
Architecture
Confi
dent
ial
RESTRICTED
Page 17
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Mongod
Mongod
Mongod
Mongos
Config
Architecture
Confi
dent
ial
RESTRICTED
Page 18
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Mongod
Mongod
Mongod
Mongos
ConfigClient
Client
Client
Architecture
Confi
dent
ial
RESTRICTED
Page 19
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Mongod
Mongod
Mongod
Mongos
ConfigClient
Client
Client
Agent
Architecture
Confi
dent
ial
RESTRICTED
Page 20
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Mongod
Mongod
Mongod
Mongos
ConfigClient
Client
Client
Agent
MMS
Architecture
Confi
dent
ial
RESTRICTED
Page 21
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Why do we use MongoDB ?Couchbase
MemcachedMongoDB
Oracle
Comparison
Confi
dent
ial
RESTRICTED
Page 22
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Memcached Couchbase MongoDB OracleDatabase Model Key-Value Key-Value Document TableData scheme Schema-free Schema-free Schema-free Schema-fullIndexesPersistencyData IntegrityTransaction
Comparison
Confi
dent
ial
RESTRICTED
Page 23
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Memcached Couchbase MongoDB OracleDatabase Model Key-Value Key-Value Document TableData scheme Schema-free Schema-free Schema-free Schema-fullIndexesPersistencyData IntegrityTransaction
Comparison
Confi
dent
ial
RESTRICTED
Page 24
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Memcached Couchbase MongoDB OracleDatabase Model Key-Value Key-Value Document TableData scheme Schema-free Schema-free Schema-free Schema-fullIndexesPersistencyData IntegrityTransaction
Comparison
Confi
dent
ial
RESTRICTED
Page 25
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Memcached Couchbase MongoDB OracleDatabase Model Key-Value Key-Value Document TableData scheme Schema-free Schema-free Schema-free Schema-fullIndexesPersistencyData IntegrityTransaction
Comparison
Confi
dent
ial
RESTRICTED
Page 26
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Memcached Couchbase MongoDB OracleDatabase Model Key-Value Key-Value Document TableData scheme Schema-free Schema-free Schema-free Schema-fullIndexesPersistencyData IntegrityTransaction
Comparison
Confi
dent
ial
RESTRICTED
Page 27
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Memcached Couchbase MongoDB OracleDatabase Model Key-Value Key-Value Document TableData scheme Schema-free Schema-free Schema-free Schema-fullIndexesPersistencyData IntegrityTransaction
Comparison
Confi
dent
ial
RESTRICTED
Page 28
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
Memcached Couchbase MongoDB OracleDatabase Model Key-Value Key-Value Document TableData scheme Schema-free Schema-free Schema-free Schema-fullIndexesPersistencyData IntegrityTransaction
Comparison
Confi
dent
ial
RESTRICTED
Page 29
Overview of MongoDB
_ Mongod is the process storing the data
Mongod
Confi
dent
ial
RESTRICTED
Page 30
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongod is the process storing the data_ 3 different types
Mongod
Confi
dent
ial
RESTRICTED
Page 31
Overview of MongoDB
_ Mongod is the process storing the data_ 3 different types• PRIMARY
Mongod
Confi
dent
ial
RESTRICTED
Page 32
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongod is the process storing the data_ 3 different types• PRIMARY• The primary is the only process that
receives write operations
Mongod
Confi
dent
ial
RESTRICTED
Page 33
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongod is the process storing the data_ 3 different types• PRIMARY• The primary is the only process that
receives write operations• SECONDARY
Mongod
Confi
dent
ial
RESTRICTED
Page 34
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongod is the process storing the data_ 3 different types• PRIMARY• The primary is the only process that
receives write operations• SECONDARY• A secondary maintains a copy of the
primary’s data set
Mongod
Confi
dent
ial
RESTRICTED
Page 35
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongod is the process storing the data_ 3 different types• PRIMARY• The primary is the only process that
receives write operations• SECONDARY• A secondary maintains a copy of the
primary’s data set• ARBITER
Mongod
Confi
dent
ial
RESTRICTED
Page 36
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongod is the process storing the data_ 3 different types• PRIMARY• The primary is the only process that
receives write operations• SECONDARY• A secondary maintains a copy of the
primary’s data set• ARBITER• An arbiter does not have a copy of
data set and cannot become a primary
Mongod
Confi
dent
ial
RESTRICTED
Page 37
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongod is the process storing the data_ 3 different types• PRIMARY• The primary is the only process that
receives write operations• SECONDARY• A secondary maintains a copy of the
primary’s data set• ARBITER• An arbiter does not have a copy of
data set and cannot become a primary
Mongod
Confi
dent
ial
RESTRICTED
Page 38
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongod is the process storing the data_ 3 different types• PRIMARY• The primary is the only process that
receives write operations• SECONDARY• A secondary maintains a copy of the
primary’s data set• ARBITER• An arbiter does not have a copy of
data set and cannot become a primary_ Heartbeats are sent every 10s
between mongods
Mongod
Heartbeat
Confi
dent
ial
RESTRICTED
Page 39
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Config servers are special mongod instances that store the metadata for a cluster (like topology)
Config Server
Confi
dent
ial
RESTRICTED
Page 40
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Config servers are special mongod instances that store the metadata for a cluster (like topology)
_ A production cluster has exactly three config servers
Config Server
Confi
dent
ial
RESTRICTED
Page 41
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongos is a connection multiplexer
_ Heartbeat are sent every 10s
_ Map/Reduce queries
Mongos
Hun
dred
s of
clie
nts
A single connection/server
Clients are not aware of the topology
Mongos hides all the complexity and standardizes
the configuration
MongosMongod
Mongod
Mongod
Config
Heartbeat
Confi
dent
ial
RESTRICTED
Page 42
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongos is a connection multiplexer
Mongos
Mongos
Confi
dent
ial
RESTRICTED
Page 43
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongos is a connection multiplexer
Mongos
Hun
dred
s of
clie
nts
MongosMongod
Mongod
Mongod
Config
Confi
dent
ial
RESTRICTED
Page 44
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongos is a connection multiplexer
Mongos
Hun
dred
s of
clie
nts
A single connection/server
MongosMongod
Mongod
Mongod
Config
Confi
dent
ial
RESTRICTED
Page 45
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongos is a connection multiplexer
Mongos
Hun
dred
s of
clie
nts
A single connection/server
Clients are not aware of the topology
MongosMongod
Mongod
Mongod
Config
Confi
dent
ial
RESTRICTED
Page 46
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongos is a connection multiplexer
Mongos
Hun
dred
s of
clie
nts
A single connection/server
Clients are not aware of the topology
Mongos hides all the complexity and standardizes
the configuration
MongosMongod
Mongod
Mongod
Config
Confi
dent
ial
RESTRICTED
Page 47
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongos is a connection multiplexer
_ Heartbeat are sent every 10s
Mongos
Hun
dred
s of
clie
nts
A single connection/server
Clients are not aware of the topology
Mongos hides all the complexity and standardizes
the configuration
MongosMongod
Mongod
Mongod
Config
Heartbeat
Confi
dent
ial
RESTRICTED
Page 48
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Mongos is a connection multiplexer
_ Heartbeat are sent every 10s
_ Map/Reduce queries
Mongos
Hun
dred
s of
clie
nts
A single connection/server
Clients are not aware of the topology
Mongos hides all the complexity and standardizes
the configuration
MongosMongod
Mongod
Mongod
Config
Heartbeat
Confi
dent
ial
RESTRICTED
Page 49
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Monitoring process
Agent
Agent
Confi
dent
ial
RESTRICTED
Page 50
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Monitoring process_ 1 active/1 passive
per cluster
Agent
Agent
Confi
dent
ial
RESTRICTED
Page 51
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Monitoring process_ 1 active/1 passive
per cluster_ Retrieves the
config servers from MMS
Agent
Agent
MMS
Confi
dent
ial
RESTRICTED
Page 52
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Monitoring process_ 1 active/1 passive
per cluster_ Retrieves the
config servers from MMS
_ Retrieves the list of processes from a config server
Agent
Config
Agent
Confi
dent
ial
RESTRICTED
Page 53
© 2
014
Amad
eus I
T Gr
oup
SA
Overview of MongoDB
_ Monitoring process_ 1 active/1 passive
per cluster_ Retrieves the
config servers from MMS
_ Retrieves the list of processes from a config server
_ Connects to all processes
Agent
Mongod
Mongod
Mongod
Mongos
Config
Agent
MMS
Confi
dent
ial
RESTRICTED
Page 54
© 2
014
Amad
eus I
T Gr
oup
SA
_ Datastore• Pros• Structured data• Flexible (Schema-less)• Indexes• Scaling is “easy”• Easy to deploy• Cons• Transactions are not yet supported• Each command is atomic but locking is at database level in 2.6 (moving to
document level in 2.8)_ Analytical search• MSX
Use CasesOverview of MongoDB
© 2
014
Amad
eus I
T Gr
oup
SA
Confi
dent
ial
RESTRICTED
Amadeus Architecture Around NoSQL2
Confi
dent
ial
RESTRICTED
Page 56
© 2
014
Amad
eus I
T Gr
oup
SA
Mongos
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongos ?
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 57
© 2
014
Amad
eus I
T Gr
oup
SA
Mongos
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongos ?
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 58
© 2
014
Amad
eus I
T Gr
oup
SA
Mongos
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongos ?_ Deployed on each AS
mongos
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 59
© 2
014
Amad
eus I
T Gr
oup
SA
Mongos
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongos ?_ Deployed on each AS_ BEs are connected via the
standard Mongo C++ client
BE BE BEBE
mongos
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 60
© 2
014
Amad
eus I
T Gr
oup
SA
Mongos
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongos ?_ Deployed on each AS_ BEs are connected via the
standard Mongo C++ client
_ Mongos is connected to all the MongoDB servers
BE BE BEBE
mongos
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 61
© 2
014
Amad
eus I
T Gr
oup
SA
Mongos
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongos ?_ Deployed on each AS_ BEs are connected via the
standard Mongo C++ client
_ Mongos is connected to all the MongoDB servers
_ Duwrapped process monitored by the local Agent
SI Agent
BE BE BEBE
mongosdwi
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 62
© 2
014
Amad
eus I
T Gr
oup
SA
Mongos
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongos ?_ Deployed on each AS_ BEs are connected via the
standard Mongo C++ client
_ Mongos is connected to all the MongoDB servers
_ Duwrapped process monitored by the local Agent
_ Equivalent of ProxyD for Memcached and Couchbase
SI Agent
BE BE BEBE
mongosdwi
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 63
© 2
014
Amad
eus I
T Gr
oup
SA
Mongos
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongos ?_ Deployed on each AS_ BEs are connected via the
standard Mongo C++ client
_ Mongos is connected to all the MongoDB servers
_ Duwrapped process monitored by the local Agent
_ Equivalent of ProxyD for Memcached and Couchbase
_ Also running on NoSQL Tier for administration tasks
SI Agent
BE BE BEBE
mongosdwi
mongos
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 64
© 2
014
Amad
eus I
T Gr
oup
SA
Mongod
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongod ?
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 65
© 2
014
Amad
eus I
T Gr
oup
SA
Mongod
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongod ?
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 66
© 2
014
Amad
eus I
T Gr
oup
SA
Mongod
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongod ?_ Deployed on cluster
nodes
Amadeus Architecture Around NoSQL
mongodmongod
mongodmongod
Confi
dent
ial
RESTRICTED
Page 67
© 2
014
Amad
eus I
T Gr
oup
SA
Mongod
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongod ?_ Deployed on cluster
nodes_ Accepts connections
from Mongos on App Tier and NoSQL Tiermongod
mongodmongod
mongod
mongos
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 68
© 2
014
Amad
eus I
T Gr
oup
SA
Mongod
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Mongod ?_ Deployed on cluster
nodes_ Accepts connections
from Mongos on App Tier and NoSQL Tier
_ A mongod can be a primary, a secondary or an arbiter
mongodmongod
mongodmongod
mongos
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 69
© 2
014
Amad
eus I
T Gr
oup
SA
Agent
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Agent?
mongodmongod
mongodmongod
mongos
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 70
© 2
014
Amad
eus I
T Gr
oup
SA
Agent
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Agent?_ Deployed on 2
cluster nodes, 1 active / 1 passive
mongodmongod
mongodmongod
mongos
agent
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 71
© 2
014
Amad
eus I
T Gr
oup
SA
Agent
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Agent?_ Deployed on 2
cluster nodes, 1 active / 1 passive
_ Connects to Mongos to monitor all processes
mongodmongod
mongodmongod
mongos
agent
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 72
© 2
014
Amad
eus I
T Gr
oup
SA
Config Server
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Config servers?
mongodmongod
mongodmongod
mongos
agent
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 73
© 2
014
Amad
eus I
T Gr
oup
SA
Config Server
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Config servers?_ Running on
dedicated nodes
mongodmongod
mongodmongod
mongos
agent
config
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 74
© 2
014
Amad
eus I
T Gr
oup
SA
Config Server
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Config servers?_ Running on
dedicated nodes_ Accepts connections
from mongosmongod
mongodmongod
mongod
mongos
agent
config
mongos
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 75
© 2
014
Amad
eus I
T Gr
oup
SA
Config Server
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
(GOS, DZB)
_ Config servers?_ Running on
dedicated nodes_ Accepts connections
from mongos_ These nodes are
treated like any other cluster nodes in our tools
mongodmongod
mongodmongod
mongos
agent
config
mongos
Amadeus Architecture Around NoSQL
Confi
dent
ial
RESTRICTED
Page 76
© 2
014
Amad
eus I
T Gr
oup
SA
Amadeus Architecture Around NoSQLConfiguration
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
MongoS port retrieved from
the MAGdwi
Duwrapper Conf(listen port)
+Configuration files
Keyfile_xxxMongos_xxx.yml
xxx.pem
_ BE Configuration• http://rndwww.nce.amadeus.n
et/wiki/wikidoc/index.php/BE_Configuration_(MongoDB)
_ Mongos Configuration• http://rndwww.nce.amadeus.n
et/wiki/wikidoc/index.php/Mongos_Configuration
_ MongoDB Configuration• Handled entirely by DZB
© 2
014
Amad
eus I
T Gr
oup
SA
Confi
dent
ial
RESTRICTED
MongoDB concepts3
Confi
dent
ial
RESTRICTED
Page 78
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Replica Set or ShardMongoDB
Confi
dent
ial
RESTRICTED
Page 79
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Replica Set or Shard• A replica set is a group of mongod
instances that host the same data set
MongoDB
Replica Set
Confi
dent
ial
RESTRICTED
Page 80
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Replica Set or Shard• A replica set is a group of mongod
instances that host the same data set
MongoDB
Replica Set
Confi
dent
ial
RESTRICTED
Page 81
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Shard Key
• Determines the distribution of the documents among the cluster’s shards
• The shard key is an indexed field that exists in every document
MongoDB
Confi
dent
ial
RESTRICTED
Page 82
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Shard Map• MongoDB partitions data in
“chunks”, chunks are associated to a replica set
MongoDB
Shard Map
Shard 1 Shard 2 Shard 3
Confi
dent
ial
RESTRICTED
Page 83
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Shard Map• MongoDB partitions data in
“chunks”, chunks are associated to a replica set• When a chunk grows over the
chunk size (64 MB), MongoDB splits the chunk in smaller chunks. The (arbitrary) limit has been chosen to avoid moving too much data during a balancing.
MongoDB
Shard Map
Shard 1 Shard 2 Shard 3
Confi
dent
ial
RESTRICTED
Page 84
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Shard Map• MongoDB partitions data in
“chunks”, chunks are associated to a replica set• When a chunk grows over the
chunk size (64 MB), MongoDB splits the chunk in smaller chunks. The (arbitrary) limit has been chosen to avoid moving too much data during a balancing.• The config server provides a
shard map with at least one mongod process per shard
MongoDB
Shard Map
Shard 1 Shard 2 Shard 3
Confi
dent
ial
RESTRICTED
Page 85
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ MongoDB functional architectureMongoDB
Confi
dent
ial
RESTRICTED
Page 86
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ MongoDB functional architecture• A MongoDB cluster contains several databases
MongoDB
Confi
dent
ial
RESTRICTED
Page 87
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ MongoDB functional architecture• A MongoDB cluster contains several databases• Each database contains several collections (equivalent of Oracle schemas)
MongoDB
Confi
dent
ial
RESTRICTED
Page 88
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ MongoDB functional architecture• A MongoDB cluster contains several databases• Each database contains several collections (equivalent of Oracle schemas)• A collection contains a set of JSON documents (equivalent of Oracle tables)
MongoDB
Confi
dent
ial
RESTRICTED
Page 89
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ MongoDB functional architecture• A MongoDB cluster contains several databases• Each database contains several collections (equivalent of Oracle schemas)• A collection contains a set of JSON documents (equivalent of Oracle tables)• Documents are distributed across all the shards
• Each sharded collection need a value to shard on (=shard key)
MongoDB
Confi
dent
ial
RESTRICTED
Page 90
© 2
014
Amad
eus I
T Gr
oup
SA
_ Example
MongoDB ConceptsMongoDB {
“name”: “Winky”}
Confi
dent
ial
RESTRICTED
Page 91
© 2
014
Amad
eus I
T Gr
oup
SA
_ Example
MongoDB ConceptsMongoDB {
“name”: “Winky”}
mongos
Confi
dent
ial
RESTRICTED
Page 92
© 2
014
Amad
eus I
T Gr
oup
SA
_ Example
MongoDB ConceptsMongoDB {
“name”: “Winky”}
mongos
SHARDSHARD SHARD SHARD
Confi
dent
ial
RESTRICTED
Page 93
© 2
014
Amad
eus I
T Gr
oup
SA
_ Example
MongoDB ConceptsMongoDB {
“name”: “Winky”}
mongos
SHARDSHARD SHARD SHARD
INSERT in my_collection
Confi
dent
ial
RESTRICTED
Page 94
© 2
014
Amad
eus I
T Gr
oup
SA
_ Example
MongoDB ConceptsMongoDB {
“name”: “Winky”}
mongos
INSERT in my_collection
SHARDSHARD SHARD SHARD
Chunk = HASH(doc.name)
Confi
dent
ial
RESTRICTED
Page 95
© 2
014
Amad
eus I
T Gr
oup
SA
_ Example
MongoDB ConceptsMongoDB {
“name”: “Winky”}
mongos
INSERT in my_collection
SHARDSHARD SHARD SHARD
Chunk = HASH(doc.name)
Confi
dent
ial
RESTRICTED
Page 96
© 2
014
Amad
eus I
T Gr
oup
SA
_ Example
MongoDB ConceptsMongoDB {
“name”: “Winky”}
mongos
INSERT in my_collection
SHARDSHARD SHARD SHARD
Chunk = HASH(doc.name)
Confi
dent
ial
RESTRICTED
Page 97
© 2
014
Amad
eus I
T Gr
oup
SA
_ Example
MongoDB ConceptsMongoDB {
“name”: “Winky”}
mongos
INSERT in my_collection
SHARDSHARD SHARD SHARD
Chunk = HASH(doc.name)
Confi
dent
ial
RESTRICTED
Page 98
© 2
014
Amad
eus I
T Gr
oup
SA
_ Example
MongoDB ConceptsMongoDB {
“name”: “Winky”}
mongos
INSERT in my_collection
SHARDSHARD SHARD SHARD
_ Remember, a shard is a replica set, it is spread over different machinesChunk = HASH(doc.name)
Confi
dent
ial
RESTRICTED
Page 99
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Data ReplicationMongoDB
Confi
dent
ial
RESTRICTED
Page 100
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Data Replication• A write goes first to the primary of a shard
MongoDB
NODE 1
Mongod(primary)
Replica Set/Shard
write
NODE 2
Mongod(secondary)
Confi
dent
ial
RESTRICTED
Page 101
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Data Replication• A write goes first to the primary of a shard• The primary writes the operation in a special collection called oplog
MongoDB
NODE 1
Mongod(primary)
Replica Set/Shard
oplogwrite
NODE 2
Mongod(secondary)
Confi
dent
ial
RESTRICTED
Page 102
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Data Replication• A write goes first to the primary of a shard• The primary writes the operation in a special collection called oplog• Secondaries periodically poll the oplog to get the latest updates
MongoDB
NODE 1
Mongod(primary)
Replica Set/Shard
oplogwrite
NODE 2
Mongod(secondary)
Confi
dent
ial
RESTRICTED
Page 103
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Data Replication• Oplog window• Time difference between the first and the last entry in the oplog collection• Replication Lag• Time between the latest entry on the primary and the latest entry on a secondary• Replication headroom• Difference between Oplog Window and Replication Lag
MongoDB
Confi
dent
ial
RESTRICTED
Page 104
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Journaling• When running with journaling, MongoDB stores and applies write
operations in memory and in the on-disk journal before the changes are present in the data files on disk• MongoDB journals operations by batch (called group commits), the size is
configurable• A group commit must block all writers during the commit• In case of a crash, mongod uses the journal to warm-up
MongoDB
Confi
dent
ial
RESTRICTED
Page 105
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Chunk splitting• Each mongos knows how much it writes to each chunk• When a mongos thinks a chunk needs to be split (=reached the max chunk
size), it queries the related mongod and check the chunk size. If needed, it will start the balancer (if enabled)
MongoDB
Confi
dent
ial
RESTRICTED
Page 106
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?MongoDB
Confi
dent
ial
RESTRICTED
Page 107
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks
MongoDB
Confi
dent
ial
RESTRICTED
Page 108
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
Confi
dent
ial
RESTRICTED
Page 109
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2
Confi
dent
ial
RESTRICTED
Page 110
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2
“I want to move this chunk”
Confi
dent
ial
RESTRICTED
Page 111
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2
Global lock, no other chunk can be moved
Confi
dent
ial
RESTRICTED
Page 112
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2
“Please move chunk 4 to shard 2”
Confi
dent
ial
RESTRICTED
Page 113
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2
Please poll this list of documents {..}
Confi
dent
ial
RESTRICTED
Page 114
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2
Polling..
Confi
dent
ial
RESTRICTED
Page 115
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2
Block read/writes
Confi
dent
ial
RESTRICTED
Page 116
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2
Block read/writes
Update chunk map
Confi
dent
ial
RESTRICTED
Page 117
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2Connects with outdated map
Confi
dent
ial
RESTRICTED
Page 118
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2
Get new map2
Confi
dent
ial
RESTRICTED
Page 119
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added
MongoDB
mongos
SHARD2
SHARD1config
mongos1 1 1 1 2 21 1 1 1 2 2 2
Confi
dent
ial
RESTRICTED
Page 120
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Scalability• If we need more nodes, we add more shards (done through one of the
mongos running on NoSQL tier)• When you add a shard to a sharded cluster, you affect the balance of
chunks among the shards. The balancer will begin migrating chunks so that the cluster will achieve balance• The balancer has an impact on performance• It moves one chunk at a time
MongoDB
Confi
dent
ial
RESTRICTED
Page 121
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic RecoveryMongoDB
Primary
Secondary Arbiter
Replicationmongos
Confi
dent
ial
RESTRICTED
Page 122
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod
MongoDB
Primary
Secondary Arbiter
Heartbeat
Replicationmongos
Confi
dent
ial
RESTRICTED
Page 123
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
Replication
PrimarySecondary
mongos
Confi
dent
ial
RESTRICTED
Page 124
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
Replication
PrimarySecondary
mongos
Confi
dent
ial
RESTRICTED
Page 125
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
PrimarySecondary
mongos
Write outage (read depends on configuration)
Confi
dent
ial
RESTRICTED
Page 126
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
PrimarySecondary
mongos
Write outage (read depends on configuration)
Election
Confi
dent
ial
RESTRICTED
Page 127
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
PrimarySecondary
mongos
Write outage (read depends on configuration)
Primary
Confi
dent
ial
RESTRICTED
Page 128
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
PrimarySecondary
mongos
isMas
ter() Primary
Confi
dent
ial
RESTRICTED
Page 129
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
Primary
Recoverymongos
Confi
dent
ial
RESTRICTED
Page 130
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
Primary
Recovery
Secondary
mongos 1. Read from journal
2. Pull oplog
Primary
Confi
dent
ial
RESTRICTED
Page 131
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
Primary
Recovery
Secondary
mongos
Primary
Secondary
Replication
Confi
dent
ial
RESTRICTED
Page 132
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
Replication
PrimarySecondary
mongos Sends its priority in the heartbeat. In case it is the highest…
Secondary
Primary
Confi
dent
ial
RESTRICTED
Page 133
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
Replication
Primary
Recovery
Primary
mongos
…self stepdown
Secondary
Confi
dent
ial
RESTRICTED
Page 134
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
Replication
Primary
RecoveryPrimary
Secondary
mongos
Confi
dent
ial
RESTRICTED
Page 135
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Automatic Recovery• Mongod-mongod
heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod
heartbeat checks the role of the processes
MongoDB
Primary
Secondary Arbiter
Heartbeat
Replication
Primary
RecoveryPrimary
Secondary
mongos
isMaster()
Confi
dent
ial
RESTRICTED
Page 136
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Rollback files• A rollback reverts write operations on a former primary when the member
rejoins its replica set after a failover. A rollback is necessary only if the primary had accepted write operations that the secondaries had not successfully replicated before the primary stepped down• When a rollback does occur, administrators must decide whether to apply
or ignore the rollback data. • Note: Rollback files can be avoided if the application uses replica
acknowledged write concern (=write successful only when written on replica)
MongoDB
Confi
dent
ial
RESTRICTED
Page 137
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Full resyncMongoDB
Primary
Secondary Arbiter
Heartbeat
Primary
Recovery
2. Pull oplogmongos 1. Read from journal
Confi
dent
ial
RESTRICTED
Page 138
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Full resync• In case the down time
is higher than the oplog window, a full resync is needed
MongoDB
Primary
Secondary Arbiter
Heartbeat
Primary
Recovery
2. Pull oplogmongos 1. Read from journal
Confi
dent
ial
RESTRICTED
Page 139
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Full resync• In case the down time
is higher than the oplog window, a full resync is needed • 1. Stop process
MongoDB
Primary
Secondary Arbiter
Heartbeat
Primary
Recovery
2. Pull oplogmongos 1. Read from journal
Confi
dent
ial
RESTRICTED
Page 140
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Full resync• In case the down time
is higher than the oplog window, a full resync is needed • 1. Stop process • 2. wipe-out data
MongoDB
Primary
Secondary Arbiter
Heartbeat
Primary
Recovery
2. Pull oplogmongos 1. Read from journal
Confi
dent
ial
RESTRICTED
Page 141
© 2
014
Amad
eus I
T Gr
oup
SA
MongoDB Concepts
_ Full resync• In case the down time
is higher than the oplog window, a full resync is needed • 1. Stop process • 2. wipe-out data • 3. start process
MongoDB
Primary
Secondary Arbiter
Heartbeat
Primary
Recovery
2. Pull oplogmongos 1. Read from journal
Confi
dent
ial
RESTRICTED
Page 142
© 2
014
Amad
eus I
T Gr
oup
SA
Training MongoDB
_ MongoDB processes• mongod (primary/secondary/arbiter), mongos, conf, agent, MMS
_ Replica Set / Shard, Shard Key, Shard Map_ Database, Collection, Document_ Replication (oplog)_ Journaling_ Capture/Restore_ Chunk Move_ Automatic Recovery / Election
Recap – What you should know by now
© 2
014
Amad
eus I
T Gr
oup
SA
Confi
dent
ial
RESTRICTED
Workflows4
Confi
dent
ial
RESTRICTED
Page 144
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – discover topologyMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
_ Mongos startsAgent
BE BE BEBE
mongos
Confi
dent
ial
RESTRICTED
Page 145
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – discover topologyMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
_ Mongos starts_ Reads Mongos.yml file
and connects to the config server
Agent
BE BE BEBE
mongos
Confi
dent
ial
RESTRICTED
Page 146
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – discover topologyMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
_ Mongos starts_ Reads Mongos.yml file
and connects to the config server
_ Retrieves the shard map
Agent
BE BE BEBE
mongos
Confi
dent
ial
RESTRICTED
Page 147
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – discover topologyMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
_ Mongos starts_ Reads Mongos.yml file
and connects to the config server
_ Retrieves the shard map_ Connects to all the
mongods
Agent
BE BE BEBE
mongos
Confi
dent
ial
RESTRICTED
Page 148
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – discover topologyMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
_ Mongos starts_ Reads Mongos.yml file
and connects to the config server
_ Retrieves the shard map_ Connects to all the
mongods_ Retrieves the complete
list of processes for each shard from the mongods
Agent
BE BE BEBE
mongos
Confi
dent
ial
RESTRICTED
Page 149
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – discover topologyMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
_ Mongos starts_ Reads Mongos.yml file
and connects to the config server
_ Retrieves the shard map_ Connects to all the
mongods_ Retrieves the complete
list of processes for each shard from the mongods
_ Retrieves the role of each process is the command isMaster()
Agent
BE BE BEBE
mongos
Confi
dent
ial
RESTRICTED
Page 150
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows - writeMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
_ The BE wants to store a document
{..}
Confi
dent
ial
RESTRICTED
Page 151
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows - writeMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
_ The BE wants to store a document
_ The document is routed according to the shard key{..}
Confi
dent
ial
RESTRICTED
Page 152
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows - writeMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
_ The BE wants to store a document
_ The document is routed according to the shard key
_ Written only on primaryprimary
{..}
Confi
dent
ial
RESTRICTED
Page 153
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows - writeMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
_ The BE wants to store a document
_ The document is routed according to the shard key
_ Written only on primary_ And replicated to the
secondary asynchronouslyprimary secondary{..}{..}
Confi
dent
ial
RESTRICTED
Page 154
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows - writeMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
_ The BE wants to store a document
_ The document is routed according to the shard key
_ Written only on primary_ And replicated to the
secondary asynchronously_ The client can decide to
wait for a successful write in memory, on disk or on the secondaries
primary secondary{..}{..}
Confi
dent
ial
RESTRICTED
Page 155
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – readMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
_ The BE wants to read documents
{..}
Confi
dent
ial
RESTRICTED
Page 156
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – readMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
_ The BE wants to read documents
_ The query is routed according to the shard key{..}
primary
Confi
dent
ial
RESTRICTED
Page 157
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – readMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
_ The BE wants to read documents
_ The query is routed according to the shard key
_ Depending on the query, several mongods can be targeted primary primary
{..}{..}
Confi
dent
ial
RESTRICTED
Page 158
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – readMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
_ The BE wants to read documents
_ The query is routed according to the shard key
_ Depending on the query, several mongods can be targeted
_ Depending on the configuration, mongos could read on a primary or a secondary
primary primary secondary{..}{..} {..}
Confi
dent
ial
RESTRICTED
Page 159
© 2
014
Amad
eus I
T Gr
oup
SA
Workflows – readMongoDB
AS OBE(GOS, App Support)
AppTier
NoSQLTier MongoDB
Agent
BE BE BEBE
mongos
_ The BE wants to read documents
_ The query is routed according to the shard key
_ Depending on the query, several mongods can be targeted
_ Depending on the configuration, mongos could read on a primary or a secondary
_ Mongos will process the documents returned and reply to the BE
primary primary secondary
{..}
© 2
014
Amad
eus I
T Gr
oup
SA
Confi
dent
ial
RESTRICTED
Operability & Monitoring5
Confi
dent
ial
RESTRICTED
Page 161
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Let’s check the new stuffs in the life cycle of a software in ADP
• Deployment• ACO• Logs• Monitoring• Automatic Alerts• Operations
MongoDB
Confi
dent
ial
RESTRICTED
Page 162
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ New Pega flow defined for MongoDB clusters
_ Uses Puppet for installation automation
_ http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Pega_flow_Big_Data
Deployment
Confi
dent
ial
RESTRICTED
Page 163
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ ULAM: mongos is deployed as a component of the OBE using it. _ Later, MongoDB upgrades will be done by ULAM too (not initial
installation)
Deployment
Confi
dent
ial
RESTRICTED
Page 164
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Called by ULAM during a load_ http://appcheckout.os.amadeus.net:8080/appcheckout/_ New check to be created for Mongos
Application Checkout (ACO)
Confi
dent
ial
RESTRICTED
Page 165
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Current design (might evolve)• Mongos sends its logs a binary using Tracer, this binary sends all the logs
(Error + Info) to the application logserver in “mongos_link”
Logs
Confi
dent
ial
RESTRICTED
Page 166
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ New section in InfrastructureSentinel
Confi
dent
ial
RESTRICTED
Page 167
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ http://sentinel.muc.amadeus.net/index.php?page=mongodbMonitoring_ http://sentinel.muc.amadeus.net/index.php?page=mongosMonitoring
Sentinel
Confi
dent
ial
RESTRICTED
Page 168
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Uniping• New pinger for mongo (check the status of mongod, mongos and config
processes on database side)• http://uniping.muc.amadeus.net/watchdog/
_ Scheck• Nothing new, it checks nodes via ssh
_ Topx• Nothing new, it monitors memory
_ YALB • Monitors logs and raises specialized IRs for mongos on application side
MongoDB
Confi
dent
ial
RESTRICTED
Page 169
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Trends• Tracks IRs for Ops• http://gmstst.muc.amadeus.net/trends
MongoDB
Confi
dent
ial
RESTRICTED
Page 170
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ New links to MongoDB OSAGs/OPROCsMongoDB
Confi
dent
ial
RESTRICTED
Page 171
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ OSAG• Operators and System Administrators Guide• It contains OPROCs, POCs, escalation procedures, Impact & Recovery of
configuration changes, Setup procedures and much more!• http
://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Mongos_and_MongoDB_OSAG
MongoDB
Confi
dent
ial
RESTRICTED
Page 172
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ For GOS• Escalation Procedure• http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Escalation_Proce
dure_for_MongoDB
• The link will be in each IR• We don’t expect any action from GOS on MongoDB side, recovery
procedures are fully automated. The actions available in Workbench can be used for test purposes
MongoDB
Confi
dent
ial
RESTRICTED
Page 173
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Workbench• http://houston.muc.amadeus.net/index.php?page=workbench/launch
MongoDB
Confi
dent
ial
RESTRICTED
Page 174
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & MonitoringMongoDB
Confi
dent
ial
RESTRICTED
Page 175
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & MonitoringMongoDB
_ NoSQL tab
Confi
dent
ial
RESTRICTED
Page 176
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & MonitoringMongoDB
_ NoSQL tab_ Cluster status
Confi
dent
ial
RESTRICTED
Page 177
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & MonitoringMongoDB
_ NoSQL tab_ Cluster status_ Node status
Confi
dent
ial
RESTRICTED
Page 178
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & MonitoringMongoDB
_ NoSQL tab_ Cluster status_ Node status_ Process Status
Confi
dent
ial
RESTRICTED
Page 179
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & MonitoringMongoDB
_ NoSQL tab_ Cluster status_ Node status_ Process Status_ Shard
distribution
Confi
dent
ial
RESTRICTED
Page 180
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Start/stop of a Secondary: Right click on the mongod process
_ Stepdown a primary: Right click on the mongod process
MongoDB
Confi
dent
ial
RESTRICTED
Page 181
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Shard distributionMongoDB
Confi
dent
ial
RESTRICTED
Page 182
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Shard distributionMongoDB
Number of primaries
Confi
dent
ial
RESTRICTED
Page 183
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Shard distributionMongoDB
Number of primaries
Primary
Confi
dent
ial
RESTRICTED
Page 184
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Shard distributionMongoDB
Number of primaries
Primary Secondary
Confi
dent
ial
RESTRICTED
Page 185
© 2
014
Amad
eus I
T Gr
oup
SA
Operability & Monitoring
_ Shard distribution
_ Highlight of the node hosting secondary processes for the selected node
MongoDB
Number of primaries
Primary Secondary
© 2
014
Amad
eus I
T Gr
oup
SA
Confi
dent
ial
RESTRICTED
You can follow us on:AmadeusITGroup
amadeus.com/blogamadeus.com
Thank you