Replication

InternalsFitting Everything Together

2.8, Refactored

● Architecture as of 2.8

● Unit testable; more, and faster, cpp tests

● Many changes (heartbeats, locking, future)

● Interop with 2.6

● Larger replica sets

Large Blocks

● Topology Manager (state machine)

● Replication Coordinator (repl facade)

● Applier (replicate/apply oplog)

● Executor (network, heartbeats, serialization)

● Commands (re-config, init, status, etc)

● External (writes, storage, query, commands)

Blocks

Applier

Topology Manager

Replication

Coordinator

Oplog

CFG

CM

Ds

Write

s

Query

Executo

r

Blocks

Applier

Topology Manager

Replication

Coordinator

Oplog

CFG

CM

Ds

Write

s

Query

Executo

r

Topology

● Maintains Authoritative Stateo Heartbeat, ping, member state

o Roles and transitions

● Contains Decision Logic

● Unit Testable

● Serial AccessTopology Manager

CFG

Examples

● updateConfig

● prepare*Response for commands

● getSyncSource, *

● setFollowerMode (state)

● processHeartbeat

● prepareHeartbeatResponse

PrepareHeartbeatResponseStatus TopologyCoordinatorImpl::prepareHeartbeatResponse(...) {

// Check error conditions, then set response fields …

response->setElectable(!_getMyUnelectableReason(...));

response->setHbMsg(_getHbmsg(...));

response->setTime(...);

response->setOpTime(lastOpApplied);

if (!_syncSource) {

response->setSyncingTo(_syncSource); }

… topology_coordinator_impl.cpp:628

Failover Scenario

Heart

beats P

S

S

Health Check (rsHB)Active Primary

Failover Scenario

Heart

beats P

S

S

Active PrimaryP

Failed Primary

Failover Scenario

Heart

beats Failed

P

S

Health Check (rsHB)

Blocks

Applier

Topology Manager

Replication

Coordinator

Oplog

CFG

CM

Ds

Write

s

Query

Executo

r

Replications Coordinator

● Interface to other subsystems

● Uses executor to scheduleo Commands

o Elections, Initiate, Reconfig

o Role/State Changes

● Unit Testableo With help, requires mocking out bridge for

subsystems

Replication

Coordinator

Blocks

ApplierReplication

Coordinator

OplogC

MD

s

Write

s

Query

Executo

r

Topology ManagerCFG

Examples

● process*Response for commands

● awaitReplication* (for writes or migration)

● isReplEnabled

● canAcceptWrites*

Accepting writesstatic bool checkIsMasterForDatabase(const std::string& db, ...) {

if (!getReplicationCoordinator()->canAcceptWritesForDatabase(db)){

errorDetail->setErrCode(ErrorCodes::NotMaster);

errorDetail->setErrMessage("Not primary while writing to " + ns);

return false;

}

return true;

}

Blocks

Applier

Topology Manager

Replication

Coordinator

Oplog

CFG

CM

Ds

Write

s

Query

Executo

r

Applier

● Reads from *upstream* oplog

● Applier operations transformations

● Mostly unchanged since 2.4

● Includes UpdatePosition commands

Applier

Read + Apply Decoupled

● Background oplog reader thread (net)

● Pool of oplog applier threads (by collection)

Repl Source

Applier

Pool

Buffer

DB4

DB3

DB1 DB2

Local Oplog

Network

Replication Operations

oplog entry (fields):

o = update, o2 = query

{ "ns" : "test.tags",

"op" : "u", "v" : 2, "ts": ...,

"o2" : { "_id" : 1 },

"o" : { "$set" : { "tags.4" : "e" } } }

Blocks

Applier

Topology Manager

Replication

Coordinator

Oplog

CFG

CM

Ds

Write

s

Query

Executo

r

Executor

● Serializes access to Topology state

● Serializes global state changes wrt db writes

● Processes network requests in IO pool

● Supports event/signal notification

Write Request

● Sent by user

● Interpreted by command subsystem

● Checked by replication coordinator

● Executed

● Idempotent entry recorded in oplog

● ~ Replicated

● ~ Possibly verified during user write request

Write Request

ApplierReplication

Coordinator

OplogC

MD

s

Write

s

Query

Executo

r

Topology ManagerCFG

● Topology Manager (state machine)

● Replication Coordinator (repl facade)

● Applier (replicate/apply oplog)

● Executor (network, heartbeats, serialization)

● Commands (re-config, init, status, etc)

● External (writes, storage, query, commands)

Thanks

Questions?