Retail Reference Architecture Part 2: Real-Time, Geo Distributed Inventory

Senior Consulting Engineer, MongoDB

Norman Graham

#MongoDBWorld

Retail Reference Architecture: Real-Time, Geo-Distributed Inventory

Inventory

Inventory

MongoDB

External Inventory

Internal Inventory

Regional Inventory

Purchase Orders

Fulfillment

Promotions

Inventory – Traditional Architecture

Relational DBSystem of Records

NightlyBatches

Analytics, Aggregations

,Reports

Caching Layer

Field Inventory

Internal & External

Apps

Point-in-time Loads

Inventory – Opportunities Missed

• Can’t reliably detect availability

• Can't redirect purchasers to in-store pickup

• Can’t do intra-day replenishment

• Degraded customer experience

• Higher internal expense

Inventory – Principles

• Single view of the inventory

• Used by most services and channels

• Read-dominated workload

• Local, real-time writes

• Bulk writes for refresh

• Geographically distributed

• Horizontally scalable

Requirement Challenge MongoDB

Single view of inventory

Ensure availability of inventory

information on all channels and

services

Developer-friendly, document-oriented

storage

High volume, low latency reads

Anytime, anywhere access to inventory

data without overloading the system of record

Fast, indexed readsLocal reads

Horizontal scaling

Bulk updates,intra-day deltas

Provide window-in-time consistency for

highly available services

Bulk writesFast, in-place

updatesHorizontal scaling

Rapid application development

cycles

Deliver new services rapidly to capture new opportunities

Flexible schemaRich query language

Agile-friendly iterations

Inventory – Requirements

Inventory – Target Architecture

Relational DBSystem of Records

Analytics, Aggregations

,Reports

Field Inventory

Internal & External

Apps

Inventory

Assortments

Shipments

Audits

Products

Stores

Point-in-time Loads

NightlyRefres

h

Real-timeUpdates

Horizontal Scaling

Inventory – Technical Decisions

Store

Inventory

Schema

Indexing

Inventory – Collections

Stores InventoryProducts

Audits AssortmentsShipments

> db.stores.findOne(){ "_id" : ObjectId("53549fd3e4b0aaf5d6d07f35"), "className" : "catalog.Store", "storeId" : "store0", "name" : "Bessemer store", "address" : { "addr1" : "1st Main St", "city" : "Bessemer", "state" : "AL", "zip" : "12345", "country" : "US" }, "location" : [ -86.95444, 33.40178 ], ...}

Stores – Sample Document

Stores – Sample Queries

• Get a store by storeId

db.stores.find({ "storeId" : "store0" })

• Get a store by zip code

db.stores.find({ "address.zip" : "12345" })

What’s near me?

Stores – Sample Geo Queries

• Get nearby stores sorted by distance

db.runCommand({

geoNear : "stores",

near : {

type : "Point",

coordinates : [-82.8006, 40.0908] },

maxDistance : 10000.0,

spherical : true

})

Stores – Sample Geo Queries

• Get the five nearest stores within 10 km

db.stores.find({

location : {

$near : {

$geometry : {

type : "Point",

coordinates : [-82.80, 40.09] },

$maxDistance : 10000.0 } }

}).limit(5)

Stores – Indices

• { "storeId" : 1 }, { "unique" : true }

• { "name" : 1 }

• { "address.zip" : 1 }

• { "location" : "2dsphere" }

> db.inventory.findOne(){ "_id": "5354869f300487d20b2b011d", "storeId": "store0", "location": [-86.95444, 33.40178], "productId": "p0", "vars": [ { "sku": "sku1", "q": 14 }, { "sku": "sku3", "q": 7 }, { "sku": "sku7", "q": 32 }, { "sku": "sku14", "q": 65 }, ... ]}

Inventory – Sample Document

Inventory – Sample Queries

• Get all items in a store

db.inventory.find({ storeId : "store100" })

• Get quantity for an item at a store

db.inventory.find({

"storeId" : "store100",

"productId" : "p200"

})

Inventory – Sample Queries

• Get quantity for a sku at a store

db.inventory.find(

{

"storeId" : "store100",

"productId" : "p200",

"vars.sku" : "sku11736"

},

{ "vars.$" : 1 }

)

Inventory – Sample Update

• Increment / decrement inventory for an item at a store

db.inventory.update(

{

"storeId" : "store100",

"productId" : "p200",

"vars.sku" : "sku11736"

},

{ "$inc" : { "vars.$.q" : 20 } }

)

Inventory – Sample Aggregations

• Aggregate total quantity for a product

db.inventory.aggregate( [

{ $match : { productId : "p200" } },

{ $unwind : "$vars" },

{ $group : {

_id : "result",

count : { $sum : "$vars.q" } } } ] )

{ "_id" : "result", "count" : 101752 }

Inventory – Sample Aggregations

• Aggregate total quantity for a store

db.inventory.aggregate( [

{ $match : { storeId : "store100" } },

{ $unwind : "$vars" },

{ $match : { "vars.q" : { $gt : 0 } } },

{ $group : {

_id : "result",

count : { $sum : 1 } } } ] )

{ "_id" : "result", "count" : 29347 }

Inventory – Sample Aggregations

• Aggregate total quantity for a store

db.inventory.aggregate( [

{ $match : { storeId : "store100" } },

{ $unwind : "$vars" },

{ $group : {

_id : "result",

count : { $sum : "$vars.q" } } } ] )

{ "_id" : "result", "count" : 29347 }

Inventory – Sample Geo-Query

• Get inventory for an item near a point

db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.8006, 40.0908] }, maxDistance : 10000.0, spherical : true, limit : 10, query : { "productId" : "p200", "vars.sku" : "sku11736" } } )

Inventory – Sample Geo-Query

• Get closest store with available sku

db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.800672, 40.090844] }, maxDistance : 10000.0, spherical : true, limit : 1, query : { productId : "p200", vars : { $elemMatch : { sku : "sku11736", q : { $gt : 0 } } } } } )

Inventory – Sample Geo-Aggregation

• Get count of inventory for an item near a point

db.inventory.aggregate( [ { $geoNear: { near : { type : "Point", coordinates : [-82.800672, 40.090844] }, distanceField: "distance", maxDistance: 10000.0, spherical : true, query: { productId : "p200", vars : { $elemMatch : { sku : "sku11736", q : {$gt : 0} } } }, includeLocs: "dist.location", num: 5 } }, { $unwind: "$vars" }, { $match: { "vars.sku" : "sku11736" } }, { $group: { _id: "result", count: {$sum: "$vars.q"} } }])

Inventory – Sample Indices

• { storeId : 1 }

• { productId : 1, storeId : 1 }

• { productId : 1, location : "2dsphere" }

• Why not "vars.sku"?– { productId : 1, storeId : 1, "vars.sku" : 1 }

Horizontal Scaling

Inventory – Technical Decisions

Store

Inventory

Schema

Indexing

ShardEast

ShardCentr

al

ShardWest

East DC

Inventory – Sharding TopologyWest DC Central DC

LegacyInventor

y

Primary

Primary

Primary

Inventory – Shard Key

• Choose shard key– { storeId : 1 } ?– { productId : 1, storeId : 1 } ?– { storeId : 1, productId : 1 } ?

• Set up sharding– sh.enableSharding("inventoryDB")– sh.shardCollection( "inventoryDB.inventory", { storeId : 1, productId : 1 } )

Inventory – Shard Tags

• Set up shard tags– sh.addShardTag("shard0000", "west")– sh.addShardTag("shard0001", "central")– sh.addShardTag("shard0002", "east")

• Set up tag ranges– sh.addTagRange("inventoryDB.inventory", { storeId : 0 }, { storeId : 100}, "west" )– sh.addTagRange("inventoryDB.inventory", { storeId : 100 }, { storeId : 200 }, "central" )– sh.addTagRange("inventoryDB.inventory", { storeId : 200 }, { storeId : 300 }, "east" )

Senior Consulting Engineer, MongoDB

Norman Graham

#MongoDBWorld

Thank You