Embed Size (px)
DESCRIPTION
During this session we will cover the best practices for implementing a real-time inventory with MongoDB. This includes properly model quantities and stores to avoid large numbers of documents being indexed, how to efficiently use geo-indexing to find the closest store with a specific item available and how to run aggregation to gather interesting inventory stats. We will also cover operational considerations, like how to make inventory queries and updates from anywhere be low-latency and resilient to network partitions via tag-aware sharding.
Citation preview
Senior Consulting Engineer, MongoDB
Norman Graham
#MongoDBWorld
Retail Reference Architecture: Real-Time, Geo-Distributed Inventory
Inventory
Inventory
MongoDB
External Inventory
Internal Inventory
Regional Inventory
Purchase Orders
Fulfillment
Promotions
Inventory – Traditional Architecture
Relational DBSystem of Records
NightlyBatches
Analytics, Aggregations
,Reports
Caching Layer
Field Inventory
Internal & External
Apps
Point-in-time Loads
Inventory – Opportunities Missed
• Can’t reliably detect availability
• Can't redirect purchasers to in-store pickup
• Can’t do intra-day replenishment
• Degraded customer experience
• Higher internal expense
Inventory – Principles
• Single view of the inventory
• Used by most services and channels
• Read-dominated workload
• Local, real-time writes
• Bulk writes for refresh
• Geographically distributed
• Horizontally scalable
Requirement Challenge MongoDB
Single view of inventory
Ensure availability of inventory
information on all channels and
services
Developer-friendly, document-oriented
storage
High volume, low latency reads
Anytime, anywhere access to inventory
data without overloading the system of record
Fast, indexed readsLocal reads
Horizontal scaling
Bulk updates,intra-day deltas
Provide window-in-time consistency for
highly available services
Bulk writesFast, in-place
updatesHorizontal scaling
Rapid application development
cycles
Deliver new services rapidly to capture new opportunities
Flexible schemaRich query language
Agile-friendly iterations
Inventory – Requirements
Inventory – Target Architecture
Relational DBSystem of Records
Analytics, Aggregations
,Reports
Field Inventory
Internal & External
Apps
Inventory
Assortments
Shipments
Audits
Products
Stores
Point-in-time Loads
NightlyRefres
h
Real-timeUpdates
Horizontal Scaling
Inventory – Technical Decisions
Store
Inventory
Schema
Indexing
Inventory – Collections
Stores InventoryProducts
Audits AssortmentsShipments
> db.stores.findOne(){ "_id" : ObjectId("53549fd3e4b0aaf5d6d07f35"), "className" : "catalog.Store", "storeId" : "store0", "name" : "Bessemer store", "address" : { "addr1" : "1st Main St", "city" : "Bessemer", "state" : "AL", "zip" : "12345", "country" : "US" }, "location" : [ -86.95444, 33.40178 ], ...}
Stores – Sample Document
Stores – Sample Queries
• Get a store by storeId
db.stores.find({ "storeId" : "store0" })
• Get a store by zip code
db.stores.find({ "address.zip" : "12345" })
What’s near me?
Stores – Sample Geo Queries
• Get nearby stores sorted by distance
db.runCommand({
geoNear : "stores",
near : {
type : "Point",
coordinates : [-82.8006, 40.0908] },
maxDistance : 10000.0,
spherical : true
})
Stores – Sample Geo Queries
• Get the five nearest stores within 10 km
db.stores.find({
location : {
$near : {
$geometry : {
type : "Point",
coordinates : [-82.80, 40.09] },
$maxDistance : 10000.0 } }
}).limit(5)
Stores – Indices
• { "storeId" : 1 }, { "unique" : true }
• { "name" : 1 }
• { "address.zip" : 1 }
• { "location" : "2dsphere" }
> db.inventory.findOne(){ "_id": "5354869f300487d20b2b011d", "storeId": "store0", "location": [-86.95444, 33.40178], "productId": "p0", "vars": [ { "sku": "sku1", "q": 14 }, { "sku": "sku3", "q": 7 }, { "sku": "sku7", "q": 32 }, { "sku": "sku14", "q": 65 }, ... ]}
Inventory – Sample Document
Inventory – Sample Queries
• Get all items in a store
db.inventory.find({ storeId : "store100" })
• Get quantity for an item at a store
db.inventory.find({
"storeId" : "store100",
"productId" : "p200"
})
Inventory – Sample Queries
• Get quantity for a sku at a store
db.inventory.find(
{
"storeId" : "store100",
"productId" : "p200",
"vars.sku" : "sku11736"
},
{ "vars.$" : 1 }
)
Inventory – Sample Update
• Increment / decrement inventory for an item at a store
db.inventory.update(
{
"storeId" : "store100",
"productId" : "p200",
"vars.sku" : "sku11736"
},
{ "$inc" : { "vars.$.q" : 20 } }
)
Inventory – Sample Aggregations
• Aggregate total quantity for a product
db.inventory.aggregate( [
{ $match : { productId : "p200" } },
{ $unwind : "$vars" },
{ $group : {
_id : "result",
count : { $sum : "$vars.q" } } } ] )
{ "_id" : "result", "count" : 101752 }
Inventory – Sample Aggregations
• Aggregate total quantity for a store
db.inventory.aggregate( [
{ $match : { storeId : "store100" } },
{ $unwind : "$vars" },
{ $match : { "vars.q" : { $gt : 0 } } },
{ $group : {
_id : "result",
count : { $sum : 1 } } } ] )
{ "_id" : "result", "count" : 29347 }
Inventory – Sample Aggregations
• Aggregate total quantity for a store
db.inventory.aggregate( [
{ $match : { storeId : "store100" } },
{ $unwind : "$vars" },
{ $group : {
_id : "result",
count : { $sum : "$vars.q" } } } ] )
{ "_id" : "result", "count" : 29347 }
Inventory – Sample Geo-Query
• Get inventory for an item near a point
db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.8006, 40.0908] }, maxDistance : 10000.0, spherical : true, limit : 10, query : { "productId" : "p200", "vars.sku" : "sku11736" } } )
Inventory – Sample Geo-Query
• Get closest store with available sku
db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.800672, 40.090844] }, maxDistance : 10000.0, spherical : true, limit : 1, query : { productId : "p200", vars : { $elemMatch : { sku : "sku11736", q : { $gt : 0 } } } } } )
Inventory – Sample Geo-Aggregation
• Get count of inventory for an item near a point
db.inventory.aggregate( [ { $geoNear: { near : { type : "Point", coordinates : [-82.800672, 40.090844] }, distanceField: "distance", maxDistance: 10000.0, spherical : true, query: { productId : "p200", vars : { $elemMatch : { sku : "sku11736", q : {$gt : 0} } } }, includeLocs: "dist.location", num: 5 } }, { $unwind: "$vars" }, { $match: { "vars.sku" : "sku11736" } }, { $group: { _id: "result", count: {$sum: "$vars.q"} } }])
Inventory – Sample Indices
• { storeId : 1 }
• { productId : 1, storeId : 1 }
• { productId : 1, location : "2dsphere" }
• Why not "vars.sku"?– { productId : 1, storeId : 1, "vars.sku" : 1 }
Horizontal Scaling
Inventory – Technical Decisions
Store
Inventory
Schema
Indexing
ShardEast
ShardCentr
al
ShardWest
East DC
Inventory – Sharding TopologyWest DC Central DC
LegacyInventor
y
Primary
Primary
Primary
Inventory – Shard Key
• Choose shard key– { storeId : 1 } ?– { productId : 1, storeId : 1 } ?– { storeId : 1, productId : 1 } ?
• Set up sharding– sh.enableSharding("inventoryDB")– sh.shardCollection( "inventoryDB.inventory", { storeId : 1, productId : 1 } )
Inventory – Shard Tags
• Set up shard tags– sh.addShardTag("shard0000", "west")– sh.addShardTag("shard0001", "central")– sh.addShardTag("shard0002", "east")
• Set up tag ranges– sh.addTagRange("inventoryDB.inventory", { storeId : 0 }, { storeId : 100}, "west" )– sh.addTagRange("inventoryDB.inventory", { storeId : 100 }, { storeId : 200 }, "central" )– sh.addTagRange("inventoryDB.inventory", { storeId : 200 }, { storeId : 300 }, "east" )
Senior Consulting Engineer, MongoDB
Norman Graham
#MongoDBWorld
Thank You
