PolyglotDatabases
php[tek] 2015
@majidfn
Majid Fatemian
Ubisoft Montreal
1 2 3
Non-Relational Cluster Schemaless
1. Document
2. Key-Value
3. Graph
4. Column
MongoDB, CouchBase
Memcache, Redis
Neo4J, OrientDB
Cassandra, HBase
A
C P
Availability
Consistency Partition Tolerance
Consistency vailabilityA
artitionPConsistency
vailabilityA artitionP
MongoDB
MySQL
Postgres
CouchBase
Cassandra
Riak
3M 15K 160K
Players CCU Daily Ses.
STATS
2011
2012
2013
2014
2011
2012
2013
201
Normalized
ACID
Transaction{ConsistencyRelational
Replication
Table PartitioningAvailability{Relational
OPs / DBAs Tools Devs{DevOps
Relational
2011
2012
2013
201
Aggregated
…
Detailed
Profile Community
History Leaderboards
Single Point Failure Availability} Relational
ScalabilityRelational
SchemaRelational
{ player_id: 1234, * calories: 40, * duration: 75, * activity: jumping_ropes, game_id: 4001, platform: WiiU, difficulty_level: medium, repetition: 178, score: 450, endurance:3}
{ player_id: 1234, calories: 75, duration: 120, activity: boxing }
Player Game Cals Duration Activity Difficulty Score Reps.
1000 1 100 80 Box - - -
2000 2 90 100 Jump H 4.5 180
1000 2 80 120 Pushup M 4 130
2000 1 120 110 Situps - - -
Player Game Cals Duration Activity Difficulty Score Reps.
1000 1 100 80 Box - - -
2000 2 90 100 Jump H 4.5 180
1000 2 80 120 Pushup M 4 130
2000 1 120 110 Situps - - -
DistributionRelational
Data CData B
Data A
Data Access Layer
ACID Transaction Partitioning Replication
Tools OPs
Data Volume Fixed Schema Scale Up Manual Sharding
+ -
mongoDB
1 2 3
OPs DEVs Community
SchemalessNon-Relational
player_id: 1234, * calories: 40, * duration: 75, * activity: jumps, game_id: 4001, platform: WiiU, difficulty_level: 2, repetition: 178, score: 450, endurance:3
player_id: 1234, calories: 75, duration: 120, activity: boxing
ScalabilityNon-Relational
ScalabilityNon-Relational
+
ShardingNon-Relational
ShardingNon-Relational
Data AA Data BB Data CC
• DB Managed
• Balanced Load
ShardingNon-Relational
• Sharding key
• Bouncing
• Decreased resilience
Map/ReduceNon-Relational
R1 R2 R3
Map/ReduceNon-Relational
QueryQueryQuery
R1 R2 R3
Map/ReduceNon-Relational
QueryQuery Query
R1 R2 R3
Map
Map/ReduceNon-Relational
QueryQuery Query
R1 R2 R3
ResultReduce
Map
DenormalizedNon-Relationalread / }write
Disk Space
QueryNon-RelationalData Model
Aggregation}
Eventual- Consistency
Non-RelationalX1
X1 X1
Eventual- Consistency
Non-Relational
X2
W
X1 X1
X2
Eventual- Consistency
Non-Relational
X2
W
X1 X1
R
X1
X2
Eventual- Consistency
Non-Relational
X2
W
X2
X2
R
X2
X2
LockingNon-RelationalDB Level (2.2)}Document Level (3.0)
BackupNon-Relational
Point-in-time
Denormalized data Disk Space Expertise Complex QueryingEventual Consistency Resource Usage DB-Level Locking
-
Schema-Less Aggregated Data Large Scale Data
Sharding Map/Reduce
Memory Storage Journaling
+
Random ProfileSearch(1M)
Full History Retrieval(6M)
Insert(6M)
AggregationMap/Reduce
(6M)
0
0.003
0.005
0.008
0.01
MySQLMongoDB
PersistencePolyglot
Profile (Total Sum) Leaderboards
…
Activity Details Activity Aggregations
Master
Slave
Config Server
Shard 1Replicaset
Shard 2Replicaset
Data Access Layer
Data Access Layer
Commit2-Phase
Data Access Layer
W
Pending
Commit2-Phase
Data Access Layer
Pending
W Commit2-Phase
Data Access Layer
Done
Commit2-Phase
Consistency vs.
Latency2
Data Access Layer
Data Access Layer
Data Access Layer
Unit{TEST IntegrationFunctional
Profile
Activity }3rd Party
{ profile: { GUID:…, platform: { name:’xbox’, xuid:… } }, activity:…, duration:…, datetime:… }
1 One-Shot Migration
2 Profile Versioning4 hrs !!
Live !!
2011 2012 2013 2014
> Go Hybrid, wisely!
> Go Hybrid, wisely!
Denormalized dataEventual Consistency
Evolving
Schema-Less Large Scale Data
Sharding Map/Reduce
ACID Transaction Replication Tools, OPs
Data Volume Fixed Schema Scaling
ThankYOU!
@majidfn
http://joind.in/13728