PolyglotDatabases

php[tek] 2015

@majidfn

Majid Fatemian

Ubisoft Montreal

Relational

DATA

BIGDATA

NoSQL

1 2 3

Non-Relational Cluster Schemaless

1. Document

2. Key-Value

3. Graph

4. Column

MongoDB, CouchBase

Memcache, Redis

Neo4J, OrientDB

Cassandra, HBase

A

C P

Availability

Consistency Partition Tolerance

Consistency vailabilityA

artitionPConsistency

vailabilityA artitionP

MongoDB

MySQL

Postgres

CouchBase

Cassandra

Riak

Fitness Evolved

Fitness Evolved

Fitness Evolved

Fitness Evolved

3M 15K 160K

Players CCU Daily Ses.

STATS

2011

2012

2013

2014

2011

2012

2013

201

Application

Normalized

ACID

Transaction{ConsistencyRelational

Replication

Table PartitioningAvailability{Relational

OPs / DBAs Tools Devs{DevOps

Relational

2011

2012

2013

201

Aggregated

…

Detailed

Profile Community

History Leaderboards

Single Point Failure Availability} Relational

ScalabilityRelational

SchemaRelational

{ player_id: 1234, * calories: 40, * duration: 75, * activity: jumping_ropes, game_id: 4001, platform: WiiU, difficulty_level: medium, repetition: 178, score: 450, endurance:3}

{ player_id: 1234, calories: 75, duration: 120, activity: boxing }

Player Game Cals Duration Activity Difficulty Score Reps.

1000 1 100 80 Box - - -

2000 2 90 100 Jump H 4.5 180

1000 2 80 120 Pushup M 4 130

2000 1 120 110 Situps - - -

Player Game Cals Duration Activity Difficulty Score Reps.

1000 1 100 80 Box - - -

2000 2 90 100 Jump H 4.5 180

1000 2 80 120 Pushup M 4 130

2000 1 120 110 Situps - - -

DistributionRelational

Data CData B

Data A

Data Access Layer

ACID Transaction Partitioning Replication

Tools OPs

Data Volume Fixed Schema Scale Up Manual Sharding

+ -

NoSQL...

mongoDB

1 2 3

OPs DEVs Community

SchemalessNon-Relational

player_id: 1234, * calories: 40, * duration: 75, * activity: jumps, game_id: 4001, platform: WiiU, difficulty_level: 2, repetition: 178, score: 450, endurance:3

player_id: 1234, calories: 75, duration: 120, activity: boxing

ScalabilityNon-Relational

ScalabilityNon-Relational

+

ShardingNon-Relational

ShardingNon-Relational

Data AA Data BB Data CC

• DB Managed

• Balanced Load

ShardingNon-Relational

• Sharding key

• Bouncing

• Decreased resilience

Map/ReduceNon-Relational

R1 R2 R3

Map/ReduceNon-Relational

QueryQueryQuery

R1 R2 R3

Map/ReduceNon-Relational

QueryQuery Query

R1 R2 R3

Map

Map/ReduceNon-Relational

QueryQuery Query

R1 R2 R3

ResultReduce

Map

DenormalizedNon-Relationalread / }write

Disk Space

QueryNon-RelationalData Model

Aggregation}

Eventual- Consistency

Non-RelationalX1

X1 X1

Eventual- Consistency

Non-Relational

X2

W

X1 X1

X2

Eventual- Consistency

Non-Relational

X2

W

X1 X1

R

X1

X2

Eventual- Consistency

Non-Relational

X2

W

X2

X2

R

X2

X2

LockingNon-RelationalDB Level (2.2)}Document Level (3.0)

BackupNon-Relational

Point-in-time

Denormalized data Disk Space Expertise Complex QueryingEventual Consistency Resource Usage DB-Level Locking

-

Schema-Less Aggregated Data Large Scale Data

Sharding Map/Reduce

Memory Storage Journaling

+

Random ProfileSearch(1M)

Full History Retrieval(6M)

Insert(6M)

AggregationMap/Reduce

(6M)

0

0.003

0.005

0.008

0.01

MySQLMongoDB

PersistencePolyglot

Profile (Total Sum) Leaderboards

…

Activity Details Activity Aggregations

Master

Slave

Config Server

Shard 1Replicaset

Shard 2Replicaset

Data Access Layer

TheChallenge

Inconsistency1

Data Access Layer

Commit2-Phase

Data Access Layer

W

Pending

Commit2-Phase

Data Access Layer

Pending

W Commit2-Phase

Data Access Layer

Done

Commit2-Phase

Consistency vs.

Latency2

Operations3

Availability4

Data Access Layer

Data Access Layer

Data Access Layer

Data Integrity5

Unit{TEST IntegrationFunctional

DATAMigration

Profile

Activity }3rd Party

{ profile: { GUID:…, platform: { name:’xbox’, xuid:… } }, activity:…, duration:…, datetime:… }

1 One-Shot Migration

2 Profile Versioning4 hrs !!

Live !!

+1

R

W

V+11

R

W

V

+1

2

FinalResults

2011 2012 2013 2014

9x response time

3 years uptime

> Go Hybrid, wisely!

> Go Hybrid, wisely!

Denormalized dataEventual Consistency

Evolving

Schema-Less Large Scale Data

Sharding Map/Reduce

ACID Transaction Replication Tools, OPs

Data Volume Fixed Schema Scaling

ThankYOU!

@majidfn

http://joind.in/13728