[Rakuten TechConf2014] [B-1] Performance at scale

Performance at ScaleOctober 2014

Amir Ish-ShalomSenior Solutions Architect

About Viber

The Viber service• Free, cross platform text messaging• Free, cross platform VoIP calls

(voice and video)• Photo, video and location sharing• Stickers and Emoticons• Group communication platform

(up to 100 participants)• Push To Talk

Monetization

Viber out Sticker market

Simplicity and User Experience• No registration needed

• User ID = your mobile number• Automatic friends detection

(no add a friend)

• Always on. No battery impact• 32 languages• Multiple devices experience:

Mobile, Tablets and Desktop

Viber in numbers

Viber in numbers• Over 400 million users• 1 million added daily• Billions of messages every day• Billions of talking minutes every month

Viber Growth

20

13

-12

20

13

-11

20

13

-10

20

13

-09

20

13

-08

20

13

-07

20

13

-06

20

13

-05

20

13

-04

20

13

-03

20

13

-02

20

13

-01

20

12

-12

20

12

-11

20

12

-10

20

12

-09

20

12

-08

20

12

-07

20

12

-06

20

12

-05

20

12

-04

20

12

-03

20

12

-02

20

12

-01

20

11

-12

20

11

-11

20

11

-10

20

11

-09

20

11

-08

20

11

-07

20

11

-06

20

11

-05

20

11

-04

20

11

-03

20

11

-02

In 2013 vs. 2012 there was:• Over 3x growth in talking minutes• Over 5x growth in messages• Over 12x growth in group messages

Viber DB Architecture

Viber DB Architecture – 1st Generation

In-housein-memory DB

Application ServersViberClients

Viber DB Architecture – 2nd Generation

Redis Cache MongoDB Cluster

Redis Cluster

Application ServersViberClients

Redis Sharder

Redis Sharder

2nd generation DB architecture advantages• Got us through the first few years of extreme growth• Never lost data from MongoDB• Redis performance

2nd generation DB architecture problems• MongoDB performance• MongoDB does not scale well with many application servers• Redis – In-memory database with no sharding• Redis Sharder – Not manageable and robust enough

3rd generation DB architecture requirements• High performance• Large data sets• Scalable• Robust• Backed-up• Always on• Easy to monitor• Prefer single DB solution

Solution:

Viber DB Architecture – 3rd Generation

Application ServersViberClients

CouchbaseBackupClusterCouchbase

Clusters

XDCR

Migrating from 2nd to 3rd generation DB’s• Migrate a live system• Zero downtime• No data loss• Consistent data

How did we migrate?• Stage 1: Add new CB cluster in parallel to existing cluster

Only delete keys from CB• Stage 2: Read only from MongoDB

Write/Delete to both CB & MongoDB• Stage 3: Background process that copies all data from

MongoDB to CB (if it doesn’t exist)• Stage 4: Validate data (both DB’s should be identical)• Stage 5: Read only from CB

Write/Delete from both CB & MongoDB• Stage 6: Remove MongoDB and use only CB

Couchbase Cluster with 60 nodes

Migration

Back-end servers• Over 500 application servers• 2nd generation DB architecture:

• MongoDB – 1 cluster with 150 servers (master + 2 slaves)• Redis – 3 clusters with a total of 144 servers (master + 1 slave)

• 3rd generation DB architecture:• 7 Couchbase clusters (up to 60 nodes each)• 1 – 2 replicas, XDCR & external backup• Total of less than 200 Couchbase servers

Increased performance using fewer DB servers!

Interesting facts1. We have recently doubled our Couchbase instance sizes to

cope with increased usage2. Total of ~1.5 million DB operations per second3. Bigger clusters don’t necessarily do more ops4. Highest performing clusters have 100% of their data in

memory

Couchbase Cluster with 12 nodes & XDCR

Questions?

Thank you