C* Summit EU 2013: The Cassandra Experience at Orange

Jean Armel Luce

Orange France/DSIF/DF/SDF

V1.0

The Cassandra Experience at Orange Project PnS 3.0

2 Cassandra Summit Europe – October 17 2013

#CassandraEU

Summary

§  Short descripGon of PnS. Why did we choose C* ?

§  Our migraGon strategy

§  AMer the migraGon …

§  AnalyGcs with Hadoop/Pig/Hive over Cassandra

§  ContribuGons and open sourced modules from Orange & conclusions

Jean Armel Luce - Orange-DOP-PnS 3.0

Short description of PnS3


#CassandraEU

PnS – Short description

§  PnS means Profiles and Syndication : PnS is a highly available service for collecting and serving live data about Orange customers

§  End users of PnS are :

–  Orange customers (logged to Portal www.orange.fr) –  Sellers in Orange shops –  Some services in Orange (advertisements, …)



#CassandraEU

PnS – The Big Picture


End users

Millions of HTTP requests (Rest or Soap) Fast and highly available

Database

WebService to get or set data stored by pns : - postProcessing(data1) - postProcessing(data2) - postProcessing(data3) - postProcessing(datax) - …

PNS Data providers

Thousands of files (Csv or Xml) Scheduled data injection

DB Queries R/W operations


#CassandraEU

§  Until 2012, data were stored in 2 differents backends :

ü  MySQL cluster (for volatile data)

ü  PostGres « cluster » (sharding and replication)

§  and web services

(read and writes)

§  for batch updates

PnS2 – Architecture


Bagnolet

Sophia Antipolis

2 DCs architecture for high availability


#CassandraEU

Timeline – Key dates of PnS 3.0


PNS 2

2010 to 2012

•  Study phase We did a large study about a few NoSQL databases (Cassandra, MongoDB, Riak, Hbase, Hypertable, …) è We chose Cassandra as the single backend for PnS

06/2012 •  Design phase

We started the design phase of PnS3.0

09/2012

•  Proof Of Concept We started a 1st (small) Cassandra cluster in producGon for a non criGcal applicaGon : 1 table, key value access

04/2013 •  Produc7on phase

MigraGon of the 1st subset of data of PnS from mysql cluster to Cassandra in produc7on

05/2013

to 12/2013

•  Complete migra7on MigraGon of all other subsets of data from Mysql cluster and Postgres to Cassandra Add new nodes in the cluster (From 8 nodes in each DC to 16 nodes in each DC) Add a 3rd datacenter for AnalyGcs


#CassandraEU

PnS – Why did we choose Cassandra ?

§  Cassandra fits our requirements :

–  Very high availability

–  Low latency

–  Scalability

§  And also :

–  Ease of use : Cassandra is easy to administrate and operate –  Some features that I like (rack aware, CL per request, …) –  Cassandra is very efficient for simple requests :

« SELECT mycol1, mycol2, …, mycolx FROM mytable WHERE myprimarykey = ‘mycustomerid’ »


PnS2 = 99,95% availability we want to improve it !!!

20 ms < RT PnS2 web service < 150 ms we want to improve it !!!

Higher load, higher volume next years ?

unpredictable; better scalability brings new businesses

Our migration strategy


#CassandraEU


The migraGon -‐ Input

§  During the migration, we need to :

§  maintain a very high availability

§  maintain (or lower) the latency during the migration

§  guarantee no functional regression

§  Question :

§  How can we migrate the data to Cassandra without any interruption of service ?


#CassandraEU

S ubdivision intosubsets

Migration data of the subset

Check/validation of the migration

S witch queriesto Cassandra for the subset

goto 1st subset

goto next subset

§  Subdivision of data into many subsets according to many criteria : §  Same source of data

§  Relationships between data

§  And then, migrate each subset 1 by 1

§  Definition of a generic process for all the subsets


The migraGon : Step by step processing


#CassandraEU


The migraGon : Tools and Gps

§  The strategy of migration is based on 2 main facilities :

the timestamp management by Cassandra Each data stored in C* is timestamped.

It is possible to set this timestamp when inserting/updating/deleting a data in Cassandra. When Cassandra retrieves a data item, it returns the value having the most recent timestamp. We use this feature to distinguish the values stored before the migration started and the values inserted during or inserted after the migration

mod_dup An Apache module developped (and open sourced) by Orange teams. Mod_dup can duplicate web requests, filter them on some criteria, substitute characters (regexp), and send the duplicated requests to another pool of web servers. Used in order to fill legacy (relational) database and Cassandra database simultaneously during the migration of the subset

HTTP Req

PNS 2

PNS 3

mod_dup


#CassandraEU

The migration : initial state


Data providers

End users

HTTP Rest/Soap Read

Files transfer via FTP or CFT PNS 2

DB

HTTP Rest/Soap Write

SQL Read/Write

WebServer

BatchInjector

Step 0


#CassandraEU


The migration : double feed

PNS 3 Cassandra

DB

CQL update

Duplicate streams (files) from data providers BatchInjector

WebServer

Duplicate HTTP update streams from end users

Data providers

mod_dup

Step 1


#CassandraEU


The migration : copy data form PnS2 to PnS3

PNS 3 Cassandra

DB BatchInjector

WebServer

Batch Injection TimeStamp = start date of extraction

mod_dup

Data providers

HTTP Write

Step 2


#CassandraEU


The migration : control

PNS 3 Cassandra

DB BatchInjector

WebServer

Synchro Control

SQL

CQL

mod_dup

Data providers

HTTP Write

Step 3


#CassandraEU


The migration : switch reads

PNS 3 Cassandra

DB BatchInjector

WebServer End users

Files transfer via FTP or CFT PNS 2 DB

HTTP Rest/Soap Write WebServer

BatchInjector

100 % read now on Cassandra HTTP Read requests

HTTP Rest/Soap Read

mod_dup

Data providers

HTTP Write

Step 4


#CassandraEU


The migration : stop double feed

PNS 3 Cassandra

DB BatchInjector

WebServer End users

PNS 2 DB

WebServer

BatchInjector

100 % read on Cassandra 100% write on Cassandra

for HTTP request

100% write on Cassandra for Data injection

Files transfer

HTTP Rest/Soap Write

HTTP Read/Write requests

Data providers

Files transfer via FTP or CFT

mod_dup

HTTP Write

HTTP Read requests

Step 5


#CassandraEU


The migraGon

§  Using this procedure :

§  We can migrate to Cassandra without any interruption of service

§  It is possible to switch progressively to Cassandra rather than doing a one shot switch

§  During the control phase, we can take time (a few days, a few weeks) to check that everything is OK before switching to Cassandra

§  It is possible to easily rollback the migration of a subset if errors are found during the control phase, without losing any update

§  Doesn’t work if the queries are not idempotent.

§  After the migration, we can easily duplicate production requests (entirely or partially) and send them to a bench platform thanks to mod_dup

After the migration …


#CassandraEU

§  Comparison before/after migration to Cassandra

§  Some graphs about the latency of the web services are very explicit :

Service push mail Service push webxms


The latency

dates of migration to C*


#CassandraEU

§  Read and write latencies are now in microseconds in the datanodes :

Thanks to and

This latency will be improved by (tests in progress) : ALTER TABLE syndic WITH compacGon = { 'class' : 'LeveledCompacGonStrategy', 'sstable_size_in_mb' : ?? };


The latency


#CassandraEU

•  We got a few hardware failures and network outages

•  No impact on QoS :

•  no error returned by the application

•  no real impact on latency


The availability


#CassandraEU

•  PnS activity is always increasing (volume of data and requests/sec)

•  How to measure the capacity of a cluster ?

Capacity of a C* cluster = capacity of a node * number of nodes (true if all nodes are identical)

•  there are 2 ways to deal with the expansion of activity :

Ø  scale up (add more resources such as CPU, disks, RAM to each node)

Ø  scale out (add new nodes in the cluster)


The scalability


#CassandraEU

•  Thanks to vnodes (available since Cassandra 1.2), it is easy to scale out

With NetworkTopologyStrategy, make sure to distribute evenly the nodes in the racks


The scalability

Analytics with Hadoop/ Pig/Hive over Cassandra


#CassandraEU


Basic architecture of the Cassandra cluster

§  Cluster without Hadoop : 2 datacenters, 16 nodes in each DC

§  RF (DC1, DC2) = (3, 3)

§  Requests from web servers in DC1 are sent to C* nodes in DC1

§  Requests from web servers in DC2 are sent to C* nodes in DC2

Poolofweb

serversDC1

Poolofweb

serversDC2

DC1 DC2


#CassandraEU


Architecture of the Cassandra cluster with the datacenter for analyGcs

§  Cluster with Hadoop : 3 datacenters, 16 nodes in DC1, 16 nodes in DC2, 4 nodes in DC3

§  RF (DC1, DC2, DC3) = (3, 3, 1)

§  Because RF = 1 in DC3, we shall need less storage space in this datacenter

§  We favor cheaper disks (SATA) in DC3 rather than SSDs or FusionIo cards

§  Works better with HSHA Thrift server (tests in progress)


#CassandraEU


Architecture of the Cassandra cluster with the datacenter for analyGcs

DC1 DC2

DC3

Poolofweb

serversDC1

Poolofweb

serversDC2

Contributions and open sourced modules from Orange & conclusions


#CassandraEU


ContribuGons and open sourced modules

§  Open sources by Orange

§  PHP driver for Cassandra :

https://github.com/Orange-OpenSource/YACassandraPDO

Thanks to Sandro Lex & Mathieu Lornac

§  Mod_dup (Migration to Cassandra)

§ https://github.com/Orange-OpenSource/mod_dup

Thanks to Jonas Wustrack & Emmanuel Courreges

§  Other contributions

§  C driver (libdbi driver) :

http://libdbi-drivers.cvs.sourceforge.net/viewvc/libdbi-drivers/libdbi-drivers/?pathrev=Branch-2012-07-02-cassandra

Thanks to Emmanuel Courreges


#CassandraEU

§  With Cassandra, we have improved our QoS

§  We are able to open our service to new opportunities

§  There is an ecosystem around C* (Hadoop, Hive, Pig, Storm, Shark, …), which offers more capabilities. However, we would love to have some of the components (Hive) integrated in C* core (as Pig)

§  PnS3 works better and hopefully cheaper than PnS2


Conclusions


#CassandraEU


Thank you


#CassandraEU


Questions


#CassandraEU


A few answers about hardware/OS version /Java version/Cassandra version

§  Hardware :

§  16 nodes in each DC at the end of 2013 :

§  6 CPU Intel® Xeon® 2.00 GHz

§  24 GB RAM

§  FusionIO 320 GB MLC

§  OS :

§  Ubuntu Precise (12.04 LTS)

§  Cassandra version :

§  1.2.2 (with a few patches backported from 1.2.3)

§  Java version :

§  Java7u7 : not recommended, upgrade scheduled soon


#CassandraEU


A few answers about data and requests

§  Data types : §  Volume : 6 TB at the end of 2013

§  elementary types : boolean, integer, string, date

§  collection types

§  complex types : json, xml (between 1 and 20 KB)

§  Requests : §  10.000 requests/sec at the end of 2013

§  80% get

§  20% set

§  Consistency level used by PnS : §  ONE (95% of the queries)

§  LOCAL_QUORUM (5% of the queries)

Technology

C* Summit EU 2013: The Cassandra Experience at Orange