Cassandra - A decentralized storage system

Cassandra - A Decentralized Structured Storage System

Presented ByTejaswi Ganne Latha Muddu

Khulud AlsultanRajaramya Janagama

Marmik Patel Arunit Gupta

Chaitanya Sai

Prashant MalikFacebook

Avinash LakshmanFacebook

Outline• Data Model.• Cassandra API.• System Architecture:

• Partitioning.• Replication.• Membership and Failure Detection. • Bootstrapping and Scaling the Cluster.• Local Persistence.

Introduction and Data Model

-Tejaswi Ganne

Introduction•Apache Cassandra is an open source distributed storage system.•Manages very large data spread across many commodity servers located across many data centers.•Named after the Greek mythological prophet Cassandra.•Initially developed at Facebook to power their Inbox Search feature, later Facebook open sourced it as Apache Incubator project.•Features – •High Scalability •High Availability •Fault Tolerant

Features• High Scalability: There is no downtime or interruptions to

applications as read and write throughput increases linearly as new machines get added.

• High Availability: It refers to systems which are durable and likely to operate continuously without any failure for a long time.

• Fault Tolerant: Data is automatically replicated to multiple nodes, where failed nodes can be replaced within no time. Replication is supported across multiple data centers.

Data Model

•Uses a simple data model instead of a full Relational data model.• A table in Cassandra is a distributed multi dimensional map indexed by a key.•Value is a structured object.•Operations are atomic on each row per replica.

Data Model

*Figure taken from http://www.tutorialspoint.com/cassandra/cassandra_data_model.htm

Data Model Contd…•Each row can have different number of columns.•Each column has < Name, Value, Timestamp >.•Columns can be ordered by names and timestamps.•Columns are grouped into Column Families (CF).•Two types of CFs – •Simple Column Family –Has columns. –A Column can be accessed using the convention - »column_family : column •Super Column Family –CF within a CF. –Has a Simple CF, or another Super CF in it. –A Column can be accessed using the convention - »column_family : super_column : column

Key-Value Model

• It is column oriented NoSQL system

• Row is collection of columns labeled with a name

• Key is the column name and a row must contain at least 1 column

https://10kloc.wordpress.com/2012/12/25/cassandra-chapter-three-data-model/

Related Work

•Amazon Dynamo Dynamo is a storage system that is used by Amazon to store and retrieve user shopping carts. It requires both read and write operations for managing timestamps.•Google Chubby GFS uses a simple design with a single master server for hosting the entire metadata and where the data is split into chunks and stored in chunk servers. It is made fault tolerant using the Chubby abstraction. Chubby achieves fault-tolerant through replication.

References• https://en.wikipedia.org/wiki/Apache_Cassandra• http://cassandra.apache.org/• http://www.planetcassandra.org/what-is-apache-cassandra

/• http://

docs.datastax.com/en/cassandra/2.1/cassandra/gettingStartedCassandraIntro.html

Questions???

Cassandra API

-Muddu Latha

Cassandra Query Language

• The Cassandra Query Language (CQL) is the primary language for communicating with the Cassandra database.

CQL Statements :Data Definition StatementsData Manipulation StatementsQueries

Data Definition Statements• Create Keyspace• Use • Alter Keyspace • Drop Keyspace • Create Table• Alter Table • Drop Table• Create Type

• Alter Type• Drop Type• Create Trigger • Drop Trigger• Create Function• Drop Function• Create Aggregate• Drop Aggregate

Data Definition Statements(Cont..)

1. Create Keyspace : cqlsh> CREATE KEYSPACE sample_demo with replication = {‘ class ’: ‘ SimpleStrategy ‘ , ‘ replicaton_factor ‘ : 3 };2.Use Keyspace : cqlsh> USE sample_demo ;3.Alter Keyspace :cqlsh>ALTER KEYSPACE sample_demo WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 5};

Data Definition Statements(Cont..)

4. Drop Keyspace : DROP KEYSPACE sample_demo ;5. Create Table :CREATE TABLE presentors_list ( firstname text, lastname text, classid int, email text, PRIMARY KEY (lastname));6. Alter Table :ALTER TABLE presentors_list ADD city text ;

Data Definition Statements(Cont..)

7 . Drop Table :removes a table.DROP TABLE presentors_list;

8. Truncate Table : removes all data from a table.TRUNCATE presentors_list;

CQL Statements Execution Screenshots

Data Manipulation Statements

Insert ,Update , Delete , Batch1.Insert :INSERT INTO presentors_list( firstname , lastname , classid ,email ) VALUES( ‘ lakshmi‘ , ‘ upadrasta ‘ , 29 ,mltx9@mail.umkc.edu);2.Update :UPDATE presentors_list TTL 400 set firstname = ‘ dileep ’ , classid=‘28’, WHERE lastname =‘ upadrasta ’ ;

Data Manipulation Statements(Cont..)

3.Delete : DELETE FROM presentors_list USING TIMESTAMP WHERE lastname=‘upadrasta’ ;4.Batch :BEGIN BATCH

INSERT QUERY;UPDATE QUERY;DELETEQUERY;

APPLY BATCH

Queries

Select SELECT * FROM presentors_list ;SELECT firstname , email WHERE lastname = ‘upadrasta ’;

References

1. http://www.planetcassandra.org/try-cassandra/

2. https://cassandra.apache.org/doc/cql3/CQL.html#createKeyspaceStmt

3. https://cassandra.apache.org/doc/cql3/CQL-2.2.html

Questions???

System ArchitecturePartitioning

- Khulud Alsultan

System Architecture

• The core distributed systems techniques: Partitioning Replication Membership and Failure handling Bootstrapping and Scaling the Cluster. Local Persistence.

System Architecture • These modules work in synchrony to handle

read/write requests.• Read/write request for a key gets routed to any

node in the cluster. • The node determines the replica for this

particular key.

System Architecture • For writes:Routes the requests to the replicas and waits for a quorum of replicas to acknowledge the completion of the writes. • For reads:

• Routes the requests to the closest replica OR • Routes the requests to all replicas and waits for a

quorum of responses

Partitioning

• Scale incrementally. • Dynamically partition the data over the set of nodes

in cluster.• Partitions data using consistent hashing.• Uses an order preserving hash function.• Output range is treated as ring.• Each node is assigned a random value which

represents its position on the ring.

Partitioning

http://www.ibm.com/developerworks/library/os-apache-cassandra/index.html#resources

Consistent Hashing • Example:

Cassandra assigns a hash value to each partition key:

if you have the following data:

https://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureDataDistributeHashing_c.html

Consistent Hashing

https://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureDataDistributeHashing_c.html

Consistent Hashing • Cassandra places the data on each node according to the value of

the partition key and the range that the node is responsible for.

https://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureDataDistributeHashing_c.html

Consistent hashingAdvantage:• Departure or arrival of a node only affects its

immediate neighbors and others remain unaffected.

Some Challenges:• The random position assignment of each node on

the ring leads to non-uniform data and load distribution.

• The heterogeneity in the performance of nodes.

Partitioning

• Two ways to address this issue: Nodes get assigned to multiple positions in the

circle.

Analyze load information on the ring and have

lightly loaded nodes move on the ring to alleviate

heavily loaded nodes

Questions???

Replication

-Rajaramya Janagama

Replication

• How data is duplicated across nodes.

• Why replication?

• To achieve high availability and durability.• Ensure fault tolerance and no failure by

replicating one or more copies of every row in a column family across nodes in cluster

Replication

• How to achieve replication?

• Each data item is replicated at N (replication factor) nodes.

• Coordinator node is responsible for the replication of data items.

• It also replicates keys across N-1 nodes.

Replication Policies

– Various options to replicate data

• Rack Unaware • Rack Aware• Datacenter Aware

Rack Unware

• Replicate data at N-1 successive nodes after its coordinator

Rack Aware

Replica 1

Replica 2

Rack 1

N1 N2 N3

Rack2

N4 N5 N6

• No two replicas should lie in the same rack.

N- Nodes

Data Center Aware

• No two replicas should lie in the same datacenter.

Rack1

Rack2

Datacenter 1

N1 N2 N4N3

N6N5 N7 N8

Datacenter 2

Replica1

Replica2

Advantages

• Cassandra provides durability guarantees in the presence of node failures and network partitions.

• The storage nodes are spread across multiple datacenters and are connected through high speed data links.

• This scheme of replicating across multiple datacenters allows us to handle entire data center failures without any outage.

Questions???

Membership And Failure Detection

- Marmik Patel

What is Membership?• Can be split into two parts:

1. Service Discovery2. Failure Detection

Service Discovery• Service Discovery comes into picture when new node is set

up and added to cluster• Based on Scuttlebutt Reconciliation, a very efficient anti-

antropy gossip protocol based mechanism• Scuttlebutt has very efficient utilization of CPU and gossip

channel

Gossip Protocol and Scuttlebutt Reconciliation

Gossip Protocol• Protocol that Cassandra uses to discover information about other nodes• Information transferred from node to the node it knows about• Not only for Membership, but also used to disseminate other system

related to control state such as health, tokens, addresses, data size etc.

Scuttlebutt Reconciliation• Not necessary that two participants in a gossip exchange most recent

mapping than those of the peer• Inspired by real life rumor spreading• Repair replicated data by comparing differences

Robbert van Renesse, Dan Mihai Dumitriu, Valient Gough, and Chris Thomas. Efficient reconciliation and flow control for anti-entropy protocols

Failure Detection

• Comes into picture when  the node is was taken down for maintenance, or fails due to an error

• Mechanism by which a node can locally determine if any other node in system is up or down

• Also used to avoid attempts to communicate with unreachable node

• Uses failure detector which is modified version of Φ Accrual Failure Detector

• Gossip protocol is used for exchanging information

Φ Accrual Failure Detector

• Based on very simple principle• Does not emit a Boolean value stating a node is up or down,

but emits a value which represents a suspicion level for nodes• Value is defines as Φ• Idea is to express the value of Φ on a scale that is dynamically

adjusted to reflect network and load condition• Difference between traditional failure detector and accrual

failure detector is which component of the system does what part of failure detection

Traditional Failure Detector vs Accrual Failure Detector

• In Traditional Failure Detector, the monitoring and interpretation are combined and output of this combination is Boolean.

• Application cannot do any interpretation as monitored information is already being interpreted

Traditional Failure Detector

Accrual Failure Detector

• Accrual Failure Detector provides lower level abstraction that avoids the interpretation of monitoring information

• Value associated with each process represents suspicion level which is left for application to interpret

http://www.jaist.ac.jp/~defago/files/pdf/IS_RR_2004_010.pdf

Properties of Φ

• Φ represents likelihood that node A is wrong about node B’s state• Assume when Φ = 1, A will make mistake in deciding state of B

is 10%, then likelihood is about 1% when Φ = 2 , 0.1% when Φ = 3 and so on

• Node maintains a sliding window of inter-arrival times of gossip messages to calculate the value of Φ

• Φ is very good in accuracy and speed• Also adjust well to network conditions and server load conditions• Cassandra approximate Φ using exponential distribution

Questions???

Bootstrapping

in

-Arunit Gupta

BootstrappingWhat is Bootstrapping?

Adding new nodes is called “Bootstrapping”

Ways of Adding new nodeThere are two ways of adding node :

– New node gets assigned a random token which gives its position in the ring. It gossips its location to rest of the ring where the information is exchanged about one another.

– New node reads its config file to contact it’s initial contact points.

• New nodes are added manually by administrator via CLI or Web interface provided by Cassandra.

http://s3.amazonaws.com/ppt-download/cassandraekaterinberg2013-131212053553-phpapp01.pdf?response-content-disposition=attachment&Signature=7pB%2BhMgGqV1vxcRUaqCbCt2%2BH6o%3D&Expires=1458678552&AWSAccessKeyId=AKIAJ6D6SEMXSASXHDAQ

Bootstrapping Contd..

• These initial contact points are known as Seeds, which is basically used by newly added node to know each other, where ultimate goal for all nodes in the cluster is to discover one another.

• Seeds can also come from configuration service like Zookeeper, which is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

“Because Coordinating Distributed Systems is a Zoo”

Google images

Facts!!! • Comparison with Amazon’s Dynamo which is a highly available key-value structured storage system.

“Dynamo’s load is no where close to what we see in practice over here at Facebook.” –Avinash Lakshman

nosqlmatters2012-130102154135-phpapp01.pdf

Configuration

In addition to seeds, you'll also need to configure the IP interface to listen on for Gossip and CQL, (listen_address and rpc_address respectively).Use listen_address that will be reachable from the listen_address used on all other nodes, and a rpc_address that will be accessible to clients.Once everything is configured and the nodes are running, use the bin/nodetool status utility to verify a properly connected cluster. For example:

https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html

Environment• Node outages occurred are often transient but may last for extended

intervals.• A network outage rarely signifies a permanent departure and should not

result in re-balancing of the partition assignment or repair of the unreachable replicas.

• Manual errors could result in unintentional startup of new Cassandra nodes. As a result an explicit mechanism is considered appropriate to initiate the addition and removal of nodes from a Cassandra Instance.

• Administrator – uses a command line tool or a browser to connect to a Cassandra node and issue a membership change to join or leave the cluster.

Scaling the cluster• Whenever a new node is added into the system, it gets assigned a token such that it can alleviate heavily loaded

node.• New node will take the range which other node were responsible for before.• Cassandra bootstrapping algorithm is initiated from any other node in the system either using a command line

utility or web dashboard.• The node giving up the data streams the data over to the new nodes using kernel copy techniques.

Cassandra Ring showing scalability.

Scaling the Cluster

https://www.google.com/search?q=scalability+in+cassandra&biw=1366&bih=667&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjtztnc4dfLAhWEzoMKHQ1zD4MQ_AUIBygC#imgrc=qusi2veDeVAH4M%3A

FutureWhat is the Future?

• Operational experience has shown that data can be transferred at the rate of 40MB/sec from a single node. Work is going on to have multiple replicas take part in the bootstrap transfer by parallelizing the effort, similar to bit torrent which is a p2p system used to transfer large files to thousands of location in a short period of time

• Facebook uses bit torrent to distribute updates to Facebook servers.

“Bit Torrent is fantastic for this, it’s really great,” Cook said. “It’s ‘super-duper’ fast and it allows us to alleviate a lot of scaling concerns we’ve had in the past, where it took forever to get code to the webservers before you could even boot it up and run it.”

Virtual nodes in Cassandra• One of the new features slated for Cassandra 1.2’s release was virtual nodes (vnodes) where

there was paradigm change from one token or range per node, to many per node. Within a cluster these can be randomly selected and be non-contiguous, giving us many smaller ranges that belong to each node.

Advantage?

Use of Heterogeneous machines in a cluster. Node Failures and backing up.

http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2

Questions???

Local Persistence

-Chaitanya Sai Manne

Local Persistence– Cassandra depends on the local file system for data

persistence.– The data is represented on disk using a format that lends

itself to efficient data retrieval.– For a data store to be considered persistent, it must write to

non-volatile storage.

Cassandra – more than one server

• All the nodes participate in a cluster

• They are independent – share nothing

• Add or remove as needed• If you need more capacity?

Add a server

Focus on singer server

http://www.slideshare.net/patrickmcfadin/introduction-to-cassandra-2014

Write operation• Firstly, it writes into the commit log• Then it puts into the in-memory data

structure i.e. memtable• The memtable is identified by the

primary key• Acknowledge back to the client• This is a simple process and that’s

what make scaling is easier• As memtable start to fill up there is a

flush process• Flush process writes the memtable to

a file called SS table i.e. Sorted String

• The writes here are sequential writeshttp://www.slideshare.net/sameiralk/cassendra

Example

Update usersSet firstname = ‘Chaitanya’Where id = ‘cm7cd’

write Rowkey,Column (id = ‘cm7cd’,

firstname = ‘Chaitanya’)

http://www.slideshare.net/patrickmcfadin/introduction-to-cassandra-2014

http://www.slideshare.net/patrickmcfadin/introduction-to-cassandra-2014

http://www.slideshare.net/patrickmcfadin/introduction-to-cassandra-2014

http://www.slideshare.net/patrickmcfadin/introduction-to-cassandra-2014

http://www.slideshare.net/patrickmcfadin/introduction-to-cassandra-2014

http://www.slideshare.net/patrickmcfadin/introduction-to-cassandra-2014

Compaction• Compaction is process which takes all the SSTables, does a

sequential reads back into the memtable of both files, do merge sort, picks the latest timestamp file and writes a brand new file.

• It deletes the old files.

http://www.slideshare.net/patrickmcfadin/introduction-to-cassandra-2014

Compaction

http://www.slideshare.net/patrickmcfadin/introduction-to-cassandra-2014

Read Operation

www.slideshare.net/sameiralk/cassendra

Read Operation• It look up in the memtable before going into the files on the disk• Look up is done in order of newest to oldest• Cassandra checks an in-memory data structure called Bloom filter• Bloom filter can quickly tell you whether the key exists in a file• A key in a column family have many columns so in order to prevent

scanning all the columns it maintain column indices• In a cluster, client can ask any node to retrieve the dataConsistency Levels• Set every read and write like ONE, TWO, ALL, QUORUM->51% etc.

Read Operation

http://www.slideshare.net/patrickmcfadin/introduction-to-cassandra-2014

Summary• Established high scalability, performance and wide

applicability• Very high update throughput, delivering low latency• Future works:

– Adding compression– Support atomicity across keys– Secondary index support

Questions???

Lakshman, Avinash, and Prashant Malik. "Cassandra: a decentralized structured storage system." ACM SIGOPS Operating Systems Review 44.2 (2010): 35-40.

For More Information

Thank You