75
Jonathan Ellis [email protected] / @spyced The Cassandra Distributed Database Tuesday, November 30, 2010

Cassandra devoxx 2010

  • Upload
    jbellis

  • View
    3.425

  • Download
    0

Embed Size (px)

DESCRIPTION

Introduction to Cassandra at Devoxx 2010

Citation preview

Page 1: Cassandra devoxx 2010

Jonathan [email protected] / @spyced

The Cassandra Distributed Database

Tuesday, November 30, 2010

Page 2: Cassandra devoxx 2010

Bigtable, 2006 Dynamo, 2007

OSS, 2008

Incubator, 2009 TLP, 2010

Tuesday, November 30, 2010

Page 3: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 4: Cassandra devoxx 2010

Why Cassandra?

✤ Relational databases are not designed to scale

✤ B-trees are slow

✤ and require read-before-write

Tuesday, November 30, 2010

Page 5: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 6: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 7: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 8: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 9: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 10: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 11: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 12: Cassandra devoxx 2010

(“The eBay Architecture,” Randy Shoup and Dan Pritchett)

Tuesday, November 30, 2010

Page 13: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 14: Cassandra devoxx 2010

eBay: NoSQL pioneer

✤ “BASE is diametrically opposed to ACID. Where ACID is pessimistic and forces consistency at the end of every operation, BASE is optimistic and accepts that the database consistency will be in a state of flux. Although this sounds impossible to cope with, in reality it is quite manageable and leads to levels of scalability that cannot be obtained with ACID.”✤ ”BASE: An Acid Alternative,” Dan Pritchett, eBay

Tuesday, November 30, 2010

Page 15: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 16: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 17: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 18: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 19: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 20: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 21: Cassandra devoxx 2010

Commitlog

MemtableWriterReader

The Log-Structured Merge-Tree,Bigtable: A Distributed Storage System for Structured Data

Tuesday, November 30, 2010

Page 22: Cassandra devoxx 2010

Myth 1

✤ “Cassandra is for people who don’t understand {SQL, denormalization, query tuning, ...}”

✤ Similarly: “Only users of [database X] are turning to Cassandra, because X sucks.”

Tuesday, November 30, 2010

Page 23: Cassandra devoxx 2010

Myth 2

✤ “Only huge social media sites care about scalability.”

Tuesday, November 30, 2010

Page 24: Cassandra devoxx 2010

Cassandra in production

✤ Digital Reasoning: NLP + entity analytics

✤ OpenX: largest publisher-side ad network in the world

✤ Cloudkick: performance data & aggregation

✤ SimpleGEO: location-as-API

✤ Ooyala: video analytics and business intelligence

✤ ngmoco: massively multiplayer game worlds

Tuesday, November 30, 2010

Page 25: Cassandra devoxx 2010

Myth 3

✤ “Cassandra is only appropriate for unimportant data.”

Tuesday, November 30, 2010

Page 26: Cassandra devoxx 2010

Durabilty

✤ Write to commitlog

✤ fsync is cheap since it’s append-only

✤ Write to memtable

✤ [amortized] flush memtable to sstable

Tuesday, November 30, 2010

Page 27: Cassandra devoxx 2010

Commitlog

MemtableWriterReader

The Log-Structured Merge-Tree,Bigtable: A Distributed Storage System for Structured Data

Tuesday, November 30, 2010

Page 28: Cassandra devoxx 2010

SSTable format, briefly

<row data 0><row data 1>

...<row data 127>

...<row data 255>

...

<key 127><key 255>

...

Sorted [clustered] by row key

Tuesday, November 30, 2010

Page 29: Cassandra devoxx 2010

Scaling

Tuesday, November 30, 2010

Page 30: Cassandra devoxx 2010

A

L

T

W

Tuesday, November 30, 2010

Page 31: Cassandra devoxx 2010

A

L

T

W

F

Tuesday, November 30, 2010

Page 32: Cassandra devoxx 2010

A

L

T

W

F(A-L]

Tuesday, November 30, 2010

Page 33: Cassandra devoxx 2010

A

L

T

W

F(A-F]

(F-L]

Tuesday, November 30, 2010

Page 34: Cassandra devoxx 2010

A

L

T

W

F

Key “C”

Tuesday, November 30, 2010

Page 35: Cassandra devoxx 2010

Reliability

✤ No single points of failure

✤ Multiple datacenters

✤ Monitorable

Tuesday, November 30, 2010

Page 37: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 38: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 39: Cassandra devoxx 2010

Good architecture solves multiple problems at once

✤ Availability in single datacenter

✤ Availablility in multiple datacenters

Tuesday, November 30, 2010

Page 40: Cassandra devoxx 2010

A

LT

W

F

P

YKey “C”

U

Tuesday, November 30, 2010

Page 41: Cassandra devoxx 2010

A

LT

W

F

P

YKey “C”

U

Tuesday, November 30, 2010

Page 42: Cassandra devoxx 2010

A

LT

W

F

P

YKey “C”

U

Tuesday, November 30, 2010

Page 43: Cassandra devoxx 2010

A

LT

W

F

P

YKey “C”

U

Tuesday, November 30, 2010

Page 44: Cassandra devoxx 2010

A

LT

W

F

P

YKey “C”

U

XTuesday, November 30, 2010

Page 45: Cassandra devoxx 2010

A

LT

W

F

P

YKey “C”

U

Xhint

Tuesday, November 30, 2010

Page 46: Cassandra devoxx 2010

A

LT

W

F

P

YKey “C”

U

Xhint

Tuesday, November 30, 2010

Page 47: Cassandra devoxx 2010

A

LT

W

F

P

Y

U

Tuesday, November 30, 2010

Page 48: Cassandra devoxx 2010

A

LT

W

F

P

Y

U

Tuesday, November 30, 2010

Page 49: Cassandra devoxx 2010

A

LT

W

F

P

Y

U

Tuesday, November 30, 2010

Page 50: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 51: Cassandra devoxx 2010

A

LT

W

F

P

YKey “C”

U

Tuesday, November 30, 2010

Page 52: Cassandra devoxx 2010

A

LT

W

F

P

Y

U

Key “C”

Tuesday, November 30, 2010

Page 53: Cassandra devoxx 2010

Tuneable consistency

✤ ONE, QUORUM, ALL

✤ R + W > N

✤ Choose availability vs consistency (and latency)

Tuesday, November 30, 2010

Page 54: Cassandra devoxx 2010

Monitorable

Tuesday, November 30, 2010

Page 55: Cassandra devoxx 2010

JMX

Tuesday, November 30, 2010

Page 56: Cassandra devoxx 2010

Ripcord

Tuesday, November 30, 2010

Page 57: Cassandra devoxx 2010

Data model tradeoffs

✤ Twitter: “Fifteen months ago, it took two weeks to perform ALTER TABLE on the statuses [tweets] table.”

Tuesday, November 30, 2010

Page 58: Cassandra devoxx 2010

A static ColumnFamily

Tuesday, November 30, 2010

Page 59: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 60: Cassandra devoxx 2010

A dynamic ColumnFamily

Tuesday, November 30, 2010

Page 61: Cassandra devoxx 2010

SELECT * FROM tweetsWHERE user_id IN (SELECT follower FROM followers WHERE user_id = ?)

followers

?

tweets

timeline

?

uuid:tweet

Tuesday, November 30, 2010

Page 62: Cassandra devoxx 2010

SuperColumns = full denormalization

Tuesday, November 30, 2010

Page 63: Cassandra devoxx 2010

A little deeper

✤ http://twissandra.com

✤ http://github.com/jhermes/twissjava

Tuesday, November 30, 2010

Page 64: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 65: Cassandra devoxx 2010

Tuesday, November 30, 2010

Page 66: Cassandra devoxx 2010

Mutator<String> m = createMutator("Twissandra", stringExtractor);

MutationResult mr = m.insert(tweetId, "Tweet", createStringColumn("uname", uname))   .insert(tweetId, "Tweet", createStringColumn("body", body));

for (String follower : getFollowers(uname)) { mr.insert(follower, "Timeline", createColumn(timestamp, tweetId, longExtractor, stringExtractor));}

m.execute()

Tuesday, November 30, 2010

Page 67: Cassandra devoxx 2010

SliceQuery<String, String, String> q = createSliceQuery("Twissandra", stringExtractor, stringExtractor, stringExtractor);

q.setColumnFamily("Timeline") .setKey(uname) .setRange(startTimestamp, null, true, 40);

ColumnSlice<String, String> slice = q.execute().get();

Tuesday, November 30, 2010

Page 68: Cassandra devoxx 2010

API cake

✤ libpq

✤ JDBC

✤ JPA

✤ Thrift

✤ Pelops, Hector

✤ Kundera, ?

Tuesday, November 30, 2010

Page 69: Cassandra devoxx 2010

Analytics in Cassandra

✤ @afex: “Cassandra + Pig (Hadoop) is very exciting. A 7 line script to analyze data from my entire cluster transparently, with no ETL? Yes, please”

Tuesday, November 30, 2010

Page 70: Cassandra devoxx 2010

TaskTracker

JobTracker

Tuesday, November 30, 2010

Page 71: Cassandra devoxx 2010

0.7

✤ More control over replica placement

✤ Hadoop refinements

✤ Secondary indexes

✤ Online schema changes

✤ Large row support (> 2GB)

✤ Dynamic routing around slow nodes

Tuesday, November 30, 2010

Page 72: Cassandra devoxx 2010

When do you need Cassandra?

✤ Ian Eure: “If you’re deploying memcache on top of your database, you’re inventing your own ad-hoc, difficult to maintain NoSQL data store”

Tuesday, November 30, 2010

Page 73: Cassandra devoxx 2010

Not Only SQL

✤ Curt Monash: “ACID-compliant transaction integrity commonly costs more in terms of DBMS licenses and many other components of TCO (Total Cost of Ownership) than [scalable NoSQL]. Worse, it can actually hurt application uptime, by forcing your system to pull in its horns and stop functioning in the face of failures that a non-transactional system might smoothly work around. Other flavors of “complexity can be a bad thing” apply as well. Thus, transaction integrity can be more trouble than it’s worth.” [Curt’s emphasis]

Tuesday, November 30, 2010

Page 75: Cassandra devoxx 2010

Questions

Tuesday, November 30, 2010