Cassandra nice use cases and worst anti patterns

@doanduyhai

Cassandra nice use-cases and worst anti-patterns DuyHai DOAN, Technical Advocate

@doanduyhai

Agenda!

2

Anti-patterns •  Queue-like designs •  CQL null values •  Intensive update on same column •  Design around dynamic schema

@doanduyhai

Agenda!

3

Nice use-cases •  Rate-limiting •  Anti Fraud •  Account validation •  Sensor data timeseries

Worst anti-patterns!

Queue-like designs!CQL null!

Intensive update on same column!Design around dynamic schema!

!

@doanduyhai

Failure level!

5

☠

☠☠

☠☠☠

☠☠☠☠

@doanduyhai

Queue-like designs!

6

Adding new message ☞ 1 physical insert

@doanduyhai

Queue-like designs!

7

Adding new message ☞ 1 physical insert Consuming message = deleting it ☞ 1 physical insert (tombstone)

@doanduyhai

Queue-like designs!

8

Adding new message ☞ 1 physical insert Consuming message = deleting it ☞ 1 physical insert (tombstone) Transactional queue = re-inserting messages ☞ physical insert * <many>

@doanduyhai

Queue-like designs!

9

A

FIFO queue

{ A }

@doanduyhai

Queue-like designs!

10

A B

FIFO queue

{ A, B }

@doanduyhai

Queue-like designs!

11

A B C

FIFO queue

{ A, B, C }

@doanduyhai

Queue-like designs!

12

A B C A

FIFO queue

{ B, C }

@doanduyhai

Queue-like designs!

13

A B C A D

FIFO queue

{ B, C, D }

@doanduyhai

Queue-like designs!

14

A B C A D B

FIFO queue

{ C, D }

@doanduyhai

Queue-like designs!

15

A B C A D B C

FIFO queue

{ D }

@doanduyhai

Queue-like designs!

16

A A A A A A A A A A

FIFO queue, worst case

{ }

@doanduyhai

Failure level!

17

☠☠☠

@doanduyhai

CQL null semantics!

18

Reading null value means •  value does not exist (has never bean created) •  value deleted (tombstone)

SELECT age FROM users WHERE login = ddoan; à NULL

@doanduyhai

CQL null semantics!

19

Writing null means •  delete value (creating tombstone) •  even though it does not exist

UPDATE users SET age = NULL WHERE login = ddoan;

@doanduyhai

CQL null semantics!

20

Seen in production: prepared statement

UPDATE users SET age = ?, … geo_location = ?, mood = ?, … WHERE login = ?;

@doanduyhai

CQL null semantics!

21

Seen in production: bound statement

preparedStatement.bind(33, …, null, null, null, …);

null ☞ tombstone creation on each update …

jdoe age name geo_loc mood status

33 John DOE ý ý ý

@doanduyhai

Failure level!

22

☠

@doanduyhai

Intensive update!

23

Context •  small start-up •  cloud-based video recording & alarm •  internet of things (sensor) •  10 updates/sec for some sensors

@doanduyhai

Intensive update on same column!

24

Data model

sensor_id value

45.0034

CREATE TABLE sensor_data ( sensor_id long, value double, PRIMARY KEY(sensor_id));

@doanduyhai


25

Updates

sensor_id value (t1)

45.0034

UPDATE sensor_data SET value = 45.0034 WHERE sensor_id = …; UPDATE sensor_data SET value = 47.4182 WHERE sensor_id = …; UPDATE sensor_data SET value = 48.0300 WHERE sensor_id = …;


47.4182 sensor_id

value (t36)

48.0300

@doanduyhai


26

Read

SELECT sensor_value from sensor_data WHERE sensor_id = …;

read N physical columns, only 1 useful …


45.0034 sensor_id

value (t13)

47.4182 sensor_id

value (t36)

48.0300

@doanduyhai


27

Solution 1: leveled compaction! (if your I/O can keep up)


45.0034 sensor_id

value (t13)

47.4182 sensor_id

value (t36)

48.0300


48.0300

@doanduyhai


28

Solution 2: reversed timeseries & DateTiered compaction strategy

CREATE TABLE sensor_data ( sensor_id long, date timestamp, sensor_value double, PRIMARY KEY((sensor_id), date)) WITH CLUSTERING ORDER (date DESC);

@doanduyhai


29

Data cleaning by configuration (max_sstable_age_days)

SELECT sensor_value FROM sensor_data WHERE sensor_id = … LIMIT 1;

sensor_id date3(t3) date2(t2) date1(t1) ...

48.0300 47.4182 45.0034 …

@doanduyhai

Failure level!

30

☠☠

@doanduyhai

Design around dynamic schema!

31

Customer emergency call •  3 nodes cluster almost full •  impossible to scale out •  4th node in JOINING state for 1 week •  disk space is filling up, production at risk!

@doanduyhai


32

After investigation •  4th node in JOINING state because streaming is stalled •  NPE in logs

@doanduyhai


33

After investigation •  4th node in JOINING state because streaming is stalled •  NPE in logs Cassandra source-code to the rescue

@doanduyhai


34

public class CompressedStreamReader extends StreamReader { … @Override public SSTableWriter read(ReadableByteChannel channel) throws IOException { … Pair<String, String> kscf = Schema.instance.getCF(cfId); ColumnFamilyStore cfs = Keyspace.open(kscf.left).getColumnFamilyStore(kscf.right);

NPE here

@doanduyhai


35

The truth is •  the devs dynamically drop & recreate table every day •  dynamic schema is in the core of their design Example:

DROP TABLE catalog_127_20140613; CREATE TABLE catalog_127_20140614( … );

@doanduyhai


36

Failure sequence

n1

n2

n4

n3

catalog_x_y

catalog_x_y

catalog_x_y

catalog_x_y

1 4

2

3

5

6

@doanduyhai


37

Failure sequence

n1

n2

n4

n3

catalog_x_y

catalog_x_y

catalog_x_y

catalog_x_y

1 4

2

3

5

6

catalog_x_z

catalog_x_z

catalog_x_z

catalog_x_z

@doanduyhai


38

Failure sequence

n1

n2

n4

n3

1 4

2

3

5

6

catalog_x_z

catalog_x_z

catalog_x_z

catalog_x_z

catalog_x_y ????

@doanduyhai


39

Consequences •  joining node got always stuck •  à cannot extend cluster •  à changing code takes time •  à production in danger (no space left) •  à sacrify analytics data to survive

@doanduyhai


40

Nutshell •  dynamic schema change as normal operations is not recommended •  concurrent schema AND topology change is an anti-pattern

@doanduyhai

Failure level!

41

☠☠☠☠

Q & R

! " !

Nice Examples!

Rate limiting!Anti Fraud!

Account Validation!Sensor Data Timeseries!

@doanduyhai

Rate limiting!

44

Start-up company, reset password feature

1) /password/reset

2) SMS with token A0F83E63DB935465CE73DFE….

Phone number Random token

3) /password/new/<token>/<password>

@doanduyhai

Rate limiting!

45

Problem 1 •  account created with premium phone number

@doanduyhai

Rate limiting!

46

Problem 1 •  account created with premium phone number •  /password/reset x 100

@doanduyhai

Rate limiting!

47

« money, money, money, give money, in the richman’s world » $$$

@doanduyhai

Rate limiting!

48

Problem 2 •  massive hack

@doanduyhai

Rate limiting!

49

Problem 2 •  massive hack •  106 /password/reset calls from few accounts

@doanduyhai

Rate limiting!

50

Problem 2 •  massive hack •  106 /password/reset calls from few accounts •  SMS messages are cheap

@doanduyhai

Rate limiting!

51

Problem 2 •  ☞ but not at the 106/per user/per day scale

@doanduyhai

Rate limiting!

52

Solution •  premium phone number ☞ Google libphonenumber

@doanduyhai

Rate limiting!

53

Solution •  premium phone number ☞ Google libphonenumber •  massive hack ☞ rate limiting with Cassandra

@doanduyhai

Cassandra Time To Live!

54

Time to live •  built-in feature •  insert data with a TTL in sec •  expires server-side automatically •  ☞ use as sliding-window

@doanduyhai

Rate limiting in action!

55

Implementation •  threshold = max 3 reset password per sliding 24h

@doanduyhai

Rate limiting in action!

56

Implementation •  when /password/reset called •  check threshold •  reached ☞ error message/ignore

•  not reached ☞ log the attempt with TTL = 86400

Rate limiting demo

@doanduyhai

Anti Fraud!

58

Real story •  many special offers available •  30 mins international calls (50 countries)

•  unlimited land-line calls to 5 countries •  …

@doanduyhai

Anti Fraud!

59

Real story •  each offer has a duration (week/month/year) •  only one offer active at a time

@doanduyhai

Anti Fraud!

60

Cassandra TTL •  check for existing offer before

SELECT count(*) FROM user_special_offer WHERE login = ‘jdoe’;

@doanduyhai

Anti Fraud!

61

Cassandra TTL •  then grant new offer

INSERT INTO user_special_offer(login, offer_code, …) VALUES(‘jdoe’, ’30_mins_international’,…) USING TTL <offer_duration>;

@doanduyhai

Account Validation!

62

Requirement •  user creates new account •  sends sms/email link with token to validate account •  10 days to validate

@doanduyhai

Account Validation!

63

How to ? •  create account with 10 days TTL

INSERT INTO users(login, name, age) VALUES(‘jdoe’, ‘John DOE’, 33) USING TTL 864000;

@doanduyhai

Account Validation!

64

How to ? •  create random token for validation with 10 days TTL

INSERT INTO account_validation(token, login, name, age) VALUES(‘A0F83E63DB935465CE73DFE…’, ‘jdoe’, ‘John DOE’, 33) USING TTL 864000;

@doanduyhai

Account Validation!

65

On token validation •  check token exist & retrieve user details

SELECT login, name, age FROM account_validation WHERE token = ‘A0F83E63DB935465CE73DFE…’;

•  re-insert durably user details without TTL INSERT INTO users(login, name, age) VALUES(‘jdoe’, ‘John DOE’, 33);

@doanduyhai

Sensor Data Timeseries!

66

Requirements •  lots of sensors (103 – 106) •  medium to high insertion rate (0.1 – 10/secs) •  keep good load balancing •  fast read & write

@doanduyhai

Bucketing!

67

CREATE TABLE sensor_data ( sensor_id text, date timestamp, raw_data blob, PRIMARY KEY(sensor_id, date));

sensor_id date1 date2 date3 date4 …

blob1 blob2 blob3 blob4 …

@doanduyhai

Bucketing!

68

Problems: •  limit of 2.109 physical columns •  bad load balancing (1 sensor = 1 node) •  wide row spans over many files

sensor_id date1 date2 date3 date4 …


@doanduyhai

Bucketing!

69

Idea: •  composite partition key: sensor_id:date_bucket •  tunable date granularity: per hour/per day/per month …

CREATE TABLE sensor_data ( sensor_id text, date_bucket int, //format YYYYMMdd date timestamp, raw_data blob, PRIMARY KEY((sensor_id, date_bucket), date));

@doanduyhai

Bucketing!

70

sensor_id:2014091014 date1 date2 date3 date4 …


Idea: •  composite partition key: sensor_id:date_bucket •  tunable date granularity: per hour/per day/per month …



Buckets

@doanduyhai

Bucketing!

71

Advantage: •  distribute load: 1 bucket = 1 node •  limit partition width (max x columns per bucket)

Buckets





@doanduyhai

Bucketing!

72

But how can I select raw data between 14:45 and 15:10 ?

14:45 à ?

15:00 à 15:10





@doanduyhai

Bucketing!

73

SELECT * FROM sensor_data WHERE sensor_id = xxx AND date_bucket IN (2014091014 , 2014091015) AND date >= ‘2014-09-10 14:45:00.000‘ AND date <= ‘2014-09-10 15:10:00.000‘

Solution •  use IN clause on partition key component •  with range condition on date column ☞ date column should be monotonic function (increasing/decreasing)

@doanduyhai

Bucketing Caveats!

74

IN clause for #partition is not silver bullet ! •  use scarcely •  keep cardinality low (≤ 5)

n1

n2

n3

n4

n5

n6

n7

n8

coordinator

sensor_id:2014091014


@doanduyhai

Bucketing Caveats!

75

IN clause for #partition is not silver bullet ! •  use scarcely •  keep cardinality low (≤ 5) •  prefer // async queries •  ease of query vs perf

n1

n2

n3

n4

n5

n6

n7

n8

n1 Async client



Q & R

! " !

@doanduyhai

Cassandra developers!

77

If you don’t know, ask for help (me, Cassandra ML, PlanetCassandra, stackoverflow, …)

Rule n°1 !

@doanduyhai


78

Do not blind-guess troubleshooting alone in production

(ask for help, see rule n°1)

Rule n°2 !

@doanduyhai


79

Share with the community (your best use-cases … and worst failures)

Rule n°3 !

http://planetcassandra.org/

Thank You @doanduyhai

[email protected]

Technology

Cassandra nice use cases and worst anti patterns