Libon cassandra summiteu2014

Billion Records from SQL to Cassandra, lessons learned DuyHai Doan Brice Dutheil

#CassandraSummit @doanduyhai @BriceDutheil

Who are we ?

Brice Dutheil

Mockito Java Track Lead @ Devoxx France Independant contractor @ Libon (Orange-Vallée)

DuyHai Doan

Achilles Cassandra Technical Advocate Former Java Developer @ Libon

2


Agenda •  Libon context

•  Migration strategy

•  Business code migration

•  Data Modeling

•  Take Away

3


Libon Context


What is Libon ? •  Messaging app

•  VOIP (out)

•  Custom voicemail & greetings

•  SMS/chat/file transfer

•  Contacts matching

5


Contact Matching

6

Libon User


Contact Matching

7

Libon User Friend


Contact Matching

8

Libon User Friend

Contact matching


Contact Matching

9

Libon User Friend

Accept link


Project Context •  Application grew over the years

10


Project Context •  Application grew over the years

•  Already using Cassandra to handle events

•  messaging / file sharing / SMS / notifications

•  Cassandra R/W latencies ≈ 0,4 ms

•  server response time under 10 ms

11


Project Context •  About contacts …

12



•  stored as relational model in RDBMS (Oracle)

13




•  1 user ≈ 300 contacts

14





•  with millions users ☞ billions of contacts to handle

15





•  with millions users ☞ billions of contacts to handle

•  query latency unpredictable

16

#CassandraSummit @doanduyhai @BriceDutheil 17


Fixing the problem •  Tune the RDBMS

18



•  indices

19



•  indices

•  partitioning

20



•  indices

•  partitioning

•  less joins, simplified relational model

21



•  indices

•  partitioning


•  hardware capacity increased

22



•  indices

•  partitioning



That worked

23



•  indices

•  partitioning



That worked but …

24


Back-end application

RDBMS Cassandra

25


Next Challenges •  High Availability (DB failure, site failure …)

26



•  Predictable performance at scale

27



•  Predictable performance at scale

•  Going to multi data-centers

28


Going for Cassandra •  Denormalize (if possible …)

29



•  Know your business ☞ know your queries

30




•  Linear scaling out

31




•  Linear scaling out

•  Consistent performance

32


Data Migration Strategy


Objectives •  No downtime

34



•  No concurrency corner-cases

35




•  Safe rollback possible

36




•  Safe rollback possible

•  Replay-ability & resume-ability

37


Strategy •  3 phases

38



•  Write contacts to both data stores

39




•  Old contacts migration

40




•  Old contacts migration

•  Switch to Cassandra …

•  … and deprecate SQL

41


Migration Phase 1 Back end server

· ·

·

SQL SQL SQL

C*

C*

C* C*

C*

Write

contactUUID

42

contactId … contactUUID 129363 123e4567-

e89b-12d3… 834849

contacId(long) + contactUUID



· ·

·

SQL SQL SQL

C*

C*

C* C*

C*

Read

43


Migration Phase 2

SQL SQL SQL

C*

C*

C* C*

C*

For each batch of users SELECT * FROM contacts WHERE user_id = … AND contact_uuid IS NULL

•  On live production, migrate old contacts

44

Old contacts created before phase 1


Migration Phase 2

SQL SQL SQL

C*

C*

C* C*

C*

For each batch of users SELECT * FROM contacts WHERE user_id = … AND contact_uuid IS NULL

Logged batches of INSERT INTO contacts(..) VALUES(…) USING TIMESTAMP now() - 1 week

•  On live production, migrate old contacts

45

Old contacts created before phase 1


Migration Phase 2

USING TIMESTAMP now() - 1 week 😳

46


Migration Phase 2 •  During data migration …

47



•  … concurrent writes from the migration batch …

48



•  … concurrent writes from the migration batch …

•  … and updates from production for the same contact

49


Migration Phase 2

contact_uuid name (now -1 week) … name (now) …

Johny … Johnny …

Insert from batch (to the past)

Update from production

50


Migration Phase 2

contact_uuid name (now -1 week) … name (now) …

Johny … Johnny …

Future reads pick the most up-to-date value

51


Migration Phase 2

"Write to the Past… to save the Future"

Libon – 2014/10/08

52



· ·

·

SQL SQL SQL

C*

C*

C* C*

C*

Write

❌ 53


Business Code Refactoring


Code Inventory •  Written for RDBMS

55



•  Lots of joins (no surprise)

56




•  Designed around transactions

57




•  Designed around transactions

•  Spring @Transactional everywhere

58


Code Inventory cont. •  Entities go through Services & Repositories

59

Repositories

Services

ContactEntity


Code Inventory cont. •  Hibernate is auto-magic

60


Code Inventory cont. •  Hibernate is auto-magic

•  lazy loading

•  1st level cache

•  N+1 select

61

Repositories

Services

ContactEntity


Which options ? •  Throw existing code …

•  … and re-design from scratch for Cassandra

62


Which options ? •  Throw existing code …

•  … and re-design from scratch for Cassandra No way !

63


Code Quality •  Existing business code has…

•  … ≈ 3500 unit tests

64


Code Quality •  Existing business code has…

•  … ≈ 3500 unit tests

•  and ≈600+ integration tests

65


Code Quality •  We are TDD aficionados …

66


Code Quality •  We are TDD aficionados …

•  … and we love our code coverage

67


Code Quality

"The code coverage is one of your most

valuable technical asset" Libon – since beginning

68


Repositories

Services

Refactoring Strategy

ContactMatchingService ContactService ContactSync

ContactEntity

n 1 n n

69


Repositories

Services


ContactMatchingService ContactService

ContactNoSQLEntity

ContactSync

ContactEntity

n 1 n n

70

Proxy


Repositories

Services


ContactMatchingService ContactService

ContactNoSQLEntity

ContactSync

ContactEntity

n 1 n n

Denorm2 … DenormN Denorm1

71

Proxy


Refactoring Strategy •  Use CQRS

•  ContactReadRepository

•  ContactWriteRepository

•  ContactUpdateRepository

•  ContactDeleteRepository

72


Refactoring Strategy •  ContactReadRepository

•  direct sequential read

•  no joins

•  1 read ≈ 1 SELECT

73


Refactoring Strategy •  ContactWriteRepository

•  write to all denormalized tables

•  using CQL logged batches

•  use TTLs

74


Refactoring Strategy •  ContactUpdateRepository

•  read-before-write most of the time 😟

•  rare updates ☞ acceptable perf penalty

75


Refactoring Strategy •  ContactDeleteRepository

•  delete

•  update contact modification date

76


Outcome •  5 months of 2 men work

77



•  Many iterations to fix bugs (thanks to IT)

78




•  Lots of performance benchmarks using Gatling

79


Gatling Output

80





☞ data model & code validation

81





☞ data model & code validation

•  … we are almost there for production

82


Data Model


Denormalization, the good •  Support fast reads

•  1 read ≈ 1 SELECT

•  Worthy because mostly read, few updates

84


Denormalization, the bad •  Updating mutable data can be nightmare

•  Data model bound by existing client-facing API

•  Update paths very error-prone without tests

85


Data model in detail

Contacts_by_id

Contacts_by_identifiers

Contacts_in_profiles

Contacts_by_modification_date

Contacts_by_firstname_lastname

Contacts_linked_user

86



Contacts_by_id






87

user_id always component

of partition key


Scalable design

88

n1

n2

n3

n4

n5

n6

n7

n8

A

B

C

D

E

F

G

H

user_id1

user_id2

user_id3

user_id4

user_id5


Scalable design

89

n1

n2

n3

n4

n5

n6

n7

n8

A

B

C

D

E

F

G

H

user_id1 user_id2

user_id3

user_id4

user_id5


Bloom filters in action

90

•  For some tables, partition key = (user_id, contact_id)

☞ fast look-up, leverages Bloom filters

☞ touches 1 SSTable most of the time



Contacts_by_id






91

Wide partition Bucketed


A "queue" story

92

•  contacts_by_modification_date

•  queue-like pattern 😭


A "queue" story

93

•  contacts_by_modification_date

•  queue-like pattern 😭

☞ buckets to the rescue

user_id:2014-12 date35 date12 … … date47

… … … …

user_id:2014-11 date11 date12 … … date34

… … … …


Data model summary •  7 tables for denormalization

94



•  Normalize some tables because rare access

95



•  Normalize some tables because rare access

•  Read-before write in most update scenarios 😟

96


Notes on contact_id •  In SQL, auto-generated long using sequence

•  In Cassandra, auto-generated timeuuid

97


Notes on contact_id •  How to store both types ?

98



•  As text ? ☞ easy solution …

99



•  As text ? ☞ easy solution …

•  … but waste of space !

•  because encoded as UTF-8 or ASCII in Cassandra

100


Notes on contact_id •  Long ☞ 8 bytes

•  Long as text(UTF-8: 1 byte) ☞ "digits count" bytes

101


Notes on contact_id •  UUID ☞ 16 bytes

•  32 hex chars + 4 hyphens = 36 chars

•  UUID as text(UTF-8: 1 byte) ☞ 36 bytes

•  Bytes overhead = 36 – 16 = 20 bytes

102


Notes on contact_id •  20 bytes wasted per contact uuid

103



•  × 7 denormalizations = 140 bytes per contact uuid

104



•  × 7 denormalizations = 140 bytes per contact uuid

•  × 109 contacts = 140 GB wasted

😠 105

not even counting replication factor …


Notes on contact_id •  ☞ just save contact id as byte[ ]

106



•  Achilles @TypeTransformer for automatic conversion (see later)

107



•  Achilles @TypeTransformer for automatic conversion (see later)

•  Use blobAsBigInt( ) or blobAsUUID( ) to view data

108


Achilles •  Advanced "object mapper"

•  Fluent API

•  Tons of features

•  TDD friendly

109


Achilles •  Dirty checking, why is it important ?

110



•  1 contact ≈ 8 mutable fields

111




•  × 7 denormalizations = 56 update combinations …

112




•  × 7 denormalizations = 56 update combinations …

•  and not even counting multiple fields updates …

113


Achilles •  Are you going to manually generate 56+ prepared

statements for all possible updates ?

114


Achilles •  Are you going to manually generate 56+ prepared

statements for all possible updates ?

•  Or just use dynamic plain string statements and get some perf penalty ?

115


Achilles •  Dirty check in action

//No read-before-write ContactEntity proxy = manager.forUpdate(ContactEntity.class, contactId); proxy.setFirstName(…); proxy.setLastName(…); //type-safe updates proxy.setAddress(…);

manager.update(proxy);

116


Achilles

117

Empty Entity

DirtyMap

Proxy Setters interception

PrimaryKey


Achilles •  Dynamic statements generation

UPDATE contacts SET firstname=?, lastname=?,address=? WHERE contact_id=?

118

prepared statements are cached, of course


Achilles •  Insert strategy, what is it ?

119


Achilles •  Simple INSERT prepared statement

INSERT INTO contacts(contact_id,name,age,address,gender,avatar,…) VALUES(?, ?, ?, ? … ?);

120


Achilles •  Runtime values binding

•  some columns are optional

preparedStatement.bind(49374,’John DOE’,33, null, null, …, null);

121


Achilles

Wait … are you saying inserting null in CQL???

😳

122


Achilles

Inserting null ≡ creating tombstones

123


Achilles

Inserting null ≡ creating tombstones × 7 denormalizations

124


Achilles

Inserting null ≡ creating tombstones × 7 denormalizations

× billions of contacts created

😱 125

not even counting replication factor …


Achilles

@Entity(table = "contacts_by_id ») @Strategy(insert = InsertStrategy.NOT_NULL_FIELDS) public class ContactById {

}

126

•  Simple annotation


Achilles •  Runtime dynamic INSERT statement

INSERT INTO contacts(contact_id, name, age, address,) VALUES(:contact_id, :name, :age, :address);

127

prepared statements are cached, of course


Achilles

@PartitionKey @Column(name = "contact_id") @TypeTransformer(valueCodecClass = ContactIdToBytes.class) private ContactId contactId;

128

•  Remember the contactId ⇄ byte[ ] conversion ?

BYOC ☞ Bring Your Own Codec


Achilles public interface Codec<FROM, TO> { Class<FROM> sourceType(); Class<TO> targetType(); TO encode(FROM fromJava) FROM decode(TO fromCassandra); }

129


Achilles

130

2014-12-01 14:25:20,554 Bound statement : [INSERT INTO contacts.contacts_by_modification_date(user_id,month_bucket,modification_date,...) VALUES (:user_id,:month_bucket,:modification_date,...) USING TTL :ttl;] with CONSISTENCY LEVEL [LOCAL_QUORUM] 2014-12-01 14:25:20,554 bound values : [222130151, 2014-12, e13d0d50-7965-11e4-af38-90b11c2549e0, ...]

2014-12-01 14:25:20,701 Bound statement : [SELECT birthday,middlename,avatar_size,... FROM contacts.contacts_by_modification_date WHERE user_id=:user_id AND month_bucket=:month_bucket AND (modification_date)>=(:modification_date) ORDER BY modification_date ASC;] with CONSISTENCY LEVEL [LOCAL_QUORUM] 2014-12-01 14:25:20,701 bound values : [222130151, 2014-10, be6bc010-6109-11e4-b385-000038377ead]

•  Dynamic logging in action


Achilles

131

•  Dynamic logging

•  runtime activation

•  no need to recompile/re-deploy

•  save us hours of debugging

•  TRACE log level ☞ query tracing


Take Away


Conditions for success •  Data modeling is crucial

133



•  Double-run strategy & timestamp trick FTW

134




•  Data type conversion can be tricky

135





•  Benchmark !

136





•  Benchmark !

•  Mindset shifts for the team

137

Thank You

Technology

Libon cassandra summiteu2014