Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences

Riak on Drugs(and the other way around)

Kresten Krab Thorup, Jørn LarsenTrifork

@drkrab

onsdag den 12. oktober 2011

In this talk...

2

■ About Common Medicine Card

■ Building a Decentralized Architecture

■ Conclusion


A Medicine Card

3

■ For a person

■ List of current drug treatments

■ With prescriptions and related events


Common Medicine Card

4


Common Medicine Card

5


6

15-30 existing systems+150k users

SOAP


7

Master

MySQL

Slave

MySQL

Front Endweb services/app



Architecture


Overload

8onsdag den 12. oktober 2011

Database

9

Prescriptions Replica

Treatments+ Events

Front Ends

SOAP endpointWeb applicationBusiness Logic

Prescription Service

Front Ends



Distributed Architecture

10

■ Availability: Run in multiple data centers

■ Scalability: Prepare the system forexpected growth


Riak Data Store

■ Fit the general requirements

■ Availability + Scalability

■ Operational improvements

■ Challenges

■ Key/Value Store, vs Relational Model

■ New technology, many unknowns

11


12


12


12


12


12


12


13


13


13


14


14

coordinate


15

sync


16

scalable and available

system captures write conflicts

resolve lazily (read repair)


Database

17

Prescriptions Replica

Drug Treatments

Front Ends



Front Ends



MySQL

Master/Slave

18

Front End



Riak

4 nodes x2 data centers

Front Ends



Challenges

19

■ Data model: To go from Relational model to Key/Value model

■ Experiences with Riak’s backends

■ A true war story


Data Model

■ Integrity without transactions

■ Riak’s default storage keeps all keys in memory

■ Dealing with Write Conflicts

20


First Attempt: Using Links

■ Ran very poorly on Virtual Hardware

■ Trying to figure out how to handle conflicts

21

PersonKey: Person-IDLinks: Prescription-ID

PrescriptionKey: Prescription-IDContent: Protobuf+GZip

~5 million ~200 million


Second Take

■ Very simple: read - resolve - modify - write

■ Integrity: 1 person ⬌ 1 record

■ Performance good: 1 disk access

■ All keys fit in memory22

PrescriptionsKey: Person-IDContent: Medicine Record

~5 million


Read Repair

■ On every read, we handle write conflicts

■ If so, auto-merge[*], store and re-read

■ Resolve: Merging is business logic; some merge actions need user attention, others don’t.

23


Conclusions

■ 1 key per person 500MB in memory

■ Availability, hot replace of HW

■ Continuous Deployment

■ 10x in performance gain and Scalable

■ 2500 TPS with 10% load of nodes

■ Fully deployed 5000 TPS needed

■ Dramatic reduction in cost of HW and Licences24


Thank You.

25

@drkrab


Documents

Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences