Dynamo and RiakPresented by Beatriz Aguilar Gallo

1. Introduction Dynamo

• It is highly available key-value storage system that some of Amazon’s core services use to provide an “always-on” experience.

!• Amazon uses a highly decentralized, loosely coupled, service oriented

architecture consisting of hundreds of services.!

• Amazon’s software systems need to handle as the normal case the failures!

• To achieve scalability and availability —> Data is partitioned and replicated (consistent hashing)

2

Introduction | Background | System Architecture | Implementation | Conclusions

1. Introduction Riak

• Riak is a highly available, scalable, open source key/value database.

• Riak offers several query methods in addition to the standard key/value interface

3

Introduction | Background | System Architecture | Implementation | Conclusions

2. Background. Dynamo

Requirements for Dynamo

• Query Model simple read and write operations to a data item. State is stored as binary objects identified by unique keys.

• ACID Properties

• Efficiency —> Stringent latency requirements, 99.9th

percentile of the distribution. (SLAs)

4

Introduction | Background | System Architecture | Implementation | Conclusions

2. Background. RiakRequirements for Riak

• Extended Dynamo's proposed query model in several ways:

1. Standard key/value access

2. MapReduce querying

3. Secondary Indexing

4. Full-text Sear

• No traditional "ACID" semantics around transactions. It's built to be "eventually consistent.

• Efficiency —> Riak database would always be available to serve requests. • (SLA)

5

Introduction | Background | System Architecture | Implementation | Conclusions

2. Background. Dynamo & RiakDesign Considerations Dynamo and Riak

• When to perform the process of resolving update conflicts?

• Amazon services can not reject customers updates —> push complexity of conflict resolution to the reads

• Who performs the process of resolving update conflicts?

• Data store —> limited

• The application —> best suited for client’s experience

!

6

Introduction | Background | System Architecture | Implementation | Conclusions

4. System Architecture. Dynamo

7

• Dynamo stores objects associated with a key through a simple interface —> get() and put().

• The get(key) operation locates the object replicas associated with the key in the storage system

• The put(key, context, object) operation determines where the replicas of the object should be placed based on the associated key ! !

!!!!!

Introduction | Background | System Architecture | Implementation | Conclusions

4. System Architecture. Riak

8

• Partitioning —> Riak uses consistent hashing to distribute data around ring to partitions responsible for storing data

• Replication —> Replication in Riak, like in Dynamo, is fundamental and automatic.

• Versioning —> Riak is an "eventually consistent" database

• Membership —> Riak's ring state holds membership information, and is propagated via gossiping

• Failure Handling and Scaling —> Riak operators can trigger node management via the riak-admin command-line tool.

Introduction | Background | System Architecture | Implementation | Conclusions

5. Implementation. Dynamo• In Dynamo, each storage node has three main software components:

• Request coordination —>build on top of an event-driven messaging substrate. All communications are implemented using Java NIO channels. !!

• Membership and failure Detection

• Local Persistence Engine —> Allows for different storage engines to be plugged in. (Berkeley Database (BDB) Transactional Data Store2, BDB Java Edition, MySQL).

• Main goal: choose the storage engine best suited for an application’s access patterns !!

• Implemented in JAVA

9

Introduction | Background | System Architecture | Implementation | Conclusions

5. Implementation. Riak

• Riak ships with various backend options.

• Bitcask is the default, LevelDB and Main Memory are also used heavily in production (in that order). !

• Implemented in Erlang

10

Introduction | Background | System Architecture | Implementation | Conclusions

6. Conclusions

• The consistency among replicas: quorum-like technique and a decentralized replica synchronization protocol.

• Gossip based distributed failure detection and membership protocol. 

• Decentralized system with minimal need for manual administration. 

• Storage nodes can be added and removed from Dynamo without requiring any manual partitioning or redistribution.

11

• Riak employs consistent hashing to partition and replicate data around the ring

• Riak makes use of gossiping in the same way that Dynamo does

• Nodes can be added and removed from the Riak cluster as needed.

Introduction | Background | System Architecture | Implementation | Conclusions

Dynamo Riak

Dynamo and RiakPresented by Beatriz Aguilar Gallo