SCALING COUCHDB WITH BIGCOUCH

a. kocoloski

O’Reilly WebcastOctober 2010

2

• Introductions and plugs

• Brief intro to CouchDB

• BigCouch Overview

• Building and Using a CouchDB Cluster

• BigCouch Internals

• Future Plans and How you can Help

OUTLINE

3

INTRODUCTIONS

Cloudant CTOCouchDB Committer Since 2008

PhD physics MIT 2010

BigCouchPutting the “C” back in CouchDB

Open Core: 2 years Development

Contactadam@cloudant.com

kocolosk in #cloudant or #couchdb@kocolosk

Hashtag of the Day: #bigcouch

4

PLUGS

Hosting, Management, Support for CouchDB and BigCouch

http://cloudant.com

5

COUCHDB IN A SLIDE

• Schema-free document database management systemDocuments are JSON objectsAble to store binary attachments

• RESTful APIhttp://wiki.apache.org/couchdb/reference

• Views: Custom, persistent representations of your dataIncremental MapReduce with results persisted to diskFast querying by primary key (views stored in a B-tree)

• Bi-Directional ReplicationMaster-slave and multi-master topologies supportedOptional ‘filters’ to replicate a subset of the dataEdge devices (mobile phones, sensors, etc.)

• Futon Web Interface

6

WHY BIGCOUCH?

“CouchDB is built of the web... This is what software of the future looks

like” -J. Kaplan-Moss

CouchDB is Awesome

...But somewhat incompleteClusterOfUntrustedCommodityHardware

“CouchDB is not a distributed database” -J. Ellis

“Without the Clustering, it’s just OuchDB”

7

WHAT WE TALK ABOUT WHEN WE TALK ABOUT SCALING

• Horizontal scaling: more servers creates more capacity

• Transparent to the application: adding more capacity should not a!ect the business logic of the application.

• No single point of failure.

http://adam.heroku.com/past/2009/7/6/sql_databases_dont_scale/

Pseudo Scalars

8

BIGCOUCH = COUCH+SCALING• Horizontal Scalability

Easily add storage capacity by adding more serversComputing power (views, compaction, etc.) scales with more servers

• No SPOFAny node can handle any requestIndividual nodes can come and go

• Transparent to the ApplicationAll clustering operations take place “behind the curtain”‘looks’ like a single server instance of Couch, just with more awesomeasterisks and caveats discussed later

9

GRAPHICAL REPRESENTATION

hash(blah) = E

Load Balancer

PUT http://kocolosk.cloudant.com/dbname/blah?w=2

N=3W=2R=2

Node 1

A B C DNode 2B

CD

E

Node 3

C

D

E

F

Node 4

D

E

F

G

Node 24

XY

ZA

• Clustering in a ring (a la Dynamo)

• Any node can handle a request

• O(1) lookup• Quorum system (N, R, W)• Views distributed like

documents• Distributed Erlang• Masterless

10

• Shopping List3 networked macbook prosUsual CouchDB DependenciesBigCouch Code

• http://github.com/cloudant/bigcouch

BUILDING YOUR FIRST CLUSTER

11

BUILDING YOUR FIRST CLUSTER

foo.example.com bar.example.com baz.example.com

Build and Start BigCouch

Pick one node and add the others to the local “nodes” DB

Make sure they all agree on the magic cookie (rel/etc/vm.args)

12

QUORUM: IT’S YOUR FRIEND• BigCouch databases are governed by 4 parameters

Q: Number of shardsN: Number of redundant copies of each shardR: Read quorum constantW: Write quorum constant(NB: Also consider the number of nodes in a cluster)

For the next few examples, consider a 5

node cluster

1

2

34

5

13

Q• Q: The number of shards over which a DB will be spread

consistent hashing space divided into Q piecesSpecified at DB creation timepossible for more than one shard to live on a nodeDocuments deterministically mapped to a shard

Q=1 Q=4

1

2

34

5

14

N• N: The number of redundant copies of each document

Choose N>1 for fault-tolerant clusterSpecified at DB creationEach shard is copied N timesRecommend N>2

1

2

34

5

N=3

15

W• W: The number of document copies that must be saved

before a document is “written”W must be less than or equal to NW=1, maximize throughputW=N, maximize consistencyAllow for “201” created responseCan be specified at write time

1

2

34

5

W=2‘201 Created’

16

R• R: The number of identical document copies that must be

read before a read request is okR must be less than or equal to NR=1, minimize latencyR=N, maximize consistencyCan be specified at query timeInconsistencies are automatically repaired

1

2

34

5

R=2

VIEWS• So far, so good, but what about secondary indexes?

Views are built locally on each node, for each DB shardMergesort at query time using exactly one copy of each shardRun a final rereduce on each row if a the view has a reduce

• _changes feed works similarly, but hasno global orderingSequence numbers convertedto strings to encode moreinformation

17

1

2

34

5

18

HACKER PORTIONThe BigCouch Stack

CHTTPD

Fabric

Rexi Mem3

Embedded CouchDBMochiweb, Spidermonkey, etc.

19

MEM3

CHTTPD

Fabric

Rexi Mem3

Embedded CouchDB

• Maintains the shard mapping for each clustered database in a node-local CouchDB database

• Changes in the node registration and shard mapping databases are automatically replicated to all cluster nodes

20

REXI

CHTTPD

Fabric

Rexi Mem3

Embedded CouchDB

• BigCouch makes a large number of parallel RPCs

• Erlang RPC library not designed for heavy parallelismpromiscuous spawning of processesresponses directed back through single process on remote noderequests block until remote ‘rex’ process is monitored

• Rexi removes some of the safeguardsin exchange for lower latenciesno middlemen on the local node remote process responds directly to clientremote process monitoring occursout-of-band

21

FABRIC / CHTTPD

CHTTPD

Fabric

Rexi Mem3

Embedded CouchDB

• FabricOTP library application (no processes) responsible for clustered versions of CouchDB core API callsQuorum logic, view merging, etc.Provides a clean Erlang interface to BigCouchNo HTTP awareness

• ChttpdCut-n-paste of couch_httpd, but usingfabric for all data access

22

SUMMARY

• BigCouch: putting the ‘C’ back in CouchDB

• http://github.com/cloudant/bigcouch

• https://cloudant.com/future