CouchDB: the last RESTful JSON store you'll ever need

CouchDB: the last RESTful JSON service you’ll ever needCarlo Cabanilla

CouchDB is a RESTful JSON store

CouchDB is a RESTful JSON store

Create

> PUT /mydb/stuff> Content-type: application/json>> {"thing_count":1}

Create

> PUT /mydb/stuff> Content-type: application/json>> {"thing_count":1}

< HTTP/1.1 201 Created<< {"_id":"stuff", < "_rev":"1-e86a94c1...", < "thing_count":1}

Create

Read

> GET /mydb/stuff

Read

> GET /mydb/stuff

< HTTP/1.1 200 OK<< {"_id":"stuff", < "_rev":"1-e86a94c1...", < "thing_count":1}

Read

Update

> PUT /mydb/stuff> Content-type: application/json>> {"_id":"stuff", > "_rev":"1-e86a94c1...", > "thing_count":13}

Update

> PUT /mydb/stuff> Content-type: application/json>> {"_id":"stuff", > "_rev":"1-e86a94c1...", > "thing_count":13}

Update

< HTTP/1.1 201 Created<< {"_id":"stuff", < "_rev":"2-39a9bf12...", < "thing_count":13}

Delete

> DELETE /mydb/stuff?rev=2-39a9bf12...

Delete

> DELETE /mydb/stuff?rev=2-39a9bf12...

Delete

< HTTP/1.1 200 OK

a RESTful JSON store


PUT /look/ma/im/doing/rest

PUT /look/ma/im/doing/rest

Do you support Conditional GET?

Conditional GET is the

holy grail of

REST

Conditional GET is the holy grail of REST

> GET /that/money


> GET /that/money

< HTTP/1.1 200 OK<< {“that_money”: “$5”}



> GET /that/money


> GET /that/money

(wait for a few ms)


> GET /that/money


(wait for a few ms)


( )> GET /that/money


(wait for a few ms)

x 1,000,000


( )> GET /that/money


(wait for a few ms)

x 1,000,000 = slow!



> GET /that/money


> GET /that/money

(Uncached: slow)


> GET /that/money

< HTTP/1.1 200 OK< Etag: "1-24c87859"<< {“that_money”: “$5”}

(Uncached: slow)


> GET /that/money

< HTTP/1.1 200 OK< Etag: "1-24c87859"<< {“that_money”: “$5”}

(Uncached: slow)



> GET /that/money> If-None-Match: "1-24c87859"





(Cache hit: fast)



< HTTP/1.1 304 Not Modified< Etag: "1-24c87859"< Content-Length: 0

(Cache hit: fast)






(Data changed so cache miss )



< HTTP/1.1 200 Ok< Etag: "2-535d9fb2"< < {“that_money”: “$1,000,000”}

(Data changed so cache miss )




GET /my_db/some_id

GET /my_db/some_id

{

GET /my_db/some_id

{ "_id": "some_id",

GET /my_db/some_id

{ "_id": "some_id", "_rev": "1-24c8785964763c21d...",

GET /my_db/some_id

{ "_id": "some_id", "_rev": "1-24c8785964763c21d...", "my_field1": "json does strings",

GET /my_db/some_id

{ "_id": "some_id", "_rev": "1-24c8785964763c21d...", "my_field1": "json does strings", "numbers too": 123.12,

GET /my_db/some_id

{ "_id": "some_id", "_rev": "1-24c8785964763c21d...", "my_field1": "json does strings", "numbers too": 123.12, "and arrays": ["ain't", "it", "cool?"],

GET /my_db/some_id

{ "_id": "some_id", "_rev": "1-24c8785964763c21d...", "my_field1": "json does strings", "numbers too": 123.12, "and arrays": ["ain't", "it", "cool?"], "dicts too!": { "don't":"try",

GET /my_db/some_id

{ "_id": "some_id", "_rev": "1-24c8785964763c21d...", "my_field1": "json does strings", "numbers too": 123.12, "and arrays": ["ain't", "it", "cool?"], "dicts too!": { "don't":"try", "this":"in",

GET /my_db/some_id

{ "_id": "some_id", "_rev": "1-24c8785964763c21d...", "my_field1": "json does strings", "numbers too": 123.12, "and arrays": ["ain't", "it", "cool?"], "dicts too!": { "don't":"try", "this":"in", "Oracle!":null }

GET /my_db/some_id

{ "_id": "some_id", "_rev": "1-24c8785964763c21d...", "my_field1": "json does strings", "numbers too": 123.12, "and arrays": ["ain't", "it", "cool?"], "dicts too!": { "don't":"try", "this":"in", "Oracle!":null }}

Schema-less

How do I know what a document means??

Duck-type your database


Duck-typing

def document_factory(doc):


Duck-typing

def document_factory(doc): if doc['look'] == Duck.look()


Duck-typing

def document_factory(doc): if doc['look'] == Duck.look() and doc['swim'] == Duck.swim()


Duck-typing

def document_factory(doc): if doc['look'] == Duck.look() and doc['swim'] == Duck.swim() and doc['quack'] == Duck.quack():


Duck-typing

def document_factory(doc): if doc['look'] == Duck.look() and doc['swim'] == Duck.swim() and doc['quack'] == Duck.quack(): return Duck(doc)


Duck-typing


Python



Python

def document_factory(doc): if doc['type'] == 'Duck':


Python

def document_factory(doc): if doc['type'] == 'Duck': return Duck(doc)


Python

def document_factory(doc): if doc['type'] == 'Duck': return Duck(doc) elif doc['type'] == 'Dog':


Python

def document_factory(doc): if doc['type'] == 'Duck': return Duck(doc) elif doc['type'] == 'Dog': return Dog(doc)


Python

def document_factory(doc): if doc['type'] == 'Duck': return Duck(doc) elif doc['type'] == 'Dog': return Dog(doc) elif ...:


Python

def document_factory(doc): if doc['type'] == 'Duck': return Duck(doc) elif doc['type'] == 'Dog': return Dog(doc) elif ...: ...


Python

def document_factory(doc): if doc['type'] == 'Duck': return Duck(doc) elif doc['type'] == 'Dog': return Dog(doc) elif ...: ... else:


Python

def document_factory(doc): if doc['type'] == 'Duck': return Duck(doc) elif doc['type'] == 'Dog': return Dog(doc) elif ...: ... else: return GenericDocument(doc)


Python


Python



Python

def document_factory(doc): import document_types


Python

def document_factory(doc): import document_types doc_class = getattr(doc, doc['type'],


Python

def document_factory(doc): import document_types doc_class = getattr(doc, doc['type'], 'GenericDocument')


Python

def document_factory(doc): import document_types doc_class = getattr(doc, doc['type'], 'GenericDocument') return doc_class(doc)


Python


Java

public static CouchObj documentFactory(CouchDoc doc) {


Java

public static CouchObj documentFactory(CouchDoc doc) { ClassLoader loader = CouchObj.class.getClassLoader();


Java

public static CouchObj documentFactory(CouchDoc doc) { ClassLoader loader = CouchObj.class.getClassLoader(); dispatcherClass = loader.loadClass(doc.get("type"));


Java

public static CouchObj documentFactory(CouchDoc doc) { ClassLoader loader = CouchObj.class.getClassLoader(); dispatcherClass = loader.loadClass(doc.get("type")); return (CouchObj) dispatcherClass.newInstance(doc);


Java

public static CouchObj documentFactory(CouchDoc doc) { ClassLoader loader = CouchObj.class.getClassLoader(); dispatcherClass = loader.loadClass(doc.get("type")); return (CouchObj) dispatcherClass.newInstance(doc);}


Java

That’s silly. How do I query something that has no schema?

That’s silly. How do I query something that has no schema?

You write a map reduce function in Javascript!

What the heck is map reduce (in CouchDB)?


What the heck is map reduce (in CouchDB)?

map reduce ≈ programmatically building an index



In CouchDB, a map/reduce pair is called a view.


In CouchDB, a map/reduce pair is called a view.(different from a RDBMS view!)



On creation => full database scan then cacheOn db write => incrementally update the cache



On creation => full database scan then cacheOn db write => incrementally update the cache

(different from Hadoop map reduce!)



You can tell CouchDB to store any data in the view


You can tell CouchDB to store any data in the view(RDBMSs only store indexed values and row ids)


You can tell CouchDB to store any data in the view(RDBMSs only store indexed values and row ids)

CouchDB view ≈ RDBMS index + materialized view


Yo, data structures nerds,



a CouchDB view is a B+ tree



a CouchDB view is a B+ treemap function determines how keys map to values




keys are stored in sorted order





O(log n) for reads, writes, deletes and range queries





O(log n) for reads, writes, deletes and range queries

keys are stored close to values



For example:

Create an view called people on the names of all person documents.


function map(doc) { if (doc['type'] == 'person') { emit(doc['name'], doc); }}

For example:



function map(doc) { if (doc['type'] == 'person') { emit(doc['name'], doc); }}

function reduce(keys, values) { return values.length; }

For example:




SELECT count(*) FROM people



GET /people


SELECT name, count(*) FROM people GROUP BY name ORDER BY name ASC


GET /people



GET /people?group=true


GET /people





GET /people

SELECT * FROM people ORDER BY name ASC





GET /people

SELECT * FROM people ORDER BY name ASC

GET /people?reduce=false



SELECT * FROM people WHERE name = “bjorn”



GET /people?reduce=false&key=”bjorn”




SELECT * FROM ( SELECT p.*, rownum rn FROM people p ORDER BY name ASC) WHERE rn BETWEEN 50 AND 60




SELECT * FROM ( SELECT p.*, rownum rn FROM people p ORDER BY name ASC) WHERE rn BETWEEN 50 AND 60

GET /people?reduce=false&limit=10&skip=50


What about joins??????



j/k.

Joins will be in version 0.11 (trunk)


j/k.

Joins will be in version 0.11 (trunk)

function map(doc) { for (i in doc['comment_ids']) { var comment_id = doc['comment_ids'][i]; emit(doc['_id'], {'_id': comment_id }); }}


[ {'_id': 1, 'comment_ids': [3, 4]}, {'_id': 2, 'comment_ids': [5]}, {'_id': 3, 'text': 'whoa'}, {'_id': 4, 'text': 'omg'}, {'_id': 5, 'text': 'ponies'}]

Documents in my blog db




GET /myblog/comments? reduce=false&include_docs=true




GET /myblog/comments? reduce=false&include_docs=true

[ {'key': 1, 'value': {'_id': 3, 'text': 'whoa'}}, {'key': 1, 'value': {'_id': 4, 'text': 'omg'}}, {'key': 5, 'value': {'_id': 5, 'text': 'ponies'}}]

Ok great, so it can pretty much do what a RDBMS can do.

Ok great, so it can pretty much do what a RDBMS can do.

Why not just use a RDBMS?

The real question you should be asking yourself is:

Why keep rewriting HTTP protocol semantics and data object accessorsto front your RDBMS?

The real question you should be asking yourself is:

Middle tier-less web apps!(well, kind of)

No, forreals,

No, forreals,

Why not just use a RDBMS?

ORACLE

OMGReallyAsininelyCostlyLicencingExpenses

ORACLE

ClusterOf Unreliable Commodity Hardware DataBase

ClusterOf Unreliable Commodity Hardware DataBase

?

Cluster?

Replication!

Cluster?

Replication!

Incremental,

Cluster?

Replication!

Incremental,Fault Tolerant,

Cluster?

Replication!

Incremental,Fault Tolerant,Over HTTP

replication is expensive

Mmm, yeah how much did we pay for the LSB?


Mmm, yeah how much did we pay for the LSB?


Seems to work well tho(fingers crossed)

Postgres replication is . . . ?

No native replication support yet (coming in v9.0)


No native replication support yet (coming in v9.0)


3rd party solutions are trigger-based (yuck)

Replication!

Managing replication conflicts (abends)

Replication!


Deterministically pick a version to “win”

Replication!



Add a “\_conflict” property to the document

Replication!




App must resolve the conflict

Replication!


Much better explanation here:http://books.couchdb.org/relax/reference/conflict-management



App must resolve the conflict

http://books.couchdb.org/relax/reference/conflict-management




Let’s see some appz!

• Design philosophy

• Transforming views with list functions

• Lucene integration

• Performance benchmarks

• Erlang under the hood

• Many to many relationships

• Changes feed

• Potential applications at WGen

• Elaborate on middle tier-less web apps

Not enough time :(

http://couchdb.apache.org/

http://couchdb.apache.org

http://couchdb.apache.org