Couchbase App Development: Documents, schemas and their relationships

1

Developing With Documents: Schemas and Rela7onships

Chris AndersonArchitect

www.couchbase.com/webinars

2

http://www.couchbase.com/webinars

http://www.couchbase.com/webinars

3

What we’ll talk about

• Working with JSON documents• RunBme-‐driven paFerns for• linking between documents• embedding data• fetching mulBple documents

• Map reduce indexing paFerns

4

JSON DOCUMENTS

5

Couchbase Server is a Document Database

hFp://marBnfowler.com/bliki/AggregateOrientedDatabase.html

This synergy between the programming model and the distribution model is very valuable. It allows the database to use its knowledge of how the application programmer clusters the data to help performance across the cluster.

http://martinfowler.com/bliki/AggregateOrientedDatabase.html

http://martinfowler.com/bliki/AggregateOrientedDatabase.html

6

Document Database

• Easy to distribute data• Makes sense to applica5on programmers

o::1001{

uid: ji22jd,customer: Ann,line_items: [

{ sku: 0321293533, quan: 3, unit_price: 48.0 },{ sku: 0321601912, quan: 1, unit_price: 39.0 },{ sku: 0131495054, quan: 1, unit_price: 51.0 }

],payment: { type: Amex, expiry: 04/2001,

last5: 12345 }}

7

JSON Documents

• Maps more closely to external API• CRUD Opera5ons, lightweight schema

• Stored under a unique iden'fier key

myDocument = { “fields” : [“with basic types”, 3.14159, true], “like” : “your favorite language.”, “status”: { “apis” : true, “databases” : “document” }}

client.set(“mydocumentid”, myDocument);mySavedDocument = client.get(“mydocumentid”);

Object to JSON back to Object

8

User Objectstring uid

string firstname

string lastname

int age

array favorite_colors

string email

u::[email protected]{

“uid”: 123456,“firstname”: “jasdeep”,“lastname”: “Jaitla”,“age”: 22,“favorite_colors”: [“blue”, “black”],“email”: “[email protected]”

}


string firstname

string lastname

int age


string email



}

set()

get()


8


string firstname

string lastname

int age


string email



}


string firstname

string lastname

int age


string email



}

set()

get()


8


string firstname

string lastname

int age


string email



}


string firstname

string lastname

int age


string email



}

set()

get()

Meta + Document Body

9

{ "brewery": "New Belgium Brewing", "name": "1554 Enlightened Black Ale", "abv": 5.5, "descrip'on": "Born of a flood...", "category": "Belgian and French Ale", "style": "Other Belgian-‐Style Ales", "updated": "2010-‐07-‐22 20:00:20"}

{ "id" : "beer_Enlightened_Black_Ale”, ...{

Documentuser data,

can be anything

unique ID

Metadataidentifier,

expiration, etc

“vintage” date format from an SQL dump >_<

10

No More ALTER table

• No More alter table• More producBve developers!• Emergent schema—the next session is about views

• This session is about working directly with documents from interacBve applicaBon code

• Schema driven by code

11

LIVE DATA

12

RealBme InteracBve CRUD

• Requirement: high-‐performance with high-‐concurrency and dynamic scale

• Key/value API is the interac've low-‐latency path

14#

Couchbase#Server#2.0#Architecture#

Heartbeat#

Process#m

onito

r#

Glob

al#singleton#supe

rviso

r#

Confi

gura?o

n#manager#

on#each#node#

Rebalance#orchestrator#

Nod

e#he

alth#m

onito

r#

one#per#cluster#

vBucket#state#and

#replica?

on#m

anager#

h"p$

REST$m

anagem

ent$A

PI/W

eb$UI$

HTTP#8091$

Erlang#port#mapper#4369$

Distributed#Erlang#21100$>$21199$

Erlang/OTP$

storage#interface#

Couchbase$EP$Engine$

11210$Memcapable##2.0#

Moxi$

11211$Memcapable##1.0#

Memcached$

$$$$$CouchStore$Persistence$Layer$

8092$Query#API#

Que

ry$Engine$

Couchbase Basic OperaBons

13

• get (key)– Retrieve a document

• set (key, value)– Store a document, overwrites if exists

• add (key, value)– Store a document, error/excepBon if exists

• replace (key, value)– Store a document, error/excepBon if doesn’t exist

• cas (key, value, cas)– Compare and swap, mutate document only if it hasn’t changed while execuBng this operaBon

Couchbase Basic OperaBons (cont’d)

14

Atomic Counters are a special structure in Couchbase, they are executed in order and are PosiBve Integer Values• set (key, value)– Use set to iniBalize the counter• cb.set(“my_counter”, 1)

• incr (key)– Increase an atomic counter value, default by 1• cb.incr(“my_counter”) # now it’s 2

• decr (key)– Decrease an atomic counter value, default by 1• cb.decr(“my_counter”) # now it’s 1

Simple Example in Ruby

15

# user.rbrequire “rubygems”require “couchbase”

class UseraFr_accessor :name, :email, :Btle, :twiFer

def ini7alize(aFr = {}) aFr.each do |name, value|

seFer = "#{name}=" next unless respond_to?(seFer) send(seFer, value)

endend

def saveC = Couchbase.bucketC. set(@email.downcase, self.to_json)

endend

# example.rbrequire “./user.rb”

u1 = User.new({ :email => “[email protected]”,:name => “Jasdeep Jaitla”,:Btle => “Scalability Sherpa”,:twiFer => “@scalabl3

})

u1.save

RunBme Driven Schema

• What’s in the database looks more like your code• Thinking about throughput, latency, update and read paFerns is the new data modeling

• Data flows get more aFenBon than data at rest• Performance Oriented Architecture hFp://jchrisa.net/drl/_design/sofa/_list/post/post-‐page?startkey=%5B%22Performance-‐Oriented-‐Architecture%22%5D

• When should I split a data-‐structure into mulBple documents?• Generally the more useful your document is as a standalone enBty, the beFer.• Documents that grow without bound are bad

16

17

LINK BETWEEN DOCUMENTS

Let’s Add Comments and RaBngs to the Beer

• Challenge linking items together• Whether to grow an exis5ng item or store independent documents

• No transac5onality between documents!

{ "brewery": "New Belgium Brewing", "name": "1554 Enlightened Black Ale", "abv": 5.5, "descrip'on": "Born of a flood...", "category": "Belgian and French Ale", "style": "Other Belgian-‐Style Ales", "updated": "2010-‐07-‐22 20:00:20"}

I give that a 5!

good w/ burgers

tastes like college!

Let’s Add Comments and RaBngs to the Beer

• We’ll put comments in their own document• And add the ra5ngs to the beer document itself.

{ "type": "comment", "about_id": "beer_Enlightened_Black_Ale", "user_id": 525, "text": "tastes like college!", "updated": "2010-‐07-‐22 20:00:20"}

I give that a 5!

{ "brewery": "New Belgium Brewing", "name": "1554 Enlightened Black Ale", "abv": 5.5, "descrip'on": "Born of a flood...", "category": "Belgian and French Ale", "style": "Other Belgian-‐Style Ales", "updated": "2010-‐07-‐22 20:00:20", “ra'ngs” : { “525” : 5, “30” : 4, “1044” : 2 }, “comments” : [ “f1e62”, “6ad8c” ]}

{ "type": "user", "user_id": 525, "name": "Chris", "email": "[email protected]"}


I give that a 5!

{ "brewery": "New Belgium Brewing", "name": "1554 Enlightened Black Ale", "abv": 5.5, "descrip7on": "Born of a flood...", "category": "Belgian and French Ale", "style": "Other Belgian-‐Style Ales", "updated": "2010-‐07-‐22 20:00:20", “ra7ngs” : { “525” : 5, “30” : 4, “1044” : 2 }, “comments” : [ “f1e62”, “6ad8c” ]}

21

Do it: save the comment document

• Set at the id “f1e62”

"type": "comment", "about_id": "beer_Enlightened_Black_Ale", "user_id": 525, "text": "tastes like college!", "updated": "2010-‐07-‐22 20:00:20"

client.set(“f1e62”,{

});

create a new document

{ "id": "f1e62"}

{ "brewery": "New Belgium Brewing", "name": "1554 Enlightened Black Ale", "abv": 5.5, "descrip'on": "Born of a flood...", "category": "Belgian and French Ale", "style": "Other Belgian-‐Style Ales", "updated": "2010-‐07-‐22 20:00:20", “ra'ngs” : { “525” : 5, “30” : 4, “1044” : 2 }, “comment_ids” : [ “f1e62”, “6ad8c” ]}

Link between comments and beers


link to comments

link to beer

{ "id": "f1e62"}

23

How to: look up comments from a beer

• SERIALIZED LOOP

figure hFp://www.ibm.com/developerworks/webservices/library/ws-‐sdoarch/

beer = client.get(“beer:A_cold_one”);beer.comment_ids.each { |id| comments.push(client.get(id));}

• FAST MULTI-‐KEY LOOKUPbeer = client.get(“beer:A_cold_one”);comments = client.multiGet(beer.comment_ids)

• ASYNC VIEW QUERYcomments = client.query(“myapp”,“by_comment_on”, {:key => “beer:A_cold_one”});

http://www.ibm.com/developerworks/webservices/library/ws-sdoarch/


How to: add a raBng to a beer

• Other users are raBngs beers also, so we use a CAS update– we don’t want to accidentally overwrite another users raBng that is being saved at the same Bme as ours

• Best pracBce is to use a lambda so the client can retry

24

cb.cas("mykey") do |doc| doc["ratings"][current_user.id] = my_rating docend

Actor 1 Actor 2

Couchbase Server

CAS mismatch & retry

Success

25

Object Graph With Shared InteracBve Updates

• Challenge: higher level data structures• Objects shared across mul5ple users• Mixed object sets (upda5ng some private and some shared objects)

figure hFp://www.ibm.com/developerworks/webservices/library/ws-‐sdoarch/



Get With Lock (GETL)

• Ozen referred to as “GETL”• PessimisBc concurrency control• Locks have a short TTL• Locks released with CAS operaBons• Useful when working with object graphs

27

Developing with Views:See Inside the Data

J Chris AndersonArchitect

28

What we’ll talk about

• Lifecycle of a view• Index definiBon, build, and query phase• Consistency opBons (async by default)

• Emergent Schema -‐ Views and Documents• PaFerns:• Secondary index• Basic aggregaBons (avg raBngs by brewery)• Time-‐based analyBcs with group_level• Leaderboard

29

VIEW LIFECYCLE:DEFINE -‐ BUILD -‐ QUERY

View DefiniBon (in JavaScript)

like:CREATE INDEX city ON brewery city;

30

Distributed Index Build Phase

• OpBmized for lookups, in-‐order access and aggregaBons• All view reads from disk (different performance profile)• View builds against every document on every node–This is why you should group them in a design document

• AutomaBcally kept up to date

31

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

Doc 9

Doc 7

Doc 8 Doc 6

Doc 3

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

Doc 9

Doc 5

DOC

DOC

DOC

Doc 1

Doc 8 Doc 2

Replica Docs Replica Docs Replica Docs

AcBve Docs AcBve Docs AcBve Docs

32

• Efficiently fetch an row or group of related rows.• Queries use cached values from B-‐tree inner nodes when possible• Take advantage of in-‐order tree traversal with group_level queries

Dynamic Range Queries with OpBonal AggregaBon

Doc 4

Doc 2

Doc 5

SERVER 1

Doc 6

Doc 4

SERVER 2

Doc 7

Doc 1

SERVER 3

Doc 3

Doc 9

Doc 7

Doc 8 Doc 6

Doc 3

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

DOC

Doc 9

Doc 5

DOC

DOC

DOC

Doc 1

Doc 8 Doc 2

Replica Docs Replica Docs Replica Docs

AcBve Docs AcBve Docs AcBve Docs

?startkey=“J”&endkey=“K”{“rows”:[{“key”:“Juneau”,“value”:null}]}

33

EMERGENT SCHEMA

34

Emergent Schema

JSON.org

Github API

Twioer API

"Capture the user's intent"

• Falls out of your key-‐value usage• Helps to know what's efficient• Mostly you can relax

35

QUERY PATTERN:FIND BY ATTRIBUTE

36

Find documents by a specific aFribute

• Lets find beers by brewery_id!

37

The index definiBon

38

The result set: beers keyed by brewery_id

39

QUERY PATTERN:BASIC AGGREGATIONS

40

Use a built-‐in reduce funcBon with a group query

• Lets find average abv for each brewery!

41

We are reducing doc.abv with _stats

42

Group reduce (reduce by unique key)

43

QUERY PATTERN:TIME-‐BASED ROLLUPS

Find paFerns in beer comments by Bme


{ "id": "f1e62"}

7mestamp

Query with group_level=2 to get monthly rollups

45

dateToArray() is your friend

46

• String or Integer based Bmestamps• Output opBmized for group_level queries• array of JSON numbers:

[2012,9,21,11,30,44]

dateToArray()

group_level=2 results

47

• Monthly rollup• Sorted by 5me—sort the query results in your applica4on if you want to rank by value—no chained map-‐reduce

group_level=3 -‐ daily results -‐ great for graphing

48

• Daily, hourly, minute or second rollup all possible with the same index.

• h\p://crate.im/posts/couchbase-‐views-‐reddit-‐data/

http://crate.im/posts/couchbase-views-reddit-data/

http://crate.im/posts/couchbase-views-reddit-data/

49

QUERY PATTERN:LEADERBOARD

50

Aggregate value stored in a document

• Lets find the top-‐rated beers!{ "brewery": "New Belgium Brewing", "name": "1554 Enlightened Black Ale", "abv": 5.5, "descrip'on": "Born of a flood...", "category": "Belgian and French Ale", "style": "Other Belgian-‐Style Ales", "updated": "2010-‐07-‐22 20:00:20", “ra'ngs” : { “jchris” : 5, “scalabl3” : 4, “damienkatz” : 1 }, “comments” : [ “f1e62”, “6ad8c” ]}

ra7ngs

51

Sort each beer by its average raBng

• Lets find the top-‐rated beers!

average

52

WHAT NOT TO WRITE

Most common mistakes

• Reduces that don’t reduce• Trying to do too many things with one view• Emi�ng too much data into a view value• ExpecBng view query performance to be as fast as get/set• Recursive queries require applicaBon code.

53

54

GEOGRAPHIC INDEX

55

Experimental Status

• Not yet using Superstar trees • (only fast on large clusters)

• OpBmized for bulk loading

56

FULL TEXT INDEX

ElasBc Search Adapter

57

Elas7cSearch

• ElasBc Search is good for ad-‐hoc queries and faceted browsing• Our adapter is aware of changing Couchbase topology• Indexed by ElasBc Search azer stored to disk in Couchbase

58

THANK YOU!HTTP://WWW.COUCHBASE.COM

@JCHRIS

http://www.couchbase.com

http://www.couchbase.com

Technology

Couchbase App Development: Documents, schemas and their relationships