Upload
couchbase
View
1.230
Download
2
Embed Size (px)
Citation preview
Couchbase Server 2.0Introduction
Couchbase Open Source Project
•Leading NoSQL database project focused on distributed database technology and surrounding ecosystem
•Supports both key-value and document-oriented use cases
•All components are available under the Apache 2.0 Public License
•Obtained as packaged software in both enterprise and community editions.
Couchbase Open Source Project
Easy Scalability
Consistent High Performance
Always On
24x365
Grow cluster without application changes, without downtime with a single click
Consistent sub-millisecond read and write response times
with consistent high throughput
No downtime for software upgrades, hardware maintenance, etc.
Flexible Data Model
JSON document model with no fixed schema.
Couchbase Server Core Principles
New in 2.0
JSON support Indexing and Querying
Cross Datacenter ReplicationIncremental Map Reduce
JSONJSONJSON
JSONJSON
• Database stays online, no downtime for schema changes• Records can have different structures, there is no fixed schema• Model Data easier and faster -- agile rapid application
development
{ “ID”: 1, “FIRST”: “Dipti”, “LAST”: “Borkar”, “ZIP”: “94040”, “CITY”: “MV”, “STATE”: “CA”
}JSON
JSONJSON JSON
Flexible Data Model
Architecture
Couchbase Server 2.0 Architecture
Hea
rtbe
at
Proc
ess
mon
itor
Glo
bal s
ingl
eton
sup
ervi
sor
Confi
gura
tion
man
ager
on each node
Reba
lanc
e or
ches
trat
or
Nod
e he
alth
mon
itor
one per cluster
vBuc
ket s
tate
and
repl
icati
on m
anag
er
httpRE
ST m
anag
emen
t API
/Web
UI
HTTP8091
Erlang port mapper4369
Distributed Erlang21100 - 21199
Erlang/OTP
storage interface
Couchbase EP Engine
11210Memcapable 2.0
Moxi
11211Memcapable 1.0
Memcached
New Persistence Layer
8092Query API
Que
ry E
ngin
e
Data Manager Cluster Manager
Couchbase Server 2.0 Architecture
Hea
rtbe
at
Proc
ess
mon
itor
Glo
bal s
ingl
eton
sup
ervi
sor
Confi
gura
tion
man
ager
on each node
Reba
lanc
e or
ches
trat
or
Nod
e he
alth
mon
itor
one per cluster
vBuc
ket s
tate
and
repl
icati
on m
anag
er
http
REST
man
agem
ent A
PI/W
eb U
I
HTTP8091
Erlang port mapper4369
Distributed Erlang21100 - 21199
Erlang/OTP
storage interface
Couchbase EP Engine
11210Memcapable 2.0
Moxi
11211Memcapable 1.0
Built-in Cache
New Persistence Layer
8092Query API
Que
ry E
ngin
e
Couchbase Operations
3333 22
Single node - Couchbase Write Operation
Managed Cache
Dis
k Q
ueue
Disk
Replication Queue
App Server
Couchbase Server Node
Doc 1Doc 1
Doc 1
To other node
3333 22
Single node - Couchbase Update Operation
Managed Cache
Dis
k Q
ueue
Replication Queue
App Server
Doc 1’
Doc 1
Doc 1’Doc 1
Doc 1’
Disk
To other node
Couchbase Server Node
GET
Doc
1
3333 22
Single node - Couchbase Read Operation
Dis
k Q
ueue
Replication Queue
App Server
Doc 1
Doc 1Doc 1
Managed Cache
Disk
To other node
Couchbase Server Node
3333 22
Single node - Couchbase Cache Eviction2
Dis
k Q
ueue
Replication Queue
App Server
Couchbase Server Node
Doc 1
Doc 6Doc 5Doc 4Doc 3Doc 2
Doc 1
Doc 6 Doc 5 Doc 4 Doc 3 Doc 2
Managed Cache
Disk
To other node
3333 22
Single node – Couchbase Cache Miss2
Dis
k Q
ueue
Replication Queue
App Server
Couchbase Server Node
Doc 1
Doc 3Doc 5 Doc 2Doc 4
Doc 6 Doc 5 Doc 4 Doc 3 Doc 2
Doc 4
GET
Doc
1
Doc 1
Doc 1
Managed Cache
Disk
To other node
Couchbase Server Cluster
Cluster-wide Basic Operation
• Docs distributed evenly across servers
• Each server stores both active and replica docsOnly one server active at a time
• Client library provides app with simple interface to database
• Cluster map provides map to which server doc is onApp never needs to know
• App reads, writes, updates docs
• Multiple app servers can access same document at same time
User Configured Replica Count = 1
READ/WRITE/UPDATE
Active
Doc 5
Doc 2
Doc
Doc
Doc
Server 1
Active
Doc 4
Doc 7
Doc
Doc
Doc
Server 2
Doc 8
Active
Doc 1
Doc 2
Doc
Doc
Doc
REPLICA
Doc 4
Doc 1
Doc 8
Doc
Doc
Doc
REPLICA
Doc 6
Doc 3
Doc 2
Doc
Doc
Doc
REPLICA
Doc 7
Doc 9
Doc 5
Doc
Doc
Doc
Server 3
Doc 6
App Server 1
COUCHBASE Client LibraryCOUCHBASE Client Library
Cluster Map
COUCHBASE Client LibraryCOUCHBASE Client Library
Cluster Map
App Server 2
Doc 9
Add Nodes to Cluster
• Two servers addedOne-click operation
• Docs automatically rebalanced across clusterEven distribution of docsMinimum doc movement
• Cluster map updated
• App database calls now distributed over larger number of servers
REPLICA
Active
Doc 5
Doc 2
Doc
Doc
Doc 4
Doc 1
Doc
Doc
Server 1
REPLICA
Active
Doc 4
Doc 7
Doc
Doc
Doc 6
Doc 3
Doc
Doc
Server 2
REPLICA
Active
Doc 1
Doc 2
Doc
Doc
Doc 7
Doc 9
Doc
Doc
Server 3 Server 4 Server 5
REPLICA
Active
REPLICA
Active
Doc
Doc 8 Doc
Doc 9 Doc
Doc 2 Doc
Doc 8 Doc
Doc 5 Doc
Doc 6
READ/WRITE/UPDATE READ/WRITE/UPDATE
App Server 1
COUCHBASE Client LibraryCOUCHBASE Client Library
Cluster Map
COUCHBASE Client LibraryCOUCHBASE Client Library
Cluster Map
App Server 2
Couchbase Server Cluster
User Configured Replica Count = 1
Fail Over Node
REPLICA
Active
Doc 5
Doc 2
Doc
Doc
Doc 4
Doc 1
Doc
Doc
Server 1
REPLICA
Active
Doc 4
Doc 7
Doc
Doc
Doc 6
Doc 3
Doc
Doc
Server 2
REPLICA
Active
Doc 1
Doc 2
Doc
Doc
Doc 7
Doc 9
Doc
Doc
Server 3 Server 4 Server 5
REPLICA
Active
REPLICA
Active
Doc 9
Doc 8
Doc Doc 6 Doc
Doc
Doc 5 Doc
Doc 2
Doc 8 Doc
Doc
• App servers accessing docs
• Requests to Server 3 fail
• Cluster detects server failedPromotes replicas of docs to activeUpdates cluster map
• Requests for docs now go to appropriate server
• Typically rebalance would follow
Doc
Doc 1 Doc 3
App Server 1
COUCHBASE Client LibraryCOUCHBASE Client Library
Cluster Map
COUCHBASE Client LibraryCOUCHBASE Client Library
Cluster Map
App Server 2
User Configured Replica Count = 1
Couchbase Server Cluster
DEMO
Indexing and Querying – The basics
• Define materialized views on JSON documents and then query across the data set
• Using views you can define• Primary indexes • Simple secondary indexes (most common use case)• Complex secondary, tertiary and composite indexes• Aggregations (reduction)
• Indexes are eventually indexed
• Queries are eventually consistent
• Built using Map/Reduce technology • Map and Reduce functions are written in Javascript
3333 22
Eventually indexed Views – Data flow2
Managed Cache
Dis
k Q
ueue
Disk
Replication Queue
App Server
Couchbase Server Node
Doc 1Doc 1
Doc 1
To other node
View engine
Doc 1
Couchbase Server Cluster
Indexing and Querying
User Configured Replica Count = 1
Active
Doc 5
Doc 2
Doc
Doc
Doc
Server 1
REPLICA
Doc 4
Doc 1
Doc 8
Doc
Doc
Doc
App Server 1
COUCHBASE Client LibraryCOUCHBASE Client Library
Cluster Map
COUCHBASE Client LibraryCOUCHBASE Client Library
Cluster Map
App Server 2
Doc 9
• Indexing work is distributed amongst nodes
• Large data set possible
• Parallelize the effort
• Each node has index for data stored on it
• Queries combine the results from required nodes
Active
Doc 5
Doc 2
Doc
Doc
Doc
Server 2
REPLICA
Doc 4
Doc 1
Doc 8
Doc
Doc
Doc
Doc 9
Active
Doc 5
Doc 2
Doc
Doc
Doc
Server 3
REPLICA
Doc 4
Doc 1
Doc 8
Doc
Doc
Doc
Doc 9
Query
Cross Data Center Replication – The basics
•Replicate your Couchbase data across clusters
•Clusters may be spread across geos
•Configured on a per-bucket (per-database) basis
•Supports unidirectional and bidirectional operation
•Application can read and write from both clusters Active – Active replication
•Replication throughput scales out linearly
•Different from intra-cluster replication
3333 22
Cross data center replication – Data flow2
Managed Cache
Dis
k Q
ueue
Disk
Replication Queue
App Server
Couchbase Server Node
Doc 1Doc 1
Doc 1
To other node
XDCR Queue
Doc 1
To other cluster
Cross Data Center Replication (XDCR)Couchbase Server Cluster
NY DATA CENTERActive
Doc
Doc 2
Server 1
Doc 9
Server 2 Server 3
RAM
Doc Doc Doc
Active
Doc
Doc
Doc RAM
Active
Doc
Doc
DocRAM
DISK
Doc Doc Doc
DISK
Doc Doc Doc
DISK
Couchbase Server ClusterSF DATA CENTER
Active
Doc
Doc 2
Server 1
Doc 9
Server 2 Server 3
RAM
Doc Doc Doc
Active
Doc
Doc
Doc RAM
Active
Doc
Doc
DocRAM
DISK
Doc Doc Doc
DISK
Doc Doc Doc
DISK
Elasticsearch Integration (via XDCR)
RAM CACHERAM CACHE
Doc 1
Doc 2
Doc
Doc
Doc
Doc
Doc
Doc
Doc
Doc
Doc
SERVER 1
Doc 6
DISKDISK
RAM CACHERAM CACHE
Doc 1
Doc 2
Doc
Doc
Doc
Doc
Doc
Doc
Doc
Doc
Doc
SERVER 2
Doc 6
DISKDISK
RAM CACHERAM CACHE
Doc 1
Doc 2
Doc
Doc
Doc
Doc
Doc
Doc
Doc
Doc
Doc
SERVER 3
Doc 6
DISKDISK
Couchbase Cluster
ES SERVER 1
Elasticsearch Cluster
18
01
80
18
01
80
18
01
80
18
01
80
18
01
80
18
01
80
ES SERVER 2 ES SERVER 3
CouchbaseCouchbaseTransport Transport
PluginPlugin
CouchbaseCouchbaseTransport Transport
PluginPlugin
CouchbaseCouchbaseTransport Transport
PluginPlugin
DEMO
Demo: Data for Social Game
3 Objects (documents) within game:• Players•Monsters• Items
Gameplay:• Players fight monsters•Monsters drop items• Players own items
Player Document
{"jsonType": "player","uuid": "35767d02-a958-4b83-8179-616816692de1","name": "Keith4540","hitpoints": 75,"experience": 663,"level": 4,"loggedIn": false
}
Player ID
Item Document
{"jsonType": "item","name": "Katana_e5890c94-11c6-65746ce6c560","uuid": "e5890c94-11c6-4856-a7a6-65746ce6c560","ownerId": "Dale9887"
}
Item ID
Player ID
Monster Document
{"jsonType": "monster","name": "Bauchan9932","uuid": "d10dfc1b-0412-4140-b4ec-affdbf2aa5ec","hitpoints": 370,"experienceWhenKilled": 52,"itemProbability": 0.5050581341872865
}
Monster ID
Official SDKs
Community SDKs
www.couchbase.com/develop
PythonRuby
Go Clojure
Q & A