View
1
Download
0
Category
Preview:
Citation preview
Couchbase - The Engagement DatabaseAbbas Ahmed - Principal Database Engineer @ Rentalcars.com
Today’s Agenda
• Introduction • What is an engagement database? • Couchbase Architecture Overview & essential theory
• Couchbase Lab setup • Lab Exercises
2
Who am I
3
Who am I• One of six DB Engineers @ Rentalcars.com
3
Who am I• One of six DB Engineers @ Rentalcars.com• Have been a MySQL user/admin since 1999 (version 3.1)
3
Who am I• One of six DB Engineers @ Rentalcars.com• Have been a MySQL user/admin since 1999 (version 3.1)• Have worked with Oracle, SQL Server & Foxpro :)
3
Who am I• One of six DB Engineers @ Rentalcars.com• Have been a MySQL user/admin since 1999 (version 3.1)• Have worked with Oracle, SQL Server & Foxpro :)• Like doing solution designs
3
Who am I• One of six DB Engineers @ Rentalcars.com• Have been a MySQL user/admin since 1999 (version 3.1)• Have worked with Oracle, SQL Server & Foxpro :)• Like doing solution designs• Started using Couchbase in 2013
3
Who am I• One of six DB Engineers @ Rentalcars.com• Have been a MySQL user/admin since 1999 (version 3.1)• Have worked with Oracle, SQL Server & Foxpro :)• Like doing solution designs• Started using Couchbase in 2013• Was honoured as a Couchbase Champion in early 2017
3
More Introductions
4
More Introductions• Let’s introduce ourselves one at a time.
4
More Introductions• Let’s introduce ourselves one at a time.• Please highlight your experience with NoSQL or SQL
4
A Useful Quote
5
• “Different databases are designed to solve different problems. Using a single database engine for all of the requirements usually leads to non- performant solutions; storing transactional data, caching session information, traversing graph of customers and the products their friends bought are essentially different problems.”
― Pramod J. Sadalage
― Pramod J. Sadalage
To which general NoSQL category does Couchbase belong?
6
To which general NoSQL category does Couchbase belong?
6
• Key Value Store (Binary Values)
To which general NoSQL category does Couchbase belong?
6
• Key Value Store (Binary Values)• Yes
To which general NoSQL category does Couchbase belong?
6
• Key Value Store (Binary Values)• Yes
• Document Store (JSON Values)
To which general NoSQL category does Couchbase belong?
6
• Key Value Store (Binary Values)• Yes
• Document Store (JSON Values)• Yes
To which general NoSQL category does Couchbase belong?
6
• Key Value Store (Binary Values)• Yes
• Document Store (JSON Values)• Yes
• Column Family
To which general NoSQL category does Couchbase belong?
6
• Key Value Store (Binary Values)• Yes
• Document Store (JSON Values)• Yes
• Column Family• No
To which general NoSQL category does Couchbase belong?
6
• Key Value Store (Binary Values)• Yes
• Document Store (JSON Values)• Yes
• Column Family• No
• Graph
To which general NoSQL category does Couchbase belong?
6
• Key Value Store (Binary Values)• Yes
• Document Store (JSON Values)• Yes
• Column Family• No
• Graph• No
Core Couchbase PrinciplesThe core database engine provides the fundamental capabilities for document management and indexing. Based on a memory-first, async everywhere architecture, this includes core capabilities of the database engine, like caching, data persistence, and inter-node replication.
7
Where did Couchbase come from?
8
Where did Couchbase come from?
8
• Memcached -> Membase -> Couchbase
Where did Couchbase come from?
8
• Memcached -> Membase -> Couchbase• Storage Engine written in C++
Where did Couchbase come from?
8
• Memcached -> Membase -> Couchbase• Storage Engine written in C++
• ForestDB, Plasma
Where did Couchbase come from?
8
• Memcached -> Membase -> Couchbase• Storage Engine written in C++
• ForestDB, Plasma• Cluster Manager & XDCR Originally written in
Erlang
Where did Couchbase come from?
8
• Memcached -> Membase -> Couchbase• Storage Engine written in C++
• ForestDB, Plasma• Cluster Manager & XDCR Originally written in
Erlang• Being ported to Go lang
Where did Couchbase come from?
8
• Memcached -> Membase -> Couchbase• Storage Engine written in C++
• ForestDB, Plasma• Cluster Manager & XDCR Originally written in
Erlang• Being ported to Go lang
• Lots of performance and reliability improvements seen
Where did Couchbase come from?
8
• Memcached -> Membase -> Couchbase• Storage Engine written in C++
• ForestDB, Plasma• Cluster Manager & XDCR Originally written in
Erlang• Being ported to Go lang
• Lots of performance and reliability improvements seen
••
The Couchbase Data Platform
9
• An end to end data management & delivery system • Supports the emerging “Engagement” use cases very well • What is Engagement?
Couchbase Server architecture
Mobile & IoT Analytics
Preview
KV Query Search
Elastic Scale Architecture Memory-first Architecture
Unified Programming Interface
Core Database Engine
Any hardware | Any Cloud | Any Container
Cro
ss S
tack
Sec
urity
SQL & Big D
ata Integrations
How does the CAP theorem see it?
11
How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system
11
How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system
• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.
11
How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system
• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.
• It tolerates network partitions, but never at the cost of consistency
11
How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system
• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.
• It tolerates network partitions, but never at the cost of consistency• Intra-cluster replication is from Memory to Memory using DCP protocol.
11
How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system
• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.
• It tolerates network partitions, but never at the cost of consistency• Intra-cluster replication is from Memory to Memory using DCP protocol.
• When you think in terms of XDCR replicated clusters, however it is an AP system across clusters.
11
How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system
• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.
• It tolerates network partitions, but never at the cost of consistency• Intra-cluster replication is from Memory to Memory using DCP protocol.
• When you think in terms of XDCR replicated clusters, however it is an AP system across clusters.• Because XDCR is eventually consistent
11
How does the CAP theorem see it?• A single Couchbase cluster is a ‘CP’ system
• All changes to any documents are immediately visible to other nodes in the same cluster, hence providing a consistent view.
• It tolerates network partitions, but never at the cost of consistency• Intra-cluster replication is from Memory to Memory using DCP protocol.
• When you think in terms of XDCR replicated clusters, however it is an AP system across clusters.• Because XDCR is eventually consistent• XDCR has to hit the disk before before data is pushed to remote. (Not
true for upcoming Ephemeral Buckets)
11
What services does a node run?
12
What services does a node run?
12
• memcached (The main storage)
What services does a node run?
12
• memcached (The main storage)• goxdcr (XDCR process)
What services does a node run?
12
• memcached (The main storage)• goxdcr (XDCR process)• indexer (The Indexer)
What services does a node run?
12
• memcached (The main storage)• goxdcr (XDCR process)• indexer (The Indexer)• cbq-engine (N1QL Query Engine)
What services does a node run?
12
• memcached (The main storage)• goxdcr (XDCR process)• indexer (The Indexer)• cbq-engine (N1QL Query Engine)• And other auxiliary ones
Scaling Choices for Couchbase
13
Multi-dimensional Scaling (MDS)
14
Data Organization in Couchbase
15
Data Organization in Couchbase
• What do you call a database or schema?
15
Data Organization in Couchbase
• What do you call a database or schema?• Bucket
15
Data Organization in Couchbase
• What do you call a database or schema?• Bucket
• How does a Bucket get distributed over multiple nodes?
15
Data Organization in Couchbase
• What do you call a database or schema?• Bucket
• How does a Bucket get distributed over multiple nodes?• Automatic Shards called vBuckets (1-1024)
15
Data Organization in Couchbase
• What do you call a database or schema?• Bucket
• How does a Bucket get distributed over multiple nodes?• Automatic Shards called vBuckets (1-1024)
• Clients maintain two connections to the cluster
15
Data Organization in Couchbase
• What do you call a database or schema?• Bucket
• How does a Bucket get distributed over multiple nodes?• Automatic Shards called vBuckets (1-1024)
• Clients maintain two connections to the cluster• One for Data
15
Data Organization in Couchbase
• What do you call a database or schema?• Bucket
• How does a Bucket get distributed over multiple nodes?• Automatic Shards called vBuckets (1-1024)
• Clients maintain two connections to the cluster• One for Data• One for Management (Cluster Map Updates, Topology
change updates)15
The Couchbase cluster map
16
Inter-node replication
17
Tunable Durability for Clients
18
Sizing a Couchbase cluster
19
Sizing a Couchbase cluster
19
• Outcomes: Number and Size of nodes (RAM, CPU & Disk Space)
Sizing a Couchbase cluster
19
• Outcomes: Number and Size of nodes (RAM, CPU & Disk Space)
• Inputs: Number of items, average size of items and number of replicas required.
Sizing a Couchbase cluster
19
• Outcomes: Number and Size of nodes (RAM, CPU & Disk Space)
• Inputs: Number of items, average size of items and number of replicas required.
• Max document size = 20MBytes
Time for Lab Exercises
20
Time for Lab Exercises
20
• How to plan a cluster
Time for Lab Exercises
20
• How to plan a cluster• How to start a new cluster
Time for Lab Exercises
20
• How to plan a cluster• How to start a new cluster• How to expand an existing cluster
Time for Lab Exercises
20
• How to plan a cluster• How to start a new cluster• How to expand an existing cluster• How to perform node failover
Time for Lab Exercises
20
• How to plan a cluster• How to start a new cluster• How to expand an existing cluster• How to perform node failover• Learn how to monitor a Couchbase cluster
Time for Lab Exercises
20
• How to plan a cluster• How to start a new cluster• How to expand an existing cluster• How to perform node failover• Learn how to monitor a Couchbase cluster
Time for Lab Exercises
20
• How to plan a cluster• How to start a new cluster• How to expand an existing cluster• How to perform node failover• Learn how to monitor a Couchbase cluster
• Come back before XDCR Lab
XDCR in Couchbase
21
XDCR in Couchbase
21
• Uses data Push instead of Pull (unlike MySQL)
XDCR in Couchbase
21
• Uses data Push instead of Pull (unlike MySQL)• Supports many topologies, linear, star or mesh, even you
can replicate to the same cluster in a different bucket
XDCR in Couchbase
21
• Uses data Push instead of Pull (unlike MySQL)• Supports many topologies, linear, star or mesh, even you
can replicate to the same cluster in a different bucket• Sends data over an encrypted channel, keys created
when you setup XDCR
XDCR in Couchbase
21
• Uses data Push instead of Pull (unlike MySQL)• Supports many topologies, linear, star or mesh, even you
can replicate to the same cluster in a different bucket• Sends data over an encrypted channel, keys created
when you setup XDCR• Big Data & Elasticsearch connectivity uses XDCR as well.
XDCR in Couchbase
21
• Uses data Push instead of Pull (unlike MySQL)• Supports many topologies, linear, star or mesh, even you
can replicate to the same cluster in a different bucket• Sends data over an encrypted channel, keys created
when you setup XDCR• Big Data & Elasticsearch connectivity uses XDCR as well.
XDCR in Couchbase
22
Backup & Recovery
23
Backup & Recovery
23
• Two methods available.
Backup & Recovery
23
• Two methods available.• cbbackup with/without cbbackupwrapper
Backup & Recovery
23
• Two methods available.• cbbackup with/without cbbackupwrapper• cbbackupmgr (recommended)
Backup & Recovery
23
• Two methods available.• cbbackup with/without cbbackupwrapper• cbbackupmgr (recommended)
• Both full and Incremental backups available.
Backup & Recovery
23
• Two methods available.• cbbackup with/without cbbackupwrapper• cbbackupmgr (recommended)
• Both full and Incremental backups available.• cbbackupmgr offers an enterprise grade solution.
Backup & Recovery
23
• Two methods available.• cbbackup with/without cbbackupwrapper• cbbackupmgr (recommended)
• Both full and Incremental backups available.• cbbackupmgr offers an enterprise grade solution.
Recommended