Couchbase Day

Wi-Fi: Network – Matrix | Password - 11223344

©2015 Couchbase Inc. 2

Today’s AgendaRegistration and Continental Breakfast§ Labs and Environment Setup

Introduction to NoSQL§ Why NoSQL?§ Introduction to Couchbase

Couchbase Architecture§ Single Node Operations§ Cluster Operations

Break

HA and DR§ Availability and XDCR§ Backup/Restore§ Security12:00 – Lunch Break

Developing with Couchbase§ Couchbase SDKs

Couchbase Indexing§ Couchbase GSI and Views§ Views Overview

Break

Indexing and Query§ Couchbase Indexes (GSI/MOI)§ Couchbase Query (N1QL)

Couchbase Mobile§ Overview

Adjourn§ Close

Why NoSQL?


Why companies need a new database solution

§ What are the biggest problems with Relational Databases that are driving adoption of NoSQL?

LACK OF FLEXIBILITY/ HAS RIGID SCHEMAS

INABILITY TO SCALE OUT

PERFORMANCE CHALLENGES

49%

69%

50%

47%

44%

COST

ALL OF THE ABOVE

35%

29%

16%

12%


DynamoOctober2007

CassandraAugust2008

BigtableNovember2006

VoldemortFebruary2009

Veryfeworganizationswantto(fewercan)buildandmaintaindatabasesoftwaretechnology.Buteveryorganizationparticipatinginthedigitaleconomyneedsthistechnology.

Why NoSQL?

New data solutions to address emerging needs:§ Performance and Scalability§ High Availability with Data Synchronization (multi-datacenter)§ Flexible Schema to accommodate frequent changes§ Transparent/Auto-Sharding without application participation§ Distributed System for Querying§ Managed Cache

©2015 Couchbase Inc. 6Scaling out flattens the cost and performance curves

NoSQL Database Scales OutCost and performance mirrors app tier

Users

NoSQLDistributedDataStore

Web/AppServerTier

Application Scales OutJust add more commodity web servers

Users

System CostApplication Performance

Application Performance System Cost

WhyNoSQL


How has Couchbase Server evolved beyond NoSQL?

§ Discreet, scalable services (data, index, query)§ Filtering for cross data center replication§ SQL-based query language and secondary indexing§ Geospatial search and indexing§ CRUD, prepared statements, and covering indexes§ Memory-optimized Indexes§ Array Indexing within Nested JSON§ Graphical Query W0rkbench§ Sub-Document API§ LDAP integration and auditing; RBAC, X-509 Certificates


Next Generation Data Management Architecture

DataSources

Enterprise Applications

Social MediaWeb &

EcommerceMobile AppsSensor Data Mainframe

External Data

System Logs

DataIntegration

Speed Layer Batch Layer

DataManagement

Data Lake Data Warehouse

Business Intelligence Dashboards Query ToolsDataAccess

In Memory Cache

Data Serving Layer


Next Generation Data Management Architecture

Enterprise Applications

Social MediaWeb &

EcommerceMobile AppsSensor Data MainframeExternal DataSystem Logs

Speed Layer Batch Layer

Data Lake

Business Intelligence Dashboards Query Tools

In Memory Cache

Data Warehouse

N1QL Query Workbench

DataSources

DataIntegration

DataManagement

DataAccess

Data Serving Layer

Real Time Data Ingestion


Couchbase Ecosystem


How Does Couchbase Fit In Big Data?

Couchbase and Hadoop are complementary.


Couchbase and Hadoop

New Data Stream

Merged View

All DataPrecompute

Views (Map Reduce)

Process Stream

Incremental Views

Partial Aggregate

Partial Aggregate

Partial Aggregate

Real-Time Data

BatchRecompute

Batch Views

Real-Time Views

Real-TimeIncrement

Merge

Batch Layer

Serving Layer

Speed Layer

Couchbase HadoopConnector (Sqoop)


Couchbase HadoopConnector (Sqoop)

Couchbase and Hadoop

New Data Stream

Merged View

All DataPrecompute

Views (Map Reduce)

Process Stream

Incremental Views

Partial Aggregate

Partial Aggregate

Partial Aggregate

Real-Time Data

BatchRecompute

Batch Views

Real-Time Views

Real-TimeIncrement

Merge

Batch Layer

Serving Layer

Speed Layer

Stream / Data Ingestion

Store

Incremental Data / Stream

processing

Serving merged results /

responses


Couchbase Server on Docker and Orchestration Technologies Official Docker Support with MacOS, Ubuntu is coming

Couchbase & Containers


Platform as a Service

§ Couchbase runs on PaaS environments

Current Focus Future Plan


Infrastructure as a Service

Couchbase supports IaaS environments

§ Available in AWS marketplace

§ BYOL or hourly pricing

§ Available in Azure marketplace

§ BYOL or hourly pricing

§ Marketplace image in the works

§ BYOL at the moment§ Hourly pricing soon

Other providers supported

Couchbase OverviewWhat is Couchbase?


Couchbase Server Defined

§ Couchbase Server 4.X is the first NoSQL Database to enable you to develop with agility and operate at any scale.

Managed Cache Key-Value Store Document Database Embedded Database Sync Management


The Power of the Flexible JSON Schema

Ability to store data in multiple ways• Denormalized single document, as opposed to normalizing data across multiple table• Dynamic Schema to add new values when needed


Document Fundamentals - Composition

§ Similar to primary keys in relational databases§ Documents are partitioned based on the

document ID§ ID based document lookup is extremely fast § Must be unique

§ JSON§ Binary - integers, strings, booleans§ Common binary values include serialized objects,

compressed XML, compressed text, encrypted values

Document ID or Key

Value

§ CAS Value (unique identifier for concurrency)§ TTL§ Flags (optional client library metadata)§ Revision #

Metadata

Document

Document ID / Key

MetadataValue


Storing And Retrieving Documents


Couchbase Data Access

• Everything is layered on top of Key Value

• A Document store is a special case of Key-Value

• Views provide aggregation and real-time analytics through incremental map-reduce

• Global Secondary Indexes provide low latency/high throughput indexes

• N1QL is a language that provides a powerful and expressive way of accessing documents

Couchbase ArchitectureSingle Node

23


Couchbase Architecture

• Data Service – builds and maintains Distributed secondary indexes (MapReduce Views)

• Indexing Engine – builds and maintains Global Secondary Indexes

• Query Engine – plans, coordinates, and executes queries against either Global or Distributed indexes

• Cluster Manager – configuration, heartbeat, statistics, RESTful Management interface


Couchbase Read Operation

APPLICATION SERVER

MANAGED CACHE

DISK

DISKQUEUE

REPLICATIONQUEUE

DOC 1

GETDOC 1

DOC 1


APPLICATION SERVER

MANAGED CACHE

DISK

DISKQUEUE

REPLICATIONQUEUE

Write Operation

DOC 1

DOC 1DOC 1


Cache Ejection

APPLICATION SERVER

MANAGED CACHE

DISK

DISKQUEUE

REPLICATIONQUEUE

DOC 1

DOC 2DOC 3DOC 4DOC 5

DOC 1

DOC 2 DOC 3 DOC 4 DOC 5


APPLICATION SERVER

MANAGED CACHE

DISK

DISKQUEUE

REPLICATIONQUEUE

DOC 1

Cache Miss



GETDOC 1

DOC 1

DOC 1

Couchbase ArchitectureCluster


Auto sharding – Bucket and vBuckets

vB

Data buckets

vB

1 ….. 1024

Active Virtual buckets

vB vB

1 ….. 1024

Replica Virtual buckets


Cluster Map

Couchbase Cluster

CLUSTER MAP

Hashing Algorithm

CRC32

vBucket2

vBucket3

vBucket4

vBucket5

vBucket6

vBucket7

vBucket1024

vBucket1

...

Couchbase SDK

Couchbase Cluster

CLUSTER MAP

Hashing Algorithm

CRC32

vBucket2

vBucket3

vBucket4

vBucket5

vBucket6

vBucket7

vBucket1024

vBucket1

...

Couchbase SDK


ACTIVE ACTIVE ACTIVE

REPLICA REPLICA REPLICA

Couchbase Server 1 Couchbase Server 2 Couchbase Server 3

Basic Operation

SHARD5

SHARD2

SHARD9

SHARD SHARD SHARD

SHARD4

SHARD7

SHARD8

SHARD SHARD SHARD

SHARD1

SHARD3

SHARD6

SHARD SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD SHARD

SHARD6

SHARD3

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD SHARD


Rebalance




ACTIVE ACTIVE

REPLICA REPLICA

Couchbase Server 4 Couchbase Server 5

SHARD5

SHARD2

SHARD SHARD

SHARD4

SHARD SHARD

SHARD1

SHARD3

SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD SHARD

SHARD6

SHARD3

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD SHARD

SHARD7

SHARD

SHARD6

SHARD

SHARD8

SHARD9

SHARD

READ/WRITE/UPDATE


Fail Over




ACTIVE ACTIVE

REPLICA REPLICA


SHARD5

SHARD2

SHARD SHARD

SHARD4

SHARD SHARD

SHARD1

SHARD3

SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD

SHARDSHARD6

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD

SHARD

SHARD7

SHARD

SHARD6

SHARDSHARD8

SHARD9

SHARD

SHARD3

SHARD1

SHARD3

SHARD

High Availability with Couchbase

35


Recovering from a Fail Over




ACTIVE ACTIVE

REPLICA REPLICA


SHARD5

SHARD2

SHARD SHARD

SHARD4

SHARD SHARD

SHARD1

SHARD3

SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD

SHARDSHARD6

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD

SHARD

SHARD7

SHARD

SHARD6

SHARDSHARD8

SHARD9

SHARD

SHARD3

SHARD1

SHARD3

SHARD


Recovering from Failover

Three options:

1. Remove the node

2. Full recovery

3. Delta node recovery


Delta Node Recovery

Recover Failed Nodes with incremental Catch-up

§ Improve recovery time for nodes with large datasets

§ Add a failed node into cluster and recover data from where it left off


Rack Zone Awareness – Server Groups

Informs Couchbase as to the physical distribution of nodes in

racks or availability groups.

The Active vBuckets in Server 1will have its replica’s in a different group. Never in the same group.

Couchbase Server 3

Couchbase Server 2

Couchbase Server 6

Couchbase Server 5

Couchbase Server 4

Couchbase Server 9

Couchbase Server 8

Couchbase Server 7

Ensures that replica vBuckets are distributed across groups

REPLICA

REPLICA

Group: A Nodes 1,2,3

Group: B Nodes 4,5,6

Group: C Nodes 7,8,9

Couchbase Server 1

ACTIVE

Lab Time!Labs 1-4

Cross Datacenter Replication (XDCR)

41



TO OTHER NODE

APPLICATION SERVER

MANAGED CACHE

DISK

DISKQUEUE

REPLICATIONQUEUE

XDCR QUEUE

TO OTHER CLUSTER

DOC 1

DOC 1DOC 1DOC 1



TO OTHER NODE

APPLICATION SERVER

MANAGED CACHE

DISK

DISKQUEUE

REPLICATIONQUEUE

XDCR QUEUE

TO OTHER CLUSTER

DOC 1

DOC 1DOC 1DOC 1

Single-node type means easier administration and scaling

CLUSTER REPLICATION

vBucket REPLICAS


XDCR: Cluster Topology Aware

• Applications can access both Clusters (active-active)

• Can provide filtered replication

• One-one, one-many, many-one• Differently sized clusters• Different HW platforms • Mixed versions supported (Upgrades)



Unidirectional or Bidirectional Replication

Unidirectional§ Hot spare / Disaster Recovery§ Development/Testing copies§ Connectors (Solr, Elasticsearch) § Integrate to custom consumer

Bidirectional§ Multiple Active Masters§ Disaster Recovery§ Datacenter Locality

Backup and Restore

46


Key Features

§ Zero downtime backup and restore§ Single utility to manage multiple clusters§ Restore from any point, to any bucket or topology§ Fully differential – merge backups to maintain desired restore points§ Built-in concurrency control§ Resume interrupted backups§ Compaction reduces storage requirements


Enterprise Backup and Restore

§ >10x Faster Backups§ Better Data Protection with Enterprise Backup Restore– Scale to larger datasets– Fast backups and restores– Better Consistency in backup data– Resilient to failures/Topology Changes

288

25

0

50

100

150

200

250

300

350

Back

Up

MB/

Sec

4.5 Backup Speed 4.1 Backup Speed


Data Import

§ Create backup repo/opt/couchbase/bin/cbbackupmgr config --archive /tmp/backup --repo demo

§ Import of sample data/opt/couchbase/bin/cbbackupmgr json --host couchbase://172.23.107.27:8091 --username Administrator --password password --bucket travel-sample --dataset /opt/couchbase/samples/travel-sample.zip --format sample

§ Import of list data/opt/couchbase/bin/cbbackupmgr json --host couchbase://172.23.107.27:8091 --username Administrator --password password --bucket chicago --dataset file:///root/data/companies.json --format list --generate-key %permalink%::#MONO_INCR#

Demo

Security


Security with Couchbase

§ Administrative Security:§ Admin user (full access) vs. Read-only user (monitoring, developer access)§ SSL encryption to REST API and Web UI§ HTTP access log

§ Data Security:§ Applications connect via SASL with single user/pass§ Data-at-Rest encryption via partnership with Vormetric§ SSL encryption for over-the-wire

§ Couchbase 4.0:§ LDAP integration§ More extensive administrative action auditing

More at: http://docs.couchbase.com/admin/admin/security/security-best-practices.html


Couchbase authentication overview

§ Application authentication § Buckets are protected with challenge-response SASL protocol§ AuthN happens over CRAM-MD5

§ Admin authentication § Authentication through admin username and password§ Authentication through LDAP (New in 4.0)

AUTHENTICATION


Couchbase authorization overview

§ Application data access § Full access to the bucket application is connected to

§ Admin access§ Full administrator has full privileges on the cluster § Read-only administrator cannot change cluster settings

AUTHORIZATION


Couchbase encryption overview

§ Data-in-motion encryption§ Client-server communication can be encrypted using SSL § Secure admin access using SSL over port 18091§ Secure view access using SSL over port 18092§ Secure XDCR for encryption across datacenters

Track all Access SERVER3SERVER1 SERVER2CouchbaseServer– NewYork

SERVER3SERVER1 SERVER2CouchbaseServer– London

SSL

Client applications

SecureXDCR over

SSL

Admin access

over port 18091

SSL

View access

over port 18092

SSL

https://couchbase_server:18091/…

https://couchbase_server:18092/…

ENCRYPTION


Couchbase encryption overview

§ Transparent data-at-rest encryption solution ENCRYPTION

Storage

Database

Application

User

File Systems

VolumeManagers

DSM

VormetricData Security Manager

on Enterprise premise or in cloudvirtual or physical appliance

• Centrally manage keys and policy• Virtual and physical appliance • High-availability with cluster• Multi-tenant and strong separation of duties• Proven 10,000+ device and key management scale• Web, CLI, API Interfaces• FIPS 140-2 certified

Secure Personally Identifiable Information• User profile information• Login Credentials• IP Addresses


External identity management using LDAP

Centralized identity management § Define multiple read-only admins and full-admins

§ Centralized security policy management for admin accounts for stronger passwords, password rotation, and auto lockouts

Individual accountability. Simplified compliance.§ Define UIDs in LDAP, and map UIDs to read-only / full admin role in

Couchbase

§ Comprehensive audit trails with LDAP UIDs in audit records


New UI for authorizing LDAP administrators

Turn on/off LDAP

Add UIDs to read-only admins

Add UIDs to full admins

Set default behavior if UID is not mapped

Testing credentials to verify what level

of access

Plus REST, and CLI integration for programmatic setup


Admin Auditing in Couchbase

Rich audit events§ Over 25 different, detailed admin audit events

§ Auditing for tools including backup

Configurable auditing § Configurable file target

§ Support for time based log rotation and audit filtering

Easy integration§ JSON format allows for easy integration with downstream

systems using flume, logstash, and syslogd


Auditing a successful login

{"timestamp":"2015-02-20T08:48:49.408-08:00", "id":8192, "name":"login success", "description":"Successful login to couchbase cluster", "role":"admin", "real_userid": {

"source":"ns_server","user":"bjones”

},"sessionid":"0fd0b5305d1561ca2b10f9d795819b2e", "remote":{"ip":"172.23.107.165", "port":59383}

}

WHEN

WHO

WHAT

HOW

Lab Time!Labs 5-7

Lunch

Couchbase SDK

63


Languages and Interfaces for Couchbase

§ Official SDKs– Java– .NET– Node.js– Python

§ For each of these we have– Full Document support– Interoperability– Common Programming Model

Others: Erlang, Perl, TCL, Clojure, Scala

- PHP- C- Go- Ruby

JDBC and ODBC


Interacting with Data

Cluster

Bucket

CRUDView

QueryN1QL Query


Couchbase SDKs

§ What does it mean to be a Couchbase SDK?

Cluster

Bucket

CRUD

FunctionalGive the application developer a concurrent API for basic (k-v) or document management

APIget()insert()replace()upsert()remove()


The Document

§ Documents are integral to the SDKs.§ All SDKs support JSON format§ In addition: Serialized objects, Unquoted Strings, Binary pass-through§ A Document contains:

67

Property Description

ID The bucket-unique identifier

Content The value that is stored

Expiry An expiration time

CAS Check-and-Set identifier


Bucket API – Modifying Documents

§ insert() the document if it does not exist§ replace() the document if it does exist§ upsert() the document (insert or replace)§ remove() the document

§ append() data to the document§ prepend() data to the document

§ counter() for increment/decrement type operations


Bucket API – Retrieving Documents

§ get() the document§ getFromReplica() if the master is not available

§ getAndTouch() to load the document and reset the expiry§ getAndLock() to load the document and write-lock it


Unified API – DML, CAS Example1. Two Clients retrieve the same document "XYZ"2. Client A retrieves it first.3. Client B then retrieves XYZ. Both clients will have the same CAS value for document XYZ4. Client B tries to perform an update to document XYZ. The update succeeds as the CAS value

was unchanged from when Client B initially retrieved the document. Once the update succeeds, the CAS value for XYZ changes.

5. Client A then tries to perform an update on XYZ immediately after Client B. The update will fail as Client A's CAS value is out of date. When Client B updated XYZ, the CAS value changed.

©2014 Couchbase, Inc.


Durability Options for writes

APPLICATION SERVER

CACHE

DISK

XDCR QUEUETO OTHER CLUSTER

CACHE

DISK DISKQUEUE

REPLICATIONQUEUE

APPLICATION SERVER

DISK DISKQUEUE

CACHEREPLICATION

QUEUE

DISKQUEUE

DOC

DOCDOC DOCDOCDOC DOC

DOC

CLUSTER 1NODE 2

CLUSTER 2NODE …

CLUSTER 1NODE 1

DOC

( PUT ) By default, a write operation is successful when the data has been written to the memory of the node with the active vBucket. Eventually (but pretty quickly), the data will also be written to persistent storage and to replica vBuckets.

( replicateTo ) write operation is successful when data has been written to both the active vBucket and replica vBuckets.

( persistTo ) write operation is successful when data has been written to persistent storage• Can be set to require writes to persistent storage for replicas.• Using persistTo will slow performance.


Bucket API – Querying

§ query() is possible for§ Views§ N1QL

§ Streams N response rows as they arrive from the server§ Loads more than one Document based on Criteria§ Typically used to satisfy secondary and advanced querying use cases

Lab Time!Labs 8-10

GSI and VIEWS

74


Which to choose – GSI vs Views

Workloads New GSI in v4.X Map/Reduce Views

ComplexReporting

Just In Time Aggregation Pre-aggregated

Ad-hoc Querying

Ad-hoc View per Query

Flexible Indexing Logic

N1QL Functions Javascript

Secondary Lookups

Faster with Single Node Lookup

Slower due to Scatter-Gather

Range Scans

Faster with Single Node Lookup

Slower due to Scatter-Gather


Comparing GSI and ViewsCapabilities Global Secondary Indexes Map/Reduce Views

Partitioning Model Independent – Indexing Service Aligned to Data – Data Service

Scale Model Independently Scale Index Scale with Data Service

Secondary Lookup Single Node Scatter-Gather

Range Scan Single Node Scatter-Gather

Grouping, Aggregates With N1QL Built-in with Views API

Caching Managed Not Managed

Storage ForestDB Couchstore

Availability Identical Indexes load balanced Replica Based


Indexing Differences

§ Local secondary indexes (Views)– Co-located with data– Higher write performance / Lower read performance: scatter-gather

– Map/Reduce Views: programmable indexer for complex reporting and indexing logic.

– Spatial Views: Incremental R-tree indexing for powerful bounding-box queries– Full partition alignment and paired scalability with Data Service

§ Global secondary indexes (GSI / MOI) – Independently partitioned and independently scalable indexes in Indexing Service– Isolated from Key-Value operations– Low latency queries without compromising on mutation performance (insert/update/delete)– Higher query performance– Async writes to a large number of global indexes– Memory Optimized Indexes (MOI) alternate storage option to standard GSI - small memory

footprint, optimized for lowest latency queries


Indexing with N1QL

Indexing in N1QLGlobal and Local Indexes

Indexing Types for N1QL

View Based Indexes• Local Indexes:

• Partitioned Aligned to Data• Auto Partitioning• Built in Replicas• Simple Scaling Model

Global Secondary Indexes (GSI)• Global Indexes:

• Independently Partitioned Indexes• Predictable High Performance at Scale• Sophisticated Scaling Model

Controlled with USING VIEW|GSI in CREATE INDEX. Set per Index. Drop and Recreate the index using DROP INDEX & CREATE INDEX to change Index Type.

Views

79


A little about Views…

§ Using views you can define– Primary indexes

– Simple secondary indexes (most common use case)

– Complex secondary, tertiary and composite indexes

– Aggregations (reduction)

§ Built using MapReduce technology – Map and Reduce functions are written in Javascript


A little about Views…

§ Captures user-defined key/value pairs about a document into an index§ Can execute logic on data during indexing§ Able to express relationships too cumbersome to maintain in Key/Value§ Incremental indexing essentially pre-computes query results§ Higher latency compared to key/value lookups

§ Common Use Cases– Materialized view: project an alternate view of your document– Complex aggregates– Lightweight reporting


Queries run against stale indexes by default

§ stale=update_after (default if nothing is specified)• always get fastest response

• can take two queries to read your own writes

§ stale=ok• auto update will trigger eventually

• might not see your own writes for a few minutes

• least frequent updates -> least resource impact

§ stale=false• Use with “set with persistence” if data needs to be included in view results

• BUT be aware of delay it adds, only use when really required


Lightweight Reporting Example

Document:{“date”: “2015-08-03T16:38:05.835Z”,“source”: “192.168.56.101”,“logtype”: “error”,“message”: “An error occurred during startup”

}

Map:function(doc, meta) {

emit(dateToArray(doc.date), doc.logtype);}



function(key, values, rereduce) {

var response = {"warning" : 0, "error": 0, "fatal" : 0 };

for(i=0; i<values.length; i++) {if (rereduce) {

response.warning = response.warning + values[i].warning;response.error = response.error + values[i].error;

response.fatal = response.fatal + values[i].fatal;

} else {if (values[i] == "warning")

response.warning++;if (values[i] == "error" )

response.error++;

if (values[i] == "fatal" )response.fatal++;

}}

return response;

}



Query: ?group_level=2&startkey=[2010,7]&endkey=[2011,4]

{"rows":[{"key":[2010,7], "value":{"warning":4,"error":2,"fatal":0}},{"key":[2010,8], "value":{"warning":4,"error":3,"fatal":0}},{"key":[2010,9], "value":{"warning":4,"error":6,"fatal":0}},{"key":[2010,10],"value":{"warning":7,"error":6,"fatal":0}},{"key":[2010,11],"value":{"warning":5,"error":8,"fatal":0}},{"key":[2010,12],"value":{"warning":2,"error":2,"fatal":0}},{"key":[2011,1], "value":{"warning":5,"error":1,"fatal":0}},{"key":[2011,2], "value":{"warning":3,"error":5,"fatal":0}},{"key":[2011,3], "value":{"warning":4,"error":4,"fatal":0}},{"key":[2011,4], "value":{"warning":3,"error":6,"fatal":0}}]

}

Indexing

86


Data Service

Projector & Router

Indexing Service

Query ServiceIndex Service

SupervisorIndex maintenance &

Scan coordinator

Index#2Index#1

Query Processorcbq-engine

Bucket#1 Bucket#2

DCP StreamIndex#4Index#3

...

Bucket#2

Bucket#1

Projector and Router: 1 Projector and Router per node1 stream of changes per bucket per supervisor

ForestDBStorage Engine

Supervisor1 Supervisor per nodeMany indexes per Supervisor

MemDB Storage Engine for MOI


Types of Indexes

§ Composite Indexes§ Covering Indexes§ Filtered Indexes§ Function Based Indexes§ Sub-Document Indexes§ Incremental MapReduce Views§ Spatial Views§ Full Text Indexes§ Prepared Statements§ N1QL Complete SQL CRUD support: Insert, Delete, Merge, Etc.


create index idx_odspilot on ods(DISTINCT ARRAY p.FILEN in p in PILOT END);

Array Indexing

§ Index only required elements or attributes in the array

§ Benefits are lot more significant for nested arrays/objects

§ Efficient on Index storage & search time

Transportation Company POC• 422,137 documents.• Query2: BEFORE array indexing • Primary index scan• 38.91 seconds.

• Query2: AFTER array indexing • Array index scan [DistinctScan]• 8.51 millisecond• Improvement of 4572 Times


Step by Step – Building indexes

§ Building and Index1. Issue “CREATE INDEX …”2. Place on the “next node” if WITH {“nodes”:[]}not specified – “next node” picked by round robin3. Create metadata for the index on the node– “next node” picked by round robin4. If deferred build = true– Mark Index Status = “Pending” – Wait for “BUILD INDEX ...”5. Else– Mark Index Status = “Pending” – Start DCP Backfill with Projector– When Done – Mark Index Status = “Online”


Step by Step – Ingest a Mutation

§ Ingest a Mutation on Data1. DCP Feed communicates the mutation to Projector on the node.2. Projector picks up the mutation for a bucket– Projector strips the mutation to check relevance to existing indexes on each indexer– Projector Sends stripped mutation to indexers on all index service nodes3. Indexer receive the mutation on each index service node– Queue the mutation– Extraction Worker Prepares and queues the update to each index.– Persistence Worker persist the mutation to the index


Global Secondary Indexes – Storage Options

Global Secondary IndexPredictable low latency with Query and Key Based Operations,

even in presence of large number of indexes

Storage Option for GSI

Standard Global Secondary Indexes• Optimized for Storage + Memory: efficient spill-

to-disk with ForestDB as the storage engine• Advanced IO Performance: New circular-writes

that minimize compaction overhead

Memory-Optimized Global Secondary Indexes (MOI)• Optimized for Memory: small memory footprint,

optimized for lowest latency queries• Faster Indexing: fresh indexes under heavy mutations with

lock free index maintenance

Controlled with Cluster Settings (Settings Tab). Cluster-wide Setting for all Indexes and Require removing all GSI indexes and index service nodes to change storage options.

(New in 4.5)


Memory Optimized (MOI) Global Secondary Indexes

• Optimized for Memory: small memory footprint, optimized for lowest latency queries

• Faster Indexing: fresh indexes under heavy mutations with lock free index maintenance


Which to Choose – Standard vs Memory Optimized GSI

Standard GSI Memory Optimized GSI

Initial Index Build Times Slower Faster

Ongoing Index Build Times

Slower Faster

Ability to Keep up with Mutation Rate

Lower Higher

Memory Demand for Larger Indexes

Lower Higher

Behavior on OOM(out of memory)

Spill to disk Stop processing mutations


Indexing & HA

§ Mirrored Indexes are load balanced and used for HA– Size at least 2 indexes for production– Add more indexes if all requests cannot be handled by a single index to protect against

failures


Indexes & Partitioning

§ Why Partition Indexes?– Index may not fit on the node– Fit index to the nodes you have– Distribute Index load to many nodes– Minimize the tree scanned– Large/Tall indexes will take longer to scan

§ How to Partition Indexes?– By list of values – great for scan latency– ... WHERE type = “customer” – ... WHERE type = “product”– By range –– ... WHERE id BETWEEN 10 AND 20– ... WHERE id BETWEEN 20 AND 30– By function – no range scan ability


Efficient IO with Circular Writes in Standard GSI

Standard Global Secondary Indexes§ Reduced Disk IO Requirements– Append-Only Writes with frequent full compaction (Version 4.1 & Earlier)– Circular-Reuse Writes with reduced full compactions (New in 4.5: )– Reused orphaned blocks in the index file– Reduce the need for frequent full-compactions of the index file

Index Mutation

Append-Only WriteCircular Write

File for Global Secondary Indexes

Index Mutation

Index Mutation

N1QL - Query


What is N1QL?

§ N1QL is a new query language that systematically extends SQL to document-oriented data

§ N1QL extends SQL to handle data that is:– Nested: Contains nested objects, arrays

– Heterogeneous: Schema-optional, non-uniform

– Distributed: Partitioned across a cluster


N1QL (EXPRESSIVE)

§ Access to every part of JSON document§ Scalar & Aggregate functions § Issue subquery in any expressions§ Subqueries§ Subqueries in the FROM clause

Give developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data.


N1QL (POWERFUL)

§ Access to every part of JSON document§ JOINS, Aggregations, standard scalar functions§ Aggregation on arrays§ NEST & UNNEST operations§ Covering Index



N1QL (QUERYING)

§ INSERT§ UPDATE§ DELETE§ MERGE§ SELECT§ EXPLAIN



N1QL (TRANSFORMING & MANIPULATING)

§ Full Transformation of the data via Query.§ INSERT§ INSERT single & multiple documents§ INSERT result a SELECT statement§ DELETE documents based on complex filter§ UPDATE any part of JSON document & use complex filter.§ MERGE two sets of documents using traditional MERGE statement§ SUBQUERIES



Optimizing Query Speed

§ There are four options for N1QL queries:

Primary Index

Secondary Index(GSI or MOI)

Covered Index

“ON KEYS”


Query Execution Flow

1. Application submits N1QL query

2. Query is parsed, analyzed and plan is created

1

2



3. Query Service makes request to Index Service

4. Index Service returns document keys and data

3

4



5. If Covering Index, skip step 6

6. If filtering is required, fetch documents from Data Service56



7. Apply final logic (e.g. SORT, ORDER BY)

8. Return formatted results to application

7

8


Inside a Query Service (without covering index)

Client

FetchParse Plan Join FilterPre-Aggregate

Offset Limit ProjectSortAggregateScan

Query Service

Index Servic

e

Data Servic

e

Data-parallel — Query is N data streams over N cores*

Memory-based

Pluggable architecture — datastore, index…

Request Response


Query Consistency

Strict Request-Time Consistency

(request_plus)Query execution is delayed until all indexes process mutations up to T3

RYOW Consistency(at_plus)

Query execution is delayed until all indexes process mutations up to T1

T1: insert (k1, v1)

T3: issue query on (k1,v1)

T2: do other business logic computation

…

…


Joins (Referential)


Joins (Referential)


Joins (Referential)


Joins (Referential)


Document-oriented Extensions

§ Nested

– Paths—user.profile.email, children[0], children[0:2]

– NEST, UNNEST

– Ranging—EVERY child.age > 5 FOR child IN children END

– Transformations—SELECT {“name”: first_name || last_name}

§ Non-uniform

– IS MISSING

– Type checking and conversion

§ Distributed– Direct lookup—USE KEYS

– Efficient joins—ON KEYS


SELECT statement

SELECT customers.id,customers.NAME.lastname, customers.NAME.firstnameSum(orderline.amount)

FROM orders UNNEST orders.lineitems AS orderline JOIN customers ON KEYS orders.custid

WHERE customers.state = 'NY'GROUP BY customers.id,

customers.NAME.lastnameHAVING sum(orderline.amount) > 10000 ORDER BY sum(orderline.amount) DESC

Dotted sub-document referenceNames are CASE-SENSITIVE

UNNEST to flatten the arrays

JOINS with Document KEY of customers


UNNEST example: Customer Document{

"C_ZIP" : "828011111","C_STATE" : "vt",

"C_FIRST" : "ykfdbqku","C_CREDIT" : "GC","C_DELIVERY_CNT" : 0,"C_W_ID" : 1,"C_CITY" : "quhpismkzumehqhr","C_STREET_1" : "rmtxadlsxqefdcwf","C_D_ID" : 1,"ORDERS" : [

{"ORDER_LINE" : [

{"OL_AMOUNT" : 0,"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z","OL_DIST_INFO" : "yptiwgjdelfxmathbjzirvye","OL_I_ID" : 35828,"OL_SUPPLY_W_ID" : 1,"OL_QUANTITY" : 5

},{

"OL_AMOUNT" : 0,"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z","OL_DIST_INFO" : "dxhqulhcgksjgqsicujzqhdb","OL_I_ID" : 26024,"OL_SUPPLY_W_ID" : 1,"OL_QUANTITY" : 5

},}

….


UNNEST example: N1QL Code{

"C_ZIP" : "828011111","C_STATE" : "vt",

"C_FIRST" : "ykfdbqku","C_CREDIT" : "GC","C_DELIVERY_CNT" : 0,"C_W_ID" : 1,"C_CITY" : "quhpismkzumehqhr","C_STREET_1" : "rmtxadlsxqefdcwf","C_D_ID" : 1,"ORDERS" : [

{"ORDER_LINE" : [

{"OL_AMOUNT" : 0,"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z","OL_DIST_INFO" : "yptiwgjdelfxmathbjzirvye","OL_I_ID" : 35828,"OL_SUPPLY_W_ID" : 1,"OL_QUANTITY" : 5

},{

"OL_AMOUNT" : 0,"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z","OL_DIST_INFO" : "dxhqulhcgksjgqsicujzqhdb","OL_I_ID" : 26024,"OL_SUPPLY_W_ID" : 1,"OL_QUANTITY" : 5

},}

….

SELECT COUNT(my_order_line)AS total_orders,MAX(my_order_line.ol_delivery_d)AS max_delivery_date,

MAX(my_order_line.ol_quantity)AS max_order_quantity,MAX(my_orders.o_entry_d) AS max_customer_entry,MAX(my_orders.o_ol_cnt) AS max_orderline_entry,COUNT(customer)AS total_customers

FROM CUSTOMER MY_CUSTOMERUNNEST ORDERS AS my_ordersUNNEST my_orders.order_line AS my_order_line;


Named Prepared Statement

url="http://localhost:8093/query"s = requests.Session()s.keep_alive = Trues.auth = ('Administrator','password')

query = {'statement':’prepare select * from `beer-sample` where type=’beer’ and name = 'Guinness’}

r = s.post(url, data=query, stream=False)prepared = str(r.json()['results'][0]['name'])query = {'prepared': '"' + prepared + '"' }

for i in range (0, 5000):r = s.post(url, data=query, stream=False)print i, r.json()['metrics']['executionTime']


Named Prepared Statement

url="http://localhost:8093/query"s = requests.Session()s.keep_alive = Trues.auth = ('Administrator','password')

query = {'statement':'prepare select * from `beer-sample` where name = [$1]’}r = s.post(url, data=query, stream=False)prepared = str(r.json()['results'][0]['name'])

for i in range (0, 5):query={'prepared': '"' + prepared + '"', 'args': '["old_hat_brewery"]' }r = s.post(url, data=query, stream=False)print i, r.json()['metrics']['executionTime']

Prepare Once

Bind Values Many Times


Query Workbench – Query Editing

Query Editor

AutocompleteSyntax highlightingMulti-line formattingHistoryCreate, edit, save, run

Results

JSON / Table / Tree


Query Workbench – Schema Browsing

Schema

InferredDocument typesElement typesFrequencySample data

Bucket

SchemaIndexes


Query Workbench – Query Monitoring

Catalogs

VitalsActive RequestsPrepared StatementsCompleted Requests

Interfaces

Query WorkbenchCLI (Query Shell)REST API


Monitoring: Active requests

§ List / Delete requests currently being run by the query service§ Through N1QL– SELECT * FROM system:active_requests– DELETE FROM system:active_requests WHERE…

§ Through REST– GET http://localhost:8093/admin/active_requests– GET http://localhost:8093/admin/active_requests/<request_id>– DELETE http://localhost:8093/admin/active_requests/<request_id>


Monitoring: Prepared statements

§ List / Delete requests prepared on the query node§ Through N1QL– SELECT * FROM system:prepareds– DELETE FROM system:prepareds WHERE…

§ Through REST– GET http://localhost:8093/admin/prepareds– GET http://localhost:8093/admin/prepareds/<request_id>– DELETE http://localhost:8093/admin/prepareds/<request_id>


Monitoring: Completed requests

§ List / Delete completed requests deemed to be of high cost§ Through N1QL– SELECT * FROM system:completed_requests– DELETE FROM system:completed_requests where…

§ Through REST– GET http://localhost:8093/admin/completed_requests– GET http://localhost:8093/admin/completed_requests/<request_id>– DELETE http://localhost:8093/admin/completed_requests/<request_id>

§ Provide an overall health picture of the query service– Using REST: GET http://localhost:8093/admin/vitals

Lab Time!Labs 10-12

Couchbase MobileIntroduction

128


Key Challenges When Building Mobile Apps

According to study done by Bock & Company Enterprise Developers Rank Critical Requirements for Success in the Coming Mobile App Explosion


What is Couchbase Mobile

• Removed continual network dependency• Faster development cycles• Less long term maintenance than

traditional solutions

• Enterprise class mobile/embedded NoSQL database + sync platform

• Fast and consistent access to data

Built-in enterprise level security throughout the entire stack includes user authentication, user and role based data access control (RBAC), secure transport (TLS), and 256-bit AES full database

encryption.

SECURITY


Couchbase Lite Overview

§ Cross-platform support for all major operating systems and platforms

§ Built native from the ground up

§ 500kb for most platforms

§ 256-bit AES full database encryption

131

Couchbase LiteEmbedded Database


Sync Gateway Overview

§ Synchronize data between Couchbase Lite and Couchbase Server

§ REST, Stream, Batch, and Event APIs

§ Pluggable authentication

§ Fine grained user and role based access control

§ Elastically scalable in real-time

132

Sync GatewaySynchronization


Couchbase Server Overview

§ Scale easily to thousands of nodes

§ Consistent high performance that supports millions of concurrent users

§ Flexible JSON data model

§ 24x365 always-on availability

133

Couchbase ServerDatabase Server


End-to-End Security

ü Pluggable authentication

ü Fine grained user and role based access control

ü Transport layer security (TLS)

ü 256-bit AES full database encryption on device

ü File system encryption in the cloud Security


Who’s using Couchbase Mobile today


Common Mobile Use CasesCommon Mobile Use Cases


Offline Data Access

The Problem• When an app doesn’t work offline, users are cut off from important transactions

• Mobile apps need to work regardless of network connection

How Couchbase Mobile helps• Couchbase Lite stores data locally so your app isn’t dependent on the network

• If a network connection is lost, the database continues to operate and is automatically synced with the cloud when the network is restored


Local Caching

The Problem• Today’s mobile applications require fast access to data

• Slow access to data results in slow mobile apps, unhappy users and loss of productivity

How Couchbase Mobile helps• By storing and caching application data locally in the embedded Couchbase Lite

database, you eliminate the need to constantly request data from the server

• You can synchronize data automatically with Sync Gateway


Digitization of Paper Assets

The Problem• Reducing operational time and costs is critical for any business, and mobile

application deployments are a key component of making it happen

• By digitizing paper assets, you save time and resources, reduce storage, and cut logistics costs

How Couchbase Mobile helps• Store paper assets with Couchbase Mobile, both locally and in the cloud

• Your data is securely synced between devices and the cloud


Content Delivery & Storage

The Problem• Guaranteed delivery and storage of assets is crucial for a great user experience

• Users need real-time data updates

How Couchbase Mobile helps• Couchbase Mobile gives you the ability to store and access data locally so content is

always available

• Data can be synced between the cloud and device, allowing business-critical updates to be reflected in real-time


Real-time Asset Tracking

The Problem• Real-time asset tracking is a key requirement in mobile applications across many

industries

• Tracking, storing, and delivering real-time updates can be tricky to manage

How Couchbase Mobile helps• Asset tracking data can be shared with supervisors and/or across field employees

• Deliver updates on shipments, routing, service orders, ETAs, or anything else in real-time with Couchbase Mobile


Peer-to-Peer Usage

The Problem• Without a network connection, devices can’t talk to each other

• In isolated environments like retail stores, airplanes, the field, or any other places where the network isn’t reliable, this can cause lost revenue and productivity

How Couchbase Mobile helps• With Couchbase Mobile, connect devices directly to each other with peer-to-peer

replication

• Apps can connect to exchange data and will continue to work regardless of network availability


User Data/Preferences Storage

The Problem• Storing user data and preferences across devices and the cloud is complex for

games, CRM systems, e-commerce apps, or any other app

• Making sure user information is secure, both on-device and in transport, is difficult

How Couchbase Mobile helps• Couchbase Mobile gives your users a consistent experience across all of their

devices

• Securely store and sync user data like preferences, profiles, sign-in, game state and more across devices

Lab Time!Bonus Labs


Couchbase Server and Intel’s high-performance processors and storage subsystems combine to power millions of operations/sec for the world’s

largest customer 360 systems and profile stores, popular online games, and more. Additional information about Couchbase and Intel can be found here:

http://www.couchbase.com/partners/intel

Power with Intel


Consulting and Expert Services

Packaged Services– Quick Start– Architecture Reviews– Health Checks– Upgrade Services– Managed Services

Custom Consulting– Migration– Application Integration– Big-Data Integration, Analytics, Reporting– Others

Email inquiries: [email protected] OR Contact your Account Representative


Couchbase Hands-On Training

For System Admins:§ CS300 Couchbase NoSQL Server Administration, 4-day For Developers:§ CD220 Developing Couchbase NoSQL Applications, 3-dayFor Anyone:§ Free Online On-Demand Courses: http://training.couchbase.com/online

Email inquiries: [email protected]


Additional Resources

148

§ Getting Started: http://www.couchbase.com/download

§ Couchbase Labs: http://github.com/couchbaselabs§ Developer Portal: http://developer.couchbase.com§ General Docs: http://docs.couchbase.com§ Query Portal: http://query.couchbase.com

§ Blog: http://blog.couchbase.com§ Forum: http://forums.couchbase.com

§ Sample Applications:§ https://github.com/couchbaselabs?utf8=%E2%9C%93&query=try§ https://github.com/couchbaselabs?utf8=%E2%9C%93&query=beer

Technology

Couchbase Day