Couchbase Day

Wi-Fi: Network – Matrix | Password - 11223344

Today’s AgendaRegistration and Continental Breakfast§ Labs and Environment Setup

Introduction to NoSQL§ Why NoSQL?§ Introduction to Couchbase

Couchbase Architecture§ Single Node Operations§ Cluster Operations

HA and DR§ Availability and XDCR§ Backup/Restore§ Security12:00 – Lunch Break

Developing with Couchbase§ Couchbase SDKs

Couchbase Indexing§ Couchbase GSI and Views§ Views Overview

Indexing and Query§ Couchbase Indexes (GSI/MOI)§ Couchbase Query (N1QL)

Couchbase Mobile§ Overview

Adjourn§ Close

Why NoSQL?

Why companies need a new database solution

§ What are the biggest problems with Relational Databases that are driving adoption of NoSQL?

LACK OF FLEXIBILITY/ HAS RIGID SCHEMAS

INABILITY TO SCALE OUT

PERFORMANCE CHALLENGES

ALL OF THE ABOVE

DynamoOctober2007

CassandraAugust2008

BigtableNovember2006

VoldemortFebruary2009

Veryfeworganizationswantto(fewercan)buildandmaintaindatabasesoftwaretechnology.Buteveryorganizationparticipatinginthedigitaleconomyneedsthistechnology.

Why NoSQL?

New data solutions to address emerging needs:§ Performance and Scalability§ High Availability with Data Synchronization (multi-datacenter)§ Flexible Schema to accommodate frequent changes§ Transparent/Auto-Sharding without application participation§ Distributed System for Querying§ Managed Cache

NoSQL Database Scales OutCost and performance mirrors app tier

NoSQLDistributedDataStore

Web/AppServerTier

Application Scales OutJust add more commodity web servers

System CostApplication Performance

Application Performance System Cost

WhyNoSQL

How has Couchbase Server evolved beyond NoSQL?

§ Discreet, scalable services (data, index, query)§ Filtering for cross data center replication§ SQL-based query language and secondary indexing§ Geospatial search and indexing§ CRUD, prepared statements, and covering indexes§ Memory-optimized Indexes§ Array Indexing within Nested JSON§ Graphical Query W0rkbench§ Sub-Document API§ LDAP integration and auditing; RBAC, X-509 Certificates

Next Generation Data Management Architecture

DataSources

Enterprise Applications

Social MediaWeb &

EcommerceMobile AppsSensor Data Mainframe

External Data

System Logs

DataIntegration

Speed Layer Batch Layer

DataManagement

Data Lake Data Warehouse

Business Intelligence Dashboards Query ToolsDataAccess

In Memory Cache

Data Serving Layer

Next Generation Data Management Architecture

Enterprise Applications

Social MediaWeb &

EcommerceMobile AppsSensor Data MainframeExternal DataSystem Logs

Speed Layer Batch Layer

Data Lake

Business Intelligence Dashboards Query Tools

In Memory Cache

Data Warehouse

N1QL Query Workbench

DataSources

DataIntegration

DataManagement

DataAccess

Data Serving Layer

Real Time Data Ingestion

Couchbase Ecosystem

How Does Couchbase Fit In Big Data?

Couchbase and Hadoop are complementary.

Couchbase and Hadoop

New Data Stream

Merged View

All DataPrecompute

Views (Map Reduce)

Process Stream

Incremental Views

Partial Aggregate

Real-Time Data

BatchRecompute

Batch Views

Real-Time Views

Real-TimeIncrement

Batch Layer

Serving Layer

Speed Layer

Couchbase HadoopConnector (Sqoop)

Couchbase and Hadoop

New Data Stream

Merged View

All DataPrecompute

Views (Map Reduce)

Process Stream

Incremental Views

Partial Aggregate

Real-Time Data

BatchRecompute

Batch Views

Real-Time Views

Real-TimeIncrement

Batch Layer

Serving Layer

Speed Layer

Stream / Data Ingestion

Incremental Data / Stream

processing

Serving merged results /

responses

Couchbase Server on Docker and Orchestration Technologies Official Docker Support with MacOS, Ubuntu is coming

Couchbase & Containers

Platform as a Service

§ Couchbase runs on PaaS environments

Current Focus Future Plan

Infrastructure as a Service

Couchbase supports IaaS environments

§ Available in AWS marketplace

§ BYOL or hourly pricing

§ Available in Azure marketplace

§ BYOL or hourly pricing

§ Marketplace image in the works

§ BYOL at the moment§ Hourly pricing soon

Other providers supported

Couchbase OverviewWhat is Couchbase?

Couchbase Server Defined

§ Couchbase Server 4.X is the first NoSQL Database to enable you to develop with agility and operate at any scale.

Managed Cache Key-Value Store Document Database Embedded Database Sync Management

The Power of the Flexible JSON Schema

Ability to store data in multiple ways• Denormalized single document, as opposed to normalizing data across multiple table• Dynamic Schema to add new values when needed

Document Fundamentals - Composition

§ Similar to primary keys in relational databases§ Documents are partitioned based on the

document ID§ ID based document lookup is extremely fast § Must be unique

§ JSON§ Binary - integers, strings, booleans§ Common binary values include serialized objects,

compressed XML, compressed text, encrypted values

Document ID or Key

§ CAS Value (unique identifier for concurrency)§ TTL§ Flags (optional client library metadata)§ Revision #

Metadata

Document

Document ID / Key

MetadataValue

Storing And Retrieving Documents

Couchbase Data Access

• Everything is layered on top of Key Value

• A Document store is a special case of Key-Value

• Views provide aggregation and real-time analytics through incremental map-reduce

• Global Secondary Indexes provide low latency/high throughput indexes

• N1QL is a language that provides a powerful and expressive way of accessing documents

Couchbase ArchitectureSingle Node

Couchbase Architecture

• Data Service – builds and maintains Distributed secondary indexes (MapReduce Views)

• Indexing Engine – builds and maintains Global Secondary Indexes

• Query Engine – plans, coordinates, and executes queries against either Global or Distributed indexes

• Cluster Manager – configuration, heartbeat, statistics, RESTful Management interface

Couchbase Read Operation

APPLICATION SERVER

MANAGED CACHE

DISKQUEUE

REPLICATIONQUEUE

GETDOC 1

APPLICATION SERVER

MANAGED CACHE

DISKQUEUE

REPLICATIONQUEUE

Write Operation

DOC 1DOC 1

Cache Ejection

APPLICATION SERVER

MANAGED CACHE

DISKQUEUE

REPLICATIONQUEUE

DOC 2DOC 3DOC 4DOC 5

DOC 2 DOC 3 DOC 4 DOC 5

APPLICATION SERVER

MANAGED CACHE

DISKQUEUE

REPLICATIONQUEUE

Cache Miss

DOC 2 DOC 3 DOC 4 DOC 5

GETDOC 1

Couchbase ArchitectureCluster

Auto sharding – Bucket and vBuckets

Data buckets

1 ….. 1024

Active Virtual buckets

1 ….. 1024

Replica Virtual buckets

Cluster Map

Couchbase Cluster

CLUSTER MAP

Hashing Algorithm

vBucket2

vBucket3

vBucket4

vBucket5

vBucket6

vBucket7

vBucket1024

vBucket1

Couchbase SDK

Couchbase Cluster

CLUSTER MAP

Hashing Algorithm

vBucket2

vBucket3

vBucket4

vBucket5

vBucket6

vBucket7

vBucket1024

vBucket1

Couchbase SDK

ACTIVE ACTIVE ACTIVE

REPLICA REPLICA REPLICA

Couchbase Server 1 Couchbase Server 2 Couchbase Server 3

Basic Operation

SHARD5

SHARD2

SHARD9

SHARD SHARD SHARD

SHARD4

SHARD7

SHARD8

SHARD SHARD SHARD

SHARD1

SHARD3

SHARD6

SHARD SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD SHARD

SHARD6

SHARD3

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD SHARD

Rebalance

ACTIVE ACTIVE

REPLICA REPLICA

Couchbase Server 4 Couchbase Server 5

SHARD5

SHARD2

SHARD SHARD

SHARD4

SHARD SHARD

SHARD1

SHARD3

SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD SHARD

SHARD6

SHARD3

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD SHARD

SHARD7

SHARD6

SHARD8

SHARD9

READ/WRITE/UPDATE

Fail Over

ACTIVE ACTIVE

REPLICA REPLICA

SHARD5

SHARD2

SHARD SHARD

SHARD4

SHARD SHARD

SHARD1

SHARD3

SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD

SHARDSHARD6

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD

SHARD7

SHARD6

SHARDSHARD8

SHARD9

SHARD3

SHARD1

SHARD3

High Availability with Couchbase

Recovering from a Fail Over

ACTIVE ACTIVE

REPLICA REPLICA

SHARD5

SHARD2

SHARD SHARD

SHARD4

SHARD SHARD

SHARD1

SHARD3

SHARD SHARD

SHARD4

SHARD1

SHARD8

SHARD SHARD

SHARDSHARD6

SHARD2

SHARD SHARD SHARD

SHARD7

SHARD9

SHARD5

SHARD SHARD

SHARD7

SHARD6

SHARDSHARD8

SHARD9

SHARD3

SHARD1

SHARD3

Recovering from Failover

Three options:

1. Remove the node

2. Full recovery

3. Delta node recovery

Delta Node Recovery

Recover Failed Nodes with incremental Catch-up

§ Improve recovery time for nodes with large datasets

§ Add a failed node into cluster and recover data from where it left off

Rack Zone Awareness – Server Groups

Informs Couchbase as to the physical distribution of nodes in

racks or availability groups.

The Active vBuckets in Server 1will have its replica’s in a different group. Never in the same group.

Couchbase Server 3

Couchbase Server 2

Couchbase Server 6

Couchbase Server 5

Couchbase Server 4

Couchbase Server 9

Couchbase Server 8

Couchbase Server 7

Ensures that replica vBuckets are distributed across groups

REPLICA

Group: A Nodes 1,2,3

Group: B Nodes 4,5,6

Group: C Nodes 7,8,9

Couchbase Server 1

ACTIVE

Lab Time!Labs 1-4

Cross Datacenter Replication (XDCR)

TO OTHER NODE

APPLICATION SERVER

MANAGED CACHE

DISKQUEUE

REPLICATIONQUEUE

XDCR QUEUE

TO OTHER CLUSTER

DOC 1DOC 1DOC 1

TO OTHER NODE

APPLICATION SERVER

MANAGED CACHE

DISKQUEUE

REPLICATIONQUEUE

XDCR QUEUE

TO OTHER CLUSTER

DOC 1DOC 1DOC 1

Single-node type means easier administration and scaling

CLUSTER REPLICATION

vBucket REPLICAS

XDCR: Cluster Topology Aware

• Applications can access both Clusters (active-active)

• Can provide filtered replication

• One-one, one-many, many-one• Differently sized clusters• Different HW platforms • Mixed versions supported (Upgrades)

Unidirectional or Bidirectional Replication

Unidirectional§ Hot spare / Disaster Recovery§ Development/Testing copies§ Connectors (Solr, Elasticsearch) § Integrate to custom consumer

Bidirectional§ Multiple Active Masters§ Disaster Recovery§ Datacenter Locality

Backup and Restore

Key Features

§ Zero downtime backup and restore§ Single utility to manage multiple clusters§ Restore from any point, to any bucket or topology§ Fully differential – merge backups to maintain desired restore points§ Built-in concurrency control§ Resume interrupted backups§ Compaction reduces storage requirements

Enterprise Backup and Restore

§ >10x Faster Backups§ Better Data Protection with Enterprise Backup Restore– Scale to larger datasets– Fast backups and restores– Better Consistency in backup data– Resilient to failures/Topology Changes

4.5 Backup Speed 4.1 Backup Speed

Data Import

§ Create backup repo/opt/couchbase/bin/cbbackupmgr config --archive /tmp/backup --repo demo

§ Import of sample data/opt/couchbase/bin/cbbackupmgr json --host couchbase://172.23.107.27:8091 --username Administrator --password password --bucket travel-sample --dataset /opt/couchbase/samples/travel-sample.zip --format sample

§ Import of list data/opt/couchbase/bin/cbbackupmgr json --host couchbase://172.23.107.27:8091 --username Administrator --password password --bucket chicago --dataset file:///root/data/companies.json --format list --generate-key %permalink%::#MONO_INCR#

Security

Security with Couchbase

§ Administrative Security:§ Admin user (full access) vs. Read-only user (monitoring, developer access)§ SSL encryption to REST API and Web UI§ HTTP access log

§ Data Security:§ Applications connect via SASL with single user/pass§ Data-at-Rest encryption via partnership with Vormetric§ SSL encryption for over-the-wire

§ Couchbase 4.0:§ LDAP integration§ More extensive administrative action auditing

More at: http://docs.couchbase.com/admin/admin/security/security-best-practices.html

Couchbase authentication overview

§ Application authentication § Buckets are protected with challenge-response SASL protocol§ AuthN happens over CRAM-MD5

§ Admin authentication § Authentication through admin username and password§ Authentication through LDAP (New in 4.0)

AUTHENTICATION

Couchbase authorization overview

§ Application data access § Full access to the bucket application is connected to

§ Admin access§ Full administrator has full privileges on the cluster § Read-only administrator cannot change cluster settings

AUTHORIZATION

Couchbase encryption overview

§ Data-in-motion encryption§ Client-server communication can be encrypted using SSL § Secure admin access using SSL over port 18091§ Secure view access using SSL over port 18092§ Secure XDCR for encryption across datacenters

Track all Access SERVER3SERVER1 SERVER2CouchbaseServer– NewYork

SERVER3SERVER1 SERVER2CouchbaseServer– London

Client applications

SecureXDCR over

Admin access

over port 18091

View access

over port 18092

https://couchbase_server:18091/…

https://couchbase_server:18092/…

ENCRYPTION

Couchbase encryption overview

§ Transparent data-at-rest encryption solution ENCRYPTION

Storage

Database

Application

File Systems

VolumeManagers

VormetricData Security Manager

on Enterprise premise or in cloudvirtual or physical appliance

• Centrally manage keys and policy• Virtual and physical appliance • High-availability with cluster• Multi-tenant and strong separation of duties• Proven 10,000+ device and key management scale• Web, CLI, API Interfaces• FIPS 140-2 certified

Secure Personally Identifiable Information• User profile information• Login Credentials• IP Addresses

External identity management using LDAP

Centralized identity management § Define multiple read-only admins and full-admins

§ Centralized security policy management for admin accounts for stronger passwords, password rotation, and auto lockouts

Individual accountability. Simplified compliance.§ Define UIDs in LDAP, and map UIDs to read-only / full admin role in

Couchbase

§ Comprehensive audit trails with LDAP UIDs in audit records

New UI for authorizing LDAP administrators

Turn on/off LDAP

Add UIDs to read-only admins

Add UIDs to full admins

Set default behavior if UID is not mapped

Testing credentials to verify what level

of access

Plus REST, and CLI integration for programmatic setup

Admin Auditing in Couchbase

Rich audit events§ Over 25 different, detailed admin audit events

§ Auditing for tools including backup

Configurable auditing § Configurable file target

§ Support for time based log rotation and audit filtering

Easy integration§ JSON format allows for easy integration with downstream

systems using flume, logstash, and syslogd

Auditing a successful login

{"timestamp":"2015-02-20T08:48:49.408-08:00", "id":8192, "name":"login success", "description":"Successful login to couchbase cluster", "role":"admin", "real_userid": {

"source":"ns_server","user":"bjones”

},"sessionid":"0fd0b5305d1561ca2b10f9d795819b2e", "remote":{"ip":"172.23.107.165", "port":59383}

Lab Time!Labs 5-7

Couchbase SDK

Languages and Interfaces for Couchbase

§ Official SDKs– Java– .NET– Node.js– Python

§ For each of these we have– Full Document support– Interoperability– Common Programming Model

Others: Erlang, Perl, TCL, Clojure, Scala

- PHP- C- Go- Ruby

JDBC and ODBC

Interacting with Data

Cluster

Bucket

CRUDView

QueryN1QL Query

Couchbase SDKs

§ What does it mean to be a Couchbase SDK?

Cluster

Bucket

FunctionalGive the application developer a concurrent API for basic (k-v) or document management

APIget()insert()replace()upsert()remove()

The Document

§ Documents are integral to the SDKs.§ All SDKs support JSON format§ In addition: Serialized objects, Unquoted Strings, Binary pass-through§ A Document contains:

Property Description

ID The bucket-unique identifier

Content The value that is stored

Expiry An expiration time

CAS Check-and-Set identifier

Bucket API – Modifying Documents

§ insert() the document if it does not exist§ replace() the document if it does exist§ upsert() the document (insert or replace)§ remove() the document

§ append() data to the document§ prepend() data to the document

§ counter() for increment/decrement type operations

Bucket API – Retrieving Documents

§ get() the document§ getFromReplica() if the master is not available

§ getAndTouch() to load the document and reset the expiry§ getAndLock() to load the document and write-lock it

Unified API – DML, CAS Example1. Two Clients retrieve the same document "XYZ"2. Client A retrieves it first.3. Client B then retrieves XYZ. Both clients will have the same CAS value for document XYZ4. Client B tries to perform an update to document XYZ. The update succeeds as the CAS value

was unchanged from when Client B initially retrieved the document. Once the update succeeds, the CAS value for XYZ changes.

5. Client A then tries to perform an update on XYZ immediately after Client B. The update will fail as Client A's CAS value is out of date. When Client B updated XYZ, the CAS value changed.

Durability Options for writes

APPLICATION SERVER

XDCR QUEUETO OTHER CLUSTER

DISK DISKQUEUE

REPLICATIONQUEUE

APPLICATION SERVER

DISK DISKQUEUE

CACHEREPLICATION

DISKQUEUE

DOCDOC DOCDOCDOC DOC

CLUSTER 1NODE 2

CLUSTER 2NODE …

CLUSTER 1NODE 1

( PUT ) By default, a write operation is successful when the data has been written to the memory of the node with the active vBucket. Eventually (but pretty quickly), the data will also be written to persistent storage and to replica vBuckets.

( replicateTo ) write operation is successful when data has been written to both the active vBucket and replica vBuckets.

( persistTo ) write operation is successful when data has been written to persistent storage• Can be set to require writes to persistent storage for replicas.• Using persistTo will slow performance.

Bucket API – Querying

§ query() is possible for§ Views§ N1QL

§ Streams N response rows as they arrive from the server§ Loads more than one Document based on Criteria§ Typically used to satisfy secondary and advanced querying use cases

Lab Time!Labs 8-10

GSI and VIEWS

Which to choose – GSI vs Views

Workloads New GSI in v4.X Map/Reduce Views

ComplexReporting

Just In Time Aggregation Pre-aggregated

Ad-hoc Querying

Ad-hoc View per Query

Flexible Indexing Logic

N1QL Functions Javascript

Secondary Lookups

Faster with Single Node Lookup

Slower due to Scatter-Gather

Range Scans

Faster with Single Node Lookup

Slower due to Scatter-Gather

Comparing GSI and ViewsCapabilities Global Secondary Indexes Map/Reduce Views

Partitioning Model Independent – Indexing Service Aligned to Data – Data Service

Scale Model Independently Scale Index Scale with Data Service

Secondary Lookup Single Node Scatter-Gather

Range Scan Single Node Scatter-Gather

Grouping, Aggregates With N1QL Built-in with Views API

Caching Managed Not Managed

Storage ForestDB Couchstore

Availability Identical Indexes load balanced Replica Based

Indexing Differences

§ Local secondary indexes (Views)– Co-located with data– Higher write performance / Lower read performance: scatter-gather

– Map/Reduce Views: programmable indexer for complex reporting and indexing logic.

– Spatial Views: Incremental R-tree indexing for powerful bounding-box queries– Full partition alignment and paired scalability with Data Service

§ Global secondary indexes (GSI / MOI) – Independently partitioned and independently scalable indexes in Indexing Service– Isolated from Key-Value operations– Low latency queries without compromising on mutation performance (insert/update/delete)– Higher query performance– Async writes to a large number of global indexes– Memory Optimized Indexes (MOI) alternate storage option to standard GSI - small memory

footprint, optimized for lowest latency queries

Indexing with N1QL

Indexing in N1QLGlobal and Local Indexes

Indexing Types for N1QL

View Based Indexes• Local Indexes:

• Partitioned Aligned to Data• Auto Partitioning• Built in Replicas• Simple Scaling Model

Global Secondary Indexes (GSI)• Global Indexes:

• Independently Partitioned Indexes• Predictable High Performance at Scale• Sophisticated Scaling Model

Controlled with USING VIEW|GSI in CREATE INDEX. Set per Index. Drop and Recreate the index using DROP INDEX & CREATE INDEX to change Index Type.

A little about Views…

§ Using views you can define– Primary indexes

– Simple secondary indexes (most common use case)

– Complex secondary, tertiary and composite indexes

– Aggregations (reduction)

§ Built using MapReduce technology – Map and Reduce functions are written in Javascript

A little about Views…

§ Captures user-defined key/value pairs about a document into an index§ Can execute logic on data during indexing§ Able to express relationships too cumbersome to maintain in Key/Value§ Incremental indexing essentially pre-computes query results§ Higher latency compared to key/value lookups

§ Common Use Cases– Materialized view: project an alternate view of your document– Complex aggregates– Lightweight reporting

Queries run against stale indexes by default

§ stale=update_after (default if nothing is specified)• always get fastest response

• can take two queries to read your own writes

§ stale=ok• auto update will trigger eventually

• might not see your own writes for a few minutes

• least frequent updates -> least resource impact

§ stale=false• Use with “set with persistence” if data needs to be included in view results

• BUT be aware of delay it adds, only use when really required

Lightweight Reporting Example

Document:{“date”: “2015-08-03T16:38:05.835Z”,“source”: “192.168.56.101”,“logtype”: “error”,“message”: “An error occurred during startup”

Map:function(doc, meta) {

emit(dateToArray(doc.date), doc.logtype);}

function(key, values, rereduce) {

var response = {"warning" : 0, "error": 0, "fatal" : 0 };

for(i=0; i<values.length; i++) {if (rereduce) {

response.warning = response.warning + values[i].warning;response.error = response.error + values[i].error;

response.fatal = response.fatal + values[i].fatal;

} else {if (values[i] == "warning")

response.warning++;if (values[i] == "error" )

response.error++;

if (values[i] == "fatal" )response.fatal++;

return response;

Query: ?group_level=2&startkey=[2010,7]&endkey=[2011,4]

{"rows":[{"key":[2010,7], "value":{"warning":4,"error":2,"fatal":0}},{"key":[2010,8], "value":{"warning":4,"error":3,"fatal":0}},{"key":[2010,9], "value":{"warning":4,"error":6,"fatal":0}},{"key":[2010,10],"value":{"warning":7,"error":6,"fatal":0}},{"key":[2010,11],"value":{"warning":5,"error":8,"fatal":0}},{"key":[2010,12],"value":{"warning":2,"error":2,"fatal":0}},{"key":[2011,1], "value":{"warning":5,"error":1,"fatal":0}},{"key":[2011,2], "value":{"warning":3,"error":5,"fatal":0}},{"key":[2011,3], "value":{"warning":4,"error":4,"fatal":0}},{"key":[2011,4], "value":{"warning":3,"error":6,"fatal":0}}]

Indexing

Data Service

Projector & Router

Indexing Service

Query ServiceIndex Service

SupervisorIndex maintenance &

Scan coordinator

Index#2Index#1

Query Processorcbq-engine

Bucket#1 Bucket#2

DCP StreamIndex#4Index#3

Bucket#2

Bucket#1

Projector and Router: 1 Projector and Router per node1 stream of changes per bucket per supervisor

ForestDBStorage Engine

Supervisor1 Supervisor per nodeMany indexes per Supervisor

MemDB Storage Engine for MOI

Types of Indexes

§ Composite Indexes§ Covering Indexes§ Filtered Indexes§ Function Based Indexes§ Sub-Document Indexes§ Incremental MapReduce Views§ Spatial Views§ Full Text Indexes§ Prepared Statements§ N1QL Complete SQL CRUD support: Insert, Delete, Merge, Etc.

create index idx_odspilot on ods(DISTINCT ARRAY p.FILEN in p in PILOT END);

Array Indexing

§ Index only required elements or attributes in the array

§ Benefits are lot more significant for nested arrays/objects

§ Efficient on Index storage & search time

Transportation Company POC• 422,137 documents.• Query2: BEFORE array indexing • Primary index scan• 38.91 seconds.

• Query2: AFTER array indexing • Array index scan [DistinctScan]• 8.51 millisecond• Improvement of 4572 Times

Step by Step – Building indexes

§ Building and Index1. Issue “CREATE INDEX …”2. Place on the “next node” if WITH {“nodes”:[]}not specified – “next node” picked by round robin3. Create metadata for the index on the node– “next node” picked by round robin4. If deferred build = true– Mark Index Status = “Pending” – Wait for “BUILD INDEX ...”5. Else– Mark Index Status = “Pending” – Start DCP Backfill with Projector– When Done – Mark Index Status = “Online”

Step by Step – Ingest a Mutation

§ Ingest a Mutation on Data1. DCP Feed communicates the mutation to Projector on the node.2. Projector picks up the mutation for a bucket– Projector strips the mutation to check relevance to existing indexes on each indexer– Projector Sends stripped mutation to indexers on all index service nodes3. Indexer receive the mutation on each index service node– Queue the mutation– Extraction Worker Prepares and queues the update to each index.– Persistence Worker persist the mutation to the index

Global Secondary Indexes – Storage Options

Global Secondary IndexPredictable low latency with Query and Key Based Operations,

even in presence of large number of indexes

Storage Option for GSI

Standard Global Secondary Indexes• Optimized for Storage + Memory: efficient spill-

to-disk with ForestDB as the storage engine• Advanced IO Performance: New circular-writes

that minimize compaction overhead

Memory-Optimized Global Secondary Indexes (MOI)• Optimized for Memory: small memory footprint,

optimized for lowest latency queries• Faster Indexing: fresh indexes under heavy mutations with

lock free index maintenance

Controlled with Cluster Settings (Settings Tab). Cluster-wide Setting for all Indexes and Require removing all GSI indexes and index service nodes to change storage options.

(New in 4.5)

Memory Optimized (MOI) Global Secondary Indexes

• Optimized for Memory: small memory footprint, optimized for lowest latency queries

• Faster Indexing: fresh indexes under heavy mutations with lock free index maintenance

Which to Choose – Standard vs Memory Optimized GSI

Standard GSI Memory Optimized GSI

Initial Index Build Times Slower Faster

Ongoing Index Build Times

Slower Faster

Ability to Keep up with Mutation Rate

Lower Higher

Memory Demand for Larger Indexes

Lower Higher

Behavior on OOM(out of memory)

Spill to disk Stop processing mutations

Indexing & HA

§ Mirrored Indexes are load balanced and used for HA– Size at least 2 indexes for production– Add more indexes if all requests cannot be handled by a single index to protect against

failures

Indexes & Partitioning

§ Why Partition Indexes?– Index may not fit on the node– Fit index to the nodes you have– Distribute Index load to many nodes– Minimize the tree scanned– Large/Tall indexes will take longer to scan

§ How to Partition Indexes?– By list of values – great for scan latency– ... WHERE type = “customer” – ... WHERE type = “product”– By range –– ... WHERE id BETWEEN 10 AND 20– ... WHERE id BETWEEN 20 AND 30– By function – no range scan ability

Efficient IO with Circular Writes in Standard GSI

Standard Global Secondary Indexes§ Reduced Disk IO Requirements– Append-Only Writes with frequent full compaction (Version 4.1 & Earlier)– Circular-Reuse Writes with reduced full compactions (New in 4.5: )– Reused orphaned blocks in the index file– Reduce the need for frequent full-compactions of the index file

Index Mutation

Append-Only WriteCircular Write

File for Global Secondary Indexes

Index Mutation

N1QL - Query

What is N1QL?

§ N1QL is a new query language that systematically extends SQL to document-oriented data

§ N1QL extends SQL to handle data that is:– Nested: Contains nested objects, arrays

– Heterogeneous: Schema-optional, non-uniform

– Distributed: Partitioned across a cluster

N1QL (EXPRESSIVE)

§ Access to every part of JSON document§ Scalar & Aggregate functions § Issue subquery in any expressions§ Subqueries§ Subqueries in the FROM clause

Give developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data.

N1QL (POWERFUL)

§ Access to every part of JSON document§ JOINS, Aggregations, standard scalar functions§ Aggregation on arrays§ NEST & UNNEST operations§ Covering Index

N1QL (QUERYING)

§ INSERT§ UPDATE§ DELETE§ MERGE§ SELECT§ EXPLAIN

N1QL (TRANSFORMING & MANIPULATING)

§ Full Transformation of the data via Query.§ INSERT§ INSERT single & multiple documents§ INSERT result a SELECT statement§ DELETE documents based on complex filter§ UPDATE any part of JSON document & use complex filter.§ MERGE two sets of documents using traditional MERGE statement§ SUBQUERIES

Optimizing Query Speed

§ There are four options for N1QL queries:

Primary Index

Secondary Index(GSI or MOI)

Covered Index

“ON KEYS”

Query Execution Flow

1. Application submits N1QL query

2. Query is parsed, analyzed and plan is created

3. Query Service makes request to Index Service

4. Index Service returns document keys and data

5. If Covering Index, skip step 6

6. If filtering is required, fetch documents from Data Service56

7. Apply final logic (e.g. SORT, ORDER BY)

8. Return formatted results to application

Inside a Query Service (without covering index)

Client

FetchParse Plan Join FilterPre-Aggregate

Offset Limit ProjectSortAggregateScan

Query Service

Index Servic

Data Servic

Data-parallel — Query is N data streams over N cores*

Memory-based

Pluggable architecture — datastore, index…

Request Response

Query Consistency

Strict Request-Time Consistency

(request_plus)Query execution is delayed until all indexes process mutations up to T3

RYOW Consistency(at_plus)

Query execution is delayed until all indexes process mutations up to T1

T1: insert (k1, v1)

T3: issue query on (k1,v1)

T2: do other business logic computation

Joins (Referential)

Document-oriented Extensions

§ Nested

– Paths—user.profile.email, children[0], children[0:2]

– NEST, UNNEST

– Ranging—EVERY child.age > 5 FOR child IN children END

– Transformations—SELECT {“name”: first_name || last_name}

§ Non-uniform

– IS MISSING

– Type checking and conversion

§ Distributed– Direct lookup—USE KEYS

– Efficient joins—ON KEYS

SELECT statement

SELECT customers.id,customers.NAME.lastname, customers.NAME.firstnameSum(orderline.amount)

FROM orders UNNEST orders.lineitems AS orderline JOIN customers ON KEYS orders.custid

WHERE customers.state = 'NY'GROUP BY customers.id,

customers.NAME.lastnameHAVING sum(orderline.amount) > 10000 ORDER BY sum(orderline.amount) DESC

Dotted sub-document referenceNames are CASE-SENSITIVE

UNNEST to flatten the arrays

JOINS with Document KEY of customers

UNNEST example: Customer Document{

"C_ZIP" : "828011111","C_STATE" : "vt",

"C_FIRST" : "ykfdbqku","C_CREDIT" : "GC","C_DELIVERY_CNT" : 0,"C_W_ID" : 1,"C_CITY" : "quhpismkzumehqhr","C_STREET_1" : "rmtxadlsxqefdcwf","C_D_ID" : 1,"ORDERS" : [

{"ORDER_LINE" : [

{"OL_AMOUNT" : 0,"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z","OL_DIST_INFO" : "yptiwgjdelfxmathbjzirvye","OL_I_ID" : 35828,"OL_SUPPLY_W_ID" : 1,"OL_QUANTITY" : 5

"OL_AMOUNT" : 0,"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z","OL_DIST_INFO" : "dxhqulhcgksjgqsicujzqhdb","OL_I_ID" : 26024,"OL_SUPPLY_W_ID" : 1,"OL_QUANTITY" : 5

UNNEST example: N1QL Code{

"C_ZIP" : "828011111","C_STATE" : "vt",

"C_FIRST" : "ykfdbqku","C_CREDIT" : "GC","C_DELIVERY_CNT" : 0,"C_W_ID" : 1,"C_CITY" : "quhpismkzumehqhr","C_STREET_1" : "rmtxadlsxqefdcwf","C_D_ID" : 1,"ORDERS" : [

{"ORDER_LINE" : [

{"OL_AMOUNT" : 0,"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z","OL_DIST_INFO" : "yptiwgjdelfxmathbjzirvye","OL_I_ID" : 35828,"OL_SUPPLY_W_ID" : 1,"OL_QUANTITY" : 5

"OL_AMOUNT" : 0,"OL_DELIVERY_D" : "2015-02-11T14:55:25.480Z","OL_DIST_INFO" : "dxhqulhcgksjgqsicujzqhdb","OL_I_ID" : 26024,"OL_SUPPLY_W_ID" : 1,"OL_QUANTITY" : 5

SELECT COUNT(my_order_line)AS total_orders,MAX(my_order_line.ol_delivery_d)AS max_delivery_date,

MAX(my_order_line.ol_quantity)AS max_order_quantity,MAX(my_orders.o_entry_d) AS max_customer_entry,MAX(my_orders.o_ol_cnt) AS max_orderline_entry,COUNT(customer)AS total_customers

FROM CUSTOMER MY_CUSTOMERUNNEST ORDERS AS my_ordersUNNEST my_orders.order_line AS my_order_line;

Named Prepared Statement

url="http://localhost:8093/query"s = requests.Session()s.keep_alive = Trues.auth = ('Administrator','password')

query = {'statement':’prepare select * from `beer-sample` where type=’beer’ and name = 'Guinness’}

r = s.post(url, data=query, stream=False)prepared = str(r.json()['results'][0]['name'])query = {'prepared': '"' + prepared + '"' }

for i in range (0, 5000):r = s.post(url, data=query, stream=False)print i, r.json()['metrics']['executionTime']

Named Prepared Statement

url="http://localhost:8093/query"s = requests.Session()s.keep_alive = Trues.auth = ('Administrator','password')

query = {'statement':'prepare select * from `beer-sample` where name = [$1]’}r = s.post(url, data=query, stream=False)prepared = str(r.json()['results'][0]['name'])

for i in range (0, 5):query={'prepared': '"' + prepared + '"', 'args': '["old_hat_brewery"]' }r = s.post(url, data=query, stream=False)print i, r.json()['metrics']['executionTime']

Prepare Once

Bind Values Many Times

Query Workbench – Query Editing

Query Editor

AutocompleteSyntax highlightingMulti-line formattingHistoryCreate, edit, save, run

Results

JSON / Table / Tree

Query Workbench – Schema Browsing

Schema

InferredDocument typesElement typesFrequencySample data

Bucket

SchemaIndexes

Query Workbench – Query Monitoring

Catalogs

VitalsActive RequestsPrepared StatementsCompleted Requests

Interfaces

Query WorkbenchCLI (Query Shell)REST API

Monitoring: Active requests

§ List / Delete requests currently being run by the query service§ Through N1QL– SELECT * FROM system:active_requests– DELETE FROM system:active_requests WHERE…

§ Through REST– GET http://localhost:8093/admin/active_requests– GET http://localhost:8093/admin/active_requests/<request_id>– DELETE http://localhost:8093/admin/active_requests/<request_id>

Monitoring: Prepared statements

§ List / Delete requests prepared on the query node§ Through N1QL– SELECT * FROM system:prepareds– DELETE FROM system:prepareds WHERE…

§ Through REST– GET http://localhost:8093/admin/prepareds– GET http://localhost:8093/admin/prepareds/<request_id>– DELETE http://localhost:8093/admin/prepareds/<request_id>

Monitoring: Completed requests

§ List / Delete completed requests deemed to be of high cost§ Through N1QL– SELECT * FROM system:completed_requests– DELETE FROM system:completed_requests where…

§ Through REST– GET http://localhost:8093/admin/completed_requests– GET http://localhost:8093/admin/completed_requests/<request_id>– DELETE http://localhost:8093/admin/completed_requests/<request_id>

§ Provide an overall health picture of the query service– Using REST: GET http://localhost:8093/admin/vitals

Lab Time!Labs 10-12

Couchbase MobileIntroduction

Key Challenges When Building Mobile Apps

According to study done by Bock & Company Enterprise Developers Rank Critical Requirements for Success in the Coming Mobile App Explosion

What is Couchbase Mobile

• Removed continual network dependency• Faster development cycles• Less long term maintenance than

traditional solutions

• Enterprise class mobile/embedded NoSQL database + sync platform

• Fast and consistent access to data

Built-in enterprise level security throughout the entire stack includes user authentication, user and role based data access control (RBAC), secure transport (TLS), and 256-bit AES full database

encryption.

SECURITY

Couchbase Lite Overview

§ Cross-platform support for all major operating systems and platforms

§ Built native from the ground up

§ 500kb for most platforms

§ 256-bit AES full database encryption

Couchbase LiteEmbedded Database

Sync Gateway Overview

§ Synchronize data between Couchbase Lite and Couchbase Server

§ REST, Stream, Batch, and Event APIs

§ Pluggable authentication

§ Fine grained user and role based access control

§ Elastically scalable in real-time

Sync GatewaySynchronization

Couchbase Server Overview

§ Scale easily to thousands of nodes

§ Consistent high performance that supports millions of concurrent users

§ Flexible JSON data model

§ 24x365 always-on availability

Couchbase ServerDatabase Server

End-to-End Security

ü Pluggable authentication

ü Fine grained user and role based access control

ü Transport layer security (TLS)

ü 256-bit AES full database encryption on device

ü File system encryption in the cloud Security

Who’s using Couchbase Mobile today

Common Mobile Use CasesCommon Mobile Use Cases

Offline Data Access

The Problem• When an app doesn’t work offline, users are cut off from important transactions

• Mobile apps need to work regardless of network connection

How Couchbase Mobile helps• Couchbase Lite stores data locally so your app isn’t dependent on the network

• If a network connection is lost, the database continues to operate and is automatically synced with the cloud when the network is restored

Local Caching

The Problem• Today’s mobile applications require fast access to data

• Slow access to data results in slow mobile apps, unhappy users and loss of productivity

How Couchbase Mobile helps• By storing and caching application data locally in the embedded Couchbase Lite

database, you eliminate the need to constantly request data from the server

• You can synchronize data automatically with Sync Gateway

Digitization of Paper Assets

The Problem• Reducing operational time and costs is critical for any business, and mobile

application deployments are a key component of making it happen

• By digitizing paper assets, you save time and resources, reduce storage, and cut logistics costs

How Couchbase Mobile helps• Store paper assets with Couchbase Mobile, both locally and in the cloud

• Your data is securely synced between devices and the cloud

Content Delivery & Storage

The Problem• Guaranteed delivery and storage of assets is crucial for a great user experience

• Users need real-time data updates

How Couchbase Mobile helps• Couchbase Mobile gives you the ability to store and access data locally so content is

always available

• Data can be synced between the cloud and device, allowing business-critical updates to be reflected in real-time

Real-time Asset Tracking

The Problem• Real-time asset tracking is a key requirement in mobile applications across many

industries

• Tracking, storing, and delivering real-time updates can be tricky to manage

How Couchbase Mobile helps• Asset tracking data can be shared with supervisors and/or across field employees

• Deliver updates on shipments, routing, service orders, ETAs, or anything else in real-time with Couchbase Mobile

Peer-to-Peer Usage

The Problem• Without a network connection, devices can’t talk to each other

• In isolated environments like retail stores, airplanes, the field, or any other places where the network isn’t reliable, this can cause lost revenue and productivity

How Couchbase Mobile helps• With Couchbase Mobile, connect devices directly to each other with peer-to-peer

replication

• Apps can connect to exchange data and will continue to work regardless of network availability

User Data/Preferences Storage

The Problem• Storing user data and preferences across devices and the cloud is complex for

games, CRM systems, e-commerce apps, or any other app

• Making sure user information is secure, both on-device and in transport, is difficult

How Couchbase Mobile helps• Couchbase Mobile gives your users a consistent experience across all of their

devices

• Securely store and sync user data like preferences, profiles, sign-in, game state and more across devices

Lab Time!Bonus Labs

Couchbase Server and Intel’s high-performance processors and storage subsystems combine to power millions of operations/sec for the world’s

largest customer 360 systems and profile stores, popular online games, and more. Additional information about Couchbase and Intel can be found here:

http://www.couchbase.com/partners/intel

Power with Intel

Consulting and Expert Services

Packaged Services– Quick Start– Architecture Reviews– Health Checks– Upgrade Services– Managed Services

Custom Consulting– Migration– Application Integration– Big-Data Integration, Analytics, Reporting– Others

Email inquiries: Services@couchbase.com OR Contact your Account Representative

Couchbase Hands-On Training

For System Admins:§ CS300 Couchbase NoSQL Server Administration, 4-day For Developers:§ CD220 Developing Couchbase NoSQL Applications, 3-dayFor Anyone:§ Free Online On-Demand Courses: http://training.couchbase.com/online

Email inquiries: training@couchbase.com

Additional Resources

§ Getting Started: http://www.couchbase.com/download

§ Couchbase Labs: http://github.com/couchbaselabs§ Developer Portal: http://developer.couchbase.com§ General Docs: http://docs.couchbase.com§ Query Portal: http://query.couchbase.com

§ Blog: http://blog.couchbase.com§ Forum: http://forums.couchbase.com

§ Sample Applications:§ https://github.com/couchbaselabs?utf8=%E2%9C%93&query=try§ https://github.com/couchbaselabs?utf8=%E2%9C%93&query=beer

Couchbase Day

Technology

Tunable Memory in Couchbase Server 3.0: Couchbase Connect 2014

Couchbase 3.0 Beta Webinar: Simplified administration with Couchbase 3.0

Couchbase Chennai Meetup 2 - Couchbase - Mobile

CCB12 Concur's journey to 1 billion Couchbase operations per day

Couchbase Mobile 101: How to Build Your First Mobile App with Couchbase Mobile: Couchbase Connect 2015

Introduction to Couchbase Sync Gateway: Couchbase Connect 2014

Couchbase at LinkedIn: Couchbase Connect 2015

Sizing Your Couchbase Cluster: Couchbase Connect 2015

What's new in couchbase 4.0: couchbase connect 2015

Couchbase Live Europe 2015: Couchbase in Amadeus Critical Systems

Couchbase 101: Couchbase Connect 2014

Building Hybrid Apps with Couchbase Mobile: Couchbase Connect 2015

Couchbase Mobile 102 – Couchbase Live New York 2015

Introduction to Couchbase Mobile: Couchbase Connect 2014

Webinar: Couchbase for Mobile - Introduction to Couchbase Lite

Couchbase Mobile 103: Building a Peer-to-Peer App with Couchbase Mobile: Couchbase Connect 2015

Couchbase in the Digital Economy – Couchbase Connect 2016

Couchbase Chennai Meetup: Developing with Couchbase- made easy

Couchbase 102 - Developing with Couchbase

Couchbase Server 101: Couchbase Connect 2015