Accessing IoT Data with Couchbase Server, Couchbase Mobile and Kaa: Couchbase Connect 2015

Accessing IoT data with Couchbase Server, Couchbase mobile, and Kaa

Andrew Kokhanovskyi

CTO, CyberVision, Inc.

@akokhanovskyikaaproject.org© 2015 CyberVision, Inc. All rights reserved.

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.

Endpoints

Cloud

Web dashboardsMobile applications Monitoring systems

Data

Aggregated Data

Conceptual diagram

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.

Server / Lite

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.

➢ Fully-featured IoT middleware platform➢ 10 Kb RAM footprint (with C SDK)➢ Guaranteed data delivery and reliable local storage➢ Built-in transport security➢ Efficient data serialization➢ Horizontally scalable and fault tolerant➢ 100% open-source (Apache license 2.0)➢ Rapid application development using C / C++ / Java SDKs➢ Integration with popular device platforms

Why Kaa?

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.

➢ Flexible document-oriented data model➢ Elastic scalability➢ Consistent high performance➢ Real-time big data analytics➢ Always-on availability➢ Open-source (Apache license 2.0)➢ Real-time Server synchronization with Couchbase Lite in user apps

Why Couchbase?

Architecture

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.

Endpoints

Web dashboardsMobile applications Monitoring systems

Telemetrydata

Lots of telemetry data

Aggregated data

Server

Lite

Kaa cluster

Data flow: Kaa perspective

© 2015 CyberVision, Inc. All rights reserved.

Your application

KaaSDK

Sensors

Raw Data

Your device

Operations server node

Couchbaseappender

Kaaservices

Structured Data

Structured Data

kaaproject.org

Mobile application

CouchbaseLite

Applicationlogic

Data flow: Couchbase perspective

© 2015 CyberVision, Inc. All rights reserved.

Couchbase cluster

Documents(huge volume)

View

Documents(summary)

Structured Data

Sync Gateway

CouchbaseLite

Applicationlogic

Mobile application

kaaproject.org

The secret sauce:structured data e2e

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.

Power plant

Power plant: telemetry data collection

© 2015 CyberVision, Inc. All rights reserved.

kaa::KaaUserLogRecord voltageReport;voltageReport.timestamp = getCurrentTime();

samples.reserve(configuration.panelCount);for (std::int32_t i = 0; i < configuration.panelCount; ++i) { kaa_log::VoltageSample sample; sample.zoneId = solarPanels_[i].getZoneId(); sample.panelId = solarPanels_[i].getPanelId(); // Analog-to-digital converter factor sample.voltage = solarPanels_[i].getVoltage(); samples.push_back(sample);}voltageReport.samples = std::move(samples);

// upload log recordkaaClient_.addLogRecord(report);

Data record schema C++ client code snippet

kaaproject.org

{ "type": "record", "name": "VoltageReport", "namespace": "org.kaaproject.kaa.examples.powerplant", "fields": [ {"name": "timestamp", "type": "long"}, {"name": "samples", "type": { "type": "array", "items": { "type": "record", "name": "VoltageSample", "fields": [ {"name": "zoneId", "type": "int"}, {"name": "panelId", "type": "int"}, {"name": "voltage", "type": "double"} ] }}}]}

© 2015 CyberVision, Inc. All rights reserved. kaaproject.org

Power plant: same client code in Java and pure C

Java snippet C snippet

Long timestamp = System.currentTimeMillis();

List<VoltageSample> samples = new ArrayList<>();for (int i = 0; i < KAA_DEMO_MAX_ZONE_ID; i++) {

Integer zoneId = i;Integer panelId = getPanelId(i);

Double voltage = getVoltage(panelId); VoltageSample sample = new VoltageSample(zoneId, panelId, voltage); samples.add(sample);}VoltageReport report = new VoltageReport(timestamp,

samples);kaaClient.addLogRecord(report);

kaa_user_log_record_t *log_record = kaa_logging_voltage_report_create();

log_record->timestamp = time(NULL) * 1000;log_record->samples = kaa_list_create();

for (size_t i = 0; i < KAA_DEMO_MAX_ZONE_ID; i++) { kaa_logging_voltage_sample_t *sample = kaa_logging_voltage_sample_create();

sample->zone_id = i; sample->panel_id = getPanelId(i); sample->voltage = getVoltage(sample->panel_id); kaa_list_push_back(log_record->samples, sample);}kaa_logging_add_record(log_collector, log_record);log_record->destroy(log_record);

© 2015 CyberVision, Inc. All rights reserved. kaaproject.org

Power plant: sample log document

© 2015 CyberVision, Inc. All rights reserved. kaaproject.org

Power plant: Couchbase view

© 2015 CyberVision, Inc. All rights reserved. kaaproject.org

Power plant: Couchbase Java daemon

© 2015 CyberVision, Inc. All rights reserved. kaaproject.org

database = manager.getDatabase(localDBName);URL dbURL = new URL(dbURLStr + gatewayName);pull = database.createPullReplication(dbURL);List<String> channels = Arrays.asList("totals");Authenticator auth = new BasicAuthenticator(username, password);pull.setAuthenticator(auth);pull.setChannels(channels);pull.setContinuous(true);pull.start();

sync gateway pull replication

View reportView = database.getView(VIEW_NAME);reportView.setMap(new Mapper() { @Override public void map(Map<String, Object> document, Emitter emitter) {

Long timestamp = (Long) document.get(TIMESTAMP_FIELD);

if (timestamp != null) { emitter.emit(timestamp, document);}

}}, "1");

local view

Power plant: Couchbase Lite Android

Video goes here

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.

kaaproject.org© 2015 CyberVision, Inc. All rights reserved.

➢ 2 days implementation➢ Reliable telemetry data delivery➢ Average e2e latency < 1-2 sec➢ Horizontal scalability➢ Clear and simple integration➢ Collected endpoint data can be easily queried and analyzed using

Couchbase views➢ Couchbase Lite pull replication: synchronizing data to apps with ~ zero

effort

Results

Andrew Kokhanovskyi, CTO, CyberVisionak@cybervisiontech.com

kaaproject.orgcybervisiontech.com

THANK YOU FOR YOUR ATTENTION

© 2015 CyberVision, Inc. All rights reserved.

Memory usage

Operations at 80% CPU utilization

1 node 2 nodes 4 nodes 16 nodes

Notifications / sec

45 k 86 k 173k 710 k

Configuration updates / sec

30 k 58 k 122 k 465 k

Events / sec 120 k 150 k 410 k 1 500 k

Endpoint count 100 k 200 k 400 k 800 k

RAM usage 2Gb 3Gb 5Gb 9.5Gb

Key performance indicators

kaaproject.org

© 2015 CyberVision, Inc. All rights reserved.

Zookeeper quorum

Endpoints

Control servers

standby

Bootstrap servers

Operations servers

Fault-tolerance and horizontal scalability

kaaproject.org

active

© 2015 CyberVision, Inc. All rights reserved. kaaproject.org

Power plant: sync gateway configuration

{ "interface":":4984", "adminInterface":":4985", "log":["CRUD", "REST+", "Access"], "databases":{ "sync_gateway":{ "server":"http://localhost:8091", "bucket":"sync_gateway", "shadow": { "server": "http://localhost:8091", "bucket": "powerplant" }, "sync":`function(doc) { if (doc.zones != undefined) {channel("totals"); } }` } }}