Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 1
NoSQL , JSON and TimeSeries Data - CON8862
Anuj Sahni, Principal Product Manager
Ashok Holla, Senior Sales Consultant
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 3
Safe Harbor
The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated into
any contract. It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making purchasing decisions.
The development, release, and timing of any features or functionality
described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 4
Agenda
Why JSON ?
Its role in NoSQL and Big Data
NoSQL Use Cases
Oracle NoSQL DB overview
Architecture
Data Modeling using JSON schema
Time Series case study and demo
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 5
Brief History of data Interchange
10 years ago XML was primary data exchange format
– Vast improvement over SGML (Standard Generalized Markup Language)
– Enabled people to exchange documents across HTTP
Language of Internet
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 6
Brief History of data Interchange (cont…)
Last few years, a bold transformation happening in world of data
interchange
– JSON (JavaScript Object Notation) emerged as an alternative to XML
looks more data-structure like,
Light weight/bandwidth-non-intensive,
language independent,
Language of Internet
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 7
What made JSON Cool ?
API
– No business value gain operating in silos
– REST replacing SOAP as data transfer protocol
The Internet of Things
– “JSON better adapted to devices with
limited capabilities”
Full-stack JavaScript
– JavaScript is new hot
– Node.js gone mainstream
Big Data
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 8
Thoughts Things Processes
Newer Challenges
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 9
Get Fast Answers to New Questions
What is Big data ?
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 10
NoSQL and Big Data Where did it come from?
SQL
JDBC,
ODBC
General
Purpose
Managed
Schemas
Security,
Backups
Analytics
…
Distributed
Processing
Distributed,
Replicated
File System
Driver
Application
NoSQL databases
Flexible
Schemas
Sharded,
Replicated
Database
High Speed,
Simple Ops
More Flexible Schema
Management
Globally Distributed,
“Always On” data
Competitive Advantages
of “Fast Data”
Lower TCO,
commodity HW scale-out
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 11
Kinds of NoSQL Database Based on Storage Model
Key Value Columnar Document Graph
Oracle NoSQL DB
Riak
Dynamo
Voldemort
Cassandra
HBase
MongoDB
CouchBase
Neo4J
GraphDB
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 12
Few common characteristics
They all store data in de-normalized fashion
– Don’t support PK-> FK relationship (no cascade deletes)
– Don’t support complex Joins
– Don’t manage constraints (application’s job)
Maintains data relationships using JSON/BSON representation
– Simple lookup operations by primary keys are generally enough
In NoSQL Databases
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 13
Agenda
Why JSON ?
Its role in NoSQL and Big Data
NoSQL Use Cases
Oracle NoSQL DB overview
Architecture
Data Modeling using JSON schema
Time Series case study and demo
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 14
Point Flexibility Point
Web-Scale
Transaction Processing
High velocity, volume,
variety, low information
density data capture
Web browsing, Shopping
Carts, CDR processing,
Sensor data, Stock data
Web-Scale
Personalization
Guaranteed low latency
lookups for end-customers
Advertising, Product
Recommendations, Online
Catalogs, Social Media, Profile
Management, Personalization
Real-Time
Event Processing
Real time events trigger rule
that perform low latency lookups
Medical Monitoring, Factory
Automation, Oil & Gas, Geo-
location
NoSQL Database Use Case Summary
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 15
Use Case – Online Social Gaming
Problem
– Very low latency requirements – Player movement must feel like a real time operation, while
being tracked on the server
– Extreme data velocity – Popular games, large scale user base (Farmville boasts 80 million active
users)
– Highly available – 24/7 sites
– Write heavy workloads
Solution – Where to use a NoSQL Database?
– Player interaction data store – Database to track player movement and game interaction
– Game play statistics – Per player usage statistics
– Persistent chat store – For games that allow player communication via chat, the NoSQL
database is used as a persistent message store (auditing and COPA compliance)
Web Scale Personalization
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 16
Use Case – Online Social Gaming Web Scale Personalization
JSON maps well
with HTML5/JS
based Client
Player interaction
stored as simple
JSON events
Easy to store
entire player state
reliably
NoS
QL D
B D
river
Applic
ation
Shard 2
Shard N
Shard 1
Schema Evolution
for agile
development
Transparent load
balancing On-demand
cluster expansion
REST
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 17
Use Case – Smart Meter Analytics
Problem
– Real Time Access to time-series data – identity patterns hidden within the terabytes of data
Identify theft,
predict system failures ,
better manage operations
– Large volume of machine data – 1000 fold increase in data collection
Write heavy workloads
– Highly available – 24/7 sites
Solution – Where to use a NoSQL Database?
– Fast time based queries – Daily, weekly, yearly consumption trends.
– Flexible data model – can adapt to various kind of sensors or changes in sensor format
– Scalable and Reliable performance – Easy to scale as workload increases because of more
sensors
Web Scale Transactions
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 18
Smart Meter Analytics Web Scale Transactions
Analytic
Engine
User
Interface
Export
back to
Smart
Grid N
oS
QL D
B D
river
Applic
ation
Shard 2
Shard N
Shard 1
Time Series Data
Data coming in
many different
format
Data
coming
fast
Horizontally
scalable
database
JSON
Easy to model
keys for time
bases queries
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 19
Agenda
Why JSON ?
Its role in NoSQL and Big Data
Introduction to NoSQL
Oracle NoSQL DB overview
Architecture
Data Modeling using JSON schema
Time Series case study and demo
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 20
Flexible Key-Value Data Model
ACID transactions
Horizontally Scalable
Highly Available
Elastic Configuration
Simple administration
Intelligent Driver
Commercial grade software and support
Features
Oracle NoSQL Database
Application
Storage Nodes Datacenter B
Storage Nodes Datacenter A
Application
NoSQL DB Driver
Application
NoSQL DB Driver
Application
Java SE 6 (JDK 1.6.0 u25)+; Solaris or Linux
Scalable, Highly Available, Key-Value Database
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 21
Simple Data Model
Simple, flexible key-value paradigm
Simple operations – read/insert/update/delete, RMW support
Ordered scan of key ranges
Unordered scan of all data (non-transactional)
Streaming API for LOBs
Java and C APIs
Key-value pairs
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 22
Compact, highly efficient serialization
Synergy with Hadoop
Supports serialization from and to JSON strings
Bindings from serialized formats to language constructs
Easy to use mechanism for schema evolution
Schema definition tracked with schema ID of data writer
Schema versions can be opaque to readers
Oracle NoSQL – Why AVRO?
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 23
JSON support in Oracle NoSQL DB Schema Creation
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 24
JSON support in Oracle NoSQL DB (cont…) Data Manipulation
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 25
JSON support in Oracle NoSQL DB (cont…) Schema Evolution
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 26
JSON support in Oracle NoSQL DB (cont…) Agile development
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 27
Time Series Analysis using
Oracle NoSQL database
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 28
Big Data Challenge
High velocity of stock tick data generated
at a massive volume each day
Millions of customers trading stocks on-
line based on the current value and the
historical trends.
How do you ensure these buy/sell
transactions happens in real time?
How do you store large volume of tick
data in a quick and consistent manner?
– So you can run the trend analysis on the time
series data.
UC - Stock Tick Analysis
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 29
Introduction – Oracle Investments Inc. Goal
– Ensure fast and consistent storage of
large volume of tick data
– Deliver a platform where customers can:
Analyze historical trends by plotting
time-series data
Trade stocks in real time by placing
buy/sell orders
Challenge
– 100k’s customers trading concurrently.
– 10k’s of ticks each second
– Real time response required for order
execution
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 30
Introduction – Oracle Investments Inc.
Value
– Real time:
Data capture and random access to
time sensitive data
Order processing
– Automatic data partitioning for easy
scalability
– Lowest $/ops for a consistent storage
– Highly Available system for 24/7 operation
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 31
Big Data Appliance
Oracle NoSQL DB
Oracle NoSQL DB as Data Store
NoSQL Driver
Read, Update
Stores all key interactions required to
drive application. For example:
User Profile
Stock Tick Data
Transaction information
Why Oracle NoSQL Database?
Highly Scalable
Extremely performant
Super low-latency
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 32
Architecture
Business Layer
Big Data Appliance
Oracle NoSQL DB
NoSQL Driver
Stock Tick Generator
Tick data every 5 seconds
Trading
Read, Update
Analytics
Time Series
Analysis
Order requests
User Profile Updates
Order processing
Historical Data
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 33
NoSQL Data Representation
{"lasttimestamp":"1367251181219“,"lastclose":"45.17","lasthigh":"32.25","lastlow":"31.94","lastopen":"32.21","lastvolume":"26"
,
"stockArray":
[
{"timestamp":"1367245601219“, "value":"32.21“, "open":"32.23“, "high":"32.27“, "low":"32.14“,"volume":"4700"}
{"timestamp":"1367245602220“, "value":"32.15“, "open":"32.14“, "high":"32.27“, "low":"32.09“,"volume":"4000"}
{"timestamp":"1367245603319“, "value":"32.11“, "open":"32.09“, "high":"32.27“, "low":"32.14“,"volume":"3500“}
]
}
Denormalized Data
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 34
NoSQL Data Representation
Get Stock Information for a Company
– Key = /SYMBOL/YYYY-MM/-/DD
– (Ex: /ORCL/2013-05/-/02 , Stock data for 2nd of May 13)
Write = put(key, value)
Read = get(key)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 35
Demo
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 36
Oracle NoSQL DB Resources
Support via OTN forums and Oracle Support process
OTN Forum:
– Forum Home » Big Data » NoSQL Database
– forums.oracle.com/forums/forum.jspa?forumID=1388
Oracle.com:
– www.oracle.com/us/products/database/nosql/overview/index.html
OTN (including documentation and download):
– www.oracle.com/technetwork/products/nosqldb/overview/index.html
Support
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 37
Oracle NoSQL DB Resources
On OTN and in download
– docs.oracle.com/cd/NOSQL/html/index.html
Getting Started Guides
Programmatic API
Installation & Release Notes
FAQ
Documentation
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 38
References
Why JSON will continue to push XML out of the picture
Interactive Matter Lab
Architecting the Internet of Things(p. 102)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 39
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 40
Appendix
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 41
Configurable Durability per operation
Configurable Consistency per
operation
ACID by default
Transaction scope is single API call
Records share same major key
Multiple operations supported
Greater Flexibility
Features – Configurable ACID Transactions
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 42
Developing Applications Data Modeling
Subject ID Qualifier – Value Type Range
Binary
JSON/Avro
RDF Triples
Tables/Rows
Queries
sensors by section
Pressure Sensor Sensor ID Left Front What a List The Value
Value Types
=
A specific sensor
All measures of a sensor A range of sensor measures
Major Key
/ / / / / /
Minor Key
CT 123-PS3234 LF – psi, irate Timestamp Array of Int = / / / / / /
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 43
Increase Data Capacity
– Add more storage nodes
– New shards automatically created
Increase Data Throughput
– More shards = better write throughput
– More replicas/shard = better read throughput
On Demand
Elasticity
NoSQL DB Driver
Application
Master
Replica
Replica
StorageNode StorageNode StorageNode
Shard-1
Master
Replica
Replica
Shard-2
On-Demand Cluster Expansion
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 44
Supports heterogeneous storage
topology
Replicas move from over-utilized to
under-utilized storage nodes
Number of shards and replication
factor remain unchanged
Improve Performance
Rebalance an Unbalanced Store
Storage Node 1 Storage Node 2 Storage Node 3
Represents a data fragment
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 45
What’s Really Important?
Technical Feature Importance Why
Storage Model Not really Will merge over time
Specific Features Somewhat Application requirements?
Performance Somewhat Rapid changes, YMWV
Integration Critical Long term, Repetitive cost
Reliability/Support Critical Early products, Product
direction
Predictability Critical Production reqs & SLAs
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 46
Query NoSQL data from Oracle Database
Access NoSQL data from Hadoop for DW and analytics
Share data with Coherence for extensible in-memory cache grid
Persist history & event streams for processing with OEP
Store & query RDF data using Oracle RDF for NoSQL
Integration Oracle NoSQL Database: Integrated out of the box
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 47
Reliability & Support
Decades of widespread, reliable deployment experience
15+ years of mission-critical non-relational database technology
Oracle Support available for both Enterprise and Community Edition
Oracle NoSQL Database: Enterprise-Grade Software & Support
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 48
Automatic election of new Master
Rejoining nodes automatically
synchronize with the Master
Isolated nodes can still service reads
All nodes are symmetric
Automatic Failover
Highly Reliable
Replication factor = 5
Rep Node
Master
Rep Node
Replica Rep Node
Replica
Rep Node
Replica
Rep Node
Replica
New Master