SmartCity StreamApp Platform: Real-time Information for Smart Cities and Transportation

Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 1

!Damian Black, CEO SQLstream Strata + Hadoop World, 2014!October 15-17 2014, New York

!SmartCity StreamApp: An Internet of Things Service for Real-‐;me Traffic Management


!Product: SQLstream Blaze Stream Processor Focus: Powering smart services for the IoT/IoE Located: San Francisco

!Real-time Action in the Internet of Everything


“We predict that the Internet of Everything will be a $19 trillion dollar market over the next several years.”

John Chambers, CEO, Cisco

“When you look at the Internet of Things, it is clear 2014 will be a tipping point in the evolution of the Internet.”

Marissa Mayer, CEO and President, Yahoo

“We are equipping our products with sensors that constantly measure performance so our customers see major productivity gains and minimize unplanned downtime.”

Jeff Immelt, Chairman and CEO, GE

A world transformed by an explosion of sensors


A World of Big Data §  Big Data – datasets (3V) beyond the ability of traditional data management systems §  Driven by Hadoop: massively scalable, multi-server systems on low cost hardware §  Designed for semi-structured and unstructured data – ideal for sensors §  Driving expectation for real-time responses and action


Internet Wireless Sensors Smartphones


0!

45!

90!

135!

180!

225!

2014 2015 2016 2017 2018 2019 2020

Analytics will be dominated by IoT Data

IoE (net IoT) 9.5% CAGR [IDC]

IoT (real-time) 44% CAGR [Gartner] $

Billio

ns

Characteris;cs of Internet of Things sensor data >  Very high data volumes >  Systems must react to each and every record >  Mostly ‘business as usual’ – no need to warehouse every record


Stream Processing!

Relational Databases!

Hadoop !

In-memory Databases!

Big Data Technology: From Human Time to Real-time

6

High Low

Speed of Response

Throughput Capacity

Low

High

Store First

Process First


Stream Processing versus Stored Data Queries

7

Databases store then query. Stream Processors query then store.

Query

Query

Databases The query traverses the table. With every record, the table is updated and the process must start again.!

Stream Processing The incoming streams of data) move through a continuous, windowed SQL query. Unlimited capacity and continuous, real-time results.!

Storage(Optional — RDBMS, Hadoop, NoSQL …)

Performance Metrics

§  1 million EPS/core §  Multi-server scale-out §  Latency < 10 milliseconds


A real-time data hub for the Internet of Things

IoT Smart Srvcs

Smart Srvcs

Smart Srvcs

Smart Srvcs

Smart Srvcs

Mobile Apps Apps Apps Apps

Server Server Server Server

Internet

Switch Proxy Router

Billing Fraud SLA


SmartCity StreamApp"Roads & Maritime Services Case Study


We live in a Modern World §  800 million vehicles on the world’s roads today, estimated to increase to 4 billion by 2050 §  Environmental concerns and air pollution driving innovation, but just the beginning §  The expectation of on-demand, real-time and accurate information has not been realized §  The Quality of Life for travelers and commuters is deteriorating


Conges4on Environment Poor Informa4on Quality of Life?


The Internet of Things Smart People to Smart Cities with Smart Services

Traffic Analy;cs Mul;-‐Modal Smart City Real-‐;me Informa;on

§  Step 1: Real-time traffic flow and congestion for transportation agencies §  Step 2: Extend for multi-modal Travel Time with public transportation §  Step 3: Smart City Smart Services – dyanamic traffic management §  Step 4: Traveler and Commuter Smartphone App access


Traffic Analy4cs Mul4-‐Modal Smart City Real-‐4me Informa4on


From Big Data to real-time actions §  Stream processing means high throughput with low latency actions at Big Data scale §  A real-time data hub delivers multiple applications on a single stream processing platform §  The platform for the real-time SmartCity StreamApp §  Delivering a better traveler experience


Stream Processing Real-‐4me Data Hub Smart City PlaIorm A BeKer Experience


Case Study: Big Data Innovation for ITS

TT5 : Real-‐4me SmartCity StreamApp Traffic App Hackathon (Powered by SQLstream, Sept 20-‐21 2014)

•  GPS fleet data – low solu;on cost, faster ;me to value

•  Real-‐;me traffic flow & conges;on predic;on

•  All classified roads in NSW networks

•  Opera;onal dashboards based on Google Maps

•  Public API (web service feed and JDBC app connect)

•  Encourage innova;on and app development

•  Focus on conges;on hotspots

•  Funding for the winning team

•  Building on SQLstream TT5 API



Calculate Road Segment (10m) Mean Travel Time

Vehicle 1 Vehicle 2 Vehicle 3 Vehicle 4

Vehicle N

Aggregate Mean Travel Time per Road Element

Vehicle GPS record Vehicle Travel Time (green light) Vehicle Travel Time (red light)


Travel Time Calculation using Vehicle GPS Data Streams

Calculate Vehicle Travel Time along specified route bounded by REi and REj: •  RE: Road Element (10 meter segments)

•  l: Length of RE (typically 10 meters)

•  D: Distance traveled by vehicle between GPS reports

•  T: Time taken by vehicle to travel between GPS reports

•  n: Number of GPS reports

TTREi−REj =1nRE

Tv,rlREsDv,rr(reports)

∑#

$%%

&

'((

v(vehicles)∑

#

$%%

&

'((

REs∑


Update average speed incrementally across any subset of network over rolling time windows

Code Example #2: Continuous Average Speed Calculation

CREATE OR REPLACE VIEW "SpeedZoneStats"DESCRIPTION ‘Rolling averages for multiple windows partitioned by zone' AS SELECT STREAM "zone", -- zone id "segmentid", -- parent road segment "speedlimit", -- speed limit for zone AVG("Speed")OVER last1Min AS "avgSpeed1", -- 1-min running average AVG("Speed")OVER last5Min AS "avgSpeed5", -- 5-min running average AVG("Speed")OVER last10Min AS "avgSpeed10" -- 10-mn running average FROM "RoadPositionInfo" WINDOW last1Min AS (PARTITION BY "zone" RANGE INTERVAL '1' MINUTE PRECEDING), last5Min AS (PARTITION BY "zone" RANGE INTERVAL '5' MINUTE PRECEDING), last10Min AS (PARTITION BY "zone" RANGE INTERVAL '10' MINUTE PRECEDING);


Improve System Accuracy"Join streaming analytics with historical trends from the data warehouse

0:00

1:00

2:00

3:00

4:00

5:00

6:00

7:00

8:00

9:00

10:00

11:00

12:00

13:00

14:00

15:00

16:00

17:00

18:00

19:00

20:00

21:00

22:00

23:00

0:00

Average (Trend) Real-‐;me >  Trend data comparisons help

to differentiate exceptional

event from normal behavior

>  Predicting extent and

duration of an event can still

be difficult

>  Assessing the traveler

sentiment from Social Media

analytics offers additional

insight

Average Travel Time over a 24 hour period


SmartCity StreamApp"Architecture


SQLstream Blaze Distributed SQL stream processing engine with

intelligent guide discovery and real-‐;me dashboards.

s-Visualizer real-time dashboards for Enterprise Power Users

s-Server Distributed SQL Stream

Processor

s-Dashboard HTML5 real-time

dashboards for Developers

Storm Adapter s-Studio Developer & Admin

StreamLab Intelligent guided data stream discovery, analytics and visualization without coding

Enterprise-Class Real-time Data HubStream Processing for Operational Intelligence and the Internet of Things

Industry StreamApps

Pre-‐built libraries for rapid development of industry

stream processing solu;ons.

Telecoms Smart City Emergency Services Oil & Gas


Control Systems

SQL Optimizer Parallel Scheduler Real-time Indexing

RT Memory Manager Dynamic Java Analytics (UDX)

Streaming Data Protocol

Interac4ve Stream Discovery and Visualiza4on

Stream Processing Engine

(HTML5)!Discovery

API Connect

SmartCity StreamApp – Core Platform Architecture

Devices & Apps

Hadoop Ingestion (MB/s)

Events/Sec/Core)

Native Tables

Repor4ng Tools

JDBC Web Agent

Web Sockets REST

(HTML5)!Dashboards

(Flash)!Dashboards

JDBC

Remote Systems

Enterprise Systems

Agents

Adapters

Devices & Apps

JDBC

Machine Data

Enterprise Systems

Agents

Adapters

Hadoop & NoSQL Enterprise BI

Data Warehouse

SQL Database

Predictive Analytics

Hadoop / HDFS

HBase

Storm & Kafka


SmartCity StreamApp – Software Architecture

Data Stream Acquisi4on Toolbox

GIS & Geospatial Library

Data Quality & Density Analysis

Data Parsing & Transformation

Data Enrichment

StreamApp Analy4cs Toolbox

Congestion Prediction Library

Congestion Detection Algorithms

Real-time Traffic Flow

Object Tracking

Integra4on & Delivery Toolbox

Alerting Module

Control System Metadata

KML & Ruby/Rails Library

Email & Workforce Activation

Customer Extensions Customer Extensions Customer Extensions

Travel Time App

Travel Time Subsystem

Customer Extensions

Conges4on App

Congestion Subsystem

Customer Extensions

Network KPIs App

Network KPIs Subsystem

Customer Extensions

Custom App

Custom App Subsystem

Distributed SQL & UDX(Java) Execution Engine

Query Management Distributed Query Optimization Query Scheduler

Platform - Distributed Server Cluster

JDBC Driver Adapter Plug-ins UDX (Java) Libraries Agent Framework

StreamApp End-user Applications

StreamApp Toolkits & External Libraries

StreamApp Execution Platform


Real-‐4me Data Hub for SmartCity StreamApp

SmartCity StreamApp – Solution Architecture Roadside Furniture

Traffic !Lights

Speed Signs

Agents

Vehicle GPS Bus & Train Roads Agents

Weather Feeds Agents

Social Media

Twitter Feeds Agents

Historical Trends Data!

Warehouse Predictive Analytics

Adapters

Logs

XML, JSON

XML, JSON

JDBC, API

Write API Live Road Updates Traffic !Lights

Speed Signs

HTTP RT Travel Time Apps

HTTP RT Conges4on Maps

Smart phone Device

Website

KML/Ruby RT Traffic Opera4ons

Real-time Dashboards

KML/Ruby Planning

KPI Reports

Real-time updates to traffic light phasing, speed signs and dynamic lane configuration

Commuters on the move

Commuters travel planning

Network Operations

Network Planning

SmartCity StreamApp

Streaming Integration Layer Distributed Stream Processing

Execution Environment

Operational Servers

Internet!Servers

Historical Trends Data!

Warehouse Predictive Analytics

Adapters JDBC, API


>  Minimum Time to Value Combination of subscription-based GPS vehicle feeds with pre-built StreamApp means operational system for

first requirements available in a few weeks.

>  Strong Return on Investment Estimated $20million saving on initial project, with > $100 million over project’s lifetime.

>  Traffic congestion reduction Real-time operational displays and greater accuracy for long term forecasting means a significant reduction in

congestion at key hotspots.

>  Improved commuter experience Accurate and reliable Travel Time estimates with real-time updates means commuters are better prepared and better able to plan re-routes.

SmartCity StreamApp – Operational Results

Data & Analytics

SmartCity StreamApp Platform: Real-time Information for Smart Cities and Transportation