Upload
cassidy-ming
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
Solving Manufacturing EquipmentMonitoring Through Efficient
Complex Event Processing
Tilmann Rabl, Kaiwen Zhang, Mohammad Sadoghi, Navneet Kumar Pandey, Aakash Nigam, Chen Wang, Hans-Arno Jacobsen
Middleware Systems Research Group, University of Toronto
DEBS Grand Challenge 2012
msrg.org2
Agenda
Complex Event Processing Scenarios
System Architecture
Evaluation
Demo
18/07/2012
msrg.org3
Motivation Course “Large-Scale Data Management”
Big data storage Large scale event processing
Complex event processing scenarios Application performance management Smart traffic monitoring Energy monitoring
Resulting team project DEBS Grand Challenge
18/07/2012
msrg.org4
Scenario I: Application Performance Management
Monitoring of enterprise systems Find bottlenecks, problems Trace transactions, measure utilization
18/07/2012
msrg.org5
Scenario II: Smart Traffic Monitoring
Traffic data from cars, mobile devices, road sensors
Event aggregation, filtering, correlation Traffic status, accident detection, etc.
18/07/2012
msrg.org6
Scenario III: Energy Monitoring
Green computing Application-level energy monitoring API-based energy consumption estimation
Operating System & Hardware
Applications
CPUNetwor
kI/O
Formulae
Sensors
…
Store
API
18/07/2012
msrg.org7
Common Denominators High data rates
1000 – 1000000+ events / sec Small data points
< 1 KB Complex queries
Filtering, aggregation, correlation Persistent storage Distributed setup
DEBS Grand Challenge
18/07/2012
msrg.org8
Continuous Query
Evaluation
High-Level Architecture Monitoring Service
Input data stream, marshalling Event Dissemination Substrate
(Optional) pub/sub layer, queues Continuous Query Evaluation
Consumes input, computes results Storage Manager
Stores data, enables querying Client
Visualizes query results Java-based implementation
Monitoring Service
Storage Manager
Client
Event Disseminati
on Substrate
Stable Storag
e
18/07/2012
msrg.org9
Storage Architecture Data is stored in
tables Key-value pairs
Table Index Fast lookup
Compressor Efficient data storage Run length encoding
Grand Challenge data Run-length: 20000 Compression: 99.99%
18/07/2012
msrg.org10
Client
Google Web Toolkit-based Client Java-code compiled to JavaScript Displays all Grand Challenge results
Plots, results, alarms
18/07/2012
msrg.org11
Evaluation Distributed setup
Separated servers for data generator and monitoring tool
Configuration 2 servers 2 x dual core Xeon processor 4 GB RAM Gigabit Ethernet
Data set: 5 min + 18 days + synthetic Synthetic data
Many errors (every 2 - 3 min) Maximum throughput (no network)
Metrics: latency & throughput18/07/2012
msrg.org12
Processing Overhead Real Workload
0 – 200 queries (Q1,Q2 repeated) Linear in the number of queries Stable with increasing generator speedup
18/07/2012
msrg.org13
ThroughputReal Workload
Throughput controlled by data generator Data generator does not saturate system Peak 9000 events/sec ~ 0.11 ms arrival rate Well below maximum throughput
18/07/2012
msrg.org14
LatencySynthetic Workload
Maximum throughput Minimum latency
0.019 ms (10 queries) Maximum latency
0.63 ms (1400 queries)
18/07/2012
msrg.org15
Throughput Synthetic Workload
Maximum throughput 40000 events / sec (20 queries)
Minimum throughput 1500 events / sec (1400 queries)
18/07/2012
msrg.org16
Demo
Live!
18/07/2012
msrg.org17
Conclusions Complex event processing scenarios
Application performance management Smart traffic monitoring Energy monitoring DEBS Grand Challenge
Efficient implementation of the Grand Challenge Java-based Google Web Toolkit GUI Synthetic data generator
Up to 40000 events per second Up to 1400 queries
18/07/2012
msrg.org19
Back-Up: DEBS Grand Challenge Manufacturing equipment monitoring Large, real data set
18 days, 100 Hz 2 queries
State of sensors and valves Difference Threshold
Power consumption Range Average Threshold
18/07/2012