GAMPS COMPRESSING MULTI SENSOR DATA BY

GROUPING & AMPLITUDE SCALING

Sorabh Gandhi, UC Santa Barbara Suman Nath, Microsoft Research Subhash Suri, UC Santa Barbara

Jie Liu, Microsoft Research

Fine Grained Sensing & Data Glut

Advances in sensing technology fine grained ubiquitous sensing of environment

Many applications, but the issue is data glutAutomated Data Center Cooling: [MSFT DCGenome project] physical parameters ex. humidity, temperature etc 1000s of sensors, 10 bytes/sensors/sec 10s of GBs/dayServer Performance Monitoring: [MSFT server farm monitoring] performance counters ex. cpu utilization, memory usage etc 100s of counters, 1000s of servers, few bytes/counter/sec TBs/day

Focus and Objectives

Data archival + (reliable and fast) query processing Centralized setting Point query: report value for sensor x, time t Similarity query: report sensors ‘similar’ to sensor x in time range

Obvious solution: compression, data is set of time series

Initial idea: approximate every time series individually Many approximation techniques known ex. DFT, DCT, piecewise linear Focus: L1 error [guarantee on point queries]

ex techniques wavelets, piecewise constant/linear approximations

Compression not enough!! Gives upto an order of magnitude improvement, we want more

Signals are Correlated!

Server dataset: 40 signals, 1 day, sampling once every 30 seconds, counter: # of connected users

# Co

nnec

ted

Use

rs

Time

Similar signals in a group

Shifted/Scaled groups Dynamic groups

We propose GAMPS, which exploits linear

correlations among multiple signals while compressing them together, and gives L1 guarantees

Compression both along time and across signals

We propose an index structure for compressed data which can give fast responses to a lot of relevant queries

Through simulations on real data, we show that on large datasets, GAMPS can achieve upto an order of magnitude improvement over state of the art compression techniques

Contributions

State of the art: Single Signal Optimal L1 approximations

Problem: Given a time series S and input parameter ² approximate S with piecewise constant segments such that the L1 error is <= ²

Greedy algorithm (PCGreedy(S, ²))

Problem: Given a time series S and input parameter ², approximate S with piecewise constant segments such that the L1 error is <= ²

Greedy algorithm (PCGreedy(S, ²))

2²

Original Time SeriesApproximation

ICDE’03 Lazardis et al.

State of the art: Single Signal Optimal L1 approximations

GAMPS Overview

GAMPS take as input, the set of time series and approximation parameter ²

Compression Partition phase: partitions the data into contiguous time

intervals Group phase: divides a given partition into groups of similar

signals Amplitude scaling phase: compression happens with

sharing of representations

Amplitude Scaling Phase

Partition Phase

DataGrouping

PhaseIndex

Structure

COMPRESSION INDEXING

DataCompressed

Compression by Amplitude Scaling

Given a group of k ‘similar’ signals Let the signals be denoted by set X = {X1, X2, …,

Xk}

Key idea: express all signals Xi as scaled function of some signal Xj: Xi = AiXj

Ai is the ratio/amplitude signal and Xj is the base signal

If signal Xi is a perfectly scaled version of Xj then Ai = constant

To reconstruct Xi, we only need to store the constant and Xj

In reality, no perfect correlation

However, we found that if there are enough linearly correlated signals smartly approximating Ais and Xj can give very good compression factors!

Illustration: Amplitude Scaling on Real Dataset

DataCenter dataset 6 signals shown for ~3 days each, parameter: relative humidity

Input: X = {X1, X2, …, X6}, ² = 1% Need to choose base signal and divide ² among

base signal (²b) and ratio signal approximations (²r) Oracle: X4 is base signal, also provides values ²b and ²r

Run PCGreedy(X4, ²b) and PCGreedy(Ai, ²r) for signals other than the base signal

DataCenter Dataset

Illustration: Amplitude Scaling on Real Dataset

Leftmost figure, all signals use PCGreedy() with ² = 1.0% Middle figure, higher fidelity base signal, ²b =0.4% Rightmost figure: Ratio signals

Very sparse (small number of segments to represent)

Base signal approx

Y-axis: Relative Humidity

Ratio signal approx

Y-axis: Ratio

Individual approx

Y-axis: Relative Humidity

Compression factor = M1/M2

M1 = number of segments in individual signal approximations M2 = number of segments in (base signal + ratio signal) approximations

For this illustrative dataset, compression factor (1% error) is 1.9

Quantitative Comparison for Amplitude Scaling

Comparison with optimal individual approximations

Facility location problem Problem is modeled as a graph G(V, E) Opening a facility at node j costs c(j) Serving a demand point j using facility i costs w(i,j) Objective is to choose F µ V

Minimize j 2 F c(i) + i 2 V w(i,j)

Grouping & amplitude scaling is modeled as facility location

Complete graph, every signal is represented by a node Cost opening a facility: # segments needed to represent base

signal Cost of serving a demand point: # segments needed to represent

the ratio signal

Grouping and Amplitude Scaling by Facility Location

Graph

Implementation Setup

We set ²b = 0.4² [error allocation for base signal]

Facility location : NP hard We show results with exact solution (integer linear program) Approximation solutions are with 90% of the results shown Time taken to solve the linear program is <= few seconds

We use three different datasets Server dataset: 240 signals, 1 day data [CPU utilization

counter] DataCenter dataset: 24 signals, 3 days of data [humidity

sensors] IBT dataset: 45 signals, 1 day of data [temperature sensors

in a building in Berkeley]

Quantitative Evaluation: GAMPS

Figure on the left shows compression factor over raw data

For 1.5% error, 300 for server data, 50 for the other two

Figure on the right: compression factor over individual approximations

For 1.5% error, between factor 2-10

Compression factor high for Server dataset Average group size is highest (60 as compared to 4.5 & 6)

Scaling versus Group size

We extracted 60 signals in the same group for the Server dataset

Compression factor (versus individual approximations) increases as group size increases

Advantage of Grouping

Demonstrate the advantage of having multiple groups Datasets IBT and Server Hybrid: algorithm which allows only 1 group

Every signal is either in the group or approximated individually For both datasets, for all errors, grouping gives great

advantage Compression Factor: 1.5 (IBT) - 9 (Server) [Error 1.5%]

Grouping: Geographical Locality

IBT dataset, 1 day, error = 1.5%

GAMPS runs the grouping on entire days data Picture on left shows sensor layout in the Intel Berkeley lab Hexagons are sensor positions, crosses are sensors without data

for the one day, rectangles are outliers (individual approximations)

Simple region boundaries conform our intuition

Sensor Layout Group Layout

Indexing Compressed Data

1

2

3

4

5

Skip-list of groups

Ptr. to base signalSkip-list of approx. lines for ratio signal

Propose Skip list based index structure Point query: log(n) Range query : log(n) + range Similarity query : log(n) + #groups in range

Future Work

How to distribute error among base and ratio signals ?

How about generic linear transformations ? We use only ratio signal (scaling) : Xi = AiXj

Maybe we can get much better compression by using Xi = AiXj + Bi

How about piecewise linear signals ? Underlying algorithm is not so trivial (convex hulls)

Can we apply this technique to 2D signals ? Consider a video, every pixel value in time time series Every pixel-time-series, correlated with neighboring pixel-time-

series

Thanks for your attention

Example Query: Similarity Query

Based on grouping we can define similarity coefficient for a given time range (t1, t2)

= 1, if signals Si and Sj are in the same group at time t

Part of IBT dataset Similarity Query

Compression by Interval Sharing

Key Idea: If two sensors have near overlapping time series they can share a part of the approximation

Let number of signals be k and desired error be ² (®, ¯) approximation algorithm

For given error ² say optimal algorithm taken OPT (®, ¯) algorithm has error no more than ®² and uses no more

than ¯OPT segments

We propose polynomial time (5, log k + log OPT) approximation algorithm for approximation with PC segments using interval sharing

Signal 1Signal 2

Representation can be shared

Multiple Correlated Signals: Example 1

Instant messaging service – Server dataset 240 servers, 2 weeks, >= 100 performance counters 40 signals shown (normalized) for one day, counter: #connected

users, sampling rate once in 30 seconds

Signals are correlated (almost overlapping) with each other, can we exploit this in compression ?

Server Dataset

Multiple Correlated Signals: Example 2

Data center monitoring 24 sensors, 2 years, 2 parameters: humidity, temperature 6 signals shown for ~3 days each, parameter: relative humidity,

sampling rate once every 30 seconds Signals not overlapping, but still correlated

Shifting or scaling may help Question: Can we exploit this correlation ? We propose a technique to compress multiple

signals along both time and across signals

DataCenter Dataset

Partition Determination

Use double-half-same size heuristic Start with some initial batch size (say 100 data points) For next batch run group and compress with 200, 100 & 50 data

points For 200, compare with two batches of size 100, whichever one

takes less memory is chosen Similarly for 50, compare two batch sizes of 50 with one batch size

100

Memory taken = # segments + Cluster delta Cluster delta: Every time clusters change, we need to update the

base signals and base-ratio signal relationships

Base signals

Ratio signals

2

4

135

GAMPS Illustration

12

3

45

12

34

5

Partition

12

3

45

(Similar signals together)Grouping

Select Base and Ratio Signals

GAMPS Compression Illustration

12

3

45

12

34

5

Partition

12

3

45

(Similar signals together)Grouping

Compress by Amplitude

Scaling

(To overcome varying correlations)