Online Piece-wise Linear Approximation of Numerical Streams with Precision Guarantees

Hazem ElmeleegyPurdue University

Ahmed Elmagarmid (Purdue) Emmanuel Cecchet (UMass) Walid Aref (Purdue)

Willy Zwaenepoel (EPFL)

Outline

Introduction

Swing & Slide Filters

Experiments

Conclusion

Application Scenario

Transmitter Receiver

Some Common Applications: Cluster Monitoring Sensor Networks Stock Market

The Problem

Goal Minimize amount of transmitted data

Saves bandwidth Saves storage (at the receiver side) Saves battery life (esp. for sensor networks)

Using piece-wise linear approximation

Assumptions Receiver can tolerate:

A bounded error for each data point received (max error = ) A maximum lag behind the transmitter

Terminology We refer to any algorithm to solve this problem as a filtering

technique, or simply a filter

Existing Techniques

t1 t2 t3 t4 t5

Time

Val

ue

x1

x2x3

x4

x5

Cache Filter The transmitter caches

the last transmitted value.

A new value is transmitted only if it is more than away from the cached value.

Piece-wise constant approximation

Existing Techniques

t1 t2 t3 t4 t5

Time

Val

ue

x1

x2x3

x4

x5

Cache Filter The transmitter caches

the last transmitted value.

A new value is transmitted only if it is more than away from the cached value.

Piece-wise constant approximation

Existing Techniques

t1 t2 t3 t4 t5

Time

Val

ue

x1

x2x3

x4

x5

Linear Filter The transmitter maintains

a line segment that can approximate the last observed data points.

The line segment is updated only when a new data point falls more than away from the maintained line.

Outline

IntroductionIntroduction

Swing & Slide Filters

Experiments

Conclusion

Swing and Slide Filters

Key Idea

Maintain a set of candidate line segments at any given time

Postpone the selection decision as late as possible to accommodate more points

Swing Filter

t1 t2 t3 t4 t5

Time

Val

ue

x1

x2x3

x4

x5

Connected line segments

Complexity Maintains upper and lower

segments only O(1) space and time

complexity

Lag If max lag is exceeded,

switch to linear filter

Correctness Proof of correctness in the

paper

Slide Filter

t1 t2 t3 t4 t5

Time

Val

ue

x1

x2x3

Slide Filter

t1 t2 t3 t4 t5

Time

Val

ue

x1

x2x3

x4

Slide Filter

t1 t2 t3 t4 t5

Time

Val

ue

x1

x2x3

x4

x5

Optimization #1 Connect line segments whenever

possible

Optimization #2 Do not maintain all the data points

currently being approximated Maintain their convex hull only

Complexity O(h) space and time complexity h is the number of data points on the

convex hull --- very small in practice

Lag If max lag is exceeded, switch to linear

filter

Correctness Proof of correctness in the paper

Outline

IntroductionIntroduction

Swing & Slide FiltersSwing & Slide Filters

Experiments

Conclusion

Compression Ratios for the Sea Temperature Signal

20.5

21

21.5

22

22.5

23

23.5

24

24.5

0 2000 4000 6000 8000 10000 12000Time (minutes)

Tem

per

atu

re (

C)

0

10

20

30

40

50

60

70

80

90

0 0.1 0.316 1 3.16 10

Precision width (% of range)

Co

mp

ress

ion

rat

io

cache linear

sw ing slide

Sea Surface Temperature

Effect of Signal Behavior (Degree of Monotonicity)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

0 0.1 0.2 0.3 0.4 0.5

Probability of decrease in value per data point

Co

mp

ress

ion

rat

io

cache linear

sw ing slide

Synthetic Signal:

Random walk with probability p to increase and (1-p) to decrease.

Overhead for the Sea Temperature Signal

0

10

20

30

40

50

60

0 0.1 0.316 1 3.16 10 31.6 100

Precision width (% of range)

Pro

cess

ing

tim

e (

ms

/ dat

a p

oin

t)cache linearsw ing non-optimized slideoptimized slide

Outline

IntroductionIntroduction

Swing & Slide FiltersSwing & Slide Filters

ExperimentsExperiments

Conclusion

Conclusion

We introduced two new filtering techniques: the swing and slide filters

They have significantly higher compression ratios compared to earlier techniques, especially the slide filter

They have a small overhead, and hence are suitable for overhead-sensitive applications

Thank you

Questions?