1

How to Summarize the Universe:Dynamic Maintenance of Quantiles

By:Anna C. GilbertYannis Kotidis

S. MuthukrishnanMartin J. Strauss

2

Quantiles Median, quartiles, … The general case:

Uses Statistics Estimating result set size Partitioning …

/1,...,2,1for NNk

3

Computing static quantiles Blum, Floyd, Pratt, Rivest & Tarjan

Find the i’th element Comparison based Similar to QuickSort O(n) – worst case time

4

Problems with massive data sets

O(n) time – not good enough… O(n) space – usually not affordable Dynamic environment

Cancellations are especially troublesome Usually recomputed periodically

May be very inaccurate until recomputed

Some kind of approximation is the only choice !…

5

Common approaches Deterministically chosen sample Randomization – probability of

failure Maintaining a backing sample Wavelets Most of the above approaches work

well for the incremental case, but deletions may cause inaccuracy.

6

GK – Greenwald-Khanna (‘01)

Fill the available memory with values Maintain rank ranges on values is memory. When a new value is inserted, kick a value

out of memory. Insert-only algorithm Can be extended to support deletes

(“GK2”). Maintain two instances – one for insertions and

one for deletions.

7

Maintenance of Equi-Depth Histograms (using a backing sample)

Gibbons, Matias, Poosala – ’97 Scan the dataset and choose values for

the sample using the “reservoir” method. Treat insertions as a “continuous” scan. When a deletion from the sample is

necessary – rescan only if number of items drops below a specified minimum.

Works well for a mostly-insertions enviornment.

8

The authors’ main result The RSS algorithm

RSS – Random Subset Sum Space – polylogarithmic in universe

size Proportional time A priori guarantee of accuracy within

a user specified error ε, with a user specified probability of failure δ.

9

Some formalism… The universe: U = {0, …, |U |-1} Number of tuples in data set: ||A||=N Data set can be thought of as an

array:A[i] – number of tuples with value i

Our goal for computing Ф-quantiles – find a jk such that:

NkiNkkji

)(][A)(

10

Some assumptions The universe’s size is known

Later we’ll throw that assumption away

Update = Delete + Insert

11

Computing quantiles Let’s say A[i] is known for every i.

Easy to maintain through updates Summing up array items ?

Not a very good complexity…

12

Computing quantiles (cont.) We need a method of reducing

summation overhead. We should be able to compute any

sum of items in A in logarithmic time.

The solution: Keeping computed sums of intervals.

13

Dyadic intervals - defined Atomic dyadic interval – a single point. Ij,k = [k*2log(|U|)-j,(k+1)*2log(|U|)-j-1] j – resolution level Example:

0 1 2 3 4 5 6 7I(3,0) I(3,1) I(3,2) I(3,3) I(3,4) I(3,5) I(3,6) I(3,7)

I(2,0) I(2,1) I(2,2) I(2,3)

I(1,0) I(1,1)

I(0,0)

14

Let’s say we have sums for all dyadic intervals as in the above example.

We want to compute A[0,6].

A[0,6] = I(1,0) + I(2,2) + I(3,6)

Computing an arbitrary interval

0 1 2 3 4 5 6 7I(3,0) I(3,1) I(3,2) I(3,3) I(3,4) I(3,5) I(3,6) I(3,7)

I(2,0) I(2,1) I(2,2) I(2,3)

I(1,0) I(1,1)

I(0,0)

15

Dyadic intervals - observations Log(|U|) + 1 resolution levels 2|U| - 1 dyadic intervals altogether

O(|U|) space needed to keep them all O(log(|U|)) time needed to

compute any arbitrary interval.

16

Computing quantiles (Cont.) We can now efficiently compute

any arbitrary interval in A. A ф-quantile for any k can be

computed thus: We need a jk s.t.:

A[0,jk) < kФN < a[0,jk+1) Use binary search to find it !

17

But… Keeping O(|U|) of data presents a

real space complexity problem. We need a way of estimating A[i]

on demand. … And also of estimating any

dyadic interval on demand.

18

Introducing random sets Let S be a random set of values

from U. Each value has a probability of ½

of being in S. Expectation of the number of items

in S is ½|U|.

19

Random subset sums Define ||AS|| as the number of

items in A with values in S.

Expectation of ||AS|| is ½||A||=½N. Now consider only subsets S

containing a certain value i.

Si

S A[i] A

}\{S A2

1 ]A[]A[E iUiSi

20

Random subset sums (cont.) Suppose we keep a number of

random sets S, each containing random values from U – each with probability ½.

We maintain ||AS|| for each such set. Easy to maintain during updates.

How can we now estimate A[i] ?

21

Random subset sums (cont.) We can estimate A[i] for any i with:

A[i] = 2||AS|| - ||A|| Proof:

The authors prove that repeating the process O(1/ε2) times yields the required accuracy.

][AA)A2

1]A[(2)AAE(2 }\{S ii iU

22

Random subset sums (cont.) We can also estimate any dyadic

interval Ij,k using the same method. Improvement: We can compute the

sums for dyadic intervals from a certain level.

We can now estimate any arbitrary interval in the universe…

23

Space Considerations Keeping a set of expected size ½|

U| is still O(|U|). We need a method of “keeping” a

set without actually keeping it… The technique: instead of sets,

keep random seeds of size o(log|U|) bits and compute whether a given iєS on demand.

24

Extended Hamming Code Used for generating the random sets. Provides sufficient “randomness” For example:

|U| = 8 Seed size: log|U|+1 = 4

G(seed, i) = seed X i’th column

10101010

11001100

11110000

11111111

25

RSS Algorithm Summary To compute a dyadic interval.

Compute 2||AS|| - ||A|| for sets containing the given dyadic interval.

To compute an arbitrary interval. Write it as a disjoint union of dyadic

intervals, estimate them and take a median over possible results (simplified).

To compute the quantiles. Use binary search and compute the

intervals until found.

26

Algorithm Complexity Claim The RSS algorithm’s space

complexity (for t quantile queries):

Time complexity for inserts, deletes and computing each quantile on demand is proportional to the space used.

)/))log(

log()((log 22 Ut

UO

27

Proof Outline Declare random variable

Xk=2||AIk|| if Ik is in S and 0 otherwise X – Sum of all Xk’s in a certain set Y – Sum of all X’s in a given interval Z – A number of repetitions of X.

28

Proof Outline (Cont.) In a similar fashion to previous

slides, show that Y and ||A|| can be used to compute ||AI||.

Compute the variance. Use Chebyshev’s and then

Chernoff’s inequalities, together with the computed variance, to achieve the required result.

29

What If U Is Unknown ? In practice, the universe U is not

always known. Predict a range [0, u-1] for U. Given an inserted (or updated) value

i s.t. (i > u-1), add another instance of RSS with range [u, u2-1], and so on…

Estimating dyadic intervals can be done in a single instance of RSS.

Increased cost factor: log2log(|U|).

30

Some RSS Properties RSS may return as a quantile a value

which is not really in the dataset. Order of insertions and deletions

does not affect result and accuracy. Can be parallelized quite easily (as

long as random subsets are pre-agreed).

31

Experimental Results Experiments

Static artificial dataset Dynamic artificial dataset Dynamic real dataset

Participants Naïve[l] RSS[l] GK GK2 – an improvement for GK

32

Static Artificial Dataset |U| = 220

Compute 15 quantiles at position (1/16)k for k = 1,2,…,15.

3 different distributions Uniform Zipf Normal[m,v]

Algorithm used: RSS[7] (11K footprint).

33

Errors for Zipf data

34

Errors for Normal[U/2, U/50] Distribution

35

Dynamic Artificial Dataset Insert N=104,858 items from

uniform dist. D1=Uni[1,U], U=220. Insert αN more items from uniform

dist. D2=Uni[U/2-U/32, U/2+U/32]. Delete all values from the first

insertion. Parameter α controls the mass of

the second insertion with respect to the first.

36

Dynamic Artificial Dataset Results

37

Dynamic Real Dataset Based on true Call Detail Records

(CDRs) from AT&T. Dataset used includes 4.42 million

CDRs covering a period of 18 hours. Objective: find the median length of

current calls. Probe for estimates every 10,000

records. Algorithm used: RSS[6] (4K footprint).

38

Number of Active Phone Calls Over Time

39

Error in Computation of Median Over Time

40

Average Error for Last 50 Snapshots, For Deciles

41

Conclusions – RSS Algorithm for maintaining dynamic

quantiles. Works well (within a user-defined

precision) both for insertions AND deletions.

Polylogarithmic (in universe size) in space and time complexities.

42