View
216
Download
0
Category
Preview:
Citation preview
1
How to Summarize the Universe:Dynamic Maintenance of Quantiles
By:Anna C. GilbertYannis Kotidis
S. MuthukrishnanMartin J. Strauss
2
Quantiles Median, quartiles, … The general case:
Uses Statistics Estimating result set size Partitioning …
/1,...,2,1for NNk
3
Computing static quantiles Blum, Floyd, Pratt, Rivest & Tarjan
Find the i’th element Comparison based Similar to QuickSort O(n) – worst case time
4
Problems with massive data sets
O(n) time – not good enough… O(n) space – usually not affordable Dynamic environment
Cancellations are especially troublesome Usually recomputed periodically
May be very inaccurate until recomputed
Some kind of approximation is the only choice !…
5
Common approaches Deterministically chosen sample Randomization – probability of
failure Maintaining a backing sample Wavelets Most of the above approaches work
well for the incremental case, but deletions may cause inaccuracy.
6
GK – Greenwald-Khanna (‘01)
Fill the available memory with values Maintain rank ranges on values is memory. When a new value is inserted, kick a value
out of memory. Insert-only algorithm Can be extended to support deletes
(“GK2”). Maintain two instances – one for insertions and
one for deletions.
7
Maintenance of Equi-Depth Histograms (using a backing sample)
Gibbons, Matias, Poosala – ’97 Scan the dataset and choose values for
the sample using the “reservoir” method. Treat insertions as a “continuous” scan. When a deletion from the sample is
necessary – rescan only if number of items drops below a specified minimum.
Works well for a mostly-insertions enviornment.
8
The authors’ main result The RSS algorithm
RSS – Random Subset Sum Space – polylogarithmic in universe
size Proportional time A priori guarantee of accuracy within
a user specified error ε, with a user specified probability of failure δ.
9
Some formalism… The universe: U = {0, …, |U |-1} Number of tuples in data set: ||A||=N Data set can be thought of as an
array:A[i] – number of tuples with value i
Our goal for computing Ф-quantiles – find a jk such that:
NkiNkkji
)(][A)(
10
Some assumptions The universe’s size is known
Later we’ll throw that assumption away
Update = Delete + Insert
11
Computing quantiles Let’s say A[i] is known for every i.
Easy to maintain through updates Summing up array items ?
Not a very good complexity…
12
Computing quantiles (cont.) We need a method of reducing
summation overhead. We should be able to compute any
sum of items in A in logarithmic time.
The solution: Keeping computed sums of intervals.
13
Dyadic intervals - defined Atomic dyadic interval – a single point. Ij,k = [k*2log(|U|)-j,(k+1)*2log(|U|)-j-1] j – resolution level Example:
0 1 2 3 4 5 6 7I(3,0) I(3,1) I(3,2) I(3,3) I(3,4) I(3,5) I(3,6) I(3,7)
I(2,0) I(2,1) I(2,2) I(2,3)
I(1,0) I(1,1)
I(0,0)
14
Let’s say we have sums for all dyadic intervals as in the above example.
We want to compute A[0,6].
A[0,6] = I(1,0) + I(2,2) + I(3,6)
Computing an arbitrary interval
0 1 2 3 4 5 6 7I(3,0) I(3,1) I(3,2) I(3,3) I(3,4) I(3,5) I(3,6) I(3,7)
I(2,0) I(2,1) I(2,2) I(2,3)
I(1,0) I(1,1)
I(0,0)
15
Dyadic intervals - observations Log(|U|) + 1 resolution levels 2|U| - 1 dyadic intervals altogether
O(|U|) space needed to keep them all O(log(|U|)) time needed to
compute any arbitrary interval.
16
Computing quantiles (Cont.) We can now efficiently compute
any arbitrary interval in A. A ф-quantile for any k can be
computed thus: We need a jk s.t.:
A[0,jk) < kФN < a[0,jk+1) Use binary search to find it !
17
But… Keeping O(|U|) of data presents a
real space complexity problem. We need a way of estimating A[i]
on demand. … And also of estimating any
dyadic interval on demand.
18
Introducing random sets Let S be a random set of values
from U. Each value has a probability of ½
of being in S. Expectation of the number of items
in S is ½|U|.
19
Random subset sums Define ||AS|| as the number of
items in A with values in S.
Expectation of ||AS|| is ½||A||=½N. Now consider only subsets S
containing a certain value i.
Si
S A[i] A
}\{S A2
1 ]A[]A[E iUiSi
20
Random subset sums (cont.) Suppose we keep a number of
random sets S, each containing random values from U – each with probability ½.
We maintain ||AS|| for each such set. Easy to maintain during updates.
How can we now estimate A[i] ?
21
Random subset sums (cont.) We can estimate A[i] for any i with:
A[i] = 2||AS|| - ||A|| Proof:
The authors prove that repeating the process O(1/ε2) times yields the required accuracy.
][AA)A2
1]A[(2)AAE(2 }\{S ii iU
22
Random subset sums (cont.) We can also estimate any dyadic
interval Ij,k using the same method. Improvement: We can compute the
sums for dyadic intervals from a certain level.
We can now estimate any arbitrary interval in the universe…
23
Space Considerations Keeping a set of expected size ½|
U| is still O(|U|). We need a method of “keeping” a
set without actually keeping it… The technique: instead of sets,
keep random seeds of size o(log|U|) bits and compute whether a given iєS on demand.
24
Extended Hamming Code Used for generating the random sets. Provides sufficient “randomness” For example:
|U| = 8 Seed size: log|U|+1 = 4
G(seed, i) = seed X i’th column
10101010
11001100
11110000
11111111
25
RSS Algorithm Summary To compute a dyadic interval.
Compute 2||AS|| - ||A|| for sets containing the given dyadic interval.
To compute an arbitrary interval. Write it as a disjoint union of dyadic
intervals, estimate them and take a median over possible results (simplified).
To compute the quantiles. Use binary search and compute the
intervals until found.
26
Algorithm Complexity Claim The RSS algorithm’s space
complexity (for t quantile queries):
Time complexity for inserts, deletes and computing each quantile on demand is proportional to the space used.
)/))log(
log()((log 22 Ut
UO
27
Proof Outline Declare random variable
Xk=2||AIk|| if Ik is in S and 0 otherwise X – Sum of all Xk’s in a certain set Y – Sum of all X’s in a given interval Z – A number of repetitions of X.
28
Proof Outline (Cont.) In a similar fashion to previous
slides, show that Y and ||A|| can be used to compute ||AI||.
Compute the variance. Use Chebyshev’s and then
Chernoff’s inequalities, together with the computed variance, to achieve the required result.
29
What If U Is Unknown ? In practice, the universe U is not
always known. Predict a range [0, u-1] for U. Given an inserted (or updated) value
i s.t. (i > u-1), add another instance of RSS with range [u, u2-1], and so on…
Estimating dyadic intervals can be done in a single instance of RSS.
Increased cost factor: log2log(|U|).
30
Some RSS Properties RSS may return as a quantile a value
which is not really in the dataset. Order of insertions and deletions
does not affect result and accuracy. Can be parallelized quite easily (as
long as random subsets are pre-agreed).
31
Experimental Results Experiments
Static artificial dataset Dynamic artificial dataset Dynamic real dataset
Participants Naïve[l] RSS[l] GK GK2 – an improvement for GK
32
Static Artificial Dataset |U| = 220
Compute 15 quantiles at position (1/16)k for k = 1,2,…,15.
3 different distributions Uniform Zipf Normal[m,v]
Algorithm used: RSS[7] (11K footprint).
33
Errors for Zipf data
34
Errors for Normal[U/2, U/50] Distribution
35
Dynamic Artificial Dataset Insert N=104,858 items from
uniform dist. D1=Uni[1,U], U=220. Insert αN more items from uniform
dist. D2=Uni[U/2-U/32, U/2+U/32]. Delete all values from the first
insertion. Parameter α controls the mass of
the second insertion with respect to the first.
36
Dynamic Artificial Dataset Results
37
Dynamic Real Dataset Based on true Call Detail Records
(CDRs) from AT&T. Dataset used includes 4.42 million
CDRs covering a period of 18 hours. Objective: find the median length of
current calls. Probe for estimates every 10,000
records. Algorithm used: RSS[6] (4K footprint).
38
Number of Active Phone Calls Over Time
39
Error in Computation of Median Over Time
40
Average Error for Last 50 Snapshots, For Deciles
41
Conclusions – RSS Algorithm for maintaining dynamic
quantiles. Works well (within a user-defined
precision) both for insertions AND deletions.
Polylogarithmic (in universe size) in space and time complexities.
42
Recommended