Hokusai - Sketching streams in real time

HokusaiSketching streams in real time

Sergiy Matusevych1

Alexander J. Smola2

Amr Ahmed2

1Yahoo! Research, Santa Clara, CA2Google, Mountain View, CA

UAI 2012

Matusevych, Smola, Ahmed Hokusai — Sketching streams in real time UAI 2012 1

Thanks

Alex SmolaGoogle and CMU

Amr AhmedGoogle


Motivation

I Compute frequencies of elements in the data streamI Item frequencies change over time.I Number of items unkonwn and variable.I Example - logging query frequency over time.


Motivation

I Compute frequencies of elements in the data streamI Item frequencies change over time.I Number of items unkonwn and variable.I Example - logging query frequency over time.

I ApplicationsI Flow counting for IP traffic (who sent what, when and how much)I Spam detection and filtering (detect bursts immediately)I Website analytics (feedback to editors, trend detection)

I State of the artI CountMin sketch is instantaneous but does not log time.I Naive snapshotting costs linear memory.I MapReduce batch job provides exact counts but long delays.

I Resource constraintsI Fixed memory footprint for entire sketch regardless of durationI High query throughputI Real time aggregation and response


Strategy

1. Use CountMin sketch to store snapshots of data.(this solves the real time logging problem)

2. Compress snapshots linearly as they ageI We care most about recent events

I Logarithmic storage sinceT∑t=1

t−1 = O(logT )

3. Exploit CountMin data structure for efficient compressionI Variant 1: reduce storage per snapshotI Variant 2: increase timespan per snapshot

4. Interpolate between both variants for improved accuracy


CountMin Sketch (Cormode & Muthukrishnan)

M ∈ Rd×n matrix

d hash functions

n bins

hash h1 M11 M12 M13 M14 M15 M16. . . M1n

hash h2 M21 M22 M23 M24 M25 M26. . . M2n

hash h3 M31 M32 M33 M34 M35 M36. . . M3n

x

I In-memory data structure for instantaneous retrieval

I Aggregate statistic of observation interval (instantanous retrieval)

I Intuition — Bloom filter with integers

Algorithm

insert(x):for i = 1 to d doM[i , hi (x)]← M[i , hi (x)] + 1

end for

query(x):nx ← min

i∈{1,...d}M[i , hi (x)]

return nx


Guarantees

M ∈ Rd×n matrix

d hash functions

n bins

hash h1 M11 M12 M13 M14 M15 M16. . . M1n

hash h2 M21 M22 M23 M24 M25 M26. . . M2n

hash h3 M31 M32 M33 M34 M35 M36. . . M3n

x

I Approximation guaranteeFor sketch with d = dlog 1

δ e and n = d eε e we have with probability1− δ that the estimate nx deviates from the count nx via

nx ≤ nx ≤ nx + ε∑x ′

nx ′ for all x .

I Linear statistic of the dataI Power law distributions with exponent z only use O(Nε−1/z) space.


Step 1: Combining time intervals

M ∈ Rd×n matrix

d hash functions

n bins

hash h1 M11 M12 M13 M14 M15 M16. . . M1n

hash h2 M21 M22 M23 M24 M25 M26. . . M2n

hash h3 M31 M32 M33 M34 M35 M36. . . M3n

x

I MT and MT ′ sketches at time intervals T and T ′ with T ∩ T ′ = ∅.I Combine sketches by adding them up

+


Step 1: Efficient computation

I Keep aggregates for time intervals of length {1, 1, 2, 4, 8, . . . , 2m}.I Insert into the leftmost aggregation interval.

I Aggregate as cumulative sum from the left using 1 +n∑

i=0

2i = 2n+1

I Computation is∞∑n=1

n · 2−n = O(1) amortized time, O(log t) space.

4

2

1

1 1

1 2

1 1

1 1

1 1

1 1

42

4

2

1

2

1

1 1

1 1 2 4





i=0

2i = 2n+1



2

2

81

1 1

1 1

1 1

1 421

8

8

8

4

4

4

2

4

2

1 1

1 1

1 1

42

4

2

1

1 1

1 1 2 4

8

8

8

8





i=0

2i = 2n+1



2

2

81

1 1

1 1

1 1

1 421

8

8

8

4

4

4

2

4

2

1 1

1 1

1 1

42

4

2

1

1 1

1 1 2 4

8

8

8

8


Step 2: Folding over

M ∈ Rd×n matrix

d hash functions

n bins

hash h1 M11 M12 M13 M14 M15 M16. . . M1n

hash h2 M21 M22 M23 M24 M25 M26. . . M2n

hash h3 M31 M32 M33 M34 M35 M36. . . M3n

x

I Mb is sketch with n = 2b bins.

I Mb−1 can obtained as

Mb−1[i , j ] = Mb[i , j ] + Mb[i , j + 2b−1]

by “folding over” the sketch



I Halve the size of the sketch every 2t intervals.I Computation costs O(1) time and O(log t) space.

. . .

1 x 16 bins

2 x 8 bins

4 x 4 bins

interval 1

interval 2 3

4 5 6 7


Step 3: Resolution Interpolation

I Time aggregation reports good estimate over long time interval.

I Item aggregation reports poor estimate over short time interval.

I Marginals of joint distribution — assume independence & interpolate

n(t)

n(x)n I Torso and TailI Item aggregated estimate nxI Time aggregated estimate ntI Count interpolation

nxt =nx · nt

nwhere n =

∑t

nt =∑x

nx

I HeadI Sketch accuracy decreases with e · tI Use regular CountMin sketch whenever

n(x , t) > e · t · 2−b


Setup and Throughput

Web query data, 5 days sample

Term frequency

Num

ber

of u

niqu

e te

rms

100

102

104

106

97.9M unique terms,378.1M total

100 101 102 103 104 105 106

Wikipedia data

Term frequency

Num

ber

of u

niqu

e te

rms

100

101

102

103

104

105

106 4.5M unique terms,1291.5M total

100 102 104 106

Configuration

I PlatformI 64-bit LinuxI 4-core 2GHz x86I 16GB RAMI Gigabit network

I Sketch setupI 4 hash functionsI 223 binsI 211 aggregation

intervals (7 days in5 minute intervals)

I 3-gram interpolation12GB sketch with

I 3 hash functionsI 230 bins


Setup and Throughput

Web query data, 5 days sample

Term frequency

Num

ber

of u

niqu

e te

rms

100

102

104

106

97.9M unique terms,378.1M total

100 101 102 103 104 105 106

Wikipedia data

Term frequency

Num

ber

of u

niqu

e te

rms

100

101

102

103

104

105

106 4.5M unique terms,1291.5M total

100 102 104 106

Speed

I SoftwareI Client-server systemI ICE middlewareI 1 server, 10 clients

I Throughput/sI 50k insertsI 22k requests

(time aggregation)I 8.5k requests

(resolution interp.)

I Limiting FactorsI TCP/IP Overhead

Package queryI Memory latency

Random access


Accuracy (aggregate absolute error n − n)


Accuracy (stratified absolute error n − n)


Sketching for Graphical Models

I GoalI Observe stream of observationsI Estimate joint probability in O(1) timeI CountMin is good for head but interpolation better for torso and tail

I General StrategyI Markov network with junction tree: cliques C and separator sets S.I Estimate counts for xC and xS with C ∈ C and S ∈ S to generate

p(x) = n|S|−|C|∏C∈C

nxC∏S∈S

n−1xS .

I Estimates are fast — only lookup in CountMin sketch. No need tosolve convex program for graphical model inference.

I Markov Chain

p(abc) ≈ n−3 · na · nb · nc Unigrams

p(abc) ≈ n−2 · nab · nbcnb

Bigrams

Backoff smoothing (e.g. Kneser-Ney) in practice.


n-gram Interpolation

I Trigram approximation

I Wikipedia dataset (1291.5M terms, 405M unique trigrams)

Absolute error Relative error

Unigram approximation 2.50 · 107 0.266Bigram approximation 1.22 · 106 0.013Trigram sketching (CountMin) 8.35 · 106 0.089

I Sketching trigrams is not accurate enough on the tail.


Summary

I Fast and simple algorithm to aggregate statistics of data streams.I Effective compressed representation of the temporal data.I Works well for graphical models.I High-performance scalable implementation with O(1) time access.I Can be distributed over many servers.

Hokusai Katsushika

Great Wave off Kanagawa


Science

Hokusai - Sketching streams in real time