Optimal Space Lower Bounds for all Frequency

Moments

David Woodruff

Based on SODA ’04 paper

The Streaming Model [AMS96]

0113734 … Stream of elements a1, …, aq each in {1, …, m} Want to compute statistics on stream Elements arranged in adversarial order Algorithms given one pass over stream Goal: Minimum space algorithm

Frequency Moments

Notation q = stream size, m = universe size fi = # occurrences of item i

Why are frequency moments important?

F0 = # of Distinct elements F1 = q F2 = repeat rate

k-th moment

Applications

Estimating # distinct elts. w/ low space Estimate selectivity of queries to DB w/o expensive sort Routers gather # distinct destinations w/limited memory.

Estimating F2 estimates size of self-joins:

Bob x

Alice y

Bob z

Bob a

Alice b

Bob c

,

Alice b y

Bob a x

Bob a z

Bob c x

Bob c z

The Best Determininistic Algorithm

Trivial algorithm for Fk

Store/update fi for each item i, sum fi

k at end Space = O(mlog q): m items i, log q bits to count fi

Negative Results [AMS96]: Compute Fk exactly => (m) space Any deterministic alg. outputs x with |Fk – x| < must use (m) space

What about randomized algorithms?

Randomized Approx Algs for Fk

Randomized alg. -approximates Fk if outputs x s.t. Pr[|Fk – x| < Fk ] > 2/3

Can -approximate F0 [BJKST02], F2 [AMS96], Fk [CK04], k > 2 in space:(big-Oh notation suppresses polylog(1/, m, q) factors)

Ideas: Hashing: O(1)-wise independence Sampling

Example: F0 [BJKST02]

Idea: For random function h:[m] -> [0,1] and distinct elts b1, b2, …, bF0

, expect mini h(bi) ¼ 1/F0

Algorithm: Choose 2-wise indep. hash function h: [m] -> [m3] Maintain t = (1/2) distinct smallest values h(bi) Let v be t-th smallest value Output tm3/v as estimate for F0

Success prob up to 1- => take median O(log 1/) copies Space: O((log 1/)/2)

Example: F2 [AMS99]

Algorithm: Choose 4-wise indep. hash function h:[m] -> {-1,1} Maintain Z = i in [m] fi ¢ h(i) Output Y = Z2 as estimate for F2

Correctness:

Chebyshev’s inequality => O(1/2) space

Previous Lower Bounds:

[AMS96] 8 k, –approximating Fk => (log m) space

[Bar-Yossef] -approximating F0 => (1/) space

[IW03] -approximating F0 => space if

Questions: Does the bound hold for k 0? Does it hold for F0 for smaller ?

Our First Result

Optimal Lower Bound: 8 k 1, any = (m-.5), -approximate Fk => (-2) bits of space.

F1 = q trivial in log q space Fk trivial in O(m log q) space, so need = (m-.5)

Technique: Reduction from 2-party protocol for computing Hamming distance (x,y)

Use tools from communication complexity

Lower Bound Idea

x 2 {0,1}m

y 2 {0,1}m

Stream s(x) Stream s(y)

(1 § ) Fk algorithm A

(1 § ) Fk algorithm A

Internal state of A

• Compute (1 § ) Fk(s(x) ± s(y)) w.p. > 2/3 • Idea: If can decide f(x,y) w.p. > 2/3, space used by A at least randomized 1-way comm. Complexity of f

S

Alice Bob

Randomized 1-way comm. complexity

Boolean function f: X £ Y ! {0,1} Alice has x 2 X, Bob y 2 Y. Bob wants f(x,y) Only 1 message m sent: must be from Alice to Bob Communication cost = maxx,y Ecoins [|m|]

-error randomized 1-way communication complexity R(f), is cost of optimal protocol computing f with probability ¸ 1-

Ok, but how do we lower bound R(f)?

Shatter Coefficients [KNR] F = {f : X ! {0,1}} function family, f 2 F length-|X| bitstring

For S µ X, shatter coefficient SC(fS) of S :

|{f |S}f 2 F| = # distinct bitstrings when F restricted to S

SC(F, p) = maxS µ X, |S| = p SC(fS). If SC(fS) = 2|S|, S shattered

Treat f: X £ Y ! {0,1} as function family fX : fX = { fx(y) : Y ! {0,1} | x 2 X }, where fx(y) = f(x,y)

Theorem [BJKS]: For every f: X £ Y ! {0,1}, every integer p, R1/3(f) = (log(SC(fX, p)))

Warmup: (1/) Lower Bound [Bar-Yossef]

Alice input x 2R {0,1}m, wt(x) = m/2 Bob input y 2R {0,1}m, wt(y) = m s(x), s(y) any streams w/char. vectors x, y PROMISE:

(1) wt(x Æ y) = 0 OR (2) wt(x Æ y) = m f(x,y) = 0 f(x,y) = 1

F0(s(x) ± s(y)) = m/2 + m F0(s(x) ± s(y)) = m/2

R1/3(f) = (1/) [Bar-Yossef] (uses shatter coeffs) (1+’)m/2 < (1 - ’)(m/2 + m) for ’ = () Hence, can decide f ! F0 alg. uses (1/) space Too easy! Can replace F0 alg. with a Sampler!

Our Reduction: Hamming Distance Decision Problem (HDDP)

Lower bound R1/3(f) via SC(fX, t), but need a lemma

Set t = (1/2)

x 2 {0,1}t y 2 {0,1}t

Alice Bob

Promise Problem :

(x,y) · t/2 – (t1/2) (x,y) > t/2 f(x,y) = 0 OR f(x,y) = 1

Main Lemma

S µ{0,1}n

y= T= S-T

9 S µ {0,1}n with |S| = n s.t. exist 2(n) “good” sets T µ S s.t.

9 y 2 {0,1}n s.t 8 t 2 T, (y, t) · n/2 – cn1/2 for some c > 0 8 t 2 S – T, (y,t) > n/2

Lemma Resolves HDDP Complexity

Theorem: R1/3(f) = (t) = (-2). Proof:

Alice gets yT for random good set T applying main lemma with n = t.

Bob gets random s 2 S Let f: {yT }T £ S ! {0,1}. Main Lemma =>SC(f) = 2(t)

[BJKS] => R1/3(f) = (t) = (-2)

Corollary: (1/2) space for randomized 2-party protocol to approximate (x,y) between inputs

First known lower bound in terms of !

Back to Frequency Moments

Use -approximator for Fk to solve HDDP

y 2 {0,1}t s 2 S µ {0,1}t

Fk Alg Fk AlgState

ay asi-th universe element included exactlyonce in stream ay iff yi = 1 (as same)

Solving HDDP with Fk

Alice/Bob compute -approx to Fk(ay ± as) Fk(ay ± as) = 2k wt(y Æ s) + 1k (y,s) For k 1,

Conclusion: -approximating Fk(ay ± as) decides HDDP, so space for Fk is (t) = (-2)

Alice also transmits wt(y) in log m space.

Back to the Main Lemma

Recall: show 9 S µ {0,1}n with |S| = n s.t. 2(n) “good” sets T µ S s.t:

9 y 2 {0,1}n s.t 1. 8 t 2 T, (y, t) · n/2 – cn1/2 for some c > 02. 8 t 2 S – T, (y,t) > n/2

Probabilistic Method Choose n random elts in {0,1}n for S Show arbitrary T µ S of size n/2 is good with

probability > 2-zn for constant z < 1. Expected # good T is 2(n)

So exists S with 2(n) good T

Proving the Main Lemma

T ={t1, …, tn/2} µ S arbitrary Let y be majority codeword of T What is probability p that both:

1. 8 t 2 T, (y, t) · n/2 – cn1/2 for some c > 02. 8 t 2 S – T, (y,t) > n/2

Put x = Pr[8 t 2 T, (y,t) · n/2 – cn1/2] Put y = Pr[8 t 2 S-T, (y,t) > n/2] = 2-n/2

Independence => p = xy = x2-n/2

The Matrix Problem

Wlog, assume y = 1n (recall y is majority word) Want lower bound Pr[8 t 2 T, (y,t) · n/2 – cn1/2] Equivalent to matrix problem:

t1 ->t2 -> …tn/2 ->

101001000101111001100101011100011110001110111101010101101010111011100011

For random n/2 x n binary matrix M, each column majority 1, what is probablity each row ¸ n/2 + cn1/2 1s?

A First Attempt

Set family A µ 2^{0,1}n monotone increasing if S1 2 A, S1 µ S2 => S2 2 A For uniform distribution on S µ {0,1}n, and A, B monotone

increasing families, [Kleitman]Pr[A Å B] ¸ Pr[A] ¢ Pr[B]

First try: Let R be event M ¸ n/2 + cn1/2 1s in each row, C event M

majority 1 in each column Pr[8 t 2 T, (y,t) · n/2 – cn1/2] = Pr[R | C] = Pr[R Å C]/Pr[C] M characteristic vector of subset of [.5n2] => R,C monotone

increasing => Pr[R Å C]/Pr[C] ¸ Pr[R]Pr[C]/Pr[C] = Pr[R] < 2-n/2

But we need > 2-zn/2 for constant z < 1, so this fails…

A Second Attempt

Second Try: R1: M ¸ n/2 + cn1/2 1s in first m rows R2: M ¸ n/2 + cn1/2 1s in remaining n/2-m rows C: M majority 1 in each column

Pr[8 t 2 T, (y,t) · n/2 – cn1/2] = Pr[R1 Å R2 | C] = Pr[R1 Å R2 Å C]/Pr[C] R1, R2, C monotone increasing => Pr[R1 Å R2 Å C]/Pr[C] ¸ Pr[R1 Å C]Pr[R2]/Pr[C] = Pr[R1 | C] Pr[R2] Want this at least 2-zn/2 for z < 1 Pr[ Xi > n/2 + cn1/2] > ½ - c (2/pi)1/2 [Stirling] Independence => Pr[R2] > (½ - c(2/pi)1/2)n/2 - m

Remains to show Pr[R1 | C] large.

Computing Pr[R1 | C]

Pr[R1 | C] = Pr[M ¸ n/2 + cn1/2 1s in 1st m rows | C]

Show Pr[R1 | C] > 2-z’m for certain constant z’ < 1

Ingredients: Expect to get n/2 + (n1/2) 1s in each of 1st m rows | C Use negative correlation of entries in a given row => show n/2 + (n1/2) 1s in a given row w/good probability

for small enough c A simple worst-case conditioning argument on these 1st

m rows shows they all have ¸ n/2 + cn1/2 1s

Completing the Proof

Recall: what is probability p = xy, where1. x = Pr[ 8 t 2 T, (y, t) · n/2 – cn1/2]2. y = Pr[ 8 t 2 S – T, (y,t) > n/2] = 2-n/2

3. R1: M ¸ n/2 + cn1/2 1s in first m rows

4. R2: M ¸ n/2 + cn1/2 1s in remaining n/2-m rows

5. C: M majority 1 in each column x ¸ Pr[R1 | C] Pr[R2] ¸ 2-z’m (½ - c(2/pi)1/2)n/2 – m

Analysis shows z’ small so this ¸ 2-z’’n/2, z’’ < 1 Hence p = xy ¸ 2-(z’’+1)n/2 Hence expected # good sets 2n-O(log n)p = 2(n) So exists S with 2(n) good T

Bipartite Graphs

Matrix Problem Bipartite Graph Counting Problem:

How many bipartite graphs exist on n/2 by n vertices s.t. each left vertex has degree > n/2 + cn1/2 and each right vertex degree > n/2?

… …

Our Result on # of Bipartite Graphs

Bipartite graph count: Argument shows at least 2n^2/2 – zn/2 –n such bipartite

graphs for constant z < 1.

Main lemma shows # bipartite graphs on n + n vertices w/each vertex degree > n/2 is > 2n^2-zn-n

Can replace > with <

Previous knowncount: 2n^2-2n [MW – personal comm.] Follows easily from Kleitman inequality

Summary

Results:

Optimal Fk Lower Bound: 8 k 1 and any = (m-1/2), any -approximator for Fk must use (-2) bits of space.

Communication Lower Bound of (-2) for one-way communication complexity of (, )-approximating (x, y)

Bipartite Graph Count: # bipartite graphs on n + n vertices w/each vertex degree > n/2 at least 2n^2-zn-n for constant z < 1.