Lecture 5 Today, how to solve recurrences We learned “guess and proved by induction” We also learned “substitution” method Today, we learn the “master

Lecture 5

Today, how to solve recurrences We learned “guess and proved by induction” We also learned “substitution” method Today, we learn the “master theorem”

More divide and conquer: closest pair problem matrix multiplication

Master Theorem

Theorem 4.1 (CLRS, Theorem 4.1) Let a ≥ 1 and b > 1 be constants. Let f(n) be a function and let T(n) be defined on the nonnegative integers by

T(n) = aT(n/b) + f(n). Then

10

largefor )()/( AND )(

)(

)(

)(

log)(

log

log

log

log

log

c

nncfbnafnnf

nnf

nOnf

nf

nn

n

nT

a

a

a

a

a

b

b

b

b

b

Note

Only apply to a particular family of recurrences.

f(n) is positive for large n. Key is to compare f(n) with nlog_b a Case 2, more general is f(n) = Θ( nlog_b a lgkn).

Then the result is T(n) = Θ( nlog_b a lgk+1n). Sometimes it does not apply. Ex. T(n) =

4T(n/2) + n2 /logn.

Proof ideas of Master Theorem Consider a tree with T(n) at the root, and apply the recursion to

each node, until we get down to T(1) at the leaves. The first recursion is T(n) = aT(n/b) + f(n), so assign a cost of f(n) to the root. At the next level we have “a” nodes, each with a cost of T(n/b). When we apply the recursion again, we get a cost of af(n/b) for all of these. At the next level we have a2 nodes, each with a cost of T(n/b2). We get a cost of a2f(n/b2). We continue down to T(1) at the leaves. There are alog_b n leaves and each costs Θ(1), which gives Θ(alog_b n). The total cost associated with f is Σ 0 ≤ i ≤ log_b n - 1 ai f(n/bi).

Thus T(n) = Θ(n log_b a) + Σ 0 ≤ i ≤ (log_b n) - 1 ai f(n/bi). The three cases now come from deciding which term is

dominant. In case (1), the Θ term is dominant. In case (2), the terms are roughly equal (but the second term has an extra lg n factor). In case (3), the f(n) term is dominant. The details are somewhat painful, but can be found in CLRS, pp. 76-84.

f (n/b)

Idea of master theorem

f (n/b) f (n/b)

(1)

…Recursion tree:

…

f (n) a

f (n/b2)f (n/b2) f (n/b2)…ah = logbn

f (n)

a f (n/b)

a2 f (n/b2)

…#leaves = ah

= alogbn

= nlogba

nlogba(1)

Three common cases

Compare f (n) with nlogba:1. f (n) = O(nlogba – ) for some constant > 0.

• f (n) grows polynomially slower than nlogba (by an n factor).

Solution: T(n) = (nlogba) .

f (n/b)


f (n/b) f (n/b)

(1)

…Recursion tree:

…

f (n) a


f (n)

a f (n/b)

a2 f (n/b2)

…

nlogba(1)CASE 1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight. (nlogba)

These functions increasefrom top to bottomgeometrically, hence weonly need to have thelast bottom term

Case 2

Compare f (n) with nlogba:

2. f (n) = (nlogba lgkn) for some constant k 0.• f (n) and nlogba grow at similar rates.• This is clear for k=0. For k>0, the intuition is

that lgk n factor remain for constant fraction of levels, hence sum to the following

Solution: T(n) = (nlogba lgk+1n) .

f (n/b)


f (n/b) f (n/b)

(1)

…Recursion tree:

…

f (n) a


f (n)

a f (n/b)

a2 f (n/b2)

…

nlogba(1)CASE 2: (k = 0) The weight is approximately the same on each of the logbn levels.

(nlogbalg n)

All levels same

Case 3, c<1, akf(n/bk) geometrically decreases hence = Θ(f(n))

Compare f (n) with nlogba:3. f (n) = (nlogba + ) for some constant > 0.

• f (n) grows polynomially faster than nlogba (by an n factor),

and f (n) satisfies the regularity condition that a f (n/b) c f (n) for some constant c < 1.Solution: T(n) = ( f (n) ) .

f (n/b)


f (n/b) f (n/b)

(1)

…Recursion tree:

…

f (n) a


f (n)

a f (n/b)

a2 f (n/b2)

…

nlogba(1)CASE 3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight. ( f (n))

af(n/b)<(1-ε)f(n)

Examples for the Master Theorem The Karatsuba recurrence has a = 3, b = 2, f(n) = cn. Then case

1 applies, and so T(n) = Θ(n l og_2 3 ), as we found. The mergesort recurrence has a = 2, b = 2, f(n) = n. Then case

2 applies, and so T(n) = Θ(n lg n). Finally, a recurrence like T(n) = 3T(n/2) + n2 gives rise to case 3.

In this case f(n) = n2, so 3f(n/2) = 3 (n/2)2 = (3/4) n2 ≤ c n2 for c = 3/4, and so T(n) = Θ(n2).

Note that the master theorem does not cover all cases. In particular, it does not cover the case

T(n) = 2 T(n/2) + n / lg n since then the only applicable case is case 3, but then the

inequality involving f does not hold.

Closest pair problem

Input: A set of points P = {p1,…, pn} in two

dimensions Output:

The pair of points pi, pj that minimize the Euclidean distance between them.

Distances Euclidean distance

1x 2x

2y

1y 11, yx

22 , yx

221

2212211 ,, yyxxyxyx

Closest Pair Problem


Divide and Conquer

O(n2) time algorithm is easy Assumptions:

No two points have the same x-coordinates No two points have the same y-coordinates

How do we solve this problem in 1 dimension? Sort the number and walk from left to right to find

minimum gap.

Divide and Conquer

Divide and conquer has a chance to do better than O(n2).

We can first sort the points by their x-coordinates and sort also by y-coordinates


Divide and Conquer for the Closest Pair Problem

Divide by x-median

Divide

Divide by x-median

L R

Conquer

Conquer: Recursively solve L and R

L R

1

2

Combination I

Take the smaller one of 1 , 2 : = min(1 , 2 )

L R

2

Combination IIIs there a point in L and a point in R whose distance is smaller than ?

= min(1 , 2 )

L R

Combination II

If the answer is “no” then we are done!!! If the answer is “yes” then the closest such

pair forms the closest pair for the entire set

How do we determine this?


L R


Need only to consider the narrow bandO(n) time

L R


Denote this set by S, assume Sy is the sorted list of S by the y-coordinates.

L R

Combination II

There exists a point in L and a point in R whose distance is less than if and only if there exist two points in S whose distance is less than .

If S is the whole thing, did we gain anything?

CLAIM: If s and t in S have the property that ||s-t|| < , then s and t are within 15 positions of each other in the sorted list Sy.


L R

There are at most one point in each box of size δ/2 by δ/2. Thus s and t cannot be too far apart.

Closest-Pair Preprocessing:

Construct Px and Py as sorted-list by x- and y-coordinates Closest-pair(P, Px,Py)

Divide Construct L, Lx , Ly and R, Rx , Ry

Conquer Let 1= Closest-Pair(L, Lx , Ly ) Let 2= Closest-Pair(R, Rx , Ry )

Combination Let = min(1 , 2 ) Construct S and Sy For each point in Sy, check each of its next 15 points down

the list If the distance is less than , update the as this smaller

distance

Complexity Analysis

Preprocessing takes O(n lg n) time Divide takes O(n) time Conquer takes 2 T(n/2) time Combination takes O(n) time T(n) = 2T(n/2) + cn

So totally takes O(n lg n) time.

Matrix Multiplication

Suppose we multiply two NxN matrices together. Regular method is NxNxN = N3 multiplications O(N3)

Can we Divide and Conquer?

C11 = A11*B11 + A12*B21 C12 = A11*B12 + A12*B22 C21 = A21*B11 + A22*B21 C22 = A21*B12 + A22*B22

Complexity : T(N) = 8T(N/2) + O(N2) = O(Nlog

28) = O(N3)

22211211

AAAA

A =

22211211

BBBB

B =

22211211

CCCC

C= A*B =

No improvement

Strassen’s Matrix Multiplication

P1 = (A11+ A22)(B11+B22) P2 = (A21 + A22) * B11 P3 = A11 * (B12 - B22) P4 = A22 * (B21 - B11) P5 = (A11 + A12) * B22 P6 = (A21 - A11) * (B11 + B12) P7 = (A12 - A22) * (B21 + B22)

C11 = P1 + P4 - P5 + P7

C12 = P3 + P5 C21 = P2 + P4 C22 = P1 + P3 - P2 + P6

And do this recursively as usual.

VolkerStrassen

Time analysis

T(n) = 7T(n/2) + O(n2 ) = 7logn by the Master Theorem =nlog7

=n2.81

Best bound: O(n2.376) by Coppersmith-Winograd.

Best known (trivial) lower bound: Ω(n2). Open: what is the true complexity of matrix

multiplication?

Documents

Lecture 5 Today, how to solve recurrences We learned “guess and proved by induction” We also learned “substitution” method Today, we learn the “master