Recurrence and Master Theorem

8/3/2019 Recurrence and Master Theorem

1/35

1

Asymptotic Efficiency of Recurrences

Find the asymptotic bounds of recursive equations. Substitution method

domain transformation

Changing variable

Recursive tree method

Master method (master theorem) Provides bounds for: T(n) = aT(n/b)+f(n) where

a 1 (the number of subproblems). b>1, (n/b is the size of each subproblem).

f(n) is a given function.


2/35

2

Recurrences

MERGE-SORT

Contains details:

T(n) = (1) ifn=1

T( n/2 )+ T( n/2 )+ (n) ifn>1 Ignore details, T(n) = 2T(n/2)+ (n).

T(n) = (1) ifn=1 2T(n/2)+ (n) ifn>1


3/35

3

The Substitution Method

Two steps:1. Guess the form of the solution.

By experience, and creativity.

By some heuristics.

If a recurrence is similar to one you have seen before.

T(n)=2T( n/2 +17)+n, similar to T(n)=2T( n/2 )+n, , guess O(nlgn).

Prove loose upper and lower bounds on the recurrence and then reduce therange of uncertainty.

ForT(n)=2T( n/2 )+n, prove lower bound T(n)= (n), and prove upperbound T(n)= O(n2), then guess the tight bound isT(n)=O(nlg n).

By recursion tree.

1. Use mathematical induction to find the constants and show thatthe solution works.


4/35

4

Solve T(n)=2T( n/2 )+n Guess the solution: T(n)=O(nlg n),

i.e., T(n) cnlg n forsome c. Prove the solutionby induction:

Suppose this bound holds for n/2 , i.e., T( n/2 ) c n/2 lg ( n/2 ).

T(n) 2(c n/2 lg ( n/2 ))+n cn lg (n/2))+n = cn lg n - cn lg 2 +n

= cn lg n - cn +n cn lg n (as long as c 1)

Question: Is the above proof complete? Why?


5/35

5

Boundary (base) Condition

In fact, T(n) =1 ifn=1, i.e., T(1)=1.

However, cnlg n =c 1 lg 1 = 0, which is odd with

T(1)=1. Take advantage of asymptotic notation: it is

required T(n) cnlg n hold forn n0 where n0 isa constant of our choosing.

Select n0 =2, thus, n=2 and n=3 as our inductionbases. It turns out any c 2 suffices for base casesofn=2 and n=3 to hold.


6/35

6

Subtleties

Guess is correct, but induction proof not work. Problem is that inductive assumption not strong enough.

Solution: revise the guess by subtracting a lower-orderterm.

Example: T(n)=T( n/2 )+T( n/2 )+1. Guess T(n)=O(n), i.e., T(n) cn for some c. However, T(n) c n/2 +c n/2 +1 =cn+1, which does

not imply T(n) cn for any c.

Attempting T(n)=O(n2) will work, but overkill.New guess T(n) cn b will work as long as b 1. (Prove it in an exact way).


7/35

7

Avoiding Pitfall

It is easy to guess T(n)=O(n) (i.e., T(n) cn) forT(n)=2T( n/2 )+n.

And wrongly prove:

T(n) 2(c n/2 )+n cn+n

=O(n). wrongly !!!!

Problem is that it does not prove the exact

form ofT(n) cn.


8/35

8

Find bound, ceiling, floor, lower term

domain transformation Find the bound: T(n)=2T(n/2)+n (O(nlogn)) How aboutT(n)=2T( n/2 )+n ? How aboutT(n)=2T( n/2 )+n ?

T(n) 2T(n/2+1)+n Domain transformation

Set S(n)=T(n+a) and assume S(n) 2S(n/2)+O(n) (so S(n)=O(nlogn)) S(n) 2S(n/2)+O(n)T(n+a) 2T(n/2+a)+O(n) T(n) 2T(n/2+1)+n T(n+a) 2T((n+a)/2+1)+n+a Thus, set n/2+a=(n+a)/2+1, get a=2. so T(n)=S(n-2)=O((n-2)log(n-2)) = O(nlogn).

How about T(n)=2T(n/2+19)+n ?

Set S(n)=T(n+a) and get a=38.

As a result, ceiling, floor, and lower terms will not affect. Moreover, the master theorem also provides proof for this.


9/35

9

Changing Variables

Suppose T(n)=2T(n)+lg n.

Rename m=lg n. so T(2m)=2T(2m/2)+m.

Domain transformation: S(m)=T(2m), so S(m)=2S(m/2)+m.

Which is similar to T(n)=2T(n/2)+n.

Sothe solution is S(m)=O(m lg m).

Changing back to T(n) from S(m), the solution is T(n)

=T(2m)=S(m)=O(m lg m)=O(lg n lg lg n).


10/35

10

The Recursion-tree Method

Idea: Each node represents the cost of a single subproblem.

Sum up the costs with each level to get level cost.

Sum up all the level costs to get total cost.

Particularly suitable for divide-and-conquerrecurrence.

Best used to generate a good guess, toleratingsloppiness.

If trying carefully to draw the recursion-tree andcompute cost, then used as direct proof.


11/35

11

Recursion Tree forT(n)=3T( n/4 )+(n2)

T(n)

(a)

cn2

T(n/4) T(n/4) T(n/4)

(b)

cn2

c(n/4)2 c(n/4)2 c(n/4)2

T(n/16) T(n/16) T(n/16)T(n/16)T(n/16)T(n/16

)

T(n/16) T(n/16) T(n/16)

(c)cn2

c(n/4)2 c(n/4)2 c(n/4)2

c(n/16)2 c(n/16)2 c(n/16)2c(n/16)2c(n/16)2c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2

cn2

(3/16)cn2

(3/16)2

cn2

T(1)T(1) T(1)T(1) T(1)T(1) (nlog 4

3)

3log4n= nlog 43 Total O(n2)

log 4n

(d)


12/35

12

Solution to T(n)=3T( n/4 )+(n2)

The height is log 4n,

#leaf nodes = 3log 4n= nlog 43. Leaf node cost: T(1).

Total cost T(n)=cn

2

+(3/16) cn

2

+(3/16)

2

cn

2

+ +(3/16)log 4n-1cn2+ (nlog 43)=(1+3/16+(3/16)2+ +(3/16)log 4n-1) cn2 + (nlog 43)


13/35

13

Prove the above Guess

T(n)=3T( n/4 )+(n2) =O(n2).

Show T(n) dn2 for some d.

T(n) 3(d( n/4 )2) +cn2

3(d(n/4)2) +cn2

=3/16(dn2

) +cn2

dn2, as long as d (16/13)c.


14/35

14

One more example

T(n)=T(n/3)+ T(2n/3)+O(n).

Construct its recursive tree (

Figure 4.2, page 71).

T(n)=O(cnlg3/2n) = O(nlg n).

Prove T(n) dnlg n.


15/35

15

Recursion Tree ofT(n)=T(n/3)+ T(2n/3)+O(n)


16/35

16

Master Method/Theorem

Theorem 4.1 (page 73) forT(n) = aT(n/b)+f(n), n/b may be n/b or

n/b . where a 1, b>1 are positive integers,f(n) be a non-

negative function.

1. If f(n)=O(nlogba-) for some >0, then T(n)= (nlogba).

2. If f(n)= (nlog

ba

), then T(n)= (nlog

ba

lg n).3. If f(n)=(nlogba+) for some >0, and ifaf(n/b) cf(n)

for some c


17/35

17

Implications of Master Theorem Comparison betweenf(n) and nlogba ()

Must be asymptotically smaller (or larger) by apolynomial, i.e., n for some >0.

In case 3, the regularity must be satisfied, i.e.,af(n/b) cf(n) for some c


18/35

18

Application of Master Theorem

T(n) = 9T(n/3)+n;

a=9,b=3,f(n) =n

nlog

b

a

= nlog

3

9

= (n2

) f(n)=O(nlog39-) for=1

By case 1, T(n) = (n2).

T(n) = T(2n/3)+1

a=1,b=3/2,f(n) =1

nlogba = nlog3/21 = (n0) = (1)

By case 2, T(n)= (lg n).


19/35

19

Application of Master Theorem

T(n) = 3T(n/4)+nlg n;

a=3,b=4,f(n) =nlg n

nlogba = nlog43 = (n0.793)

f(n)= (nlog43+) for 0.2

Moreover, for large n, the regularity holds for

c=3/4. af(n/b) =3(n/4)lg(n/4) (3/4)nlg n =cf(n)

By case 3, T(n) = (f(n))= (nlg n).


20/35

20

Exception to Master Theorem

T(n) = 2T(n/2)+nlg n;

a=2,b=2,f(n) =nlg n

nlogba = nlog22 = (n)

f(n) is asymptotically larger than nlogba , but not

polynomially larger because

f(n)/nlogba

= lg n, which is asymptotically lessthan nfor any >0.

Therefore,this is a gap between 2 and 3.


21/35

21

Where Are the Gaps

nlogba f(n), case 2: within constant distancesc1c2

n

f(n), case 1, at least polynomially smaller

Gap between case 1 and 2

n Gap between case 3 and 2

f(n), case 3, at least polynomially larger

Note: 1. for case 3, the regularity also must hold.2. iff(n) is lg n smaller, then fall in gap in 1 and 2

3. iff(n) is lg n larger, then fall in gap in 3 and 2

4. iff(n)=(nlogbalgkn), then T(n)=(nlogbalgk+1n). (as exercise)


22/35

22

Proof of Master Theorem

The proof for the exact powers, n=bk

fork 1. Lemma 4.2 forT(n) = (1) ifn=1 aT(n/b)+f(n) ifn=bkfork 1

where a 1, b>1,f(n) be a nonnegative function, Then

T(n) = (nlogba)+ ajf(n/bj)

Proof: By iterating the recurrence

By recursion tree (See figure 4.3)

j=0

logbn-1


23/35

23

Recursion tree forT(n)=aT(n/b)+f(n)


24/35

24

Proof of Master Theorem (cont.)

Lemma 4.3: Let a 1, b>1,f(n) be a nonnegative function defined on

exact power ofb, then

g(n)= ajf(n/bj) can be bounded for exact power ofb as:

1. If f(n)=O(nlogba-) for some >0, theng(n)= O(nlogba).

2. If f(n)= (nlogba), theng(n)= (nlogba lg n).

3. If f(n)= (nlogba+) for some >0 and ifaf(n/b) cf(n) for somec


25/35

25

Proof of Lemma 4.3 For case 1: f(n)=O(nlogba-) impliesf(n/bj)=O((n /bj)logba-), so

g(n)= ajf(n/bj) =O( aj(n /bj)logba-)

= O(nlogba- aj/(blogba-)j ) = O(nlogba- aj/(aj(b-)j))

= O(nlogba- (b)j ) = O(nlogba-(((b)logbn-1)/(b-1) )

= O(nlogba-(((blogbn)-1)/(b-1)))=O(nlogba n-(n-1)/(b-1))

= O(nlogba )

j=0

logbn-1

j=0

logbn-1

j=0

logbn-1

j=0

logbn-1

j=0

logbn-1


26/35

26

Proof of Lemma 4.3(cont.) For case 2: f(n)= (nlogba) impliesf(n/bj)= ((n /bj)logba), so

g(n)= a

j

f(n/b

j

) =

( aj

(n /b

j

)

logba

)

= (nlogba aj/(blogba)j ) = (nlogba 1)

= (nlogba logbn) = (nlogbalg n)

j=0

logbn-1

j=0

logbn-1j=0

logbn-1

j=0

logbn-1


27/35

27

Proof of Lemma 4.3(cont.)

For case 3:

Sinceg(n) containsf(n),g(n) = (f(n))

Since af(n/b) cf(n), ajf(n/bj) cjf(n) , why???

g(n)= ajf(n/bj) cjf(n) f(n) cj

=f(n)(1/(1-c)) =O(f(n))

Thus,g(n)=(f(n))

j=0

logbn-1

j=0

logbn-1

j=0


28/35

28

Proof of Master Theorem (cont.)

Lemma 4.4: forT(n) = (1) ifn=1

aT(n/b)+f(n) ifn=bkfork 1

where a 1, b>1,f(n) be a nonnegative function,

1. If f(n)=O(nlogba-) for some >0, then T(n)= (nlogba).

2. If f(n)= (nlogba), then T(n)= (nlogba lg n).

3. If f(n)=(nlogba+) for some >0, and ifaf(n/b) cf(n)for some c


29/35

29

Proof of Lemma 4.4 (cont.)

Combine Lemma 4.2 and 4.3, For case 1: T(n)= (nlogba)+O(nlogba)=(nlogba).

For case 2:

T(n)= (nlogba)+(nlogba lg n)=(nlogba lg n).

For case 3:

T(n)= (nlogba)+(f(n))=(f(n)) becausef(n)= (nlogba+).


30/35

30

Floors and Ceilings T(n)=aT( n/b )+f(n) and T(n)=aT( n/b )+f(n) Want to prove both equal to T(n)=aT(n/b)+f(n)

Two results: Master theorem applied to all integers n.

Floors and ceilings do not change the result. (Note: we proved this by domain transformation too).

Since n/b n/b, and n/b n/b, upper boundfor floors and lower bound for ceiling is held.

So prove upper bound for ceilings (similar for lower

bound for floors).


31/35

31

Upper bound of proof forT(n)=aT( n/b )+f(n)

consider sequence n, n/b , n/b /b , n/b /b /b ,

Let us define nj as follows:

nj = n ifj=0

= nj-1/b ifj>0

The sequence will be n0, n1, , n logb

n

Draw recursion tree:


32/35

32

Recursion tree ofT(n)=aT( n/b )+f(n)


33/35

33

The proof of upper bound for ceiling

T(n) = (nlogba)+ ajf(nj)

Thus similar to Lemma 4.3 and 4.4, the upper boundis proven.

j=0

logbn -1


34/35

34

The simple format of master theorem

T(n)=aT(n/b)+cnk, with a, b, c, kare

positive constants, and a 1 and b 2,

O(nlogba), ifa>bk.

T(n) = O(nklogn), ifa=bk.

O(nk), ifa


35/35

35

Summary

Recurrences and their bounds Substitution

Recursion tree

Master theorem.

Proof of subtleties

Recurrences that Master theorem does not

apply to.

Documents

Recurrence and Master Theorem