02 Analysis of Algorithms: Divide and Conquer

Analysis of AlgorithmsDivide and Conquer

Andres Mendez-Vazquez

September 10, 2015

1 / 65

Outline1 Divide and Conquer: The Holy Grail!!

IntroductionSplit problems into smaller onesUsing Induction to prove algorithm correctness

2 Using Induction to prove Algorithm CorrectnessGiven an Algorithm

3 Asymptotic NotationRelation with step count

4 Examples of ComplexitiesInterpreting the NotationProperties

5 Method to Solve RecursionsSubstitution MethodThe Recursion-Tree MethodThe Master Method

2 / 65







3 / 65

Divide an Conquer

Divide et imperaA classic technique based on the multi-based recursion.

Thus, we haveThat Divide and Conquer works by recursively breaking down the probleminto subproblems and solving those subproblems recursively.

Until you reach a basis case!!!

RemarkGiven the fact of the following equivalence:

Recursion ≡ Iteration (1)

4 / 65

Divide an Conquer






4 / 65

Divide an Conquer






4 / 65

Divide an Conquer






4 / 65







5 / 65

Gauss and the Beginning

Carl Friedrich Gauss (1777–1855)He devised a way to multiply two imaginary numbers as

(a + bi) (c + di) = ac + (ad + bc) i − bd (2)

By realizing that

bc + ad = (a + b) (c + d)− ac − bd (3)

Thus minimizing the number of multiplications from four to three.

ActuallyWe can represent binary numbers like 1001 as 1000 + 01 = 22 × 10 + 01

6 / 65




By realizing that

bc + ad = (a + b) (c + d)− ac − bd (3)



6 / 65




By realizing that

bc + ad = (a + b) (c + d)− ac − bd (3)



6 / 65

Thus

We can represent numbers x , y asx = xL xR = 2n/2xL + xR

y = yL yR = 2n/2yL + yR

Thus, the multiplication can be found by usingxy =

(2n/2xL + xR

) (2n/2yL + yR

)= 2nxLyL + 2n/2(xLyR + xRyL) + xRyR (4)

Howeverif we use the Gauss’s trick, we only need xLyL, xRyR, (xL + xR)(yL + yR)to calculate the multiplication:

xLyR + xRyL = (xL + xR)(yL + yR)− xLyL − xRyR

7 / 65

Thus




(2n/2xL + xR

) (2n/2yL + yR




7 / 65

Thus




(2n/2xL + xR

) (2n/2yL + yR




7 / 65

Thus




(2n/2xL + xR

) (2n/2yL + yR




7 / 65

Thus




(2n/2xL + xR

) (2n/2yL + yR




7 / 65

Now, You have this...

We have thatxy can be calculated by using the two parts, Left and Right.

ThenThus, each xLxL, (xL + xR)(yL + yR) and xRyR can be calculated in asimilar way.

RecursionThis is know as a Recursive Procedure!!!

8 / 65





8 / 65





8 / 65

Complexities

Old Multiplication

T (n) = 4T(n

2

)+ Some Work (5)

New Multiplication

T (n) = 3T (n) + Some Work (6)

We will prove thatFor old style multiplications O

(n2).

For new style multiplications O(nlog2 3

)

9 / 65

Complexities

Old Multiplication

T (n) = 4T(n

2

)+ Some Work (5)

New Multiplication



(n2).


)

9 / 65

Complexities

Old Multiplication

T (n) = 4T(n

2

)+ Some Work (5)

New Multiplication



(n2).


)

9 / 65

Complexities

Old Multiplication

T (n) = 4T(n

2

)+ Some Work (5)

New Multiplication



(n2).


)

9 / 65

Epitaph

We can do divide and conquerIn a really unclever way!!!

Or we can go and design something betterThus, improving speedup!!!

The difference betweenA great design...Or a crappy job...

10 / 65

Epitaph




10 / 65

Epitaph




10 / 65

Epitaph




10 / 65

Recursion is the base of Divide and Conquer

This is the natural way we do many thingsWe always attack smaller versions first of the large one!!!

Stephen Cole KleeneHe defined the basics about the use of recursion.

11 / 65

Recursion is the base of Divide and Conquer

This is the natural way we do many thingsWe always attack smaller versions first of the large one!!!

Stephen Cole KleeneHe defined the basics about the use of recursion.

11 / 65

Kleene and CompanySome facts about him

Stephen Cole Kleene (January 5, 1909 – January 25, 1994) was anAmerican mathematician.One of the students of Alonzo Church!!!

I Church is best known for the lambda calculus, Church–Turing thesisand proving the undecidability of the use of an algorithm to sayYes(Valid) or No(No Valid) to a first order logic statement on a FOLSystem (Proposed by David Hilbert).

Recursion TheoryKleene, along with Alan Turing, Emil Post, and others, is best knownas a founder of the branch of mathematical logic known as recursiontheory.This theory subsequently helped to provide the foundations oftheoretical computer science.

12 / 65





12 / 65





12 / 65





12 / 65





12 / 65

Recursion

Something NotableSometimes it is difficult to define an object explicitly.It may be easy to define this object in smaller version of itself.This process is called recursion!!!

ThusWe can use recursion to define sequences, functions, and sets.

Examplean = 2n for n = 0, 1, 2, . . . =⇒ 1, 2, 4, 8, 16, 32, . . .

Thus, the sequence can be defined in a recursive way:

an+1 = 2× an (7)

13 / 65

Recursion



Examplean = 2n for n = 0, 1, 2, . . . =⇒ 1, 2, 4, 8, 16, 32, . . .


an+1 = 2× an (7)

13 / 65

Recursion



Examplean = 2n for n = 0, 1, 2, . . . =⇒ 1, 2, 4, 8, 16, 32, . . .


an+1 = 2× an (7)

13 / 65

Recursion



Examplean = 2n for n = 0, 1, 2, . . . =⇒ 1, 2, 4, 8, 16, 32, . . .


an+1 = 2× an (7)

13 / 65

Recursion



Examplean = 2n for n = 0, 1, 2, . . . =⇒ 1, 2, 4, 8, 16, 32, . . .


an+1 = 2× an (7)

13 / 65

Recursion



Examplean = 2n for n = 0, 1, 2, . . . =⇒ 1, 2, 4, 8, 16, 32, . . .


an+1 = 2× an (7)

13 / 65

Recursion



Examplean = 2n for n = 0, 1, 2, . . . =⇒ 1, 2, 4, 8, 16, 32, . . .


an+1 = 2× an (7)

13 / 65

Recursively Defined Functions

FirstAssume T is a function with the set of nonnegative integers as its domain.

SecondWe use two steps to define T :

Basis step:Specify the value of T (0).

Recursive step:Give a rule for T (x) using T (y) where 0 ≤ y < x.

ThusSuch a definition is called a recursive or inductive definition.

14 / 65







14 / 65







14 / 65







14 / 65







14 / 65







14 / 65







14 / 65

Example

Can you give me the following?Give an inductive definition of the factorial function T (n) = n!.

Base caseWhich is the base case?

Recursive caseWhat is the recursive case?

15 / 65

Example




15 / 65

Example




15 / 65

We can go further...

Recursively defined sets and structuresAssume S is a set. We use two steps to define the elements of S .

Basis StepSpecify an initial collection of elements.

Recursive StepGive a rule for forming new elements from those already known to be in S .

16 / 65





16 / 65





16 / 65

Example

ConsiderConsider S ⊆ Z defined by...

Basis Step3 ∈ S

Recursive StepIf x ∈ S and y ∈ S , then x + y ∈ S .

17 / 65

Example


Basis Step3 ∈ S


17 / 65

Example


Basis Step3 ∈ S


17 / 65

Example

Elements3 ∈ S3 + 3 = 6 ∈ S6 + 3 = 9 ∈ S6 + 6 = 12 ∈ S· · ·

18 / 65

Application: Divide and Conquer

1 Divide2 Conquer3 Combine

DivideSplit problem into a number ofsubproblems.

19 / 65



Average SequenceSolve each subproblem recursively.

19 / 65



Average SequenceThe solution of the problems into thesolution of the original problem.

19 / 65

Divide and Conquer Example: Merge Sort

Merge-Sort(A,p,r)1 if p < r then2 q ←

⌊p+r2⌋

3 Merge-Sort(A,p,q)4 Merge-Sort(A,q+1,r)5 MERGE(A,p,q,r)

ExplanationDivide part into theconquer!!!

20 / 65

Divide and Conquer Example: Merge Sort

Merge-Sort(A,p,r)1 if p < r then2 q ←

⌊p+r2⌋

3 Merge-Sort(A,p,q)4 Merge-Sort(A,q+1,r)5 MERGE(A,p,q,r)

ExplanationThe combine part

20 / 65

Merge SortMerge(A, p, q, r )

1 n1 ← q − p + 1, n2 ← r − p2 let L[1, 2, ..., n1 + 1] and

R[1, 2, ..., n2 + 1] be new arrays.3 for i ← 1 to n14 L[i]← A[p + i − 1]5 forj ← 1 to n26 R[i]← A[q + j]7 L[n1 + 1]←∞8 R[n2 + 1]←∞9 i ← 1, j ← 110 for k ← p to r11 if L[i] ≤ R[j] then12 A[k]← L[i]13 i ← i + 114 else15 A[k]← R[j]16 j ← j + 1

ExplanationCopy all to be mergedlists into two containers.

21 / 65

Merge SortMerge(A, p, q, r )

1 n1 ← q − p + 1, n2 ← r − p2 let L[1, 2, ..., n1 + 1] and

R[1, 2, ..., n2 + 1] be new arrays.3 for i ← 1 to n14 L[i]← A[p + i − 1]5 forj ← 1 to n26 R[i]← A[q + j]7 L[n1 + 1]←∞8 R[n2 + 1]←∞9 i ← 1, j ← 110 for k ← p to r11 if L[i] ≤ R[j] then12 A[k]← L[i]13 i ← i + 114 else15 A[k]← R[j]16 j ← j + 1

ExplanationMerging part.

21 / 65

The Merge Sort Recursion Cost Function

22 / 65

Thus, we have

Each Step for ONE Merging takes...A certain constant time c!!!

Thus, if we merge n elementsTotal time at level 1 of recursion:

cn (8)

In addition...We have that the recursion split each work by

12i , for i = 1, ..., log n (9)

23 / 65

Thus, we have



cn (8)


12i , for i = 1, ..., log n (9)

23 / 65

Thus, we have



cn (8)


12i , for i = 1, ..., log n (9)

23 / 65

Thus, we have the following Recursion

Base Case n = 1

T (n) = C (10)

Recursive Step n > 1

2T(n

2

)+ cn (11)

Finally

T (n) =

c if n = 12T

(n2)

+ cn if n > 1(12)

24 / 65


Base Case n = 1

T (n) = C (10)


2T(n

2

)+ cn (11)

Finally

T (n) =

c if n = 12T

(n2)

+ cn if n > 1(12)

24 / 65


Base Case n = 1

T (n) = C (10)


2T(n

2

)+ cn (11)

Finally

T (n) =

c if n = 12T

(n2)

+ cn if n > 1(12)

24 / 65







25 / 65

Recursion and Induction

Something NotableWhen a sequence is defined recursively, mathematical induction can beused to prove results about the sequence.

26 / 65

Example

We want...Show that the set S is the set A of all positive integers that are multiplesof 3.

First A ⊆ SShow that ∀k ≥ 1, P (k) =⇒ P (k + 1)

Define inductive hypothesisP (k) : 3k ∈ S

27 / 65

Example




27 / 65

Example




27 / 65

Example

We know the following:3 ∈ S

We need to proveP (k + 1) : 3 (k + 1) = 3k + 3 ∈ S

We have using the recursive definition3 + 3k ∈ S

28 / 65

Example




28 / 65

Example




28 / 65

Example

FinallyBy Mathematical Induction 3n ∈ S =⇒ A ⊆ S .

Now, show that S ⊆ AOr ∀x (x ∈ S ⇒ x ∈ A)

We know thatBasis Step: 3 ∈ S ⇒ 3 ∈ ARecursive Step: x ∈ S , y ∈ S ⇒ x + y ∈ S

29 / 65

Example




29 / 65

Example




29 / 65

Example

FirstEach element of x ∈ S is a multiple of 3 by the recursive step.

ThusIf x, y ∈ S then x + y ∈ S and x, y ∈ S

Againx + y is a multiple of 3, then x + y ∈ A

30 / 65

Example




30 / 65

Example




30 / 65

Structural induction

Something NotableInstead of mathematical induction to prove a result about a recursivelydefined sets, we can used more convenient form of induction known asstructural induction.

FirstLet n ∈ S , show P(n) is true using structural induction.

Basis StepAssume j is an element specified in the basis step of the definition.Show ∀j P(j) is true.

31 / 65





31 / 65





31 / 65


Recursive stepLet x be a new element constructed in the recursive step of thedefinition.Assume k1, k2, ..., km are elements used to construct an element x inthe recursive step of the definition.Show ∀ k1, k2, ..., km ((P (k1) ∧ P (k2) ∧ . . . ∧ P (km))→ P (x)).

32 / 65



32 / 65



32 / 65







33 / 65

Example of an AlgorithmData: Unsorted Sequence AResult: Sort Sequence AInsertion Sort(A)for j ← 2 to lenght(A) do

key ← A[j];// Insert A[j] Insert A[j] into the sorted sequence

A[1, ..., j − 1]i ← j − 1;while i > 0 and A[i] > key do

A[i + 1]← A[i];i ← i − 1;

endA[i + 1]← key

end

34 / 65

Using Induction to prove algorithm correctness

Inductive proof StructureI Initial input

F Input of nelements

I Steps1 Initialization2 Maintenance3 Termination

ExampleBe always sure about your input

35 / 65





ExampleTrue for one elementFor example, A[1] is an alreadysorted array.

35 / 65





ExampleTrue to j − 1 elements.The insertion sort mantains thesorted property during and afterthe loop.

35 / 65





ExampleTrue for n elements.At the end of the algorithmA[1, ..., n] is a sorted arraywhich is all the array.

35 / 65

In more detail: Thanks Luis Rodriguez Oracle Master 2012First, we define the following sets with sorted elements

Less = 〈x1, ..., xk |xi < key, i = 1, .., k〉Greater = 〈x1, ..., xm |xj > key, j = 1, .., m〉I = elements still not compared to the key

InitializationWe have A[1...1] with only one element ⇒ it is sorted

MaintanenceBefore we enter to the inner while loop, we have

1 A[1..j − 1] an already sorted array2 Less = ∅3 Greater = ∅4 I = A[1...j − 1].

36 / 65






36 / 65






36 / 65






36 / 65






36 / 65






36 / 65






36 / 65






36 / 65






36 / 65

Then

Case IYou never enter in the inner loop, thus A[j − 1] < key ⇒Less = A[1..j − 1], thus A[1..j] is a sorted array.

Case IIYou entered the inner while loop. Thus at each iteration we have thefollowing structure A[1...j] = I A[i] Greater whereGreater = 〈A[i], A[i + 1], · · · , A[j − 1]〉.

Note: I and Greater are sorted such that A[1...j] is sorted by itselfat this moment in the inner loop

37 / 65

Then

Case IYou never enter in the inner loop, thus A[j − 1] < key ⇒Less = A[1..j − 1], thus A[1..j] is a sorted array.

Case IIYou entered the inner while loop. Thus at each iteration we have thefollowing structure A[1...j] = I A[i] Greater whereGreater = 〈A[i], A[i + 1], · · · , A[j − 1]〉.

Note: I and Greater are sorted such that A[1...j] is sorted by itselfat this moment in the inner loop

37 / 65

Now

Thus, we get out of the inner loop once I = ∅.We have that A[1...j] = Less A[i + 1] Greater , whereA[i + 2] == A[i + 1].Thus, A[1...j] is sorted before inserting the key into the positionA[i + 1].Then, because elements of A[1...j] are sorted, we have that afterinserting the key at position i + 1 in A[1...j] that the array is stillsorted after iteration j.

38 / 65

Now


38 / 65

Now


38 / 65

Finally, Termination

TerminationOnce j > length(A), we get out of the outer loop and j = n + 1.Then, using the maintenance procedure we have that the subarrayA [1...n] is sorted as we wanted

39 / 65

Finally, Termination

TerminationOnce j > length(A), we get out of the outer loop and j = n + 1.Then, using the maintenance procedure we have that the subarrayA [1...n] is sorted as we wanted

39 / 65

Actually

This is known asLoop Invariance!!!

Why is this important?A computational system that can compute every Turing-computablefunction is called Turing complete (or Turing powerful).

PropertiesA Turing-complete system is called Turing equivalent if every function itcan compute is also Turing computable.

It computes precisely the same class of functions as do Turingmachines.

40 / 65

Actually





40 / 65

Actually





40 / 65

Actually





40 / 65

Recursion ≡ Iteration

ThenSince you can build a Turing complete language using strictly iterativestructures and a Turning complete language using only recursivestructures, then the two are therefore equivalent.

Proof From Lambda CalculusAssume languages IT (with Iterative constructs only) and REC (withRecursive constructs only).Simulate a universal Turing machine using IT, then simulate auniversal Turing machine using REC.The existence of the simulator programs guarantees that both IT andREC can calculate all the computable functions.

41 / 65




41 / 65




41 / 65




41 / 65

Nevertheless

ImportantWe use iterative procedures, when we begin to solve new problems sowe can understand them.Then, we move everything to iterative for speed!!!

42 / 65

Nevertheless

ImportantWe use iterative procedures, when we begin to solve new problems sowe can understand them.Then, we move everything to iterative for speed!!!

42 / 65

Big O, Big Ω and Θ

1 O UpperBound

2 Ω LowerBound

3 Θ TightBound

4 Theoremabout Θ

DefinitionFor a given function g(n)

O(g(n)) = f (n)| There exists c > 0 and n0 > 0s.t. 0 ≤ f (n) ≤ cg(n) ∀n ≥ n0

Examplecg(n)

f(n)

43 / 65


1 O UpperBound

2 Ω LowerBound

3 Θ TightBound

4 Theoremabout Θ


Ω(g(n)) = f (n)| There exists c > 0 and n0 > 0s.t. 0 ≤ cg(n) ≤ f (n) ∀n ≥ n0

Examplecg(n)f(n)

43 / 65


1 O UpperBound

2 Ω LowerBound

3 Θ TightBound

4 Theoremabout Θ


Θ(g(n)) = f (n)| There exists c1 > 0, c2 > 0 and n0 > 0s.t. 0 ≤ c1g(n) ≤ f (n) ≤ c2g(n) ∀n ≥ n0

Examplec1g(n)f(n)c2g(n)

43 / 65


1 O UpperBound

2 Ω LowerBound

3 Θ TightBound

4 Theoremabout Θ

TheoremFor any two functions f (n) and g(n), we have thatf (n) = Θ(g(n)) if and only if f (n) = O(g(n)) andf (n) = Ω(g(n)).

43 / 65







44 / 65

Can we relate this with practical examples?

You could sayThis is too theoretical!

However, this is not the case!!Look at this java code...

45 / 65

Can we relate this with practical examples?

You could sayThis is too theoretical!

However, this is not the case!!Look at this java code...

45 / 65

Example: Step count of Insertion Sort in JavaCounting when A.length = n

// Sor t A assume i s f u l lp u b l i c i n t [ ] I n s e r t i o n S o r t ( i n t [ ] A) Step// I n i t i a l V a r i a b l e s 0i n t B [ ] = new i n t [A . l e n g t h ] ; 1i n t s i z e = 1 ; 1i n t i , j , t ; 1// I n i t i a l i z e the Array B 0B[0]=A [ 0 ] ; 1f o r ( i = 1 ; i < A. l e n g t h ; i ++) n

t = A[ i ] ; n−1f o r ( j=s i z e −1;

j>=0&&t<B[ j ] ; j−−) i +1

// s h i f t to the r i g h t 0B[ j +1]=B[ j ] ; i

B [ j +1]=t ; n−1s i z e ++; n−1

r e t u r n B ; 1

46 / 65

The Result

Step count for body of for loop is

6 + 3 (n − 1) + n +n−1∑i=1

(i + 1) +n−1∑j=1

(i) (13)

The summationThey have the quadratic terms n2.

ComplexityInsertion sort complexity is O

(n2)

47 / 65

The Result


6 + 3 (n − 1) + n +n−1∑i=1

(i + 1) +n−1∑j=1

(i) (13)



(n2)

47 / 65

The Result


6 + 3 (n − 1) + n +n−1∑i=1

(i + 1) +n−1∑j=1

(i) (13)



(n2)

47 / 65

What does this means for insertion sort?

We have

6 + 3 (n − 1) + n +n−1∑i=1

(i + 1) +n−1∑j=1

(i) = ...

3 + 4n + n (n − 1)2 + n − 1 + n (n − 1)

2 = ...

2 + 5n + n(n − 1) = ...

n2 + 4n + 2 ≤ n2 + 4n2 + 2n2

Thusn2 + 4n + 2 ≤ 7n2 (14)

With Tinsertion(n) = n2 + 4n + 2 describing the number of steps forinsertion when we have n numbers.

48 / 65


We have

6 + 3 (n − 1) + n +n−1∑i=1

(i + 1) +n−1∑j=1

(i) = ...

3 + 4n + n (n − 1)2 + n − 1 + n (n − 1)

2 = ...

2 + 5n + n(n − 1) = ...

n2 + 4n + 2 ≤ n2 + 4n2 + 2n2

Thusn2 + 4n + 2 ≤ 7n2 (14)


48 / 65


We have

6 + 3 (n − 1) + n +n−1∑i=1

(i + 1) +n−1∑j=1

(i) = ...

3 + 4n + n (n − 1)2 + n − 1 + n (n − 1)

2 = ...

2 + 5n + n(n − 1) = ...

n2 + 4n + 2 ≤ n2 + 4n2 + 2n2

Thusn2 + 4n + 2 ≤ 7n2 (14)


48 / 65


We have

6 + 3 (n − 1) + n +n−1∑i=1

(i + 1) +n−1∑j=1

(i) = ...

3 + 4n + n (n − 1)2 + n − 1 + n (n − 1)

2 = ...

2 + 5n + n(n − 1) = ...

n2 + 4n + 2 ≤ n2 + 4n2 + 2n2

Thusn2 + 4n + 2 ≤ 7n2 (14)


48 / 65


We have

6 + 3 (n − 1) + n +n−1∑i=1

(i + 1) +n−1∑j=1

(i) = ...

3 + 4n + n (n − 1)2 + n − 1 + n (n − 1)

2 = ...

2 + 5n + n(n − 1) = ...

n2 + 4n + 2 ≤ n2 + 4n2 + 2n2

Thusn2 + 4n + 2 ≤ 7n2 (14)


48 / 65

Actually

For n0 = 2

22 + 4× 2 + 2 = 14 < 7× 22 = 28 (15)

Graphically

49 / 65

ActuallyFor n0 = 2

22 + 4× 2 + 2 = 14 < 7× 22 = 28 (15)

Graphically

49 / 65

Meaning

FirstTime or number of operations does not exceed cn2 for a constant c onany input of size n (n suitably large).

QuestionsIs O(n2) too much time?Is the algorithm practical?

For this imagine, we have a machine able to make109instructions/seconds

50 / 65

Meaning




50 / 65

Meaning




50 / 65

Then

We have the followingn n n log n n2 n3 n4

1000 1 micros 10 micros 1 milis 1 second 17 minutes10,000 10 micros 130 micros 100 milis 17 minutes 116 days

106 1 milis 20 milis 17 minutes 32 years 3× 107years

It is much worse

51 / 65

Then

We have the followingn n n log n n2 n3 n4

1000 1 micros 10 micros 1 milis 1 second 17 minutes10,000 10 micros 130 micros 100 milis 17 minutes 116 days

106 1 milis 20 milis 17 minutes 32 years 3× 107years

It is much worsen n10 2n

1000 3.2× 1013 years 3.2× 10283 years10,000 ??? ???

106 ????? ?????The Reign of the Non Polynomial Algorithms

51 / 65

ASYMPTOTIC NOTATION , THE “LITTLE BOUNDS”

1 “little o”Bound.

2 “little ω”Bound.


o(g(n)) = f (n)| For any c > 0 there exists n0 > 0s.t. 0 ≤ f (n) < cg(n) ∀n ≥ n0

ObservationsIt is not tight.Under the definition, we have for any f (n) ∈ o(g(n))

limn→∞

f (n)g(n) = 0

52 / 65

ASYMPTOTIC NOTATION , THE “LITTLE BOUNDS”

1 “little o”Bound.

2 “little ω”Bound.


ω(g(n)) = f (n)| For any c > 0 there exists n0 > 0 s.t.0 ≤ cg(n) < f (n) ∀n ≥ n0

ObservationsIt is not tight.Under the definition, we have for any f (n) ∈ ω(g(n))

limn→∞

f (n)g(n) =∞

52 / 65







53 / 65

Interpretation

1 How do you interpretn = O(n2)

2 How do you interpret2n2 + 3n + 1 =2n2 + Θ(n).

InterpretationIt means n belongs to O(n2)

54 / 65

Interpretation

1 How do you interpretn = O(n2)

2 How do you interpret2n2 + 3n + 1 =2n2 + Θ(n).

Interpretation∃f (n) ∈ Θ(n) such that

2n2 + 3n + 1 = 2n2 + f (n)= 2n2 + Θ(n)

54 / 65







55 / 65

Properties

1 Transitivity2 Reflexitibity3 Symmetry4 Transpose Symmetry.

Propertyf (n) = Θ(g(n)) and g(n) = Θ(h(n)) thenf (n) = Θ(h(n))

56 / 65

Properties


Propertyf (n) = Θ(f (n))

56 / 65

Properties


Propertyf (n) = Θ(g(n))⇐⇒ g(n) = Θ(f (n))

56 / 65

Properties


Propertyf (n) = O(g(n))⇐⇒ g(n) = Ω(f (n))

56 / 65

Ok, we have the basics...

Now...What do we do?

We will look at methods to solve recursions!!!1 Substitution Method2 Recursion-Tree Method3 Master Method

57 / 65




57 / 65




57 / 65




57 / 65







58 / 65

The Substitution Method

1 Substitution Method.2 Making a Good Guess.3 Guess4 Solve by substitution and

induction.5 Subtleties

The MethodGuess the form of the solution.Use mathematical induction to findthe constants and show that thesolution works.

59 / 65




The MethodSolve the recurrence

T(n) = 2T(⌊n

2

⌋)+ n

59 / 65




The MethodGuess that T(n) = O(n log n)

59 / 65




InductionWe assume that the bound holds for

⌊n2

⌋,

T(⌊n

2

⌋)≤ c

⌊n2

⌋log(⌊n

2

⌋),

substitute

T(n) ≤ c⌊n

2

⌋log(⌊n

2

⌋)+ n

≤ cn log(n

2

)+ n

≤ cn log n − cn + n≤ cn log n

as long c ≥ 1

59 / 65




SubtletiesConsider the recurrence

T(n) = T(⌊n

2

⌋)+ T

(⌈n2

⌉)+ 1

59 / 65







60 / 65

The Recursion-Tree Method

SurpriseSometimes is hard to do a good guess. For example T (n) = 3T

(n4)

+ cn2

Therefore, we draw the recursion tree

61 / 65

The Recursion-Tree MethodSurpriseSometimes is hard to do a good guess. For example T (n) = 3T

(n4)

+ cn2

Therefore, we draw the recursion tree

61 / 65

The Recursion-Tree MethodCounting Again!!!

A subproblem for a node at depth i is n/4i , then once n/4i = 1⇒ i = log4 n

At each level i = 0, 1, 2, ..., log4 n − 1 the cost of each node is c(

n4i

)2

At each level i = 0, 1, 2, ..., log4 n − 1 the total cost of the work is 3ic(

n4i

)2=(

316

)icn2

At depth log4 n, we have this many nodes 3log4 n = nlog4 3

Then, we have that

T(n) =log4n−1∑

i=0

( 316

)icn2 + nlog4 3

<

∞∑i=0

( 316

)icn2 + nlog4 3

=1

1− (3/16)cn2 + nlog4 3

=O(n2)

62 / 65




n4i

)2


n4i

)2=(

316

)icn2


Then, we have that

T(n) =log4n−1∑

i=0

( 316

)icn2 + nlog4 3

<

∞∑i=0

( 316

)icn2 + nlog4 3

=1

1− (3/16)cn2 + nlog4 3

=O(n2)

62 / 65




n4i

)2


n4i

)2=(

316

)icn2


Then, we have that

T(n) =log4n−1∑

i=0

( 316

)icn2 + nlog4 3

<

∞∑i=0

( 316

)icn2 + nlog4 3

=1

1− (3/16)cn2 + nlog4 3

=O(n2)

62 / 65




n4i

)2


n4i

)2=(

316

)icn2


Then, we have that

T(n) =log4n−1∑

i=0

( 316

)icn2 + nlog4 3

<

∞∑i=0

( 316

)icn2 + nlog4 3

=1

1− (3/16)cn2 + nlog4 3

=O(n2)

62 / 65




n4i

)2


n4i

)2=(

316

)icn2


Then, we have that

T(n) =log4n−1∑

i=0

( 316

)icn2 + nlog4 3

<

∞∑i=0

( 316

)icn2 + nlog4 3

=1

1− (3/16)cn2 + nlog4 3

=O(n2)

62 / 65




n4i

)2


n4i

)2=(

316

)icn2


Then, we have that

T(n) =log4n−1∑

i=0

( 316

)icn2 + nlog4 3

<

∞∑i=0

( 316

)icn2 + nlog4 3

=1

1− (3/16)cn2 + nlog4 3

=O(n2)

62 / 65




n4i

)2


n4i

)2=(

316

)icn2


Then, we have that

T(n) =log4n−1∑

i=0

( 316

)icn2 + nlog4 3

<

∞∑i=0

( 316

)icn2 + nlog4 3

=1

1− (3/16)cn2 + nlog4 3

=O(n2)

62 / 65




n4i

)2


n4i

)2=(

316

)icn2


Then, we have that

T(n) =log4n−1∑

i=0

( 316

)icn2 + nlog4 3

<

∞∑i=0

( 316

)icn2 + nlog4 3

=1

1− (3/16)cn2 + nlog4 3

=O(n2)

62 / 65







63 / 65

The Master Theorem

Theorem - Cookbook for solving T (n) = aT(

nb

)+ f (n)

Let a ≥ 1 and b > 1 be constants, let f (n) be a function, and let T (n) be defined onthe non-negative integers by the recurrence

T(n) = aT(n

b

)+ f (n) (16)

where we interpret nb as

⌊nb

⌋or⌈

nb

⌉. Then T (n) can be bounded asymptotically as

follows:1 If f (n) = O

(nlogb a−ε) for some constant ε > 0. Then T (n) = Θ

(nlogb a) .

2 If f (n) = Θ(nlogb a), then T (n) = Θ

(nlogb a lg n

).

3 If f (n) = Ω(nlogb a+ε) for some constant ε > 0 and if af

(nb

)≤ cf (n) for some

c < 1 and all sufficiently large n, then T (n) = Θ (f (n)) .

64 / 65

The Master Theorem


nb

)+ f (n)


T(n) = aT(n

b

)+ f (n) (16)


⌊nb

⌋or⌈

nb




(nlogb a) .


(nlogb a lg n

).


(nb



64 / 65

The Master Theorem


nb

)+ f (n)


T(n) = aT(n

b

)+ f (n) (16)


⌊nb

⌋or⌈

nb




(nlogb a) .


(nlogb a lg n

).


(nb



64 / 65

The Master Theorem


nb

)+ f (n)


T(n) = aT(n

b

)+ f (n) (16)


⌊nb

⌋or⌈

nb




(nlogb a) .


(nlogb a lg n

).


(nb



64 / 65

The Master Theorem


nb

)+ f (n)


T(n) = aT(n

b

)+ f (n) (16)


⌊nb

⌋or⌈

nb




(nlogb a) .


(nlogb a lg n

).


(nb



64 / 65

Using the Master Theorem

1 Consider...2 We have...3 Then...4 We use then the

case 1 of theMaster Theorem

The MethodT(n) = 9T

(n3

)+ n

65 / 65




The Methoda = 9, b = 3 and f (n) = n

65 / 65




The Methodnlog3 9 = Θ(n2) and f (n) = O(nlog3 9−ε) with ε = 1

65 / 65




FinallyT(n) = Θ(n2).

65 / 65

Engineering

02 Analysis of Algorithms: Divide and Conquer