RAIK 283: Data Structures & Algorithms

Design and Analysis of Algorithms – Chapter 5 1

Divide and Conquer (I)

Dr. Ying [email protected]

RAIK 283: Data Structures & RAIK 283: Data Structures & AlgorithmsAlgorithms


Giving credit where credit is due:Giving credit where credit is due:• Most of the lecture notes are based on the slides from Most of the lecture notes are based on the slides from

the Textbook’s companion websitethe Textbook’s companion website– http://www.aw-bc.com/info/levitinhttp://www.aw-bc.com/info/levitin

• Some examples and slides are based on lecture notes Some examples and slides are based on lecture notes created by Dr. Ben Choi, Louisiana Technical University created by Dr. Ben Choi, Louisiana Technical University and Dr. Chuck Cusack, Hope College and Dr. Chuck Cusack, Hope College

• I have modified many of their slides and added new I have modified many of their slides and added new slides.slides.

RAIK 283: Data Structures & RAIK 283: Data Structures & AlgorithmsAlgorithms


Divide and ConquerDivide and Conquer

The most well known algorithm design strategy:The most well known algorithm design strategy:1.1. Divide instance of problem into two or more smaller Divide instance of problem into two or more smaller

instancesinstances

2.2. Solve smaller instances recursivelySolve smaller instances recursively

3.3. Obtain solution to original (larger) instance by combining Obtain solution to original (larger) instance by combining these solutionsthese solutions


Divide-and-conquer TechniqueDivide-and-conquer Technique

subproblem 2 of size n/2

subproblem 1 of size n/2

a solution to subproblem 1

a solution tothe original problem

a solution to subproblem 2

a problem of size n


Divide and Conquer ExamplesDivide and Conquer Examples

Sorting: mergesort and quicksortSorting: mergesort and quicksort

Tree traversalsTree traversals

Matrix multiplication-Strassen’s algorithmMatrix multiplication-Strassen’s algorithm

Closest pair problemClosest pair problem

Convex hull-QuickHull algorithmConvex hull-QuickHull algorithm

Review MergesortReview Mergesort



MergesortMergesort

Algorithm:Algorithm: Split array A[1..Split array A[1..nn] in two and make copies of each half in ] in two and make copies of each half in

arrays B[1.. arrays B[1.. nn/2 ] and C[1.. /2 ] and C[1.. nn/2 ]/2 ] Sort arrays B and CSort arrays B and C Merge sorted arrays B and C into array AMerge sorted arrays B and C into array A


Using Divide and Conquer: Using Divide and Conquer: MergesortMergesort

Mergesort StrategyMergesort Strategy

Sorted

Merge

Sorted Sorted

Sort recursively by Mergesort

Sort recursively by Mergesort

first last

(first last)2


MergesortMergesort


arrays B[1.. arrays B[1.. nn/2 ] and C[1.. /2 ] and C[1.. nn/2 ]/2 ] Sort arrays B and CSort arrays B and C Merge sorted arrays B and C into array AMerge sorted arrays B and C into array A


MergesortMergesort


arrays B[1.. arrays B[1.. nn/2 ] and C[1.. /2 ] and C[1.. nn/2 ]/2 ]

Sort arrays B and CSort arrays B and C Merge sorted arrays B and C into array A as follows:Merge sorted arrays B and C into array A as follows:

• Repeat the following until no elements remain in one of the arrays:Repeat the following until no elements remain in one of the arrays:

– compare the first elements in the remaining unprocessed compare the first elements in the remaining unprocessed portions of the arraysportions of the arrays

– copy the smaller of the two into A, while incrementing the index copy the smaller of the two into A, while incrementing the index indicating the unprocessed portion of that array indicating the unprocessed portion of that array

• Once all elements in one of the arrays are processed, copy the Once all elements in one of the arrays are processed, copy the remaining unprocessed elements from the other array into A.remaining unprocessed elements from the other array into A.


Algorithm: MergesortAlgorithm: Mergesort

Input:Input: Array E and indices first and last, such that the Array E and indices first and last, such that the elements E[i] are defined for first elements E[i] are defined for first i i last. last.

Output:Output: E[first], …, E[last] is a sorted rearrangement of E[first], …, E[last] is a sorted rearrangement of the same elementsthe same elements

void mergeSort(Element[] E, int first, int last)void mergeSort(Element[] E, int first, int last)if (first < last)if (first < last)

int mid = int mid = (first+last)/2(first+last)/2;;mergeSort(E, first, mid);mergeSort(E, first, mid);mergeSort(E, mid+1, last);mergeSort(E, mid+1, last);merge(E, first, mid, last);merge(E, first, mid, last);

return;return;


In-Class ExerciseIn-Class Exercise

P175 Problem 5.1.6P175 Problem 5.1.6

Apply Mergesort to sort the list E, X, A, M, P, L, E in Apply Mergesort to sort the list E, X, A, M, P, L, E in alphabetical order. alphabetical order.


Algorithm: MergesortAlgorithm: Mergesort

void mergeSort(Element[] E, int first, int last)void mergeSort(Element[] E, int first, int last)if (first < last)if (first < last)

int mid = int mid = (first+last)/2(first+last)/2;;mergeSort(E, first, mid);mergeSort(E, first, mid);

mergeSort(E, mid+1, last);mergeSort(E, mid+1, last);

merge(E, first, mid, last);merge(E, first, mid, last);

return;return;

Time EfficiencyTime Efficiency


Mergesort ComplexityMergesort Complexity

Mergesort always partitions the array equally.Mergesort always partitions the array equally. Thus, the recursive depth is always Thus, the recursive depth is always (lg (lg nn) ) The amount of work done at each level is The amount of work done at each level is ((nn) ) Intuitively, the complexity should be Intuitively, the complexity should be ((nn lg lg nn) )

We have, We have, • TT((nn) =2) =2TT((nn/2) /2) ((n) for n>1, T(1)=0 n) for n>1, T(1)=0 ((nn lg lg nn))

The amount of extra memory used is The amount of extra memory used is ((nn))


Efficiency of MergesortEfficiency of Mergesort

Number of comparisons is close to theoretical minimum for Number of comparisons is close to theoretical minimum for comparison-based sorting: comparison-based sorting: • lg lg n n ! ≈ ! ≈ nn lg lg n n - 1.44- 1.44 n n

Space requirement: Space requirement: ΘΘ( ( n n ) () (NOTNOT in-place) in-place)

Can be implemented without recursion (bottom-up)Can be implemented without recursion (bottom-up)

All cases have same efficiency: All cases have same efficiency: ΘΘ( ( n n log log nn) )


AnimationAnimation

http://math.hws.edu/TMCM/java/xSortLab/index.html


The master theoremThe master theorem

Given: a Given: a divide and conquerdivide and conquer algorithm algorithm

Then, the Master Theorem gives us a cookbook for the Then, the Master Theorem gives us a cookbook for the algorithm’s running time:algorithm’s running time:


A general divide-and-conquer A general divide-and-conquer recurrencerecurrence

TT((nn) = ) = aTaT((n/bn/b) + ) + f f ((nn)) where where f f ((nn)) ∈∈ ΘΘ((nndd))

T(1) = cT(1) = c

a < ba < bdd T T((nn) ) ∈∈ ΘΘ((nndd)) a = ba = bdd T T((nn) ) ∈∈ ΘΘ((nnd d lg lg n n )) a > ba > bdd T T((nn) ) ∈∈ ΘΘ((nnlog log b b aa))

Please refer to Appendix B for the proofPlease refer to Appendix B for the proof


In-class exerciseIn-class exercise

Page 175: Problem 5Page 175: Problem 5 5. Find the order of growth for solutions of the following 5. Find the order of growth for solutions of the following

recurrences.recurrences.• a. T(n) = 4T(n/2) + n, T(1) = 1a. T(n) = 4T(n/2) + n, T(1) = 1

• b. T(n) = 4T(n/2) + nb. T(n) = 4T(n/2) + n2222, T(1) = 1, T(1) = 1

• c. T(n) = 4T(n/2) + nc. T(n) = 4T(n/2) + n3333, T(1) = 1, T(1) = 1

In-Class ExerciseIn-Class Exercise

5.1.9 Let A[0, n-1] be an array of n distinct real 5.1.9 Let A[0, n-1] be an array of n distinct real numbers. A pair (A[i], A[j]) is said to be an numbers. A pair (A[i], A[j]) is said to be an inversion if these numbers are out of order, i.e., i<j inversion if these numbers are out of order, i.e., i<j but A[i]>A[j]. Design an O(nlogn) algorithm for but A[i]>A[j]. Design an O(nlogn) algorithm for counting the number of inversions. counting the number of inversions.


Review QuicksortReview Quicksort



Quicksort by Hoare (1962)Quicksort by Hoare (1962)

Select a Select a pivotpivot (partitioning element) (partitioning element) Rearrange the list so that all the elements in the positions Rearrange the list so that all the elements in the positions

before the pivot are smaller than or equal to the pivot and before the pivot are smaller than or equal to the pivot and those after the pivot are larger than or equal to the pivot those after the pivot are larger than or equal to the pivot

Exchange the pivot with the last element in the first (i.e., ≤) Exchange the pivot with the last element in the first (i.e., ≤) sublist – the pivot is now in its final positionsublist – the pivot is now in its final position

Sort the two sublists recursivelySort the two sublists recursively

p

A[i]≤p A[i]p


Quicksort by Hoare (1962)Quicksort by Hoare (1962)

Select a Select a pivotpivot (partitioning element) (partitioning element) Rearrange the list so that all the elements in the positions Rearrange the list so that all the elements in the positions

before the pivot are smaller than or equal to the pivot and before the pivot are smaller than or equal to the pivot and those after the pivot are larger than or equal to the pivot those after the pivot are larger than or equal to the pivot

Exchange the pivot with the last element in the first (i.e., ≤) Exchange the pivot with the last element in the first (i.e., ≤) sublist – the pivot is now in its final positionsublist – the pivot is now in its final position

Sort the two sublists recursivelySort the two sublists recursively

Apply quicksort to sort the list 7 2 9 10 5 4Apply quicksort to sort the list 7 2 9 10 5 4

p

A[i]≤p A[i]p


Quicksort ExampleQuicksort Example

Recursive implementation with the left most array Recursive implementation with the left most array entry selected as the pivot element.entry selected as the pivot element.


QuicksortQuicksort

Animated Example: Animated Example: http://math.hws.edu/TMCM/java/xSortLab/index.html


Quicksort AlgorithmQuicksort Algorithm

Input: Input: Array E and indices first, and last, s.t. elements E[i] are defined for Array E and indices first, and last, s.t. elements E[i] are defined for

first first i i last last Ouput: Ouput:

E[first], …, E[last] is a sorted rearrangement of the arrayE[first], …, E[last] is a sorted rearrangement of the array

Void quickSort(Element[] E, int first, int last)Void quickSort(Element[] E, int first, int last)if (first < last)if (first < last)

Element pivotElement = E[first];Element pivotElement = E[first];int splitPoint = partition(E, pivotElement, first, last);int splitPoint = partition(E, pivotElement, first, last);quickSort (E, first, splitPoint –1 );quickSort (E, first, splitPoint –1 );quickSort (E, splitPoint +1, last );quickSort (E, splitPoint +1, last );

return;return;


Quicksort AnalysisQuicksort Analysis

Partition can be done in O(Partition can be done in O(nn) time, where ) time, where nn is the is the size of the arraysize of the array

Let Let TT((nn) be the number of comparisons required ) be the number of comparisons required by Quicksortby Quicksort

If the pivot ends up at position If the pivot ends up at position kk, then we have, then we have• TT((nn) ) TT((nnkk) ) TT((kk 1) 1) nn

To determine best-, worst-, and average-case To determine best-, worst-, and average-case complexity we need to determine the values of complexity we need to determine the values of kk that correspond to these cases.that correspond to these cases.


Quicksort AnalysisQuicksort Analysis

Partition can be done in O(Partition can be done in O(nn) time, where ) time, where nn is the is the size of the arraysize of the array

Let Let TT((nn) be the number of comparisons required ) be the number of comparisons required by Quicksortby Quicksort

If the pivot ends up at position If the pivot ends up at position kk, then we have, then we have• TT((nn) ) TT((nnkk) ) TT((kk 1) 1) nn

To determine best-, worst-, and average-case To determine best-, worst-, and average-case complexity we need to determine the values of complexity we need to determine the values of kk that correspond to these cases.that correspond to these cases.


Best-Case ComplexityBest-Case Complexity

The best case is clearly when the pivot always The best case is clearly when the pivot always partitions the array equally.partitions the array equally.

Intuitively, this would lead to a recursive depth of Intuitively, this would lead to a recursive depth of at most lg at most lg nn calls calls

We can actually prove this. In this case We can actually prove this. In this case • TT((nn) ) TT((nn/2) /2) TT((nn/2) /2) n n ((nn lg lg nn))


Worst-Case and Average-Case Worst-Case and Average-Case ComplexityComplexity

The worst-case is when the pivot always ends up in The worst-case is when the pivot always ends up in the first or last element. That is, partitions the the first or last element. That is, partitions the array as unequally as possible.array as unequally as possible.

In this case In this case • TT((nn) ) ??


Worst-Case and Average-Case Worst-Case and Average-Case ComplexityComplexity

The worst-case is when the pivot always ends up in The worst-case is when the pivot always ends up in the first or last element. That is, partitions the the first or last element. That is, partitions the array as unequally as possible.array as unequally as possible.

In this case In this case • TT((nn) ) TT((nn11) ) TT(1(11) 1) n n TT((nn11) ) nn

nn ( (nn11) ) … + 1 … + 1

nn((nn 1)/2 1)/2 ((nn22))

Average case is rather complex, but is where the Average case is rather complex, but is where the algorithm earns its name. The bottom line is:algorithm earns its name. The bottom line is:

)lg(lg386.1)( nnnnnA


Summary of QuicksortSummary of Quicksort

Best caseBest case: split in the middle — : split in the middle — ΘΘ( ( n n log log nn) ) Worst caseWorst case: sorted array! — : sorted array! — ΘΘ( ( nn22) ) Average caseAverage case: random arrays — : random arrays — ΘΘ( ( n n log log nn))




Memory requirement?Memory requirement?




Memory requirement?Memory requirement? In-place sorting algorithmIn-place sorting algorithm Considered as the method of choice for internal sorting for Considered as the method of choice for internal sorting for

large files (large files (nn ≥ 10000) ≥ 10000)




Considered as the method of choice for internal sorting for Considered as the method of choice for internal sorting for large files (large files (nn ≥ 10000) ≥ 10000)

Improvements:Improvements:• better pivot selection: median of three partitioning avoids worst better pivot selection: median of three partitioning avoids worst

case in sorted filescase in sorted files• switch to insertion sort on small subfilesswitch to insertion sort on small subfiles• elimination of recursionelimination of recursionthese combine to 20-25% improvementthese combine to 20-25% improvement

In-class exercisesIn-class exercises


Documents

RAIK 283: Data Structures & Algorithms