Design and Analysis of Algorithms
Dr. Aarij Mahmood Hussaan
Course Objective
The last ½ century has witnessed the development of a beautiful and elegant new scientific field: the design and analysis of algorithms.
This course will teach you how to “design” and “analyze” algorithms.
History
• The name of “algorithm” comes from al-Khowarizmi, 9th century Persian mathematician and astronomer who wrote a book, eventually translated into Latin in the 12th century as Algoritmi de numero Indorum ("al-Khowarizmi's book on Indian numbers"), that gave our modern pencil-and-paper algorithms for addition, subtraction, multiplication, and division. • Every algorithm should have:
1. Input, 2. Output, 3. Deterministic 4. Effective, 5. Finite.
Why study algorithms
• Many aspects of “programming” are more important than algorithms: • user-friendliness, correctness, extensibility, maintainability, simplicity,
scalability, cost, …
• But good algorithm is fundamental to all of above.
Algorithm
• A procedure for solving a (mathematical) problem in a finite number of steps that frequently involves repetition of an operation; broadly: a step-by-step method for accomplishing some task.
Algorithm: formal definition──Intro CS
• A well-ordered collection of unambiguous and effectively computable operations that, when, executed, produces a result and halts in a finite amount of time.
A person does not really understand something until after teaching it to a computer
Donald Knuth
The Role of the Algorithms
in ComputING
Algorithms
• Algorithm: Any well-defined computation procedure that takes some value, or set of values, as input and produces some value, or set of values, as output.• Or: tool for solving well specific computational problem.
• Example: Sorting problem
Problem
• Example: Sorting problem • Input: A sequence of n numbers • Output: A permutation of the input sequence such that .
• Sorting algorithm: • The procedure that produces the sorted outputs for all possible inputs.
a a an1 2, ,...,
a a an1 2' ' ', ,...,
a a an1 2' ' '...
Properties
• An instance of a problem consists of all inputs needed to compute a solution to the problem.
• An algorithm is said to be correct if, for every input instance, it halts with the correct output.
• A correct algorithm solves the given computational problem. An incorrect algorithm might not halt at all on some input instance, or it might halt with other than the desired answer.
What kind of problem can be solved by algorithm?• The Human Genome Project• The Internet Applications• Electronic Commerce with Public-key cryptography and digital
signatures• Manufacturing and other commercial settings
Interesting problems
• How to solve these problems• Shortest path (Ch. 24)• Longest common subsequence (Ch. 15)• Convex hull (CH. 33)
• Characteristics of interesting problem• Find the best solutions from the candidate ones.• Practical application.
Hard Problems
• Efficiency is speed, i.e., how long an algorithm takes to produce its result.• NP-Complete• no efficient algorithm for an NP-complete problem has ever been found,• NP-complete problems has the remarkable property that if an efficient
algorithm exists for any one of them
Algorithm choice
• There are a large number of good sorting algorithms.• Which algorithm is best for a given application• • It depends on:• The number of items to be sorted.• The extent to which the items are already sorted.• Possible restrictions on item values.• The kind of storage device to be used.
• An algorithm is said to be correct if, for every input instance, it gives the correct output.
Algorithm efficiency
• Algorithms devised to solve the same problem often differ in their efficiency.• These differences can be much more significant than differences due
to hardware and software.
Example:• Insertion sort algorithm takes time = c1n2 to sort n numbers, where
c1 is a constant.• Merge sort algorithm takes time = c2 n lg n to sort n numbers, where
c2 is a constant.
Algorithm efficiency
• c1 < c2, but constants less significant in the running time than the input size.• lg n < n, but insertion sort is usually faster than merge sort for small
input sizes.• Once the input size n becomes large enough, merge sort will become
faster than insertion sort.
Algorithm efficiency
• • Example:• Computer A executes one billion instruction per second (1 GHz) and
executes insertion sort (c1n2).• Computer B executes one million instruction per second (1 MHz) and
executes merge sort (c2 n lg n).• Assume c1=2 and c2=50.• Assume we need to sort 1 million numbers.
Algorithm efficiency
• Computer A takes= 2*(106)2 instructions = 2000 s 109 instructions/second
• Computer B takes= 50*106* lg 106 instructions = 100 s 107 instructions/second
Algorithm efficiency
• Imagine that we need to sort 10 million numbers;• Insertion sort takes 2.3 days.• Merge sort takes 20 minutes.
• We analyze algorithms to improve them and choose the best one for a given problem.
Algorithm Questions
• How to devise algorithms?• there are various design techniques.• example: divide and conquer.
• How to analyze algorithms?• the task of determining how much computing time and storage and
algorithm requires.• it needs mathematical skills.• best case, average case, and worst case.
Algorithm Questions
• How to validate algorithms?• validation: showing that the algorithm computes the correct answer
for all legal inputs.• after validation, verification process takes place.
Insertion sort
• solves the sorting problem• Input: A sequence of n numbers • Output: A permutation of the input sequence such that .
a a an1 2, ,...,
a a an1 2' ' ', ,...,
a a an1 2' ' '...
Insertion sort
Insertion sort – pseudo code
• InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
David Luebke 26
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 10 40 20
1 2 3 4
i = j = key = A[j] = A[j+1] =
David Luebke 27
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 10 40 20
1 2 3 4
i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 10
David Luebke 28
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 30 40 20
1 2 3 4
i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30
David Luebke 29
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 30 40 20
1 2 3 4
i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30
David Luebke 30
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 30 40 20
1 2 3 4
i = 2 j = 0 key = 10A[j] = A[j+1] = 30
David Luebke 31
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 30 40 20
1 2 3 4
i = 2 j = 0 key = 10A[j] = A[j+1] = 30
David Luebke 32
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 2 j = 0 key = 10A[j] = A[j+1] = 10
David Luebke 33
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 0 key = 10A[j] = A[j+1] = 10
David Luebke 34
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 0 key = 40A[j] = A[j+1] = 10
David Luebke 35
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 0 key = 40A[j] = A[j+1] = 10
David Luebke 36
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40
David Luebke 37
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40
David Luebke 38
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40
David Luebke 39
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 4 j = 2 key = 40A[j] = 30 A[j+1] = 40
David Luebke 40
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40
David Luebke 41
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40
David Luebke 42
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20
David Luebke 43
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20
David Luebke 44
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 40
1 2 3 4
i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40
David Luebke 45
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 40
1 2 3 4
i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40
David Luebke 46
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 40
1 2 3 4
i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40
David Luebke 47
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 40
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40
David Luebke 48
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 40
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40
David Luebke 49
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 30 40
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30
David Luebke 50
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 30 40
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30
David Luebke 51
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 30 40
1 2 3 4
i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30
David Luebke 52
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 30 40
1 2 3 4
i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30
David Luebke 53
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 20 30 40
1 2 3 4
i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20
David Luebke 54
04/20/2023
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 20 30 40
1 2 3 4
i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20
Done!
Loop invariants and Correctness
• Initialization: It is true prior to the first iteration of the loop.• Maintenance: If it is true before an iteration of the loop, it remains
true before the next iteration.• Termination: When the loop terminates, the invariant gives us a
useful property that helps show that the algorithm is correct.
David Luebke 56
04/20/2023
Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
What is the preconditionfor this loop?
David Luebke 57
04/20/2023
Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}How many times will this loop execute?
Loop Invariants
• How to say that this algorithm is correct?• an invariant of a loop is a property that holds before (and after) each
repetition
• We must show three things about a loop invariant:• Initialization: It is true prior to the first iteration of the loop.• Maintenance: If it is true before an iteration of the loop, it remains true
before the next iteration.• Termination: When the loop terminates, the invariant gives us a useful
property that helps show that the algorithm is correct.
Correctness of Insertion Sort
Invariant• At the start of each iteration of the for loop of lines 1–8, the sub-array A[1 .. j-1] consists of
the elements originally in A[1 .. j-1] , but in sorted order.
• Initialization• Maintenance: • the loop moves the A[1 .. j-2] .. Elements to the right until it finds a suitable
place for A[j]. The sub-array A[1 .. j] then consists of the elements originally in A[1 .. j] , but in sorted order
• Termination• In the end we have j = n +1, substitute this in the “invariant” wording
David Luebke 60
04/20/2023
Analysis Insertion Sort
David Luebke 61
04/20/2023
Analyzing Insertion Sort• T(n) = c1n + c2(n-1) + c4(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1)) + c7(n-1)
= c8T + c9n + c10
• What can T be?• Best case -- inner loop body never executed
• ti = 1 T(n) is a linear function• Worst case -- inner loop body executed for all previous
elements• ti = i T(n) is a quadratic function
• Average case• ???
Analysis
• Simplifications• Ignore actual and abstract statement costs• Order of growth is the interesting measure:
• Highest-order term is what counts• Remember, we are doing asymptotic analysis• As the input size grows larger it is the high order term that dominates
Designing algorithms
• Insertion Sort – Incremental Approach• The divide-and-conquer approach• Recursive in nature
• Three parts• Divide the problem into a number of sub-problems that are smaller instances
of the same problem.• Conquer the sub-problems by solving them recursively. If the sub-problem
sizes are small enough, however, just solve the sub-problems in a straightforward manner.• Combine the solutions to the sub-problems into the solution for the original
problem.
Merge Sort
• Divide: Divide the n-element sequence to be sorted into two subsequences of n=2 elements each.• Conquer: Sort the two subsequences recursively using merge sort.• Combine: Merge the two sorted subsequences to produce the sorted
answer.
Merge Sort - Example
Merge sort – Merge procedure
Invariants of Merge Procedure
At the start of each iteration of the for loop of lines 12–17, the sub-array A[p..k-1] contains the k -p smallest elements of L[1 .. N1+1] and R[1 .. N2+1], in sorted order. Moreover, L[i] and R[j] are the smallest elements of their arrays that have not been copied back into A.
• Initialization: Prior to the first iteration of the loop, we have k = p, so that the sub-array A[p..k-1] is empty.• Maintenance • Termination
Merge Sort procedure
Analyzing divide-and-conquer algorithms• recurrence equation or recurrence, which describes the overall
running time on a problem of size n in terms of the running time on smaller inputs.• Suppose that our division of the problem yields ‘a’ sub-problems,
each of which is ‘1/b’ the size of the original.• It takes time T (n/b) to solve one sub-problem of size n=b, and so it
takes time ‘a T (n/b)’ to solve ‘a’ of them.• D(n) : time to divide, C(n) : time to combine
Analysis of merge sort
• Divide: The divide step just computes the middle of the sub-array, which takes constant time. Thus, D(n) = Θ(1).• Conquer: We recursively solve two sub-problems, each of size n=2,
which contributes 2T(n/2) to the running time.• Combine: The combine step also takes constant time. Thus, C(n) =
Θ(1).
• The solution to this recurrence equation is T(n) = Θ(n lg n)
Why Θ(n lg n) ?
• Rewriting the initial recurrence
Why Θ(n lg n) ?
Cost calculation
• Top level cost = cn• Next level => cn = cn/2 +cn/2; next level => cn = cn/4+cn/4+cn/4+cn/4• At any level below the top we have 2i node each with a cost c(n/2i)• At any ith level below the tops has a total cost 2i c(n/2i) = cn• The bottom level has n nodes, each contributing a cost of c, for a total cost of cn.• Total number of levels is lg n + 1 (verify for 1, 2i, and 2i+1 )• For the total cost we simply add up the costs of all the levels.
Recursion tree has lg n + 1 levels, each costing cnÞ cn(lg n + 1) => cn lg n + cn Þ ignoring lower order terms and constants we have Θ(n lg n)