Design and Analysis of Algorithms

Design and Analysis of Algorithms

Dr. Aarij Mahmood Hussaan

Course Objective

The last ½ century has witnessed the development of a beautiful and elegant new scientific field: the design and analysis of algorithms.

This course will teach you how to “design” and “analyze” algorithms.

History

• The name of “algorithm” comes from al-Khowarizmi, 9th century Persian mathematician and astronomer who wrote a book, eventually translated into Latin in the 12th century as Algoritmi de numero Indorum ("al-Khowarizmi's book on Indian numbers"), that gave our modern pencil-and-paper algorithms for addition, subtraction, multiplication, and division. • Every algorithm should have:

1. Input, 2. Output, 3. Deterministic 4. Effective, 5. Finite.

http://en.wikipedia.org/wiki/Muhammad_ibn_ME+sD%01_al-KhwD%01rizmD+

http://en.wikipedia.org/wiki/Muhammad_ibn_ME+sD%01_al-KhwD%01rizmD+

Why study algorithms

• Many aspects of “programming” are more important than algorithms: • user-friendliness, correctness, extensibility, maintainability, simplicity,

scalability, cost, …

• But good algorithm is fundamental to all of above.

Algorithm

• A procedure for solving a (mathematical) problem in a finite number of steps that frequently involves repetition of an operation; broadly: a step-by-step method for accomplishing some task.

Algorithm: formal definition──Intro CS

• A well-ordered collection of unambiguous and effectively computable operations that, when, executed, produces a result and halts in a finite amount of time.

A person does not really understand something until after teaching it to a computer

Donald Knuth

The Role of the Algorithms

in ComputING

Algorithms

• Algorithm: Any well-defined computation procedure that takes some value, or set of values, as input and produces some value, or set of values, as output.• Or: tool for solving well specific computational problem.

• Example: Sorting problem

Problem

• Example: Sorting problem • Input: A sequence of n numbers • Output: A permutation of the input sequence such that .

• Sorting algorithm: • The procedure that produces the sorted outputs for all possible inputs.

a a an1 2, ,...,

a a an1 2' ' ', ,...,

a a an1 2' ' '...

Properties

• An instance of a problem consists of all inputs needed to compute a solution to the problem.

• An algorithm is said to be correct if, for every input instance, it halts with the correct output.

• A correct algorithm solves the given computational problem. An incorrect algorithm might not halt at all on some input instance, or it might halt with other than the desired answer.

What kind of problem can be solved by algorithm?• The Human Genome Project• The Internet Applications• Electronic Commerce with Public-key cryptography and digital

signatures• Manufacturing and other commercial settings

Interesting problems

• How to solve these problems• Shortest path (Ch. 24)• Longest common subsequence (Ch. 15)• Convex hull (CH. 33)

• Characteristics of interesting problem• Find the best solutions from the candidate ones.• Practical application.

Hard Problems

• Efficiency is speed, i.e., how long an algorithm takes to produce its result.• NP-Complete• no efficient algorithm for an NP-complete problem has ever been found,• NP-complete problems has the remarkable property that if an efficient

algorithm exists for any one of them

Algorithm choice

• There are a large number of good sorting algorithms.• Which algorithm is best for a given application• • It depends on:• The number of items to be sorted.• The extent to which the items are already sorted.• Possible restrictions on item values.• The kind of storage device to be used.

• An algorithm is said to be correct if, for every input instance, it gives the correct output.

Algorithm efficiency

• Algorithms devised to solve the same problem often differ in their efficiency.• These differences can be much more significant than differences due

to hardware and software.

Example:• Insertion sort algorithm takes time = c1n2 to sort n numbers, where

c1 is a constant.• Merge sort algorithm takes time = c2 n lg n to sort n numbers, where

c2 is a constant.


• c1 < c2, but constants less significant in the running time than the input size.• lg n < n, but insertion sort is usually faster than merge sort for small

input sizes.• Once the input size n becomes large enough, merge sort will become

faster than insertion sort.


• • Example:• Computer A executes one billion instruction per second (1 GHz) and

executes insertion sort (c1n2).• Computer B executes one million instruction per second (1 MHz) and

executes merge sort (c2 n lg n).• Assume c1=2 and c2=50.• Assume we need to sort 1 million numbers.


• Computer A takes= 2*(106)2 instructions = 2000 s 109 instructions/second

• Computer B takes= 50*106* lg 106 instructions = 100 s 107 instructions/second


• Imagine that we need to sort 10 million numbers;• Insertion sort takes 2.3 days.• Merge sort takes 20 minutes.

• We analyze algorithms to improve them and choose the best one for a given problem.

Algorithm Questions

• How to devise algorithms?• there are various design techniques.• example: divide and conquer.

• How to analyze algorithms?• the task of determining how much computing time and storage and

algorithm requires.• it needs mathematical skills.• best case, average case, and worst case.

Algorithm Questions

• How to validate algorithms?• validation: showing that the algorithm computes the correct answer

for all legal inputs.• after validation, verification process takes place.

Insertion sort

• solves the sorting problem• Input: A sequence of n numbers • Output: A permutation of the input sequence such that .

a a an1 2, ,...,

a a an1 2' ' ', ,...,

a a an1 2' ' '...

Insertion sort

Insertion sort – pseudo code

• InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

David Luebke 26

04/20/2023

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {


A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

30 10 40 20

1 2 3 4

i = j = key = A[j] = A[j+1] =

David Luebke 27

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

30 10 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 10

David Luebke 28

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

30 30 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30

David Luebke 29

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

30 30 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30

David Luebke 30

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

30 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = A[j+1] = 30

David Luebke 31

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

30 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = A[j+1] = 30

David Luebke 32

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = A[j+1] = 10

David Luebke 33

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 10A[j] = A[j+1] = 10

David Luebke 34

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 40A[j] = A[j+1] = 10

David Luebke 35

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 40A[j] = A[j+1] = 10

David Luebke 36

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40

David Luebke 37

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40

David Luebke 38

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40

David Luebke 39

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 40A[j] = 30 A[j+1] = 40

David Luebke 40

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

David Luebke 41

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

David Luebke 42

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20

David Luebke 43

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20

David Luebke 44

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40

David Luebke 45

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40

David Luebke 46

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40

David Luebke 47

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

David Luebke 48

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 40 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

David Luebke 49

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 30 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30

David Luebke 50

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 30 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30

David Luebke 51

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30

David Luebke 52

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 30 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30

David Luebke 53

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 20 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20

David Luebke 54

04/20/2023




A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

10 20 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20

Done!

Loop invariants and Correctness

• Initialization: It is true prior to the first iteration of the loop.• Maintenance: If it is true before an iteration of the loop, it remains

true before the next iteration.• Termination: When the loop terminates, the invariant gives us a

useful property that helps show that the algorithm is correct.

David Luebke 56

04/20/2023

Insertion Sort



A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}

What is the preconditionfor this loop?

David Luebke 57

04/20/2023

Insertion Sort



A[j+1] = A[j]j = j - 1

}A[j+1] = key

}

}How many times will this loop execute?

Loop Invariants

• How to say that this algorithm is correct?• an invariant of a loop is a property that holds before (and after) each

repetition

• We must show three things about a loop invariant:• Initialization: It is true prior to the first iteration of the loop.• Maintenance: If it is true before an iteration of the loop, it remains true

before the next iteration.• Termination: When the loop terminates, the invariant gives us a useful

property that helps show that the algorithm is correct.

Correctness of Insertion Sort

Invariant• At the start of each iteration of the for loop of lines 1–8, the sub-array A[1 .. j-1] consists of

the elements originally in A[1 .. j-1] , but in sorted order.

• Initialization• Maintenance: • the loop moves the A[1 .. j-2] .. Elements to the right until it finds a suitable

place for A[j]. The sub-array A[1 .. j] then consists of the elements originally in A[1 .. j] , but in sorted order

• Termination• In the end we have j = n +1, substitute this in the “invariant” wording

David Luebke 60

04/20/2023

Analysis Insertion Sort

David Luebke 61

04/20/2023

Analyzing Insertion Sort• T(n) = c1n + c2(n-1) + c4(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1)) + c7(n-1)

= c8T + c9n + c10

• What can T be?• Best case -- inner loop body never executed

• ti = 1 T(n) is a linear function• Worst case -- inner loop body executed for all previous

elements• ti = i T(n) is a quadratic function

• Average case• ???

Analysis

• Simplifications• Ignore actual and abstract statement costs• Order of growth is the interesting measure:

• Highest-order term is what counts• Remember, we are doing asymptotic analysis• As the input size grows larger it is the high order term that dominates

Designing algorithms

• Insertion Sort – Incremental Approach• The divide-and-conquer approach• Recursive in nature

• Three parts• Divide the problem into a number of sub-problems that are smaller instances

of the same problem.• Conquer the sub-problems by solving them recursively. If the sub-problem

sizes are small enough, however, just solve the sub-problems in a straightforward manner.• Combine the solutions to the sub-problems into the solution for the original

problem.

Merge Sort

• Divide: Divide the n-element sequence to be sorted into two subsequences of n=2 elements each.• Conquer: Sort the two subsequences recursively using merge sort.• Combine: Merge the two sorted subsequences to produce the sorted

answer.

Merge Sort - Example

Merge sort – Merge procedure

Invariants of Merge Procedure

At the start of each iteration of the for loop of lines 12–17, the sub-array A[p..k-1] contains the k -p smallest elements of L[1 .. N1+1] and R[1 .. N2+1], in sorted order. Moreover, L[i] and R[j] are the smallest elements of their arrays that have not been copied back into A.

• Initialization: Prior to the first iteration of the loop, we have k = p, so that the sub-array A[p..k-1] is empty.• Maintenance • Termination

Merge Sort procedure

Analyzing divide-and-conquer algorithms• recurrence equation or recurrence, which describes the overall

running time on a problem of size n in terms of the running time on smaller inputs.• Suppose that our division of the problem yields ‘a’ sub-problems,

each of which is ‘1/b’ the size of the original.• It takes time T (n/b) to solve one sub-problem of size n=b, and so it

takes time ‘a T (n/b)’ to solve ‘a’ of them.• D(n) : time to divide, C(n) : time to combine

Analysis of merge sort

• Divide: The divide step just computes the middle of the sub-array, which takes constant time. Thus, D(n) = Θ(1).• Conquer: We recursively solve two sub-problems, each of size n=2,

which contributes 2T(n/2) to the running time.• Combine: The combine step also takes constant time. Thus, C(n) =

Θ(1).

• The solution to this recurrence equation is T(n) = Θ(n lg n)

Why Θ(n lg n) ?

• Rewriting the initial recurrence

Why Θ(n lg n) ?

Cost calculation

• Top level cost = cn• Next level => cn = cn/2 +cn/2; next level => cn = cn/4+cn/4+cn/4+cn/4• At any level below the top we have 2i node each with a cost c(n/2i)• At any ith level below the tops has a total cost 2i c(n/2i) = cn• The bottom level has n nodes, each contributing a cost of c, for a total cost of cn.• Total number of levels is lg n + 1 (verify for 1, 2i, and 2i+1 )• For the total cost we simply add up the costs of all the levels.

Recursion tree has lg n + 1 levels, each costing cnÞ cn(lg n + 1) => cn lg n + cn Þ ignoring lower order terms and constants we have Θ(n lg n)

Documents

Design and Analysis of Algorithms