design of algo

CA729 DESIGN AND ANALYSIS OF ALGORITHMS

The Course

• CA729 DESIGN AND ANALYSIS OF ALGORITHMS• Objective: – A rigorous introduction to the design and analysis of

algorithms– To learn about Time Complexity and various algorithmic

design methodologies• Textbook: Introduction to Algorithms, Cormen,

Leiserson, Rivest, Stein– Third edition

• Instructor:

Algorithms as a technology• Suppose computers were infinitely fast and computer memory

was free. • Would you have any reason to study algorithms? • The answer is yes

– You would still like to demonstrate that your solution method terminates and does so with the correct answer.

• If computers were infinitely fast, any correct method for solves the problem. – Implementation to be within the bounds of good software

engineering practice (i.e., well designed and documented)– Among many available algorithms, people use whichever method

was the easiest to implement.

Algorithms as a technology

• Of course, computers may be fast, but they are not infinitely fast.

• Memory may be cheap, but it is not free. • Computing time is therefore a bounded

resource, and so is space in memory. These resources should be used cleverly.

• Algorithms that are efficient in terms of time or space will help you do so.

Algorithms as a technology• Algorithms devised to solve the same problem often differ dramatically in their

Efficiency.• Consider two algorithms

– Insertion sort Merge sort

c1n2

• C1 and C2 are not depends on n• Insertion sort usually has a smaller constant factor than merge sort,

so that c1 < c2• Constant factors can be far less significant in the running time.• Insertion sort is usually faster than merge sort for small input sizes.• For large n, merge sort runs faster.

c2 n log n

Algorithms as a technology• Consider two Computers

Computer A• Insertion Sort• One billion instructions per

second• 100 times faster• C1n2

• c1 = 2• 2n2

Computer B• Merge Sort• Ten million instructions per

second

• C2 n log n• c2 = 50• 50 n log n

Algorithms as a technology• By using an algorithm whose running time grows more slowly, even with

computer B runs 20 times faster than computer A!• The advantage of merge sort is even more pronounced when we sort ten

million numbers: – Where insertion sort takes approximately 2.3 days, merge sort takes

under 20 minutes.• In general, as the problem size increases, so does the relative advantage of

merge sort increases.• The example above shows that algorithms, like computer hardware, are a

technology.• Total system performance depends on choosing efficient algorithms as much

as on choosing fast hardware.• Just as rapid advances are being made in other computer technologies, they

are being made in algorithms as well.

Insertion Sort

Sorting a hand of cards using insertion sort

Insertion SortPseudocode

Insertion SortExample

Insertion Sort• At beginning of each iteration,

– the subarray A[1…j-1] constitutes sorted array– the subarray A[j+1…n] corresponds to unsorted array

• Loop Invariant:– At the start of each iteration elements A[1…j-1] are the elements originally in positions from 1 to j-1– But, in sorted order

• Correctness of Algorithm:– Loop invariant is used

• Initialization: – J=2, A[1…j-1] = A[1]– Only one element

• Maintenance:– For any j value, body of the for loop moves A[j-1], A[j-2], A[j-3],…. one position right until find a proper

position fie A[j].– Then inserts A[j] in correct position– Incrementing j for next iteration also preserves the loop invariant

• Termination:– When for loop terminates?– Termination condition j > A.length = n– When j = n+1, we have the subarray A[1…j-1] in sorted order which is equal to A[1…]

Analysis of Algorithms

• Analysis is performed with respect to a computational model

• We will usually use a generic uniprocessor random-access machine (RAM)– All memory equally expensive to access– No concurrent operations– All reasonable instructions take unit time

• Except, of course, function calls– Constant word size

• Unless we are explicitly manipulating bits

Input Size

• Time and space complexity– This is generally a function of the input size• E.g., sorting, multiplication

– How we characterize input size depends:• Sorting: number of input items• Multiplication: total number of bits• Graph algorithms: number of nodes & edges• Etc

Running Time

• Number of primitive steps that are executed– Except for time of executing a function call– Most statements roughly require the same

amount of time• y = m * x + b• c = 5 / 9 * (t - 32 )• z = f(x) + g(y)

• We can be more exact if need be

Analysis

• Worst case– Provides an upper bound on running time– An absolute guarantee

• Average case– Provides the expected running time– Very useful, but treat with care: what is

“average”?• Random (equally likely) inputs• Real-life inputs

Analysis

• Why it matters• Growth of functions

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {A[j+1] = A[j]j = j - 1}A[j+1] = key}

}

30 10 40 20

1 2 3 4

i = j = key = A[j] = A[j+1] =



}

30 10 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 10



}

30 30 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30



}

30 30 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30



}

30 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = A[j+1] = 30



}

30 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = A[j+1] = 30



}

10 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = A[j+1] = 10



}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 10A[j] = A[j+1] = 10



}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 40A[j] = A[j+1] = 10



}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 40A[j] = A[j+1] = 10



}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40



}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40



}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40



}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 40A[j] = 30 A[j+1] = 40



}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40



}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40



}

10 30 40 20

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20



}

10 30 40 20

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20



}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40



}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40



}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40



}

10 30 40 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40



}

10 30 40 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40



}

10 30 30 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30



}

10 30 30 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30



}

10 30 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30



}

10 30 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30



}

10 20 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20



}

10 20 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20

Done!

Insertion SortInsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

What is the preconditionfor this loop?

Insertion SortInsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

How many times will this loop execute?

Insertion SortStatement Effort

InsertionSort(A, n) {for i = 2 to n { c1nkey = A[i] c2(n-1)j = i - 1; c3(n-1)while (j > 0) and (A[j] > key) { c4TA[j+1] = A[j] c5(T-(n-1))j = j - 1 c6(T-(n-1))} 0A[j+1] = keyc7(n-1)} 0

}T = t2 + t3 + … + tn where ti is number of while expression evaluations for the ith for loop iteration

Analyzing Insertion Sort• T(n) = c1n + c2(n-1) + c3(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1)) + c7(n-1)

= c8T + c9n + c10

• What can T be?– Best case -- inner loop body never executed• ti = 1 T(n) is a linear function

– Worst case -- inner loop body executed for all previous elements• ti = i T(n) is a quadratic function

– Average case• ???

Analysis

• Simplifications– Ignore actual and abstract statement costs– Order of growth is the interesting measure:• Highest-order term is what counts

– Remember, we are doing asymptotic analysis– As the input size grows larger it is the high order term that

dominates

David Luebke 51

04/22/2023

Upper Bound Notation

• We say InsertionSort’s run time is O(n2)– Properly we should say run time is in O(n2)– Read O as “Big-O” (you’ll also hear it as “order”)

• In general a function– f(n) is O(g(n)) if there exist positive constants c and n0

such that f(n) c g(n) for all n n0

• Formally– O(g(n)) = { f(n): positive constants c and n0 such that

f(n) c g(n) n n0

Insertion Sort Is O(n2)

• Proof– Suppose runtime is an2 + bn + c • If any of a, b, and c are less than 0 replace the constant

with its absolute value– an2 + bn + c (a + b + c)n2 + (a + b + c)n + (a + b + c)– 3(a + b + c)n2 for n 1– Let c’ = 3(a + b + c) and let n0 = 1

• Question– Is InsertionSort O(n3)?– Is InsertionSort O(n)?

Documents

design of algo