2IL05 2008 1 Introduction

8/8/2019 2IL05 2008 1 Introduction

1/34

Introduction to Algorithms

Introduction


2/34

Algorithms

Algorithma well-defined computational procedure that takes some value, or aset of values, as input and produces some value, or a set of values,as output.

Algorithmsequence of computational steps that transform the input into theoutput.

Algorithms research

design and analysis of algorithms and data structures forcomputational problems.


3/34

Data structures

Data Structurea way to store and organize data to facilitate access andmodifications.

Abstract data type

describes functionality (which operations are supported).

Implementationa way to realize the desired functionality

how is the data stored (array, linked list, )

which algorithms implement the operations


4/34

Sorting

lets get started


5/34

The sorting problem

Input: a sequence ofn numbers a1, a2, , anOutput: a permutation of the input such that ai1 ain

The input is typically stored in arrays

Numbers Keys

Additional information (satellite data) may be stored with keys

We will study several solutions algorithms for this problem


6/34

Describing algorithms

A complete description of an algorithm consists ofthree parts:

1. the algorithm

(expressed in whatever way is clearest and most concise,can be English and / or pseudocode)

2. a proof the algorithms correctness

3. a derivation of the algorithms running time


7/34

InsertionSort

Like sorting a hand of playing cards: start with empty left hand, cards on table

remove cards one by one, insert intocorrect position

to find position, compare to cards in hand from left to right

cards in hand are always sorted

InsertionSort is

a good algorithm to sort a small number of elements

an incremental algorithm

Incremental algorithmsprocess the input elements one-by-one and maintain the solutionfor the elements processed so far.


8/34

Incremental algorithms

Incremental algorithmsprocess the input elements one-by-one and maintain the solutionfor the elements processed so far.

In pseudocode:

IncAlg(A)

incremental algorithm which computes the solution of a problemwith input A = {x1,,xn}

1. initialize: compute the solution for {x1}

2. fori 2 to n3. do compute the solution for {x1,,xj} using the (already

computed) solution for {x1,,xj-1}

comment

assignment (Pascal :=)

no begin - end, just indentation

Check book for more pseudocode conventions


9/34

InsertionSort

InsertionSort(A) incremental algorithm that sorts array A[1..n] in non-decreasing

order1. initialize: sort A[1]2. forj 2 to length[A]

3. do sort A[1..j] using the fact that A[1.. j-1] is already sorted


10/34

InsertionSort

InsertionSort(A) incremental algorithm that sorts array A[1..n] in non-decreasing

order1. initialize: sort A[1]2. forj 2 to length[A]

3. do key A[j]4. i j -15. while i > 0 and A[i] > key6. do A[i+1] A[i]7. i i -18. A[i +1] key

1 3 14 17 28 6

1 j n

InsertionSort is an in place algorithm:the numbers are rearranged within the

array with only constant extra space.


11/34

Correctness proof

Use a loop invariant to understand why an algorithm gives thecorrect answer.

Loop invariant (for InsertionSort)At the start of each iteration of the outer forloop (indexed by j) the

subarray A[1..j-1] consists of the elements originally in A[1..j-1] butin sorted order.


12/34

Correctness proof

To proof correctness with a loop invariant we need to show threethings:

InitializationInvariant is true prior to the first iteration of the loop.

MaintenanceIf the invariant is true before an iteration of the loop, it remains truebefore the next iteration.

TerminationWhen the loop terminates, the invariant (usually along with thereason that the loop terminated) gives us a useful property thathelps show that the algorithm is correct.


13/34

Correctness proof

InsertionSort(A)1. initialize: sort A[1]2. forj 2 to length[A]3. do key A[j]4. i j -15. while i > 0 and A[i] > key

6. do A[i+1] A[i]7. i i -18. A[i +1] key

InitializationJust before the first iteration, j = 2 A[1..j-1] = A[1], which is theelement originally in A[1], and it is trivially sorted.

Loop invariantAt the start of each iteration of theouter forloop (indexed by j) thesubarray A[1..j-1] consists of theelements originally in A[1..j-1] butin sorted order.


14/34

Correctness proof


6. do A[i+1] A[i]7. i i -18. A[i +1] key

MaintenanceStrictly speaking need to prove loop invariant for inner while loop.Instead, note that body ofwhile loop moves A[j-1], A[j-2], A[j-3],and so on, by one position to the right until proper position of key isfound (which has value of A[j]) invariant maintained.



15/34

Correctness proof


6. do A[i+1] A[i]7. i i -18. A[i +1] key

TerminationThe outerforloop ends when j > n; this is when j = n+1 j-1 = n.Plug n for j-1 in the loop invariant the subarray A[1..n] consistsof the elements originally in A[1..n] in sorted order.



16/34

Another sorting algorithm

using a different paradigm


17/34

MergeSort

A divide-and-conquersorting algorithm.

Divide-and-conquerbreak the problem into two or more subproblems, solve thesubproblems recursively, and then combine these solutions to

create a solution to the original problem.


18/34

D&CAlg(A) divide-and-conquer algorithm that computes the solution of a

problem with input A = {x1,,xn}

1. if # elements of A is small enough (for example 1)

2. then compute Sol (the solution for A) brute-force

3. else

4. split A in, for example, 2 non-empty subsets A1 and A25. Sol1 D&CAlg(A1)

6. Sol2 D&CAlg(A2)

7. compute Sol (the solution for A) from Sol1 and Sol28. return Sol

Divide-and-conquer


19/34

MergeSort(A) divide-and-conquer algorithm that sorts array A[1..n]

1. iflength[A] = 1

2. then compute Sol (the solution for A) brute-force

3. else4. split A in 2 non-empty subsets A1 and A25. Sol1 MergeSort(A1)

6. Sol2 MergeSort(A2)

7. compute Sol (the solution for A) from Sol1 en Sol2

MergeSort


20/34

MergeSort(A) divide-and-conquer algorithm that sorts array A[1..n]

1. iflength[A] = 1

2. then skip

3. else4. n length[A] ; n1 n/2 ; n2 n/2 ;

copy A[1.. n1] to auxiliary array A1[1.. n1]

copy A[n1+1..n] to auxiliary array A2[1.. n2]

5. MergeSort(A1)

6. MergeSort(A2)7. Merge(A, A1, A2)

MergeSort


21/34


22/34

MergeSort

Merging

1 3 4 7 8 14 17 21 28 35A

1 3 14 17 28A1 4 7 8 21 35A2


23/34

MergeSort: correctness proof

Induction on n (# of input elements)

proof that the base case (n small) is solved correctly

proof that if all subproblems are solved correctly, then the

complete problem is solved correctly


24/34

MergeSort: correctness proof

Proof(by induction on n)

Base case: n = 1, trivial

Inductive step: assume n > 1. Note that n1 < n and n2 < n.Inductive hypothesis arrays A1 and A2 are sorted correctly

Remains to show: Merge(A, A1, A2) correctly constructs a sortedarray A out of the sorted arrays A1 and A2 etc.

MergeSort(A)

1. iflength[A] = 1

2. then skip

3. else

4. n length[A] ; n1 n/2 ; n2 n/2 ;

copy A[1.. n1] to auxiliary array A1[1.. n1]

copy A[n1+1..n] to auxiliary array A2[1.. n2]

5. MergeSort(A1)6. MergeSort(A2)

7. Merge(A, A1, A2)

LemmaMergeSort sorts the array A[1..n] correctly.


25/34

QuickSortanother divide-and-conquer sorting

algorithm


26/34

QuickSort

QuickSort(A) divide-and-conquer algorithm that sorts array A[1..n]

1. iflength[A] 1

2. then skip

3. else

4. pivot A[1]5. move all A[i] with A[i] < pivot into auxiliary array A16. move all A[i] with A[i] > pivot into auxiliary array A27. move all A[i] with A[i] = pivot into auxiliary array A3

8. QuickSort(A1)9. QuickSort(A2)

10. A A1 followed by A3 followed by A2


27/34

Analysis of algorithms

some informal thoughts for now


28/34


Can we say something about the running time of an algorithmwithout implementing and testing it?

InsertionSort(A)

1. initialize: sort A[1]2. forj 2 to length[A]3. do key A[j]4. i j -15. while i > 0 and A[i] > key6. do A[i+1] A[i]

7. i i -18. A[i +1] key


29/34


Analyze the running time as a function ofn (# of input elements) best case

average case

worst case

elementary operationsadd, subtract, multiply, divide, load, store, copy, conditional andunconditional branch, return

An algorithm has worst case running time T(n) if for anyinput of size n the maximal number ofelementary operationsexecuted is T(n).


30/34

Analysis of algorithms: example

InsertionSort: 15 n2 + 7n 2

MergeSort: 300 n lg n + 50 n

n=10 n=100 n=10001568 150698 1.5 x 107

10466 204316 3.0 x 106

InsertionSort6 x faster

InsertionSort1.35 x faster

MergeSort

5 x faster

n = 1,000,000 InsertionSort 1.5 x 1013

MergeSort 6 x 109 2500 x faster!


31/34


It is extremely important to have efficient algorithms for large inputs

The rate of growth (ororder of growth) of the running time is farmore important than constants

InsertionSort: (n2)

MergeSort: (n log n)


32/34

-notation

Intuition: concentrate on the leading term, ignore constants

19 n3 + 17 n2 - 3n becomes (n3)

2 n lg n + 5 n1.1 - 5 becomes

n - n n becomes

(precise definition next week )

(n1.1)

---


33/34

Some rules and notation

log n denotes log2 n

We have fora, b, c > 0 :

1. logc (ab) =

logc a + logc b

2. logc (ab) =

b logc a

3. loga b =

logc b / logc a


34/34

Find the leading term

lg35n vs. n ?

logarithmic functions grow slower than polynomial functions

lga n grows slower than nb for all constants a > 0 and b > 0

n100 vs. 2 n ?

polynomial functions grow slower than exponential functions

na grows slower than bn for all constants a > 0 and b > 1

Documents

2IL05 2008 1 Introduction