View
223
Download
2
Tags:
Embed Size (px)
Citation preview
eleg667/2001-f/Topic-1a 1
A Brief Review of
Algorithm Design and Analysis
eleg667/2001-f/Topic-1a 2
Outline
Basic concepts; Case study; Analyzing algorithms Comparing algorithms Problem complexity
eleg667/2001-f/Topic-1a 3
Algorithm - Definition
Sequence of steps to carry out a particular task
Examples: Recipe; Assembly instructions; ……
eleg667/2001-f/Topic-1a 4
Characteristics of Algorithms
Algorithms are precise. Each step has a clearly defined meaning;
Algorithms are effective. The task is always done as required;
Algorithms have a finite number of steps; Algorithms must terminate.
eleg667/2001-f/Topic-1a 5
Any algorithm can be stated using the following logic constructs
Sequence: steps must follow each other in a logical sequence;
Decision: there may be alternative steps that may be taken subject to a particular condition;
Repetition: certain steps may be repeated while, or until, a certain condition is true.
eleg667/2001-f/Topic-1a 6
Why study algorithms?
A motivating example…
eleg667/2001-f/Topic-1a 7
Example: sorting playing cards
Start with one card on the left hand and
the others cards facing down on the table;
while there are cards left on the table do
{ take one card from the table;
compare it with each of the cards
already in the hand, from left to right,
inserting it into the right position;
}
Sequence
Decision
Repetition
Insertion Sort
eleg667/2001-f/Topic-1a 8
Expressing computer algorithms
It should be expressed in a language more precise, less ambiguous, and more compact than a “natural language” such as English;
Algorithms are usually written in a pseudocode language and later translated to a real programming language.
eleg667/2001-f/Topic-1a 9
Insertion Sort – An Intuitive Picture
j Q K5
…
Q
i
1 1
5252
key
A A1
…
eleg667/2001-f/Topic-1a 10
Insertion Sort in Pseudocode
Insertion-Sort(A) ; A is an array of numbers for j = 2 to length(A) key = A[j] i = j - 1 while i > 0 and A[i] > key A[i+1] = A[i] i = i - 1 A[i+1] = key
SequenceRepetition
Decision
eleg667/2001-f/Topic-1a 11
Algorithms and Machine Models
Sequential algorithms and machine
models
Parallel algorithms and machine models
eleg667/2001-f/Topic-1a 12
Analyzing algorithms
Estimate the running time of algorithms;
= F(Problem Size)
= F(Input Size)
= number of primitive operations used (add, multiply, compare etc)
eleg667/2001-f/Topic-1a 13
Analysis for Insertion Sort
Insertion-Sort(A) Cost Times (Iterations) 1 for j = 2 to length(A) { c1
2 key = A[j] c2
3 i = j – 1 c3 4 while i > 0 and A[i] > key c4
5 A[i+1] = A[i] c5
6 i = i – 1 c6
7 A[i+1] = key c7 }
n
n - 1
n - 1
n - 1
eleg667/2001-f/Topic-1a 14
Insertion Sort Analysis (cont.)
Best Case:
Array already sorted, tj = 1 for all j
eleg667/2001-f/Topic-1a 15
Insertion Sort Analysis (cont.)
Worst Case:
Array in reverse order, tj = j for all j
Note that
eleg667/2001-f/Topic-1a 16
Insertion Sort Analysis (cont.)
Average Case:
Check half of array on average,
tj = j/2 for all j
T(n) = a.n2 + bn + c (quadractic in n)
We are usually interested in the worst-case running timeWe are usually interested in the worst-case running time
eleg667/2001-f/Topic-1a 18
Growth rates
constant growth rate: T(n) = c linear growth rate : T(n) = c*n logarithmic growth rate : T(n) = c*log n quadratic growth rate : T(n) = c*n2
exponential growth rate : T(n) = c*2n
eleg667/2001-f/Topic-1a 19
Asymptotic Analysis
Ignoring constants in T(n) Analyzing T(n) as n "gets large"
The running time grows roughly on the order of
Notationally, =
" n
T n O n
3
3
"
( ) ( )
As grows larger, is MUCH large than , , and ,so it dominates
n n n n nT n
n 23 log( )
T n n n n n n( ) log 13 42 2 43 2Example:
The big-oh (O) Notation
eleg667/2001-f/Topic-1a 20
Formally…
T(n) = O(f(n)) if there are constants c and n such that T(n) < c.f(n) when n n0
c.f(n)
T(N)
n0 n
f(n) is an upper bound for T(n)
eleg667/2001-f/Topic-1a 21
O ( big-oh)
Describes an upper bound for the running time of an algorithm
Upper bounds for Insertion Sort running times:
•worst case: O(n2) T(n) = c1.n2 + c2.n + c3
•best case: O(n) T(n) = c1.n + c2
•average case: O(n2) T(n) = c1.n2 + c2.n + c3
Time Complexity
eleg667/2001-f/Topic-1a 22
Some properties of the O notation
Fastest growing function dominates a sum O(f(n)+g(n)) is O(max{f(n), g(n)})
Product of upper bounds is upper bound for the product If f is O(g) and h is O(r) then fh is O(gr)
f is O(g) is transitive If f is O(g) and g is O(h) then f is O(h)
Hierarchy of functions O(1), O(logn), O(n1/2), O(nlogn), O(n2), O(2n), O(n!)
eleg667/2001-f/Topic-1a 23
Times on a 1-billion-steps-per-second computer
eleg667/2001-f/Topic-1a 24
Simple statement sequence s1; s2; …. ; sk
O(1) as long as k is constant
Simple loops for(i=0;i<n;i++) { s; } where s is O(1) Time complexity is n O(1) or O(n)
Analyzing an Algorithm
eleg667/2001-f/Topic-1a 25
Analyzing an Algorithm (cont.)
Nested loops for(i=0;i<n;i++) for(j=0;j<n;j++) { s; }
Complexity is n O(n) or O(n2)
Recursion Search(n) if middle_element = key return it
else if it’s greater search_left(n/2) else search_right(n/2) Solve recurrence equation: T(n) = T(1) + T(n/2) Complexity is O(log2n)
eleg667/2001-f/Topic-1a 26
Reasonable and Unreasonable Algorithms
Polynomial Time algorithms An algorithm is said to be polynomial if it is
O( nc ), c >1 Polynomial algorithms are said to be reasonable
They solve problems in reasonable times!
Exponential Time algorithms An algorithm is said to be exponential if it is
O( rn ), r > 1 Exponential algorithms are said to be unreasonable
eleg667/2001-f/Topic-1a 27
Another example: Quick Sort
Uses recursion to repetitively divide list in 1/2 and sort the two halves
QuickSort partitions the list into a left and a right sublist, where all values in the left sublist are less than all values in the right sublist. Then the QuickSort is applied recursively to the sublists until each sublist has only one element.
eleg667/2001-f/Topic-1a 28
Quick Sort (cont.)
9 1 25 4 15 4 1 9 25 15
becomes
4 1
25 15 becomes
becomes 1
4 15 25
1 4 15 25
eleg667/2001-f/Topic-1a 29
Quick Sort (cont.)
QuickSort (list(0,n)):
If length (list) > 1 then
partition into 2 sublists from
0..split-1 and from split + 1..n
QuickSort (list (0,split-1))
QuickSort (list (split + 1, n)
eleg667/2001-f/Topic-1a 30
1. Pick an array value, pivot.2. Use two indices, i and j.
a. Begin with i = left and j =right + 1b. As long as i is less than j do 3 steps:
i. Keep increasing i until we come to an element A[i] >= pivot.ii. Keep decreasing j until we come to an element A[j] <= pivot.iii. Swap A[i] and A[j].
Quick Sort – Partition phase
eleg667/2001-f/Topic-1a 31
The Operation of Quicksort
eleg667/2001-f/Topic-1a 32
Quick Sort – Complexity
Assume the list size (n) is a power of 2 and that pivot is always chosen such that it divides the list into two equal pieces
At each step we are cutting in half the size to be sorted
O(nlog2n)
eleg667/2001-f/Topic-1a 33
Quick Sort demo…
eleg667/2001-f/Topic-1a 34
Comparing the Sort Algorithms…
eleg667/2001-f/Topic-1a 35
Problem complexity
problems
sort
. . .Insertion Shellsort QuicksortAlgorithms:
eleg667/2001-f/Topic-1a 36
Definitions
A problem’s upper bound refers to the best that we know how to do. It is determined by the best algorithmic solution that we have found for a given problem.
Example: For the sorting problem O(n.logn) is the upper bound.
eleg667/2001-f/Topic-1a 37
Definitions (cont.)
A problem’s lower bound refers to the best algorithmic solution that is theoretically possible.
Example: It can be proved that sorting an unsorted list cannot be done in less than O(n.log n), unless we make assumptions about the particular input data. Thus O(n.log n) is the lower bound for sorting problems.
eleg667/2001-f/Topic-1a 38
Definitions (cont.)
Upper bounds tell us the best we’ve been able to do so far;
Lower bounds tell us the best we can hope to do.
For a given problem, if the upper and lower bounds are the same, then we refer to that problem as a closed problem.
eleg667/2001-f/Topic-1a 39
Tractable and Intractable problems
Tractable problems have both upper and lower bounds polynomial – O(logn), O(n), O(n.logn), O(nk);
Intractable problems have both upper and lower bounds exponential - O(2n), O(n!), O(nn);
eleg667/2001-f/Topic-1a 40
Problems that cross the line
We have found only exponential solutions (i.e. from the standpoint of our algorithms, it appears to be intractable);
We cannot prove the necessity of an exponential solution (i.e. from the standpoint of our proofs, we cannot say that it is intractable).
eleg667/2001-f/Topic-1a 41
NP-complete problems
The upper bound suggests that the problem is intractable;
No proof exists that shows these problems are intractable;
Once a solution is found, it can be checked in polynomial time;
They are all reducible to one another;
eleg667/2001-f/Topic-1a 42
Example
Traveling salesman: give a weighted graph (e.g., cities with distances between them) and some distance k (e.g., the maximum distance you want to travel), is there some tour that visits all the points (e.g., cities) and returns home such that distance k?
…
eleg667/2001-f/Topic-1a 43
P = NP ?
Win the Turing Award
and
get a $1 million prize!http://www.claymath.org/prize_problems/p_vs_np.htm