CSS342: Algorithm Analysis1 Professor: Munehiro Fukuda

CSS342: Algorithm Analysis 1

CSS342: Algorithm Analysis

Professor: Munehiro Fukuda


Today’s Topics

1. Mathematical analysis of algorithm complexity

2. Examples of algorithm complexity

3. Algorithm improvement from O(N3) to O(N)

4. Efficiency of search algorithms


Measuring Algorithm Efficiency• Cost of a computer program

– Human cost:

Developing good problem-solving skills and programming style

– Execution cost:

Algorithm efficiency Then, what in efficiency should we focus?

Clever trick in coding? Execution time? How should we compare programs?

Based on a particular computer? What data should be given?

Mathematical Analysis


Counting an Algorithm’s operationsfor ( i = 1; i <= n; i++ )

for ( j = 1; j <= i; j++ )

for ( k = 1; k <= 5; k++ )

task( );

Task: t operations ( or may take t seconds)

Loop on k: 5 * t

Loop on j: 5 * t (i = 1), 5 * t * 2 (i = 2), … 5 * t * n (i = n)

n

Loop on i: ∑(5 * t * i) = 5 * t * n * (n + 1) / 2i=1



Algorithm Growth Rates• Nested loop i/j/k requires 5n(n+1)/2 time units

to process a problem of size n.– Units may be seconds, minutes, and even dependent

on what machines we use!

• Nested loop i/j/k requires time proportional to 5n(n+1)/2– Focus on how quickly the algorithm’s time

requirement grows as a function of the problem size.

Growth rate



Order-of-Magnitude Analysis

Let Algorithm A require time proportional to f(n).

• A is of order at most g(n), O(g(n)) if f(n) <= k * g(n), for a positive constant k and n > n0

• A is of order at least g(n), Ω(g(n)) iff(n) >= k * g(n), for a positive constant k and n > n0

• A is of order g(n), Θ(g(n)) iff(n) = O(g(n)) and f(n) = Ω(g(n))



Example of O(g(n)), Ω(g(n)) ,andΘ(g(n))

Suppose Algorithm A requires time proportional to f(n) = n2 – 3n + 10

• f(n) < k* n2 , if k = 3 and n >= 2, and thusf(n) = O(n2 )

• f(n) > k * n2 , if k = 0.5 n>= 0, and thusf(n) = Ω(n2)

• Thereforef(n) = Θ(n2)

27104.53

12822

380.51

01000

3n2N2 -3N+100.5n2n



Example of Program Analysisj = n

while ( j >= 1 ) {

for ( i = 1; i <= j; i++ )

x = x + 1; // How many times is x=x+1 executed?

j = j / 2;

}

T(n): # x=x+1 executed

In the 1st while-loop, j = n. for-loop executes x=x+1 n times. T(n)=Ω(n)

In the 2nd while-loop, j <= n/2, n/2 times

In the 3rd while-loop, j <= n/4, n/4 times

In the kth while-loop, j <= n/2k-1, n/2k-1times



Example of Program Analysis Cont’d

Geometric Sum:

a + ar1 + ar2 + …. + arn = a(rn+1 – 1)/(r – 1), where r != 1

(Proof is given in the week 4, “Induction”)

Thus,

T(n) <= n + n/2 + n/4 + … + n/2k-1

= n + n*(1/2) + n*(1/2)2 + … + n * (1/2)(k-1)

= n((1/2)k-1)/(1/2 – 1) = n(1-1/2k)/(1-1/2)

= 2n(1-1/2k) <= 2n

T(n) = O(n)

Therefore, T(n) = Θ(n)



Focusing on Big O Notation

• Upper-bound information is the main concern • In most cases, O(g(n))=Ω(g(n)) =Θ(g(n))

• O(1) < O(log2n) < O(n) < O(n*log2n) < O(n2) < O(n3) < O(2n)

• Focus on the dominant factor:– Low-order terms can be ignored. O(n3+4n) = O(n3)

– Multiplicative constant in the high-order term can be ignored. O(3n3+4) = O(n3)

• O(f(n)) + O(g(n)) = O(f(n) + g(n))• If f(n) = O(g(n)) and g(n) = O(h(n)) , then f(n)=O(h(n))



Intuitive interpretation of growth-rate functions

Increase too rapidly to be practicalExponentialO(2n)

Ex. Three nested loopsCubicO(n3)

Ex. Two nested loopsQuadraticO(n2)

Increase more rapidly than a liner algorithm. Ex. Merge sort

O(nlog2n)

Increase directly with sizeLinearO(n)

Increase slowly as size increases. Ex. Binary search

LogarithmicO(log2n)

Independent of problem sizeConstantO(1)



Examples of Algorithm ComplexityMinimum Element in an Array

• Given an array of N items, find the smallest item.• What order (in big O) will this problem be bound to?

3 9 1 8 5 2 0 7 4 6 -1 10 -2

N1

Examples


Examples of Algorithm ComplexityClosest Points in the Plane

• Given N points in a plane (that is, an x-y coordinate system), find the pair of points that are closest together.

• What order (in big O) will this problem be bound to?

2,1

x

y1,41,6

4,74,5

5,45,6

7,18,4

6,76,3

sqrt( (x2 – x1)2 + (y2 – y1)2 )

Examples


Examples of Algorithm ComplexityColinear Points in the Plane

• Given N points in a plane (that is, an x-y coordinate system), determine if any tree form a straight line.

• What order (in big O) will this problem be bound to?

2,1

x

y1,41,6

4,74,5

5,45,6

7,18,4

6,76,3

Find y = ax + b for two points.Check if the 3rd point satisfies this equation

Examples


The Maximum Contiguous Subsequence Sum Problem

• Given a sequence of integers I1..IN, find and identify the sub sequence corresponding to the maximum value of ∑k=i

jAk.

Algorithm Improvement

-2 11 -4 13 -5 2 1 -3 4 -2 -1 6 -2

N

20

1


O(N3) AlgorithmAlgorithm Improvement

-2 11 -4 13 -5 2 1 -3 4 -2 -1 6 -2

N1

i j

∑k=ijAk

The most inner loop

The intermediate loop

j++The most outer loop i++

int maxSum = 0;for ( int i = 0; i < n; i++ ) for ( int j = i; j < n; j++ ) { int thisSum = 0; for ( int k = i; k <= j; k++ ) thisSum += a[k];

if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } }

We don’t need this loop!Simply increment j and add a new integer to the sum.


O(N2) AlgorithmAlgorithm Improvement

-2 11 -4 13 -5 2 1 -3 4 -2 -1 6 -2

N1

i j

∑j=ijAj

The intermediate loop

j++The most outer loop i++

int maxSum = 0;for ( int i = 0; i < n; i++ ) { int thisSum = 0; for ( int j = i; j < n; j++ ) { thisSum += a[j];

if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } }}


O(N) AlgorithmAlgorithm Improvement

N1∑ < 0

Ignore this partFrom here, restart the summation.

∑j=inAj

Generally, the summation increases but if it gets below 0

int thisSum = 0;int maxSum = 0;for ( int i = 0, j = 0; j < n; j++ ) { thisSum += a[j];

if ( thisSum > maxSum ) { maxSum = thisSum; seqStart = i; seqEnd = j; } else if ( thisSun < 0 ) { i = j + 1; thisSum = 0; }}


Worst, Best, and Average-case Analysis

• Worst-case analysis:– Algorithm A requires no more than k * f(n) time units.

– It might happen rarely, if at all, in practice.

• Best-case analysis:– Like worst-case analysis, it might happen rarely.

• Average-case analysis:– A requires no more than k * f(n) time units for all but a finite

number of values of n

– Difficulties: determining probability and distribution of size n and input data

Efficiency of Search Algorithms


Efficiency of Sequential Search

29 10 14 1337 4652 75 5 43 6921 188

Best caseO(1)

Hit!

Worst caseHit!

O(n)

Average caseHit!

O(n/2) = O(n)(Assume desired items are uniformly distributed.)



Efficiency of Binary Search

2910 1413 37 46 52 755 43 69211 88 9991


template <class Comparable>int binarySearch( const vector<Comparable> &a, const Comparable &x ) { int low = 0; int high = a.size( ) – 1; int mid;

while ( low <= high ) { mid = ( low + high ) / 2;

if ( a[mid] < x ) low = mid + 1; else if ( a[mid] > x ) high = mid – 1; else return mid; } return NOT_FOUND; // NOT_FOUND = -1}

low =0 high=15mid = (15 + 0)/2 = 7

10x

anew high=7–1=6


Efficiency of Binary Search

N = 2 K = 1N = 4 K = 2N = 8 K = 3N = 16 K = 4N = 2k K = K2K-1 < N < 2K

K – 1 < log2N < K

K < 1 + log2N < K + 1 K = O(log2N)

2910 1413 37 46 52 755 43 69211 88

1st step

9991

2nd step3rd step

4th stepHit!



Efficiency of Interpolation Search

• A way you start your search for “Hank Aaron” from the first 20 pages in a 500-page white page:next = low + ┌ (x – a[low]) / (a[high] – a[low]) * (high – low – 1)┐Example:

Assume low = 0, high = 999, a[low] = 1000, a[high]=1000000.If you search for the value 12000:next = 0 + ┌(12000 – 1000)/(1000000 – 1000) * (999 – 0 – 1) ┐ = 10

• Two assumptions:– Each access must be very expensive like a disk access.– Data must be uniformly distributed

• Complexity– Average: O(log log N)– Worst: O(N)



Some Note for Algorithm Analysis

• Checking an algorithm analysis1. Empirically observe running time as increasing N.2. Divide the time by O(f(N)).3. The dividend should converge to a positive constant.

• Limitations of Big-Oh Analysis– Does not work for a small N– Worst-case analysis may be uncommon– Average-case analysis is more difficult than worse

case– Empirically observation depends on memory

hierarchy and multiprogramming level, etc..

Limitations


Intractable, Unsolvable, and NP problems

• Good algorithms:– Their worst-case time is proportional to a polynomial.

• Intractable problems:– Their worst-case time cannot be bounded to a polynomial.

• Unsolvable algorithms:– They have no algorithms at all. Ex. Halting problem

• NP(Non-Polynomial) problems:– They are thought to be intractable, but not yet proved.

– Ex1. Traveling salesperson problem

– Ex2. Hamiltonian cycle

Limitations

Documents

CSS342: Algorithm Analysis1 Professor: Munehiro Fukuda