Lecture 01236

CSE 830:Design and Theory of AlgorithmsDr. Eric Torng

Welcome to CSE 830, the Design and Theory of Algorithms.

My name is Charles, and I will be your instructor for this class. The goal of the class is to teach you the fundamentals of creating and analyzing computer algorithms.

Before we get to the actual material, however, I need to go through a few administrative details. While I take attendance, I have some handouts for you to be reading over.

OutlineDefinitionsAlgorithmsProblemsCourse ObjectivesAdministrative stuff Analysis of Algorithms

What is an Algorithm?Algorithms are the ideas behind computer programs.

An algorithm is the thing that stays the same whether the program is in C++ running on a Cray in New York or is in BASIC running on a Macintosh in Katmandu!

To be interesting, an algorithm has to solve a general, specified problem.

What is a problem?DefinitionA mapping/relation between a set of input instances (domain) and an output set (range)Problem SpecificationSpecify what a typical input instance isSpecify what the output should be in terms of the input instanceExample: SortingInput: A sequence of N numbers a1anOutput: the permutation (reordering) of the input sequence such that a1 a2 an .

Types of Problems

Search: find X in the input satisfying property YStructuring: Transform input X to satisfy property YConstruction: Build X satisfying YOptimization: Find the best X satisfying property YDecision: Does X satisfy Y?Adaptive: Maintain property Y over time.

Two desired properties of algorithmsCorrectnessAlways provides correct output when presented with legal inputEfficiencyWhat does efficiency mean?

Example: Odd NumberInput: A number nOutput: Yes if n is odd, no if n is evenWhich of the following algorithms solves Odd Number best? Count up to that number from one and alternate naming each number as odd or even. Factor the number and see if there are any twos in the factorization. Keep a lookup table of all numbers from 0 to the maximum integer. Look at the last bit (or digit) of the number.

Example: TSPInput: A sequence of N cities with the distances dij between each pair of citiesOutput: a permutation (ordering) of the cities that minimizes the expression

S{j =1 to n-1} dj,j+1 + dn,1

Possible Algorithm: Nearest neighbor

Not Correct!

A Correct AlgorithmWe could try all possible orderings of the points, then select the ordering which minimizes the total length:

d = For each of the n! permutations, Pi of the n points,if cost(Pi) < d thend = cost(Pi)Pmin = Pireturn Pmin

Course ObjectivesLearning classic algorithmsHow to devise correct and efficient algorithms for solving a given problemHow to express algorithmsHow to validate/verify algorithmsHow to analyze algorithmsHow to prove (or at least indicate) no correct, efficient algorithm exists for solving a given problemWriting clear algorithms and

Classic AlgorithmsLots of wonderful algorithms have already been developedI expect you to learn most of this from reading, though we will reinforce in lecture

How to devise algorithmsSomething of an art formCannot be fully automatedWe will describe some general techniques and try to illustrate when each is appropriate

Expressing AlgorithmsImplementationsPseudo-codeEnglishMy main concern here is not the specific language used but the clarity of your expression

In this class, most algorithms will be expressed in plain English with pseudo-code used to clear up any ambiguities.

Verifying algorithm correctnessProving an algorithm generates correct output for all inputsOne technique covered in textbookLoop invariantsWe will do some of this in the course, but it is not emphasized as much as other objectives

Analyzing algorithmsThe process of determining how much resources (time, space) are used by a given algorithmWe want to be able to make quantitative assessments about the value (goodness) of one algorithm compared to anotherWe want to do this WITHOUT implementing and running an executable version of an algorithmQuestion: How can we study the time complexity of an algorithm if we dont run it or even choose a specific machine to measure it on?

Proving hardness resultsWe believe that no correct and efficient algorithm exists that solves many problems such as TSPWe define a formal notion of a problem being hardWe develop techniques for proving hardness results

Algorithm Analysis OverviewRAM model of computationConcept of input sizeThree complexity measuresBest-case, average-case, worst-caseAsymptotic analysisAsymptotic notation

The RAM ModelRAM model represents a generic implementation of the algorithmEach simple operation (+, -, =, if, call) takes exactly 1 step.Loops and subroutine calls are not simple operations, but depend upon the size of the data and the contents of a subroutine. We do not want sort to be a single step operation.Each memory access takes exactly 1 step.

Basically, any basic operation that takes a small, fixed amount of time we assume to take just one step.

We measure the run time of an algorithm by counting the number of steps it takes.

Why does this work? For the same reason that the Flat Earth model works. In our day-to-day lives we assume the Earth to be flat!

Now, lets look at how we can use this.

Input SizeIn general, larger input instances require more resources to process correctlyWe standardize by defining a notion of size for an input instanceExamplesWhat is the size of a sorting input instance?What is the size of an Odd number input instance?

Measuring ComplexityThe running time of an algorithm is the function defined by the number of steps required to solve input instances of size nF(1) = 3F(2) = 5F(3) = 7F(n) = 2n+1What potential problems do we have with the above definition when applied to real algorithms solving real problems?

Case study: Insertion SortCount the number of times each line will be executed:

Num Exec.for i = 2 to n(n-1) + 1key = A[i] n-1j = i - 1 n-1while j > 0 AND A[j] > key ?A[j+1] = A[j] ?j = j -1 ?A[j+1] = key n-1

Measuring Complexity AgainThe worst case running time of an algorithm is the function defined by the maximum number of steps taken on any instance of size n.The best case running time of an algorithm is the function defined by the minimum number of steps taken on any instance of size n. The average-case running time of an algorithm is the function defined by an average number of steps taken on any instance of size n. Which of these is the best to use?

Average case analysisDrawbacksBased on a probability distribution of input instancesHow do we know if distribution is correct or not?Usually more complicated to compute than worst case running timeOften worst case running time is comparable to average case running time(see next graph)Counterexamples to above: Quicksortsimplex method for linear programming

Best, Worst, and Average Case

Worst case analysisTypically much simpler to compute as we do not need to average performance on many inputsInstead, we need to find and understand an input that causes worst case performanceProvides guarantee that is independent of any assumptions about the inputOften reasonably close to average case running timeThe standard analysis performed

Motivation for Asymptotic AnalysisAn exact computation of worst-case running time can be difficult Function may have many terms: 4n2 - 3n log n + 17.5 n - 43 n + 75 An exact computation of worst-case running time is unnecessaryRemember that we are already approximating running time by using RAM model

SimplificationsIgnore constants4n2 - 3n log n + 17.5 n - 43 n + 75 becomesn2 n log n + n - n + 1Asymptotic Efficiencyn2 n log n + n - n + 1 becomes n2End Result: (n2)

Why ignore constants?RAM model introduces errors in constantsDo all instructions take equal time?Specific implementation (hardware, code optimizations) can speed up an algorithm by constant factorsWe want to understand how effective an algorithm is independent of these factorsSimplification of analysisMuch easier to analyze if we focus only on n2 rather than worrying about 3.7 n2 or 3.9 n2

Asymptotic AnalysisWe focus on the infinite set of large n ignoring small values of nUsually, an algorithm that is asymptotically more efficient will be the best choice for all but very small inputs.

0

infinity

Big Oh NotationO(g(n)) =

{f(n) : there exists positive constants c and n0 such that " nn0, 0 f(n) c g(n) }

What are the roles of the two constants?n0:c:

Set Notation CommentO(g(n)) is a set of functions.However, we will use one-way equalities like

n = O(n2)This really means that function n belongs to the set of functions O(n2)Incorrect notation: O(n2) = nAnalogyA dog is an animal but not an animal is a dog

Three Common Setsf(n) = O(g(n)) means c g(n) is an Upper Bound on f(n)

f(n) = (g(n)) means c g(n) is a Lower Bound on f(n)

f(n) = (g(n)) means c1 g(n) is an Upper Bound on f(n) and c2 g(n) is a Lower Bound on f(n)

These bounds hold for all inputs beyond some threshold n0.

Asymptotic or Big-O notation.

O Omega Sigma

O(g(n))

(g(n))

O(f(n)) and (g(n))

Example Functionf(n) = 3n2 - 100n + 6

Quick Questions c n0 3n2 - 100n + 6 = O(n2)3n2 - 100n + 6 = O(n3) 3n2 - 100n + 6 O(n)

3n2 - 100n + 6 = (n2)3n2 - 100n + 6 (n3)3n2 - 100n + 6 = (n)

3n2 - 100n + 6 = (n2)?3n2 - 100n + 6 = (n3)?3n2 - 100n + 6 = (n)?

[ Do on blackboard!!!!! ]

f(n) = 3n2 - 100n + 6

Little Oh Notationo(g(n)) =

{f(n) : "c >0 $n0 > 0 such that "n n0 0 f(n) < cg(n)}Intuitively, limn f(n)/g(n) = 0f(n) < c g(n)

Two Other Setsf(n) = o(g(n)) means c g(n) is a strict upper bound on f(n)

f(n) = w(g(n)) means c g(n) is a strict lower bound on f(n)

These bounds hold for all inputs beyond some threshold n0 where n0 is now dependent on c.


O Omega Sigma

Common Complexity FunctionsComplexity 10 20 30 40 50 60n 110-5 sec 210-5 sec 310-5 sec 410-5 sec 510-5 sec 610-5 secn2 0.0001 sec 0.0004 sec 0.0009 sec 0.016 sec 0.025 sec 0.036 secn3 0.001 sec 0.008 sec 0.027 sec 0.064 sec 0.125 sec 0.216 secn5 0.1 sec 3.2 sec 24.3 sec 1.7 min 5.2 min 13.0 min2n 0.001sec 1.0 sec 17.9 min 12.7 days 35.7 years 366 cent3n 0.59sec 58 min 6.5 years 3855 cent 2108cent 1.31013cent

log2 n310-6 sec 410-6 sec 510-6 sec 510-6 sec 610-6 sec 610-6 secn log2 n310-5 sec 910-5 sec 0.0001 sec 0.0002 sec 0.0003 sec 0.0004 sec

Example Problems1. What does it mean if:f(n) O(g(n)) and g(n) O(f(n)) ???

2. Is 2n+1 = O(2n) ? Is 22n = O(2n) ?

3. Does f(n) = O(f(n)) ?

4. If f(n) = O(g(n)) and g(n) = O(h(n)), can we say f(n) = O(h(n)) ?

Welcome to CSE 830, the Design and Theory of Algorithms.

My name is Charles, and I will be your instructor for this class. The goal of the class is to teach you the fundamentals of creating and analyzing computer algorithms.

Before we get to the actual material, however, I need to go through a few administrative details. While I take attendance, I have some handouts for you to be reading over.

In this class, most algorithms will be expressed in plain English with pseudo-code used to clear up any ambiguities.

Basically, any basic operation that takes a small, fixed amount of time we assume to take just one step.

We measure the run time of an algorithm by counting the number of steps it takes.

Why does this work? For the same reason that the Flat Earth model works. In our day-to-day lives we assume the Earth to be flat!

Now, lets look at how we can use this.


O Omega Sigma

[ Do on blackboard!!!!! ]

f(n) = 3n2 - 100n + 6Asymptotic or Big-O notation.

O Omega Sigma

Documents

Lecture 01236