UMass Lowell Computer Science 91.404

Analysis of Algorithms Prof. Karen Daniels

Spring, 2001

UMass Lowell Computer Science 91.404

Analysis of Algorithms Prof. Karen Daniels

Spring, 2001

FinalFinal ReviewReview

Mon. 5/14-Wed. 5/16Mon. 5/14-Wed. 5/16

Overview of Next 2 LecturesOverview of Next 2 Lectures

Review of some key course materialReview of some key course material Final Exam: Final Exam:

Course GradeCourse Grade Logistics, Coverage, FormatLogistics, Coverage, Format Handout for basis of 40% of testHandout for basis of 40% of test

Course EvaluationsCourse Evaluations

Review of Key Course MaterialReview of Key Course Material

What’s It All About?What’s It All About?

AlgorithmAlgorithm:: steps for the computer to follow to steps for the computer to follow to

solve a problemsolve a problem Problem Solving GoalsProblem Solving Goals::

recognize structure of some common recognize structure of some common problemsproblems

understand important characteristics of understand important characteristics of algorithms to solve common problemsalgorithms to solve common problems

select appropriate algorithm & data select appropriate algorithm & data structures to solve a problemstructures to solve a problem

tailor existing algorithmstailor existing algorithms create new algorithmscreate new algorithms

Some Algorithm Application AreasSome Algorithm Application Areas

Computer GraphicsComputer Graphics

Geographic Geographic Information SystemsInformation Systems

RoboticsRobotics

BioinformaticsBioinformatics

AstrophysicsAstrophysicsMedical ImagingMedical Imaging

TelecommunicationsTelecommunications

Design

Apply

Analyze

Tools of the TradeTools of the Trade

Algorithm Design Patterns such as:Algorithm Design Patterns such as: binary searchbinary search divide-and-conquerdivide-and-conquer

Data Structures such as:Data Structures such as: trees, linked lists, hash tables, graphs trees, linked lists, hash tables, graphs

Theoretical Computer Science Theoretical Computer Science principles such as:principles such as: NP-completeness, hardnessNP-completeness, hardness

Growth of FunctionsGrowth of Functions

SummationsSummations

RecurrencesRecurrences

SetsSets

ProbabilityProbability

MATHMATHProofsProofs

Discrete Math Review Chapters 1-6

Discrete Math Review Chapters 1-6

Growth of Functions, Summations, Growth of Functions, Summations, Recurrences, Sets, Counting, ProbabilityRecurrences, Sets, Counting, Probability

TopicsTopics

Discrete Math Review Discrete Math Review : : Chapters 1-6Chapters 1-6 Solving Summations & RecurrencesSolving Summations & Recurrences Sets, Basic Tree & Graph conceptsSets, Basic Tree & Graph concepts Counting: Permutations/CombinationsCounting: Permutations/Combinations Probability: Basics, including Expectation of a Random VariableProbability: Basics, including Expectation of a Random Variable Proof Techniques: InductionProof Techniques: Induction

Basic Algorithm Analysis TechniquesBasic Algorithm Analysis Techniques: : Chapters 1-6Chapters 1-6 Asymptotic Growth of FunctionsAsymptotic Growth of Functions Types of Input: Best/Average/WorstTypes of Input: Best/Average/Worst Bounds on Algorithm vs. Bounds on ProblemBounds on Algorithm vs. Bounds on Problem Algorithmic Paradigms/Design Patterns: Divide-and-ConquerAlgorithmic Paradigms/Design Patterns: Divide-and-Conquer Analyze pseudocode running time to form summations &/or Analyze pseudocode running time to form summations &/or

recurrencesrecurrences

What are we measuring?What are we measuring?

Some Analysis Criteria:Some Analysis Criteria: ScopeScope

The problem itself? The problem itself? A particular algorithm that solves the problem?A particular algorithm that solves the problem?

““Dimension”Dimension” Time Complexity? Space Complexity?Time Complexity? Space Complexity?

Type of BoundType of Bound Upper? Lower? Both?Upper? Lower? Both?

Type of InputType of Input Best-Case? Average-Case? Worst-Case?Best-Case? Average-Case? Worst-Case?

Type of ImplementationType of Implementation Choice of Data StructureChoice of Data Structure

Function Order of GrowthFunction Order of Growth

O( ) upper boundO( ) upper bound

( ) lower bound( ) lower bound

( ) upper & lower bound( ) upper & lower bound

nn11 n lg(n) n lg(n) n lgn lg22(n) (n) 22nnnn55 lg(n)lg(n)lglg(n) lglg(n) nn22

know how to use asymptotic complexity notationknow how to use asymptotic complexity notation

to describe time or space complexityto describe time or space complexity

know how to order functions know how to order functions asymptoticallyasymptotically

(behavior as n becomes large)(behavior as n becomes large)

Types of Algorithmic InputTypes of Algorithmic Input

Best-Case InputBest-Case Input: of all possible algorithm inputs of size n, it : of all possible algorithm inputs of size n, it generates the “best” resultgenerates the “best” result

for Time Complexity: “best” is smallest running timefor Time Complexity: “best” is smallest running time

Best-Case Input Produces Best-Case Running TimeBest-Case Input Produces Best-Case Running Time provides a provides a lower bound on the algorithm’s asymptotic running timelower bound on the algorithm’s asymptotic running time

(subject to any implementation assumptions)(subject to any implementation assumptions)

for Space Complexity: “best” is smallest storagefor Space Complexity: “best” is smallest storage

Average-Case InputAverage-Case Input

Worst-Case InputWorst-Case Input

these are defined similarlythese are defined similarly

Best-Case Time <= Average-Case Time <= Worst-Case TimeBest-Case Time <= Average-Case Time <= Worst-Case Time

Bounding Algorithmic Time(using cases)

Bounding Algorithmic Time(using cases)

nn11 n lg(n) n lg(n) n lgn lg22(n) (n) 22nnnn55 lg(n)lg(n)lglg(n) lglg(n) nn22

T(n) = T(n) = (1)(1) T(n) = T(n) = (2(2nn))very loose bounds are not very useful!very loose bounds are not very useful!

Worst-CaseWorst-Case time of T(n) = time of T(n) = (2(2nn) tells us that worst-case inputs cause the algorithm to ) tells us that worst-case inputs cause the algorithm to take take at mostat most exponential time (i.e. exponential time is exponential time (i.e. exponential time is sufficientsufficient). ).

But, can the algorithm every really take exponential time? (i.e. is exponential timeBut, can the algorithm every really take exponential time? (i.e. is exponential time necessarynecessary?)?)

If, If, for arbitrary nfor arbitrary n, we find a worst-case input that forces the algorithm to use exponential , we find a worst-case input that forces the algorithm to use exponential time, then this time, then this tightenstightens the the lower boundlower bound on the on the worst-caseworst-case running time. If we can force running time. If we can force the lower and upper bounds on the worst-case time to match, then we can say that, for the lower and upper bounds on the worst-case time to match, then we can say that, for the worst-case running time, T(n) = the worst-case running time, T(n) = (2(2n n ) (i.e. we’ve found the ) (i.e. we’ve found the minimum upper boundminimum upper bound, , so the bound is so the bound is tight.) tight.)

Using “case” we can discuss lower and/or upper bounds on:Using “case” we can discuss lower and/or upper bounds on:best-case running timebest-case running time or or average-case running timeaverage-case running time or or worst-case running timeworst-case running time

Bounding Algorithmic Time(tightening bounds)

Bounding Algorithmic Time(tightening bounds)

nn11 n lg(n) n lg(n) n lgn lg22(n) (n) 22nnnn55 lg(n)lg(n)lglg(n) lglg(n) nn22

TTWW (n) = (n) = (2(2nn))

for example...for example...

Here we denote best-case time by THere we denote best-case time by TBB(n); worst-case time by T(n); worst-case time by TWW(n) (n)

TTBB(n) = (n) = (1)(1)1st attempt1st attempt 1st attempt1st attempt

TTBB (n) = (n) = (n)(n)

1st attempt1st attempt

2nd attempt2nd attempt

TTWW (n) = (n) = (n(n22))

TTBB(n) = (n) = (n)(n)2nd attempt2nd attempt

1st attempt1st attempt

TTWW(n) = (n) = (n(n22))

Algorithm BoundsAlgorithm Bounds

Explore the Explore the problem problem to gain intuition:to gain intuition: Describe it: What are the assumptions? Describe it: What are the assumptions? (model of computation, (model of computation,

etc...)etc...) Has it already been solved?Has it already been solved? Have similar problems been solved? (more on this later)Have similar problems been solved? (more on this later) What does best-case input look like?What does best-case input look like? What does worst-case input look like?What does worst-case input look like?

Establish worst-case Establish worst-case upper boundupper bound on the on the problemproblem using using an algorithman algorithm Design a (simple) algorithm and find an upper bound on its worst-case Design a (simple) algorithm and find an upper bound on its worst-case

asymptotic running time; this tells us problem asymptotic running time; this tells us problem can becan be solved in a certain solved in a certain amount of time. Algorithms taking more than this amount of time may amount of time. Algorithms taking more than this amount of time may exist, but won’t help us.exist, but won’t help us.

Establish worst-case Establish worst-case lower boundlower bound on the on the problemproblem Tighten each bound to form a worst-case “sandwich” Tighten each bound to form a worst-case “sandwich”

Approach

increasing worst-case asymptotic running time as a function of nincreasing worst-case asymptotic running time as a function of n

nn11 22nnnn22 nn33 nn44 nn55

Know the Difference!Know the Difference!

nn11 22nnnn55

worst-case boundsworst-case bounds on problemon problem

An inefficient algorithm for An inefficient algorithm for the problem might exist the problem might exist that takes this much time, that takes this much time, but would not help us.but would not help us.

No algorithm for the problem No algorithm for the problem exists that can solve it for exists that can solve it for worst-case inputs in less worst-case inputs in less than linear time .than linear time .

Strong BoundStrong Bound: This : This worst-case worst-case lowerlower bound bound on the problem holds for on the problem holds for everyevery algorithm that algorithm that solves the problem and solves the problem and abides by our problem’s abides by our problem’s assumptions.assumptions.

Weak BoundWeak Bound: This worst-case : This worst-case upperupper bound on the problem bound on the problem comes from just considering comes from just considering oneone algorithm. Other, less algorithm. Other, less efficient algorithms that solve efficient algorithms that solve this problem might exist, but this problem might exist, but we don’t care about them!we don’t care about them!

Both the upper and lower bounds Both the upper and lower bounds are probably are probably looseloose (i.e. probably (i.e. probably can be can be tightenedtightened later on). later on).

Master TheoremMaster Theorem

Master Theorem : Let with a > 1 and b > 1 .

Then : Case 1: If f(n) = O ( n (log

b a) - ) for some > o

 then T ( n ) = ( n log

b a )

  Case 2: If f (n) = (n log

b a )

 then T ( n ) = (n log

b a * log n )

  Case 3: If f ( n ) = (n (log

b a) + ) for some > o and if

a f( n/b) < c f ( n ) for some c < 1 , n > N0 then T ( n ) = ( f ( n ) )

)()()( nfb

naTnT

)()()( nfb

naTnT

Use ratio test to Use ratio test to distinguish distinguish between cases:between cases:

f(n)/ f(n)/ n log b

a

Look for Look for “polynomially “polynomially larger” dominance.larger” dominance.

SortingChapters 7-10

SortingChapters 7-10

Heapsort, Quicksort, LinearTime-Sorting, Heapsort, Quicksort, LinearTime-Sorting, MediansMedians

TopicsTopics

SortingSorting: : Chapters 7-10Chapters 7-10 Sorting Algorithms:Sorting Algorithms:

[Insertion & MergeSort from Chapters 1-6)], Heapsort, Quicksort, [Insertion & MergeSort from Chapters 1-6)], Heapsort, Quicksort, LinearTime-Sorting, MediansLinearTime-Sorting, Medians

Comparison-Based Sorting and its lower boundComparison-Based Sorting and its lower bound Breaking the lower bound using special assumptionsBreaking the lower bound using special assumptions Tradeoffs: Selecting an appropriate sort for a given situationTradeoffs: Selecting an appropriate sort for a given situation

Time vs. Space RequirementsTime vs. Space Requirements Comparison-Based vs. Non-Comparison-BasedComparison-Based vs. Non-Comparison-Based

Comparison-Based SortingComparison-Based Sorting

In algebraic decision tree model, comparison-based In algebraic decision tree model, comparison-based sorting of n items requiressorting of n items requires (n lg n) (n lg n) time.time.

HeapSortHeapSort

To breaking the lower bound and obtain linear To breaking the lower bound and obtain linear time, forego direct value comparisons and/or time, forego direct value comparisons and/or

make stronger assumptions about input. make stronger assumptions about input.

InsertionSortInsertionSort

MergeSortMergeSort

QuickSortQuickSort

(n lg n)(n lg n) (n(n22) )

BestCaseBestCase AverageCaseAverageCase WorstCaseWorstCaseTimeTime::AlgorithmAlgorithm::

(n lg n) (n lg n) (n (n lg n) lg n)

(n lg n) (n lg n) (n lg (n lg n) n)

(n lg n)(n lg n) (n lg n) (n lg n) (n(n22) )

Data StructuresChapters 11-14Data StructuresChapters 11-14

Stacks, Queues, LinkedLists, Trees, Stacks, Queues, LinkedLists, Trees, HashTables, Binary Search Trees, Balanced HashTables, Binary Search Trees, Balanced

TreesTrees

TopicsTopics

Data StructuresData Structures: : Chapters 11-14Chapters 11-14 Abstract Data Types: their properties/invariantsAbstract Data Types: their properties/invariants

Stacks, Queues, LinkedLists, (Heaps from Chapter 7), Trees, Stacks, Queues, LinkedLists, (Heaps from Chapter 7), Trees, HashTables, Binary Search Trees, Balanced (Red/Black) TreesHashTables, Binary Search Trees, Balanced (Red/Black) Trees

Implementation/Representation choices -> data structureImplementation/Representation choices -> data structure Dynamic Set Operations:Dynamic Set Operations:

Query [does not change the data structure]Query [does not change the data structure] Search, Minimum, Maximum, Predecessor, SuccessorSearch, Minimum, Maximum, Predecessor, Successor

Manipulate: [can change data structure]Manipulate: [can change data structure] Insert, DeleteInsert, Delete

Running Time & Space Requirements for Dynamic Set Running Time & Space Requirements for Dynamic Set Operations for each Data Structure Operations for each Data Structure

Tradeoffs: Selecting an appropriate data structure for a situationTradeoffs: Selecting an appropriate data structure for a situation Time vs. Space RequirementsTime vs. Space Requirements Representation choicesRepresentation choices Which operations are crucial?Which operations are crucial?

Advanced TechniquesChapters 16-17

Advanced TechniquesChapters 16-17

Dynamic Programming, Greedy Dynamic Programming, Greedy AlgorithmsAlgorithms

TopicsTopics

Advanced TechniquesAdvanced Techniques: : Chapters 16-17Chapters 16-17 Algorithmic Paradigms/Design Patterns: Algorithmic Paradigms/Design Patterns:

Divide-and-ConquerDivide-and-Conquer Dynamic ProgrammingDynamic Programming Greedy AlgorithmsGreedy Algorithms Brute-Force/NaiveBrute-Force/Naive

Using Dynamic Programming &/or Greedy Algorithms to solve Using Dynamic Programming &/or Greedy Algorithms to solve Optimization ProblemsOptimization Problems

Optimal SubstructureOptimal Substructure Greedy Choice Property: Locally optimal -> Globally optimalGreedy Choice Property: Locally optimal -> Globally optimal

Tradeoffs: Tradeoffs: Selecting an appropriate paradigm to solve a problemSelecting an appropriate paradigm to solve a problem Tackling a problem using a sequence of paradigms:Tackling a problem using a sequence of paradigms:

Brute-Force (high running time) then improve...Brute-Force (high running time) then improve...

Problem CharacteristicsProblem Characteristics

ModularModular

Independent piecesIndependent pieces

Divide-and-ConquerDivide-and-Conquer Dynamic ProgrammingDynamic Programming Greedy AlgorithmsGreedy Algorithms

ModularModular

OptimizationOptimization

Optimal substructure: Optimal substructure: optimal solution contains optimal solution contains optimal solutions to optimal solutions to subproblemssubproblems

Overlapping subproblemsOverlapping subproblems

ModularModular

OptimizationOptimization

Optimal substructure: Optimal substructure: optimal solution optimal solution contains optimal contains optimal solutions to solutions to subproblemssubproblems

Greedy choice property: Greedy choice property: locally optimal choices locally optimal choices

lead to globallead to global optimumoptimum

Graph AlgorithmsChapters 23-25

Graph AlgorithmsChapters 23-25

DFS/BFSTraversals, Topological Sort, DFS/BFSTraversals, Topological Sort,

MinimumSpanningTrees, Shortest PathsMinimumSpanningTrees, Shortest Paths

TopicsTopics

Graph AlgorithmsGraph Algorithms: : Chapters 23-25Chapters 23-25 Undirected, Directed GraphsUndirected, Directed Graphs Connected Components of an Undirected GraphConnected Components of an Undirected Graph Representations: Adjacency Matrix, Adjacency ListRepresentations: Adjacency Matrix, Adjacency List Traversals: DFS and BFSTraversals: DFS and BFS

Differences in approach: DFS: LIFO/stack vs. BFS:FIFO/queueDifferences in approach: DFS: LIFO/stack vs. BFS:FIFO/queue Forest of spanning treesForest of spanning trees Vertex coloring, Edge classification: tree, back, forward, crossVertex coloring, Edge classification: tree, back, forward, cross Shortest paths (BFS)Shortest paths (BFS)

Topological SortTopological Sort Weighted GraphsWeighted Graphs MinimumSpanningTrees: 2 different approachesMinimumSpanningTrees: 2 different approaches Shortest Paths: Single source: Dijkstra’s algorithmShortest Paths: Single source: Dijkstra’s algorithm Tradeoffs:Tradeoffs:

Representation Choice: Adjacency Matrix vs. Adjacency ListRepresentation Choice: Adjacency Matrix vs. Adjacency List Traversal Choice: DFS or BFSTraversal Choice: DFS or BFS

Traversals: DFS, BFSTraversals: DFS, BFS

DFS backtracks DFS backtracks visit most recently visit most recently discovered vertex discovered vertex LIFO structure LIFO structure stack data structurestack data structure

BFS BFS vertices close to v are visited before vertices close to v are visited before those further away those further away FIFO structure FIFO structure queue data structurequeue data structure

FINAL EXAMFINAL EXAM

Logistics, Coverage, FormatLogistics, Coverage, Format

Handout for basis of 40% of testHandout for basis of 40% of test

Course GradingCourse Grading

Homework 40%Homework 40% Exam 1 15% (closed book)Exam 1 15% (closed book) Midterm 20% (open book)Midterm 20% (open book) Final Exam 25% (open book)Final Exam 25% (open book)

Results are scaled if necessary.

Check grade status with us before final!

Final Exam: LogisticsFinal Exam: Logistics

Friday, 12/18Friday, 12/18 Olsen 311: 8:00-11:00 a.m.Olsen 311: 8:00-11:00 a.m. Open book, open notesOpen book, open notes Closed computers, neighborsClosed computers, neighbors CumulativeCumulative Worth 25% of gradeWorth 25% of grade

Note change Note change from registrar’s from registrar’s room numberroom number

Text/Chapter/Topic CoverageText/Chapter/Topic Coverage

Discrete Math Review & Basic Algorithm Analysis Discrete Math Review & Basic Algorithm Analysis Techniques Techniques : : Chapters 1-6Chapters 1-6 Summations, Recurrences, Sets, Trees, Graph, Counting, ProbabilitySummations, Recurrences, Sets, Trees, Graph, Counting, Probability, ,

Growth of Functions, Divide-and-ConquerGrowth of Functions, Divide-and-Conquer SortingSorting: : Chapters 7-10Chapters 7-10

Heapsort, Quicksort, LinearTime-Sorting, MediansHeapsort, Quicksort, LinearTime-Sorting, Medians Data StructuresData Structures: : Chapters 11-14Chapters 11-14

Stacks, Queues, LinkedLists, Trees, HashTables, Binary Search Trees, Stacks, Queues, LinkedLists, Trees, HashTables, Binary Search Trees, Balanced (Red/Black) TreesBalanced (Red/Black) Trees

Advanced TechniquesAdvanced Techniques: : Chapters 16-17Chapters 16-17 Dynamic Programming, Greedy AlgorithmsDynamic Programming, Greedy Algorithms

Graph AlgorithmsGraph Algorithms: : Chapters 23-25Chapters 23-25 Traversal, MinimumSpanningTrees, Shortest PathsTraversal, MinimumSpanningTrees, Shortest Paths

FormatFormat

Mixture of questions of the following types:Mixture of questions of the following types:1) Multiple Choice 1) Multiple Choice

2) True/False2) True/False

3) Short Answer3) Short Answer

4) Analyze Pseudo-Code and/or Data Structure 4) Analyze Pseudo-Code and/or Data Structure

5) Solve a Problem by Designing an Algorithm5) Solve a Problem by Designing an Algorithm Select an appropriate paradigm/ design patternSelect an appropriate paradigm/ design pattern Select appropriate data structuresSelect appropriate data structures Write pseudo-codeWrite pseudo-code Justify correctnessJustify correctness Analyze asymptotic complexityAnalyze asymptotic complexity

60%60%

40%40%