UMass Lowell Computer Science 91.503 Analysis of ...kdaniels/courses/ALG_503_S10/503_lecture1c_S10.pdfUMass Lowell Computer Science 91.503 Analysis of AlgorithmsAnalysis of Algorithms

UMass Lowell Computer Science 91.503Analysis of Algorithms

UMass Lowell Computer Science 91.503Analysis of AlgorithmsAnalysis of Algorithms

Prof. Karen DanielsSpring 2010

Analysis of AlgorithmsProf. Karen Daniels

Spring 2010Spring, 2010Spring, 2010

Design PatternsDesign Patterns forforDesign Patterns Design Patterns for for Optimization ProblemsOptimization Problems

D namicD namic ProgrammingProgrammingDynamic Dynamic ProgrammingProgrammingMatrix ParenthesizingMatrix Parenthesizing

Longest Common SubsequenceLongest Common SubsequenceLongest Common SubsequenceLongest Common SubsequenceActivity SelectionActivity Selection

Algorithmic Paradigm ContextAlgorithmic Paradigm Context

Divide & Conquer

Dynamic Programming

Greedy AlgorithmConquer Programming Algorithm

View problem as collection of subproblems

“Recursive” nature Independent subproblems overlapping typically

sequential qdependence

Number of subproblems depends on partitioning

factors

typically small

Preprocessing typically sort Characteristic running time typically log depends on number often dominatedCharacteristic running time typically log

function of n depends on number and difficulty of subproblems

often dominated by nlogn sort

Primarily for optimization problems (find an optimal solution)

solution) Optimal substructure: optimal solution to problem contains within it optimal solutions to subproblems

Greedy choice property: y p p ylocally optimal produces globally optimal Heuristic version useful for bounding optimal value

Subproblem solution orderSubproblem solution orderMake choice, then solve Make choice, then solve subproblem(s)subproblem(s)

Solve subproblem(s), Solve subproblem(s), then make choicethen make choice

Dynamic Programming Approach to Optimization ProblemsDynamic Programming Approach to Optimization Problemsto Optimization Problemsto Optimization Problems

1.1. Characterize structure of an optimal Characterize structure of an optimal solution.solution.

2.2. Recursively define value of an optimal Recursively define value of an optimal solution.solution.

3.3. Compute value of an optimal solution in Compute value of an optimal solution in bottombottom--up fashion.up fashion.

4.4. Construct an optimal solution from Construct an optimal solution from computed information.computed information.

source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.

Dynamic ProgrammingDynamic Programming

Matrix ParenthesizationMatrix Parenthesization

Example: Matrix Parenthesization D fi i i

Example: Matrix Parenthesization D fi i iDefinitionsDefinitions

Given “chain” of n matrices: <AGiven “chain” of n matrices: <A AA AA >>Given chain of n matrices: <AGiven chain of n matrices: <A11, A, A22, … A, … Ann, >, >

Compute product ACompute product A11AA22… A… Ann efficientlyefficiently

Minimize “cost” = number of scalar multiplicationsMinimize “cost” = number of scalar multiplications

Multiplication order matters!Multiplication order matters!u p c o o de e s!u p c o o de e s!


Example: Matrix Parenthesization Step 1: Characterizing an Optimal SolutionExample: Matrix Parenthesization Step 1: Characterizing an Optimal SolutionStep 1: Characterizing an Optimal SolutionStep 1: Characterizing an Optimal Solution

Observation: Observation: Any parenthesization of Any parenthesization of AAiiAAi+1i+1… A… Ajj must split it between must split it between AAk k andand AAk+1 k+1 for some k.for some k.jj

THM: Optimal Matrix Parenthesization:THM: Optimal Matrix Parenthesization:If an optimal parenthesization of If an optimal parenthesization of AAiiAAi+1i+1… A… Ajj splits at k, then splits at k, then parenthesi ation of prefiparenthesi ation of prefi AA AA AA m st be an optimal parenthesi ationm st be an optimal parenthesi ationparenthesization of prefix parenthesization of prefix AAiiAAi+1i+1… A… Akk must be an optimal parenthesization.must be an optimal parenthesization.

Why?Why?If existed less costly way to parenthesize prefix, then substituting thatIf existed less costly way to parenthesize prefix, then substituting thatIf existed less costly way to parenthesize prefix, then substituting that If existed less costly way to parenthesize prefix, then substituting that

parenthesization would yield less costly way to parenthesize parenthesization would yield less costly way to parenthesize AAiiAAi+1i+1… A… Ajj , , contradicting optimality of that parenthesization. contradicting optimality of that parenthesization.


common DP proof technique: common DP proof technique: “cut“cut--andand--paste” proof by contradictionpaste” proof by contradiction

Example: Matrix Parenthesization Step 2: A Recursive SolutionExample: Matrix Parenthesization Step 2: A Recursive SolutionStep 2: A Recursive SolutionStep 2: A Recursive Solution

Recursive definition of minimum parenthesization Recursive definition of minimum parenthesization ppcost:cost:

0 if i = jm[i,j]= min{m[i,k] + m[k+1,j] + pi-1pkpj} if i < j

0 if i j

i <= k < j

How many distinct subproblems?How many distinct subproblems?

each matrix Ai has dimensions pi-1 x pi


Example: Matrix Parenthesization Step 3: Computing Optimal CostsExample: Matrix Parenthesization Step 3: Computing Optimal CostsStep 3: Computing Optimal CostsStep 3: Computing Optimal Costs

2 6252 625

2,5002,500

1 0001 00000

2,6252,625 1,0001,000

s: value of k that achieves optimal s: value of k that achieves optimal cost in computing m[i, j]cost in computing m[i, j]


Example: Matrix Parenthesization Step 4: Constructing an Optimal SolutionExample: Matrix Parenthesization Step 4: Constructing an Optimal SolutionStep 4: Constructing an Optimal SolutionStep 4: Constructing an Optimal Solution

PRINTPRINT--OPTIMALOPTIMAL--PARENS(s, i, j)PARENS(s, i, j)if i = jif i = j

then print “A”then print “A”iielse print “(“else print “(“

PRINTPRINT--OPTIMALOPTIMAL--PARENS(s, i, s[i, j])PARENS(s, i, s[i, j])PRINTPRINT--OPTIMALOPTIMAL--PARENS(s, s[i, j]+1, j)PARENS(s, s[i, j]+1, j)print “)“print “)“


Example: Matrix Parenthesization Memoization

Example: Matrix Parenthesization MemoizationMemoizationMemoization

Provide Dynamic Programming Provide Dynamic Programming source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.y g gy g gefficiencyefficiencyButBut with with toptop--downdown strategystrategy

Use recursionUse recursion LOOKUPLOOKUP--CHAIN(p,i,j) CHAIN(p,i,j) Use recursionUse recursionFill in table “on demand”Fill in table “on demand”

Example: Example:

(p j)(p j)1 1 ifif m[i,j] < m[i,j] < 2 2 then return then return m[i,j]m[i,j]3 3 ifif i=j i=j

∞

RECURSIVERECURSIVE--MATRIXMATRIX--CHAIN:CHAIN:

MEMOIZEDMEMOIZED--MATRIXMATRIX--CHAIN(p)CHAIN(p)1 l h[ ]1 l h[ ] 11

jj44 then then m[i,j] 0m[i,j] 05 5 elseelse forfor k i to jk i to j--1166 dodo q LOOKUPq LOOKUP--CHAIN(p i k)CHAIN(p i k)1 n length[p] 1 n length[p] -- 11

2 2 forfor i 1 i 1 toto nn3 3 dodo for j i for j i toto nn

6 6 dodo q LOOKUPq LOOKUP--CHAIN(p,i,k)CHAIN(p,i,k)+ LOOKUP+ LOOKUP--CHAIN(p,k+1,j)CHAIN(p,k+1,j)+ p+ pii--1 1 ppk k ppjj

77 ifif [i j][i j]44 dodo m[i,j] m[i,j] 5 return LOOKUP5 return LOOKUP--CHAIN(p,1,n)CHAIN(p,1,n)

∞ 77 ifif q < m[i,j] q < m[i,j] 88 thenthen m[i,j] qm[i,j] q9 9 returnreturn m[i,j] m[i,j]

Dynamic ProgrammingDynamic Programming

Longest Common SubsequenceLongest Common Subsequence

Example: Longest Common Subsequence (LCS): MotivationExample: Longest Common Subsequence (LCS): MotivationSubsequence (LCS): MotivationSubsequence (LCS): Motivation

Strand of DNA: string over finite set {A,C,G,T}Strand of DNA: string over finite set {A,C,G,T}each element of set is a base: each element of set is a base: adenine, guanine, cytosine or thymineadenine, guanine, cytosine or thymine

Compare DNA similaritiesCompare DNA similaritiesSS1 1 = = ACCACCGGGGTCGTCGAGAGTTGGCGCGCCGGGGAAGCCGGCCGAAAAGCCGGCCGAA

SS2 2 = = GTCGTGTCGTTTCGGAACGGAATTGCCGGCCGTTTTGCGCTTCCTTGGTTAAAAAAOne measure of similarity:One measure of similarity:yy

find the longest string Sfind the longest string S33 containing bases that also appear (not containing bases that also appear (not necessarily necessarily consecutivelyconsecutively) in S) in S11 and Sand S22

SS3 3 = = GTCGTCGGAAGCCGGCCGAAGTCGTCGGAAGCCGGCCGAA


Example: LCS Definitions

Example: LCS DefinitionsDefinitionsDefinitions

SequenceSequence is a is a subsequencesubsequence of of source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.

qq qqif if (strictly increasing (strictly increasing

indices of X)indices of X) such thatsuch thatexample: example: is subsequence of is subsequence of

with index sequencewith index sequence

Z i b f X d Y if Z iZ i b f X d Y if Z iZ is common subsequence of X and Y if Z is Z is common subsequence of X and Y if Z is subsequence of both X and Ysubsequence of both X and Y

example:example:example:example:common subsequence but not common subsequence but not longestlongestcommon subsequence. Longest?common subsequence. Longest?

Longest Common Subsequence ProblemLongest Common Subsequence Problem: Given 2 sequences : Given 2 sequences X, Y, find maximumX, Y, find maximum--length common subsequence Z.length common subsequence Z.

Example: LCS Step 1: Characterize an LCS

Example: LCS Step 1: Characterize an LCSStep 1: Characterize an LCSStep 1: Characterize an LCS

THM 15.1: Optimal LCS Substructure THM 15.1: Optimal LCS Substructure 5 Opt a CS Subst uctu e5 Opt a CS Subst uctu eGiven sequences:Given sequences:For any LCSFor any LCS of X and Y: of X and Y:

1 1 if if thenthen and Zand Zkk--11 is an LCS of Xis an LCS of Xmm--11 and Yand Ynn--11

2 2 if if thenthen Z is an LCS of XZ is an LCS of Xmm--11 and Y and Y 3 if3 if thenthen Z is an LCS of X and YZ is an LCS of X and Ynn 113 if 3 if thenthen Z is an LCS of X and YZ is an LCS of X and Ynn--11

PROOF: based on producing PROOF: based on producing contradictionscontradictions

1 a) 1 a) Suppose . Appending to Z contradicts Suppose . Appending to Z contradicts longest longest nature of Z.nature of Z.b) To establish b) To establish longest longest nature of Znature of Zkk--11, suppose common subsequence W of , suppose common subsequence W of XXmm--11 and and YYnn--11

has length > khas length > k--1. Appending to W yields common subsequence of length > k = contradiction.1. Appending to W yields common subsequence of length > k = contradiction.

22 C b W fC b W f XX dd YY f l th k ld l b bf l th k ld l b b2 2 Common subsequence W of Common subsequence W of XXmm--11 and and Y Y of length > k would also be common subsequence of length > k would also be common subsequence of of XXmm, Y, contradicting , Y, contradicting longest longest nature of Z.nature of Z.

3 3 Similar to proof of (2)Similar to proof of (2)source: 91.503 textbook Cormen, et al.source: 91.503 textbook Cormen, et al.

Example: LCS Step 2: A Recursive Solution

Example: LCS Step 2: A Recursive SolutionStep 2: A Recursive SolutionStep 2: A Recursive Solution

Implications of Theorem 15 1:Implications of Theorem 15 1:Implications of Theorem 15.1:Implications of Theorem 15.1:

?? nono??yesyes nono

Fi d LCS(X Y )Find LCS(Xm-1, Yn-1) Find LCS(Xm-1, Y) Find LCS(X, Yn-1)

LCS1(X, Y) = LCS(Xm-1, Yn-1) + xm LCS2(X, Y) = max(LCS(Xm-1, Y), LCS(X, Yn-1))

Example: LCS Step 2: A Recursive Solution (continued)

Example: LCS Step 2: A Recursive Solution (continued)Step 2: A Recursive Solution (continued)Step 2: A Recursive Solution (continued)

Overlapping subproblem structure:Overlapping subproblem structure:source: 91.503 textbook Cormen, et asource: 91.503 textbook Cormen, et app g ppp g p),(),( 111 YXLCSYXLCS mnm −−− ⊆

)()( YXLCSYXLCS),( YXLCS⊆

),(),( 111 −−− ⊆ nnm YXLCSYXLCSΘΘ(mn) distinct (mn) distinct subproblemssubproblems

Recurrence for length of optimal solution:Recurrence for length of optimal solution:0 if i=0 or j=0

c[i,j]= c[i-1,j-1]+1 if i,j > 0 and xi=yj

max(c[i,j-1], c[i-1,j]) if i,j > 0 and xi=yj

0 if i=0 or j=0

Conditions of problem can exclude some subproblems!

max(c[i,j 1], c[i 1,j]) if i,j > 0 and xi yj

Example: LCS Step 3: Compute Length of an LCSExample: LCS Step 3: Compute Length of an LCSStep 3: Compute Length of an LCSStep 3: Compute Length of an LCS

BB CC BB AA

What is the What is the asymptotic worstasymptotic worst--

case time case time complexity?complexity?

BB

CC

0

12

complexity?complexity?

CC

BB

AA

2

3

c tablec table

4


(represent b table)(represent b table)

Example: LCS Step 4: Construct an LCSExample: LCS Step 4: Construct an LCSStep 4: Construct an LCSStep 4: Construct an LCS


Dynamic ProgrammingDynamic Programmingl di t G d Al ithl di t G d Al ith…leading to a Greedy Algorithm……leading to a Greedy Algorithm…

Activity SelectionActivity Selection

Activity SelectionOptimization ProblemActivity SelectionOptimization ProblemOptimization Problem Optimization Problem

Problem Instance:Problem Instance:Problem Instance:Problem Instance:Set Set SS = {1,2,...,= {1,2,...,nn} of } of nn activitiesactivitiesEach activityEach activity ii has:has:Each activity Each activity ii has:has:

start time: start time: ssii

finish time:finish time: ff ii fs ≤finish time: finish time: ffii

Activities Activities i, ji, j are compatible iff nonare compatible iff non--overlapping: overlapping: )[ fs ≤ )[ fs ≤

Objective:Objective:select aselect a maximummaximum--sizedsized set of mutually compatible activitiesset of mutually compatible activities

)[ ii fs ≤ )[ jj fs ≤

select a select a maximummaximum--sizedsized set of mutually compatible activities set of mutually compatible activities



Activity Time DurationActivity Time Duration11 22 33 44 55 66 77 88 99 1010 12121111 1313 1414 1515 1616

11

Activity Time DurationActivity Time Duration

Activity Activity NumberNumber

22

33

4444

66

55

88

77


}{ ffSS ≤<≤ }:{ jkkikij sfsfSaS ≤<≤∈=Solution to Solution to SSijij including including aakk produces 2 subproblems:produces 2 subproblems:1)1) SS (start after(start after aa finishes; finish beforefinishes; finish before aa starts)starts)1) 1) SSikik (start after (start after aaii finishes; finish before finishes; finish before aakk starts)starts)2) 2) SSkjkj (start after (start after aakk finishes; finish before finishes; finish before aajj starts) starts)

cc[[i ji j] size of maximum] size of maximum size subset ofsize subset of

⎫⎧ /if

cc[[i,ji,j]=size of maximum]=size of maximum--size subset of size subset of mutually compatible activities in mutually compatible activities in SSijij..

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

/≠++/=

=∈<< 0 if}1],[],[{max

0if0],[

; ijSajki

ij

SjkckicS

jicijk


Documents

UMass Lowell Computer Science 91.503 Analysis of ...kdaniels/courses/ALG_503_S10/503_lecture1c_S10.pdfUMass Lowell Computer Science 91.503 Analysis of AlgorithmsAnalysis of Algorithms