DESIGN AND ANALYSIS OF ALGORITHMS (DAA 2019) · 2019-09-18 · ALGORITHMS (DAA 2019) ... Dynamic Programming Week 3 (Chapter 15 in book) 2 Design and Analysis of Algorithms 2019 week

Master’s Programme in Computer Science

Juha Kärkkäinen

Based on slides by Veli Mäkinen

DESIGN AND ANALYSIS OF ALGORITHMS (DAA 2019)

18/09/2019 1 Design and Analysis of Algorithms 2019 week 3

Dynamic Programming

Week 3 (Chapter 15 in book)

18/09/2019 Design and Analysis of Algorithms 2019 week 3 2


RECURRENCES WITH REPEATING SUBSTRUCTURES

• For divide-and-conquer algorithms, the time complexity is expressed as a recurrence

• Often the problem or a solution can be expressed as a recurrence

• Example: Fibonacci numbers are defined by the recurrence: F(n) = F(n-2) + F(n-1) and F(0)=F(1)=1

• The value of a recurrence can be easily computed by a simple recursive algorithm, but this can be slow

• Computing F(n) recursively takes O(F(n)) time because F(n-i) is computed F(i) times in different branches of the recursion F(n) = F(n-2)+F(n-1) = F(n-4)+F(n-3)+F(n-3)+F(n-2) = …



RECURRENCES WITH REPEATING SUBSTRUCTURES

• By storing values, we can avoid computing the same value many times

• In memoization, the recursive algorithm is modified to check if a value is already computed and stored:

• If it is, use the stored value and avoid a recursive call

• If not, make a recursive call to compute the value and then store it

• In dynamic programming, the evaluation order of the values is designed so that no recursion is needed

• Compute and store each value before it is needed in computing another value

• Example: Compute Fibonacci numbers from smallest to largest in O(n) time: F(0)=F(1)=1, F(2)=F(0)+F(1)=1+1=2, F(3)=F(1)+F(2)=1+2=3, …


Longest Increasing Subsequence

Example on typical steps for deriving a dynamic

programming algorithm



LONGEST INCREASING SUBSEQUENCE (LIS)

•



LIS EXAMPLE

A=1, 5, 3, 6, 4, 7

L=1, 2, 2, 3,

max

1<4 3<4

3

A=1, 5, 3, 6, 4, 7

L=1, 2, 2, 3, 3, 4

L[5]=? L[6]=?

+1 max

+1

A=1, 5, 3, 6, 4, 7

L=1, 2, 2, 3, 3, 4

Traceback all optimal solutions:

A=1, 5, 3, 6, 4, 7 L=1, 2, 2, 3, 3, 4

A=1, 5, 3, 6, 4, 7

L=1, 2, 2, 3, 3, 4

18/09/2019 Design and Analysis of Algorithms 2019 week 3


LEN, J, L = LIS(A[1..N])



A’= TRACEBACKLIS(A[1..N], L[1..N],J)

A’ = empty stack

A’.Push(A[j])

For i = j-1 downto 1 do

If A[i]<A[j] and L[i]+1=L[j] then

A’.Push(A[i])

j = i;

Return A’

• Popping from A’ produces a LIS of A in proper order

• This finds one LIS but not all of them



LIS: CORRECTNESS AND ANALYSIS

• Assume by induction that L[1..j-1] is correctly computed by the recurrence.

• Any subsequence ending with A[i], A[i]<A[j], can be extended with A[j] to obtain a subsequence of length L[i]+1. Thus L[j] ≥L[i]+1 for all 0≤i<j and L[j] ≥ max0≤i<j {L[i]+1 : A[i]<A[j]}.

• If L[j] > max0≤i<j {L[i]+1 : A[i]<A[j]}, there would be a subsequence ending with some A[i], 0<i<j, of length L[j]-1, contradicting the correctness of L[i].

• Hence the recurrence is correct. (Check the base case.)

• The time complexity is O(n2): O(n) values, O(n) time per value



LIS: OPTIMAL SUBPROBLEM PROPERTY

• Suppose we instead computed the table L’[1..n]:

• L’[j] = length of longest LIS in A[1..j] (ending anywhere)

• Now there is no fast way to compute L’[j] from L’[1..j-1] because we don’t know if the LIS corresponding to L’[i], i<j, can be extended with A[j].

• Suppose we computed the table L’’[1..n,min..max]:

• L’’[j,k] = length of longest LIS in A[1..j] ending with value ≤k

• Now we have enough information to compute L[j,*] from L[j-1,*].

• The tables L and L’’ have the optimal subproblem property:

• Every optimal subproblem solution is equivalent with respect to extending the solution.

• When this is not the case (L’) we can try to add parameters to subproblem (L’’).


Dependency DAG and shortest paths

Shortest path algorithms and dynamic programming


DYNAMIC PROGAMMING AS A DAG

• To determine a proper evaluation order for dynamic programming, we can represent the dependencies between the values as a graph

• Example: Fibonacci numbers

• The graph must be a directed acyclic graph (DAG) to avoid cyclic dependencies

• A proper evaluation order is a topological order in the DAG

• Optimization problems (e.g. LIS) can often be reduced to a shortest path problem in the graph


1 1 2 3 5 …


SHORTEST S-T PATH IN A DAG

Find shortest path from s to t in DAG G=(V,E)

• Base case: d(s) = 0

• Recurrence: d(v) = min{d(u)+c(u,v) : u is an in-neighbor of v}

• Output: d(t)

• Topological sort, compute each d(v) in |N-(v)| time, total O(|V|+|E|) time, where V and E are the set of vertices and edges of the DAG, respectively.

• Traceback to output an optimal path


s t

v u

…

N-(v)

… …


SHORTEST PATH IN A DAG (EXAMPLE)


Topological sort

5

10

12

5

10

21

10

5 4 5

12

3

11

1. Topological sort




Topological sort

5

10

12

5

10

21

10

5 4 5

12

3

11

0 10 15 19 24 26

1. Topological sort

2. Compute distances

(from left to right)




Topological sort

5

10

12

5

10

21

10

5 4 5

12

3

11

0 10 15 19 24 26

1. Topological sort

2. Compute distances

3. Traceback

(from right to left)




Topological sort

5

10

12

5

10

21

10

5 4 5

12

3

11

26


REDUCTION OF LIS TO SHORTEST PATH

… … …

i j

n arcs … c(i,j)=-1

0 n+1

n arcs



ANOTHER REDUCTION OF LIS TO SHORTEST PATH

… … …

i j

n arcs … c(i,j)=j-i-1

0 n+1

n arcs

Maximizing LIS length

= minimizing number of elements not in LIS


Segmentation problem

Another example of the power of dynamic programming



SEGMENTATION PROBLEM

• Clustering in 1D

• http://stackoverflow.com/questions/11513484/1d-number-array-clustering

• Clustering in ≥2D is NP-hard

• Clustering in 1D is known as the segmentation problem and can be solved efficiently by dynamic programming


http://stackoverflow.com/questions/11513484/1d-number-array-clustering








SEGMENTATION PROBLEM: MOTIVATION


A company owns

real estates

and the CEO

requests a map

with colour codes

showing

different property

price ranges


SEGMENTATION PROBLEM: MOTIVATION


1 000 000 – 3 000 000

500 000 – 750 000 200 000 – 400 000 75 000 – 150 000 30 000 – 50 000


SEGMENTATION: MOTIVATION 2

• Compression boosting:

• Partition a string A[1..n] into segments that compress better by tailored compressor for each segment.

‒ E.g. ababbbacdaccddbbbaabaab

• Let c(i,j) give the compression result for segment A[i..j].

• Then an optimal partitioning of A is one with the sum of its segment costs being minimum.



SEGMENTATION: DEFINITION

• Input: an array A[1..n], an integer k>0, and a cost function c(i,j) that assigns a cost to any segment (subarray) A[i..j]

• Output: a segmentation of A[1..n] into k segments A[1..n] = A[1..i1]A[i1+1..i2]…A[ik-1..n] so that the sum of the segment costs is minimized



SEGMENTATION: RECURRENCE

•



1D CLUSTERING: DEFINITION

k clusters definition?



1D CLUSTERING: L2 AND AVERAGE

segment= 1d cluster

k clusters

c1



1D CLUSTERING: L2 AND AVERAGE

0 4 5

AVG(0,4,5)=3

cL2(1..3)=(3-0)2+(3-4)2+(3-5)2=14

A[1..3]=0,4,5

Clustering with k=1



1D CLUSTERING: RECURRENCE

• devMin[j,k’]=”minimum total deviation after segmenting A[1..j] into k’ segments”

• devMin[j,k’]=mini<j devMin [i,k’-1]+cL2(i+1..j)

• minC sumdev(A,C) = devMin[n,k] (optimal solution)

• O(n3k) running time with direct implementation

• Optimizations, extensions, correctness, initialization, traceback in study groups & exercises.

k’-1 clusters

i i+1 j

+



1D CLUSTERING: EVALUATION ORDER

devMin[j,k’] = mini<j devMin [i,k’-1]+cL2(i+1..j)

Evaluation order:

k’

j

i

k’-1

…

Column-by-column (above), row-by-row, …

min (…) + cost of arc

18/09/2019 Design and Analysis of Algorithms 2019 week 3


1D CLUSTERING: RECAP+INITIALIZATION

k’

j

i

k’-1

mini<j devMin[..]

+ cost of arc

0 0

devMin[0..n,0..k]

optimum

solution:

traceback

following

arcs

A

n

0 k

…

k’-1

AVG

min cost to

segment A[1..i]

to k’-1 pieces

cost to add

segment

A[i+1..j]

min cost to

segment A[1..j]

to k’ pieces

0 segments only possible for empty sequence: d[i,0]=∞ except d[0,0]=0

more segments

than points:

devMin[i,k’]=∞

for i<k’

Optimal chain of matrix multiplications

Dynamic programming is not always a shortest path

problem



OPTIMAL CHAIN OF MATRIX MULTIPLICATIONS

x = n

m

m

p

n

p

O(npm) time

x n

m

m

p

x

q

p

=

n

q




x n

m

m

p

x

q

p

=

n

q

x n

m

m

p

x

q

p

=

n

q

npm+nqp

multiplications

n=10, m=15, p=7,

q=4:

1330 multiplications

mqp+nqm

multiplications

n=10, m=15, p=7,

q=4:

1470 multiplications




x =


the multiplication

performed last


OPTIMAL CHAIN: PSEUDOCODE



OPTIMAL CHAIN: DYNAMIC PROGRAMMING

• Dynamic programming instead of memoization:

• Compute M(i,j) in increasing order of interval length j-i+1

• Then M(i,k) and M(k+1,j) always computed before M(i,j)

• Subproblem dependencies form a DAG

• Each M(i,j) computed once in O(n) time: total O(n3) time

M(i,j) M(i,k)

M(k+1,j)



OPTIMAL CHAIN: NOT SHORTEST PATH PROBLEM

• Optimal M(i,j) depends on two nodes M(i,k) and M(k+1,j)

• Thus a solution is not a path but a tree


x

x x

x x A B

C D E F

((AxB)x((CxD)x(ExF)))

Documents

DESIGN AND ANALYSIS OF ALGORITHMS (DAA 2019) · 2019-09-18 · ALGORITHMS (DAA 2019) ... Dynamic Programming Week 3 (Chapter 15 in book) 2 Design and Analysis of Algorithms 2019 week