Master’s Programme in Computer Science
Juha Kärkkäinen
Based on slides by Veli Mäkinen
DESIGN AND ANALYSIS OF ALGORITHMS (DAA 2019)
18/09/2019 1 Design and Analysis of Algorithms 2019 week 3
Dynamic Programming
Week 3 (Chapter 15 in book)
18/09/2019 Design and Analysis of Algorithms 2019 week 3 2
Master’s Programme in Computer Science
RECURRENCES WITH REPEATING SUBSTRUCTURES
• For divide-and-conquer algorithms, the time complexity is expressed as a recurrence
• Often the problem or a solution can be expressed as a recurrence
• Example: Fibonacci numbers are defined by the recurrence: F(n) = F(n-2) + F(n-1) and F(0)=F(1)=1
• The value of a recurrence can be easily computed by a simple recursive algorithm, but this can be slow
• Computing F(n) recursively takes O(F(n)) time because F(n-i) is computed F(i) times in different branches of the recursion F(n) = F(n-2)+F(n-1) = F(n-4)+F(n-3)+F(n-3)+F(n-2) = …
18/09/2019 Design and Analysis of Algorithms 2019 week 3 3
Master’s Programme in Computer Science
RECURRENCES WITH REPEATING SUBSTRUCTURES
• By storing values, we can avoid computing the same value many times
• In memoization, the recursive algorithm is modified to check if a value is already computed and stored:
• If it is, use the stored value and avoid a recursive call
• If not, make a recursive call to compute the value and then store it
• In dynamic programming, the evaluation order of the values is designed so that no recursion is needed
• Compute and store each value before it is needed in computing another value
• Example: Compute Fibonacci numbers from smallest to largest in O(n) time: F(0)=F(1)=1, F(2)=F(0)+F(1)=1+1=2, F(3)=F(1)+F(2)=1+2=3, …
18/09/2019 Design and Analysis of Algorithms 2019 week 3 4
Longest Increasing Subsequence
Example on typical steps for deriving a dynamic
programming algorithm
18/09/2019 Design and Analysis of Algorithms 2019 week 3 5
Master’s Programme in Computer Science
LONGEST INCREASING SUBSEQUENCE (LIS)
•
18/09/2019 Design and Analysis of Algorithms 2019 week 3 6
Master’s Programme in Computer Science
LIS EXAMPLE
A=1, 5, 3, 6, 4, 7
L=1, 2, 2, 3,
max
1<4 3<4
3
A=1, 5, 3, 6, 4, 7
L=1, 2, 2, 3, 3, 4
L[5]=? L[6]=?
+1 max
+1
A=1, 5, 3, 6, 4, 7
L=1, 2, 2, 3, 3, 4
Traceback all optimal solutions:
A=1, 5, 3, 6, 4, 7 L=1, 2, 2, 3, 3, 4
A=1, 5, 3, 6, 4, 7
L=1, 2, 2, 3, 3, 4
18/09/2019 Design and Analysis of Algorithms 2019 week 3
Master’s Programme in Computer Science
LEN, J, L = LIS(A[1..N])
18/09/2019 Design and Analysis of Algorithms 2019 week 3 8
Master’s Programme in Computer Science
A’= TRACEBACKLIS(A[1..N], L[1..N],J)
A’ = empty stack
A’.Push(A[j])
For i = j-1 downto 1 do
If A[i]<A[j] and L[i]+1=L[j] then
A’.Push(A[i])
j = i;
Return A’
• Popping from A’ produces a LIS of A in proper order
• This finds one LIS but not all of them
18/09/2019 Design and Analysis of Algorithms 2019 week 3 9
Master’s Programme in Computer Science
LIS: CORRECTNESS AND ANALYSIS
• Assume by induction that L[1..j-1] is correctly computed by the recurrence.
• Any subsequence ending with A[i], A[i]<A[j], can be extended with A[j] to obtain a subsequence of length L[i]+1. Thus L[j] ≥L[i]+1 for all 0≤i<j and L[j] ≥ max0≤i<j {L[i]+1 : A[i]<A[j]}.
• If L[j] > max0≤i<j {L[i]+1 : A[i]<A[j]}, there would be a subsequence ending with some A[i], 0<i<j, of length L[j]-1, contradicting the correctness of L[i].
• Hence the recurrence is correct. (Check the base case.)
• The time complexity is O(n2): O(n) values, O(n) time per value
18/09/2019 Design and Analysis of Algorithms 2019 week 3 10
Master’s Programme in Computer Science
LIS: OPTIMAL SUBPROBLEM PROPERTY
• Suppose we instead computed the table L’[1..n]:
• L’[j] = length of longest LIS in A[1..j] (ending anywhere)
• Now there is no fast way to compute L’[j] from L’[1..j-1] because we don’t know if the LIS corresponding to L’[i], i<j, can be extended with A[j].
• Suppose we computed the table L’’[1..n,min..max]:
• L’’[j,k] = length of longest LIS in A[1..j] ending with value ≤k
• Now we have enough information to compute L[j,*] from L[j-1,*].
• The tables L and L’’ have the optimal subproblem property:
• Every optimal subproblem solution is equivalent with respect to extending the solution.
• When this is not the case (L’) we can try to add parameters to subproblem (L’’).
18/09/2019 Design and Analysis of Algorithms 2019 week 3 11
Dependency DAG and shortest paths
Shortest path algorithms and dynamic programming
Master’s Programme in Computer Science
DYNAMIC PROGAMMING AS A DAG
• To determine a proper evaluation order for dynamic programming, we can represent the dependencies between the values as a graph
• Example: Fibonacci numbers
• The graph must be a directed acyclic graph (DAG) to avoid cyclic dependencies
• A proper evaluation order is a topological order in the DAG
• Optimization problems (e.g. LIS) can often be reduced to a shortest path problem in the graph
18/09/2019 Design and Analysis of Algorithms 2019 week 3 13
1 1 2 3 5 …
Master’s Programme in Computer Science
SHORTEST S-T PATH IN A DAG
Find shortest path from s to t in DAG G=(V,E)
• Base case: d(s) = 0
• Recurrence: d(v) = min{d(u)+c(u,v) : u is an in-neighbor of v}
• Output: d(t)
• Topological sort, compute each d(v) in |N-(v)| time, total O(|V|+|E|) time, where V and E are the set of vertices and edges of the DAG, respectively.
• Traceback to output an optimal path
18/09/2019 Design and Analysis of Algorithms 2019 week 3 14
s t
v u
…
N-(v)
… …
Master’s Programme in Computer Science
SHORTEST PATH IN A DAG (EXAMPLE)
18/09/2019 Design and Analysis of Algorithms 2019 week 3 15
Topological sort
5
10
12
5
10
21
10
5 4 5
12
3
11
1. Topological sort
Master’s Programme in Computer Science
SHORTEST PATH IN A DAG (EXAMPLE)
18/09/2019 Design and Analysis of Algorithms 2019 week 3 16
Topological sort
5
10
12
5
10
21
10
5 4 5
12
3
11
0 10 15 19 24 26
1. Topological sort
2. Compute distances
(from left to right)
Master’s Programme in Computer Science
SHORTEST PATH IN A DAG (EXAMPLE)
18/09/2019 Design and Analysis of Algorithms 2019 week 3 17
Topological sort
5
10
12
5
10
21
10
5 4 5
12
3
11
0 10 15 19 24 26
1. Topological sort
2. Compute distances
3. Traceback
(from right to left)
Master’s Programme in Computer Science
SHORTEST PATH IN A DAG (EXAMPLE)
18/09/2019 Design and Analysis of Algorithms 2019 week 3 18
Topological sort
5
10
12
5
10
21
10
5 4 5
12
3
11
26
Master’s Programme in Computer Science
REDUCTION OF LIS TO SHORTEST PATH
… … …
i j
n arcs … c(i,j)=-1
0 n+1
n arcs
18/09/2019 Design and Analysis of Algorithms 2019 week 3 19
Master’s Programme in Computer Science
ANOTHER REDUCTION OF LIS TO SHORTEST PATH
… … …
i j
n arcs … c(i,j)=j-i-1
0 n+1
n arcs
Maximizing LIS length
= minimizing number of elements not in LIS
18/09/2019 Design and Analysis of Algorithms 2019 week 3 20
Segmentation problem
Another example of the power of dynamic programming
18/09/2019 Design and Analysis of Algorithms 2019 week 3 21
Master’s Programme in Computer Science
SEGMENTATION PROBLEM
• Clustering in 1D
• http://stackoverflow.com/questions/11513484/1d-number-array-clustering
• Clustering in ≥2D is NP-hard
• Clustering in 1D is known as the segmentation problem and can be solved efficiently by dynamic programming
18/09/2019 Design and Analysis of Algorithms 2019 week 3 22
Master’s Programme in Computer Science
SEGMENTATION PROBLEM: MOTIVATION
18/09/2019 Design and Analysis of Algorithms 2019 week 3 23
A company owns
real estates
and the CEO
requests a map
with colour codes
showing
different property
price ranges
Master’s Programme in Computer Science
SEGMENTATION PROBLEM: MOTIVATION
18/09/2019 Design and Analysis of Algorithms 2019 week 3 24
1 000 000 – 3 000 000
500 000 – 750 000 200 000 – 400 000 75 000 – 150 000 30 000 – 50 000
Master’s Programme in Computer Science
SEGMENTATION: MOTIVATION 2
• Compression boosting:
• Partition a string A[1..n] into segments that compress better by tailored compressor for each segment.
‒ E.g. ababbbacdaccddbbbaabaab
• Let c(i,j) give the compression result for segment A[i..j].
• Then an optimal partitioning of A is one with the sum of its segment costs being minimum.
18/09/2019 Design and Analysis of Algorithms 2019 week 3 25
Master’s Programme in Computer Science
SEGMENTATION: DEFINITION
• Input: an array A[1..n], an integer k>0, and a cost function c(i,j) that assigns a cost to any segment (subarray) A[i..j]
• Output: a segmentation of A[1..n] into k segments A[1..n] = A[1..i1]A[i1+1..i2]…A[ik-1..n] so that the sum of the segment costs is minimized
18/09/2019 Design and Analysis of Algorithms 2019 week 3 26
Master’s Programme in Computer Science
SEGMENTATION: RECURRENCE
•
18/09/2019 Design and Analysis of Algorithms 2019 week 3 27
Master’s Programme in Computer Science
1D CLUSTERING: DEFINITION
k clusters definition?
18/09/2019 Design and Analysis of Algorithms 2019 week 3 28
Master’s Programme in Computer Science
1D CLUSTERING: L2 AND AVERAGE
segment= 1d cluster
k clusters
c1
18/09/2019 Design and Analysis of Algorithms 2019 week 3 29
Master’s Programme in Computer Science
1D CLUSTERING: L2 AND AVERAGE
0 4 5
AVG(0,4,5)=3
cL2(1..3)=(3-0)2+(3-4)2+(3-5)2=14
A[1..3]=0,4,5
Clustering with k=1
18/09/2019 Design and Analysis of Algorithms 2019 week 3 30
Master’s Programme in Computer Science
1D CLUSTERING: RECURRENCE
• devMin[j,k’]=”minimum total deviation after segmenting A[1..j] into k’ segments”
• devMin[j,k’]=mini<j devMin [i,k’-1]+cL2(i+1..j)
• minC sumdev(A,C) = devMin[n,k] (optimal solution)
• O(n3k) running time with direct implementation
• Optimizations, extensions, correctness, initialization, traceback in study groups & exercises.
k’-1 clusters
i i+1 j
+
18/09/2019 Design and Analysis of Algorithms 2019 week 3 31
Master’s Programme in Computer Science
1D CLUSTERING: EVALUATION ORDER
devMin[j,k’] = mini<j devMin [i,k’-1]+cL2(i+1..j)
Evaluation order:
k’
j
i
k’-1
…
Column-by-column (above), row-by-row, …
min (…) + cost of arc
18/09/2019 Design and Analysis of Algorithms 2019 week 3
Master’s Programme in Computer Science
1D CLUSTERING: RECAP+INITIALIZATION
k’
j
i
k’-1
mini<j devMin[..]
+ cost of arc
0 0
devMin[0..n,0..k]
optimum
solution:
traceback
following
arcs
A
n
0 k
…
k’-1
AVG
min cost to
segment A[1..i]
to k’-1 pieces
cost to add
segment
A[i+1..j]
min cost to
segment A[1..j]
to k’ pieces
0 segments only possible for empty sequence: d[i,0]=∞ except d[0,0]=0
more segments
than points:
devMin[i,k’]=∞
for i<k’
Optimal chain of matrix multiplications
Dynamic programming is not always a shortest path
problem
18/09/2019 Design and Analysis of Algorithms 2019 week 3 34
Master’s Programme in Computer Science
OPTIMAL CHAIN OF MATRIX MULTIPLICATIONS
x = n
m
m
p
n
p
O(npm) time
x n
m
m
p
x
q
p
=
n
q
18/09/2019 Design and Analysis of Algorithms 2019 week 3 35
Master’s Programme in Computer Science
OPTIMAL CHAIN OF MATRIX MULTIPLICATIONS
x n
m
m
p
x
q
p
=
n
q
x n
m
m
p
x
q
p
=
n
q
npm+nqp
multiplications
n=10, m=15, p=7,
q=4:
1330 multiplications
mqp+nqm
multiplications
n=10, m=15, p=7,
q=4:
1470 multiplications
18/09/2019 Design and Analysis of Algorithms 2019 week 3 36
Master’s Programme in Computer Science
OPTIMAL CHAIN OF MATRIX MULTIPLICATIONS
x =
18/09/2019 Design and Analysis of Algorithms 2019 week 3 37
the multiplication
performed last
Master’s Programme in Computer Science
OPTIMAL CHAIN: PSEUDOCODE
18/09/2019 Design and Analysis of Algorithms 2019 week 3 38
Master’s Programme in Computer Science
OPTIMAL CHAIN: DYNAMIC PROGRAMMING
• Dynamic programming instead of memoization:
• Compute M(i,j) in increasing order of interval length j-i+1
• Then M(i,k) and M(k+1,j) always computed before M(i,j)
• Subproblem dependencies form a DAG
• Each M(i,j) computed once in O(n) time: total O(n3) time
M(i,j) M(i,k)
M(k+1,j)
18/09/2019 Design and Analysis of Algorithms 2019 week 3 39
Master’s Programme in Computer Science
OPTIMAL CHAIN: NOT SHORTEST PATH PROBLEM
• Optimal M(i,j) depends on two nodes M(i,k) and M(k+1,j)
• Thus a solution is not a path but a tree
18/09/2019 Design and Analysis of Algorithms 2019 week 3 40
x
x x
x x A B
C D E F
((AxB)x((CxD)x(ExF)))