CS 332: Algorithms

David Luebke 1 04/21/23

CS 332: Algorithms

Dijkstra’s Algorithm Continued

Disjoint-Set Union

Return to MST (Kruskal)

Amortized Analysis


Review: Minimum Spanning Tree

Problem: given a connected, undirected, weighted graph, find a spanning tree using edges that minimize the total weight

1410

3

6 4

5

2

9

15

8


Review: Minimum Spanning Tree

MSTs satisfy the optimal substructure property: an optimal tree is composed of optimal subtrees

If T is MST of G, and A T is a subtree of T, and (u,v) is the min-weight edge connecting A to V-A, then (u,v) T


Review: Prim’s Algorithm

MST-Prim(G, w, r)

Q = V[G];

for each u Q key[u] = ; key[r] = 0;

p[r] = NULL;

while (Q not empty)

u = ExtractMin(Q);

for each v Adj[u] if (v Q and w(u,v) < key[v]) p[v] = u;

key[v] = w(u,v);

5 2 9

0 8 15

3

4

1410

3

6 45

2

9

15

8

u


Review: Prim’s Algorithm

MST-Prim(G, w, r)

Q = V[G];

for each u Q key[u] = ; key[r] = 0;

p[r] = NULL;

while (Q not empty)

u = ExtractMin(Q);

for each v Adj[u] if (v Q and w(u,v) < key[v]) p[v] = u;

key[v] = w(u,v);

What will be the running time?A: Depends on queue binary heap: O(E lg V) Fibonacci heap: O(V lg V + E)


Review: Single-Source Shortest Path

Problem: given a weighted directed graph G, find the minimum-weight path from a given source vertex s to another vertex v “Shortest-path” = minimum weight Weight of path is sum of edges E.g., a road map: what is the shortest path from

Chapel Hill to Charlottesville?


Review: Shortest Path Properties

Optimal substructure: the shortest path consists of shortest subpaths

Let (u,v) be the weight of the shortest path from u to v. Shortest paths satisfy the triangle inequality: (u,v) (u,x) + (x,v)

In graphs with negative weight cycles, some shortest paths will not exist


Review: Relaxation

Key technique: relaxation Maintain upper bound d[v] on (s,v):Relax(u,v,w) {

if (d[v] > d[u]+w) then d[v]=d[u]+w;

}

952

752

Relax

652

652

Relax


Review: Bellman-Ford Algorithm

BellmanFord()

for each v V d[v] = ; d[s] = 0;

for i=1 to |V|-1

for each edge (u,v) E Relax(u,v, w(u,v));

for each edge (u,v) E if (d[v] > d[u] + w(u,v))

return “no solution”;

Relax(u,v,w): if (d[v] > d[u]+w) then d[v]=d[u]+w

Initialize d[], whichwill converge to shortest-path value

Relaxation: Make |V|-1 passes, relaxing each edge

Test for solution:have we converged yet?Ie, negative cycle?


Review: Bellman-Ford Algorithm

BellmanFord()

for each v V d[v] = ; d[s] = 0;

for i=1 to |V|-1

for each edge (u,v) E Relax(u,v, w(u,v));

for each edge (u,v) E if (d[v] > d[u] + w(u,v))

return “no solution”;

Relax(u,v,w): if (d[v] > d[u]+w) then d[v]=d[u]+w

What will be the running time?


Review: Bellman-Ford

Running time: O(VE) Not so good for large dense graphs But a very practical algorithm in many ways

Note that order in which edges are processed affects how quickly it converges


Review: DAG Shortest Paths

Problem: finding shortest paths in DAG Bellman-Ford takes O(VE) time. Do better by using topological sort

If were lucky and processes vertices on each shortest path from left to right, would be done in one pass

Every path in a dag is subsequence of topologically sorted vertex order, so processing verts in that order, we will do each path in forward order (will never relax edges out of vert before doing all edges into vert).

Thus: just one pass. Running time: O(V+E)


Review: Dijkstra’s Algorithm

Dijkstra(G)

for each v V d[v] = ; d[s] = 0; S = ; Q = V; while (Q ) u = ExtractMin(Q);

S = S {u}; for each v u->Adj[] if (d[v] > d[u]+w(u,v))

d[v] = d[u]+w(u,v);RelaxationStepNote: this

is really a call to Q->DecreaseKey()


Review: Dijkstra’s Algorithm

Dijkstra(G)



d[v] = d[u]+w(u,v);Running time: O(E lg V) using binary heap for QCan acheive O(V lg V + E) with Fibonacci heaps


Dijkstra’s Algorithm

Dijkstra(G)



d[v] = d[u]+w(u,v);Correctness: we must show that when u is removed from Q, it has already converged


Correctness Of Dijkstra's Algorithm

Note that d[v] (s,v) v Let u be first vertex picked s.t. shorter path than d[u] d[u] > (s,u) Let y be first vertex V-S on actual shortest path from su d[y] = (s,y)

Because d[x] is set correctly for y's predecessor x S on the shortest path, and When we put x into S, we relaxed (x,y), giving d[y] the correct value

s

xy

up2

p2


Correctness Of Dijkstra's Algorithm

Note that d[v] (s,v) v Let u be first vertex picked s.t. shorter path than d[u] d[u] > (s,u) Let y be first vertex V-S on actual shortest path from su d[y] = (s,y) d[u] > (s,u)

= (s,y) + (y,u) (Why?)= d[y] + (y,u) d[y] But if d[u] > d[y], wouldn't have chosen u. Contradiction.

s

xy

up2

p2


Disjoint-Set Union Problem

Want a data structure to support disjoint sets Collection of disjoint sets S = {Si}, Si Sj =

Need to support following operations: MakeSet(x): S = S {{x}} Union(Si, Sj): S = S - {Si, Sj} {Si Sj}

FindSet(X): return Si S such that x Si

Before discussing implementation details, we look at example application: MSTs


Kruskal’s Algorithm

Kruskal()

{

T = ; for each v V MakeSet(v);

sort E by increasing edge weight w

for each (u,v) E (in sorted order) if FindSet(u) FindSet(v) T = T {{u,v}}; Union(FindSet(u), FindSet(v));

}



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1?

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2? 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5?

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148?

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9?

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13?

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

14?8

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

17?25

148

21

Run the algorithm:



Kruskal()

{




}

2 19?

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21?

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725?

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:



Kruskal()

{




}

2 19

9

1

5

13

1725

148

21

Run the algorithm:


Correctness Of Kruskal’s Algorithm

Sketch of a proof that this algorithm produces an MST for T: Assume algorithm is wrong: result is not an MST Then algorithm adds a wrong edge at some point If it adds a wrong edge, there must be a lower weight

edge (cut and paste argument) But algorithm chooses lowest weight edge at each step.

Contradiction Again, important to be comfortable with cut and

paste arguments



Kruskal()

{




}

What will affect the running time?



Kruskal()

{




}

What will affect the running time? 1 Sort

O(V) MakeSet() callsO(E) FindSet() callsO(V) Union() calls

(Exactly how many Union()s?)


Kruskal’s Algorithm: Running Time

To summarize: Sort edges: O(E lg E) O(V) MakeSet()’s O(E) FindSet()’s O(V) Union()’s

Upshot: Best disjoint-set union algorithm makes above 3

operations take O(E(E,V)), almost constant Overall thus O(E lg E), almost linear w/o sorting


Disjoint Set Union

So how do we implement disjoint-set union? Naïve implementation: use a linked list to

represent each set:

MakeSet(): ??? time FindSet(): ??? time Union(A,B): “copy” elements of A into B: ??? time


Disjoint Set Union

So how do we implement disjoint-set union? Naïve implementation: use a linked list to represent each

set:

MakeSet(): O(1) time FindSet(): O(1) time Union(A,B): “copy” elements of A into B: O(A) time

How long can a single Union() take? How long will n Union()’s take?


Disjoint Set Union: Analysis

Worst-case analysis: O(n2) time for n Union’sUnion(S1, S2) “copy” 1 element

Union(S2, S3) “copy” 2 elements

…

Union(Sn-1, Sn) “copy” n-1 elements

O(n2)

Improvement: always copy smaller into larger Why will this make things better? What is the worst-case time of Union()?

But now n Union’s take only O(n lg n) time!


Amortized Analysis of Disjoint Sets

Amortized analysis computes average times without using probability

With our new Union(), any individual element is copied at most lg n times when forming the complete set from 1-element sets Worst case: Each time copied, element in smaller set

1st time resulting set size 2

2nd time 4

…

(lg n)th time n


Amortized Analysis of Disjoint Sets

Since we have n elements each copied at most lg n times, n Union()’s takes O(n lg n) time

We say that each Union() takes O(lg n) amortized time Financial term: imagine paying $(lg n) per Union At first we are overpaying; initial Union $O(1) But we accumulate enough $ in bank to pay for later

expensive O(n) operation. Important: amount in bank never goes negative


Amortized Analysis

Book describes 3 views of amortized analysis in Chapter 18: Aggregate Accounting Potential


Amortized Analysis: Aggregate Method

Aggregate method This is what we just did for Union() n operations take time T(n) Average cost of an operation = T(n)/n Not very precise


Amortized Analysis: Accounting Method

Accounting method We have done this with graph algorithms Charge each operation an amortized cost

Usually just guess/invent this cost Amount not used stored in “bank” Later operations can used stored work Balance must not go negative


Amortized Analysis:Potential Method

Potential method “Stored work” of accounting method is viewed as

“potential energy” Most flexible and powerful approach See book if interested; we won’t go into (or test)


Amortized Analysis Example: Dynamic Tables

Implementing a table (e.g., hash table) for dynamic data, want to make it small as possible

Problem: if too many items inserted, table may be too small

Idea: allocate more memory as needed


Dynamic Tables

1. Init table size m = 1

2. Insert elements until number n > m

3. Generate new table of size 2m

4. Reinsert old elements into new table

5. (back to step 2)What is the worst-case cost of an insert?One insert can be costly, but the total?


Analysis Of Dynamic Tables

Let ci = cost of ith insert

ci = i if i-1 is exact power of 2, 1 otherwise

Example: Operation Table Size Cost

Insert(1) 1 1 1






Insert(1) 1 1 1Insert(2) 2 1 + 1 2






Insert(1) 1 1 1Insert(2) 2 1 + 1 2Insert(3) 4 1 + 2 3






Insert(1) 1 1 1Insert(2) 2 1 + 1 2Insert(3) 4 1 + 2 3Insert(4) 4 1 4






Insert(1) 1 1 1Insert(2) 2 1 + 1 2Insert(3) 4 1 + 2 3Insert(4) 4 1 4Insert(5) 8 1 + 4 5






Insert(1) 1 1 1Insert(2) 2 1 + 1 2Insert(3) 4 1 + 2 3Insert(4) 4 1 4Insert(5) 8 1 + 4 5Insert(6) 8 1 6






Insert(1) 1 1 1Insert(2) 2 1 + 1 2Insert(3) 4 1 + 2 3Insert(4) 4 1 4Insert(5) 8 1 + 4 5Insert(6) 8 1 6Insert(7) 8 1 7






Insert(1) 1 1 1Insert(2) 2 1 + 1 2Insert(3) 4 1 + 2 3Insert(4) 4 1 4Insert(5) 8 1 + 4 5Insert(6) 8 1 6Insert(7) 8 1 7Insert(8) 8 1 8






Insert(1) 1 1 1Insert(2) 2 1 + 1 2Insert(3) 4 1 + 2Insert(4) 4 1Insert(5) 8 1 + 4Insert(6) 8 1Insert(7) 8 1Insert(8) 8 1Insert(9) 16 1 + 8

123456789


Aggregate Analysis

n Insert() operations cost

Average cost of operation = (total cost)/(# operations) < 3

Asymptotically, then, a dynamic table costs the same as a fixed-size table Both O(1) per Insert operation

nnnncn

j

jn

ii 3)12(2

lg

01


Accounting Analysis

Charge each operation $3 amortized cost Use $1 to perform immediate Insert() Store $2

When table doubles $1 reinserts old item, $1 reinserts another old item Point is, we’ve already paid these costs Upshot: constant (amortized) cost per operation


Accounting Analysis

Suppose must support insert & delete, table should contract as well as expand Table overflows double it (as before) Table < 1/2 full halve it: BAD IDEA (Why?) Better: Table < 1/4 full halve it Charge $3 for Insert (as before) Charge $2 for Delete

Store extra $1 in emptied slot Use later to pay to copy remaining items to new table when

shrinking table


The End


Exercise 1 Feedback

First, my apologies… Harder than I thought Too late to help with midterm

Proof by substitution: T(n) = T(n/2 + n) + n Most people assumed it was O(n lg n)…why?

Resembled proof from class: T(n) = 2T(n/2 + 17) + n The correct intuition: n/2 dominates n term, so it resembles

T(n) = T(n/2) + n, which is O(n) by m.t. Still, if it’s O(n) it’s O(n lg n), right?


Exercise 1: Feedback

So, prove by substitution thatT(n) = T(n/2 + n) + n = O(n lg n)

Assume T(n) cn lg n Then T(n) c(n/2 + n) lg (n/2 + n)

c(n/2 + n) lg (n/2 + n) c(n/2 + n) lg (3n/2)

…

Documents

CS 332: Algorithms