Algorithm Design and Analysisudel.edu/~amotong/teaching/algorithm design...Randomized Algorithms Summer 2018 Amo G. Tong 25. Randomized Algorithms Summer 2018 Amo G. Tong 26 𝐸𝑇𝑛=

Algorithm Design and Analysis

Summer 2018 Amo G. Tong 1

Lecture 4Randomized Algorithms• Randomized Quick Sort

• Randomized Selection


• Quick Sort:• Take an element 𝑥 = 𝐴[𝑞] in 𝐴[1,… , 𝑛].

• Partition into two subarrays 𝐴[1,… , 𝑞 − 1] and 𝐴[𝑞 + 1,… , 𝑛] such that each element of 𝐴[1,… , 𝑞 − 1] is less than or equal to 𝑥 which is less than or equal to each element of 𝐴[𝑞 + 1,… , 𝑛].

• Sort the subarrays recursively.

• No need to combine.

Randomized Algorithms


Input: an array 𝐴[1,… , 𝑛] of numbers

• Quick Sort:




PARTITION (𝐴, 𝑝, 𝑟)1 select an element 𝑥 = 𝐴[𝑘] in 𝐴[𝑝. . 𝑟]2 exchange A[k] with A[r];3 𝑖 = 𝑝 − 14 for 𝑗 = 𝑝 to 𝑟 − 15 if 𝐴 𝑗 ≤ 𝑥6 𝑖 = 𝑖 + 17 exchange 𝐴[𝑖] with 𝐴[𝑗]8 end9 end10 exchange 𝐴[𝑖 + 1] with 𝐴[𝑟]11 return 𝑖 + 1;

• Quick Sort:




0 9 5 2 1 𝑖 = 0, exchange 5 with 1

0 1 2 5 9


• Quick Sort:




0 9 5 2 10 9 1 2 5

𝑖 = 0, exchange 5 with 1


0 1 2 5 9


• Quick Sort:




0 9 5 2 10 9 1 2 50 9 1 2 5



𝑖 = 1

0 1 2 5 9


• Quick Sort:




0 9 5 2 10 9 1 2 50 9 1 2 50 9 1 2 5



𝑖 = 1


0 1 2 5 9


• Quick Sort:




0 9 5 2 10 9 1 2 50 9 1 2 50 9 1 2 50 1 9 2 5



𝑖 = 1



0 1 2 5 9


• Quick Sort:




0 9 5 2 10 9 1 2 50 9 1 2 50 9 1 2 50 1 9 2 50 1 2 9 5



𝑖 = 1



exchange 9 with 5

0 1 2 5 9


• Quick Sort:




0 9 5 2 10 9 1 2 50 9 1 2 50 9 1 2 50 1 9 2 50 1 2 9 50 1 2 5 9



𝑖 = 1



exchange 9 with 5

Return 4

0 1 2 5 9


• Quick Sort:




QUICKSORT (𝐴, 𝑝, 𝑟)1 if 𝑝 < 𝑟2 𝑞 =PARTITION(A, p, r)3 QUICKSORT (𝐴, 𝑝, 𝑞 − 1);4 QUICKSORT (𝐴, 𝑞 + 1, 𝑟);5 end


0 9 5 2 1

0 1 2 5 9

• Quick Sort:






Which element should we take?

• Quick Sort:• Take an element 𝑥 in 𝐴[1,… , 𝑛].

• Partition into two subarrays 𝐴1[1, … , 𝑎] and 𝐴2[1, … , 𝑏] such that each element of 𝐴1 is less than or equal to x which is less than or equal to each element of 𝐴2. (𝑎 + 𝑏 = 𝑛 − 1)

• Sort the subarrays recursively.

• No need to combine.

• 𝑥 decides the size of the two subarrays.• 𝑇(𝑛) = 𝑇(𝑎) + 𝑇(𝑏) + Θ(𝑛);



• 𝑥 decides the size of the two subarrays.• 𝑇 𝑛 = 𝑇 𝑎 + 𝑇 𝑏 + Θ 𝑛 .

• If the array is always evenly partitioned:• 𝑥=median





• 𝑇 𝑛 = 𝑇 𝑛/2 + 𝑇 𝑛/2 + Θ 𝑛 .

• 𝑇(𝑛) = Θ(𝑛 log 𝑛)





• 𝑇 𝑛 = 𝑇 𝑛/2 + 𝑇 𝑛/2 + Θ 𝑛 .


• If the array is always partitioned with a fixed proportion 𝒂:• x= the 𝑎𝑛 -th smallest element

• 𝑇(𝑛) = 𝑇(𝑎𝑛) + 𝑇((1 − 𝑎)𝑛) + Θ(𝑛) where 0 < 𝑎 < 1;





• If one of the subarray is always empty:• 𝑥= the maximum or the minimum

• 𝑇(𝑛) = 𝑇(𝑛 − 1) + 𝑇(0) + Θ(𝑛)

• 𝑇(𝑛) = Θ(𝑛2)




• If one of the subarray is always empty:• 𝑥= the maximum or the minimum

• 𝑇(𝑛) = 𝑇(𝑛 − 1) + 𝑇(0) + Θ(𝑛)

• 𝑇(𝑛) = Θ(𝑛2)

• If one of the subarray has a constant length 𝒂:• x= the a-th smallest element

• 𝑇(𝑛) = 𝑇(𝑛 − 1 − 𝑎) + 𝑇(𝑎) + Θ(𝑛) where 0 < 𝑎 < 𝑛;

• 𝑇(𝑛) =? ? (try it yourself)



• If we always select the last element as the pivot:



• If we always select the last element as the pivot:• Worst-case running time:

• 𝑇(𝑛) = 𝑇(𝑛 − 1) + 𝑇(0) + Θ(𝑛)

• 𝑇(𝑛) = 𝑂(𝑛2)




• 𝑇(𝑛) = 𝑇(𝑛 − 1) + 𝑇(0) + Θ(𝑛)

• 𝑇(𝑛) = 𝑂(𝑛2)

• Expected running time:• Assume each of the possible permutations appears with the same

probability.




• 𝑇(𝑛) = 𝑇(𝑛 − 1) + 𝑇(0) + Θ(𝑛)

• 𝑇(𝑛) = 𝑂(𝑛2)


probability.

• With probability 1/n, the last element is the 𝒊-th smallest.




• 𝑇(𝑛) = 𝑇(𝑛 − 1) + 𝑇(0) + Θ(𝑛)

• 𝑇(𝑛) = 𝑂(𝑛2)


probability.

• With probability 1/n, the last element is the 𝑖-th smallest.

• If the last element is the 𝑖-th smallest, 𝑇 𝑛 = T(𝑖 − 1) + 𝑇(𝑛 − 𝑖) + 𝑐𝑛




• 𝑇(𝑛) = 𝑇(𝑛 − 1) + 𝑇(0) + Θ(𝑛)

• 𝑇(𝑛) = 𝑂(𝑛2)


probability.

• With probability 1/n, the last element is the 𝑖-th smallest.

• If the last element is the 𝑖-th smallest, 𝑇 𝑛 = T(𝑖 − 1) + 𝑇(𝑛 − 𝑖) + 𝑐𝑛

• The expected value is

𝐸 𝑇 𝑛 = σ𝑖=1𝑛 1

𝑛(𝐸[𝑇 𝑖 − 1 ] + 𝐸[𝑇 𝑛 − 𝑖 ] + 𝑐𝑛)






𝑛(𝑇(𝑖 − 1) + 𝑇(𝑛 − 𝑖) + 𝑐𝑛)⇒ 𝐸 𝑇 𝑛 = O(n log n)

Proof: Let 𝑇 𝑘 ≤ 𝑎 for 𝑘 < 2.





Proof: Let 𝑇 𝑘 ≤ 𝑎 for 𝑘 < 2.Inductive hypothesis: 𝐸 𝑇 𝑛 ≤ 𝑏 𝑛 log𝑛






Basic: 𝐸[𝑇(2)] =1

2(𝑇(0) + 𝑇(1) + 𝑐 ⋅ 2) +

1

2(𝑇(1) + 𝑇(0) + 𝑐 ⋅ 2) = 2𝑎 + 2𝑐.

We need 2𝑎 + 2𝑐 < 2𝑏.







2(𝑇(0) + 𝑇(1) + 𝑐 ⋅ 2) +

1

2(𝑇(1) + 𝑇(0) + 𝑐 ⋅ 2) = 2𝑎 + 2𝑐.

We need 2𝑎 + 2𝑐 < 2𝑏.Induction: Suppose 𝐸 𝑇 𝑛 ≤ 𝑏 𝑛 log 𝑛 for all 2 < 𝑛 < 𝑘.







2(𝑇(0) + 𝑇(1) + 𝑐 ⋅ 2) +

1

2(𝑇(1) + 𝑇(0) + 𝑐 ⋅ 2) = 2𝑎 + 2𝑐.


𝑬[𝑻(𝒌)] = σ𝒊=𝟏𝒌 𝟏

𝒌(𝑬[𝑻 𝒊 − 𝟏 ] + 𝑬[𝑻 𝒌 − 𝒊 ] + 𝒄𝒌)







2(𝑇(0) + 𝑇(1) + 𝑐 ⋅ 2) +

1

2(𝑇(1) + 𝑇(0) + 𝑐 ⋅ 2) = 2𝑎 + 2𝑐.


𝐸[𝑇(𝑘)] = σ𝑖=1𝑘 1

𝑘(𝐸[𝑇 𝑖 − 1 ] + 𝐸[𝑇 𝑘 − 𝑖 ] + 𝑐𝑘)

=𝟏

𝒌(𝟐σ𝒊=𝟎

𝒌−𝟏𝑬[𝑻 𝒊 ] + 𝒄𝒌𝟐)







2(𝑇(0) + 𝑇(1) + 𝑐 ⋅ 2) +

1

2(𝑇(1) + 𝑇(0) + 𝑐 ⋅ 2) = 2𝑎 + 2𝑐.


𝐸[𝑇(𝑘)] = σ𝑖=1𝑘 1

𝑘(𝐸[𝑇 𝑖 − 1 ] + 𝐸[𝑇 𝑘 − 𝑖 ] + 𝑐𝑘)

=1

𝑘(2σ𝑖=0

𝑘−1𝐸[𝑇 𝑖 ] + 𝑐𝑘2)

=𝟏

𝒌(𝟐σ𝒊=𝟐

𝒌−𝟏𝑬[𝑻 𝒊 ] + 𝟐𝑻(𝟎) + 𝟐𝑻(𝟏) + 𝒄𝒌𝟐)







2(𝑇(0) + 𝑇(1) + 𝑐 ⋅ 2) +

1

2(𝑇(1) + 𝑇(0) + 𝑐 ⋅ 2) = 2𝑎 + 2𝑐.


𝐸[𝑇(𝑘)] = σ𝑖=1𝑘 1

𝑘(𝐸[𝑇 𝑖 − 1 ] + 𝐸[𝑇 𝑘 − 𝑖 ] + 𝑐𝑘)

=1

𝑘(2σ𝑖=0

𝑘−1𝐸[𝑇 𝑖 ] + 𝑐𝑘2)

=1

𝑘(2σ𝑖=2

𝑘−1𝐸[𝑇 𝑖 ] + 2𝑇(0) + 2𝑇(1) + 𝑐𝑘2)

≤𝟏

𝒌(𝟐σ𝒊=𝟐

𝒌−𝟏𝒃 𝒊 𝐥𝐨𝐠 𝒊) + (𝟒𝒂 + 𝒄𝒌𝟐)/𝒌







2(𝑇(0) + 𝑇(1) + 𝑐 ⋅ 2) +

1

2(𝑇(1) + 𝑇(0) + 𝑐 ⋅ 2) = 2𝑎 + 2𝑐.


𝐸[𝑇(𝑘)] = σ𝑖=1𝑘 1

𝑘(𝐸[𝑇 𝑖 − 1 ] + 𝐸[𝑇 𝑘 − 𝑖 ] + 𝑐𝑘)

=1

𝑘(2σ𝑖=0

𝑘−1𝐸[𝑇 𝑖 ] + 𝑐𝑘2)

=1

𝑘(2σ𝑖=2

𝑘−1𝐸[𝑇 𝑖 ] + 2𝑇(0) + 2𝑇(1) + 𝑐𝑘2)

≤2𝑏

𝑘(σ𝑖=2

𝑘−1 𝑖 log 𝑖) + (4𝑎 + 𝑐𝑘2)/𝑘

(see additional reading material)

≤2b

𝑘(

𝑘2

2log 𝑘 −

𝑘2

4+

1

4) + (4𝑎 + 𝑐𝑘2)/𝑘



We need this to be ≤ 𝑏 𝑘 log 𝑘.





2(𝑇(0) + 𝑇(1) + 𝑐 ⋅ 2) +

1

2(𝑇(1) + 𝑇(0) + 𝑐 ⋅ 2) = 2𝑎 + 2𝑐.


𝐸[𝑇(𝑘)] = σ𝑖=1𝑘 1

𝑘(𝐸[𝑇 𝑖 − 1 ] + 𝐸[𝑇 𝑘 − 𝑖 ] + 𝑐𝑘)

=1

𝑘(2σ𝑖=0

𝑘−1𝐸[𝑇 𝑖 ] + 𝑐𝑘2)

=1

𝑘(2σ𝑖=2

𝑘−1𝐸[𝑇 𝑖 ] + 2𝑇(0) + 2𝑇(1) + 𝑐𝑘2)

≤2𝑏

𝑘(σ𝑖=2

𝑘−1 𝑖 log 𝑖) + (4𝑎 + 𝑐𝑘2)/𝑘

(see additional reading material)

≤2b

𝑘(

𝑘2

2log 𝑘 −

𝑘2

4+

1

4) + (4𝑎 + 𝑐𝑘2)/𝑘


• 𝑇(𝑛) = 𝑇(𝑛 − 1) + 𝑇(0) + Θ(𝑛)

• 𝑇(𝑛) = 𝑂(𝑛2)


probability.

• The expected running time is

𝐸 𝑇 𝑛 = Θ(𝑛 log 𝑛)




• 𝑇(𝑛) = 𝑇(𝑛 − 1) + 𝑇(0) + Θ(𝑛)

• 𝑇(𝑛) = 𝑂(𝑛2)


probability.


𝐸 𝑇 𝑛 = Θ(𝑛 log 𝑛)



This is not always true.We want a Θ(𝑛 log 𝑛)algorithm which does not rely on this assumption

• Randomized Quick Sort;• Select the pivot uniformly at random from the given array.




• With probability 1/𝑛, 𝐴[𝑖] is selected as the pivot.

• With probability 1/𝑛, the 𝑖-th smallest element is selected.








RANDOM-PARTITION (𝐴, 𝑝, 𝑟)1 randomly select an element x = 𝐴[𝑘] in 𝐴[𝑝. . 𝑟],

where each element has the same probability to be selected.2 exchange A[k] with A[r];3 𝑖 = 𝑝 − 14 for 𝑗 = 𝑝 to 𝑟 − 15 if 𝐴 𝑗 ≤ 𝑥6 𝑖 = 𝑖 + 17 exchange 𝐴[𝑖] with 𝐴[𝑗]8 end9 end10 exchange 𝐴[𝑖 + 1] with 𝐴[𝑟]11 return 𝑖 + 1;







RANDOM-QUICKSORT (𝐴, 𝑝, 𝑟)1 if 𝑝 < 𝑟2 𝑞 =RANDOM-PARTITION(A, p, r)3 RANDOM-QUICKSORT (𝐴, 𝑝, 𝑞 − 1);4 RANDOM-QUICKSORT (𝐴, 𝑞 + 1, 𝑟);5 end




• Expected running time:• No assumption on the input.



𝑛(𝐸[𝑇 𝑖 − 1 ] + 𝐸[𝑇 𝑛 − 𝑖 ] + 𝑐𝑛)



The same as that in page 24.

• Selection:



Input: an array 𝐴[1,… , 𝑛] of distinct numbers and an integer 𝑖, 1 ≤ 𝑖 ≤ 𝑛.

Output: the 𝑖-th smallest element in 𝐴[1,… , 𝑛] .

• Selection:



Pseudocode:RANDOMIZED-SELECT(𝐴, 𝑝, 𝑟, 𝑖)1 if 𝑝 == 𝑟2 return A[p]3 end4 q=RANDOM-PARTITION(𝐴, 𝑝, 𝑟)5 𝑘 = 𝑝 − 𝑞 + 1;6 if 𝑖 == 𝑘7 return 𝐴[𝑞]8 end9 if 𝑖 < 𝑘10 return RANDOM-SELECT(𝐴, 𝑝, 𝑞 − 1, 𝑖)11 else12 return RANDOM-SELECT(𝐴, 𝑞 + 1, 𝑟, 𝑖 − 𝑘)13 end

• Selection:



Pseudocode:RANDOMIZED-SELECT(𝐴, 𝑝, 𝑟, 𝑖)1 if 𝑝 == 𝑟2 return A[p]3 end4 𝑞 =RANDOM-PARTITION(𝐴, 𝑝, 𝑟)5 𝑘 = 𝑝 − 𝑞 + 1;6 if 𝑖 == 𝑘7 return 𝐴[𝑞]8 end9 if 𝑖 < 𝑘10 return RANDOMIZED-SELECT(𝐴, 𝑝, 𝑞 − 1, 𝑖)11 else12 return RANDOMIZED-SELECT(𝐴, 𝑞 + 1, 𝑟, 𝑖 − 𝑘)13 end

Θ(𝑛)

𝑇(max(𝑞 − 𝑝, 𝑟 − 𝑞))

T(𝑛)

• Randomized selection;• Select the pivot uniformly at random from the given array.

• With probability1/𝑛, 𝐴[𝑖] is selected as the pivot.

• With probability1/𝑛, the 𝑖-th smallest element is selected.

• Running time:• If the 𝑖-th smallest element is selected as the pivot, the running time is

𝑇 𝑛 = 𝑇 max 𝑖 − 1, 𝑛 − 𝑖 + Θ(𝑛)

• So the expected running time is:

𝐸 𝑇 𝑛 =

𝑖=0

𝑛−11

𝑛(𝐸[𝑇 max 𝑖 − 1, 𝑛 − 𝑖 ] + Θ(𝑛) )





𝐸 𝑇 𝑛

=

𝑖=1

𝑛1

𝑛(𝐸[𝑇 max 𝑖 − 1, 𝑛 − 𝑖 ] + Θ(𝑛) )

=

𝑖=1

𝑛/2 −11

𝑛(𝐸[𝑇 max 𝑖 − 1, 𝑛 − 𝑖 ] + Θ(𝑛) ) +

𝑖= 𝑛/2 −1

𝑛1

𝑛(𝐸[𝑇 max 𝑖 − 1, 𝑛 − 𝑖 ] + Θ(𝑛) )

=

𝑖=1

𝑛/2 −11

𝑛(𝐸[𝑇 𝑛 − 𝑖 ] + Θ(𝑛) ) +

𝑖= 𝑛/2

𝑛1

𝑛(𝐸[𝑇 𝑖 − 1 ] + Θ(𝑛) )

=

𝑖= 𝑛/2

𝑛2

𝑛(𝐸[𝑇 𝑖 − 1 ] + Θ(𝑛) )

𝐸 𝑇 𝑛 =

𝑖= 𝑛/2

𝑛2

𝑛(𝐸[𝑇 𝑖 − 1 ] + Θ(𝑛) )

Prove 𝐸[𝑇(𝑛)] = 𝑂(𝑛)

• Expected running time:• The algorithm is not randomized. The expectation is taken over the

distribution of the inputs.





• The algorithm is randomized. The expected running time is 𝑶(𝒇(𝒏)) for any given input.





• The algorithm is randomized. The expected running time is 𝑂(𝑓(𝑛)) for any given input.

• The algorithm is randomized and the input is also sampled.





• The algorithm is randomized. The expected running time is 𝑂(𝑓(𝑛)) for any given input.

• The algorithm is randomized and the input is also sampled.• Consider the Randomized Quick Sort with an input uniformly sampled

from {all the permutations over {1, … , 𝑛}}.

• What is the running time? (exercise)



Documents

Algorithm Design and Analysisudel.edu/~amotong/teaching/algorithm design...Randomized Algorithms Summer 2018 Amo G. Tong 25. Randomized Algorithms Summer 2018 Amo G. Tong 26 𝐸𝑇𝑛=