Sorting Algorithms

We have already seen:I Selection-sort

I Insertion-sort

I Heap-sort

We will see:I Bubble-sort

I Merge-sort

I Quick-sort

We will show that:I O(n · log n) is optimal for comparison based sorting.

Bubble-Sort

The basic idea of bubble-sort is as follows:I exchange neighboring elements that are in wrong order

I stops when no elements were exchanged

bubbleSort(A):n = length(A)

swapped = truewhile swapped == true do

swapped = falsefor i = 0 to n − 2 do

if A[i ] > A[i + 1] thenswap(A[i ],A[i + 1])

swapped = truedonen = n − 1

done

Bubble-Sort: Example

5 4 3 7 1

4 5 3 7 1

4 3 5 7 1

4 3 5 7 1

4 3 5 1 7

3 4 5 1 7

4 3 5 1 7

4 3 1 5 7

3 4 1 5 7

3 1 4 5 7

1 3 4 5 7

Unsorted Part

Sorted Part

Bubble-Sort: Properties

Time complexity:I worst-case:

(n − 1) + . . .+ 1 =(n − 1)2 + n − 1

2∈ O(n2)

(caused by sorting an inverse sorted list)

I best-case: O(n)

Bubble-sort is:I slow

I in-place

Divide-and-Conquer

Divide-and-Conquer is a general algorithm design paradigm:I Divide: divide the input S into two disjoint subsets S1, S2

I Recur: recursively solve the subproblems S1, S2

I Conquer: combine solutions for S1, S2 to a solution for S(the base case of the recursion are problems of size 0 or 1)

Example: merge-sort

7 2 | 9 4 7→ 2 4 7 9

7 | 2 7→ 2 7

7 7→ 7 2 7→ 2

9 | 4 7→ 4 9

9 7→ 9 4 7→ 4

I | indicates the splitting point

I 7→ indicates merging of the sub-solutions

Merge Sort

Merge-sort of a list S with n elements works as follows:I Divide: divide S into two lists S1, S2 of ≈ n/2 elements

I Recur: recursively sort S1, S2

I Conquer: merge S1 and S2 into a sorting of S

Algorithm mergeSort(S,C):Input: a list S of n elements and a comparator COutput: the list S sorted according to C

if size(S) > 1 then(S1,S2) = partition S into size bn/2c and dn/2emergeSort(S1,C)

mergeSort(S2,C)

S = merge(S1,S2,C)

Merging two Sorted Sequences

Algorithm merge(A,B,C):Input: sorted lists A, BOutput: sorted lists containing the elements of A and B

S = empty listwhile ¬A.isEmtpy() or ¬B.isEmtpy() do

if A.first().element < B.first().element thenS.insertLast(A.remove(A.first()))

elseS.insertLast(B.remove(B.first()))

donewhile ¬A.isEmtpy() do S.insertLast(A.remove(A.first()))while ¬B.isEmtpy() do S.insertLast(B.remove(B.first()))

Performance:I Merging two sorted lists of length about n/2 is O(n) time.

(for singly linked lists, double linked lists, and arrays)

Merge-sort: Example

a n e x | a m p l e

a n | e x

a | n

a n

e | x

e x

a m | p l e

a | m

a m

p | l e

p l | e

l e

a n e x a m e l

e l pa e n x

a e l m p

a a e e l m n p x

Divide(split)

Conquer(merge)

Merge-Sort Tree

An execution of merge-sort can be displayed in a binary tree:I each node represents recursive call and stores:

I unsorted sequence before execution, its partition

I sorted sequence after execution

I leaves are calls on subsequences of size 0 or 1

a n e x | a m p l e 7→ a a e e l m n p x

a n | e x 7→ a e n x

a | n 7→ a n

a 7→ a n 7→ n

e | x 7→ e x

e 7→ e x 7→ x

a m | p l e 7→ a e l m p

a | m 7→ a m

a 7→ a m 7→ m

p | l e 7→ e l p

p 7→ p l | e 7→ e l

l 7→ l e 7→ e

Merge-Sort: Example Execution

7 1 2 9 | 6 5 3 8 7→ 1 2 3 5 6 7 8 9

7 1 | 2 9 7→ 1 2 7 9

7 | 1 7→ 1 7

7 7→ 7 1 7→ 1

2 | 9 7→ 2 9

2 7→ 2 9 7→ 9

6 5 | 3 8 7→ 3 5 6 8

6 | 5 7→ 5 6

6 7→ 6 5 7→ 5

3 | 8 7→ 3 8

3 7→ 3 8 7→ 8

Finished merge-sort tree.

Merge-Sort: Running Time

The height h of the merge-sort tree is O(log2 n):I each recursive call splits the sequence in half

The work all nodes together at depth i is O(n):I partitioning, and merging of 2i sequences of n/2i

I 2i+1 ≤ n recursive calls

10 n

21 n/2

2ii n/2i

. . .. . . . . .

depth nodes size

Thus the worst-case running time is O(n · log2 n).

Quick-Sort

Quick-sort of a list S with n elements works as follows:I Divide: pick random element x (pivot) from S and split S in:

I L elements less than x

I E elements equal than x

I G elements greater than x

7 2 1 9 6 5 3 85

I Recur: recursively sort L, and G

2 1 3 5 7 9 6 8

L E G

I Conquer: join L, E , and G

1 2 3 5 6 7 8 9

Quick-Sort: The Partitioning

The partitioning runs in O(n) time:I we traverse S and compare every element y with x

I depending on the comparison insert y in L, E or G

Algorithm partition(S,p):Input: a list S of n elements, and positon p of the pivotOutput: list L, E , G of lists less, equal or greater than pivot

L,E ,G = empty listsx = S.elementAtRank(p)

while ¬S.isEmpty() doy = S.remove(S.first())if y < x then L.insertLast(y)

if y == x then E .insertLast(y)

if y > x then G.insertLast(y)

donereturn L, E , G

Quick-Sort Tree

An execution of quick-sort can be displayed in a binary tree:I each node represents recursive call and stores:

I unsorted sequence before execution, and its pivot

I sorted sequence after execution

I leaves are calls on subsequences of size 0 or 1

1 6 2 9 4 0 7→ 0 1 2 4 6 9

1 2 0 7→ 0 1 2

0 7→ 0 2 7→ 2

6 9 7→ 6 9

9 7→ 9

Quick-Sort: Example

8 2 9 3 1 5 7 6 4 7→ 1 2 3 4 5 6 7 8 9

2 3 1 5 4 7→ 1 2 3 4 5

2 3 5 4 7→ 2 3 4 5

2 7→ 2 5 4 7→ 4 5

4 7→ 4

8 9 7 7→ 7 8 9

7 7→ 7 9 7→ 9

Quick-Sort: Worst-Case Running Time

The worst-case running time occurs when:I the pivot is always the minimal or maximal element

I then one L and G has size n − 1, the other size 0

Then the running time is O(n2):

n + (n − 1) + (n − 2) + . . .+ 1 ∈ O(n2)

n

0 n − 1

0 n − 2

0 ...1

0 0

Quick-Sort: Average Running Time

Consider a recursive call on a list of size m:I Good call: if both L and G are each less then 3

4 · s size

I Bad call: one of L and G is greater than 34 · s size

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

bad calls good calls bad calls

A good call has probability 12 :

I half of the pivots give rise to good calls

Quick-Sort: Average Running Time

For a node at depth i, we expect (average):I i/2 ancestors are good calls

I the size of the sequence is ≤ (3/4)i/2 · nAs a consequence:

I for a node at depth 2 · log3/4 n the expected input size is 1

I the expected height of the quick-sort tree is O(log n)

O(log n)

O(n)

O(n)

O(n)

sa

sb

sd

. . . . . .

se

. . . . . .

sc

sf

. . . . . .

sg

. . . . . .

The amount of work at depth i is O(n).Thus the expected (average) running time is O(n · log n).

In-Place Quick-Sort

Quick-Sort can be sorted in-place (but then non-stable):

Algorithm inPlaceQuickSort(A, l, r):Input: list A, indices l and rOutput: list A where elements from index l to r are sorted

if l ≥ r then returnp = A[r] (take rightmost element as pivot)l ′ = l and r ′ = rwhile l ′ ≤ r ′ do

while l ′ ≤ r ′ and A[l ′] ≤ p do l ′ = l ′ + 1 (find > p)while l ′ ≤ r ′ and A[r ′] ≥ p do r ′ = r ′ − 1 (find < p)if l ′ < r ′ then swap(A[l ′],A[r ′]) (swap < p with > p)

doneswap(A[r],A[l ′]) (put pivot into the right place)inPlaceQuickSort(A, l, l ′ − 1) (sort left part)inPlaceQuickSort(A, l ′ + 1, r) (sort right part)

Considered in-place although recursion needs O(log n) space.

In-Place Quick-Sort: Example

5 8 3 7 1 6

5 1 3 7 8 6

5 1 3 6 8 7

1 5 3 6 8 7

1 3 5 6 8 7

1 3 5 6 8 7

1 3 5 6 8 7

1 3 5 6 7 8

1 3 5 6 7 8

Unsorted Part

Sorted Part

Pivot

Sorting: Lower Bound

Many sorting algorithms are comparison based:I sort by comparing pairs of objects

I Examples: selection-sort, insertion-sort, bubble-sort,heap-sort, merge-sort, quick-sort,. . .

xi < xj ?noyes

No comparison based sorting algorithm can be faster than

Ω(n · log n)

time (worst-case).

We will prove this lower bound on the next slides. . .

Sorting: Lower Bound (Decision Tree)

We will only count comparisons (sufficient for lower bound):I Assume input is a permutation of the numbers 1, 2, . . . , n.

I Every execution corresponds to a path in the decision tree:

xa < xb?

xc < xd?

xg < xh?

. . . . . .

xi < xj?

. . . . . .

xe < xf ?

xk < xl?

. . . . . .

xm < xo?

. . . . . .

I Algorithm itself maybe does not have this tree structure,but this is the maximal information gained by the algorithm.

Sorting: Lower Bound (Leaves)

Every leaf corresponds to exactly one input permutation:I application of the same swapping steps on two different

input permutations, e.g. . . . ,6,. . . ,7,. . . and . . . ,7,. . . ,6,. . . ,yields different results (not both results can be sorted)

xa < xb?

xc < xd?

xg < xh?

. . . . . .

xi < xj?

. . . . . .

xe < xf ?

xk < xl?

. . . . . .

xm < xo?

. . . . . .

Sorting: Lower Bound (Height of the Decision Tree)

The height of the tree is a lower bound on the running time:I There are n! = n · (n − 1) · · · 1 permutations of 1, 2, . . . , n.

I Thus the height of the tree is at least: log2(n!).

n!

log2(n!)

xa < xb?

xc < xd?

xg < xh?

. . . . . .

xi < xj?

. . . . . .

xe < xf ?

xk < xl?

. . . . . .

xm < xo?

. . . . . .

Sorting: Lower Bound

Hence any comparison-based sorting algorithm takes at least

log2(n!) ≥ log2(n/2)n/2

=n2

log2n2

∈ Ω(n · log n)

time in worst-case.

Summary of Comparison-Based Sorting Algorithms

Algorithm Time Notes

selection-sort O(n2) I slow (but good for small lists)I in-place, stableI insertion-sort good for online

sorting and nearly sorted lists

insertion-sort O(n2)

bubble-sort O(n2)

heap-sort O(n · log2 n)I in-place, not stable, fastI good for large inputs (1K − 1M)

merge-sort O(n · log2 n)

I fast, stable, usually not in-placeI sequential data accessI good for large inputs (> 1M)

quick-sort O(n · log2 n)

(expected)I in-place, randomized, not stableI fastest, good for huge inputs

Quick-sort usually performs fastest, although worst-case O(n2).

Sorting: Comparison of Runtime

Algorithm 25.000sorted

100.000sorted

25.000not sorted

100.000not sorted

selection-sort 1.1 19.4 1.1 19.5

insertion-sort 0 0 1.1 19.6

bubble-sort 0 0 5.5 89.8

Algorithm 5 millionsorted

20 millionsorted

5 millionnot sorted

20 millionnot sorted

insertion-sort 0.03 0.13 timeout timeout

heap-sort 3.6 15.6 8.3 42.2

merge-sort 2.5 10.5 3.7 16.1

quick-sort 0.5 2.2 2.0 8.7

I Source: Gumm, Sommer Einführung in die Informatik.