1

Analysis of Algorithms

Chapter - 03Sorting Algorithms

2

This Chapter Contains the following Topics:

1. Simple Sortsi. Bubble Sortii. Insertion Sort

2. Heap sorti. Heapsii. The Heap Propertyiii. Maintaining the Heap Propertyiv. Building a Heapv. The Heapsort Algorithmvi. Priority Queues

3

Simple Sorts

4

Sorting ProblemInput: A sequence of n numbers (a1, a2,…, an). Output: A permutation of n numbers (reordering) (a'1, a'2,…, a'n) of the input sequence such that a'1≤ a'2 ≤… ≤ a'n.

Bubble Sort It is a popular sorting algorithmIt swaps adjacent elements that are out of order

Algorithm BubbleSort (A, n) { for i:=1 to n-1 do {

for j := n downto i+1 do{ if ( A[j]<A[j-1]) then

exchange A[j] ↔ A[j-1];}

} }

Illustrate the method. What is the worst-case complexity of Bubble sort?

Bubble Sort

5

Insertion Sort

Insert an element to a sorted array such that the order of the resultant array be not changed.

Algorithm InsertionSort (A, n){ for i:=2 to n do { key:=A[i]; // Insert A[i] into the sorted sequence A[1…i-1]. j:=i-1; while ( (j>0) and (A[j]>key) ) do {

A[j+1]:=A[j]; j:=j-1;

} A[j+1]:=key; }}

What is the worst-case complexity of Insertion sort?

6

Analyzing Simple Sorts

Bubble sort -O(n2)- Very simple code

Insertion sort- Slightly better than bubble sort- Fewer comparisons required.- Also O(n2)

Bubble Sort and Insertion Sort- Use when n is small, (n ~10);- Simple code compensates for low efficiency.

n^2 and n log n

0

500

1000

1500

2000

2500

0 10 20 30 40 50 60

n

Tim

e n log n

n^2

7

Heap sort

8

A heap can be seen as a complete binary tree:

What makes a binary tree complete? Is the example above complete?

16

14 10

8 7 9 3

2 4 1

Heaps

9

A heap can be seen as a complete binary tree:

They are called “nearly complete” or “full” binary trees.

We can think the unfilled slots as null pointers.

16

14 10

8 7 9 3

2 4 1 1 1 111

Heaps (Contd.)

10

In practice, heaps are usually implemented as arrays.

To represent a complete binary tree as an array: The root node is A[1] Node i is A[i] The parent of node i is A[i/2] (note: integer

divide) The left child of node i is A[2i] The right child of node i is A[2i + 1]

For example, the array

can be represented as the following heap.

16

14 10

8 7 9 3

2 4 1

16 14 10 8 7 9 3 2 4 1A =

Heaps (Contd.)

11

Referencing Heap Elements

So…

Parent(i)

{

return i/2; }

Left(i)

{

return 2*i;

}

right(i)

{

return 2*i + 1;

}

12

Heaps also satisfy the max-heap property: A[Parent(i)] A[i] for all nodes i > 1 In other words, the value of a node is at most

the value of its parent. In min-heap,

A[Parent(i)] ≤ A[i] for all nodes i > 1 In other words, the value of a parent node is at

most the value of its child nodes. Where are the largest elements in a max-heap

and min-heap stored? Definitions:

The height of a node in a head = the number of edges on the longest simple downward path from the node to a leaf.

The height of the heap = the height of its root. Since a heap of n-elements is based on a

complete binary tree, its height is Θ(lg n) The basic operations on heaps run in time at

most proportional to the height of the tree and thus take O(lg n).

The Heap Property

13

MaxHeapify(): maintains the max-heap property. Given: a node i in the heap with children l

(left) and r (right). Given: two subtrees rooted at l and r,

assumed to be heaps. Problem: The subtree rooted at i may

violate the heap property (How?) Action: let the value of the parent node

A[i] “float down” so subtree rooted at index i becomes a max-heap.

What do you suppose will be the basic operation between i, l, and r?

Maintaining the Heap Property

14

Algorithm MaxHeapify(A, i)

{

l := Left(i);

r := Right(i);

if ((l <= heap_size(A)) and (A[l] > A[i])) then

largest := l;

else

largest := i;

if ((r <= heap_size(A)) and (A[r] > A[largest])) then

largest := r;

if (largest ≠ i) then

{

exchange A[i]↔A[largest];

MaxHeapify(A, largest);

}

}

The Algorithm MaxHeapify()

15

16

4 10

14 7 9 3

2 8 1 16 4 10 14 7 9 3 2 8 1A =

Illustration by an Example

16

4 10

14 7 9 3

2 8 1 16 10 14 7 9 3 2 8 1A = 4

16

16

4 10

14 7 9 3

2 8 1 16 10 7 9 3 2 8 1A = 4 14

Example (Contd.)

16

14 10

4 7 9 3

2 8 1 16 14 10 4 7 9 3 2 8 1A =

17

16

14 10

4 7 9 3

2 8 1 16 14 10 7 9 3 2 8 1A = 4

Example (Contd.)

16

14 10

4 7 9 3

2 8 1 16 14 10 7 9 3 2 1A = 4 8

18

16

14 10

8 7 9 3

2 4 1 16 14 10 8 7 9 3 2 4 1A =

Example (Contd.)

16

14 10

8 7 9 3

2 4 1 16 14 10 8 7 9 3 2 1A = 4

19

16

14 10

8 7 9 3

2 4 1

16 14 10 8 7 9 3 2 4 1A =

Example (Contd.)

20

Analyzing MaxHeapify()

The running time of the algorithm on a subtree of size n rooted at given node i is the

(1) time to fix up the relationships among the elements A[i], A[l] and A[r], Plus the time to run MaxHeapify() on a subtree rooted at one of the children of node i.

If the heap at i has n elements, how many elements can the subtrees at l or r have?

Answer: 2n/3 The worst case occurs when the last row

of the tree is exactly half full. So time taken by MaxHeapify() is given

by T(n) T(2n/3) + (1)

By case 2 of the Master Theorem,T(n) = O(lg n)

Alternatively, we can characterize the running time of MaxHeapify() on a node of height h as O(h).

21

We can build a heap in a bottom-up manner by running MaxHeapify() on successive subarrays: Fact: for array of length n, all elements in

range A[n/2 + 1 .. n] are heaps (Why?) Walk backwards through the array from

n/2 to 1, calling MaxHeapify() on each node.

Order of processing guarantees that the children of node i are heaps when i is processed.

Algorithm BuildMaxHeap(A){

heap_size(A) := length(A);for i := length[A]/2 downto 1 do

MaxHeapify(A, i);}

Work through the example A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7}

Building a Heap

22

Each call to MaxHeapify() takes O(lg n) time.

There are O(n) such calls (specifically, n/2)

Thus the running time is O(n lg n). This is an upper bound, though correct,

but is not asymptotically tight. Our tighter analysis relies on the

properties that an n-element heap has height lg n, and at most |n/2h+1| of any height h.

The time required by MaxHeapify() when called on a node of height h is O(h).

So we can express the total cost of BuildMaxHeap() as being bounded from above by the following calculation.

Analyzing BuildMaxHeap

23

Analyzing BuildMaxHeap (Contd.)

.

max,

).(22

(),

2)2/11(

2/1

2

,2/1

,1,)1(

.2

)(2

0

lg

0

02

02

lg

0

lg

01

timelinearinarrayunordered

anfromheapabuildcanweHence

nOh

nOh

nO

asboundedbecan

apBuildMaxHeoftimerunningtheThus

has

summationlastthegetwexngSubstituti

xforx

xkxthathaveweSince

hnOhO

n

hh

n

hh

hh

k

k

n

h

n

hhh

24

The Heapsort algorithm starts by using BuildMaxHeap() to build a max-heap on the input array A[1..n], where n=length[A].

The maximum element of the array is stored at the root A[1].

Swap A[1] with element at A[n] to put the maximum element into its correct position.

Then discard the node n from the heap, which will decrement the heap_size[A] by one.

The children of the root remain max-heap, but the new root element may violate the max-heap property.

So, restore the max-heap property at A[1] by calling MaxHeapify(A,1).

This will leaves a max-heap in A[1..(n-1)]. Always swap A[1] for A[heap_size(A)], and

repeat the process for the heap_size(A) down to 2.

Heapsort

25

Algorithm Heapsort(A){

BuildMaxHeap(A);for i := length(A) downto 2 do{

exchange A[1] ↔ A[i];heap_size(A) := heap_size(A) - 1;MaxHeapify(A, 1);

}} Work through the example

A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7} Analyzing the heapsort algorithm

The call to BuildMaxHeap() takes O(n) time. Each of the n - 1 calls to MaxHeapify() takes

O( lg n) time. Thus the total time taken by Heapsort()

= O(n) + (n - 1) O(lg n)= O(n) + O(n lg n)= O(n lg n)

The Heapsort Algorithm

26

Priority Queues

One of the most popular application of a heap is an efficient priority queue.

A priority queue is a data structure for maintaining a set S of elements, each with an associated value called a key.

There are two kinds of priority queues: Max-priority queues and Min-priority queues.

We will focus how to implement max-priority queues, which are based on max-heaps.

A max-priority queue supports the following operations: Insert(S, x): Inserts the element x into the set

S. This operation could be written as S:= S U {x}.

Maximum(S): Returns the element of S with the largest key.

ExtractMax(S): Removes and returns the element of S with the largest key.

IncreaseKey(S, x, k): Increases the value of element x’s key to the new value k, which is assumed to be at least as large as x’s current key value.

27

Priority Queues (Contd.)

Application of max-priority queues: Scheduling jobs on a shared computer. It keeps the track of jobs to be performed and their

relative priorities. When a job is finished or interrupted, the highest priority

job is selected from those pending using ExtractMax(). A new job can be added to the queue at any time using

Insert().

A min-priority queue supports the operations: Insert(), Minimum(), ExtractMin(), and DecreaseKey().

Application of min-priority queues: It can be used as event-driven simulator. The items in the queue are events to be simulated, each

with an associated time of occurrence that serves as its key.

The events must be simulated in order of their time of occurrences, because the simulation of an event can cause other events to be simulated in the future.

The simulation program uses ExtractMin() at each step to choose the next event to simulate.

As new events are produced, they are inserted into the min-priority queue using Insert().

28

Operations

The procedure HeapMaximum() implements the Maximum() operation in Θ(1) time.

Algorithm HeapMaximum(A)

return A[1];

The procedure HeapExtractMax() implements the ExtractMax() operation as follows: Algorithm HeapExtractMax(A)

{

if (heap_size(A) < 1) then

error “heap underflow”;

max := A[1];

A[1] := A[heap_size(A)];

heap_size(A) := heap_size(A) – 1;

MaxHeapify(A, 1);

return max;

}

The running time of HeapExtractMax() is O(lg n), since it performs only a constant amount of work on top of the O(lg n) time for MaxHeapify().

29

Operations (Contd.)

The procedure HeapIncreaseKey() implements the IncreaseKey() operation.

The priority-queue element whose key is to be increased is identified by an index i into the array.

The procedure first updates the key of element A[i] to its new value.

Increasing the key of A[i] may violate the max-heap property.

So, the procedure then traverses a path from this node towards the root to find a proper place for the newly increased key.

During this traversal, it repeatedly compares an element to its parent, exchanging their keys and continuing if the element’s key is larger, and terminating if the element’s key is smaller, since the max-heap property now holds

30

Operations (Contd.)

Algorithm HeapIncreaseKey(A, i, key)

{

if (key < A[i]) then

error “new key is smaller than current key”;

A[i] := key;

while ((i > 1) and (A[Parent(i)] < A[i]) do

{

exchange A[i] ↔ A[Parent(i)];

i := Parent(A);

}

}

Since the path traced from the node updation statement to the root has length O(lg n).

The running time of HeapIncreaseKey() on an n-element heap is O(lg n).

Illustrate the procedure.

31

Illustration

16

14 10

8 7 9 3

2 4 1

(a) The max-heap with a node whose index i is red labeled.

(b) This node has its key updated to 15.

16

14 10

8 7 9 3

2 15 1

i

i

32

Illustration (Contd.)

16

14 10

15 7 9 3

2 8 1(c) After one iteration of the while loop, the node and its parent have exchanged keys, and the index i moves up to the parent.

16

15 10

14 7 9 3

2 8 1

i

i

(d) The max-heap after one more iteration of the while loop. At this points, A[Parent(i)] ≥ A[i]. The max-heap property now holds and the procedure terminates.

33

MaxHeapInsert()

The procedure MaxHeapInsert() implements the Insert() operation.

It takes as an input the key of new element to be inserted into max-heap.

The procedure first expands the max-heap by adding to the tree a new leaf whose key is -∞.

Then it calls HeapIncreaseKey() to set the key of this new node to its correct value and maintain the max-heap property.

Algorithm MaxHeapInsert(A, key)

{

heap_size(A) := heap_size(A) + 1;

A[heap_size(A)] := -∞;

HeapIncreaseKey(A, heap_size(A), key);

} The running time of MaxHeapInsert() on an n-

element heap is O(lg n). In summary, a heap can support any priority-

queue operation on a set of size n in O(lg n) time.

34

End of

Chapter - 03