Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 9

Analysis of AlgorithmsCS 477/677

Instructor: Monica Nicolescu

Lecture 9

CS 477/677 - Lecture 9

A Job Scheduling Application

• Job scheduling

– The key is the priority of the jobs in the queue

– The job with the highest priority needs to be executed

next

• Operations

– Insert, remove maximum

• Data structures

– Priority queues

– Ordered array/list, unordered array/list

CS 477/677 - Lecture 9

PQ Implementations & Cost

Worst-case asymptotic costs for a PQ with N items

Insert Remove max

ordered array

ordered list

unordered array

unordered list

N

N

N

N

1

1

1

1

Can we implement both operations efficiently?

CS 477/677 - Lecture 9

Background on Trees

• Def: Binary tree = structure composed of a finite set of nodes that either:– Contains no nodes, or– Is composed of three disjoint sets of nodes: a root

node, a left subtree and a right subtree

2

14 8

1

16

4

3

9 10

root

Right subtreeLeft subtree

CS 477/677 - Lecture 9

Special Types of Trees

• Def: Full binary tree = a binary tree in which each node is either a leaf or has degree (number of children) exactly 2.

• Def: Complete binary tree = a binary tree in which all leaves have the same depth and all internal nodes have degree 2.

Full binary tree

2

14 8

1

16

7

4

3

9 10

12

Complete binary tree

2

1

16

4

3

9 10

CS 477/677 - Lecture 9

The Heap Data Structure

• Def: A heap is a nearly complete binary tree with the following two properties:– Structural property: all levels are full, except

possibly the last one, which is filled from left to right– Order (heap) property: for any node x

Parent(x) ≥ x

Heap

5

7

8

4It doesn’t matter that 4 in

level 1 is smaller than 5 in

level 22

CS 477/677 - Lecture 9

Definitions

• Height of a node = the number of edges on a longest simple path from the node down to a leaf

• Depth of a node = the length of a path from the root to the node

• Height of tree = height of root node

= lgn, for a heap of n elements

2

14 8

1

16

4

3

9 10

Height of root = 3

Height of (2)= 1 Depth of (10)= 2

CS 477/677 - Lecture 9

Array Representation of Heaps

• A heap can be stored as an

array A.– Root of tree is A[1]

– Left child of A[i] = A[2i]

– Right child of A[i] = A[2i + 1]

– Parent of A[i] = A[ i/2 ]– Heapsize[A] ≤ length[A]

• The elements in the subarray

A[(n/2+1) .. n] are leaves

• The root is the maximum

element of the heap

CS 477/677 - Lecture 9

Heap Types

• Max-heaps (largest element at root), have the

max-heap property: – for all nodes i, excluding the root:

A[PARENT(i)] ≥ A[i]

• Min-heaps (smallest element at root), have the

min-heap property:– for all nodes i, excluding the root:

A[PARENT(i)] ≤ A[i]

CS 477/677 - Lecture 9

Operations on Heaps

• Maintain the max-heap property– MAX-HEAPIFY

• Create a max-heap from an unordered array– BUILD-MAX-HEAP

• Sort an array in place– HEAPSORT

• Priority queue operations

CS 477/677 - Lecture 9

Operations on Priority Queues

• Max-priority queues support the following

operations:

– INSERT(S, x): inserts element x into set S

– EXTRACT-MAX(S): removes and returns element of

S with largest key

– MAXIMUM(S): returns element of S with largest key

– INCREASE-KEY(S, x, k): increases value of element

x’s key to k (assume k ≥ current key value at x)

CS 477/677 - Lecture 9

Building a Heap

Alg: BUILD-MAX-HEAP(A)

1. n = length[A]

2. for i ← n/2 downto 1

3. do MAX-HEAPIFY(A, i, n)

• Convert an array A[1 … n] into a max-heap (n = length[A])

• The elements in the subarray A[(n/2+1) .. n] are

leaves

• Apply MAX-HEAPIFY on elements between 1 and n/2

2

14 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 10

4 1 3 2 16 9 10 14 8 7A:

CS 477/677 - Lecture 9

Example: A 4 1 3 2 16 9 10 14 8 7

2

14 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 10

14

2 8

1

16

7

4

10

9 3

1

2 3

4 5 6 7

8 9 10

2

14 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 1014

2 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 10

14

2 8

16

7

1

4

10

9 3

1

2 3

4 5 6 7

8 9 108

2 4

14

7

1

16

10

9 3

1

2 3

4 5 6 7

8 9 10

i = 5 i = 4 i = 3

i = 2 i = 1

CS 477/677 - Lecture 9

Correctness of BUILD-MAX-HEAP• Loop invariant:

– At the start of each iteration of the for loop, each node i + 1, i + 2,…, n is the root of a max-heap

• Initialization:– i = n/2: Nodes n/2+1, n/2+2, …, n are leaves

they are the root of trivial max-heaps

2

14 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 10

CS 477/677 - Lecture 9

Correctness of BUILD-MAX-HEAP• Maintenance:

– MAX-HEAPIFY makes node i a max-heap root and preserves the property that nodes i + 1, i + 2, …, n are roots of max-heaps

– Decrementing i in the for loop reestablishes the loop invariant

• Termination:– i = 0 each node 1, 2, …, n is the root of

a max-heap (by the loop invariant)

2

14 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 10

CS 477/677 - Lecture 9

Running Time of BUILD MAX HEAP

It would seem that running time is O(nlgn)

• This is not an asymptotically tight upper bound

Alg: BUILD-MAX-HEAP(A)

1. n = length[A]

2. for i ← n/2 downto 1

3. do MAX-HEAPIFY(A, i, n) O(lgn)O(n)

CS 477/677 - Lecture 9

Running Time of BUILD MAX HEAP

• HEAPIFY takes O(h) the cost of HEAPIFY on a node i is proportional to the height of the node i in the tree

Height Level

h0 = 3 (lgn)

h1 = 2

h2 = 1

h3 = 0

i = 0

i = 1

i = 2

i = 3 (lgn)

No. of nodes

20

21

22

23

hi = h – i height of the heap rooted at level ini = 2i number of nodes at level i

i

h

iihnnT

0

)( ihh

i

i 0

2 )(nO

CS 477/677 - Lecture 9

Running Time of BUILD MAX HEAP

i

h

iihnnT

0

)( Cost of HEAPIFY at level i number of nodes at that level

ihh

i

i 0

2 Replace the values of ni and hi computed before

hh

iih

ih2

20

Multiply by 2h both at the nominator and denominator and

write 2i as i2

1

h

kk

h k

0 22 Change variables: k = h - i

0 2k

k

kn The sum above is smaller than the sum of all elements to

and h = lgn

)(nO The sum above is smaller than 2

Running time of BUILD-MAX-HEAP: T(n) = O(n)

CS 477/677 - Lecture 9

Binary Search Trees• Tree representation:

– A linked data structure in which each node is an object

• Node representation:– Key field– Satellite data– Left: pointer to left child– Right: pointer to right child– p: pointer to parent (p [root [T]] =

NIL)

• Satisfies the binary search tree property

Left child Right child

L Rparent

key data

CS 477/677 - Lecture 9

Binary Search Tree Example

• Binary search tree property:

– If y is in left subtree of x,

then key [y] ≤ key [x]

– If y is in right subtree of x,

then key [y] ≥ key [x]

2

3

5

5

7

9

CS 477/677 - Lecture 9

Binary Search Trees

• Support many dynamic set operations

– SEARCH, MINIMUM, MAXIMUM, PREDECESSOR,

SUCCESSOR, INSERT, DELETE

• Running time of basic operations on binary

search trees

– On average: (lgn)• The expected height of the tree is lgn

– In the worst case: (n)• The tree is a linear chain of n nodes

CS 477/677 - Lecture 9

Red-Black Trees

• “Balanced” binary trees guarantee an O(lgn) running time on the basic dynamic-set operations

• Red-black tree– Binary tree with an additional attribute for its nodes:

color which can be red or black– Constrains the way nodes can be colored on any path

from the root to a leaf• Ensures that no path is more than twice as long as another

the tree is balanced

– The nodes inherit all the other attributes from the binary-search trees: key, left, right, p

CS 477/677 - Lecture 9

Red-Black Trees Properties

1. Every node is either red or black

2. The root is black

3. Every leaf (NIL) is black

4. If a node is red, then both its children are black• No two red nodes in a row on a simple path from the

root to a leaf

5. For each node, all paths from the node to

descendant leaves contain the same number of

black nodes

CS 477/677 - Lecture 9

Example: RED-BLACK TREE

• For convenience we use a sentinel NIL[T] to represent all the NIL nodes at the leafs– NIL[T] has the same fields as an ordinary node– Color[NIL[T]] = BLACK– The other fields may be set to arbitrary values

26

17 41

30 47

38 50

NIL NIL

NIL

NIL NIL NIL NIL

NIL

CS 477/677 - Lecture 9

Black-Height of a Node

• Height of a node: the number of edges in a longest path to a leaf

• Black-height of a node x: bh(x) is the number of black nodes (including NIL) on a path from x to leaf, not counting x

26

17 41

30 47

38 50

NIL NIL

NIL

NIL NIL NIL NIL

NIL

h = 4bh = 2

h = 3bh = 2

h = 2bh = 1

h = 1bh = 1

h = 1bh = 1

h = 2bh = 1 h = 1

bh = 1

CS 477/677 - Lecture 9

Properties of Red-Black Trees• Claim

– Any node with height h has black-height ≥ h/2• Proof

– By property 4, there are at most h/2 red nodes on the path from the node to a leaf

– Hence at least h/2 are black26

17 41

30 47

38 50

Property 4: if a node is red then both its children are black

CS 477/677 - Lecture 9

Properties of Red-Black Trees

Claim

The subtree rooted at any node x contains at least

2bh(x) - 1 internal nodes

Proof: By induction on height of x

Basis: height[x] = 0

x is a leaf (NIL[T])

bh(x) = 0

# of internal nodes: 20 - 1 = 0

NIL

x

CS 477/677 - Lecture 9

Properties of Red-Black TreesInductive step: • Let height(x) = h and bh(x) = b• Any child y of x has:

– bh (y) =– bh (y) =

26

17 41

30 47

38 50

b (if the child is red), or

b - 1 (if the child is black)

CS 477/677 - Lecture 9

Properties of Red-Black Trees• Want to prove:

– The subtree rooted at any node x contains at

least 2bh(x) - 1 internal nodes

• Assume true for children of x: – Their subtrees contain at least 2bh(x) - 1 – 1

internal nodes

• The subtree rooted at x contains at least:

(2bh(x) - 1 – 1) + (2bh(x) - 1 – 1) + 1 =

2 · (2bh(x) - 1 - 1) + 1 =

2bh(x) - 1 internal nodes

x

l r

CS 477/677 - Lecture 9

Properties of Red-Black Trees

Lemma: A red-black tree with n internal nodes has height at most 2lg(n + 1).

Proof:

n

• Add 1 to all sides and then take logs:

n + 1 ≥ 2b ≥ 2h/2

lg(n + 1) ≥ h/2 h ≤ 2 lg(n + 1)

root

l r

height(root) = hbh(root) = b

number n of internal nodes

≥ 2b - 1 ≥ 2h/2 - 1

since b h/2

CS 477/677 - Lecture 9

Operations on Red-Black Trees

• The non-modifying binary-search tree operations

MINIMUM, MAXIMUM, SUCCESSOR,

PREDECESSOR, and SEARCH run in O(h) time

– They take O(lgn) time on red-black trees

• What about TREE-INSERT and TREE-DELETE?

– They will still run in O(lgn)

– We have to guarantee that the modified tree will still be

a red-black tree

CS 477/677 - Lecture 9

INSERT

INSERT: what color to make the new node?• Red? Let’s insert 35!

– Property 4: if a node is red, then both its children are black

• Black? Let’s insert 14!– Property 5: all paths from a node to its leaves contain

the same number of black nodes

26

17 41

30 47

38 50

CS 477/677 - Lecture 9

DELETE

DELETE: what color was the

node that was removed? Red?

1. Every node is either red or black

2. The root is black

3. Every leaf (NIL) is black

4. If a node is red, then both its children are black

5. For each node, all paths from the node to descendant

leaves contain the same number of black nodes

OK!

OK!

OK!

OK! Does not createtwo red nodes in a row

OK! Does not changeany black heights

26

17 41

30 47

38 50

CS 477/677 - Lecture 9

DELETE

DELETE: what color was the

node that was removed? Black?

1. Every node is either red or black

2. The root is black

3. Every leaf (NIL) is black

4. If a node is red, then both its children are black

5. For each node, all paths from the node to descendant

leaves contain the same number of black nodes

OK!

OK!

Not OK! Could createtwo red nodes in a row

Not OK! Could change theblack heights of some nodes

26

17 41

30 47

38 50

Not OK! If removing the root and the child that replaces it is red

CS 477/677 - Lecture 9

Rotations

• Operations for restructuring the tree after insert

and delete operations on red-black trees

• Rotations take a red-black tree and a node within

the tree and:– Together with some node re-coloring they help restore

the red-black tree property

– Change some of the pointer structure

– Do not change the binary-search tree property

• Two types of rotations:– Left & right rotations

CS 477/677 - Lecture 9

Left Rotations• Assumption for a left rotation on a node x:

– The right child of x (y) is not NIL

• Idea:– Pivots around the link from x to y– Makes y the new root of the subtree– x becomes y’s left child– y’s left child becomes x’s right child

CS 477/677 - Lecture 9

Readings

• Chapter 8, Chapter 6