Analysis of AlgorithmsCS 477/677

Instructor: Monica NicolescuLecture 11

CS 477/677 - Lecture 11 2

INTEL Nervana AI Academy• Webcast: October 20, register on-line• Fundamentals of Machine Learning, Deep

Learning & Artificial Intelligence focused on newly optimized frameworks for Intel Architecture like Neon, Caffe, and TensorFlow from our technical experts.– Machine Learning & Deep Learning Fundamentals– Applications in Real Life– Deep Learning Examples, like Convolutional Neural

Networks (CNN) for Image Recognition– How Intel plans to help developers to improve

performance of Machine Learning workloads – What frameworks are optimized for Intel Architecture,

and how you can get access to them

CS 477/677 - Lecture 11 3

Red-Black Trees• “Balanced” binary trees guarantee an

O(lgn) running time on the basic dynamic-set operations

• Red-black tree– Binary tree with an additional attribute for its

nodes: color which can be red or black– Constrains the way nodes can be colored on

any path from the root to a leaf• Ensures that no path is more than twice as long as

another ⇒ the tree is balanced– The nodes inherit all the other attributes from

the binary-search trees: key, left, right, p

CS 477/677 - Lecture 11 4

Red-Black Trees Properties1. Every node is either red or black2. The root is black3. Every leaf (NIL) is black4. If a node is red, then both its children are

black• No two red nodes in a row on a simple path from

the root to a leaf5. For each node, all paths from the node to

descendant leaves contain the same number of black nodes

CS 477/677 - Lecture 11 5

Black-Height of a Node

• Height of a node: the number of edges in a longest path to a leaf

• Black-height of a node x: bh(x) is the number of black nodes (including NIL) on a path from x to leaf, not counting x

26

17 41

30 47

38 50

NIL NIL

NIL

NIL NIL NIL NIL

NIL

h = 4bh = 2

h = 3bh = 2

h = 2bh = 1

h = 1bh = 1

h = 1bh = 1

h = 2bh = 1 h = 1

bh = 1

CS 477/677 - Lecture 11 6

Augmenting Data Structures• Let’s look at two new problems:

– Dynamic order statistic– Interval search

• It is unusual to have to design all-new data structures from scratch– Typically: store additional information in an already known

data structure– The augmented data structure can support new

operations• We need to correctly maintain the new information

without loss of efficiency

CS 477/677 - Lecture 11 7

Dynamic Order Statistics• Def.: the i-th order statistic of a set of n elements,

where i ∈ {1, 2, …, n} is the element with the i-th smallest key.

• We can retrieve an order statistic from an unordered set:– Using: – In:

• We will show that:– With red-black trees we can achieve this in O(lgn)– Finding the rank of an element takes also O(lgn)

RANDOMIZED-SELECTO(n) time

CS 477/677 - Lecture 11 8

Order-Statistic Tree• Def.: Order-statistic tree: a red-black

tree with additional information stored in each node

• Node representation:– Usual fields: key[x], color[x], p[x], left[x],

right[x]– Additional field: size[x] that contains the

number of (internal) nodes in the subtree rooted at x (including x itself)

• For any internal node of the tree:size[x] =

size[left[x]] + size[right[x]] + 1

CS 477/677 - Lecture 11 9

Example: Order-Statistic Tree

710

3 8

315

1 4

1 9

111

119

key

size

CS 477/677 - Lecture 11 10

OS-SELECTGoal:• Given an order-statistic tree, return a pointer to

the node containing the i-th smallest key in the subtree rooted at x

Idea:• size[left[x]] = the number of nodesthat are smaller than x• rank’[x] = size[left[x]] + 1

in the subtree rooted at x• If i = rank’[x] Done!• If i < rank’[x]: look left• If i > rank’[x]: look right

710

3 8

315

1 4

1 9

111

119

CS 477/677 - Lecture 11 11

OS-SELECT(x, i)1. r ← size[left[x]] + 12. if i = r3. then return x4. elseif i < r5. then return OS-SELECT(left[x], i)6. else return OS-SELECT(right[x], i - r)

Initial call: OS-SELECT(root[T], i)Running time: O(lgn)

► compute the rank of x within the subtree rooted at x

12CS 477/677 - Lecture 11

Example: OS-SELECT

710

3 8

315

1 4

1 9

111

119

OS-SELECT(root, 3)r = 3 + 1 = 4OS-SELECT(‘8’, 3)

r = 1 + 1 = 2OS-SELECT(‘9’, 1)

r = 1Found!

CS 477/677 - Lecture 11 13

OS-RANKGoal:• Given a pointer to a node x in an

order-statistic tree, return the rank of x in the linear order determined by an inorder walk of T

710

3 8

315

1 4

1 9

111

119

Idea: • Add elements in the leftsubtree• Go up the tree and if a right child: add the elements in the leftsubtree of the parent + 1

x

The elements in the left subtree

Its parent plus the leftsubtree if x is a right child

CS 477/677 - Lecture 11 14

OS-RANK(T, x)1. r ← size[left[x]] + 12. y ← x3. while y ≠ root[T]4. do if y = right[p[y]]5. then r ← r + size[left[p[y]]]

+ 16. y ← p[y]7. return r

Add to the rank the elements inits left subtree + 1 for itselfSet y as a pointer that will traverse the tree

If a right child add the size of the parent’s left subtree + 1 for the parentRunning time: O(lgn)

CS 477/677 - Lecture 11 15

Example: OS-RANK2620

104

121

162

201

211

192

214

147

1712

72

31

281

351

383

391

471

305

417

141

xr = 1 + 1 = 2

y = x

r = r + 1 + 1 = 4y

r = r = 4y

y = root[T]r = r + 12 + 1 = 17

CS 477/677 - Lecture 11 16

Maintaining Subtree Sizes

• We need to maintain the size field during

INSERT and DELETE operations

• Need to maintain them efficiently

• Otherwise, might have to recompute all

size fields, at a cost of (n)𝝮

CS 477/677 - Lecture 11 17

Maintaining Size for OS-INSERT

• Insert in a red-black tree has two stages

1. Perform a binary-search tree insert

2. Perform rotations and change node colors to

restore red-black tree properties

CS 477/677 - Lecture 11 18

OS-INSERTIdea for maintaining the size field during

insert

710

3 8

315

1 4

1 9

111

119

Phase 1 (going down):• Increment size[x] for

each node x on the traversed path from the root to the leaves

• The new node gets a size of 1

• Constant work at each node, so still O(lgn)

19CS 477/677 - Lecture 11

OS-INSERTIdea for maintaining the size field during

insertPhase 2 (going up):

– During RB-INSERT-FIXUP there are:• O(lgn) changes in node colors• At most two rotations

Rotations affect the subtree sizes !!

1942

1293

6

4 7

x

y

93

7

y

42 x

6 4

LEFT-ROTATE(T, x)

size[x] = size[left[x]] +size[right[x]] + 1

size[y] = size[x]19

11

Maintain size in: O(lgn)

CS 477/677 - Lecture 11 20

Augmenting a Data Structure1. Choose an underlying data structure

2. Determine additional information to maintain

3. Verify that we can maintain additional information for existing data structure operations

4. Develop new operations

⇒ Red-black trees

⇒ size[x]

⇒ Shown how to maintain size during modifying operations

⇒ Developed OS-RANK and OS-SELECT

CS 477/677 - Lecture 11 21

Augmenting Red-Black TreesTheorem: Let f be a field that augments a

red-black tree. If the contents of f for a node can be computed using only the information in x, left[x], right[x] ⇒ we can maintain the values of f in all nodes during insertion and deletion, without affecting their O(lgn) running time.

CS 477/677 - Lecture 11 22

Examples1. Can we augment a RBT with size[x]?

Yes: size[x] = size[left[x]] + size[right[x]] + 1

2. Can we augment a RBT with height[x]?Yes: height[x] = 1 + max(height[left[x]],

height[right[x]])

3. Can we augment a RBT with rank[x]?No, inserting a new minimum will cause all n rank values to change

23

Interval TreesDef.: Interval tree = a red-black tree that

maintains a dynamic set of elements, each element x having associated an interval int[x].

• Operations on interval trees:– INTERVAL-INSERT(T, x) – INTERVAL-DELETE(T, x)– INTERVAL-SEARCH(T, i)

CS 477/677 - Lecture 11

24

Interval Properties• Intervals i and j overlap iff:

low[i] ≤ high[j] and low[j] ≤ high[i]

• Intervals i and j do not overlap iff:high[i] < low[j] or high[j] < low[i]

ij

ij

ij

ij

i j j i

CS 477/677 - Lecture 11

25

Interval Trichotomy

• Any two intervals i and j satisfy the

interval trichotomy: exactly one of the

following three properties holds:

a) i and j overlap,

b) i is to the left of j (high[i] < low[j])

c) i is to the right of j (high[j] < low[i])CS 477/677 - Lecture 11

26

Designing Interval Trees1. Underlying data structure

– Red-black trees– Each node x contains: an interval

int[x], and the key: low[int[x]]– An inorder tree walk will list intervals

sorted by their low endpoint2. Additional information

– max[x] = maximum endpoint value in subtree rooted at x

3. Maintaining the information

max[x] =

Constant work at each node, so still O(lgn) time

high[int[x]]max max[left[x]]

max[right[x]]

CS 477/677 - Lecture 11

27

Designing Interval Trees4. Develop new operations• INTERVAL-SEARCH(T, i):

– Returns a pointer to an element x in the interval tree T, such that int[x] overlaps with i, or NIL otherwise

• Idea:• Check if int[x]

overlaps with i• Max[left[x]] ≥ low[i]

– Go left• Otherwise, go right

[16, 21]30

[25, 30]30

[26, 26]26

[17, 19]20

[19, 20]20

[8, 9]23

[15, 23]23

[5, 8]10

[6, 10]10

[0, 3]3

[22, 25]

highlow

CS 477/677 - Lecture 11

28

Example

[16, 21]30

[25, 30]30

[26, 26]26

[17, 19]20

[19, 20]20

[8, 9]23

[15, 23]23

[5, 8]10

[6, 10]10

[0, 3]3

i = [11, 14] x

xx

x = NIL

i = [22, 25]

CS 477/677 - Lecture 11

29

INTERVAL-SEARCH(T, i)1. x ← root[T]2. while x ≠ nil[T] and i does not overlap

int[x]3. do if left[x] ≠ nil[T] and

max[left[x]] ≥ low[i]4. then x ← left[x]5. else x ← right[x]6. return x CS 477/677 - Lecture 11

30

TheoremAt the execution of interval search: if

the search goes right, then either:– There is an overlap in right subtree, or– There is no overlap in either subtree

• Similar when the search goes left• It is safe to always proceed in only

one direction

CS 477/677 - Lecture 11

31

Theorem• Proof: If search goes right:

– If there is an overlap in right subtree, done– If there is no overlap in right ⇒ show there is

no overlap in left– Went right because:

left[x] = nil[T] ⇒ no overlap in left, ormax[left[x]] < low[i] ⇒ no overlap in left

i

max[left[x]]

low[x]

CS 477/677 - Lecture 11

32

Theorem - ProofIf search goes left:• If there is an overlap in left subtree, done• If there is no overlap in left, show there is no overlap in

right• Went left because:

low[i] ≤ max[left[x]] = high[j] for some j in left subtree int[root]

max

[low[j], high[j]]max[j]

[low[k], high[k]]max[k]

high[j] < low[i]high[i] < low[j]

i ij

low[j] < low[k]

kNo overlap!high[i] < low[k]

max[left]

CS 477/677 - Lecture 11

CS 477/677 - Lecture 11 33

Readings

• Chapters 14, 15