Chapter 5:Algorithms and Heuristics

CS105: Great Insights in Computer Science

The Story So Far....• Introduction to Algorithms via Socks

• Decision Problems on Lists

• Analysis of Algorithms

• Song Growth Rates

• Big O Notation

• Exponential Misuses

• Graphs

The Story So Far....• Introduction to Algorithms via Socks

• Decision Problems on Lists

• Analysis of Algorithms

• Song Growth Rates

• Big O Notation

• Exponential Misuses

• Graphs

What?

The Story So Far....• Introduction to Algorithms via Socks

• Decision Problems on Lists

• Analysis of Algorithms

• Song Growth Rates

• Big O Notation

• Exponential Misuses

• Graphs

What?

How Fast?

The Story So Far....• Introduction to Algorithms via Socks

• Decision Problems on Lists

• Analysis of Algorithms

• Song Growth Rates

• Big O Notation

• Exponential Misuses

• Graphs

What?

How Fast?

Examples!

Still to Come...• Graph Algorithms

• Sorting!

• Uncomputable Things

• Halting Problem

• Huffman Codes

• Robots

Still to Come...• Graph Algorithms

• Sorting!

• Uncomputable Things

• Halting Problem

• Huffman Codes

• Robots

Examples!

Still to Come...• Graph Algorithms

• Sorting!

• Uncomputable Things

• Halting Problem

• Huffman Codes

• Robots

Examples!

Limits!

Still to Come...• Graph Algorithms

• Sorting!

• Uncomputable Things

• Halting Problem

• Huffman Codes

• Robots

Examples!

Limits!

Cool

Applications!

Graphical

A

B C

D E

F

Graphs Everywhere

Graphs Everywhere

NJPA

NY

NH

VT

ME

CT

MA

RI

Graphs Everywhere

NJPA

NY

NH

VT

ME

CT

MA

RIRI

Graphs Everywhere

NJPA

NY

NH

VT

ME

CT

MA

RIRI

Algorithms on Graphs

We can represent a graph in the computer by a list of nodes, and a function that, given a node i, returns the list of nodes to which i is linked.

Reachable• A node j is reachable from a node i if there is a path

that begins at i and ends at j.

• Let’s list all the nodes reachable from i.

• Any node that is reachable from a node that i is linked to is also reachable.

Don’t Revisit!

• What goes wrong? Once we realize we can reach some node, we should mark is as “reached” and never pursue it again.

To-Do List Helps

• Keep track of what you need to do later once your current activity is done.

• Stack. (See ‘paintcan’ video.)

Breadth First Search

A

B C

D E

F

Breadth First Search

A

B C

D E

F

ToDo:

Breadth First Search

A

B C

D E

F

ToDo: A

Breadth First Search

A

B C

D E

F

ToDo: B

Breadth First Search

A

B C

D E

F

ToDo: C,D

Breadth First Search

A

B C

D E

F

ToDo: D,F

Breadth First Search

A

B C

D E

F

ToDo: F,E

Breadth First Search

A

B C

D E

F

ToDo: E

Breadth First Search

A

B C

D E

F

ToDo:

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Depth First

A

B C

D E

F

Back to Google• So, how does Google do it?

I. Web crawl: download known pages, collect links to other pages, repeat.

II. Indexing: Build a giant index that associates each word with a list of pages on which it appears.

III.Distributed search: Use lots and lots and lots of computers to do fast lookups.

Sorting Algorithms• Another name for the lecture is “Google II”.

• Sorting is a great topic in CS:

- relatively simple

- extremely important

- illustrates lots of different algorithms and analysis techniques

There’s more than one way to skin a cat.

Three Pages of Words3 html3 head3 title3 monica3 babes3 rutgers3 title3 head3 frameset3 rows3 frame3 name3 banner3 scrolling3 no3 noresize3 target3 contents3 src3 header3 htm3 frameset3 cols3 frame3 name3 contents3 target3 main3 src3 left3 htm3 frame3 name3 main3 src3 right3 htm3 frameset3 noframes3 body

3 p3 this3 page3 uses3 frames3 but3 your3 browser3 doesn3 t3 support3 them3 p3 body3 noframes3 frameset3 html4 doctype4 html4 public4 w4 c4 dtd4 html4 final4 en4 html4 head4 title4 index4 of4 mlittman4 courses4 cs4 python4 title4 head4 body4 h4 index

4 of4 mlittman4 courses4 cs4 python4 h4 ul4 li4 a4 href4 mlittman4 courses4 cs4 parent4 directory4 a4 li4 a4 href4 add4 py4 add4 py4 a4 li4 a4 href4 differences4 old4 py4 differences4 old4 py4 a4 li4 a4 href4 differences4 py4 differences

4 py4 a4 li4 a4 href4 exception4 py4 exception4 py4 a4 li4 a4 href4 ml4 py4 ml4 py4 a4 li4 a4 href4 ml4 b4 py4 ml4 b4 py4 a4 li4 a4 href4 nim4 nim4 a4 li4 a4 href4 nim4 py4 nim

4 py4 a4 li4 a4 href4 nim4 py4 nim4 py4 a4 li4 a4 href4 nim4 py4 nim4 py4 a4 li4 a4 href4 python4 walkthru4 pdf4 python4 walkthru4 pdf4 a4 li4 a4 href4 socks4 py4 socks4 py4 a4 ul4 address4 apache4 server

4 at4 www4 cs4 rutgers4 edu4 port4 address4 body4 html9 header9 title9 google9 test9 unlinked9 page9 title9 header9 html9 h9 how9 does9 google9 find9 pages9 h9 this9 page9 has9 little9 purpose9 other9 than9 to9 include9 the9 word9 googleblackout9 this9 is9 a

9 word9 sort9 of9 that9 i9 concocted9 on9 january9 and9 verified9 that9 it9 was9 unknown9 to9 google9 your9 search9 b9 googleblackout9 b9 did9 not9 match9 any9 documents9 p9 there9 is9 a9 primary9 page9 http9 www9 cs9 rutgers9 edu9 mlittman9 courses9 cs

9 googletest9 html9 that9 is9 explicitly9 linked9 from9 the9 course9 homepage9 it9 has9 its9 own9 special9 term9 which9 consists9 of9 concatenating9 google9 and9 discovery9 p9 visit9 the9 primary9 page9 for9 the9 search9 history9 p9 this9 page9 is9 part9 of9 a9 href

9 index9 html9 great9 insights9 in9 computer9 science9 a9 html

What Can We Do?• All the information is there, and we can sift

through it.

• But, it’s slow and error-prone to skim through every page every time we want to find something.

• If there are N words (total) on the web pages, how long would it take to sift through them each time? (Use “big O” notation.)

• How can we organize the data to simplify?

Sort, Remove Duplicates4 a9 a4 add4 address9 and9 any4 apache4 at4 b9 b3 babes3 banner3 body4 body3 browser3 but4 c3 cols9 computer9 concatenating9 concocted9 consists3 contents

9 course4 courses9 courses4 cs9 cs9 did4 differences4 directory9 discovery4 doctype9 documents9 does3 doesn4 dtd4 edu9 edu4 en4 exception9 explicitly4 final9 find9 for3 frame

3 frames3 frameset9 from9 google9 googleblackout9 googletest9 great4 h9 h9 has3 head4 head3 header9 header9 history9 homepage9 how4 href9 href3 htm3 html4 html9 html

9 http9 i9 in9 include4 index9 index9 insights9 is9 it9 its9 january3 left4 li9 linked9 little3 main9 match4 ml4 mlittman9 mlittman3 monica3 name4 nim

3 no3 noframes3 noresize9 not4 of9 of4 old9 on9 other9 own3 p9 p3 page9 page9 pages4 parent9 part4 pdf4 port9 primary4 public9 purpose4 py

4 python3 right3 rows3 rutgers4 rutgers9 rutgers9 science3 scrolling9 search4 server4 socks9 sort9 special3 src3 support3 t3 target9 term9 test9 than9 that9 the3 them

9 there3 this9 this3 title4 title9 title9 to4 ul9 unknown9 unlinked3 uses9 verified9 visit4 w4 walkthru9 was9 which9 word4 www9 www3 your9 your

Unsorted / Sorted 262, 201, 918, 301, 187, 762, 397, 277, 645, 306, 765, 798, 689, 867, 276, 402, 124, 545, 907, 569, 259, 152, 399, 481, 977, 947, 774, 727, 292, 285, 173, 599, 464, 212, 147, 696, 242, 559, 155, 569, 806, 784, 415, 321, 820, 126, 469, 225, 646, 438

124, 126, 147, 152, 155, 173, 187, 201, 212, 225, 242, 259, 262, 276, 277, 285, 292, 301, 306, 321, 397, 399, 402, 415, 438, 464, 469, 481, 545, 559, 569, 569, 599, 645, 646, 689, 696, 727, 762, 765, 774, 784, 798, 806, 820, 867, 907, 918, 947, 977

Sorting Helps

• Phonebook, look for a last name vs. look for a first name.

• “Is there a pair that sums to 86?” Don’t have to consider all pairs.

• Is there a repeated number in the list?

• Not to mention min, max, median.

Selection Sort• Idea is quite simple. We go through the list

one item at a time.

• We keep track of the smallest item we’ve found.

• When we’re through the list, we pull the smallest item out and add it to a list of sorted items.

• We repeat until all the items have been removed.

Selection Code

About a 2 1/2 min. to sort 100 items.

Selection Sort Analysis

• How many comparisons does Selection Sort do in the worst case? Assume the list is length N. Hint: What song is it like? You can use “big O” notation.

• Does it matter whether the list is sorted or not?

Other Sorting Approaches

• How else can you imagine sorting?

• Fewer comparisons than O(N2)?

- bubblesort

- counting sort

- insertion sort

- Shell sort

Guess Who?• Each player picks a

character.

• Players take turns asking each other yes/no questions.

• First player to uniquely identify the other player’s character wins!

Mindreader: Set Cards

Mindreader: Set Cards

A BC

D

E F G H

I J K L

M N O P

Cross-Hatched?

Cross-Hatched?

Cross-Hatched?

Squiggle?

Squiggle?

Squiggle?

Insight

Insight• Each question splits

the remaining set of possibilities into two subsets (yes and no).

Insight• Each question splits

the remaining set of possibilities into two subsets (yes and no).

• We want to pick a question so that the larger of the two subsets is as small as possible.

Insight• Each question splits

the remaining set of possibilities into two subsets (yes and no).

• We want to pick a question so that the larger of the two subsets is as small as possible.

• Half!

Insight• Each question splits

the remaining set of possibilities into two subsets (yes and no).

• We want to pick a question so that the larger of the two subsets is as small as possible.

• Half!

• How many questions?

Insight• Each question splits

the remaining set of possibilities into two subsets (yes and no).

• We want to pick a question so that the larger of the two subsets is as small as possible.

• Half!

• How many questions?

• n=1, questions = 0

Insight• Each question splits

the remaining set of possibilities into two subsets (yes and no).

• We want to pick a question so that the larger of the two subsets is as small as possible.

• Half!

• How many questions?

• n=1, questions = 0

• n=2, questions = 1

Insight• Each question splits

the remaining set of possibilities into two subsets (yes and no).

• We want to pick a question so that the larger of the two subsets is as small as possible.

• Half!

• How many questions?

• n=1, questions = 0

• n=2, questions = 1

• n=4, questions = 2

Insight• Each question splits

the remaining set of possibilities into two subsets (yes and no).

• We want to pick a question so that the larger of the two subsets is as small as possible.

• Half!

• How many questions?

• n=1, questions = 0

• n=2, questions = 1

• n=4, questions = 2

• n=8, questions = 3

Insight• Each question splits

the remaining set of possibilities into two subsets (yes and no).

• We want to pick a question so that the larger of the two subsets is as small as possible.

• Half!

• How many questions?

• n=1, questions = 0

• n=2, questions = 1

• n=4, questions = 2

• n=8, questions = 3

• n=16, questions = 4

Insight• Each question splits

the remaining set of possibilities into two subsets (yes and no).

• We want to pick a question so that the larger of the two subsets is as small as possible.

• Half!

• How many questions?

• n=1, questions = 0

• n=2, questions = 1

• n=4, questions = 2

• n=8, questions = 3

• n=16, questions = 4

• n, questions = lg n.

Binary Search

a b c d e f g

Binary Search• Let’s say we have a

sorted list of n items.

a b c d e f g

Binary Search• Let’s say we have a

sorted list of n items.

• How many comparisons do we need to make to find where a new item belongs in the list?

a b c d e f g

Binary Search• Let’s say we have a

sorted list of n items.

• How many comparisons do we need to make to find where a new item belongs in the list?

• Can start at the bottom and compare until the new item is bigger. a b c d e f g

Binary Search• Let’s say we have a

sorted list of n items.

• How many comparisons do we need to make to find where a new item belongs in the list?

• Can start at the bottom and compare until the new item is bigger.

• Maximum number of comparisons?

a b c d e f g

Binary Search• Let’s say we have a

sorted list of n items.

• How many comparisons do we need to make to find where a new item belongs in the list?

• Can start at the bottom and compare until the new item is bigger.

• Maximum number of comparisons?

• One for each position: n.

a b c d e f g

Binary Search• Let’s say we have a

sorted list of n items.

• How many comparisons do we need to make to find where a new item belongs in the list?

• Can start at the bottom and compare until the new item is bigger.

• Maximum number of comparisons?

• One for each position: n.

• We can ask better questions: bigger than the halfway mark?

a b c d e f g

Binary Search• Let’s say we have a

sorted list of n items.

• How many comparisons do we need to make to find where a new item belongs in the list?

• Can start at the bottom and compare until the new item is bigger.

• Maximum number of comparisons?

• One for each position: n.

• We can ask better questions: bigger than the halfway mark?

• That gets us: lg (n+1)!

a b c d e f g

Binary Search Sort• Using O(lg N)

comparisons, can find where to insert the next item.

• Since we insert N items, comparisons is O(N lg N) in total.

• Can’t quite implement it that way, though: Once we find the spot, O(N) to stick it in.

• However, other algorithms are really O(N lg N).

• Hillis mentions Quick Sort and Merge Sort.

Binary Sort Code

About a 1/2 min. to sort 100 items.