ADVANCED ALGORITHM ANALYSIS LECTURE # 1 INTRODUCTION-H euristic Algorithms

ADVANCED ALGORITHM ANALYSIS

LECTURE # 1

INTRODUCTION-Heuristic Algorithms

1

ADVANCED ALGORITHM ANALYSIS

1.Introduction 2.Basic concept of Heuristic algorithms.

2.1.Example of Heuristic algorithms.2.2The Traveling Salesman Problem.2.3Traveling Salesman Problem algorithm with example.

2

Where We're Going Learn general approaches to algorithm desi

gn Divide and conquer Greedy method Dynamic Programming Basic Search and Traversal TechniqueGraph TheoryLinear ProgrammingApproximation Algorithm NP Problem

3

Where We're Going Examine methods of analyzing algorithm correctness and

efficiencyRecursion equationsLower bound techniquesO,Omega and Theta notations for best/worst/average ca

se analysis Decide whether some problems have no solution in reaso

nable timeList all permutations of n objects (takes n! steps)Travelling salesman problem

Investigate memory usage as a different measure of efficiency

4

Algorithm. (webster.com) A procedure for solving a mathematical problem (as of finding the

greatest common divisor) in a finite number of steps that frequently involves repetition of an operation.

Broadly: a step-by-step procedure for solving a problem or accomplishing some end especially by a computer.

"Great algorithms are the poetry of computation."

Etymology: "algos" = Greek word for pain. "algor" = Latin word for to be cold.

Abu Ja’far al-Khwarizmi’s = 9th century Arab scholar.his book "Al-Jabr wa-al-Muqabilah" evolved into today’s

high school algebra text 5

Example How to make a coffee with a coffee

machine:1. Open the coffee machine2. Put water in it.3. Close it4. Open the coffee container (hope it's called like that)5. Fill it with finely ground coffee.6. Close it7. Turn the machine on8. Wait 3-4 minutes9. Open it 10. Pour the coffee.11. etc. 6

Imagine: A World With No Algorithms

Fast arithmetic.Cryptography.

Quick sort. Databases.

Signal processing. Huffman codes.

Data compression. Network flow.

Routing Internet packets. Linear programming.

Planning, decision-making. 7

What is an Algorithm?- Anany Levitin An algorithm is a sequence of

unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

8

What IS A Computer Algorithm?

A computer algorithm isa detailed step-by-step method forsolving a problemby using a computer.

9

Which algorithm is better?

The algorithms are correct, but which is the best?

Measure the running time (number of operations needed).

Measure the amount of memory used.

Note that the running time of the algorithms increase as the size of the input increases.

10

Importance of Analyze Algorithm

Need to recognize limitations of various algorithms for solving a problem.

Need to understand relationship between problem size and running time.When is a running program not good enough?

Need to learn how to analyze an algorithm's running time without coding it.

Need to learn techniques for writing more efficient code.

Need to recognize bottlenecks in code as well as which parts of code are easiest to optimize.

11

WhY do we analyze about them? understand their behavior, and (Job -- Selection, performance, modif

y) improve them. (Research) Correctness

Does the input/output relation match algorithm requirement?

Amount of work done ( complexity) Basic operations to do task

Amount of space usedMemory used

Simplicity, clarityVerification and implementation.

Optimality Is it impossible to do better? 12

ALGORITHM CLASSIFICATION There are various ways to classify algorithms: 1. Classification by implementation :

Recursion or iteration: Logical: Serial or parallel or distributed: Deterministic or non-deterministic: Exact or approximate:

2.Classification by Design Paradigm : Divide and conquer. Dynamic programming. The greedy method. Linear programming. Reduction. Search and enumeration. The probabilistic and heuristic paradigm.

13

14

Solution Methods

I. Try every possibility (n-1)! possibilities – grows faster than exponentially

If it took 1 microsecond to calculate each possibility takes 10140 centuries to calculate all possibilities when n = 100

II. Optimising Methods obtain guaranteed optimal solution, but can take a

very, very, long time

III. Heuristic Methods obtain ‘good’ solutions ‘quickly’ by intuitive methods.

No guarantee of optimality.

Heuristic Algorithms The term heuristic is used for algorithms which find

solutions among all possible ones ,but they do not guarantee that the best will be found, therefore they may be considered as approximately and not accurate algorithms.

These algorithms, usually find a solution close to the best one and they find it fast and easily.

Sometimes these algorithms can be accurate, that is they actually find the best solution, but the algorithm is still called heuristic until this best solution is proven to be the best .

EXAMPLE: Heuristic algorithm for the Traveling Salesman Problem (T.S.P) .

15

16

A Salesman wishes to travel around a given set of cities, and return to the beginning, covering the smallest total distance.

Easy to State

Difficult to Solve

Traveling Salesman Problem

Traveling Salesman Problem (T.S.P)

William Rowan HamiltonThis is one of the most known problems, and is often called as a difficult problem. A salesman must visit n cities, passing through each city only once, beginning from one of them which is considered as his base, and returning to it. The cost of the transportation among the cities (whichever combination possible) is given. The program of the journey is requested, that is the order of visiting the cities in such a way that the cost is the minimum.

17


Let's number the cities from 1 to n , and let city 1 be the city-base of the salesman. Also let's assume that c(i, j) is the visiting

cost from i to j. There can be c(i, j)<>c(j, i).Apparently all the possible solutions are (n-1)!.Someone could probably determine them

systematically, find the cost for each and every one of these solutions and finally keep the one with the minimum cost.

These require at least (n-1)! steps.18


If for example there were 21 cities the steps required are (n-1)!=(21-1)!=20! steps.

If every step required a msec we would need about 770 centuries of calculations.

Apparently, the exhausting examination of all possible solutions is out of the question. Since we are not aware of any other quick algorithm that finds a best solution we will use a heuristic algorithm. 19


According to this algorithm whenever the salesman is in town i he chooses as his next city, the city j for which the c(i,j) cost, is the minimum among all c(i,k) costs, where k are the pointers of the city the salesman has not visited yet.

There is also a simple rule just in case more than one cities give the minimum cost, for example in such a case the city with the smaller k will be chosen.

This is a greedy algorithm which selects in every step the cheapest visit and does not care whether this will lead to a wrong result or not.

20

Traveling Salesman Problem AlgorithmT.S.P. algorithm:Input : Number of cities n and array of costs c(i,j) i, j=1,..n (We begin from city number 1)Output : Vector of cities and total cost.(* starting values *)C=0cost=0visits=0e=1 (*e=pointer of the visited city)(* determination of round and cost)for r=1 to n-1 do

choose of pointer j with minimum=c(e,j)=min{c(e,k);visits(k)=0 and k=1,..,n} cost=cost+ minimum e=j C(r)=j

end r-loopC(n)=1cost=cost+ c(e,1)

21

Traveling Salesman Problem Algorithm

We can find situations in which the TSP algorithm don't give the best solution.

We can also succeed on improving the algorithm.

For example we can apply the algorithm t times for t different cities and keep the best round every time.

We can also unbend the greeding in such a way to reduce the algorithm problem ,that is there is no room for choosing cheep sides at the end of algorithm because the cheapest sides have been exhausted.

22

23

If there is no condition to return to the beginning. It can still be regarded as a TSP.

Suppose we wish to go from A to B visiting all cities.

AB

24


Connect A and B to a ‘dummy’ city at zero distance(If no stipulation of start and finish cities connect all to dummy at zero distance)

AB

25


Create a TSP Tour around all cities

AB

26

A route returning to the beginning is known as a Hamiltonian Circuit

A route not returning to the beginning is known as a Hamiltonian Path

Essentially the same class of problem

Hamilton CircuitsEuler circuit/path => Visit each edge once

and only onceHamilton circuit => Visit each vertex once

and only once (except at the end, where it returns to the starting vertex)

Hamilton path => Visit each vertex once and only once

Difference: Edge (Euler) Vertex (Hamilton)

27

Examples of Hamilton circuits

A B

CD

E

Has many Hamilton circuits:1)A, B, C, D, E, A2)A, D, C, E, B, A

Has many Hamilton paths:1)A, B, C, D, E2)A, D, C, E, B

Has no Euler circuit, no Euler path => 4 vertices of odd degree

Hamilton circuits can be shortened into a Hamilton path by removal of the last edge

Graph 1

28


A B

CD

E

Has no Hamilton circuits:What ever the starting point, we are going to have to pass through vertex E more than once to close the circuit.

Has many Hamilton paths:• A, B, E, C, D• C, D, E, A, B

Has Euler circuit => each vertex has even degree

Graph 2

29


A B

CD

E

Has many Hamilton circuits:1)A, F, B, E, C, G, D, A2)A, F, B, C, G, D, E, A

Has many Hamilton paths:1)A, F, B, E, C, G, D2)A, F, B, C, G, D, E

Has Euler circuit => Every vertex has even degree

Graph 3

F

G

30


A B

CD

E

Has no Hamilton circuits:

Has no Hamilton paths:

Has no Euler circuit

Has no Euler path => more than 2 vertices of odd degreeGraph 4

F G

HI

31

Complete graph

A graph with N vertices in which every pair of vertices is joined by exactly one edge is called the complete graph.

Total no. of edges = N(N-1)/2

A B

CD

In K4, each vertex has degree 3 and the number of edges = 4 (3)/2 = 6

32

The six Hamilton circuits of K4A B

CD

A,B,C,D,A B,C,D,A,B C,D,A,B,C D,A,B,C,D

A,B,D,C,A B,D,C,A,B C,A,B,D,C D,C,A,B, D

A,C,B,D,A B,D,A,C,B C,B,D,A,C D,A,C,B,D

A,C,D,B,A B,A,C,D,B C,D,B,A, C D,B,A,C,D

A,D,B,C,A B,C,A,D,B C,A,D,B,C D,B,C,A,D

A,D,C,B,A B,A,D,C,B C,B,A,D,C D,C,B,A,D

Reference point AReference point BReference point C Reference point D

Rows => 6 Hamilton circuits Cols=> same Hamilton circuit with different reference points

Graph

33

Complete graphThe number of Hamilton circuits in a

complete graph can be computed by using factorials.

The complete graph with N vertices has N! (factorial of N) = 1x 2x3x4x … x(N-1)x N

(N-1)! Hamilton circuits.Example: The complete graph with 5

vertices has 4! = 1x2x3x4 = 24 Hamilton circuits

34

Factorial

Which of the following is true?

n! = n! x (n-1)!n! = n! + (n-1)!n! = n x (n-1)!n! = n + (n-1)!

35

Traveling Salesman ProblemTraveling Salesman Problem (TSP):

Given a complete graph with nonnegative edge costs, Find a minimum cost cycle visiting every vertex exactly once.

Application (Example): Given a number of cities and the costs of traveling from any city to any other city, what is the cheapest round-trip route that visits each city exactly once and then returns to the starting city. 36

37

Applications of the TSP

Routing around Cities

Computer Wiring - connecting together computer components using minimum

wire length

Archaeological Seriation - ordering sites in time

Genome Sequencing - arranging DNA fragments in

sequence

Job Sequencing - sequencing jobs in order to minimise total set-up time between jobs

Wallpapering to Minimise Waste

NB: First three applications generally symmetricLast three asymmetric

38

8am-10am

2pm-3pm

3am-5am7am-8am10am-1pm

4pm-7pm

Major Practical Extension of the TSP

Vehicle Routing - Meet customers demands within given time windows using lorries of limited capacity

Depot

6am-9am

6pm-7pm

Much more difficult than TSP

39

History of TSP

1800’s Irish Mathematician, Sir William Rowan Hamilton

1930’s Studied by Mathematicians Menger, Whitney, Flood etc.

1954 Dantzig, Fulkerson, Johnson, 49 cities (capitals of USA states) problem solved

1971 64 Cities

1975 100 Cities

1977 120 Cities

1980 318 Cities

1987 666 Cities

1987 2392 Cities (Electronic Wiring Example)

1994 7397 Cities

1998 13509 Cities (all towns in the USA with population > 500)

2001 15112 Cities (towns in Germany)

2004 24978 Cities (places in Sweden)

But many smaller instances not yet solved (to proven optimality)

But there are still many smaller instances which have not been solved.

Finding an Approximate Solution

Cheapest Link AlgorithmEdge with smallest weight is drawn firstEdge with second smallest weight is drawn inContinue unless it closes a smaller circuit or three

edges come out of one vertexFinished once a complete Hamilton Circuit is

drawn

40

Cheapest Link Algorithm

41


42


43


44


45

Finding an Approximate Solution

Nearest Neighbor AlgorithmStart at any given vertexTravel to edge that yields smallest weight and

has not been traveled through yetContinue until we have a complete Hamilton

circuit

46

Nearest Neighbor Algorithm

47


48


49


50


51

Comparing Approximating Methods

Cheapest Link Nearest Neighbor

Total weight for

Cheapest Link – 31

Nearest Neighbor – 33

52

Example (D`oh!): We all know that Homer is lazy. However, he has to run errands at the Kwik-E-Mart, the Retirement Home and Moe`s. Assuming that he wants to begin and end his day at home find the route that will allow him to get back to napping as soon as possible.

53

• We might represent this dilemma with the following graph:

12

8

20

1715

20

A

B

C

D

54

• We might represent this dilemma with the following graph:

12

8

20

1715

27

A

B

C

D

These numbers will represent the number of blocks between each destination.

When we place values like this on the edges of a graph we refer to them as the weights of the edges.

55

• One solution might begin as follows:

1212

8

20

1715

20

A

B

C

D

56

• . . . And continue like so. . .

1212

8

2020

1715

20

A

B

C

D

57

• . . . And so on . . .

1212

8

2020

171715

20

A

B

C

D

58

• . . . Continuing until he arrives back at home.

1212

88

2020

171715

20

A

B

C

D

59

• Following this circuit, we find that Homer has to travel 12 + 8 + 20 + 17 = 57

blocks to finish his errands and get back to napping.

• But is this the most efficient route he can take?

• How might we find this best route?

60

• `Method 1`Make a list of all possible

Hamiltonian circuits.Calculate the `cost` for each one.Select the circuit with the least

cost.

61



cost.Circuit Mirror

ImageWeight

A, B, C, D, A A, D, C, B, A 12 + 8 + 20 + 17 = 57

A, B, D, C, A A, C, D, B, A 12 + 20 + 17 + 15 = 74

A, C, B, D, A A, D, B, C, A 15 + 20 + 20 + 8 = 63 62



cost.Circuit Mirror

ImageWeight

A, B, C, D, A A, D, C, B, A 12 + 8 + 20 + 17 = 57

A, B, D, C, A A, C, D, B, A 12 + 20 + 17 + 15 = 74

A, C, B, D, A A, D, B, C, A 15 + 20 + 20 + 8 = 63

We can see that the first route We can see that the first route was indeed the most efficient--was indeed the most efficient--Homer can get back to his nap Homer can get back to his nap after traveling only 57 blocks. after traveling only 57 blocks. Woo Hoo!Woo Hoo!

63

Example: The Galactica needs to survey a set of planets (A, B, C, D, E, F, G) in order to find water for the Fleet. the Commander has asked the helm to chart the course that will use the lowest amount of tylium fuel.

Example: The Galactica needs to survey a set of planets (A, B, C, D, E, F, G) in order to find water for the Fleet. the Commander has asked the helm to chart the course that will use the lowest amount of tylium fuel.

Again, we can model this situation using a graph like the one on the screen to the right.

• this time, however, it is incredibly difficult to list all the possible paths, so the ‘homer’ method (I.e. Method I( is not a good choice here.

• in fact, even listing the weight of each edge on the graph is hard given the number of vertices.

• so, the first thing we will do is list the weights in a table. . .

A

B

C

G

FE

D

75

22

50

13

40

20

40

3065

50

80

35

29

15

48

35

60

28

30 32

28

A B C D E F G

A 75 50 28 35 15 22

B 75 30 60 80 65 50

C 50 30 40 48 35 28

D 28 60 40 20 30 29

E 35 80 48 20 40 32

F 15 65 35 30 40 13

G 22 50 28 29 32 13

THE FOLLOWING TABLE TELLS US HOW MANY TONS OF TYLIUM IT TAKES TO TRAVEL FROM ONE PLANET TO ANOTHER:

A B C D E F G

A 75 50 28 35 15 22

B 75 30 60 80 65 50

C 50 30 40 48 35 28

D 28 60 40 20 30 29

E 35 80 48 20 40 32

F 15 65 35 30 40 13

G 22 50 28 29 32 13

THE FOLLOWING TABLE TELLS US HOW MANY TONS OF TYLIUM IT TAKES TO TRAVEL FROM ONE PLANET TO ANOTHER:

METHOD I IS IMPRACTICAL HERE (WE HAVE (7-1)!=6!=720 POSSIBLE HAMILTONIAN CIRCUITS!)

SO WHAT CAN WE DO IN THIS CASE?

Milestones

1954, one major city per contiguous state and Washington, D.C.

1980, 318 vertices Took 7 years Drilling application

68

Proctor and Gamble Game

69

Milestones

Most recently: 15,112 cities in Germany

April 21, 2001 Approx. 40,500 miles Used 110 different

computers from Rice and Princeton University

Total runtime?

70

Work in Progress – World Tour 1,904,711 cities worldwide Keld Helsgaun

March 2004 .075% to .076% from estimated lower-bound

71

72

USA Towns of 500 or more population 13509 cities 1998.

73

Sweden 24978 Cities 2004

Modern Applications

Traveling Salesmen?Drilling applicationsDelivery efficiency

Semi trucksSchool bus routes

74

For any query Feel Free to ask

Documents

ADVANCED ALGORITHM ANALYSIS LECTURE # 1 INTRODUCTION-H euristic Algorithms