Explorations in Artificial Intelligence

Explorations in Artificial Intelligence

Prof. Carla P. [email protected]

Module 6Intro to Complexity

The algorithm problem

Specification ofall legal inputs

and

Specification ofdesired output

as a function of the input

Any legalinput

The algorithm

The desiredoutput

Examples of algorithmic problems

Problem 1: Input: A list L, of integers Output: The sum of the integers on L

Problem 3: Input: A road map of citieswith distances attached to the road map, and two designated cities A and B Output: A description of theshortest path between A and B

Problem 2: Input: Two texts A and B in English Output: The list of common words in both texts

Problem 5:Input: A valid linear programming problemOutput: The optimal solution to the LP problem

Problem 4: Input: A road map of cities with distances attached to the road map, and a designated initial city A. Output: A description of the shortest tour that passes through all the cities starting and ending at A.

Examples of algorithmic problems

Variants of algorithmic problems: Decision Problem

Problem: Knapsack (decision)Input: profits p0, p1, …, pn-1

weights w0, w1, …, wn-1

capacity M

target profit P

Output: YES or NO to the question:Does there exist an n-tuple [xo,…,x n-1] Є {0,1}n, such

that:

Pixn

i ip

1

0and

Mixn

i iw

1

0

Variants of algorithmic problems: Search Problem

Problem: Knapsack (search)

Input: profits p0, p1, …, pn-1


capacity M

target profit P

Output:An n-tuple [xo,…,x n-1] Є {0,1}n, such

that:Pix

n

i ip

1

0and

Mixn

i iw

1

0

Variants of algorithmic problems: Optimal Value

Problem: Knapsack (optimal value)

Input: profits p0, p1, …, pn-1


capacity M

Output:The maximum value of:

ixn

i ipP

1

0subject to:

Mixn

i iw

1

0

and [xo,…,x n-1] Є

{0,1}n

Variants of algorithmic problems: Optimization

Problem: Knapsack (optimization)Input: profits p0, p1,

…, pn-1


capacity M

Output:An n-tuple [xo,…,x n-1] Є {0,1}n, such that:

ixn

i ipP

1

0 is

maximized subject to:

Mi

xn

iiw

1

0

Instance of an algorithmic problemSize of an instance

An instance of an algorithmic problem is a concrete case of such a problem with specific input. The size of an instance is given by the size of its input.

Examples of instances:

– An instance of problem 1:

• L= 2, 5, 26, 8, 170, 79, 1002

Problem 1: Input: A list L, of integers Output: The sum of the integers on L

Size of instance length of list

Size of instance = |L| = 7

Examples of instances


1

2

3

4

5

6

2

4

2 1

3

4

2

3

2

Size of instance Number of cities and roads

A particular instance:Size of instance:6 nodes9 edges

The size of an instance is given by the size of its input.

An instance of problem 5:

max 3 x + 5 y

s.t

x ≤ 4

y ≤ 12

3 x + 2 y ≤ 18

x,y ≥ 0

Problem 5:Input: A valid linear programming problemOutput: The optimal solution to the LP problem

Size of instance:2 variables3 functional constraints

Size of instance Number of variablesand constraints (n,m)

The size of an instance is given by the size of its input.

Complexity of Algorithms

Complexity of Algorithms

The complexity of an algorithm is the number of steps that it takes to transform the input data into the desired output.

Each simple operation (+,-,*,/,=,if,etc) and each memory access corresponds to a step.(*)

The complexity of an algorithm is a function of the size of the input (or size of the instance). We’ll denote the complexity of algorithm A by CA(n), where n is the size of the input.

(*) This model is a simplification but still valid to give us a good idea of the complexity of algorithms.

Example: Insertion Sort

From: Introduction to AlgorithmsCormen et al

best cost

worst cost

I n

I n

worst cost

best cost

I n – all possible instances of size n

Different notions of complexity

Worst case complexity of an algorithm A – the maximum number of computational steps required for the execution of Algorithm A, over all the inputs of the same size, s. It provides an upper bound for an algorithm. The worst that can happen given the most difficult instance – the pessimistic view.

Best case complexity of an algorithm A -the minimum number of computational steps required for the execution of Algorithm A, over all the inputs of the same size, s. The most optimistic view of an algorithm– it tells us the least work a particular algorithm could possibly get away with for some one input of a fixed size – we have the chance to pick the easiest input of a given size.

Average case complexity of an algorithm A - i.e., the average amount of resources the algorithm consumes assuming some plausible frequency of occurrence of each input.

Figuring out the average cost is much more difficult than figuring out either the worst-

cost or best-cost e.g., we have to assume a given probability distribution for the types of inputs we get.

Different notions of complexity

In general thisis the notion that we use tocharacterize the complexity ofalgorithms

We performupper bound

analysis onalgorithms.

Growth Rates

In general we only worry about growth rates because:

Our main objective is to analyze the cost performance of algorithms.

Another obstacle to having the exact cost of algorithms is that sometimes the algorithm are quite complicated to analyze.

When analyzing an algorithm we are not that interested in the exact time the algorithm takes to run – often we only want to compare two algorithms for the same problem – the thing that makes one algorithm more desirable than another is its growth rate relative to the other algorithm’s growth rate.

Growth Rates

Two functions of n have different growth rates if as n goes to infinity their ratio either goes to infinity or goes to zero.

If their ratio stays near a non-zero constant then they are asymptotically the same function.

Big Oh Notation

Given two functions F and G, whose domain is the natural numbers, we say that the order of F is lower than or equal to the order of G if:

– F(n) ≤ c G(n) for all n > n0 (c and n0 are constants)

We say F is O(G) (F is oh of G)

Example:(3 n3 + n2 + n ) is O(n3)

In practice we just look at the fastest growing term of the expression

Function Name

c logN log2N N NlogN N2 N3 2N

Constant Logarithmic Log-squared Linear Quadratic Cubic Exponential

Typical Growth Rates

Roughly Speaking…

Size

Cost

exponentialquadratic

linear

logarithmic

constant

Good vs. Bad Algorithms

How do computer scientists differentiate between

good (efficient) and bad (not efficient) algorithms?

The yardstick is that any algorithm that runs in no more than polynomial time is an efficient algorithm; everything else is not.

Ordered functions by their growth rates

cOrder

constant 1

logarithmic 2

polylogarithmic 3

nr ,0<r<1

nsublinear 4

linear 5

nr ,1<r<2 subquadratic 6

quadratic 7

cubic 8

nc,c≥1

rn, r>1

polynomial 9

exponential 10

lg n

lgc n

n3

n2

Efficient algorithms

Not efficient algorithms

exponential

polynomial

N2

Binary B&B alg and DPLL.

Polynomial vs. exponential growth (Harel 2000)

LP’s interior pointMin. Cost Flow AlgsTransportation AlgAssignment AlgDijkstra’s alg.

Problem Complexity

Theory of NP-completeness or NP-hardnessEasy vs. hard problems

Overview of complexity

How can we show a problem is efficiently solvable?– We can show it constructively. We provide an algorithm and show that it solves the

problem efficiently. E.g.:

• Shortest path problem - Dijkstra’s algorithm runs in polynomial time, O(n2). Therefore the shortest path problem can be solved efficiently.

• Linear Programming – The Interior Point method has polynomial worst-case complexity. Therefore Linear programming can be solved efficiently.

(*) The simplex method has exponential worst case complexity/ However, in practice the simplex algorithm seems to scale as m3, where m is the number of functional constraints.

Overview of complexity

How can we show a problem is not efficiently solvable?

– How do you prove a negative? Much harder!!!

– This is the aim of complexity theory.

Easy (efficiently solvable) problems vsHard Problems

Easy Problems - we consider a problem X to be “easy” or efficiently solvable, if there is a polynomial time algorithm A for solving X. We denote by P the class of problems solvable in polynomial time.

Hard problems --- everything else. Any problem for which there is no polynomial time algorithm is an intractable problem.

.

Two problems

Problem 2: Input: A road map of citieswith distances attached to the road map, and two designated cities A and B Output: A description of thelongest path between A and B


We know that it is easy – it can be solved efficiently!

Can we come up with an efficient algorithm for the longest path?

I’d like you to develop an effcient algorithm to find the longest path between two points in a graph.

Your Longest Path Algorithm between two nodes, u and v

1

2

3

4

5

6

2

4

2 1

3

4

2

3

2

Initialization: MaxPath none ;MaxPathLength 0;

For each path P starting at 1:if P is a simple path from u to v and length(P) > MaxPath

MaxPath P;MaxPathLength length(P)

Return: MaxPath; MaxPathLength;

G=(N,E)

u v

Is that the best you can do? -- that seems to be a bad algorithm!!!

I know…If each intersection leads to two roads and we have100 intersections that canlead to 2100 paths …

I cant’ find an efficient algorithm. I guess I’m too dumb.

I can’t find an efficient algorithm, but neither can these famous researchers.

Alan Turing

In 1936, Alan Turing, a British mathematician, showed that there exists a relatively simple universal computing device that can perform any computational process. Computers use such a universal model.

Turing Machine(abstraction)

Turing also showed the limits of computation – some problems cannot be computed even with the most powerful computer and even with unlimited amount of time – e.g., Halting problem.

http://www.computer50.org/kgill/people/turpic.gif

John Von Neumann

Brilliant mathematician, synthesizer, and promoterof the stored program concept, whose logical design of the Institute of Advanced Studies (Princeton)Computer became the prototype of today’s computer (*) - the von Neumann Architecture.

(*) sequential i.e., non-parallel computers

http://www.neuralmachines.com/axon/von-neumann.gif

Richard Karp

Steven Cook

Invented the theory of NP-Completeness –proved that a simple problem - Satisfiabilityis NP-Complete.

Showed that several important problemsand applications are NP-Complete and NP-hard,including Integer Programming.

Given a propositional formula, is there an assignment to its

variables (a, b, and c – True or False) making the formula true?

)()()( cacacba

http://math.berkeley.edu/publications/newsletter/1996/F96new_images/Richard%20Karp.jpeg

David Johnson

Michael Garey

George Dantzig

Invented Linear Programming Formulations:

max 3 x + 5 y

s.t

x ≤ 4

y ≤ 12

3 x + 2 y ≤ 18

x,y ≥ 0

Invented Simplex Algorithm

Coefficients of Variables

Bas Var equation Z x1 x2 x3 x4 x5 RHSZ 0 1 -3 -5 0 0 0 0x3 1 0 1 0 1 0 0 4x4 2 0 0 2 0 1 0 12x5 3 0 3 2 0 0 1 18

basic variables x3=4 x4=12 x5=18non-basic variables x1=0 x2=0

http://www.seas.smu.edu/~jlk/images/dad.jpg

Theory of NP-completeness and NP-hardness

Easy vs. hard problems

Can satisfiability or integer programming be solved in polynomial time?

FACT: every algorithm that has ever been developed for satisfiability or integer programming takes exponential time.

Hundreds of very smart researchers have tried to come up with polynomial time algorithms for satisfiability or integer programming, and failed.

It is generally believed that there is no polynomial time algorithm for satisifiability or integer programming.

Complexity theory: deals with proving that satisfiability, integer programming, and many other problems are computationally hard.

Decision Problems

NP-Completeness theory deals with decision problems. What is a decision problem? A problem for which there is a yes/no answer.

Examples:Given a propositional formula, is there an assignment of values True

or False to its variables that makes the formula true?

)()()( cacacba

Is there a path between two nodes in a graph shorter than k?(decision version of shortest path problem)

Is there a path between two nodes in a graph longer than k?(decision version of longest path problem)

Most optimization problems can be formulated as a decision problem

Class NP Class of Problems solvable in

Nondeterministic-Polynomial Time

We say that a decision problem is solvable in Non-deterministic polynomial time if:

– The solution can be verified in polynomial time. (E.g., verifying that a path has length greater than K)

– If we imagine that we have an exponential number of processors, we can check all possible solutions simultaneously and therefore answer in polynomial time

NPP

Class NP-Complete

The first problem to be shown to be NP-Complete was Satisfiability – Cook showed that all the problems in NP could be translated (in polynomial time) as Satisfiability problems;

The word complete means that every problem in the class NP-complete can be reduced (in polynomial time) into another problem of the class NP-complete. For example all the problems in the class NP-complete can be written as Satisfiability problems.

The class of NP-Complete problems is the class of the hardest computational problems in the class NP: every NP problem can be transformed into an NP-complete problem (the reverse is not true!!!)

NP

P NP-Complete

Is P not equal to NP?$1,000,000 question

P not equal to NP? – Is that true that not all problems is NP can be solved in polynomial time?

Class of NP-Complete Problems– They all admit exponential time solutions;

– Nobody has ever been able to find a polynomial time solution for any single problem in the class;

– Nobody has ever been able to prove an exponential lower bound for any single problem in the class;

P not equal to NP?$1,000,000 question

Pictorial interpretation of this question

NP

P NP-Complete

Is this the right picture? There are problems in NP that are inherently intractableand cannot be solved in polynomial time.

P not equal to NP?$1,000,000 question

Pictorial interpretation of this question

Or is this the right picture? All the problems in NP can be solved in polynomial time. Even though at this point we don’t know of polynomial time algorithms to solve some problems in NP, they exist…

P = NP

Class of NP-Complete ProblemsOne Million Dollar Prize

Completeness – – if someone were to find a polynomial time solution for a single problem

in the class NP-complete all the problems could be solved in polynomial time!!!

– if someone were to prove an exponential lower bound for a single problem in the class NP-complete all the problems in the class would be intractable !!!

P vs. NPOne Million Dollar Prize

http://www.claymath.org/Millennium_Prize_Problems/P_vs_NP/

Class of NP-Complete ProblemsOne Million Dollar Prize

Completeness – if someone were to find a polynomial time

solution for a single problem in the class NP-complete all the problems could be solved in polynomial time!!!

– if someone were to prove an exponential lower bound for a single problem in the class NP-complete all the problems in the class would be intractable !!!

Conjecture

NP-Complete problems are inherently hard!

They are intractable !

NP

P NP-Complete

On Proving NP-Completeness results

Suppose that we want to prove that the a problem

is NP-Complete. How do we do it?

Find a known NP-complete problem, NPC.

Show that NPC can be transformed (in polynomial time) into .

Show that there is a solution to NPC if and only if there there is a solution to .

–

Proof: Hamiltonian Path is NP-complete

A hamiltonian cycle is a cycle that passes through each node exactly once.

A hamiltonian path is a path that includes every node of G.

Suppose that we know that the problem of deciding if there is a hamiltonian cycle in a graph is NP-Complete.

We will show that the problem of deciding if there is a hamiltonian path is also NP-Complete.

Proof Technique

Start with any instance of the hamiltonian cycle problem (NPC). – We denote this instance as G = (N, A).

Transformation proofs (these are standard). Create an instance G’ = (N’, A) for the hamiltonian path problem from G with the following property: there is a hamiltonian path in G’ if and only if there is a hamiltonian cycle in G.

1 1 210 22

The original networkThe transformed network: node 1 of the original network was split into nodes 1 and 21, and nodes 0 and 22 were connected to the split nodes.

From J.Orlin

Claim: If there is a hamiltonian cycle in the original graph then there is a hamiltonian path in the transformed graph.

1

A Hamiltonian Cycle.

Connect one to node 1, and the other to node 21. Add in arcs (0,1) and (21, 22).

1 210 22

Take the two arcs in G incident to the node 1.

Claim: If there is a hamiltonian path in the transformed graph then there is a hamiltonian cycle in the original graph.

1

Delete the two arcs (0, 1) and (21, 22). Then take the other arcs in G’ incident to 1 and 21, and make them incident to node 1 in G.

1 210 22

A Hamiltonian Path

NP-CompletenessLongest Path

Problem Longest Path: Input: Graph G = (V,E), length l for each edge, positive integer K ≤ |V|.Output: Is there a simple path (i.e., a path visiting each node at most once) with K or more edges?

Problem HamiltonianPath: Input: Graph G = (V,E).Output: Does it have a Hamiltonian path (i.e., a path visiting each node exactly once)?

Reduction NP-CompletenessFrom Hamiltonian Path into Longest Path

Problem Longest Path: Input: Graph G = (V,E), length l for each edge, positive integer K= |V|.Output: Is there a simple path (i.e., a path visiting each node at most once) with K or more edges?


If there is a pathof length K= |V|(Yes)

There is a HamiltonianPath in G (YES).

HamiltonianPath is NP-Complete



If there is not a pathof length K= |V|(No)

There is not a HamiltonianPath in G (NO).




If there is a HamiltonianPath in G (YES).

There is a pathof length K= |V|(YES)




If there is NOT a HamiltonianPath in G (NO).

There is NOT a pathof length K= |V|(NO)


Proof of NP-Completeness of Sudoku

Suppose that we know that the problem of deciding if we can complete a Latin Square is NP-Complete – i.e., the Latin Square Completion problem is NP-Complete.

We will show that the problem of deciding if we can complete a partial Sudoku instance is also NP-Complete.

SudokuSudoku

9 55 ~ 3x 10 52 possible completionsCan we complete this matrix using numbers from 1 to 9, with repeating a number in a row, column, or block?

0 1

1

1 2

Latin SquareCompletion

Problem

0 1Reduction

To Sudoku

3 5 6 84 7

3 5 6 84 7 0 16 87 0 1 3 54

1 7 8 6 4 35

4 5 3

7 8 6

1 7 8 6

4 5 3 1

1 2

1 2

1 2

5 3 4

5 3 4

5 3 48 6 7

8 6 7

8 6 7

0 1

1

1 2


Problem

0 1

If there isa Latin SquareCompletion

There is also a way of Completingthe Sudokumatrix

3 5 6 84 7

3 5 6 84 7 0 16 87 0 1 3 54

1 7 8 6 4 35

4 5 3

7 8 6

1 7 8 6

4 5 3 11 2

1 2

1 2

5 3 4

5 3 4

5 3 48 6 7

8 6 7

8 6 7

2

2 0

0

2

2

2

2

0

02

2 0

0

0

0

0 1

1

1 2

There isalso away of completing the Latin SquareCompletion

If there isa way of completingthe Sudokumatrix

0 1

1

1 2


Problem

0 1 3 5 6 84 7

3 5 6 84 7 0 16 87 0 1 3 54

1 7 8 6 4 35

4 5 3

7 8 6

1 7 8 6

4 5 3 11 2

1 2

1 2

5 3 4

5 3 4

5 3 48 6 7

8 6 7

8 6 7

2

2 0

0

2

2

2

2

0

02

2 0

0

0

0

0 1

1

1 2

Class NP vs. Class Co-NP

Class NP Class of decision problems whose solution can be verified in polynomial time. (E.g., satisfiability)

Class Co-NP co-NP is the complexity class that contains the complements of decision problems in the complexity class NP.

Satisfiability - Is there a value for a, b, and c such that the formula is true?

YES instances have short proofs; i.e., If the answer is YES we just have to exhibit

the values of a, b, and c and show the formula is true

)()()( cacacba

Co-NP and the asymmetry of NP

Class Co-NP co-NP is the complexity class that contains the complements of decision problems in the complexity class NP.

Unsatisfiability – (Complement of Satisfiability)- Is it true that for all values of a,

b, and c this formula is not satisfiable?

YES instances don’t have short answers: If indeed the answer to this question

is YES (i.e., we have an unsatisfiable clause) we do not have a short proof for it.

)()()( cacacba

http://en.wikipedia.org/wiki/NP_%28complexity%29

Class P vs. NP Co-NP

If a problem belongs to P, then it belongs to both NP and co-NP,so P NP Co-NP

P = NP Co-NP ? (i.e., are there problems with good characterizationbut with no polynomial time algorithm?)

Class PSPACE

What if we worry about space - i.e., memory

requirements?

Class PSPACE - the set of decision problems

that can be solved using a polynomial amount

of memory, and unlimited time.

Clearly P PSPACE

Class PSPACE

Class PSPACE - class of problems that appears

to be harder than NP and co-NP.

Why? space can be re-used while time cannot.

Examples: • Consider an algorithm that counts from 0 to 2n – 1. while this algorithm can be

implemented with a simple n-bit counter, it runs for an exponential time!

• We can also solve the satisfiability problem using only a polynomial amount of space, for example by trying all possible assignments using also an n-bit counter.

PSPACE-Complete

A decision problem is in PSPACE-complete if it is in PSAPCE, and

every problem in PSPACE can be reduced to it in polynomial time. The

problems in PSPACE-complete can be thought of as the hardest problems

in PSPACE. These problems are widely suspected to be outside of P and

NP, but that is not known.

–

Satisfiability (NP-complete):

Quantified Satisfiability (PSPACE-complete):

(The most basic PSPACE-complete problem is identical to satisfiability, except it alternates existential and universal quantifiers)

The PSPACE-complete problem resembles a game: is there some move I can make, such that for all moves my opponent might make, there will then be some move I can make to win? The question alternates existential and universal quantifiers. Not surprisingly, many puzzles turn out to be NP-complete, and many games turn out to be PSPACE-complete.

Checkers is PSPACE-complete when generalized so that it can be played on an n × n board.

Generalized versions of the games Hex and Reversi and the solitaire games such as Rush Hour, Mahjong, Atomix and Sokoban.

Note that the definition of PSPACE-complete is based on asymptotic complexity: the time it takes to solve a problem of size n, in the limit as n grows without bound. That means a game like checkers (which is played on an 8 × 8 board) could never be PSPACE-complete. That is why all the games were modified by playing them on an n × n board instead.

PSPACE

P

Co-NPNP

Some examples of NP-hard problems

Longest path

Traveling Salesman Problem

Capital Budgeting Problem (knapsack problem)

Independent Set Problem

Fire Station Problem (set covering)

0-1 Integer programming

Integer Programming

Project management with resource constraints

and thousands more

Okay – Should we give up?

Here is why:

The theory of NP-completeness is only a worst case result. Not all problem instances are as hard as the worst case.

Real problems tend to have sub-problems that are tractable and by exploiting the structure of such sub-problems using efficient algorithms such Unit Propagation, LP, Min Cost Flow, Transportation. Assignment and shortest path methods we can solve much larger problem instances.

In the 1970’s we could only solve Binary Integer Programming instances with 100 variables. By exploiting the structure we can now solve real world instances with over 120,000 variables and 4000 functional constraints.

In the 1990’s we could only solve Satisfiability instances with 50 variables and 200 clauses. By using randomization and learning to exploit the structure we can now solve real world instances with over 1,000,000 variables and 5,000,000 functional clauses.

NO WAY!!!

EXPONENTIAL FUNCTION

POLYNOMIAL FUNCTIONHard Computational

ProblemsScale Exponentially

EXPONENTIAL-TIMEALGORITHMS

EXPLOSIVECOMBINATORICS

ExperimentDesignGoal

Start

Software & HardwareVerification

Satisfiability

(A or B) (D or E or not A)

Data Analysis& Data Mining

Fiber optics routing

Capital BudgetingAnd Financial Appl. Information

Retrieval

Protein Folding

And Medical ApplicationsCombinatorial

Auctions

Planning and SchedulingAnd Supply Chain Management

Many more applications!!!

Require powerful computational and

mathematical tools!

NP-Complete andNP-Hard Problems

Documents

Explorations in Artificial Intelligence