Problem Complexity Classesstudentnet.cs.manchester.ac.uk/pgt/2014/COMP60342/... · ‘Computers and Intractability, A Guide to the Theory of NP-Completeness’, Freeman and Co., New

Problem Complexity ClassesP, NP, NP-Completeness and Complexity of Approximation

Joshua KnowlesSchool of Computer Science

The University of Manchester

COMP60342 - Week 2 2.15, March 21st 2014

In This Lecture

• The Classes P and NP

• The most difficult problems:NP-Complete and NP-hard

• Strongly and Weakly NP-hard

• An Approximation ComplexityClass: APX

Complexity / NP-Completeness 2 2.15, March 21st 2014

Quick Overview of Main Complexity Classes

P The set of problems for which a worst-case polynomial-time algorithm is known (inshort: Easy problems).

NP The set of problems for which we know how to check a proposed solution inworst-case polynomial time (in short: this set might include some harderproblems to find solutions for, but we can at least check the solutions quickly ifwe are given them).

NP-complete The set of problems in NP that we can efficiently reduce any otherproblem in NP to. (In short: this set stands for the hardest problems in NP, butwe don’t actually have a proof that they are harder than P problems!)

These definitions might seem a little tricky. Why can’t we have something simpler? ...We will hopefully see why they are like this in what follows


Polynomial-Time is Tractable

Some notation:Π A problemπ A problem instance|π| The number of symbols needed to describe

π. Also written as n, the input size to analgorithm for solving π.

Running times of algorithms are expressed as functions of the size of the input. E.g.O(n2).

Very few polynomial algorithms run in a complexity worse than say O(n5) time.

(There are no O(n1000) algorithms).

So, polynomial⇒ tractable.


The Need for Problem Complexity Classes

Efficient algorithms have been designed for many problems, e.g.:

integer arithmetic operations,sorting,matrix multiplication,shortest path in a graph,minimum spanning tree,

and so on. These problems are polynomial-time solvable: there are knownalgorithms to solve each of these problems in polynomial time in the worst case.

But for many other well-defined problems, often very simple to state, no efficient(polynomial) algorithm has been found.

Confronted with a new problem, how do we know whether or not it will be futile tolook for an efficient algorithm for it?



The ‘bandersnatch’ problem (*Garey and Johnson, 1979)

A fictitious scenario:

Your boss has asked you, her chief algorithmdesigner, to design a procedure to check whethernew components can be made for a bandersnatchmachine that meet all the design requirements.

You design an algorithm, but it keeps running tooslowly. The result is that the whole production is heldup.

What can you do, as your boss wants a faster algorithm, but you can’t think of one?

‘Computers and Intractability, A Guide to the Theory of NP-Completeness’, Freeman and Co., New York, 1979.



One strategy: If you can’t do something, prove it can’t be done.

−→ Good idea, but for many problems no such proof can be found either. This is thecase for many optimization problems and whether they are polynomial-time solvable.

...Now your bandersnatch boss asks to see you about the SLOWNESS of youralgorithm.

What can you say?



You should say:

“I can’t prove it can’t be done, but I can prove thatthe problem is as hard as all these problems:

traveling salesman problem,propositional satisfiability,solving quadratic Diophantine equations,shortest common supersequence,bounded post correspondance problem,...

And NO-ONE ELSE has found a polynomial timesolution to those either !”

→ I.e. you might be able to prove an equivalence of your problem with some othersthat seem difficult. This is a key motivation for the class of NP-Complete problems(which are problems that all have an equivalent maximum level of difficulty).



c©Bell Telephone Laboratories, 1979. (Reproduced without permission)


Optimization Problems, Evaluation Problems and DecisionProblems

Optimization Problems are Three Problems in One.

1. An optimization version instance has the form:Given the parameters of the instance, find the optimal feasible solution.

(The solution must define the values of the variables to achieve the optimum.)

2. An evaluation version of the problem instance requires only that the optimalsolution cost is returned.

3. A decision version∗ of the problem instance gives the parameters of the instanceand an additional integer L and asks for a yes/no answer to the question, Is there afeasible solution of cost c ≤ L ? +

The theory of computation studies problems like the decision version above, notoptimization problems, so the classes P, NP (and NP-Complete) are defined ondecision problems only.∗

In the course textbook this is called a recognition version.

+We assume a minimization problem, without loss of generality.


The Class P

Informally the class P is the class of decisionproblems solvable by some algorithm withina number of steps bounded by some fixedpolynomial in the length of the input.

Stephen Cook

The length of the input refers to the number of symbols written in some alphabet (orlanguage) needed to describe a problem instance.

Recall that O(n log n), which is a log-linear complexity, and not polynomial, is stillbounded by a polynomial, e.g. O(n2). Similarly other complexities involving rootswill generally be bounded by polynomials if exponents and factorials of n are absent.


The Class NPDefinition:

The class NP is the set of decision problems such that a yes certificate can bechecked in polynomial time.

Thus, the class NP encapsulates allproblems where it is easy/quick to verifya proposed ‘yes’ solution (a certificate) toevery problem instance that admits a ‘yes’solution.

Proof:pi is a YES instance

...................

...................

...................

...................

...................

.....

since:

Algorithm

Certificate is verified

Every combinatorial optimization problem in its decision version isin NP (see p.351, course textbook).


Important Note: It is not a requirement of members of NP for there to be a methodof efficiently generating the certificate.

Second Important Note: If the problem instance is a ‘no’ instance then anyproposed solution (or certificate) must be verified as being a no in polynomial time.But there is no need to be able to arrive at the NO decision for the problem (onemight have to check every certificate) quickly to be in NP.

E.g. the problem may be “Is there a route around the 50 state capitals of the UnitedStates in at most 5000 miles?”. The answer is “no”, but it can be difficult to provebecause we may have to check a lot of certificates. ALL that is required for theproblem to be in NP is that we can check each no certificate in polynomial time. E.g.we can verify that a particular tour does NOT visit all the state capitals and have alength ≤ 5000 miles.


Example of a Member of the Class NP

TSP (decision version) instance: Is there a Hamiltoninan Cycle in this graph of lengthless than L=41 ?

D

E

C

6

93

5

5

BA

7

7

6

9

8

F

G

certificate: <ABCDEGF> = 40

Certificate: a sequence of cities, such as 〈ABCDEGF 〉 Checking algorithm: checkeach edge in the certificate (e.g. A-B, B-C, . . . , F-A) exists in the given graph, andthat every vertex in the sequence appears exactly twice.


Proof (TSP (decision) problem is in NP.)

Clearly, the checking can be done in polynomial time since the number of symbols inthe certificate is less than the number in the problem instance (input), and eachcheck that must be done is a simple look-up or a simple increment to a counter.

More examples: Papadimitriou and Steiglitz, p 349–351.


All P problems are NP

Theorem:

Every (decision) problem in P is a member of NP.

Argument: Assume a problem is in P but not in NP. The consequence of being in P:the algorithm for generating the decision (yes/no) is polynomial. But the trace of thisalgorithm can serve legitimately as the certificate of the decision (a polynomialcertificate) contradicting the assumption the problem is not in NP.

See page 351 of Papadimitriou and Steiglitz for fuller treatment.


Is P a proper subset of NP ?

This is the notorious P ? NP (or P vs NP) problem.

Computer scientists have not been able to prove that problems such as TravelingSalesman, Graph Coloring (scheduling and timetabling), Diophantine Equations,cannot be solved in polynomial time.

Either:

P = NP

P = NP

OR

NP

P

P ⊂ NP


P vs NP

vs NPP

One of the Clay Millenium Prizeswww.claymath.org/millennium/P vs NP/ is offering a Million USDollars for a legitimate proof either way.


P vs NP (Wider Implications)

Apart from being a rigourously defined mathematical problem of great interest totheoretical computer science, the P vs NP problem is also taken to have wider (evenphilosophical) implications about the limits of AI, and questions around humancreativity.

If P=NP, then painting a (new) masterpiece should be as easy as recognizing that it isa masterpiece.

vs.

Check out

the painting.

I verify it is

good.

Doing a painting versus verifying a painting is good.


P vs NP

Some time in 2010...

... but Deolalikar is not a prize millionaire yet.


P vs NP

In fact, when Deolalikar announced his proof, Scott Aaronson, a respectedtheoretical computer scientist wrote on his blog

“I hereby announce the following offer:

If Vinay Deolalikar is awarded the 1,000,000-dollar Clay Millennium Prize forhis proof of P 6= NP, then I, Scott Aaronson, will personally supplement hisprize by the amount of 200,000 dollars.

I’m dead serious — and I can afford it about as well as you’d think I can.”

even before the proof had been seen or checked.

http://www.scottaaronson.com/blog/?p=456


Polynomial Transformations

There is a polynomial transformation from a problem Π1 to a problem Π2 if there is afunction F such that for every instance π1 ∈ Π1 there is an instanceπ2 ∈ Π2 = F (π1), such that π1 is a ‘yes’ instance of Π1 if and only if π2 is a ‘yes’instance of Π2, and F is a polynomially bounded function. We write Π1 ∝ Π2.

Note 1: Π2 is a least as hard a problem as Π1 since the above says that all instancesof Π1 can be mapped to instances of Π2, but not (necessarily) the other way around.

Note 2: polynomial transformation is transitive, i.e. if Π1 ∝ Π2 and Π2 ∝ Π3 thenΠ1 ∝ Π3.

We can use this idea to define a class of most difficult problems in NP.


The Definition of NP-Complete

A Problem Π is NP-complete if and only if

1. Π is in NP, and

2. Π′ ∝ Π for every other Π′ in NP.

It would be most inconvenient if, to show a problem is hard, we had to show thatevery problem in NP transforms to it.

Fortunately, we can instead prove that just one NP-complete problem transforms toit.

Why is this sufficient?


Updated View of ComplexityClasses

NP

NP−Complete

P

It has been shown that if P 6= NP then there are problems in NP that are not in P orNP-Complete, including integer factoring.


The Usefulness of the NP-Complete Class

1. No NP-Complete problem can be solved by a known polynomial-time algorithm

2. If a polynomial-time algorithm is found for one NP-Complete problem, then allNP-Complete problems will be solvable in polynomial time.

So, ‘NP-Complete’ serves as a kind of badge of difficulty.

One can now say, my problem Π is hard unless P=NP. (And most people who know,think P6= NP).

It also allows every theoretical computer scientist to be working on the sameproblem, i.e., searching for a proof that P 6= NP, or searching for a polynomial timealgorithm for any NP-Complete problem (and hence proving P=NP).


The First NP-Complete Problem(s)

Timeline:

1971 Stephen Cook proves that SATISFIABILITY is NP-Complete by showing thatevery other problem in NP can be transformed to it.

1971 Cook also proves that another problem, SUBGRAPH ISOMORPHISM isNP-Complete.

1972 Richard Karp gives NP-Completeness proofs for 21 further problems in NP.

1973 Levin gives NP-Completeness proofs for a number of natural problems ofinterest.

1978 Garey and Johnson publish the landmark book ‘Computers and Intractability...’,which contains NP-Completeness results (references to proofs) for more than300 problems across Graph theory, Network design, Sets and partitions,Sequencing and scheduling, Logic, Storage and retrieval, Games, and Numbertheory.


Generalizations and Special Cases

One easy way to prove a problem is NP-Complete is to show it is a generalization ofan existing NP-Complete problem.

E.g. The 0/1 Knapsack problem is a generalization of the problem, PARTITION.PARTITION asks whether or not it is possible to divide a set of items (each with aweight) into two sets with equal total weight. Can you show that 0/1 Knapsack is ageneralization of this?

Notice: This does not mean PARTITION is easier than 0/1 Knapsack; there mustalso be a way to transform 0/1 Knapsack to PARTITION too, so that each instance of0/1 knapsack is a ‘yes’ instance if and only if the corresponding PARTITION instanceis a ‘yes’.

Special cases (or restrictions) of NP-Complete problems need not be NP-complete.

For you to do: Give a restriction of 0/1 Knapsack that makes it easy (polynomial-timesolvable).


The Definition of NP-Hard

Recall the definition of NP-Complete:

A Problem Π is NP-complete if and only if

1. Π is in NP, and

2. Π′ ∝ Π for every other Π′ in NP.

If Π satisfies condition (2) and whether or not it satisfies condition (1), then it isNP-hard.

So, NP-hard problems are not necessarily in NP. The hard optimization problems (inthe original optimization version) are NP-hard, and not in NP.

NP-hard is the class of problems (not necessarily decision problems) that are at leastas hard as the NP-Complete problems.


Warning: Some people mistakenly think NP-Complete are really hard problems, andNP-hard are slightly less hard. This is incorrect. NP-hard problems are a larger classthan NP-Complete and at least as difficult.


The Complement of NP: co-NP

A problem is said to be in co-NP if its complement is in NP.

Hamiltonian Cycle Complement Problem:Is there NOT a Hamiltonian cycle in the given graph G?

The above problem is in co-NP (since if you remove the NOT you have a problem inNP).

The above problem is not in NP. Why? (Hint: certificates?)

Every P decision problem is in both NP and co-NP. Can you see why?


Summary of Key Information

decision problems problems with yes/no answerP set of poly.-time solvable decision problemsNP set of problems whose ‘yes’ instances can be

verified (not solved) in poly. time, given a suitablecertificate

P ⊆ NP (since solution constitutes certificate)P 6= NP probably though proof is seeming difficult!NP-Complete Problem Π 1. Π is in NP, and 2. Π′ polynomially transforms to Π

for all Π′ in NPNP-Hard Problem Π At least as hard as any problem in NP. Not

necessarily in NP; so does not need to be a decisionproblem

Π̄ complement of problem Π. Yes instances of Π areno instances of Π̄

co-NP problems where ‘no’ instances can be verified inpoly. time

P ⊆ NP ∩ co-NP (since run of polynomial algorithm is certificate forboth a problem and its complement)


Part II — Pseudopolynomial Algorithms andApproximation Algorithms


0/1 Knapsack Problem is Weakly NP-Hard

You are currently studying algorithms for the 0/1 Knapsack problem.

Later, you will study a Dynamic Programming algorithm for the problem. It works byconstructing a table of size N.C , where N is the number of items in the knapsackand C is the knapsack capacity.

Each operation for constructing the table is simple [i.e., O(1)], so the complexity ofthe algorithm as a whole is O(N.C).

But 0/1 Knapsack is NP-Hard (or NP-Complete in decision form), so how can this be?


The Dynamic Programming Approach to Knapsack

Here is a problem with Knapsack capacity=6 being solved. There are 3 items of sizes3,1 and 5. (The profits equal the sizes.)

Capacityitem 0 1 2 3 4 5 60 0 0 0 0 0 0 01 0 0 0 3 3 3 32 0 1 1 3 4 4 43 0 1 1 3 4 5 6

The optimal solution is in the last (bottom right) cell of the table.

This looks like polynomial time. So did we prove P=NP?


0/1 Knapsack is Still Exponential Time :-(

Why is the DP algorithm for Knapsack not polynomial? Because O(N.C) is notpolynomial in the input size of the instance !

Consider a problem with just 5 items but knapsack capacity 1,000,005 with items ofsizes 743,004; 225,025; 78,000; 634,451; 362,017.

What is the size of the instance? Well, count the digits in the input above. There arejust 36.

But N.C is ∼ 5, 000, 000.

What has happened here? The input size is ∼ N. logC since it takes roughly logCdigits to express each number (item weight).

So O(N.C) is exponential in the input size.

However, practically, Knapsack problems often grow in the number of items, not (somuch) by the capacity. So in practice DP approach to Knapsack is fast.


0/1 Knapsack is Pseudopolynomial Time Solvable

It seems useful to be able to describe the relative easiness of Knapsack in practice(despite it being NP-hard).

Definition:A problem can be solved in pseudopolynomial time if there is an algorithm that cansolve every instance π in a polynomial function of n and number(π).

Above, number(π) can be taken as the largest number described in the input.


Strongly NP-Hard Means Not Pseudo-Poly.Solvable

Many NP-hard problems cannot be solved in pseudopolynomial time.

E.g. dynamic programming when applied to the travelling salesman still gives arunning time exponential in |V |, so it does not matter if the weights in the graph aresmall or large. No other pseudopolynomial time algorithm is possible either unlessP=NP.

Such problems are called Strongly NP-Hard.

Definition:A problem is strongly NP-hard if it remains NP-hard for instances πn ⊂ Π that arerestricted to the set of instances where number(πn) is polynomial in |πn|.

An NP-hard problem that is not strongly NP-hard is called weakly NP-hard.


Approximation Algorithms

One approach to tackling NP-hard optimization problems, is to use polynomial-timealgorithms that give guaranteed levels of approximation of the optimal solution.

Definition:An algorithm is a p-approximation algorithm for a problem Π if it gives a solutionnot worse than p times the optimal cost on any instance π ∈ Π.

If a p-approximation algorithm is in addition a polynomial time algorithm for Π, then itis known as a polynomial time approximation algorithm for Π.


Approximation Ratios

The performance ratio of an algorithm is

maxπ∈Π

Calg(π)

Copt(π)

for a minimization problem

maxπ∈Π

Copt(π)

Calg(π)

for a maximization problem where Calg(π) is the value of the algorithm solution oninput π and Copt(π) is the value of the optimal solution on input π.


A 2-Approximation Scheme for Metric TSP

One form of metric TSP is:Given a weighted, complete graph G = (V,E) with edge weights satisfying thetriangle inequality, give a minimal weight Hamiltonian cycle in G.

A 2-Approximation Scheme for it is the following: [HOMEWORK: Prove this]

Step 1. Construct the minimum spanning tree (e.g. usingKruskal or Prim’s algorithm.) This takes O(|V |2) time.

Step 2. Create an Eulerian graph from the tree by duplicatingall edges. This step is O(|V |).Step 3. Construct an Euler cycle in the Eulerian graph byusing a cycle finding algorithm. The complexity is O(|V | +|E|).Step 4. Finally, construct a Hamiltonian Cycle in G by‘shortcutting’ the Euler cycle. That is, skip out verticesalready visited and proceed to the next unvisited vertex onthe Eulerian cycle. This takes O(|V |2) steps.


Illustration: A 2-Approximation Scheme for Metric TSP

D

E

C

6

93

5

5

BA

7

7

6

9

8

F

G

D

E

C

6

93

5

5

BA

7

7

6

9

8

F

G

2

D

E

C

BA

F

G

2

TSP Instance STEP 1

STEPS 2 & 3

D

E

C

BA

F

G

2

3

5

67

8

9

10

11

112

4

1

2

3

4

5

6

7

STEP 4

Note: the scheme isillustrated here on a sparsegraph. However, in generalit only works on completegraphs. Why?


The APX Class

Any problem that is approximable within a fixed constant factor of thebest cost is in the class APX (approximable).

In other words, if there is a polynomial time p-approximation algorithmfor a problem, then the problem is in APX.

⇒ Metric TSP is in APX.


Summary

• The P, NP and NP-Complete complexity classes are important for classifyingproblem difficulty.

• NP-hard class characterizes optimization problems (optimization versions) asbeing very difficult.

• Strongly NP-hard problems remain hard even when all the numbers (e.g.weights) in a problem instance are small (i.e. bounded by a polynomial of theinstance size).

• Weakly NP-hard problems are only hard when some of the numbers in aninstance are large. They can be solved in pseudo-polynomial time.

• Constant-factor approximation algorithms for NP-hard problems that run inpolynomial time are known as polynomial approximation algorithms. The class ofproblems for which a PTAA exists is called APX.


Further Reading

• Chapters 15, 16, 17 of Papadimitriou and Steiglitz

• Garey and Johnson (1979). Look at the NP-Complete problems listed in theback, and how they are related.

• http://personal.vu.nl/l.stougie/Courses/CO/col456CT.pdf

• http://xlinux.nist.gov/dads/HTML/npcomplete.html

• http://qwiki.stanford.edu/index.php/Complexity Zoo

Some more links are on the course webpage.


Documents

Problem Complexity Classesstudentnet.cs.manchester.ac.uk/pgt/2014/COMP60342/... · ‘Computers and Intractability, A Guide to the Theory of NP-Completeness’, Freeman and Co., New