Algorithmic Complexity

COMP30172: Advanced Algorithms

Introduction to Complexity - I

Howard Barringer

Room KB2.20: email: howard.barringer@manchester.ac.uk

April 2010

Algorithmic Complexity

Introduction to Complexity

Algorithmic ComplexityMotivationUnsolvable ProblemsSolvable ProblemsTheoretical UnderpinningsSummary

Algorithmic Complexity

Outline

Algorithmic ComplexityMotivationUnsolvable ProblemsSolvable ProblemsTheoretical UnderpinningsSummary

Algorithmic Complexity

Classifying Problems by their Difficulty

Algorithmic Complexity

Outline

Algorithmic ComplexityMotivationUnsolvable ProblemsSolvable ProblemsTheoretical UnderpinningsSummary

Algorithmic Complexity

Solvable / Unsolvable Distinction

Informally:

A problem is unsolvable if there is no algorithm that willalways terminate with the correct solution for any giveninput to the problem.

More formally:

• Define the notion of computability via a formal machine, e.g. aTuring Machine, or a Universal Register Machine, or some othermathematically equivalent computational device.

• Solving a problem is then related to computing a function (via theformal device) from a set of inputs to its set of outputs.

• Unsolvable problems are then those where no correspondingfunction can be computed on the formal device within a finiteamount of time, for any finite input.

Algorithmic Complexity

Solvable / Unsolvable Distinction

Informally:

A problem is unsolvable if there is no algorithm that willalways terminate with the correct solution for any giveninput to the problem.

More formally:

• Define the notion of computability via a formal machine, e.g. aTuring Machine, or a Universal Register Machine, or some othermathematically equivalent computational device.

• Solving a problem is then related to computing a function (via theformal device) from a set of inputs to its set of outputs.

• Unsolvable problems are then those where no correspondingfunction can be computed on the formal device within a finiteamount of time, for any finite input.

Algorithmic Complexity

Solvable / Unsolvable Distinction

Informally:

A problem is unsolvable if there is no algorithm that willalways terminate with the correct solution for any giveninput to the problem.

More formally:

• Define the notion of computability via a formal machine, e.g. aTuring Machine, or a Universal Register Machine, or some othermathematically equivalent computational device.

• Solving a problem is then related to computing a function (via theformal device) from a set of inputs to its set of outputs.

• Unsolvable problems are then those where no correspondingfunction can be computed on the formal device within a finiteamount of time, for any finite input.

Algorithmic Complexity

Solvable / Unsolvable Distinction

Informally:

A problem is unsolvable if there is no algorithm that willalways terminate with the correct solution for any giveninput to the problem.

More formally:

• Define the notion of computability via a formal machine, e.g. aTuring Machine, or a Universal Register Machine, or some othermathematically equivalent computational device.

• Solving a problem is then related to computing a function (via theformal device) from a set of inputs to its set of outputs.

• Unsolvable problems are then those where no correspondingfunction can be computed on the formal device within a finiteamount of time, for any finite input.

Algorithmic Complexity

Solvable / Unsolvable Distinction

Informally:

A problem is unsolvable if there is no algorithm that willalways terminate with the correct solution for any giveninput to the problem.

More formally:

• Define the notion of computability via a formal machine, e.g. aTuring Machine, or a Universal Register Machine, or some othermathematically equivalent computational device.

• Solving a problem is then related to computing a function (via theformal device) from a set of inputs to its set of outputs.

• Unsolvable problems are then those where no correspondingfunction can be computed on the formal device within a finiteamount of time, for any finite input.

Algorithmic Complexity

Solvable / Unsolvable Distinction

Informally:

A problem is unsolvable if there is no algorithm that willalways terminate with the correct solution for any giveninput to the problem.

More formally:

• Define the notion of computability via a formal machine, e.g. aTuring Machine, or a Universal Register Machine, or some othermathematically equivalent computational device.

• Solving a problem is then related to computing a function (via theformal device) from a set of inputs to its set of outputs.

• Unsolvable problems are then those where no correspondingfunction can be computed on the formal device within a finiteamount of time, for any finite input.

Algorithmic Complexity

Unsolvable does not mean Unsolved!

There are many unsolved (mathematical) problems.

We generally refer to these as open problems.

Occasionally, an open problem is solved, e.g. Fermat’s lasttheorem.

And sometimes, an open problem is proved to be unsolvable, e.g.Hilbert’s 10th Problem on whether there are integer solutions toDiophantine Equations (Matiyashevich - 1970)

Algorithmic Complexity

Unsolvable does not mean Unsolved!

There are many unsolved (mathematical) problems.

We generally refer to these as open problems.

Occasionally, an open problem is solved, e.g. Fermat’s lasttheorem.

And sometimes, an open problem is proved to be unsolvable, e.g.Hilbert’s 10th Problem on whether there are integer solutions toDiophantine Equations (Matiyashevich - 1970)

Algorithmic Complexity

Unsolvable does not mean Unsolved!

There are many unsolved (mathematical) problems.

We generally refer to these as open problems.

Occasionally, an open problem is solved, e.g. Fermat’s lasttheorem.

And sometimes, an open problem is proved to be unsolvable, e.g.Hilbert’s 10th Problem on whether there are integer solutions toDiophantine Equations (Matiyashevich - 1970)

Algorithmic Complexity

Unsolvable does not mean Unsolved!

There are many unsolved (mathematical) problems.

We generally refer to these as open problems.

Occasionally, an open problem is solved, e.g. Fermat’s lasttheorem.

And sometimes, an open problem is proved to be unsolvable, e.g.Hilbert’s 10th Problem on whether there are integer solutions toDiophantine Equations (Matiyashevich - 1970)

Algorithmic Complexity

The Halting Problem

This is one of the most famous unsolvable problems.Informally:

Determine whether an arbitrary given algorithm, orcomputer program, will eventually halt for any givenfinite input.

Its unsolvability was proved by Turing circa 1936.

Algorithmic Complexity

The Halting Problem

This is one of the most famous unsolvable problems.Informally:

Determine whether an arbitrary given algorithm, orcomputer program, will eventually halt for any givenfinite input.

Its unsolvability was proved by Turing circa 1936.

Algorithmic Complexity

A Tiling System

Algorithmic Complexity

An Unsolvable Tiling Problem

Given a tiling system, as above, a tiling is a mapping of tiles tosquares in the quadrant.

The problem of determining whether, given an arbitrarytiling system, there is a tiling by that system isunsolvable.

Algorithmic Complexity

A Unsolvable Logical Problem

The problem of determining whether a given arbitraryformula ϕ of first order logic, i.e. predicate calculus, isvalid, i.e. evaluates true in all possible models, isunsolvable.

In fact, the problem is partially solvable in that if the formula isindeed valid an algorithm can be constructed to determine so, butif the formula is not valid, then the algorithm may or may notterminate.

Algorithmic Complexity

A Unsolvable Logical Problem

The problem of determining whether a given arbitraryformula ϕ of first order logic, i.e. predicate calculus, isvalid, i.e. evaluates true in all possible models, isunsolvable.

In fact, the problem is partially solvable in that if the formula isindeed valid an algorithm can be constructed to determine so, butif the formula is not valid, then the algorithm may or may notterminate.

Algorithmic Complexity

Outline

Algorithmic ComplexityMotivationUnsolvable ProblemsSolvable ProblemsTheoretical UnderpinningsSummary

Algorithmic Complexity

Measuring the Difficulty of Solvable Problems

Our motivation used terms such as easy and hard.

We must make these concepts more precise.

Two critical computational resources are:

• Time — how long does it take to solve this problem

• Space — how much memory is required ...

We need to measure these quantities as functions of the problemsize.

Additionally, we can measure complexity in terms of:

• best-case

• average-case

• worst-case

Algorithmic Complexity

Measuring the Difficulty of Solvable Problems

Our motivation used terms such as easy and hard.

We must make these concepts more precise.

Two critical computational resources are:

• Time — how long does it take to solve this problem

• Space — how much memory is required ...

We need to measure these quantities as functions of the problemsize.

Additionally, we can measure complexity in terms of:

• best-case

• average-case

• worst-case

Algorithmic Complexity

Measuring the Difficulty of Solvable Problems

Our motivation used terms such as easy and hard.

We must make these concepts more precise.

Two critical computational resources are:

• Time — how long does it take to solve this problem

• Space — how much memory is required ...

We need to measure these quantities as functions of the problemsize.

Additionally, we can measure complexity in terms of:

• best-case

• average-case

• worst-case

Algorithmic Complexity

Measuring the Difficulty of Solvable Problems

Our motivation used terms such as easy and hard.

We must make these concepts more precise.

Two critical computational resources are:

• Time — how long does it take to solve this problem

• Space — how much memory is required ...

We need to measure these quantities as functions of the problemsize.

Additionally, we can measure complexity in terms of:

• best-case

• average-case

• worst-case

Algorithmic Complexity

Informal Examples - I

Consider the time complexity of an algorithm which performs alinear search of an array.

We will take the problem size as the length of the array, say, n.

In the worst case, one may have to search the whole array to find aparticular element. Assume that the time cost to check a givenarray element is c1, and some initialisation cost is k2, then theworst-case time complexity may be taken as c1n + k1.

Another search algorithm may have worst-case time complexity2c2n + k2.

However, a third algorithm may have worst-case time complexity9c3n√

n + k3.

Which is better and why?

Algorithmic Complexity

Informal Examples - I

Consider the time complexity of an algorithm which performs alinear search of an array.

We will take the problem size as the length of the array, say, n.

In the worst case, one may have to search the whole array to find aparticular element. Assume that the time cost to check a givenarray element is c1, and some initialisation cost is k2, then theworst-case time complexity may be taken as c1n + k1.

Another search algorithm may have worst-case time complexity2c2n + k2.

However, a third algorithm may have worst-case time complexity9c3n√

n + k3.

Which is better and why?

Algorithmic Complexity

Informal Examples - I

Consider the time complexity of an algorithm which performs alinear search of an array.

We will take the problem size as the length of the array, say, n.

In the worst case, one may have to search the whole array to find aparticular element. Assume that the time cost to check a givenarray element is c1, and some initialisation cost is k2, then theworst-case time complexity may be taken as c1n + k1.

Another search algorithm may have worst-case time complexity2c2n + k2.

However, a third algorithm may have worst-case time complexity9c3n√

n + k3.

Which is better and why?

Algorithmic Complexity

Informal Examples - I

Consider the time complexity of an algorithm which performs alinear search of an array.

We will take the problem size as the length of the array, say, n.

In the worst case, one may have to search the whole array to find aparticular element. Assume that the time cost to check a givenarray element is c1, and some initialisation cost is k2, then theworst-case time complexity may be taken as c1n + k1.

Another search algorithm may have worst-case time complexity2c2n + k2.

However, a third algorithm may have worst-case time complexity9c3n√

n + k3.

Which is better and why?

Algorithmic Complexity

Asymptotic Rates of Growth

In measuring algorithmic complexity, we are usually interested inthe rate of growth of the particular measure, rather than its precisevalue.

Hence, for the first two examples above, we don’t find it helpful todistinguish them. Their rate of growth is the same.

We use the Big-Oh O(f ) notation to denote a set of functions thatgrow no faster than the function f .

Formally, given f : N→ R∗, O(f ) is the set of functionsg : N→ R∗ such that for some c ∈ R+ and n0 ∈ N, g(n) ≤ cf (n)for all n ≥ n0.

We use Θ(f ) to denote the set of functions that grow at the samerate as f .

Algorithmic Complexity

Asymptotic Rates of Growth

In measuring algorithmic complexity, we are usually interested inthe rate of growth of the particular measure, rather than its precisevalue.

Hence, for the first two examples above, we don’t find it helpful todistinguish them. Their rate of growth is the same.

We use the Big-Oh O(f ) notation to denote a set of functions thatgrow no faster than the function f .

Formally, given f : N→ R∗, O(f ) is the set of functionsg : N→ R∗ such that for some c ∈ R+ and n0 ∈ N, g(n) ≤ cf (n)for all n ≥ n0.

We use Θ(f ) to denote the set of functions that grow at the samerate as f .

Algorithmic Complexity

Asymptotic Rates of Growth

In measuring algorithmic complexity, we are usually interested inthe rate of growth of the particular measure, rather than its precisevalue.

Hence, for the first two examples above, we don’t find it helpful todistinguish them. Their rate of growth is the same.

We use the Big-Oh O(f ) notation to denote a set of functions thatgrow no faster than the function f .

Formally, given f : N→ R∗, O(f ) is the set of functionsg : N→ R∗ such that for some c ∈ R+ and n0 ∈ N, g(n) ≤ cf (n)for all n ≥ n0.

We use Θ(f ) to denote the set of functions that grow at the samerate as f .

Algorithmic Complexity

Asymptotic Rates of Growth

In measuring algorithmic complexity, we are usually interested inthe rate of growth of the particular measure, rather than its precisevalue.

Hence, for the first two examples above, we don’t find it helpful todistinguish them. Their rate of growth is the same.

We use the Big-Oh O(f ) notation to denote a set of functions thatgrow no faster than the function f .

Formally, given f : N→ R∗, O(f ) is the set of functionsg : N→ R∗ such that for some c ∈ R+ and n0 ∈ N, g(n) ≤ cf (n)for all n ≥ n0.

We use Θ(f ) to denote the set of functions that grow at the samerate as f .

Algorithmic Complexity

The Not So Hard Problems — P

Assume n is the problem size.

An algorithm is said to be at worst linear if its worst-case timecomplexity is in Θ(n).

It is quadratic if it is in Θ(n2), cubic if in Θ(n3), etc..

We define the complexity class P to be the set of all decisionproblems whose worst-case time complexity is in O(nk) for somek ∈ N. Thus, P contains all the decision problems that can besolved in polynomial time.

We refer to the members of P as the not so hard problems (andsome say, easy)! These problems are polynomially time-bounded.

Algorithmic Complexity

The Not So Hard Problems — P

Assume n is the problem size.

An algorithm is said to be at worst linear if its worst-case timecomplexity is in Θ(n).

It is quadratic if it is in Θ(n2), cubic if in Θ(n3), etc..

We define the complexity class P to be the set of all decisionproblems whose worst-case time complexity is in O(nk) for somek ∈ N. Thus, P contains all the decision problems that can besolved in polynomial time.

We refer to the members of P as the not so hard problems (andsome say, easy)! These problems are polynomially time-bounded.

Algorithmic Complexity

The Not So Hard Problems — P

Assume n is the problem size.

An algorithm is said to be at worst linear if its worst-case timecomplexity is in Θ(n).

It is quadratic if it is in Θ(n2), cubic if in Θ(n3), etc..

We define the complexity class P to be the set of all decisionproblems whose worst-case time complexity is in O(nk) for somek ∈ N. Thus, P contains all the decision problems that can besolved in polynomial time.

We refer to the members of P as the not so hard problems (andsome say, easy)! These problems are polynomially time-bounded.

Algorithmic Complexity

The Not So Hard Problems — P

Assume n is the problem size.

An algorithm is said to be at worst linear if its worst-case timecomplexity is in Θ(n).

It is quadratic if it is in Θ(n2), cubic if in Θ(n3), etc..

We define the complexity class P to be the set of all decisionproblems whose worst-case time complexity is in O(nk) for somek ∈ N. Thus, P contains all the decision problems that can besolved in polynomial time.

We refer to the members of P as the not so hard problems (andsome say, easy)! These problems are polynomially time-bounded.

Algorithmic Complexity

Harder problems

Problems whose worst-case (time) complexity grow faster than anypolynomial, i.e. a problem not falling within P, are said to be hard.

For example, algorithms with exponential growth rates, or factorial,etc.

Given a propositional logic formula over n variables, thereare 2n possible valuations of the propositional variables.A naive algorithm to determine satisfiability of a givenformula may need to check all 2n possible valuations.Such an algorithm is therefore of time complexity Θ(2n).This is known as the Boolean SAT problem.

It is unknown whether there is an algorithm for SAT that falls inP. If anyone finds one, they’ll obtain a Nobel prize .... SUCH arethe consequences, as we will later see.

Algorithmic Complexity

Harder problems

Problems whose worst-case (time) complexity grow faster than anypolynomial, i.e. a problem not falling within P, are said to be hard.

For example, algorithms with exponential growth rates, or factorial,etc.

Given a propositional logic formula over n variables, thereare 2n possible valuations of the propositional variables.A naive algorithm to determine satisfiability of a givenformula may need to check all 2n possible valuations.Such an algorithm is therefore of time complexity Θ(2n).This is known as the Boolean SAT problem.

It is unknown whether there is an algorithm for SAT that falls inP. If anyone finds one, they’ll obtain a Nobel prize .... SUCH arethe consequences, as we will later see.

Algorithmic Complexity

Harder problems

Problems whose worst-case (time) complexity grow faster than anypolynomial, i.e. a problem not falling within P, are said to be hard.

For example, algorithms with exponential growth rates, or factorial,etc.

Given a propositional logic formula over n variables, thereare 2n possible valuations of the propositional variables.A naive algorithm to determine satisfiability of a givenformula may need to check all 2n possible valuations.Such an algorithm is therefore of time complexity Θ(2n).This is known as the Boolean SAT problem.

It is unknown whether there is an algorithm for SAT that falls inP. If anyone finds one, they’ll obtain a Nobel prize .... SUCH arethe consequences, as we will later see.

Algorithmic Complexity

Outline

Algorithmic ComplexityMotivationUnsolvable ProblemsSolvable ProblemsTheoretical UnderpinningsSummary

Algorithmic Complexity

Theoretical Underpinnings

Just as one can use a formal mathematical device, say a TuringMachine, to formalise the notion of non-computability, orunsolvable problems, so we can use Turing Machines to determinewhat complexity classes an algorithm falls within.

Without going into details, the algorithm is coded as a TM, andthe algorithm’s input encoded onto the input tape of the machine.

The time complexity of the machine is then determined by thenumber of steps it makes in order to terminate with the giveninput (hence a function from N→ N).

The space complexity of the machine is determined by the numberof cells of the tape that are used in order to terminate on a giveninput (again, a function from N→ N).

Algorithmic Complexity

Theoretical Underpinnings

Just as one can use a formal mathematical device, say a TuringMachine, to formalise the notion of non-computability, orunsolvable problems, so we can use Turing Machines to determinewhat complexity classes an algorithm falls within.

Without going into details, the algorithm is coded as a TM, andthe algorithm’s input encoded onto the input tape of the machine.

The time complexity of the machine is then determined by thenumber of steps it makes in order to terminate with the giveninput (hence a function from N→ N).

The space complexity of the machine is determined by the numberof cells of the tape that are used in order to terminate on a giveninput (again, a function from N→ N).

Algorithmic Complexity

Theoretical Underpinnings

Just as one can use a formal mathematical device, say a TuringMachine, to formalise the notion of non-computability, orunsolvable problems, so we can use Turing Machines to determinewhat complexity classes an algorithm falls within.

Without going into details, the algorithm is coded as a TM, andthe algorithm’s input encoded onto the input tape of the machine.

The time complexity of the machine is then determined by thenumber of steps it makes in order to terminate with the giveninput (hence a function from N→ N).

The space complexity of the machine is determined by the numberof cells of the tape that are used in order to terminate on a giveninput (again, a function from N→ N).

Algorithmic Complexity

Theoretical Underpinnings

Just as one can use a formal mathematical device, say a TuringMachine, to formalise the notion of non-computability, orunsolvable problems, so we can use Turing Machines to determinewhat complexity classes an algorithm falls within.

Without going into details, the algorithm is coded as a TM, andthe algorithm’s input encoded onto the input tape of the machine.

The time complexity of the machine is then determined by thenumber of steps it makes in order to terminate with the giveninput (hence a function from N→ N).

The space complexity of the machine is determined by the numberof cells of the tape that are used in order to terminate on a giveninput (again, a function from N→ N).

Algorithmic Complexity

Outline

Algorithmic ComplexityMotivationUnsolvable ProblemsSolvable ProblemsTheoretical UnderpinningsSummary

Algorithmic Complexity

Summary - so far

• Introduced basic notions solvable and unsolvable problems

• Introduced basic measures for determining the difficulty ofsolvable problems

• Introduced the P complexity class — the class of easydecision problems!

• Indicated there are much harder problems

Next, we will introduce the class NP — non-deterministicpolynomial time bounded algorithms.

Algorithmic Complexity

Summary - so far

• Introduced basic notions solvable and unsolvable problems

• Introduced basic measures for determining the difficulty ofsolvable problems

• Introduced the P complexity class — the class of easydecision problems!

• Indicated there are much harder problems

Next, we will introduce the class NP — non-deterministicpolynomial time bounded algorithms.

Algorithmic Complexity

Summary - so far

• Introduced basic notions solvable and unsolvable problems

• Introduced basic measures for determining the difficulty ofsolvable problems

• Introduced the P complexity class — the class of easydecision problems!

• Indicated there are much harder problems

Next, we will introduce the class NP — non-deterministicpolynomial time bounded algorithms.

Algorithmic Complexity

Summary - so far

• Introduced basic notions solvable and unsolvable problems

• Introduced basic measures for determining the difficulty ofsolvable problems

• Introduced the P complexity class — the class of easydecision problems!

• Indicated there are much harder problems

Next, we will introduce the class NP — non-deterministicpolynomial time bounded algorithms.

Algorithmic Complexity

Summary - so far

• Introduced basic notions solvable and unsolvable problems

• Introduced basic measures for determining the difficulty ofsolvable problems

• Introduced the P complexity class — the class of easydecision problems!

• Indicated there are much harder problems

Next, we will introduce the class NP — non-deterministicpolynomial time bounded algorithms.