THEORY OF COMPUTATIONDr. Adam P. AnthonyLectures 25 and 26

Overview Computer Science: do we need

computers? Computation Theory Functions Turing Machines Universal Programming Languages The Halting problem

Computer Science and ComputersComputer science is no more

about computers than astronomy is about telescopes.

Edgser W. Dijkstra

Computability Insight: the computation is separate, in concept, from

the computer A computer, then, is just some object that can carry

out the computation Humans

Brain, often supplemented by pencil, paper Charles Babbage

Difference engine, analytical engine Controlled using clockwork-type components

ENIAC Controlled using vacuum tubes

Intel 8080 Controlled using micro-transistors

Strength of Computers Can a simple calculator help you find

your way around cleveland? How about a (dumb) phone?

Aside from making calls How about a smart phone? How about a laptop? How about a desktop? Which of these count as computers?

Specific-Purpose vs. General Computers

Some ‘computers’ are designed only to achieve a limited number of specific tasks, and to do that either at high speed or at a low cost: Digital Phones (Cell and otherwise)

Encryption chips too! Various scientific measuring devices

Others are considered General Purpose Computers Anything that can be computed, can be

done on one of these machines

Theory of Computation The theory of computation aims to

answer the following questions: 1. What is a general purpose computer?2. What problems can I solve with a general

purpose computer?3. Is this specific computer general purpose?4. Given a general purpose computer, how

difficult will it be to solve a specific problem?

Alan Turing: “The Father of Computer Science” Successful mathematician Cryptographer Helped build some early

(classified) computation devices Many ideas predated the first

computers Turing Machine Computability

Helped define what is possible on a computer, and what is not

Computable Functions A function is a mapping of inputs to outputs

Sum(2+2) = 4 Feet-Centimeter(500) = 15,240 Sort([1,3,2,3,6,9,7,8,0]) = [0,1,2,3,3,6,7,8,9] Father(Bill Smith) = Edward Smith

Some functions are computable Given the input, an algorithmic process can

always be applied to get an exact answer for the output

A general purpose computer can compute any computable function, and no others

Turing Machines

Control Unit: The actual machine Tape: infinitely long memory Read/Write Head: Used to read information from the tape,

erase information, write new information Reads one character at a time Moves left/right one position at a time

State: Description of current situation, based on tape values

State = START

How a Turing Machine Works1. Each new Turing Machine has an alphabet

of characters that it understands, and a set of states that help it make decisions

2. Given the state the current character read by the the read/write head, and a program of execution, the Control Unit decides to:

a) Stop running (HALT state)b) Write over the current characterc) Move one space left/rightd) Change States

A More Complex Machine Alphabet: {0,1,*}

A single binary number is represented as *101010*

States: ADD,RETURN,CARRY,OVERFLOW,HALT

Program to increase a positive binary integer by 1:

Useful Turing Machine Facts Multiple Turing machines are no more powerful

(though possibly faster) than a single Turing Machine

Any Turing Machine can ‘simulate’ another Turing Machine

Result: We can use unambiguous complex commands in the control unit’s program! Command: “Move 5 Spaces to the left”

Turing Machine reads: “Execute the Turing Machine routine that moves 5 spaces to the left”

Theoretically speaking, one should typically demonstrate the sub-program is computable first

Church-Turing Thesis

Any function that can be computed using a Turing Machine is also computable using any other general purpose computer (i.e., the function is computable)

SO WHAT???Who Cares?

Impact of Church-Turing Thesis1. If Power = ‘number of functions I can

compute,’ then a Turing machine is the most powerful computer imaginable Or, at least, it ties with any other computer It computes ALL of the computable functions!

2. If a Turing machine can’t solve a problem, then neither can a real computer, no matter how ‘powerful’ it is

Modern Computers As Turing Machines

Control Unit = Processor Alphabet = {0,1} States = Op Codes Read/Write Head = BUS Programs = Software Tape = RAM

Infinite????? No, but for most purposes it is long enough to solve the

problem Tape = External Storage

Only limited by the number of natural resources we can obtain from the entire universe (so, probably infinite!)

Bare-Bones Computer Language Programming languages usually market their

‘features’ Meant to make programming easier

Bare-Bones Language: Only includes features that are 100% necessary to be

equivalent to a Turing machine: Variable names: all variables are in binary clear statement: set a variable = 0 (clear X;) incr statement: increase a variable by 1 ( incr X; ) decr statement: decrease a variable by 1 (decr X;) While/end: continue execution until a variable = 0

while X not 0 do;….end;

Group Work! Can you use Bare-Bones to:

Set the variable Z = 4? Add X + Y = Z? (use one variable each for

X,Y,Z) Copy the value of X into Y?

About the Bare-Bones Language Computer scientists have proven that any

computer that can execute the Bare-Bones language is equivalent in power to a Turing Machine Heaven forbid!

Useful conclusion: Any programming language does at least

the same as Bare-Bones (hopefully more!) will also be Turing Equivalent

The extra features are just for convenience

Where We’ve Been Computers are just tools for completing

computations Theory of computation: what is possible/impossible

for all computers? What is computable? Turing Machine: imaginary ‘all powerful’ computer

Church-Turing thesis states no computer can do better Modern computers are equivalent to Turing

Machines Any algorithm we implement on a computer is

computable

Where We’re headed It’d be nice to know, before we start if a

problem is noncomputable Halting problem as an example

Even if a problem is computable, it would be nice to know in advance if it is easy or hard to solve

Even if we can solve a problem, it would be nice to know how long it will take to solve it Save effort in solving complex problems Take advantage of complexity

The Halting Problem Some problems can’t be solved. Consider: Given the source code for any

computer program, can you analyze the code and decide if it will it ever stop running?

The Halting ProblemDoes this program halt?

int X = 3while( X > 0)x = x -1

The Halting ProblemDoes this program halt?

int X = 3repeatx = x +1;until x = 0

The Halting Problem

How about this program?virtual void estimate_sigmas(){sigmas = std::vector< std::vector<Matrix> >(num_clusters);for(int i = 0; i<num_clusters; i++){sigmas[i] = std::vector<Matrix>(num_clusters);}for(int i = 0; i<num_clusters; i++){for(int j = 0; j<num_clusters; j++){sigmas[i][j] = zero_matrix<double>(proper_size,proper_size);}}Vector temp_v(proper_size);edge_iterator ebg,end;int c_i,c_j;for(tie(ebg,end) = edges(data); ebg!=end; ++ebg){if(data[*ebg].type == edge_type && data[*ebg].exists){c_i = data[source(*ebg,data)].clustering();c_j = data[target(*ebg,data)].clustering();temp_v = get_edge_vector(*ebg) - ic_means[c_i][c_j]sigmas[c_i][c_j] = sigmas[c_i][c_j] + outer_prod(temp_v,trans(temp_v))/observed_edge_prob(c_i,c_j);

}}}

Computability Explained Sometimes, we can work out answers for simple,

example inputs of hard problems, but: What algorithm did you use to decide for the first

two programs? Can you generalize it to the third?

To prove something is not computable, we’ll use the following strategy: 1. Assume that there is an algorithm that can solve the

problem all the time2. Show that, regardless of how the algorithm works, that

there is at least one case where the algorithm will fail1. Contradicts part 1, which claimed it ‘always works’

The Halting Problem Is Not Computable (PROOF!)

1. Assume, for the sake of contradiction, that there exists a computer program that, given any other computer program as input, can tell us if it stops:

STOPS(Program)2. Make a new program:

Opposite(X): 1. If STOPS(X), then run forever. 2. Otherwise, stop!

3. Opposite is a program, which itself accepts program code as input. What happens when we try to run

Opposite(Oppisite)?1. If STOPS(Opposite), then Opposite will run forever2. Otherwise, stop!

Halting Problem Implications Existence proof: Since there’s one program that

exists which we can’t compute if it halts, then there may be (probably are) others

If there’s one problem that seems computable, but is not, then there are others Look up Wang Tiles for an interesting example!

Program Analysis: “Does my program compute X?” Any place in the code where X is computed, add a

HALT command Changes to “Does my program ever halt?”

Algorithmic Complexity Algorithmic Complexity refers to how

many resources (time and memory) a computer will need to solve a problem How long will it take to process all the data? How much space (Memory) will we need? If we use more space, will it take less time? Are some problems harder to solve than

others? Can we figure that out before we try to solve them?

How can we take advantage of complexity?

Problems Vs. Solutions It’s one thing to take a specific algorithm and say it’s complex (or

not):PROCEDURE Add(X,Y):

sum = X + YRETURN sum

Because solving the problem, and doing so efficiently, are two different things:

PROCEDURE BadAdd(X,Y):Z = 1000000000sum = 0REPEAT Z = Z - 1UNTIL Z = 0sum = X + YRETURN sum

To say that a PROBLEM is difficult, you need to prove that there are no easy ways to solve it

Run-Time Complexity Looked at briefly in chapter 5 Principal method for analyzing algorithm complexity:

How many steps does it take to complete the entire algorithm?

Steps are often based on the size of the input: PROCEDURE add-all(L):

sum = 0count = 0WHILE count < length(L):sum = sum + L[count]RETURN sum

How many steps to add a list with 10 numbers? 1000 numbers?

Space Complexity Many algorithms only need enough

space to hold the input data Procedure add-all (L) Only needs enough

space to store L Others, because the problem is more

difficult, use supplementary data EX: Binary Search Trees

Still others use extra memory to be faster EX: Dynamic Programming Fibonacci

sequence

Complexity Classes The most reoccurring algorithmic runtimes are (in order)

constant, log(N), N, N*log(N), N2, N3, and an

A polynomial problem is any problem for which the best known algorithm for solving it has time complexity that is no worse than a polynomial function f(N) = Nd where d can be any number and N is the size of the input.

All problems that we can solve with an exact solution is a reasonable amount of time are in the polynomial class

Problems outside this class are referred to as intractible For short, we refer to the entire set of all

polynomial problems in the world a the set P

Complexity Classes, continued. A Nondeterministic machine: is a theoretical

machine that just knows how to solve a problem, no matter how hard it may be

A Nondeterministic Polynomial Problem is a problem for which the best known algorithm for solving has a polynomial runtime, but its execution would require a nondeterministic machine

Another intuition: these are problems for which finding the solution is hard, but checking the solution for correctness is easy

We refer to the set of problems in this domain as NP

NP DOES NOT MEAN “NOT POLYNOMIAL”

NP-Completeness In General, the class of problems in NP consists of

problems that are difficult, but useful Traveling Salesman example—best solution is exponential

Within a single class, some problems are harder than others In P, it is harder to sort a list than it is to add two

numbers In the class NP, we identify a set of problems that are

the most difficult to solve in the entire set Called NP-Complete problems The speed at which we can solve these problems

determines how fast we can solve the lesser problems

P vs. NP All problems that are in the set P must also be

in the set NP Why?

Big, unknown question in Computer Science: DOES P = NP???

What does it mean if P = NP? One approach: Find a polynomial solution for

an NP-Complete problem Thousands have tried, all have failed

Most people believe, but can’t prove P NP