CSCI 4325 / 6339Theory of ComputationChapter One
Zhixiang Chen
Department of Computer Science
University of Texas-Pan American
Background
Experience in at least one high-level programming language
Data structures Discrete math Others …
Topics
What is computation? Or what is an algorithm?
Formal languagesand computing devices? Computation vs. Formal language understanding?
What can be computed? Or what cannot be computed?
What can be efficiently computed? Or what cannot be efficiently computed?
Reading Materials
Chapter 0, Introduction
The World of Theoretical Foundations of Computation
…
PARTIALLY COMPUTABLE
COMPUTABLE
PSPACE
NEXP
EXP
NP
P
CHAPTER 0Introduction
Automata, Computability, and Complexity Want to know
What are the fundamental capabilities and limitations of computers?
Complexity
What makes some problem computationally hard and others easy?
How to classify problems according to their computational difficulties?
How to understand the computational difficulties in the worst case, randomized, or approximate settings?
How to harness the computational difficulties for applications like cryptography?
Computability
What are limitations of Computers? Want to know what problems are solvable by
computers, and what are not? Note: In complexity, we are interested in
“solvable” problem, but want to know how difficult to solve.
Automata
Automata theory deals with the definitions of properties of mathematical models of computation. These models play a role in several applied areas of computer science. Finite automata = regular expression: for text processing Context-free grammar: for programming language design and
implementation, AI, natural language processing Context-sensitive grammar: for natural language processing
Sets
What is a Set? A set is a collection of objects
Ex’s? Notes:
We use capital letters to denote sets An object contained in a set is called a set element. We use lower-
case letters to denote set elements. Notations
Given a set A and an object x, if x is an element of A, we write
Otherwise we write
Ax
Ax
Finite Sets and Infinite Sets
A set is finite if it has a finite number of elements Ex’s?
A set is infinite if it has an infinite number of elements Ex’s?
Set Cardinality
Given a set A, the cardinality of A, denoted by |A|, is the number of elements
in A. Ex’s?
Empty set
The empty set is a set without elements. The empty is denoted as Ø.
Set Representation
By enumeration: List all elements. Only good for finite sets Ex’s.
A = {1, 2, 3, 4, 5}, B = {a, b, c, d, e} C = { Joe, Jose, Chen_1, Chen_ 2} D = { Black_Cat, White_Cat}
By Property: Give property for set elements. Good for infinite sets (and
finite sets as well) Ex’s.
A = {x: x is an integer} B = {y: y is a student at UTPA} C = {z: z is a student in the CS department} D = {w: and w is even}Aw
Set Operations
Union A U B is a set of elements in either A or B A U B = {x: }
Intersection is a set of elements in both A and B.
Difference A – B is a set of elements in A but not in B.
BxAx
BA
}:{ BxAxxBA
}:{ BxAxxBA
Set Operations
Subset if any implies
Proper Subset if and
Equivalence A = B if
Want example’s?
BxAxBA
BA BA BA
ABBA
Properties of Set Operation Idempotency
A U A = A Commutativity
Associativity
ABBA
ABBA
)()(
)()(
CBACBA
CBACBA
Properties of Set Operation Distributivity
Absorption
DeMorgan’s Law
)()()(
)()()(
CBCACBA
CBCACBA
AABA
AABA
)(
)(
)()()(
)()()(
CABACBA
CABACBA
Can You See DeMorgan’s Law?
A
B
C
)()()(
)()()(
CABACBA
CABACBA
Disjoint Sets and Power Sets Disjoint Sets
A and B are disjoint if Power Sets
Given a set A, the power set of A, denoted by is the collection of all subsets of A.
Ex’s.
A = {1, 2, 3} = {, {1}, {2}, {3}, {1, 2}, {1, 3}, {2, 3}, {1, 2, 3}}
.BA
A2
}.:{2 AXXA
A2
Two Notations
Let S be a set such that any element in S is also a set, define
Ex’s
What is ? What is ?
},:{
},:{
PxSPxPS
PxSPxPS
SP
SP
A
A
2
2
Set Partition
Given a nonempty set A, we say is a partition of A if Any two elements of are disjoint.
AXX
A
}{2
Set Partition Examples
Ex’s Let S = {a, b, c, d} What partition of S has the fewest members? The
most members? List all partitions of S with exactly two members?
Relations and Functions- Ordered Pairs Given any two objects a, b, let (a, b) denote the
ordered pair of a and b. The ordered pair of a and b tells that a and b is
related under some relationship The differences between (a, b) and {a, b}:
(a, b) (b, a), while {a, b} = {b, a} (a, a) is a good ordered pair, while {a, a} is not.
The equivalence (a, b) = (c, d) if and only if a = c, b = d.
Ordered Tuples
Given any n objects is an n-ary ordered tuple.
Note: The objects may not be distinct.
Ordered tuple examples?
,,,, 21 naaa ),,,( 21 naaa
,,,, 21 naaa
Cartesian Products
Given two sets A and B, the Cartesian Product of A and B is
Given n sets , the n-fold Cartesian Product of these sets is
Ex’s?
}:),{( BbAabaBA
}1,:),,,{( 2121 niAaaaaAAA iinn
nAAA ,,, 21
Relations
A binary relation is defined as
A n-ary relation is defined as
BAR :BAR
BAAAR n 21:
BAAAR n 21
Functions
A function is a binary relation on A and B such that there is exactly one such that A is called the domain of f.
For any , we denote y as f(x). f(x) is called the image (or value) of x.
The range of f is defined as f [A] = { y: y = f(x) for some }
BAf :
,Ax By .),( fyx
Ax
fyx ),(
Function Mapping
A Bf
One-to-One, Onto, Bijection
One-to-One A function is one-to-one if for any two distinct
elements Onto
A function is onto B, if
Bijection A function is a bijection if f is one-to-one and
onto
BAf :
).()(,, bfafAba
BAf :
).(,, xfyAxBy
BAf :
Inverse
The inverse of a binary relation is
The inverse of a function may not be a function any more. Why?
If a function is bijection then its inverse is still a function. Why?
BAR :}),(:),{(1 RbaabR
Composition
Let Q and R to be two relations
Then RQ is also a relation
,: BAQ CBR :
CARQ :
}),(),(,:),{( RzyQyxByzxRQ
A B C
Q R
Composition Examples
Let R = {(a, b), (a, c), (c, d), (a, a), (b, a)}, What is RR? What is ? Is R, RR or a function?
1R1R
Graph Representation of a Binary Relation Given a binary relation
for each draw a node to represent a, draw another node to represent b, draw an edge from a to b. Then, we obtain a directed graph for R.
,: AAR
,),( Rba
An Graph Representation Example Let R = {(a, b), (b, a), (a, d), (d, c), (c, c),
(c, a)}. The Graph is
a b
cd
Reflexive
A relation R: A→A is reflexive, if for any
For a reflexive relation R, the directed graph for R has a self-circle for each node
Graph example? Other examples?
The less-than-or-equal-to relation over integers
.),(, RaaAa
Symmetric
A relation R: A→A is symmetric if for any
In the directed graph of a symmetric relation, if there is an edge from node a to node b, then there must be another edge from node b to node a.
Graph examples? Other Examples?
The friend relation over a group of people.
.),(,),( RabRba
Antisymmetric
A relation R: A A is antisymmetric if for any
In the direct graph of a antisymmetric relation, if there is an edge from a to b, then there must be no edge from b to a.
Some graph examples? Other examples?
The parent relation over a group of people.
.),(,),( RabRba
Transitive
A relation R:A A is transitive if (a, b) and(a, c) are in R, then (a, c) is also in R.
In the directed graph of a transitive relation, if there a path from a to b, then there is an edge from a to b.
Some graph examples? Other examples?
The ancestor relation over a group of people. The less-than relation over integers.
Equivalence Relation
A relation R: A A is equivalent if R is reflexive, symmetric and transitive.
Let R be an equivalence relation, for any a in A, we define the equivalence class containing the element a as [a] = {x: (a, x) in R} That is, [a] is the set of all elements related to a.
Examples? The same-family relation The x=y (mod p) relation for the given p.
A Equivalence Relation Theorem Theorem. Let R: A→A to be an equivalence
relation, then the equivalence classes of R is a partition of A. That is {[a]: a in A} is a partition of A.
Proof.
AaAa
][
],[2,1 aSSAa
.2121 SSorSSeither
Partial Order A relation R: A→A is a partial order if R is
reflexive, antisymmetric, and transitive. Ex’s
Some graph examples? The boss-employee relation, assuming any
person is the boss of him/herself. Minimal elements of a partial order R: A→A
An element a in A is minimal if for any b in A, (b, a) in R implies b = a.
Ex’s
Total Order, Path, Length, Cycles Total Order
A relation R: A A is a total order, if R is a partial order, and for any a, b in A, either (a, b) in R or
(b, a) in R. Path, Length, Cycles
Easy to see from the directed graph of a relation R.
Examples?
Countable and Uncountable Sets Countable
A set S is countable if one of the following is true S is finite S is infinite but there is a bijection f: SN, where N is
the set of natural numbers. In other words, S is countable, if elements in S are
enumerable. Countable set examples? Uncountable
Sets that are not countable
Unions of Countable Sets
The union of a finite collection of countable sets is countable Proof. By dovetailing
The Union of a countably infinite collection of countable sets is countable Proof. By dovetailing
There Are Uncountable Sets
Proof. By diagonalization. Will give a proof later.
Definitions
Definitions describe the objects and notions that we use. A definition may be simple, or complex. Precision is essential to any mathematical definition. When defining some object we must make clear what constitutes that object and what does not.
Theorems
A theorem is a mathematical statement proved true. Generally a theorem is used to refer to a mathematical statement of special interest. Sometimes we prove statements that are interesting only because they assist in the proof of another, more significant statement. Such statements are called lemmas. Occasionally a theorem or its proof may allow us to conclude easily that other, related statements are true. These statements are called corollaries of the theorem.
Proofs
A proof is a convincing logical argument that a statement is true. In mathematics an argument must be airtight, that is, convincing in an absolute sense. This is rather different from the notion of proof that we use in everyday life or in the law. A mathematician demands proof beyond any doubt.
Five Fundamental Proof Techniques Proof by Mathematical Induction Proof by Construction Proof by Contradiction Proof by Pigeon Hole Principle Proof by Diagonalization
Mathematical Induction
The Principle of Math Induction: Let A be a set of natural numbers such that
and For each natural number , if then ,
then, A = N
Math Induction can be used to prove the following assertion For all natural numbers n, property P(n) is true.
A00n An 1An
Three Steps of Math Induction To show property P(n) is true for all natural
numbers, follow the following three steps: Base case: show P(0) is true. Induction hypothesis: Assume P(n) is true Induction: show P(n+1) is true.
Math Induction Example
Show that for any
Proof. (Prove in class.)
,0n
2
)1(21
nnn
Proof by Construction
Many theorems state that a particular type of object exists. One way to prove such a theorem is by demonstrating how to construct the object.
Theorem. For any even number n greater than 2, there exists a graph with n nodes such that every node has exactly 3 edges connecting to it.
Proof. By Construction.
Proof by Contradiction
We assume that the theorem is false and then show that this assumption leads to an obviously false consequence, called a contradiction.
Theorem. is irrational. Proof. By contradiction. (do it in class.)
2
The Pigeonhole Principle
The Principle. Let m pigeons to sit in n holes, if m > n then there is at least one hole with at least 2 pigeons.
Ex: Let R to be a binary relation on a finite set A, and let a, b be in A. If there is a path from a to b in R, then there is a path of length at most |A|.
Proof . By Pigeonhole Principle.
Diagonalization
The Principle. Let R to be a binary relation on a set A, define D = {a: a in A and (a, a) not in R}
For any a in A, define = {b: b in A and (a, b) in R}
Then, for any a in A,
aD
aDD
Applications of Diagonalizations
Theorem 1. Let [0, 1] denote the set of all real numbers between 0 and 1. Then it is uncountable.
Proof. By diagonalization. (Do it in class.)
Theorem 2. is uncountable, where N is the set of all nature numbers.
Proof. By diagonalization. (Do it in class.)
N2
Closures and Algorithms
The idea of reflexive transitive closure of a relation R is to find a smallest reflexive and transitive relation to cover R.
Ex.
R:
a1 O O a4
a2 O O a3
R*: a1 a4
a2 a3
R* is reflexive and transitive closure of R.
Reflexive Transitive Closure
Definition: Let R: A X A be a relation. The transitive closure of R is the relation R* = {(a, b): a, b are in A and there is a path from
a to b in R or a = b} Give R, how to find the transitive closure R*
of R? Suppose A = {a1, a2, . . . , an}
Closure Algorithm 1
Algorithm 1 Initially R* = For i = 1, . . . , n do For each i-tuple (b1, . . ., bi) in do
if (b1, . . ., bi) is a path in R then add (b1, bi) to R
iA
Closure Algorithm 2
Algorithm 2 Initially R* = R {(ai, ai) : ai in A} For j = 1, 2, . . ., n do
For i = 1, 2, . . ., n and k = 1, 2, . . ., n do If (ai, aj), (aj, ak) are in R* but (ai, ak) are not in R* then
add (ai, ak) to R*
Algorithm Analysis
How to show the correctness of the two closure algorithms? Do they halt on any given input? If halt, Do they output the right result? What are the performances of the two algorithms
Time complexity Space complexity
How to analyze the performance?
Big-O Notation
Let f, g be two functions from N to N. f(n) = O(g(n)) if there are two constants c and such that for all
Ex. But
More Ex’s ?
0nn)()( ncgnf
0n
)(16100 525 nOnn )(16100 225 nOnn
Time Complexity of the Two Closure Algorithms
Algorithm 1
Algorithm 2
Which algorithm is better?
)( 1nnO
)( 3nO
Alphabets and Languages
Consider Pascal Consider English Observation:
Language = Alphabet + Syntax + Semantics Alphabet
A finite set of symbols Ex’s?
{0, 1}, {a, . . . , z, A, . . . , Z}
Strings
A string of an alphabet A sequence of symbols from the alphabet.
Ex’s 1010 of {0, 1} ILovetheory of {a, . . . , z, A, . . . , Z}
Empty string a string without symbols denoted by e.
String Length
The length of a string w the number of occurrences of symbols in w,
denoted by |w| Ex’s
|1010|= 4 |ILoveTheory|= 11 |e|=0
String Concatenation
Concatenation of two strings x, y: xy is the string of x followed by y. |xy|=|x| +|y|
Substring, Prefix, Suffix
z is a substring of w if there are strings x, y such that w = xyz Ex. Find substrings of “Iiovetheory”
z is a prefix of w if there is a string x such that w = zx Ex. Find prefixes of “Iiovetheory”
z is a suffix of w if there is a string x such that w = xz Ex. Find suffixes of “Iiovetheory”
Power of a String
The power of a string w Given any natural number i,
.0
,0,1 iifww
iifew ii
Reversal of a String
The reversal of a string w
where a is some letter in the alphabet
Property
,
,0||,
auwifau
wifew RR
RRR xyxy )(
Σ, Σ*, and Languages
Let Σ be any finite alphabet {a1, a2, . . . , an} Σ* denote the set of all strings of Σ Any subset L of Σ* is called a language over Σ. Ex’s: Pascal, English, binary numbers, decimal
numbers, etc.
Note: Languages are defined as sets. Thus, all set operations such as union, intersection, difference, complement are all applicable to languages.
Concatenation and Kleene Star Concatenation of Languages: Given two
languages and , the concatenation
of and is
Kleene Star: the Kleene star of a language L is
Ex’s = {a, b}, = {11}
},,,,0:{ 2121* LwwwkwwwL kk
}:{ 22112121 LwLwwwLL
2L1L
1L
1L
2L
2L
The closure of a language
The closure of a language L:
Ex. Find the closures of the following languages L1 = {a, b}, L2 = {11}
},,,,1:{ 2121* LwwwkwwwLLL kk
Finite Representations of Languages The Large Picture
Given an alphabet Σ, the collection of all languages over Σ is
when , Σ* is countably infinite,
is uncountably infinite.
1||
*
2
*
2
Finite Representation of a Language A finite representation of a language
is a finite string of some other finite alphabet .
The total number of languages represented by strings of is .
is countably infinite.
Can you get some conclusion?
*
2L
|| *
*
The Language World
All languages
Finitely representable languages
Regular Expressions
Definition. Regular expressions over a given alphabet Σ are defined as follows: (1) Φ and any symbol of Σ are regular exp. (2) If α and β are regular expressions, then so are
(α β), (α U β ), and α* (3) Only those derived from (1) and (2) are regular
expressions Ex’s. Σ = {a, b}
Φ, a, b, (ab), (aa)*, (aabb)*, (aba)*(bb)*
Regular Languages
Languages defined by regular expressions Given a regular expression α, let L(α) denote the
language represented by α. We have L(Φ) = Φ, L(a) = {a} for a Σ L(α U β ) = L(α) U L(β) L(αβ) = L(α) L(β) L(α*) = L(α)*
Ex’s (give examples in class.) A language L is regular if there is a regular
expression α such that L = L(α).
Non-Regular Languages
There are languages which are not regular
Ex’s
}:{
}0:{*
wwwB
nbaAR
nn
Recognizable & Generatable
A and B are recognizable Ex.
We can write a C++ program which, given any string w Σ*, can decide whether w in A or not
A and B are generatable Ex. We can write a C++ program to enumerate all
elements of A, provided we have infinite supply of paper and computing time and space.
A New World of Languages
All languages
Finitely representable languages
Regular languages