Establishing the Equivalence between Recurrent Neural Networks and Turing Machines.
Ritesh Kumar Sinha(02d05005)
Kumar Gaurav Bijay(02005013)
“No computer has ever been designed that is everaware of what it’s doing; but most of the time, wearen’t either”
– Marvin Minsky.
Introduction
Plan : Establish Equivalence between recurrent neural network and turing machine History of Neurons Definitions Constructive Proof of equivalence
Approach : Conceptual Understanding
Motivation
Understanding the learning patterns of human brain – concept of neuron
Is Turing Machine the ultimate computing machine ?
How powerful are neural networks – DFA, PDA, Turing Machine, still higher …
Brain
The most complicated human organ sense, perceive, feel, think, believe, remember, utter Information processing Centre
Neurons : information processing units
MCP Neuron
McCulloch and Pitts gave a model of a neuron in 1943
But it's only a highly simplified model of real neuron Positive weights (activators) Negative weights (inhibitors)
Artificial Neural Neworks (ANN)
Interconnected units : model neurons
Modifiable weights (models synapse)
Types of ANN
Feed-forward Networks Signals travel in one way only Good at computing static functions
Neuro-Fuzzy Networks Combines advantages of both fuzzy reasoning
and Neural Networks. Good at modeling real life data.
Recurrent Networks
Recurrent Neural Networks
Activation Function:f(x) = x , if x > 0 0 , otherwise
Turing Machine and Turing Complete Languages
Turing Machine: As powerful as any other computer
Turing Complete Language: Programming language that can compute any
function that a Turing Machine can compute.
A language L
Four basic operations: No operation : V V Increment : V V+1 Decrement : V max(0,V-1) Conditional Branch : IF V != 0 GOTO j
(V is any variable having positive integer values, and j stands for line numbers)
L is Turing Complete
Turing Machine can encode Neural Network
Turing machine can compute any computable function, by definition
The activation function, in our case, is a simple non-linear function
Turing machine can therefore simulate our recurrent neural net
Intuitive, cant we write code to simulate our neural net
An example
Function that initiates: Y=X
L1: X X - 1
Y Y + 1
if X != 0 goto L1
An example
Function that initiates: Y=X
if X != 0 goto L1
X X + 1
if X != 0 goto L2
L1: X X - 1
Y Y + 1
if X != 0 goto L1
L2: Y Y
L is Turing Complete : Conceptual Understanding
Idea: Don’t think of C++, think of 8085 ;)
Subtraction: Y = X1 - X2
L is Turing Complete : Conceptual Understanding
Idea: Don’t think of C++, think of 8085 ;)
Subtraction: Y = X1 - X2 Yes, decrement X, increment Y, when Y=0,
stop
Multiplication, division:
L is Turing Complete : Conceptual Understanding
Idea: Don’t think of C++, think of 8085 ;)
Subtraction: Y = X1 - X2
Yes, decrement X, increment Y, when Y=0, stop
Multiplication, division:
Yes, think of the various algos you studied in Hardware Class :)
L is Turing Complete : Conceptual Understanding
If: if X=0 goto L
L is Turing Complete : Conceptual Understanding
If: if X=0 goto L
Yes,
if X != 0 goto L2
Z Z + 1 // Z is dummy
if Z != 0 goto L
L2: ...
Constructing a Perceptron Network for Language L
For each variable V : entry node NV
For each program row i : instruction node Ni
Conditional branch instruction on row i :
2 transition nodes Ni’ and Ni”
Constructions
Variable V :
NV NV
No Operation:
Ni Ni+1
Constructions Continued
Increment Operation :
Ni Ni+1
Ni NV
Decrement Operation:
Ni Ni+1
Ni NV
Constructions Continued
Conditional Branch Operation :
Ni Ni’
Ni Ni’’
Nv Ni’’
Ni’’ Ni+1
Ni’ Nj
Ni’’ Nj
Definitions
Legal State: Transition Nodes (Ni’ and Ni’’) : 0 outputs
Exactly one Ni has output=1
Final State : All instruction nodes have 0 outputs.
Properties
If yi = 1 then program counter is on line i
V = output of Nv
Changes in network state are activated by non-zero nodes.
Proof
•If row i is V V
•If row i is V V - 1
•For V V + 1, behavior is similar
Proof Continued
•If row i is : if V != 0 GOTO j
Proof Continued
•Thus a legal state leads to another legal state.
What about other activation functions?
• Feed-forward Net With Binary State Function is DFA
• Recurrent Neural Network with:1) Sigmoid is Turing Universal2) Saturated Linear Function is
Turing Universal
Conclusion
Turing machine can encode Recurrent Neural Network.
Recurrent neural net can simulate a Turing complete language.
So, Turing machines are Recurrent Neural Networks !
Project
Implementation of Push down automata (PDA) using Neural Networks.
References
[1] McCulloch , W. S., and Pitts, W. : A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5, 1943 :115-133.
[2] Hyotyniemi, H : Turing Machines are Recurrent Neural Networks. In SYmposium of Artificial Networks, Vaasa, Finland, Aug 19-23, 1926, pp. 13-24
[3] Davis, Martin D.Weyuker, Elaine J. : Computability, complexity, and languages : fundamentals of theoretical computer science, New York : Academic Press, 1983.
References Continued
[4] J. E. Hopcroft and J. D. Ullman, Introduction to Automata Theory, Languages and Computation. Addison-Wesley, 1979.
[5] Arbib, M. : Turing Machnies, Finite Automata and Neural Nets. Journal of the ACM (JACM), NY, USA, Oct 1961, Vol 8 , Issue 4 , pp. 467 - 475.