1

Zvi Kohavi and Niraj K. Jha

Memory, Definiteness, and Information Memory, Definiteness, and Information Losslessness of Finite AutomataLosslessness of Finite Automata

2

Memory SpanMemory Span

Memory span: the amount of past input and output information needed to determine the machine’s future behavior

Memory span w.r.t. input-output sequences (finite-memory machines): An FSM M is defined as a finite-memory machine of order , if is the least integer s.t. the present state of M can be determined

uniquely from the knowledge of the last input symbols and the corresponding output symbols

• I.e., every input sequence of length is a homing sequence

3

Testing Table and Testing GraphTesting Table and Testing Graph

Example: Consider machine M1 and its testing table and testing graph

AB BD CD

AC

0/0

1/1

0/00/0

1/0

0/0

BC AD0/0

Uncertainty

pair

Implied

pair

4

Conditions for Finite MemoryConditions for Finite Memory

Theorem: A sequential machine M has a finite memory if and only if its testing graph is loop-free

Example: the testing graph of M1 has two loops – hence it’s not finite memory

• An arbitrary long string of 0 input symbols: will never resolve uncertainty (CD)

• Similarly, if initial uncertainty is (AC): input sequence 0101…01 will transfer the machine to (BD), (AC), (BD), and so on

AB BD CD

AC

0/0

1/1

0/00/0

1/0

0/0

BC AD0/0

5

CorollaryCorollary

Corollary: Let G be a loop-free testing graph for machine M. If the length of the longest path in G is l, then = l + 1

Example: Machine M2 for which = (n-1)n/2

AB BC AC

CD

0/0

0/0

0/0

1/0

1/00/0

BD AD0/0

1/0

6

Determining Whether a Graph is Loop-Determining Whether a Graph is Loop-freefree

Connection matrix of directed graph G with p vertices: a pxp matrix, whose (i,j)th entry is 1 if there is an arc emanating from vertex i and terminating at vertex j, and is 0 otherwise

• If G is loop-free: then it has one or more terminal vertices• The subgraph resulting from the removal of a terminal vertex and all arcs

leading to it is also loop-free

Example: connection matrix for machine M2 with = 6

7

Memory Span wrt Input Sequences Memory Span wrt Input Sequences (Definite Machines)(Definite Machines)

A sequential machine M is called a definite machine of order if is the least integer s.t. the present state of M can be determined uniquely from the knowledge of the last input symbols to M

• A definite machine is thus said to have finite input memory• Definite machine of order : -definite machine• A -definite machine is also finite memory of order equal to or smaller

than • The knowledge of any past input values is always sufficient to

completely specify the present state of a -definite machine• Canonical realization:

x1

z

D

Combinational logic

DDxu

xx2

8

Properties of Definite MachinesProperties of Definite Machines

A machine is -definite if and only if every sequence of length is a synchronizing sequence

• Length of the longest path to a singleton uncertainty in the synchronizing tree: order of definiteness

Example: Machine M3: definite of order 3

0

0

(ABCD)

Level

0

1

(BD)(AC)

(A) (B)(C)(BD)

(C) (B)

0

1

1 10

3

2

1

9

Machine ContractionMachine Contraction

Example: Machine M4 and its contracted table M4

If M is -definite: M is ( -1)-definite– Conversely, if M is k-definite: then M is (k + 1)-definite– If M is not definite: neither is M

10

Test for DefinitenessTest for Definiteness

First test for definiteness:1. Determine the subsets of states whose Ik-successors are identical

2. Select one representative in each subset

3. Obtain the contracted table M by replacing each subset with its representative, and modifying the table accordingly

4. Regard M as a new table and repeat the previous steps until no new contractions are possible• M is definite if and only if the final contracted table consists of just a

single state

Example: Machine M4 and its contractions

11

TheoremTheorem

Theorem: If machine M is -definite, then <= n-1. Moreover the order of definiteness is equal to the number of contractions needed to

obtain a one-state machine• Since for machine M4, four contractions are necessary to obtain a one-

state machine: its = 4

12

Testing Table/Graph for DefinitenessTesting Table/Graph for Definiteness

Example: Testing table and testing graph for machine M3

Theorem: A machine is -definite if and only if its corresponding testing graph G is loop-free. If the length of the longest path in G is l, then = l+1

• Machine M3: definite of order 3

AB

BC

AC

CD

0

0

1

10BD

AD

0

1

13

Memory Span wrt Output SequencesMemory Span wrt Output Sequences

An FSM M is said to have an output memory of order if is the least integer s.t. the knowledge of the last output symbols suffices to determine the state of M at some time during the last transitions

Testing table for output memory:

Example: Machine M5

Output-successor

table

14

Example (Contd.)Example (Contd.)

Example (contd.): Testing graph for M5

Theorem: An FSM has a finite output memory if and only if its corresponding testing graph G is loop-free. Furthermore, if G is loop-free and the longest path in G is of length l, then M has an output memory of order = I + 1

• For M5: = 4

AB AD BD

CD

1

0

10

BC AC0

1 1

15

Determining the State of the MachineDetermining the State of the Machine

Example: For machine M5

• Suppose the output sequence is 1110• Initially, the machine could have been in: A, B, or D• Thus, initial uncertainty: (ABD)

16

Information Lossless MachinesInformation Lossless Machines

An FSM M is information lossless if the knowledge of the initial state, output sequence, and final state is sufficient to determine uniquely the input sequence

Conditions for lossiness:

Example: Machine M6 is lossy

Sc Sf

x1/z1

x1/z1

x2/z2

x2/z2 Sj

Si

xn/zn

xn/zn

A

A

B

B

0/0 1/0

0/01/0

17

Information Losslessness of Finite OrderInformation Losslessness of Finite Order

An FSM is said to be information lossless of finite order if the knowledge of the initial state and the first output symbols is sufficient to

determine the first input symbol uniquely

Example: Machine M7: lossless machine of first order

18

Test for Information LosslessnessTest for Information Losslessness

Two states, Si and Sj, are said to be output-compatible if there exists some state Sp s.t. both Si and Sj are its Ok-successors, or if there

exists a pair of states Sr,St, s.t. Si,Sj are their Ok-successors

• In the latter case: (SiSj) is implied by (SrSt)

Example: Machine M8: testing table for information losslessness

19

Testing GraphTesting Graph

A machine is lossless if and only if its testing table does not contain any compatible pair consisting of repeated entries

Testing graph for M8: lossless

AD BC AB

0

0

11

1

0

DE

AE

1

AC

0

20

TheoremTheorem

Theorem: A machine M is lossless of order = l + 2 if and only if its testing graph is loop-free and the length of the longest path in the

graph is l• Case of = 1: detected by the absence of compatible pairs• Case of = 2: detected by the absence of arcs in the graph

Example: Since the testing graph of M8 is not loop-free: it is not lossless of finite order

AD BC AB

0

0

11

1

0

DE

AE

1

AC

0

21

ExampleExample

Example: Machine M9: lossless of order 3

AB

AD BD

CD

1

000

BCAC

0

11

1

22

Retrieval of the Input SequenceRetrieval of the Input Sequence

Example: Consider machine M8

• Assume the machine was

initially in A• Suppose output sequence:

110001100101• The machine terminates in B

23

Inverse MachinesInverse Machines

An inverse Mi is a machine which, when excited by the output sequence of a machine M, produces (as its output) the input sequence to M,

after at most a finite delay• M must be lossless of finite order

Example: Machine M7 and its inverse M7i

24

A Deciphering SystemA Deciphering System

Schematic diagram:

z(t-u+1)D

Combinational logic

DD

x(t)

(u-1)-delay register

x(t-u+1)

Logic

Delays

Decoded message

S(t-u+1)

Copy of original machine M

Logic

Delays

Coding machine M

Input

S(t)

Outputz(t)

Inverse machine

25

Minimal Inverse MachineMinimal Inverse Machine

Example: Machine M10: lossless of third order

• Knowledge of initial state and three successive output symbols: yields the first input symbol• Possible triples: (A,0,0), (A,1,1), (B,0,1), (B,1,0), (C,0,0), (C,0,1), (D,1,0), (D,1,1)

• For every state of M10: the next inverse state is a triple whose members are obtained as follows:

– First member: the state to which M10 goes when it is initially in the state that is the first member of the present inverse state, and when it is supplied with the first input symbol

– Second member: third member of the corresponding present inverse state

– Third member: present output of M10

26

Example (Contd.)Example (Contd.)

Example (contd.): Suppose M10 is initially in state A, and produces either 00 or 11 in response to some input sequence

• Then, two time units later: M10i must be in the state that corresponds to A

and the appropriate output sequence, i.e., (A,0,0) or (A,1,1)

• Since S4 = (B,1,0) is the only state from which M10i that can reach (A,0,0)

and (A,1,1), when supplied with input sequences 00 or 11: if the initial state of M10 is A, the initial state of M10

i must be (B,1,0)

State of M10

0

A C B CD A CD BD

01 110001Input to M10

Output of M10

State of M10i

Output of M10i

0 11 010100

0 110001

S4 S5 S1 S3S5 S1 S2S6 S1S2

27

Synchronizable and Uniquely Synchronizable and Uniquely Decipherable CodesDecipherable Codes

Source alphabet: {A,B,C,…}

Code alphabet: L = {0,1,2,…}

Binary code: L = {0,1}

Code word: concatenation of a finite number of code symbols

Code: finite number of distinct code words of finite length, each representing a source symbol

Coded message: concatenation of code words, without spacing or any other punctuation

Example: L = {0,1} and set of code words = {00,01,11,10}• Sequence ABDC: coded as 00011011

1

28

Uniquely Decipherable CodeUniquely Decipherable Code

Example: Not in every case can we work backwards to find a unique sequence of source symbols for a given binary sequence

• If = {0,00,01} represents {A,B,C}: then 0001 may be decoded as AAC or BC

Uniquely decipherable code: if and only if every coded message can be decomposed into a sequence of code words in only one way

• is uniquely decipherable, is not• Whenever the number of code symbols is not the same for all code words: the code is not necessarily uniquely

decipherable• On the other hand: = {1,01,001,0001} is uniquely decipherable since symbol 1 actually serves as a separator

between successive code words– Such a separator is called a comma– Such a code is called a comma code

• Block code: in which all code words contain the same number of symbols• Variable-length code: in which the number of symbols representing code words is not the same

2

21

3

29

Instantaneous CodeInstantaneous Code

Instantaneous code: whenever each code word can be deciphered without knowing the succeeding code words

• and are instantaneous codes: while = {1,10,100} is not since sequence 10 cannot be deciphered until we verify that the next symbol is a 1

Let = be a code word: then ,m<=n, is called a prefix of • A necessary and sufficient condition for a code to be instantaneous is that no code word is a prefix of

some other code word

Reason for using variable-length codes: reduction in the average length of messages• Use shorter code words for more frequently used symbols

• Average length of code: Pili– Pi and li: probability of occurrence and length of the code word representing the ith source symbol

31 4

n ...21

m ...21

30

Test of Unique DecipherabilityTest of Unique Decipherability

A code is uniquely decipherable with a finite delay if and only if is the least integer s.t. the knowledge of the first symbols of the coded message suffices to determine its first code word

• Insert a separation symbol S at the beginning and end of each code word in

• In every code word representing source symbol N: insert symbol Ni between its ith symbol and its (i+1)st symbol

Example: If source symbols are {A,B,C} and = {0,01,1010}, then the code words are

• Separation symbol to the right of the code symbol: -successor, denoted Ri, of the left separation symbol

– C1: 1-successor of S because S1C1 occurs in the third code word

k

31

Testing Table for Unique DecipherabilityTesting Table for Unique Decipherability

Two successors, Ri and Rj, are compatible if S Ri and S Rj occur, or if Rp Ri and Rq Rj occur, and Rp and Rq are compatible

• (RiRj) is said to be the compatible implied by (RpRq)

Testing table can be constructed in the following manner:1. The column headings of the table are the symbols of the code alphabet

2. The first row heading is S. The other row headings are the compatible pairs

3. The entries in row RpRq, column , are the compatibles implied by (RpRq) under

Example: Testing table for our examplek

k

kk

kk

32

Testing Table (Contd.)Testing Table (Contd.)

If a repeated pair (SS) occurs in the table: the code is not uniquely decipherable, else it is

• Implies that there exists some compatible pair (RiRj) s.t. S is the

-successor of both Ri and Rj

• However, since both Ri and Rj are reachable from S by a binary sequence that corresponds to two or more different sequences of source symbols: the code is not uniquely decipherable

• Tracing back the compatibles which implied pair (SS): we can find shortest ambiguous messages– Ambiguous message 01010 may be interpreted as AC or BBA

S (SB1)0 01

(SC1) (SS)(SC3)(B1C2)01

33

Testing GraphTesting Graph

Example: Testing table and graph for code = {1,10,001}• Uniquely decipherable since (SS) is not generated

A code is uniquely decipherable with a finite delay if and only if its testing graph is loop-free• Delay = I + 1: l is the length of the longest path in the testing graph

S SB1 SC10 01

C1C2

34

Deciphering a Coded MessageDeciphering a Coded Message

Example: Consider code = {11,011,001,01,00}, which is known to be uniquely decipherable

• We want to decode 0011101100011010011• Scanning from left: insert a lower comma whenever a sequence, which

corresponds to a legitimate code word, is detected

• Next, scan the coded message from the right and insert an upper comma: whenever a sequence which corresponds to the inverse of a legitimate code word is scanned

• If the code is uniquely decipherable: the message can be decoded by retaining only those commas that occur in the upper and lower spaces simultaneously

0 0 , 1 , 1 , 1 , 0 1 , 1 , 0 0 , 0 1 , 1 , 0 1 , 0 0 , 1 , 1, ,, ,, ,, ,,,,,

35

Test for Synchronizability of CodesTest for Synchronizability of Codes

A code is said to be synchronizable of order if is the least integer s.t. the knowledge of any consecutive code symbols is sufficient

to determine a separation of code words within these symbols

Testing a code for synchronizability: analogous to testing an FSM for finite output memory

• A code is synchronizable if and only if it is uniquely decipherable and its testing graph is loop-free. It is synchronizable of order if and only if the longest path in the graph is of length - 1

Example: Testing table and testing graph for = {1,10,001}

1

0

SB1

SC1

C1C2

0

0

SC2B1C2

B1C1