Upload
harsh-arya
View
222
Download
0
Embed Size (px)
Citation preview
7/24/2019 Dtmc Basic
1/185
Discrete Time Markov Chains, Definition
and classification
Bo Friis Nielsen1
1Applied Mathematics and Computer Science
02407 Stochastic Processes 1, September 1 2015
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
2/185
Discrete time Markov chains
Today:
Short recap of probability theory
Markov chain introduction (Markov property) Chapmann-Kolmogorov equations
First step analysis
Next week
Random walks First step analysis revisited
Branching processes
Generating functions
Two weeks from now Classification of states
Classification of chains
Discrete time Markov chains - invariant probability
distributionBo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
3/185
Basic concepts in probability
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
4/185
Basic concepts in probability
Sample space
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
5/185
Basic concepts in probability
Sample space
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
6/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
7/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
B i i b bili
7/24/2019 Dtmc Basic
8/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
B i t i b bilit
7/24/2019 Dtmc Basic
9/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
B i t i b bilit
7/24/2019 Dtmc Basic
10/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Basic concepts in probabilit
7/24/2019 Dtmc Basic
11/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event Ac =
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Basic concepts in probability
7/24/2019 Dtmc Basic
12/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event Ac = \A
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Basic concepts in probability
7/24/2019 Dtmc Basic
13/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event Ac = \A
Union
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Basic concepts in probability
7/24/2019 Dtmc Basic
14/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event Ac = \A
Union A B
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Basic concepts in probability
7/24/2019 Dtmc Basic
15/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event Ac = \A
Union A B outcome in at least one ofAor B
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Basic concepts in probability
7/24/2019 Dtmc Basic
16/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event Ac = \A
Union A B outcome in at least one ofAor B
Intersection
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Basic concepts in probability
7/24/2019 Dtmc Basic
17/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event Ac = \A
Union A B outcome in at least one ofAor B
Intersection A B
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Basic concepts in probability
7/24/2019 Dtmc Basic
18/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event Ac = \A
Union A B outcome in at least one ofAor B
Intersection A B Outcome is inbothA and B
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Basic concepts in probability
7/24/2019 Dtmc Basic
19/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event Ac = \A
Union A B outcome in at least one ofAor B
Intersection A B Outcome is inbothA and B
(Empty) or impossible event
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Basic concepts in probability
7/24/2019 Dtmc Basic
20/185
Basic concepts in probability
Sample space set of all possible outcomesOutcome Event A, B
Complementary event Ac = \A
Union A B outcome in at least one ofAor B
Intersection A B Outcome is inbothA and B
(Empty) or impossible event
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Probability axioms and first results
7/24/2019 Dtmc Basic
21/185
Probability axioms and first results
0 P(A) 1, P() =1
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Probability axioms and first results
7/24/2019 Dtmc Basic
22/185
Probability axioms and first results
0 P(A) 1, P() =1
P(A B) = P(A) + P(B) forA B=
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Probability axioms and first results
7/24/2019 Dtmc Basic
23/185
y
0 P(A) 1, P() =1
P(A B) = P(A) + P(B) forA B=
Leading to
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Probability axioms and first results
7/24/2019 Dtmc Basic
24/185
y
0 P(A) 1, P() =1
P(A B) = P(A) + P(B) forA B=
Leading toP() =0, P(Ac) =1 P(A)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Probability axioms and first results
7/24/2019 Dtmc Basic
25/185
y
0 P(A) 1, P() =1
P(A B) = P(A) + P(B) forA B=
Leading toP() =0, P(Ac) =1 P(A)
P(A B) = P(A) + P(B) P(A B)
(inclusion- exlusion)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Conditional probability and independence
7/24/2019 Dtmc Basic
26/185
p y p
P(A|B) = P(A B)P(B)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Conditional probability and independence
7/24/2019 Dtmc Basic
27/185
p y p
P(A|B) = P(A B)P(B)
P(A B) = P(A|B)P(B)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Conditional probability and independence
7/24/2019 Dtmc Basic
28/185
p y p
P(A|B) = P(A B)P(B)
P(A B) = P(A|B)P(B)
(multiplication rule)
iBi=
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Conditional probability and independence
7/24/2019 Dtmc Basic
29/185
P(A|B) = P(A B)P(B)
P(A B) = P(A|B)P(B)
(multiplication rule)
iBi= Bi Bj= i=j
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Conditional probability and independence
7/24/2019 Dtmc Basic
30/185
P(A|B) =P
(A B)P(B)
P(A B) = P(A|B)P(B)
(multiplication rule)
iBi= Bi Bj= i=j
P(A) =
i
P(A|Bi)P(Bi) (law of total probability)
Independence:
P(A|B) = P(A|Bc) = P(A)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Conditional probability and independence
7/24/2019 Dtmc Basic
31/185
P(A|B) =P
(A B)P(B)
P(A B) = P(A|B)P(B)
(multiplication rule)
iBi= Bi Bj= i=j
P(A) =
i
P(A|Bi)P(Bi) (law of total probability)
Independence:
P(A|B) = P(A|Bc) = P(A) P(A B) = P(A)P(B)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Discrete random variables
7/24/2019 Dtmc Basic
32/185
Mapping from sample space to metric space
(Read: Real space)
Probability mass function
f(x) = P(X=x) = P ({|X() =x})
Distribution function
F(x) = P(Xx) = P ({|X() x}) =tx
f(t)
Expectation
E(X) =
x
xP(X=x), E(g(X)) =
x
g(x)P(X=x) =
x
g(x)f(x)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Joint distribution
7/24/2019 Dtmc Basic
33/185
f(x1, x2) = P(X1=x1, X2=x2), F(x1, x2) = P(X1x1, X2 x2)
fX1
(x1) = P(X1=x1) = x2
P(X1=x1, X2=x2) = x2
f(x1, x2)
FX1 (x1) =
tx1,x2
P(X1=t1, X2=x2) =F(x1,)
Straightforward to extend tonvariables
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
34/185
We can define the joint distribution of (X0, X1)through
P(X0=x0, X1 =x1) = P(X0 =x0)P(X1=x1|X0 =x0) = P(X0=x0)Px0,x1
Suppose now some stationarity in addition that X2 conditioned
onX1 is independent onX0
P(X0 =x0, X1 =x1, X2=x2) =P(X0=x0)P(X1=x1|X0=x0)P(X2=x2|X0 =x0, X1=x1) =
P(X0=x0)P(X1=x1|X0=x0)P(X2=x2|X1=x1) =
px0 Px0,x1 Px1,x2
which generalizes to arbitraryn.
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov property
7/24/2019 Dtmc Basic
35/185
P (Xn=xn|H)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov property
7/24/2019 Dtmc Basic
36/185
P (Xn=xn|H)
= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov property
7/24/2019 Dtmc Basic
37/185
P (Xn=xn|H)
= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)
= P (Xn=xn|Xn1=xn1)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov property
7/24/2019 Dtmc Basic
38/185
P (Xn=xn|H)
= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)
= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and
the time
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov property
7/24/2019 Dtmc Basic
39/185
P (Xn=xn|H)
= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)
= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and
the time
In most applications the chain is assumed to be time
homogeneous, i.e. it does not depend on time
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov property(X |H)
7/24/2019 Dtmc Basic
40/185
P (Xn=xn|H)
= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)
= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and
the time
In most applications the chain is assumed to be time
homogeneous, i.e. it does not depend on time The only parameters needed are
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov propertyP (X |H)
7/24/2019 Dtmc Basic
41/185
P (Xn=xn|H)
= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)
= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and
the time
In most applications the chain is assumed to be time
homogeneous, i.e. it does not depend on time The only parameters needed are P (Xn=j|Xn1 =i)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov propertyP (X |H)
7/24/2019 Dtmc Basic
42/185
P (Xn=xn|H)
= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)
= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and
the time
In most applications the chain is assumed to be time
homogeneous, i.e. it does not depend on time The only parameters needed are P (Xn=j|Xn1 =i) =pij
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov propertyP (X |H)
7/24/2019 Dtmc Basic
43/185
P (Xn=xn|H)
= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)
= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and
the time
In most applications the chain is assumed to be time
homogeneous, i.e. it does not depend on time The only parameters needed are P (Xn=j|Xn1 =i) =pij We collect these parameters in a matrix
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov propertyP (X x |H)
7/24/2019 Dtmc Basic
44/185
P (Xn=xn|H)
= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)
= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and
the time
In most applications the chain is assumed to be time
homogeneous, i.e. it does not depend on time The only parameters needed are P (Xn=j|Xn1 =i) =pij We collect these parameters in a matrix P={pij}
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Markov propertyP (X x |H)
7/24/2019 Dtmc Basic
45/185
P (Xn=xn|H)
= P (Xn=xn|X0=x0, X1=x1, X2=x2, . . . Xn1=xn1)
= P (Xn=xn|Xn1=xn1) Generally the next state depends on the current state and
the time
In most applications the chain is assumed to be time
homogeneous, i.e. it does not depend on time The only parameters needed are P (Xn=j|Xn1 =i) =pij We collect these parameters in a matrix P={pij}
The joint probability of the first noccurrences is
P(X0 =x0, X1 =x1, X2=x2. . . , Xn=xn) =px0 Px0,x1 Px1,x2. . . Pxn1,xn
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
A profuse number of applications
7/24/2019 Dtmc Basic
46/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
A profuse number of applications
7/24/2019 Dtmc Basic
47/185
Storage/inventory models
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
A profuse number of applications
7/24/2019 Dtmc Basic
48/185
Storage/inventory models Telecommunications systems
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
A profuse number of applications
7/24/2019 Dtmc Basic
49/185
Storage/inventory models Telecommunications systems
Biological models
Xnthe value attained at time n
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
A profuse number of applications
7/24/2019 Dtmc Basic
50/185
Storage/inventory models Telecommunications systems
Biological models
Xnthe value attained at time n Xncould be
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
A profuse number of applications
7/24/2019 Dtmc Basic
51/185
Storage/inventory models Telecommunications systems
Biological models
Xnthe value attained at time n Xncould be
The number of cars in stock
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
A profuse number of applications
7/24/2019 Dtmc Basic
52/185
Storage/inventory models Telecommunications systems
Biological models
Xnthe value attained at time n Xncould be
The number of cars in stock The number of days since last rainfall
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
A profuse number of applications
7/24/2019 Dtmc Basic
53/185
Storage/inventory models Telecommunications systems
Biological models
Xnthe value attained at time n Xncould be
The number of cars in stock The number of days since last rainfall The number of passengers booked for a flight
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
A profuse number of applications
7/24/2019 Dtmc Basic
54/185
Storage/inventory models Telecommunications systems
Biological models
Xnthe value attained at time n Xncould be
The number of cars in stock The number of days since last rainfall The number of passengers booked for a flight
See textbook for further examples
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
55/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
56/185
P=
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
57/185
P=
1 p
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
58/185
P=
1 p p
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
59/185
P=
1 p p 0 . . . 0 0 0
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
60/185
P=
1 p p 0 . . . 0 0 0q
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
61/185
P=
1 p p 0 . . . 0 0 0q 1 p q
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
62/185
P=
1 p p 0 . . . 0 0 0q 1 p q p
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
63/185
P=
1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
64/185
P=
1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0
0 q 1 p q . . . 0 0 0...
... ...
... ...
... ...
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
65/185
P=
1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0
0 q 1 p q . . . 0 0 0... ...
... ...
... ...
...
0 0 0 . . .
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
66/185
P=
1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0
0 q 1 p q . . . 0 0 0... ...
... ...
... ...
...
0 0 0 . . . q 1 p q p
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
67/185
P=
1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0
0 q 1 p q . . . 0 0 0... ...
... ...
... ...
...
0 0 0 . . . q 1 p q p0 0 0 . . . 0
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
68/185
P=
1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0
0 q 1 p q . . . 0 0 0... ...
... ...
... ...
...
0 0 0 . . . q 1 p q p0 0 0 . . . 0 q
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 1: Random walk with tworeflecting barriers0andN
7/24/2019 Dtmc Basic
69/185
P=
1 p p 0 . . . 0 0 0q 1 p q p . . . 0 0 0
0 q 1 p q . . . 0 0 0... ...
... ...
... ...
...
0 0 0 . . . q 1 p q p0 0 0 . . . 0 q 1 q
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 2: Random walk with onereflecting barrier at0
7/24/2019 Dtmc Basic
70/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 2: Random walk with onereflecting barrier at0
7/24/2019 Dtmc Basic
71/185
P=
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 2: Random walk with onereflecting barrier at0
7/24/2019 Dtmc Basic
72/185
P=
1 p p 0 0 0 . . .
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 2: Random walk with onereflecting barrier at0
7/24/2019 Dtmc Basic
73/185
P=
1 p p 0 0 0 . . .q 1 p q p 0 0 . . .
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 2: Random walk with onereflecting barrier at0
7/24/2019 Dtmc Basic
74/185
P=
1 p p 0 0 0 . . .q 1 p q p 0 0 . . .
0 q 1 p q p 0 . . .0 0 q 1 p q p . . .
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 2: Random walk with onereflecting barrier at0
7/24/2019 Dtmc Basic
75/185
P=
1 p p 0 0 0 . . .q 1 p q p 0 0 . . .
0 q 1 p q p 0 . . .0 0 q 1 p q p . . ....
... ...
... ... . . .
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 3: Random walk with twoabsorbing barriers
7/24/2019 Dtmc Basic
76/185
P=
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 3: Random walk with twoabsorbing barriers
7/24/2019 Dtmc Basic
77/185
P=
1
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 3: Random walk with twoabsorbing barriers
7/24/2019 Dtmc Basic
78/185
P=
1 0 0 0 0 . . . 0 0 0
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 3: Random walk with twoabsorbing barriers
7/24/2019 Dtmc Basic
79/185
P=
1 0 0 0 0 . . . 0 0 0
q 1 p q p 0 0 . . . 0 0 0
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 3: Random walk with twoabsorbing barriers
7/24/2019 Dtmc Basic
80/185
P=
1 0 0 0 0 . . . 0 0 0
q 1 p q p 0 0 . . . 0 0 00 q 1 p q p 0 . . . 0 0 00 0 q 1 p q p . . . 0 0 0...
... ...
... ...
... ...
... ...
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 3: Random walk with twoabsorbing barriers
7/24/2019 Dtmc Basic
81/185
P=
1 0 0 0 0 . . . 0 0 0
q 1 p q p 0 0 . . . 0 0 00 q 1 p q p 0 . . . 0 0 00 0 q 1 p q p . . . 0 0 0...
... ...
... ...
... ...
... ...
0 0 0 0 0 . . . q 1 p q p
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Example 3: Random walk with twoabsorbing barriers
7/24/2019 Dtmc Basic
82/185
P=
1 0 0 0 0 . . . 0 0 0
q 1 p q p 0 0 . . . 0 0 00 q 1 p q p 0 . . . 0 0 00 0 q 1 p q p . . . 0 0 0...
... ...
... ...
... ...
... ...
0 0 0 0 0 . . . q 1 p q p0 0 0 0 0 . . . 0 0 1
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
83/185
The matrix can be finite
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
84/185
The matrix can be finite (if the Markov chain is finite)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
85/185
The matrix can be finite (if the Markov chain is finite)
P=
p1,1 p1,2 p1,3 . . . p1,np2,1 p2,2 p2,3 . . . p2,np3,1 p3,2 p3,3 . . . p3,n
... ...
... ...
...
pn,1 pn,2 pn,3 . . . pn,n
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
7/24/2019 Dtmc Basic
86/185
The matrix can be finite (if the Markov chain is finite)
P=
p1,1 p1,2 p1,3 . . . p1,np2,1 p2,2 p2,3 . . . p2,np3,1 p3,2 p3,3 . . . p3,n
... ...
... ...
...
pn,1 pn,2 pn,3 . . . pn,n
Two reflecting/absorbing barriers
Bo Friis Nielsen Discrete Time Markov Chains Definition and classification
or infinite
7/24/2019 Dtmc Basic
87/185
or infinite
Bo Friis Nielsen Discrete Time Markov Chains Definition and classification
or infinite (if the Markov chain is infinite)
7/24/2019 Dtmc Basic
88/185
or infinite (if the Markov chain is infinite)
Bo Friis Nielsen Discrete Time Markov Chains Definition and classification
or infinite (if the Markov chain is infinite)
7/24/2019 Dtmc Basic
89/185
or infinite (if the Markov chain is infinite)
P=
p1,1 p1,2 p1,3 . . . p1,n . . .p2,1 p2,2 p2,3 . . . p2,n . . .
p3,1 p3,2 p3,3 . . . p3,n . . ....
... ...
... ...
...
pn,1 pn,2 pn,3 . . . pn,n . . ....
... ...
... ...
...
Bo Friis Nielsen Discrete Time Markov Chains Definition and classification
or infinite (if the Markov chain is infinite)
7/24/2019 Dtmc Basic
90/185
or infinite (if the Markov chain is infinite)
P=
p1,1 p1,2 p1,3 . . . p1,n . . .p2,1 p2,2 p2,3 . . . p2,n . . .
p3,1 p3,2 p3,3 . . . p3,n . . ....
... ...
... ...
...
pn,1 pn,2 pn,3 . . . pn,n . . ....
... ...
... ...
...
At most one barrier
Bo Friis Nielsen Discrete Time Markov Chains Definition and classification
The matrixPcan be interpreted as
7/24/2019 Dtmc Basic
91/185
Bo Friis Nielsen Discrete Time Markov Chains Definition and classification
The matrixPcan be interpreted as
7/24/2019 Dtmc Basic
92/185
the engine that drives the process
Bo Friis Nielsen Discrete Time Markov Chains Definition and classification
The matrixPcan be interpreted as
7/24/2019 Dtmc Basic
93/185
the engine that drives the process
the statistical descriptor of the quantitative behaviour
Bo Friis Nielsen Discrete Time Markov Chains Definition and classification
The matrixPcan be interpreted as
7/24/2019 Dtmc Basic
94/185
the engine that drives the process
the statistical descriptor of the quantitative behaviour
a collection of discrete probability distributions
Bo Friis Nielsen Discrete Time Markov Chains Definition and classification
The matrixPcan be interpreted as
7/24/2019 Dtmc Basic
95/185
the engine that drives the process
the statistical descriptor of the quantitative behaviour
a collection of discrete probability distributions For eachiwe have a conditional distribution
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The matrixPcan be interpreted as
7/24/2019 Dtmc Basic
96/185
the engine that drives the process
the statistical descriptor of the quantitative behaviour
a collection of discrete probability distributions For eachiwe have a conditional distribution
What is the probability of the next state being jknowing thatthe current state isi
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The matrixPcan be interpreted as
7/24/2019 Dtmc Basic
97/185
the engine that drives the process
the statistical descriptor of the quantitative behaviour
a collection of discrete probability distributions For eachiwe have a conditional distribution
What is the probability of the next state being jknowing thatthe current state isi pij= P (Xn=j|Xn1 =i)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The matrixPcan be interpreted as
7/24/2019 Dtmc Basic
98/185
the engine that drives the process
the statistical descriptor of the quantitative behaviour
a collection of discrete probability distributions For eachiwe have a conditional distribution
What is the probability of the next state being jknowing thatthe current state isi pij= P (Xn=j|Xn1 =i)
jpij=1
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The matrixPcan be interpreted as
7/24/2019 Dtmc Basic
99/185
the engine that drives the process
the statistical descriptor of the quantitative behaviour
a collection of discrete probability distributions For eachiwe have a conditional distribution
What is the probability of the next state being jknowing thatthe current state isi pij= P (Xn=j|Xn1 =i)
jpij=1
We say thatPis a stochastic matrix
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
More definitions and the first properties
7/24/2019 Dtmc Basic
100/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
More definitions and the first properties
7/24/2019 Dtmc Basic
101/185
We have defined rules for the behaviour from one value
and onwards
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
More definitions and the first properties
7/24/2019 Dtmc Basic
102/185
We have defined rules for the behaviour from one value
and onwards
Boundary conditions
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
More definitions and the first properties
7/24/2019 Dtmc Basic
103/185
We have defined rules for the behaviour from one value
and onwards
Boundary conditions specify e.g. behaviour ofX0
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
More definitions and the first properties
7/24/2019 Dtmc Basic
104/185
We have defined rules for the behaviour from one value
and onwards
Boundary conditions specify e.g. behaviour ofX0 X0 could be certain
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
More definitions and the first properties
7/24/2019 Dtmc Basic
105/185
We have defined rules for the behaviour from one value
and onwards
Boundary conditions specify e.g. behaviour ofX0 X0 could be certainX0 =a
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
More definitions and the first properties
7/24/2019 Dtmc Basic
106/185
We have defined rules for the behaviour from one value
and onwards
Boundary conditions specify e.g. behaviour ofX0 X0 could be certainX0 =a or random
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
More definitions and the first properties
7/24/2019 Dtmc Basic
107/185
We have defined rules for the behaviour from one value
and onwards
Boundary conditions specify e.g. behaviour ofX0 X0 could be certainX0 =a or random P (X0 =i) =pi
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
More definitions and the first properties
7/24/2019 Dtmc Basic
108/185
We have defined rules for the behaviour from one value
and onwards
Boundary conditions specify e.g. behaviour ofX0 X0 could be certainX0 =a or random P (X0 =i) =pi Once again we collect the possibly infinite many
parameters in a vectorp
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
nstep transition probabilities
7/24/2019 Dtmc Basic
109/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
nstep transition probabilities
7/24/2019 Dtmc Basic
110/185
P (Xn=j|X0=i)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
nstep transition probabilities
(n)
7/24/2019 Dtmc Basic
111/185
P (Xn=j|X0=i) =P(n)ij
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
nstep transition probabilities
(n)
7/24/2019 Dtmc Basic
112/185
P (Xn=j|X0=i) =P(n)ij
the probability of being in jat thenth transition having
started ini
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
nstep transition probabilities
(n)
7/24/2019 Dtmc Basic
113/185
P (Xn=j|X0=i) =P(n)ij
the probability of being in jat thenth transition having
started ini
Once again collected in a matrix
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
nstep transition probabilities
(n)
7/24/2019 Dtmc Basic
114/185
P (Xn=j|X0=i) =P(n)ij
the probability of being in jat thenth transition having
started ini
Once again collected in a matrix P(n) ={P(n)ij }
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
nstep transition probabilities
(n)
7/24/2019 Dtmc Basic
115/185
P (Xn=j|X0=i) =P(n)ij
the probability of being in jat thenth transition having
started ini
Once again collected in a matrix P(n) ={P(n)ij }
The rows ofP(n) can be interpreted like the rows ofP
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
nstep transition probabilities
(n)
7/24/2019 Dtmc Basic
116/185
P (Xn=j|X0=i) =P(n)ij
the probability of being in jat thenth transition having
started ini
Once again collected in a matrix P(n) ={P(n)ij }
The rows ofP(n) can be interpreted like the rows ofP
We can define a new Markov chain on a larger time scale
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
nstep transition probabilities
(n)
7/24/2019 Dtmc Basic
117/185
P (Xn=j|X0=i) =P( )ij
the probability of being in jat thenth transition having
started ini
Once again collected in a matrix P(n) ={P(n)ij }
The rows ofP(n) can be interpreted like the rows ofP
We can define a new Markov chain on a larger time scale
(Pn)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
7/24/2019 Dtmc Basic
118/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
7/24/2019 Dtmc Basic
119/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
7/24/2019 Dtmc Basic
120/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
7/24/2019 Dtmc Basic
121/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
+pq
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
7/24/2019 Dtmc Basic
122/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
+pq (1 p)p
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
7/24/2019 Dtmc Basic
123/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
+pq (1 p)p p2
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
1 0 0
7/24/2019 Dtmc Basic
124/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
+pq (1 p)p p2
0
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
1 0 0
7/24/2019 Dtmc Basic
125/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
+pq (1 p)p p2
0q(1 p)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
1 0 0
7/24/2019 Dtmc Basic
126/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
+pq (1 p)p p2
0q(1 p) 2qp
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
1 p p 0 0
7/24/2019 Dtmc Basic
127/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
+pq (1 p)p p2
0q(1 p) 2qp 0
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
1 p p 0 0
7/24/2019 Dtmc Basic
128/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
+pq (1 p)p p2
0q(1 p) 2qp 0 p2
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
1 p p 0 0
7/24/2019 Dtmc Basic
129/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
+pq (1 p)p p2
0q(1 p) 2qp 0 p2
q2 0 2qp p(1 q)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example
1 p p 0 0
7/24/2019 Dtmc Basic
130/185
P=
1 p p 0 0q 0 p 0
0 q 0 p
0 0 q 1 q
P(2) =
(1 p)2
+pq (1 p)p p2
0q(1 p) 2qp 0 p2
q2 0 2qp p(1 q)0 q2 (1 q)q (1 q)2 +qp
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Chapmann Kolmogorov equations
7/24/2019 Dtmc Basic
131/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Chapmann Kolmogorov equations
There is a generalisation of the example above
7/24/2019 Dtmc Basic
132/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Chapmann Kolmogorov equations
There is a generalisation of the example above
Suppose we start in iat time 0 and wants to get to jat time
n + m
7/24/2019 Dtmc Basic
133/185
n+m
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Chapmann Kolmogorov equations
There is a generalisation of the example above
Suppose we start in iat time 0 and wants to get to jat time
n + m
7/24/2019 Dtmc Basic
134/185
n+m
At some intermediate timenwe must be in some state k
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Chapmann Kolmogorov equations
There is a generalisation of the example above
Suppose we start in iat time 0 and wants to get to jat time
n + m
7/24/2019 Dtmc Basic
135/185
n+m
At some intermediate timenwe must be in some state k
We apply the law of total probability
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Chapmann Kolmogorov equations
There is a generalisation of the example above
Suppose we start in iat time 0 and wants to get to jat time
n + m
7/24/2019 Dtmc Basic
136/185
n+m
At some intermediate timenwe must be in some state k
We apply the law of total probability
P (B) =
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Chapmann Kolmogorov equations
There is a generalisation of the example above
Suppose we start in iat time 0 and wants to get to jat time
n + m
7/24/2019 Dtmc Basic
137/185
n+m At some intermediate timenwe must be in some state k
We apply the law of total probability
P (B) = kP (B|Ak)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Chapmann Kolmogorov equations
There is a generalisation of the example above
Suppose we start in iat time 0 and wants to get to jat time
n + m
7/24/2019 Dtmc Basic
138/185
n+m At some intermediate timenwe must be in some state k
We apply the law of total probability
P (B) = kP (B|Ak)P (Ak)P (Xn+m=j|X0 =i)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Chapmann Kolmogorov equations
There is a generalisation of the example above
Suppose we start in iat time 0 and wants to get to jat time
n+m
7/24/2019 Dtmc Basic
139/185
+ At some intermediate timenwe must be in some state k
We apply the law of total probability
P (B) = kP (B|Ak)P (Ak)P (Xn+m=j|X0 =i)
= k
P (Xn+m=j|X0 =i, Xn=k)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Chapmann Kolmogorov equations
There is a generalisation of the example above
Suppose we start in iat time 0 and wants to get to jat time
n+m
7/24/2019 Dtmc Basic
140/185
+ At some intermediate timenwe must be in some state k
We apply the law of total probability
P (B) = kP (B|Ak)P (Ak)P (Xn+m=j|X0 =i)
= k
P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
k
P (Xn+m=j|X0 =i, Xn=k)
7/24/2019 Dtmc Basic
141/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
k
P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)
7/24/2019 Dtmc Basic
142/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
k
P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)
by the Markov property we get
7/24/2019 Dtmc Basic
143/185
k
P (Xn+m=j|Xn=k)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
k
P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)
by the Markov property we get(X j|X k ) (X k |X i)
7/24/2019 Dtmc Basic
144/185
k
P (Xn+m=j|Xn=k)P (Xn=k|X0=i)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
k
P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)
by the Markov property we getP (X j|X k )P (X k |X i)
7/24/2019 Dtmc Basic
145/185
k
P (Xn+m=j|Xn=k)P (Xn=k|X0=i)
= k
P(m)
kj
P(n)
ik
=
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
k
P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)
by the Markov property we getP (X j|X k )P (X k |X i)
7/24/2019 Dtmc Basic
146/185
k
P (Xn+m=j|Xn=k)P (Xn=k|X0=i)
= k
P(m)
kj
P(n)
ik
= k
P(n)
ik
P(m)
kj
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
k
P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)
by the Markov property we getP (X j|X k )P (X k |X i)
7/24/2019 Dtmc Basic
147/185
k
P (Xn+m=j|Xn=k)P (Xn=k|X0=i)
= k
P(m)
kj
P(n)
ik
= k
P(n)
ik
P(m)
kj
which in matrix formulation is
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
k
P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)
by the Markov property we getP (X j|X k )P (X k |X i)
7/24/2019 Dtmc Basic
148/185
k
P (Xn+m=j|Xn=k)P (Xn=k|X0=i)
= k
P(m)
kj
P(n)
ik
= k
P(n)
ik
P(m)
kj
which in matrix formulation is
P(n+m) =P(n)P(m)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
k
P (Xn+m=j|X0 =i, Xn=k)P (Xn=k|X0 =i)
by the Markov property we getP (X j|X k )P (X k |X i)
7/24/2019 Dtmc Basic
149/185
k
P (Xn+m=j|Xn=k)P (Xn=k|X0=i)
= k
P(m)
kj
P(n)
ik
= k
P(n)
ik
P(m)
kj
which in matrix formulation is
P(n+m) =P(n)P(m) =Pn+m
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The probability ofXn
7/24/2019 Dtmc Basic
150/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The probability ofXn
The behaviour of the process itself - Xn
7/24/2019 Dtmc Basic
151/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The probability ofXn
The behaviour of the process itself - Xn
The behaviour conditional onX0=iis known (P(n)ij )
7/24/2019 Dtmc Basic
152/185
j
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The probability ofXn
The behaviour of the process itself - Xn
The behaviour conditional onX0=iis known (P(n)ij )
7/24/2019 Dtmc Basic
153/185
j
Define P (Xn=j)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The probability ofXn
The behaviour of the process itself - Xn
The behaviour conditional onX0=iis known (P(n)ij )
7/24/2019 Dtmc Basic
154/185
j
Define P (Xn=j) =p(n)
j
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The probability ofXn
The behaviour of the process itself - Xn
The behaviour conditional onX0=iis known (P(n)ij )
7/24/2019 Dtmc Basic
155/185
Define P (Xn=j) =p(n)
j
withp(n) ={p(n)
j }
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The probability ofXn
The behaviour of the process itself - Xn
The behaviour conditional onX0=iis known (P(n)ij )
7/24/2019 Dtmc Basic
156/185
Define P (Xn=j) =p(n)
j
withp(n) ={p(n)
j }we find
p(n)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The probability ofXn
The behaviour of the process itself - Xn
The behaviour conditional onX0=iis known (P(n)ij )
7/24/2019 Dtmc Basic
157/185
Define P (Xn=j) =p(n)
j
withp(n) ={p(n)
j }we find
p(n) =p
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
The probability ofXn
The behaviour of the process itself - Xn
The behaviour conditional onX0=iis known (P(n)ij )
7/24/2019 Dtmc Basic
158/185
Define P (Xn=j) =p(n)
j
withp(n) ={p(n)
j }we find
p(n) =pP(n) =pPn
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example - revisited
P=
1 p p 0 0q 0 p 0
0 q 0 p0 0 q 1 q
7/24/2019 Dtmc Basic
159/185
q q
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example - revisited
P=
1 p p 0 0q 0 p 0
0 q 0 p0 0 q 1 q
7/24/2019 Dtmc Basic
160/185
withp=
13 , 0, 0,
23
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example - revisited
P=
1 p p 0 0q 0 p 0
0 q 0 p0 0 q 1 q
7/24/2019 Dtmc Basic
161/185
withp=
13 , 0, 0,
23
we get
p(1)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example - revisited
P=
1 p p 0 0q 0 p 0
0 q 0 p0 0 q 1 q
7/24/2019 Dtmc Basic
162/185
withp=
13 , 0, 0,
23
we get
p(1) =
1
3, 0, 0,
2
3
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example - revisited
P=
1 p p 0 0q 0 p 0
0 q 0 p0 0 q 1 q
7/24/2019 Dtmc Basic
163/185
withp=
13 , 0, 0,
23
we get
p(1) =
1
3, 0, 0,
2
3
1 p p 0 0q 0 p 00 q 0 p
0 0 q 1 q
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example - revisited
P=
1 p p 0 0q 0 p 0
0 q 0 p0 0 q 1 q
7/24/2019 Dtmc Basic
164/185
withp=
13 , 0, 0,
23
we get
p(1) =
1
3, 0, 0,
2
3
1 p p 0 0q 0 p 00 q 0 p
0 0 q 1 q
=
1 p
3 ,
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example - revisited
P=
1 p p 0 0q 0 p 0
0 q 0 p0 0 q 1 q
7/24/2019 Dtmc Basic
165/185
withp=
13 , 0, 0,
23
we get
p(1) =
1
3, 0, 0,
2
3
1 p p 0 0q 0 p 00 q 0 p
0 0 q 1 q
=
1 p
3 ,
p
3,
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example - revisited
P=
1 p p 0 0q 0 p 0
0 q 0 p0 0 q 1 q
7/24/2019 Dtmc Basic
166/185
withp=
13 , 0, 0,
23
we get
p(1) =
1
3, 0, 0,
2
3
1 p p 0 0q 0 p 00 q 0 p
0 0 q 1 q
=
1 p
3 ,
p
3,2q
3 ,
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Small example - revisited
P=
1 p p 0 0q 0 p 0
0 q 0 p0 0 q 1 q
7/24/2019 Dtmc Basic
167/185
withp=
13 , 0, 0,
23
we get
p(1) =
1
3, 0, 0,
2
3
1 p p 0 0q 0 p 00 q 0 p
0 0 q 1 q
=
1 p
3 ,
p
3,2q
3 ,
2(1 q)
3
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
p= 1
3
, 0, 0,2
3 , (1 p)2 +pq (1 p)p p2 0
2
7/24/2019 Dtmc Basic
168/185
P2 =
q(1 p) 2qp 0 p2
q2 0 2qp p(1 q)0 q2 (1 q)q (1 q)2 +qp
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
p(2)
7/24/2019 Dtmc Basic
169/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
p(2) =
1
3, 0, 0,
2
3
7/24/2019 Dtmc Basic
170/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
p(2) =
1
3, 0, 0,
2
3
(1 p)2 +pq (1 p)p p2 0q(1 p) 2qp 0 p2
q2 0 2qp p(1 q)
7/24/2019 Dtmc Basic
171/185
q qp p( q)0 q2 (1 q)q (1 q)2 +qp
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
p(2) =
1
3, 0, 0,
2
3
(1 p)2 +pq (1 p)p p2 0q(1 p) 2qp 0 p2
q2 0 2qp p(1 q)
7/24/2019 Dtmc Basic
172/185
( )0 q2 (1 q)q (1 q)2 +qp
=
(1 p)2 +pq
3 ,
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
p(2) =
1
3, 0, 0,
2
3
(1 p)2 +pq (1 p)p p2 0q(1 p) 2qp 0 p2
q2 0 2qp p(1 q)
7/24/2019 Dtmc Basic
173/185
0 q2 (1 q)q (1 q)2 +qp
=
(1 p)2 +pq
3 ,
(1 p)p
3 ,
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
p(2) =
1
3, 0, 0,
2
3
(1 p)2 +pq (1 p)p p2 0q(1 p) 2qp 0 p2
q2 0 2qp p(1 q)
7/24/2019 Dtmc Basic
174/185
0 q2 (1 q)q (1 q)2 +qp
=
(1 p)2 +pq
3 ,
(1 p)p
3 ,
4qp
3 ,
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
p(2) =
1
3, 0, 0,
2
3
(1 p)2 +pq (1 p)p p2 0q(1 p) 2qp 0 p2
q2 0 2qp p(1 q)2 2
7/24/2019 Dtmc Basic
175/185
0 q2 (1 q)q (1 q)2 +qp
=
(1 p)2 +pq
3 ,
(1 p)p
3 ,
4qp
3 ,
2p(1 q)
3
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
First step analysis - setup
Consider the transition probability matrix
P=
1 0 0
0 0 1
7/24/2019 Dtmc Basic
176/185
0 0 1 Define
T =min {n0:Xn=0 orXn=2}
and
u= P(XT =0|X0 =1) v= E(T|X0=1)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
First step analysis - absorption probability
u = P(XT =0|X0=1)
=2
k=0
P(X1=k|X0 =1)P(XT =0|X0 =1, X1=k)
7/24/2019 Dtmc Basic
177/185
=2
k=0
P(X1=k|X0 =1)P(XT =0|X1 =k)
= P1,0 1 +P1,1 u+P1,2 0.
And we find
u= P1,0
1 P1,1=
+
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
First step analysis - time to absorption
v = E(T|X0 =1)
=2
k=0
P(X1=k|X0=1)E(T|X0 =1, X1 =k)
7/24/2019 Dtmc Basic
178/185
= 1 +2
k=0
P(X1 =k|X0=1)E(T|X0 =k)(NB!
= 1 +P1,0 0 +P1,1 v+P1,2 0.
And we find
v= 1
1 P1,1=
1
1
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
More than one transient state
P=
1 0 0 0
P1,0 P1,1 P1,2 P1,3P2,0 P2,1 P2,2 P2,3
7/24/2019 Dtmc Basic
179/185
0 0 0 1
Here we will need conditional probabilitiesui= P(XT =0|X0=i)
and conditional expectationsvi= E(T|X0 =i)
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Leading to
u1 = P1,0+P1,1u1+P1,2u2
u2 = P2,0+P2,1u1+P2,2u2
7/24/2019 Dtmc Basic
180/185
and
v1 = 1 +P1,1v1+P1,2v2
v2 = 1 +P2,1v1+P2,2v2
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
General finite state Markov chain
P
Q R
7/24/2019 Dtmc Basic
181/185
P=
0 I
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
General absorbing Markov chain
T =min {n0, Xnr}
In statejwe accumulate rewardg(j),wiis expected totalreward conditioned on start in state i
7/24/2019 Dtmc Basic
182/185
wi= ET1
n=0
g(Xn)|X0 =ileading to
wi=g(i) +
j
Pi,jwj
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Special cases of general absorbing Markov chain
7/24/2019 Dtmc Basic
183/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Special cases of general absorbing Markov chain
g(i) =1 expected time to absorption (vi)
7/24/2019 Dtmc Basic
184/185
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification
Special cases of general absorbing Markov chain
g(i) =1 expected time to absorption (vi)
7/24/2019 Dtmc Basic
185/185
g(i) =ikexpected visits to state kbefore absorption
Bo Friis Nielsen Discrete Time Markov Chains, Definition and classification