Markov Chain Specification

1 T o l z S is countable

2 Markov property3 P Xo R Ito r a C S

P Xt y I Xt a Ilse y

K y E S

ITC I should satisfy

1T x y e o D

IT K y I t ne S

YES

IT is a square matrix IT can be

yinfinite dimensional since Is I can be a

Example I Inventory system

demand periods u

mmmmI l l l l l

O

Stock replenishmentpointsn demand period numberXn inventory level at the begin of period n

G random discrete demand during period ns order levelu order upto level

1 Assume that G Gz one iidwith pm f f

2 Notice

XpEnn if s s xn e w

U Ent Xn E S

Example I Inventory system

demagd periods u

mmmmI l l l l l

O

Stock replenishmentpointss

Xn n e T is a Markov chain

with S u u t U z

T 0 1,2

and

Mocaif a u

0 OW

f x y se a e u

ITH'tfu y sees

Example III Epidemic Branching Chain

firmoffn generation number

1 v Xn number infectedIEEE IEEE in generation n

L v j

en reEEE Dv v

IEEE IEEE

S T O 1,2

Suppose each person in a generation independentlyinfects by others where pmf of G is f

Example III Epidemic Branching Chain

Md initial distribution of infected people1T 0,0 I IT 0 y O t y f O

IT r y PCG that then ywhere he Gz G are i i dwith pmf f K 31

Denote fu pmf of G test tha

O I 2

O I O O n n n

DI fill fill2 4 HDc

Example III Mutant Transmission

gene with d units med of whichare mutant

Bz Bz Bz Gen n

replicate3g BB Bae BB

I3h Bz Bz Bz Bz Gen ntl

Xn number of mutant units ingeneration n

S 0,1 d T 0 1,2

IT a y7 215 4 yeaead

0 otherwise

Ito n see o.is d given

Understand the Markov Property

Suppose Xt t e T is a

Markov chain Which of the

following are true statements

I P Xn me A X no X a Xrisen

I P X me A X an

II P Xn mE A XE Ao X EA XEAn.itn an

P Xn mEA Xn an

lI.P XnmEAXoEAo X EA glnEAn

P Xn mEA XnEAn

Why are Markov chains so useful

Let's calculate P X y IX a

IT a y P X y X a

IT ca y IT ca y

IT ca y P X y IX x X zz

P X zlXEP Xn y X z IT a z

z

IT cz.gg TCx.zZ

IT 2e Z 1T Cz yl

Why are Markov chains so useful

From 111 we see that

IT ca y The yIT Ca y IT x y

IT n'cry IT x y z

Since 2 holds fr any n y E S

we can write

IT'M IT 3

Don't confuse computational teeth

IT Cn y f CIT Ca yFH

So then

P X y

Hitting Time

Define

Tea min n o Xn EA

Ta is called the hitting time

of A Hitting time is first hittingkey Identity

time

P X y X a

nn m

P Try m 1T y yM 1

Why is this true

Hitting Time

P X y X a

n

P Try m Ty yM 1

Why is this true