Part IB Michaelmas 2009 YMS

E-mail: yms@statslab.cam.ac.uk

MARKOV CHAINS

5. Recurrence and transience

Recurrence and transience; equivalence of transience and summability of n-step transition probabilities; equivalence of recurrence and certainty of re-turn. Recurrence as a class property, relation with closed classes. Simplerandom walks in dimensions one, two and three.

Given a state i ∈ I, we call it recurrent (R) if

Pi

(Xn = i for infinitely many n

)= 1,

⇔ the series∑n≥0

p(n)ii diverges (5.1.1)

and transient (T) if

Pi

(Xn = i for infinitely many n

)= 0,

⇔ the series∑n≥0

p(n)ii converges. (5.1.2)

The last condition means that

Pi

(Xn = i for finitely many n

)= 1.

(The symbol ⇔ means equivalence.) Note that all probabilities involvedtake values 0 or 1 but not from (0, 1). The equivalence of properties listed in(5.1.1) and that of properties listed in (5.1.2) is the subject of Theorem 5.1below.

Given r = 0, 1, . . ., define the rth passage time T(r)i to state i by T

(0)i = 0

and

T(1)i = inf

[n ≥ 1 : Xn = i

], T

(r+1)i = inf

[n ≥ T

(r)i + 1 : Xn = i

], r ≥ 1,

(5.2)

1

and the length of the rth excursion to i by

S(r)i =

{T

(r)i − T

(r−1)i , if T

(r−1)i < ∞,

0, if T(r−1)i = ∞.

(5.3)

When r = 1, we simply write T(1)i = Ti; in addition, T

(r+1)i is set to equal

+∞ when T(r)i = +∞. See the diagram below.

T

i

i Ti

Ti Ti

Ti

T(1)S i

S (2)i i

S(3)

i

S(4)i S(5)

i

S

(1)(2)

(3)(4)

(5) (6)

(6)i

time

X n

Next, define the return probability to state i:

fi := Pi(Ti < +∞). (5.4)

This probability may be equal to or less than than one; in Theorem 5.1 itwill be proved that

state i is R iff fi = 1 (5.5.1)

andstate i is T iff fi < 1 (5.5.2)

2

Therefore, every state is either R or T. In Theorem 5.2 we will show thatrecurrence or transience are class properties: two states from the same classare either both R or both T.

Before we proceed further, let us analyse the structure of the series∑n≥0

p(n)ii

in (5.1.1) and (5.1.2):

∑n≥0

p(n)ii =

∑n

Pi(Xn = i) =∑n

Ei1(Xn = i)

= Ei

∑n

1(Xn = i) = EiVi

(5.6)

where

Vi = # of visits to i = time spent in i =∑

n≥0

1(Xn = i). (5.7)

Then equations (5.1.1) and (5.1.2) can be written as

Pi

(Xn = i for

{infinitely

finitelymany times

)= 1 ⇔ EiVi

{= +∞,

< +∞.(5.8)

We see that, for a Markov chain,

Pi

(Vi

{= +∞< +∞

)⇔ EiVi

{= +∞< +∞

(5.9)

and the case where Pi(Vi < +∞) = 1 but EiVi = +∞ does not occur.

Theorem 5.1 In a (λ, P )-Markov chain, ∀ state i ∈ I we have thedichotomy:

∑n≥0

p(n)ii

= EiVi

{= +∞ ⇔ Pi(Vi = +∞) = 1 ⇔ fi = Pi(Ti < +∞) = 1 : R

< +∞ ⇔ Pi(Vi < +∞) = 1 ⇔ fi = Pi(Ti < +∞) < 1 : T

(5.10)Therefore, every state is either recurrent or transient.

3

Proof The random variable Ti is a stopping time. By the strong Markovproperty,

Pi(Vi ≥ 2) = Pi (Xn = i for at least two values of n ≥ 1) = f 2i , (5.10)

and more generally, ∀ k

P(Vi ≥ k) = Pi (Xn = i for at least k values of n ≥ 1) = fki . (5.11)

Let us highlight the event under consideration:

B(i)k :=

{Vi ≥ k

}={Xn = i for at least k values of n ≥ 1

}.

Then, obviously, events B(i)k are decreasing with k: B

(i)1 ⊇ B

(i)2 ⊇ . . ., and

the event

{Vi = +∞

}={Xn = i for infinitely many values of n

}

is the intersection ∩k≥1B(i)k . Hence,

Pi(Vi = +∞) = Pi

(Xn = i for infinitely many n

)

= Pi

(∩k≥1B

(i)k

)

= limk→∞

P(B(i)k ) = lim

k→∞fk

i =

{1, fi = 1

0, fi < 1.

(5.12)

Next,

∑n≥0

p(n)ii = EiVi =

∑r≥0

Pi(Vi ≥ r)

=∑r≥0

f ri

{= +∞, fi = 1, i.e., Pi(Vi = +∞) = 1,

< +∞, fi < 1, i.e., Pi(Vi = +∞) = 0.

(5.13)

QED

Theorem 5.1 will be repeatedly used in the analysis of recurrence and tran-sience of states of various chains.

4

An alternative proof of Theorem 5.1 exploits the probability generatingfunctions of random variable Ti. Set

fi(n) = Pi(Ti = n)= Pi (Xn = i but Xl 6= i for l = 1, . . . , n − 1) , n ≥ 1,

(5.14)

andF (z)(= Fi(z)) = EzTi =

∑

n≥1

znfi(n), |z| < 1, (5.15)

then fi = limz→1

F (z).

On the other hand,

p(n)ii = Pi(Xn = i) = fi(n) + fi(n − 1)pii + . . . + fi(1)p

(n−1)ii ; (5.16)

this implies that if

U(z)(= Ui(z)) =∑

n≥1

p(n)ii zn, |z| < 1, (5.17)

then

U(z) = F (z) + F (z)U(z), i.e. U(z) =F (z)

1 − F (z).

Hence, the limiting value limz→1

U(z) is finite if and only if limz→1

F (z) < 1. That

is, (5.13) holds true if and only if fi < 1.

Remark 5.1 We established that state i is recurrent if and only ifPi(Ti < ∞) = 1, i.e. the return time to i is finite with probability 1. However,the mean EiTi can be finite ot infinite. This divides recurrent states into twodistinct categories: positive recurrent and null recurrent (see later).

For convenience we repeat the definition of a communicating class:

Definition 5.2 States i, j ∈ I belong to the same communicating class

if p(n)ij > 0 and p

(n′)ji > 0 for some n, n′ ≥ 0. The communicating classes form

5

a partition of the state space I, when I is countable, some of them may beinfinite, and the number of communicating classes can also be infinite. Next,a communicating class C is called closed if ∀ i ∈ C, if i → j then j ∈ C.Finally, we say that the chain is irreducible if it has a unique communicatingclass (automatically closed). In other words, in an irreducible chain, thewhole of the state space I is a single (closed) communicating class.

Remark 5.2 Observe that if the state space I is finite, the definition ofa transient state coincides with that of a non-essential state (i.e., a state froma non-closed communicating class). In other words, in the finite case everystate from a non-closed class is transient, and every state from a closed classis recurrent. However, as we noted in Remark 2.1, in the case of a countableDTMC a closed class can consist entirely of transient states, which are, from a‘physical’ point of view, non-essential. It shows that in the countable case theconcept of transience is more relevant than that of a closed communicatingclass.

Our aim now is to prove that recurrence and transience are class properties.This means that if states i, j lie in the same communicating class then theyare either both recurrent or both transient. We therefore could use

Definition 5.3 A communicating class is called recurrent (transient) ifall its states are recurrent (respectively, transient).

Theorem 5.2 Within the same communicating class, all states are ofthe same type. Every finite closed communicating class is recurrent.

Proof Let C be a communicating class. Then, ∀ distinct i, j ∈ C,p

(m)ij > 0 and p

(n)ji > 0 for some m, n ≥ 1. Then ∀ r ≥ 0:

p(n+m+r)ii ≥ p

(m)ij p

(r)jj p

(n)ji and p

(n+m+r)jj ≥ p

(n)ji p

(r)ii p

(m)ij , (5.18)

as the RHS in each inequality takes into account only a part of the possibil-ities of return.

6

Hence

p(r)jj ≤ p

(n+m+r)ii

p(m)ij p

(n)ji

(5.19.1)

and, for r ≥ n + m,p

(r)jj ≥ p

(n)ji p

(r−n−m)ii p

(m)ij . (5.19.2)

Then the series∑r

p(r)ii and

∑r

p(r)jj converge or diverge together.

Now let C be a finite closed communicating class, and j ∈ C. Then, withX0 = j ∈ C, Xn ∈ C ∀ n. Hence, ∃ state i ∈ C visited infinitely often, withPj(Vi = ∞) > 0. (In other words, it cannot be that Pj(Vi = ∞) = 0 forall i ∈ C: we run the chain for infinitely many times among finitely manystates, implying that

∑i∈C

Pj(Vi = ∞) = 1.) By the Strong Markov property,

Pj(Vi = ∞) = Pj(Ti < ∞) Pi(Vi = ∞).

Then the probability Pi(Vi = ∞) cannot be 0 (and hence should be 1). Thus,state i is recurrent. Then every state from C is recurrent. QED

Definition 5.2. A transition matrix P (and a (λ, P ) Markov chain)is called recurrent (transient) if every state i is recurrent (respectively, tran-sient).

We conclude this section with one more statement involving passage, orreturn, times.

Theorem 5.3 (Non-examinable) If P is irreducible and recurrent theneach random variable Tj (the passage time to state j) is finite with probability1. That is, P(Tj < ∞) = 1 ∀ j and initial distribution λ.

Proof By the Markov property

P(Tj < ∞) =∑

i

λiPi(Tj < ∞).

7

Given i, take m with p(m)ji > 0. Write

1 = Pj

(Vj = ∞

)≤ Pj

(Xn = j for some n ≥ m

)

(obviously, there is equality here, but the inequality will also do). Further,

Pj

(Xn = j for some n ≥ m

)

=∑k

p(m)jk Pj

(Xn = j for some n ≥ m

∣∣Xm = k)

=∑k

p(m)jk Pk

(Tj < ∞

)≤∑k

p(m)jk = 1.

We see that each summand p(m)jk Pk

(Tj < ∞

)must be equal to p

(m)jk ; otherwise

(i.e. when p(m)jk Pk

(Tj < ∞

)< p

(m)jk for some k) we would have that 1 < 1.

Therefore,

Pi

(Tj < ∞

)p

(m)ji = p

(m)ji , i.e. Pi

(Tj < ∞

)= 1.

This is true ∀ i, hence ∀ initial distribution λ. Also, it is true ∀ j. QED

Example 5.1 (Math Tripos, Markov Chains, Part IB, 2004, 111H) LetP = (pij) be a transition matrix. What does it mean to say that P is (a)irreducible, (b) recurrent?

Suppose that P is irreducible and recurrent and that the state space con-tains at least two states. Define a new transition matrix P̂ = (p̂ij) by

p̂ij =

{0 if i = j,

(1 − pii)−1pij if i 6= j.

Prove that P̂ is also irreducible and recurrent.

Solution A Markov chain is called irreducible if it has a single commu-nicating class. Equivalently, every pair of states i, j communicate; that is,the n-step transition probability p

(n)ij > 0 for some n(= n(i, j)) ≥ 1. Another

equivalent condition is that ∃ a finite sequence of non-repeated states i = i0,i1, ..., im = j with pilil+1

> 0. A chain is called recurrent if ∀ state i:

Pi

(Xn = i for infinitely many n

)= 1,

8

or equivalently,Pi

(Xn = i for some n ≥ 1

)= 1.

When the chain is irreducible, it is enough to check that ∃ a state i with theabove property. Also, if the chain is irreducible, it is recurrent if and only if ∀state i, the minimal non-negative solution to the equation hP = h satisfyinghi = 1 has hj = 1 ∀ j.

If P = (pij) is irreducible then pii < 1 ∀ state i (unless the total number

of states is one). Matrix P̂ describes the Markov chain obtained from theoriginal DTMC by recording the jumps to the new state only; clearly it isirreducible. Formally, take the sequence i0, ..., im as above, then p̂ilil+1

> 0.

Now check the recurrence of P̂ : if in the original chain pii = 0 then thereturn to state i occurs in both chains on the same event, hence the returnprobability to state i will be the same. If pii > 0 then in the new chain, thereturn probability is equal to

1

1 − piiPi(return to i after time 1 in the original chain)

=1

1 − pii

(1 − pii)

which is 1. Alternatively, hP̂ = h if and only if hP = h, i.e. the solutions toboth equations are the same. Hence, the minimal solution to hP̂ = h withhi = 1 is the same as that to hP = h. Therefore, it is ≡ 1, and the newchain is recurrent if and only if the original one is.

Example 5.2 Consider a homogeneous birth and death process, where

pii+1 = p, pii−1 = 1 − p, i ≥ 1, p01 = r, p00 = 1 − r.

It is often called a homogeneous random walk on the set of non-negative inte-gers Z+ = {0, 1, 2, . . .} with a barrier at 0 (absorbing when r = 0, reflectingwhen r = 1, or a random combination of the two when r ∈ (0, 1)).

Case 1: 0 < r ≤ 1 and 0 < p ≤ 1/2. Here we have a single communicationclass which is closed. That is, the chain is irreducible. For state i = 0: thereturn probability is

f0 = P0(T0 < +∞)= 1 − r + r · P1

(H(0) < +∞

)conditional on the 1st step

= 1 − r + r · h1 = 1 − r + r = 1 : state 0 is R.(5.20)

9

Hence, all states are R, and the whole process is R.

Case 2: 0 < r ≤ 1 and 1/2 < p < 1. Again, the chain is irreducible. Thereturn probability to state i = 0:

f0 = 1 − r + r · h1 = 1 − r + r1 − p

p< 1 − r + r = 1 : state 0 is T. (5.21)

The remaining cases are non-generic: either p or r equal 1 or 0.

For brevity, consider just one of them:

Case 3: 0 ≤ r ≤ 1 and p = 0: from state i ≥ 1 the process always jumpsto i − 1 while from 0 it can jump to 1 when 0 < r ≤ 1 and back to 0 when0 ≤ r < 1 (when 0 < r < 1, both possibilities occur). The chain is reducible:every state i > 1 forms an open CC and so is T. For 0 < r ≤ 1: the pair{0, 1} forms a closed CC, and both i = 0 and i = 1 are R-states. For r = 0state i = 1 is T and state i = 0 R.

Example 5.3 (Random walks on lattices) Random walks on cubiclattices are popular and interesting models of countable Markov chains. Herewe have a ‘particle’ that jumps at times n = 1, 2, . . . from its current positioni ∈ Z

d to another site j ∈ Zd with probability pij, regardless of the past

trajectory. We will mostly focus on homogeneous nearest-neighbour randomwalks where probabilities pij are > 0 only when i and j are neighbouringsites and depend only on the direction from i to j (i.e. are determined by

p0,j where j is a neighbour of the origin 0 = (0, . . . , 0)). For d = 1 lattice Zd

is simply the set of integers; a random walk here is specified by probabilitiesp and q = 1 − p of jumps to the right and the left.

pp

0 1 21_

q=1_ p

This is an intuitively appealing extended version of the drunkard model

10

(or birth-and-death process); see Example 2.1. Here, the state space is I =Z(= Z

1), and the transition probability matrix is infinite and has a distinct‘diagonal’ structure

P =

. . .. . .

. . .. . .

. . .. . .

. . .. . . q 0 p 0 . . .

. . .. . . 0 q 0 p 0

. . .. . .

. . . 0 q 0 p. . .

. . .. . .

. . .. . .

. . .. . .

. . .

, (5.22)

with entries p above and q below the main diagonal, and the rest filled withzeros.

The probability p(n)00 is obviously zero when n is odd (you can’t return after

an odd number of jumps). If n = 2k is even then the return is equivalent tohave exactly k jumps to the right and k jumps to the left. Hence,

p(2k)00 =

(2k)!

k!k!pkqk.

For k large, by Stirling formula n! ∼√

2πnnne−n,

(2k)!

k!k!=∼

√2π × 2k(2k)2ke−2k

2πkk2ke−2k=

1√πk

4k.

Hence, the series∑n

p(n) =∑k

p(2k)00 converges or diverges together with

∑

k

1√k

(4p(1 − p)

)k.

Now, the parabola p 7→ p(1− p) reaches its maximum on [0, 1] at p = 1/2;the maximal value equals 1/4. Hence,

4p(1 − p) ≤ 1, with equality iff p =1

2.

Therefore the above series diverges when p = 1/2 and converges when p 6=1/2. We conclude that for d = 1, state 0 is transient when p 6= 1/2 and

11

recurrent when p = 1/2. As the model is homogeneous, the same holdsfor each state i ∈ Z; one simply says that for p = 1/2 the one-dimensionalnearest-neighbour random walk is recurrent and for p 6= 1/2 transient. Thecase p = 1/2 is called symmetric.

For d = 2, Z2 is a plane square lattice; here we will consider the symmetric

nearest-neighbour random walk where probabilities to jump in every directionare the same and equal 1/4.

0 =(0,0)

1/4

Every closed path on Z2 must have equally many jumps to the left and the

right and equally many jumps up and down.

12

0 = (0,0)

Hence again p(n)00 = 0 when n is odd.

A useful idea is to project the random walk onto orthogonal axes rotatedby π/4.

0 =(0,0)

The moves are in one-to-one correspondence

old coordinates: chain (Xn)move by vector ±(1; 0)move by vector ±(0; 1)

↔new coordinates: chain (X ′

n)

move by vector ± (1/√

2)(1; 1)

move by vector ± (1/√

2)(−1; 1).

13

In the new coordinates, up to a factor1√2, the jumps are along diagonals of

the unit square.

1/4

It means that the chain (X ′n) in the new coordinates is formed by a pair

of independent symmetric nearest-neighbour random walks on Z (in the hor-izontal and vertical directions). Return to 0 = (0, 0) means return to 0 ineach of them. Therefore, for n = 2k

p(2k)00 =

((2k)!

k!k!

1

22k

)2

≈ 1

πk. (5.26)

Hence,∑k

p(2k)00 = ∞, and the random walk is recurrent.

For d = 3, Z3 is the three-dimensional cubic lattice; we may think that

it is an infinitely extended crystal. Then our walking particle may model asolitary quantum electron moving between heavy ions or atoms fixed at thesites of the lattice. The probability of moving to one of the six neighboursequals 1/6.

14

1/6

0 = (0,0,0)

Still, p(n)00 = 0 when n is odd. If n is even, a path returns to 0 = (0, 0, 0) if

and only if it makes equal numbers of jumps in each of three pairs of oppositedirections (up/down, east/west, north/south). So,

p(2k)00 =

∑

i, j, l ≥ 0 :i + j + l = k

(2k)!

(i!)2(j!)2(l!)2

(1

6

)2k

=(2k)!

(k!)2

∑

i, j, l ≥ 0 :i + j + l = k

(k!

i!j!l!

)2(1

6

)2k

≤ (2k)!

(k!)2

(max

k!

i!j!l!

)1

3k

1

22k

∑

i, j, l ≥ 0 :i + j + l = k

k!

i!j!l!

1

3k.

Now, the sum∑

i, j, l ≥ 0 :i + j + l = k

k!

i!j!l!= 3k, (5.27)

the number of ways to place k balls into 3 boxes. Also, for k = 3m

(3m)!

m!m!m!≥ (3m)!

i!j!l!whenever i + j + l = 3m. (5.28)

15

In fact, suppose that i < m < l. Then when you pass to i!j!l! from (m!)3,you either (a) replace the ‘tails’ (i + 1) . . .m and (j + 1) . . . l of m! by theproduct (m + 1) . . . (m + 2m − i − j), i.e. (m + 1) . . . l when j < m or (b)replace the tail (i+1) . . .m by the product (m+1) . . . j(m+1) . . . (3m−i−j),that is (m + 1) . . . j(m + 1) . . . l when j > m. Either way you increase thedenominator, hence decrease the ratio.

Then, for n = 2k = 6m:

p(6m)00 ≤ (6m)!

((3m)!

)2

(1

2

)6m(3m)!

(m!)3

(1

3

)3m

(5.29)

which, by Stirling, is

≈√

2

(1√2π

)31

m3/2. (5.30)

Hence,∑m

p(6m)00 < ∞.

But for m ≥ 1: p(6m)00 ≥ (1/6)2p

(6m−2)00 and p

(6m)00 ≥ (1/6)4p

(6m−4)00 , i.e.

p(6m−2)00 ≤ 62p

(6m)00 and p

(6m−4)00 ≤ 64p

(6m)00 .

Thus, ∑

k

p(2k)00 ≤

∑

m

p(6m)00 (1 + 62 + 64) < ∞, (5.31)

and the walk is transient.

A similar approach can be used in higher dimensions. But there is anotherway to establish transience in all dimensions d > 3. Namely, project therandom walk (Xd

n) on Zd to three dimensions by discarding all coordinates

but the first three. The projected chain(Xproj

n

)on Z

3 stays where it is withprobability (d − 3)/d (when the original walk jumps in one of the discardeddirections), but when it jumps, it behaves as the nearest-neighbour symmetricwalk in dimension 3

P

(Xproj

n+1 = i ± eα∣∣Xproj

n = i)

=1/(2d)

1 − (d − 3)/d=

1

6, α = 1, 2, 3, (5.32)

withe1 = (1; 0; 0), e2 = (0; 1; 0), e3 = (0; 0; 1).

16

d(d−3)

1 2d

Clearly, if the original d-dimensional walk returns to 0 = (0, . . . 0) thenthe projected walk returns to (0, 0, 0). Hence, if the original d-dimensionalwalk (Xd

n) is recurrent then the projected chain(Xproj

n

)is recurrent. But

then consider the random walk on Z3 obtained from

(Xproj

n

)by discarding

the stays and recording the jumps only. The latter is the nearest-neighboursymmetric random walk on Z

3 which is transient. By Theorem 6.3 below,(Xproj

n

)is also transient. Then so is (Xd

n).

Nearest-neighbour symmetric random walks are often called simple walks.Re-phrasing a famous saying, we could state that in two dimensions everyroad of a simple random walk will lead you to the origin (or any other givensite) while in three dimensions and higher it is no longer so. The differencebetween two and three dimensions emerges in a countless variety of situationsin virtually all domains of mathematics.

Example 5.4 (Math Tripos, Markov Chains, Part IIA, 2003, A101J andPart IIB, 2003, B101J) (i) Let (Xn, Yn) be a simple symmetric random walkin Z

2, starting from (0, 0), and set T = inf {n ≥ 0 : max {|Xn|, |Yn|} = 2}.Determine the quantities ET and P(XT = 2 and YT = 0).

(ii) Let (Xn)n≥0 be a Markov chain with state space I and transitionmatrix P . What does it mean to say that a state i ∈ I is recurrent? Prove

that i is recurrent if and only if∞∑

n=0

p(n)ii = ∞, where p

(n)ii denotes the (i, i)

entry in P n.

Show that the simple symmetric random walk in Z2 is recurrent.

17

Solution (i) If ki = EiT and hi = Pi(XT YT = 0) then

k(0,0) = 1 + k(−1,0),k(−1,0) = 1 + k(0,0)/4 + k(−1,−1)/2, k(−1,−1) = 1 + k(−1,0)/2,h(0,0) = h(−1,0),h(−1,0) = 1/4 + h(0,0)/4 + h(−1,−1)/2, h(−1,−1) = h(−1,0)/2,

by conditioning on the first step, the Markov property and symmetry.

O

Hence,

ET = k(0,0) =9

2, h(0,0) =

1

2.

By symmetry,

P(XT = 2 and YT = 0) =1

4h(0,0) =

1

8.

(ii) State i is recurrent if fi = Pi(Ti < ∞) = 1 where Ti = inf {n ≥ 1 :Xn = i}. If Vi is the total time spent in i then

Pi(Vi ≥ k + 1) = Pi(Vi ≥ k) Pi(Vi ≥ k + 1|Vi ≥ k)= Pi(Vi ≥ k)fi = . . . = fk+1

i .

ThenEi(Vi) =

∑

k≥1

P(Vi ≥ k) =∑

k≥0

fki .

On the other hand,

EiVi = Ei

∑

n≥0

1(Xn = i) =∑

n≥0

p(n)ii .

18

Hence, fi = 1 if and only if∑n≥0

p(n)ii = ∞.

Now let (Xn) be a simple symmetric random walk in Z2. It is irreducible,

hence it suffices to check that∑n≥0

p(n)ii = ∞ for a single i ∈ Z

2, say the origin

(0, 0). Write (X±n ) for the projection of (Xn) on the diagonal {x = ±y} in

Z2. Then (X±

n ) are independent simple symmetric random walks on1√2Z,

and return to (0, 0) in (Xn) means return to 0 in each of (X±n ). Next,

P0(X±2k = 0) =

(2kk

)1

22k,

andp

(2k)00 = P0(X

+2k = 0)P0(X

−2k = 0).

Then Stirling’s formula asserts that p(2k)00 is

≈( √

2√2πk

(2k)2k

k2k

1

22k

)2

=1

πk, as k → ∞.

Hence, ∑

n≥0

p(n)00 =

∑

k≥0

p(2k)00 = ∞.

Example 5.5 (Math Tripos, Markov Chains, Part IIA, 1994, A101K)(i) Let (Xn)n≥0 be a simple symmetric random walk on the integers. Showthat (Xn)n≥0 is recurrent.

(ii) Let (Xn)n≥0, (Yn)n≥0 and (Zn)n≥0 be simple symmetric random walkson the integers. Then Vn = (Xn, Yn, Zn) is a Markov chain. What are thetransition probabilities for this chain?

Show that, with probability one, (Vn)n≥0 visits (0, 0, 0) only finitely manytimes.

[Stirling’s formula states

n!(e/n)n/√

2πn → 1 as n → ∞.

19

Other standard results may be used without proofs if clearly stated.]

Solution. (i) We use the following result: ∀ state i:

if Pi(Ti < ∞) = 1, then

∞∑

n=0

p(n)ii = ∞, and i is R,

if Pi(Ti < ∞) < 1, then

∞∑

n=0

p(n)ii < ∞, and i is T.

Also, recurrence and transience are class properties, and the simple sym-metric random walk defines an irreducible chain. So, it suffices to checkrecurrence or transience for a single state, say 0. We have:

∑

n

p(n)00 =

∑

n

p(2n)00 =

∑

n

(2n)!

n!n!

(1

2

)2n

and use Stirling’s formula to obtain the summands√

4πn

2πn

(2n/e)2n

(n/e)2n

1

22n=

1√πn

.

As the series∑n

(1/√

πn) diverges, the random walk is R.

(ii) The transition probabilities for (Vn) are

p(i,j,k)(l,m,n) =1/8, if |i − l| = |j − m| = |l − n| = 1,0, otherwise.

,

and we aim to show that∞∑

n=0

p(n)(0,0,0)(0,0,0) < ∞.

As before, p(n)(0,0,0)(0,0,0) = 0 when n is odd. Furthermore, p

(2n)(0,0,0)(0,0,0) =

(p

(2n)00

)3

where p(2n)00 is as in part (i). Hence,

(p

(2n)00

)3

∼ 1(πn)3/2 , and the

series converges. So, (Vn) is T; hence the answer.

Example 5.6 (Math Tripos, Markov Chains, Part IIA, 1996, A301E)(i) A random sequence of non-negative integers (Fn)n≥0 is obtained by setting

20

F0 = 0 and F1 = 1 and, once F0, . . . , Fn are known, taking Fn+1 to be eitherthe sum or the difference of Fn−1 and Fn, each with probability 1/2. Is(Fn)n≥0 a Markov chain?

By considering the Markov chain Xn = (Fn−1, Fn), find the probabilitythat (Fn)n≥0 reaches 3 before first returning to 0.

(ii) Draw enough of the flow diagram for (Xn)n≥0 to establish a generalpattern. Hence, using the strong Markov property, show that the hittingprobability for (1, 1), starting from (1, 2), is (3 −

√5)/2.

Solution. See the diagram.

..

...

.

.

(1,0)

(1,1)

(1,2)(2,1)

(2,3)

.

....

(1,3)

. ... .(3,2)

..

(3,1)

..(0,1)

(1,1)

(1,2)

(2,3)

(0,1)

(2,1)

.

.(1,3)

(1,2)

(0,1)

(3,5)

(Fn) is not a Markov chain, as Fn+1 depends on Fn and Fn−1, but thepair (Fn−1, Fn) is. The initial part of the diagram shows that the level Fn =3 can be reached from (F0, F1) = (0, 1) either at (2, 3) or (1, 3). To hitthis level before visiting level Fn = 0 (i.e., (1, 0)), we have two straightpaths, supplemented with a number of adjacent triangular cycles. The firstpossibility gives the probability

1 · 1

2· 1

2

(1 +

1

8+

1

82+ . . .

)=

2

7,

and the second

1 · 1

2· 1

2· 1

2

(1 +

1

8+

1

82+ . . .

)=

1

7,

which adds to 3/7.

21

One can see a triangular ‘pattern’ emerging from the diagram, with tree-like symmetries. In particular,

P(1,2)

(hit (1, 1)

)= P(2,3)

(hit (1, 2)

)= P(1,3)

(hit (2, 1)

):= p

andP(2,1)

(hit (1, 1)

)= P(3,2)

(hit (2, 1)

):= p′.

Conditioning on the first jump, by the strong Markov property, we can write

p =1

2p′ +

1

2P(2,3)

(hit (1, 1)

)=

1

2p′ +

1

2p2,

and

p′ =1

2+

1

2P(1,3)

(hit (1, 1)

)=

1

2+

1

2pp′ whence p′ =

1

2 − p.

This yields

p =1

2(2 − p)+

1

2p2, i.e., p3 − 4p2 + 4p − 1 = (p − 1)(p2 − 3p + 1) = 0.

The roots are p = 1 and (3 ±√

5)/2. We are interested in the minimalnon-negative root, i.e., p = (3 −

√5)/2.

Consequently, p′ = 2/(1 +√

5). Obviously, 0 < p, p′ < 1.

22