Chapter II: Markov Jump Processes

MFF UK

IntroductionIntensity Matrix and Kolmogorov Differential Equations

Stationary DistributionTime Reversibility

Chapter II: Markov Jump Processes

Jakub Cerny

Department of Probability and Mathematical Statistics

Stochastic Modelling in Economics and Finance

October 15, 20121 / 25

MFF UK



Contents

1 Introduction

2 Intensity Matrix and Kolmogorov Differential Equations

3 Stationary Distribution

4 Time Reversibility

2 / 25

MFF UK



Notation and Markov property

(Ω,F ,P) . . . probability spaceE . . . finite or countable state spaceS0,S1,S2, . . . . . . times of jumps, S0 := 0Tn = Sn+1 − Sn . . . holding times, n ∈ N

Y0,Y1,Y2 . . . . . . visited states from state space E

Definition (Markov Chain with Continuous Time)

The system of random variables Xt , t ≥ 0 defined on (Ω,F ,P) iscalled Markov chain with continous time and countable (finite)state space (Markov jump process) E if

P(Xt = j |Xs = i , . . . ,Xt1 = i1) = P(Xt = j |Xs = i)

for all j , i , i1, . . . , in ∈ E and 0 < t1 < · · · < tn < s < t whereP(Xs = i ,Xtn = in, . . . ,Xt1 = i1) > 0.

3 / 25

MFF UK



Basic Characteristics

Assuming that Markov jump process is time-homogenous, i.e.

P(Xs+t = j |Xs = i) = P(Xt = j |X0 = i) = Pi(Xt = j) = ptij ,

let us denote the transition semigroup by the family

ptij , t ≥ 0,∑

j∈E

ptij = 1 =

Pt , t ≥ 0

.

Chapman-Kolmogorov equations are analogic to discrete timeMarkov chains

pt+sij =

∑

k∈E

psikptkj

and in the matrix form Pt+s = PtPs which can be also intepretedas usual matrix multiplication.

4 / 25

MFF UK



Example of Application in Insurance

healthy

ill

dead

S S S S1 2 3 4S0

T T T T1 2 3 4

5 / 25

MFF UK



Absorbtion State

Two possible phenomena of the Markov jump process may occur.First,the process may be absorbed at some state, for eg. state i . Itmeans that there exists a last finite jump time Sn. ThenTj = ∞, j = n, n + 1, . . . and Yk = i , k = n, n+ 1, . . .

healthy

ill

dead

S S S S1 2 3 4S0

absorbtion state

last finite jump time

6 / 25

MFF UK



Explosion Time

Second, the jumps of the process may accumulate, i.e. the processexplodes. Jump times of the process are defined as

S1 = inft > 0,Xt 6= X0

S2 = inft > S1,Xt 6= XS1

. . .

Sn = inft > Sn−1,Xt 6= XSn−1

and jump times can be rewritten by holding times as

Sn =n

∑

k=1

Tk , ξ = supSn =∞∑

k=1

Tk ,

Random variable ξ is called explosion time.

7 / 25

MFF UK



Example - Explosion Process

S S S S1 2 3 4S0

Y1

Y2

bbb

b b b ξ

8 / 25

MFF UK



Intensity Matrix

Theorem

For every state i ∈ E exists a limit

limh→0+

1− phiih

:= Λi ≤ ∞

and for every i , j ∈ E, i 6= j exists a limit

limh→0+

phijh

:= Λij < ∞

and for every i ∈ E is∑

i 6=j

Λij ≤ Λi .

Nonnegative numbers Λij are called transition intensities from thestate i to the state j , Λii = −Λi , Λi is called total intensity. Thematrix Λ = Λij , i , j ∈ E is called intensity matrix. 9 / 25

MFF UK



Intensity Matrix

When state space E is finite then

Λi =∑

i 6=j

Λij ⇒∑

j∈E

Λij = 0.

If state space E is infinite then

Λi ≥∑

i 6=j

Λij .

Theorem

If Λi = 0 then ptii = 1. If 0 < Λi < ∞ then the holding time instate i has exponential distribution with expected value equal 1/Λi .

10 / 25

MFF UK



Reuter’s Explosion Condition

The following result is of gives a necessary and sufficient condition,known as Reuter’s condition, for a Markov jump process to beexplosive (nontrivial application are birth-death processes).

Theorem

A Markov jump process is nonexplosive if and only if the onlynonngeative bounded solution k = (ki )i∈E to the set of equations

Λk = k

is k = 0.

11 / 25

MFF UK



Kolmogorov Differential Equations

Let Λ be an intensity matrix on E , Λi < ∞ and Xt , t ≥ 0 is theMarkov jump process defined on (Ω,F ,P), then E × E -matricesPt satisfy the backward equation, i.e.

(ptij)′

=∑

k∈E

Λikptkj = −Λip

tij +

∑

k 6=i

Λikptkj

(Pt)′

= ΛPt ,

and forward equation, i.e.

(ptij )′

=∑

k∈E

ptikΛkj = −ptijΛj +∑

k 6=i

ptikΛkj

(Pt)′

= PtΛ,

assuming thatphij

h→ Λij converges uniformly in i .

12 / 25

MFF UK



Kolmogorov Differential Equations

Theorem

If E is finite and Λ = Λij , 0 ≤ i , j ≤ n is a matrix whereΛij ≥ 0, i 6= j and Λi =

∑

i 6=j

Λij . Then exists a unique solution of

both Kolmogorov differential equations which satisfies the initialcondition P0 = I. The solution in matrix form is

Pt = eΛt

where eΛt is an exponential matrix function defined as

eΛt =

∞∑

k=0

Λktk

k!

13 / 25

MFF UK



Example - Kolmogorov Differential Equations

Suppose that E has p = 2 states Y1,Y2 and intensities Λ(Y1),Λ(Y2) are not zero. Then Λ has eigenvalues 0 andΛ = −Λ(Y1)− Λ(Y2) with corresponding right eigenvectors(1, 1)T , (Λ(Y1),−Λ(Y2)). Hence

Λ =

(

−Λ(Y1) Λ(Y1)Λ(Y2) −Λ(Y2)

)

= B

(

0 00 Λ

)

B−1, B =

(

1 Λ(Y1)1 −Λ(Y2)

)

using eigendecomposition.

Pt = eΛt =

∞∑

n=0

tn

n!B

(

0 00 Λn

)

B−1 = B

(

0 00 eΛt

)

B−1

=1

Λ(Y1) + Λ(Y2)

(

Λ(Y2) + Λ(Y1)eΛt Λ(Y1)− Λ(Y1)e

Λt

Λ(Y2)− Λ(Y2)eΛt Λ(Y1) + Λ(Y2)e

Λt .

)

.

14 / 25

MFF UK



Stationary Measure and Distribution

Forward and backward equations are quite limited utility even incases when state space is finite (for eg. complex eigenvalues). Onecommon application is to look for a stationary distribution.

Definition

Let Xt , t ≥ 0 be Markov jump process with transition matrix Pt .If vector π = πj , j ≥ 0 satisfies

πT = π

TPt

is called stationary measure of the process Xt , t ≥ 0 on E due toPt , t ≥ 0. If π is also probability distribution on E then it iscalled stationary distribution.

15 / 25

MFF UK



Classification of States

Let Λ∗ = Λ∗ij , i , j ∈ E be matrix with

Λ∗ij =

Λij/Λi , Λi > 00, Λi = 0

, i 6= j

Λ∗ii =

0, Λi > 01, Λi = 0

and we know that jump times of Xt , t ≥ 0 are given by thesequence S0,S1, . . . . We define

Z0 = X0,Zn = XSn , n = 1, 2, . . .

It can be shown that Zn, n ∈ N0 is discrete Markov chain withtransition probabilities Λ∗

ij defined above.

16 / 25

MFF UK



Classification of States

Theorem

The following properties are equivalent

(i) Zn, n ∈ N0 is irreducible,

(ii) for any i , j ∈ E we have ptij > 0 for some t > 0

(iii) for any i , j ∈ E we have ptij > 0 for all t > 0.

We define Xt , t ≥ 0 to be irreducible if one of the properties(i)-(iii) hold. Similarly it is seen that we can define i to betransient (reccurent) for Xt , t ≥ 0 if either (i) the sett : Xt = i is bounded (unbounded) Pi -a.s. (ii) i is transient(reccurent) for Zn, n ∈ N0 or (iii)Pi(inft > 0,Xt = i , lims→t Xs 6= i) < 1(= 1).

17 / 25

MFF UK



Stationary Measure

Theorem

Suppose that Xt , t ≥ 0 is irreducible and recurrent on E. Thenthere exists one, and up to a multiplicative factor only one,invariant measure π. This has the property πj > 0 for all j and canbe found in either of the following ways:

(i) for some fixed but arbitrary state i , πj is the expected timespent in j between successive entrances to i . That is, withω(i) = inft > 0,Xt = i , lims↑t Xs 6= i

πj = Ei

ω(i)∫

0

I(Xt = j)dt

(ii) πj = µj/Λj where µ is stationary for Zn

(iii) as a solution of πΛ = 0.18 / 25

MFF UK



Ergodicity

An irreducible recurrent process with the stationary measurehaving finite mass is called ergodic, and

Theorem

An irreducible nonexplosive Markov jump process is ergodic if andonly if exists a probability solution π (

∑

i∈E

πi = 1, πi ∈ [0, 1]) to

πΛ = 0. In that case π is the stationary distribution.

Theorem

If Xt , t ≥ 0 is ergodic and π is the stationary distribution, thenptij → πj , t → ∞ for all states i , j ∈ E.

19 / 25

MFF UK



Ergodicity

As in discrete time, time-average properties like

1

T

T∫

0

f (Xt)dta.s.→ π(f ) = Eπ(f (Xt)) =

∑

i∈E

πi f (i)

hold under suitable conditions on f . It means that the timeaverage converges to the spatial average if the process is ergodic.

Corollary

If Xt , t ≥ 0 is irreducible recurrent but not ergodic (stationarymeasure does not have finite mass), then ptij → 0 for all statesi , j ∈ E.

20 / 25

MFF UK



Time Reversibility

Time reversibility (or just reversibility) of a process means looselythat the process evolves in just the same way irrespective ofwhether time is read forward (as usual) or backward.

Definition (Reversibility)

Process Xt , t ∈ R is time reversible if for finite dimensionaldistributions and for all t is

(Xt1 ,Xt2 , . . . ,Xtn)D= (Xt−t1 ,Xt−t2 , . . . ,Xt−tn),

Lemma

If the process Xt , t ∈ R is time reversible then it is also timestationary.

21 / 25

MFF UK



Detailed Balance Condition

Theorem

Stationary Markov jump process Xt , t ∈ R with intensity matrixΛ is time reversible if and only if exists probability distribution π onE satisfying

π(x)Λ(x , y) = π(y)Λ(y , x), x , y ∈ E .

In this case π is stationary distribution.

Previous theorem is also called detailed balance conditionbecause the flow between every two states is in balance. The termπ(x)Λ(x , y) is the probability flow from x to y .

22 / 25

MFF UK



Full Balance Condition

In contrast to detailed balance condition the equilibrium equationπΛ = 0 gives the condition of full balance. More precisely,rewriting πΛ = 0

∑

x 6=y

π(x)Λ(x , y) =∑

x 6=y

π(y)Λ(y , x) for all states.

This condition loosely means that everything that flows into somestate also flows out of it. Left hand side is the inflow of the state xand right hand side is the outflow.

23 / 25

MFF UK



References

Asmussen S.: Applied Probability and Queues, Springer, NewYork, 2003.

Lachout P., Praskova Z.: Zaklady nahodnych procesu,Karolinum, Praha, 2005.

Uncovsky L.: Stochasticke modely operacnej analyzy, Alfa,Bratislava, 1980.

24 / 25

MFF UK



Thank you for your attention!

Jakub Cerny

25 / 25

Documents

Chapter II: Markov Jump Processes