View
2
Download
0
Category
Preview:
Citation preview
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Chapter II: Markov Jump Processes
Jakub Cerny
Department of Probability and Mathematical Statistics
Stochastic Modelling in Economics and Finance
October 15, 20121 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Contents
1 Introduction
2 Intensity Matrix and Kolmogorov Differential Equations
3 Stationary Distribution
4 Time Reversibility
2 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Notation and Markov property
(Ω,F ,P) . . . probability spaceE . . . finite or countable state spaceS0,S1,S2, . . . . . . times of jumps, S0 := 0Tn = Sn+1 − Sn . . . holding times, n ∈ N
Y0,Y1,Y2 . . . . . . visited states from state space E
Definition (Markov Chain with Continuous Time)
The system of random variables Xt , t ≥ 0 defined on (Ω,F ,P) iscalled Markov chain with continous time and countable (finite)state space (Markov jump process) E if
P(Xt = j |Xs = i , . . . ,Xt1 = i1) = P(Xt = j |Xs = i)
for all j , i , i1, . . . , in ∈ E and 0 < t1 < · · · < tn < s < t whereP(Xs = i ,Xtn = in, . . . ,Xt1 = i1) > 0.
3 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Basic Characteristics
Assuming that Markov jump process is time-homogenous, i.e.
P(Xs+t = j |Xs = i) = P(Xt = j |X0 = i) = Pi(Xt = j) = ptij ,
let us denote the transition semigroup by the family
ptij , t ≥ 0,∑
j∈E
ptij = 1 =
Pt , t ≥ 0
.
Chapman-Kolmogorov equations are analogic to discrete timeMarkov chains
pt+sij =
∑
k∈E
psikptkj
and in the matrix form Pt+s = PtPs which can be also intepretedas usual matrix multiplication.
4 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Example of Application in Insurance
healthy
ill
dead
S S S S1 2 3 4S0
T T T T1 2 3 4
5 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Absorbtion State
Two possible phenomena of the Markov jump process may occur.First,the process may be absorbed at some state, for eg. state i . Itmeans that there exists a last finite jump time Sn. ThenTj = ∞, j = n, n + 1, . . . and Yk = i , k = n, n+ 1, . . .
healthy
ill
dead
S S S S1 2 3 4S0
absorbtion state
last finite jump time
6 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Explosion Time
Second, the jumps of the process may accumulate, i.e. the processexplodes. Jump times of the process are defined as
S1 = inft > 0,Xt 6= X0
S2 = inft > S1,Xt 6= XS1
. . .
Sn = inft > Sn−1,Xt 6= XSn−1
and jump times can be rewritten by holding times as
Sn =n
∑
k=1
Tk , ξ = supSn =∞∑
k=1
Tk ,
Random variable ξ is called explosion time.
7 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Example - Explosion Process
S S S S1 2 3 4S0
Y1
Y2
bbb
b b b ξ
8 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Intensity Matrix
Theorem
For every state i ∈ E exists a limit
limh→0+
1− phiih
:= Λi ≤ ∞
and for every i , j ∈ E, i 6= j exists a limit
limh→0+
phijh
:= Λij < ∞
and for every i ∈ E is∑
i 6=j
Λij ≤ Λi .
Nonnegative numbers Λij are called transition intensities from thestate i to the state j , Λii = −Λi , Λi is called total intensity. Thematrix Λ = Λij , i , j ∈ E is called intensity matrix. 9 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Intensity Matrix
When state space E is finite then
Λi =∑
i 6=j
Λij ⇒∑
j∈E
Λij = 0.
If state space E is infinite then
Λi ≥∑
i 6=j
Λij .
Theorem
If Λi = 0 then ptii = 1. If 0 < Λi < ∞ then the holding time instate i has exponential distribution with expected value equal 1/Λi .
10 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Reuter’s Explosion Condition
The following result is of gives a necessary and sufficient condition,known as Reuter’s condition, for a Markov jump process to beexplosive (nontrivial application are birth-death processes).
Theorem
A Markov jump process is nonexplosive if and only if the onlynonngeative bounded solution k = (ki )i∈E to the set of equations
Λk = k
is k = 0.
11 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Kolmogorov Differential Equations
Let Λ be an intensity matrix on E , Λi < ∞ and Xt , t ≥ 0 is theMarkov jump process defined on (Ω,F ,P), then E × E -matricesPt satisfy the backward equation, i.e.
(ptij)′
=∑
k∈E
Λikptkj = −Λip
tij +
∑
k 6=i
Λikptkj
(Pt)′
= ΛPt ,
and forward equation, i.e.
(ptij )′
=∑
k∈E
ptikΛkj = −ptijΛj +∑
k 6=i
ptikΛkj
(Pt)′
= PtΛ,
assuming thatphij
h→ Λij converges uniformly in i .
12 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Kolmogorov Differential Equations
Theorem
If E is finite and Λ = Λij , 0 ≤ i , j ≤ n is a matrix whereΛij ≥ 0, i 6= j and Λi =
∑
i 6=j
Λij . Then exists a unique solution of
both Kolmogorov differential equations which satisfies the initialcondition P0 = I. The solution in matrix form is
Pt = eΛt
where eΛt is an exponential matrix function defined as
eΛt =
∞∑
k=0
Λktk
k!
13 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Example - Kolmogorov Differential Equations
Suppose that E has p = 2 states Y1,Y2 and intensities Λ(Y1),Λ(Y2) are not zero. Then Λ has eigenvalues 0 andΛ = −Λ(Y1)− Λ(Y2) with corresponding right eigenvectors(1, 1)T , (Λ(Y1),−Λ(Y2)). Hence
Λ =
(
−Λ(Y1) Λ(Y1)Λ(Y2) −Λ(Y2)
)
= B
(
0 00 Λ
)
B−1, B =
(
1 Λ(Y1)1 −Λ(Y2)
)
using eigendecomposition.
Pt = eΛt =
∞∑
n=0
tn
n!B
(
0 00 Λn
)
B−1 = B
(
0 00 eΛt
)
B−1
=1
Λ(Y1) + Λ(Y2)
(
Λ(Y2) + Λ(Y1)eΛt Λ(Y1)− Λ(Y1)e
Λt
Λ(Y2)− Λ(Y2)eΛt Λ(Y1) + Λ(Y2)e
Λt .
)
.
14 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Stationary Measure and Distribution
Forward and backward equations are quite limited utility even incases when state space is finite (for eg. complex eigenvalues). Onecommon application is to look for a stationary distribution.
Definition
Let Xt , t ≥ 0 be Markov jump process with transition matrix Pt .If vector π = πj , j ≥ 0 satisfies
πT = π
TPt
is called stationary measure of the process Xt , t ≥ 0 on E due toPt , t ≥ 0. If π is also probability distribution on E then it iscalled stationary distribution.
15 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Classification of States
Let Λ∗ = Λ∗ij , i , j ∈ E be matrix with
Λ∗ij =
Λij/Λi , Λi > 00, Λi = 0
, i 6= j
Λ∗ii =
0, Λi > 01, Λi = 0
and we know that jump times of Xt , t ≥ 0 are given by thesequence S0,S1, . . . . We define
Z0 = X0,Zn = XSn , n = 1, 2, . . .
It can be shown that Zn, n ∈ N0 is discrete Markov chain withtransition probabilities Λ∗
ij defined above.
16 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Classification of States
Theorem
The following properties are equivalent
(i) Zn, n ∈ N0 is irreducible,
(ii) for any i , j ∈ E we have ptij > 0 for some t > 0
(iii) for any i , j ∈ E we have ptij > 0 for all t > 0.
We define Xt , t ≥ 0 to be irreducible if one of the properties(i)-(iii) hold. Similarly it is seen that we can define i to betransient (reccurent) for Xt , t ≥ 0 if either (i) the sett : Xt = i is bounded (unbounded) Pi -a.s. (ii) i is transient(reccurent) for Zn, n ∈ N0 or (iii)Pi(inft > 0,Xt = i , lims→t Xs 6= i) < 1(= 1).
17 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Stationary Measure
Theorem
Suppose that Xt , t ≥ 0 is irreducible and recurrent on E. Thenthere exists one, and up to a multiplicative factor only one,invariant measure π. This has the property πj > 0 for all j and canbe found in either of the following ways:
(i) for some fixed but arbitrary state i , πj is the expected timespent in j between successive entrances to i . That is, withω(i) = inft > 0,Xt = i , lims↑t Xs 6= i
πj = Ei
ω(i)∫
0
I(Xt = j)dt
(ii) πj = µj/Λj where µ is stationary for Zn
(iii) as a solution of πΛ = 0.18 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Ergodicity
An irreducible recurrent process with the stationary measurehaving finite mass is called ergodic, and
Theorem
An irreducible nonexplosive Markov jump process is ergodic if andonly if exists a probability solution π (
∑
i∈E
πi = 1, πi ∈ [0, 1]) to
πΛ = 0. In that case π is the stationary distribution.
Theorem
If Xt , t ≥ 0 is ergodic and π is the stationary distribution, thenptij → πj , t → ∞ for all states i , j ∈ E.
19 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Ergodicity
As in discrete time, time-average properties like
1
T
T∫
0
f (Xt)dta.s.→ π(f ) = Eπ(f (Xt)) =
∑
i∈E
πi f (i)
hold under suitable conditions on f . It means that the timeaverage converges to the spatial average if the process is ergodic.
Corollary
If Xt , t ≥ 0 is irreducible recurrent but not ergodic (stationarymeasure does not have finite mass), then ptij → 0 for all statesi , j ∈ E.
20 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Time Reversibility
Time reversibility (or just reversibility) of a process means looselythat the process evolves in just the same way irrespective ofwhether time is read forward (as usual) or backward.
Definition (Reversibility)
Process Xt , t ∈ R is time reversible if for finite dimensionaldistributions and for all t is
(Xt1 ,Xt2 , . . . ,Xtn)D= (Xt−t1 ,Xt−t2 , . . . ,Xt−tn),
Lemma
If the process Xt , t ∈ R is time reversible then it is also timestationary.
21 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Detailed Balance Condition
Theorem
Stationary Markov jump process Xt , t ∈ R with intensity matrixΛ is time reversible if and only if exists probability distribution π onE satisfying
π(x)Λ(x , y) = π(y)Λ(y , x), x , y ∈ E .
In this case π is stationary distribution.
Previous theorem is also called detailed balance conditionbecause the flow between every two states is in balance. The termπ(x)Λ(x , y) is the probability flow from x to y .
22 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Full Balance Condition
In contrast to detailed balance condition the equilibrium equationπΛ = 0 gives the condition of full balance. More precisely,rewriting πΛ = 0
∑
x 6=y
π(x)Λ(x , y) =∑
x 6=y
π(y)Λ(y , x) for all states.
This condition loosely means that everything that flows into somestate also flows out of it. Left hand side is the inflow of the state xand right hand side is the outflow.
23 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
References
Asmussen S.: Applied Probability and Queues, Springer, NewYork, 2003.
Lachout P., Praskova Z.: Zaklady nahodnych procesu,Karolinum, Praha, 2005.
Uncovsky L.: Stochasticke modely operacnej analyzy, Alfa,Bratislava, 1980.
24 / 25
MFF UK
IntroductionIntensity Matrix and Kolmogorov Differential Equations
Stationary DistributionTime Reversibility
Thank you for your attention!
Jakub Cerny
25 / 25
Recommended