Upload
darius
View
52
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Flows and Networks Plan for today (lecture 2):. Questions? Continuous time Markov chain Birth-death process Example: pure birth process Example: pure death process Simple queue General birth-death process: equilibrium Reversibility, stationarity Truncation Kolmogorov’s criteria - PowerPoint PPT Presentation
Citation preview
Flows and Networks
Plan for today (lecture 2):
• Questions?• Continuous time Markov chain• Birth-death process• Example: pure birth process• Example: pure death process• Simple queue• General birth-death process:
equilibrium• Reversibility, stationarity• Truncation• Kolmogorov’s criteria• Summary / Next• Exercises
Discrete time Markov chain: summary
• stochastic process X(t) countable or finite state space S
Markov property
time homogeneous
independent t
irreducible: each state in S reachable from any other state in S
transition probabilities
Assume ergodic (irreducible, aperiodic) global balance equations (equilibrium eqns)
solution that can be normalised is equilibrium distributionif equilibrium distribution exists, then it is unique and is limiting distribution
))(|)((
))(,...,)(|)((
11
1111
nnnn
nnnn
jtXjtXP
jtXjtXjtXP
))(|)(( jtXktXP
))(|)1((),( jtXktXPkjp
1),(
kjpSk
).()()( jkpkjSk
)())0(|)((lim kjXktXPt
Random walk
http://www.math.uah.edu/stat/
• Gambling game over infinite time horizon: on any turn– Win €1 w.p. p– Lose €1 w.p. 1-p – Continue to play
– Xn= amount after n plays
– State space S = {…,-2,-1,0,1,2,…}
– Time homogeneous Markov chain
– For each finite time n:
– But equilibrium?
piip
iXjXPpiip nn
1)1,(
)|()1,( 1
)|( 0 iXjXP n
Continuous time Markov chain• stochastic process X(t)
countable or finite state space S
Markov property
transition probability
irreducible: each state in S reachable from any other state in S
Chapman-Kolmogorov equation
transition rates or jump rates
))(|)((
))(...,)(,)(|)(( 11
itXjstXP
jtXjtXitXjstXP nn
))0(|)((),( iXjtXPjiPt
),(),(),( jkPkiPjiP stk
st
jih
jiPjiq h
h
),(lim),(
0
)(),(),( hohjiqjiPh
Continuous time Markov chain
• Chapman-Kolmogorov equation
transition rates or jump rates
• Kolmogorov forward equations: (REGULAR)
Global balance equations
),(),(),( jkPkiPjiP stk
st
jih
jiPjiq h
h
),(lim),(
0
)],()(),()([0
)],(),(),(),([),('
)],(),(),(),([
]1),()[,(),(),(),(),(
),(),(),(
kjqjjkqk
kjqjiPjkqkiPjiP
kjPjiPjkPkiP
jjPjiPjkPkiPjiPjiP
jkPkiPjiP
jk
ttjk
t
hthtjk
hthtjk
tht
htk
ht
Continuous time Markov chain: summary
• stochastic process X(t) countable or finite state space S
Markov property
transition rates
independent t
irreducible: each state in S reachable from any other state in S
Assume ergodic and regular global balance equations (equilibrium
eqns)
π is stationary distribution
solution that can be normalised is equilibrium distributionif equilibrium distribution exists, then it is unique and is limiting distribution
)],()(),()([0 kjqjjkqkjk
)())0(|)((lim kjXktXPt
))(|)((
))(...,)(,)(|)(( 11
itXjstXP
jtXjtXitXjstXP nn
jih
jiPjiq h
h
),(lim),(
0
Flows and Networks
Plan for today (lecture 2):
• Questions?• Birth-death process• Example: pure birth process• Example: pure death process• Simple queue• General birth-death process:
equilibrium• Reversibility, stationarity• Truncation• Kolmogorov’s criteria• Summary / Next• Exercises
Birth-death process• State space
• Markov chain, transition rates
• Bounded state space:
q(J,J+1)=0 then states space bounded above at J
q(I,I-1)=0 then state space bounded below at I
• Kolmogorov forward equations
• Global balance equations
otherwise
jkjj
ratedeathjkj
ratebirthjkj
kjq
0
)()(
1)(
1)(
),(
ZS
)1()1()]()()[()1()1(0
)1()1,()]()()[,()1()1,(),(
jjjjjjj
jjiPjjjiPjjiPdt
jidPttt
t
Flows and Networks
Plan for today (lecture 2):
• Questions?• Birth-death process• Example: pure birth process• Example: pure death process• Simple queue• General birth-death process:
equilibrium• Reversibility, stationarity• Truncation• Kolmogorov’s criteria• Summary / Next• Exercises
Example: pure birth process• Exponential interarrival times, mean 1/
• Arrival process is Poisson process• Markov chain? • Transition rates : let t0<t1<…<tn<t
• Kolmogorov forward equations for P(X(0)=0)=1
• Solution for P(X(0)=0)=1
jk
jkkjq
hohjtXjhtXP
hojtXjhtXP
hohjtXjhtXP
jntnXjtXjtXjhtXP
1),(
)(1))(|)((
)())(|2)((
)())(|1)((
))(,...,0)0(,)(|1)((
),0()0,0(
),0()1,0(),0(
jPdt
dP
jPjPdt
jdP
tt
ttt
0,...,2,1,0,!
)(),0( tje
j
tjP t
j
t
Flows and Networks
Plan for today (lecture 2):
• Questions?• Birth-death process• Example: pure birth process• Example: pure death process• Simple queue• General birth-death process:
equilibrium• Reversibility, stationarity• Truncation• Kolmogorov’s criteria• Summary / Next• Exercises
Example: pure death process• Exponential holding times, mean 1/
• P(X(0)=N)=1, S={0,1,…,N}
• Markov chain? • Transition rates : let t0<t1<…<tn<t
• Kolmogorov forward equations for P(X(0)=N)=1
• Solution for P(X(0)=N)=1
jkj
jkjkjq
hohjjtXjhtXP
hojtXjhtXP
hohjjtXjhtXP
jntnXjtXjtXjhtXP
1),(
)(1))(|)((
)())(|2)((
)())(|1)((
))(,...,0)0(,)(|1)((
),0(),(
),0()1,0()1(),(
NPNdt
NNdP
jPjjPjdt
jNdP
tt
ttt
0,,...,2,1,0,1),(
tNjeej
NjNP
jNtjtt
Flows and Networks
Plan for today (lecture 2):
• Questions?• Birth-death process• Example: pure birth process• Example: pure death process• Simple queue• General birth-death process:
equilibrium• Reversibility, stationarity• Truncation• Kolmogorov’s criteria• Summary / Next• Exercises
Simple queue• Poisson arrival proces rate , single server
exponential service times, mean 1/
• Assume initially empty:
P(X(0)=0)=1, S={0,1,2,…,}
• Markov chain? • Transition rates :
0,
0,][
0,1
1
),(
)(][1))(|)((
)())(|1)((
)())(|1)((
jjk
jjk
jjk
jk
kjq
hohhjtXjhtXP
hohjtXjhtXP
hohjtXjhtXP
Simple queue• Poisson arrival proces rate , single server
exponential service times, mean 1/
• Kolmogorov forward equations, j>0
• Global balance equations, j>0
0,
0,][
0,1
1
),(
jjk
jjk
jjk
jk
kjq
)1()0(0
)1(])[()1(0
)1,()0,()0,(
)1,(])[,()1,(),(
jjj
iPiPdt
idP
jiPjiPjiPdt
jidP
ttt
tttt
Simple queue (ctd)
j j+1
Equilibrium distribution: <
Stationary measure; summable eq. distrib.
Proof: Insert into global balance
Detailed balance!
j
jj
)/)(/1(
)/)(0()(
).1()1()1,()( jjqjjjqj
Flows and Networks
Plan for today (lecture 2):
• Questions?• Birth-death process• Example: pure birth process• Example: pure death process• Simple queue• General birth-death process:
equilibrium• Reversibility, stationarity• Truncation• Kolmogorov’s criteria• Summary / Next• Exercises
Birth-death process• State space
• Markov chain, transition rates
• Definition: Detailed balance equations
• Theorem: A distribution that satisfies detailed balance is a stationary distribution
• Theorem: Assume that
then
is the equilibrium distrubution of the birth-death prcess X.
0,)0(
0,)()(
0,1)(
1)(
),(
jjk
jjkjj
ratedeathjjkj
ratebirthjkj
kjq
,...}2,1,0{ NS
).1()1()1,()( jjqjjjqj
1
1 )1,(
),1()0(
rrq
rrqj
rSj
Sjrrq
rrqj
j
r
,)1,(
),1()0()(
1
Flows and Networks
Plan for today (lecture 2):
• Questions?• Birth-death process• Example: pure birth process• Example: pure death process• Simple queue• General birth-death process:
equilibrium• Reversibility, stationarity• Truncation• Kolmogorov’s criteria• Summary / Next• Exercises
Reversibility; stationarity• Stationary process: A stochastic process is
stationary if for all t1,…,tn,
• Theorem: If the initial distribution is a stationary distribution, then the process is stationary
• Reversible process: A stochastic process is reversible if for all t1,…,tn,
NOTE: labelling of states only gives suggestion of one dimensional state space; this is not required
))(),...,(),((~))(),...,(),(( 2121 nn tXtXtXtXtXtX
1)(
jSj
))(),...,(),((~))(),...,(),(( 2121 nn tXtXtXtXtXtX
Reversibility; stationarity• Lemma: A reversible process is stationary.
• Theorem: A stationary Markov chain is reversible if and only if there exists a collection of positive numbers π(j), jS, summing to unity that satisfy the detailed balance equations
When there exists such a collection π(j), jS, it is the equilibrium distribution
• Proof
Skjjkqkkjqj ,),,()(),()(
Flows and Networks
Plan for today (lecture 2):
• Questions?• Birth-death process• Example: pure birth process• Example: pure death process• Simple queue• General birth-death process:
equilibrium• Reversibility, stationarity• Truncation• Kolmogorov’s criteria• Summary / Next• Exercises
Lemma 1.9 / Corollary 1.10:
If the transition rates of a reversible Markov process with
state space S and equilibrium distribution are
altered by changing q(j,k) to cq(j,k) for
where c>0 then the resulting Markov process is
reversible in equilibrium and has equilibrium distribution
where B is the normalizing constant.
If c=0 then the reversible Markov process
is truncated to A and the resulting Markov
process is reversible with equilibrium distribution
Truncation of reversible processes
Sjj ),(
10
ASkAj \,
ASjjBc
AjjB
\)(
)(
Ajk
j
Ak
)(
)(
A
S\A
Time reversed processX(t) reversible Markov process X(-t) also, butLemma 1.11: tijdshomogeneity not inherited for
non-stationary process
Theorem 1.12 : If X(t) is a stationary Markov process with transition rates q(j,k), and equilibrium distribution π(j), jS, then the reversed processX(-t) is a stationary Markov process with transition rates
and the same equilibrium distribution
Theorem 1.13: Kelly’s lemmaLet X(t) be a stationary Markov processwith transition rates q(j,k). If we can find a collection of numbers q’(j,k) such that q’(j)=q(j), jS, and a collection of positive numbers (j), jS, summing to unity, such that
then q’(j,k) are the transition rates of the time-reversed process, and (j), jS, is the equilibrium distribution of both processes.
)(
),()(),('
j
jkqkkjq
Skj ,
),(')(),()( jkqkkjqj Skj ,
Flows and Networks
Plan for today (lecture 2):
• Questions?• Birth-death process• Example: pure birth process• Example: pure death process• Simple queue• General birth-death process:
equilibrium• Reversibility, stationarity• Truncation• Kolmogorov’s criteria• Summary / Next• Exercises
Kolmogorov’s criteria• Theorem 1.8:
A stationary Markov chain is reversible iff
for each finite sequence of states
Notice that
),(),()...,(),(
),(),()...,(),(
122311
113221
jjqjjqjjqjjq
jjqjjqjjqjjq
nnn
nnn
)0,(),(),()...,(),(
),(),()...,(),(),0()0()(
112231
132211
jqjjqjjqjjqjjq
jjqjjqjjqjjqjqj
nnn
nnn
Sjjj n ,...,, 21
Flows and Networks
Plan for today (lecture 2):
• Questions?• Birth-death process• Example: pure birth process• Example: pure death process• Simple queue• General birth-death process:
equilibrium• Reversibility, stationarity• Truncation• Kolmogorov’s criteria• Summary / Next• Exercises
Summary / next:
• Birth-death process• Simple queue• Reversibility, stationarity• Truncation• Kolmogorov’s criteria
• Nextinput / output simple queuePoisson procesPASTAOutput simple queueTandem netwerk
Exercises[R+SN] 1.3.2, 1.3.3, 1.3.5, 1.5.1, 1.5.2, 1.5.5,
1.6.2, 1.6.3, 1.6.4