Markov processes - III

• review of steady-state behavior

• probability of blocked phone calls

• calculating absorption probabilities

• calculating expected time to absorption

1

'V i

)(0=' • - - ',;. ;'-,

1.,,­__ 0 _.. ; " \..­

review of steady state behavior

• Markov chain with a single class of recurrent states. aperiodic; and some transient states; then.

lim " i)'(n) = lim P ( X " = j I Xo = i) = " J' n-+oo 11-+00

(P(l(...:.j .; ~ r1J ... ,. I ... /"1fj" 4-. 1P~O-)1.... ­

• can be found as the unique solution to the balance equations

j = 1, .. . ,

• together with L " j = 1 J

2

• •

on the use of steady state probabilities, example III"O.S~1I'"\.J'O.'l. ~ 1

, o.qqB!).,8'"

iTi -to TTl. =d Od~ I M

assume process

____

starts in state 1

P (Xl = 1 ilnd XIOO = 1 1 Xo = 1) =

IrCx,=.,/'ICo,.l).'p(X,oo='/ )(,:~~) t. f" J' r" Cq'l) 'R:i f" ~ 1Tj OS~~Co

P (XIOO = 1 and XIOI = 2 1 Xo = 1) = .N ) jP(X.o--I/ '1.0.')'" W(X.o. ~2...J XOOO= ~ ~I

f,,(14)O) )I f't. W1f,"'l'l... ~ 2....,.00$ P (XlOO = 1 and X f OO = I J XO = 1) = :}~./ )

,i>OC.,._. Xo•• ). 0'( X?Q)~II X,eo ..'... ~, t. = (,,{.oo) .... r.,(IOO) ~-,y;"',,. :'TI;'-: (~)

, O.S

.' F'2./r ________3>

'" 5""" ",:,.5 ..e ~ct.(, ',,1 ....0 ~..., 10 5 cl."> J

t) O/d..t" .:> ~ .... ..,..~..

:£> PI' -Wc.o/F ...

(c:. \ O<"C < I

_<::0.3 _ C ::: o.~'}

3

••

••

design of _a~hone system (8 r '"",)

• calls originate as a Poisson process. rate oX • each call duration is exponential (parameter f.J.) • need to decide on how many lines, B~

t , ,• tIme

• for time slots of small duration at! ".l I I

l..,.S P(a new call arrives) :::::: Ad if you have i active calls. then P{a departure) ::::::: i J.L§

v;lI. j~ _ t,.~

( ,,,,,I<) -- ­

" .n

" ." "

••

_ _ .1111 -< 7 ~1111

•• • • • ~ II , . M II ~

4

design of a phone system, a discrete time approximation ,

time

• approximation: discrete time slots of (small) duration 8

!P(i. •...,"'")~U ; TI(1."'II"'~ll":b.sy)~<I"S;

o

1 ). ;;::.]0 c.cJ/, I..:... ~ #=->3

J. ""',....It.t 'fo calls• balance e uations

A7fi_ l = iJ1-7ri

11" "

'13", '0 ? "! Ai

- 11"01 -i"1H

~---..,

1

110 , 1r. -) -,rEo - --~- - ­

• P(arriving customer finds busy system) is@

5

-

calculating absorption probabilities

• absorbing state: recurrent state k with Pkk = b..:

• what is the probability @ that the chain eventually settles inA'given it started in i ?

S

1

i = 4, ai = I i =5 , ai = O

3 OJ I

otherwise, a i = ! ., a.: ,BAd" elL'" tons06

1 r-__..:-.<lI::,..:l ::;.1>/t.8-08 ..., • unique solution from i = L Pij aj

.'Ib. .... .l . : I ().s =- -:L. / • -d '--­ 6

expected time to absorption

• find expected number of transitions I-'i until reaching 4, given that the initial state is i

" "

" "

J.Li=o for i = 4o.t­for all others, j..1.'i = ?

, Ih:.IIO/B'0.4

_ /-<'1.:: qt;~ :. ~

~1= 1Iy&

• unique solution from i = 1 + L PijI.Lj .

J

7

mean first passage and recurrence times

• chain with one recurrent class; fix a recurrent state s

"

8

••

gambler's example

• a gambler starts with i dollars; each time. she bets $1 in a fair game, until she either has 0 or n dollars. • what is the probability ai that she ends up with having n '&lIars? , $.v ••r..,. t

• = n, Ui =

~~ •••

>'-­,~~ a...:

, I• expected wealth at the end!

'-' • how long does the gambler expect to stay in the game! a.;. =- % .""A-llcO

/.Li = expected number of plays. starting from i

for i = 0 , n: J-li = 0 \ <.: c .. ~ :. i + I ,/".1+1. -+ - in general

J-Li = 1 + L Pijl.Lj J

• in case of unfavorable odds? f ;: O·s 1._1....• r:: ,-t ~...: =­

f I - r"" 9

MIT OpenCourseWarehttps://ocw.mit.edu

Resource: Introduction to ProbabilityJohn Tsitsiklis and Patrick Jaillet

The following may not correspond to a p articular course on MIT OpenCourseWare, but has been provided by the author as an individual learning resource.

For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.