DjVu DocumentJOHN WILEY & SONS. INC. New York • Chichester • Brisbane • Toronto • Singapore
ACQUISITIONS EDITOR Brad Wiley II MARKETING MANAGER Debra Riegert SENIOR PRODUCfION EDITOR Tony VenGraitis MANUFACfURING MANAGER Dorothy Sinclair TEXT AND COVER DESIGN A Good Thing, Inc PRODUCfION COORDINATION Elm Street Publishing Services, Inc
This book was set in Times Roman by Bi-Comp, Inc and printed and bound by Courier/Stoughton The cover was printed by Phoenix Color
Recognizing the importance of preserving what has been written, it is a policy of John Wiley & Sons, Inc to have books of enduring value published in the United States printed on acid-free paper, and we exert our best efforts to that end
The paper in this book was manufactured by a mill whose forest management programs include sustained yield harvesting of its timberlands Sustained yield harvesting principles ensure that the number of trees cut each year does not exceed the amount of new growth
Copyright © 1996, by John Wiley & Sons, Inc
All rights reserved Published simultaneously in Canada
Reproduction or translation of any part of this work beyond that permitted by Sections 107 and 108 of the 1976 United States Copyright Act without the permission of the copyright owner is unlawful Requests for permission or further information shOUld be addressed to the Permissions Department, John Wiley & Sons, Inc
Library of Congress Cataloging-in-Publication Data: Ross, Sheldon M
Stochastic processes/Sheldon M Ross -2nd ed p cm
Includes bibliographical references and index ISBN 0-471-12062-6 (cloth alk paper) 1 Stochastic processes I Title
QA274 R65 1996 5192-dc20
10 9 8 7 6 5 4 3 2
95-38012 CIP
On March 30, 1980, a beautiful six-year-old girl died. This book is dedicated to the memory of
Nichole Pomaras
Preface to the First Edition
This text is a nonmeasure theoretic introduction to stochastic processes, and as such assumes a knowledge of calculus and elementary probability_ In it we attempt to present some of the theory of stochastic processes, to indicate its diverse range of applications, and also to give the student some probabilistic intuition and insight in thinking about problems We have attempted, wherever possible, to view processes from a probabilistic instead of an analytic point of view. This attempt, for instance, has led us to study most processes from a sample path point of view.
I would like to thank Mark Brown, Cyrus Derman, Shun-Chen Niu, Michael Pinedo, and Zvi Schechner for their helpful comments
SHELDON M. Ross
Preface to the Second Edition
The second edition of Stochastic Processes includes the following changes' (i) Additional material in Chapter 2 on compound Poisson random vari­
ables, including an identity that can be used to efficiently compute moments, and which leads to an elegant recursive equation for the probabilIty mass function of a nonnegative integer valued compound Poisson random variable;
(ii) A separate chapter (Chapter 6) on martingales, including sections on the Azuma inequality; and
(iii) A new chapter (Chapter 10) on Poisson approximations, including both the Stein-Chen method for bounding the error of these approximations and a method for improving the approximation itself.
In addition, we have added numerous exercises and problems throughout the text. Additions to individual chapters follow:
In Chapter 1, we have new examples on the probabilistic method, the multivanate normal distnbution, random walks on graphs, and the complete match problem Also, we have new sections on probability inequalities (includ­ ing Chernoff bounds) and on Bayes estimators (showing that they are almost never unbiased). A proof of the strong law of large numbers is given in the Appendix to this chapter.
New examples on patterns and on memoryless optimal coin tossing strate­ gies are given in Chapter 3.
There is new matenal in Chapter 4 covering the mean time spent in transient states, as well as examples relating to the Gibb's sampler, the Metropolis algonthm, and the mean cover time in star graphs.
Chapter 5 includes an example on a two-sex population growth model. Chapter 6 has additional examples illustrating the use of the martingale
stopping theorem. Chapter 7 includes new material on Spitzer's identity and using it to compute
mean delays in single-server queues with gamma-distnbuted interarrival and service times.
Chapter 8 on Brownian motion has been moved to follow the chapter on martingales to allow us to utilIZe martingales to analyze Brownian motion.
ix
x PREFACE TO THE SECOND EDITION
Chapter 9 on stochastic order relations now includes a section on associated random variables, as well as new examples utilizing coupling in coupon collect­ ing and bin packing problems.
We would like to thank all those who were kind enough to write and send comments about the first edition, with particular thanks to He Sheng-wu, Stephen Herschkorn, Robert Kertz, James Matis, Erol Pekoz, Maria Rieders, and Tomasz Rolski for their many helpful comments.
SHELDON M. Ross
CHAPTER 1. PRELIMINARIES 1
1.1. Probability 1 1.2. Random Variables 7 1.3. Expected Value 9 1.4 Moment Generating, Characteristic Functions, and
Laplace Transforms 15 1.5. Conditional ExpeCtation 20
1.5.1 Conditional Expectations and Bayes Estimators 33
1.6. The Exponential Distribution, Lack of Memory, and Hazard Rate Functions 35
1.7. Some Probability Inequalities 39 1.8. Limit Theorems 41 1.9. Stochastic Processes 41
Problems 46 References 55 Appendix 56
CHAPTER 2. THE POISSON PROCESS 59
2 1. The Poisson Process 59 22. Interarrival and Waiting Time Distributions 64 2.3 Conditional Distribution of the Arrival Times 66
2.31. The MIGII Busy Period 73 2.4. Nonhomogeneous Poisson Process 78 2.5 Compound Poisson Random Variables and
Processes 82 2.5 1. A Compound Poisson Identity 84 25.2. Compound Poisson Processes 87
xi
Problems 89 References 97
CHAPTER 3. RENEWAL THEORY 98
3.1 Introduction and Preliminaries 98 32 Distribution of N(t) 99 3 3 Some Limit Theorems 101
331 Wald's Equation 104 332 Back to Renewal Theory 106
34 The Key Renewal Theorem and Applications 109 341 Alternating Renewal Processes 114 342 Limiting Mean Excess and Expansion
of met) 119 343 Age-Dependent Branching Processes 121
3.5 Delayed Renewal Processes 123 3 6 Renewal Reward Processes 132
3 6 1 A Queueing Application 138 3.7. Regenerative Processes 140
3.7.1 The Symmetric Random Walk and the Arc Sine Laws 142
3.8 Stationary Point Processes 149
Problems 153 References 161
41 Introduction and Examples 163 42. Chapman-Kolmogorov Equations and Classification of
States 167 4 3 Limit Theorems 173 44. Transitions among Classes, the Gambler's Ruin Problem,
and Mean Times in Transient States 185 4 5 Branching Processes 191 46. Applications of Markov Chains 193
4 6 1 A Markov Chain Model of Algorithmic Efficiency 193
462 An Application to Runs-A Markov Chain with a Continuous State Space 195
463 List Ordering Rules-Optimality of the Transposition Rule 198
A ••• CONTENTS XIII
Problems 219 References 230
CHAPTER 5. CONTINUOUS-TIME MARKOV CHAINS 231
5 1 Introduction 231 52. Continuous-Time Markov Chains 231 5.3.-Birth and Death Processes 233 5.4 The Kolmogorov Differential Equations 239
5.4.1 Computing the Transition Probabilities 249 5.5. Limiting Probab~lities 251 5 6. Time Reversibility 257
5.6.1 Tandem Queues 262 5.62 A Stochastic Population Model 263
5.7 Applications of the Reversed Chain to Queueing Theory 270 57.1. Network of Queues 271 57 2. The Erlang Loss Formula 275 573 The MIG/1 Shared Processor System 278
58. Uniformization 282
Problems 2R6 References 294
CHAPTER 6. MARTINGALES 295
Introduction 295 6 1 Martingales 295 62 Stopping Times 298 6 3. Azuma's Inequality for Martingales 305 64. Submartingales, Supermartingales. and the Martingale
Convergence Theorem 313 65. A Generalized Azuma Inequality 319
Problems 322 References 327
Introduction 32R 7 1. Duality in Random Walks 329
xiv CONTENTS
7.2 Some Remarks Concerning Exchangeable Random Variables 338
73 Using Martingales to Analyze Random Walks 74 Applications to GI Gil Queues and Ruin
Problems 344 7.4.1 The GIGll Queue 7 4 2 A Ruin Problem
344 347
Problems 352 References 355
CHAPTER 8. BROWNIAN MOTION AND OTHER MARKOV PROCESSES 356
8.1 Introduction and Preliminaries 356 8.2. Hitting Times, Maximum Variable, and Arc Sine
Laws 363 83. Variations on Brownian Motion 366
83.1 Brownian Motion Absorbed at a Value 8.3.2 Brownian Motion Reflected at the Origin 8 3 3 Geometric Brownian Motion 368 8.3.4 Integrated Brownian Motion 369
8.4 Brownian Motion with Drift 372 84.1 Using Martingales to Analyze Brownian
Motion 381
366 368
85 Backward and Forward Diffusion Equations 383 8.6 Applications of the Kolmogorov Equations to Obtaining
Limiting Distributions 385 8.61. Semi-Markov Processes 385 862. The MIG/1 Queue 388 8.6.3. A Ruin Problem in Risk Theory
87. A Markov Shot Noise Process' 393 392
88 Stationary Processes 396
Problems 399 References 403
Introduction 404 9 1 Stochastically Larger 404
CONTENTS xv
9 2. Coupling 409 9.2 1. Stochastic Monotonicity Properties of Birth and
Death Processes 416 9.2.2 Exponential Convergence in Markov
Chains 418 0.3. Hazard Rate Ordering and Applications to Counting
Processes 420 94. Likelihood Ratio Ordering 428 95. Stochastically More Variable 433 9.6 Applications of Variability Orderings 437
9.6.1. Comparison of GIGl1 Queues 439 9.6.2. A Renewal Process Application 440 963. A Branching Process Application 443
9.7. Associated Random Variables 446
Probl ems 449 References 456
CHAPTER 10. POISSON APPROXIMATIONS 457
Introduction 457 10.1 Brun's Sieve 457 10.2 The Stein-Chen Method for Bounding the Error of the
Poisson Approximation 462 10.3. Improving the Poisson Approximation 467
Problems 470 References 472
INDEX 505
CHAPTER 1
1. 1 PROBABILITY
A basic notion in probability theory is random experiment an experiment whose outcome cannot be determined in advance. The set of all possible outcomes of an experiment is called the sample space of that experiment, and we denote it by S.
An event is a subset of a sample space, and is said to occur if the outcome of the experiment is an element of that subset. We shall suppose that for each event E of the sample space S a number P(E) is defined and satisfies the following three axioms*:
Axiom (1) O:s;; P(E) ~ 1.
Axiom (2) P(S) = 1 Axiom (3) For any sequence of events E., E2 , •• that are mutually
exclusive, that is, events for which E,E, = cf> when i ~ j (where cf> is the null set),
P ( 9 E,) = ~ P(E,).
We refer to P(E) as the probability of the event E. Some simple consequences of axioms (1). (2). and (3) are.
1.1.1. If E C F, then P(E) :S P(F). 1.1.2. P(EC) = 1 - P(E) where EC is the complement of E.
1.1.3. P(U~ E,) = ~~ P(E,) when the E, are mutually exclusive. 1.1.4. P(U~ E,) :S ~~ P(E,).
The inequality (114) is known as Boole's inequality
* Actually P(E) will only be defined for the so-called measurable events of S But this restriction need not concern us
1
2 PRELIMIN ARIES
An important property of the probability function P is that it is continuous. To make this more precise, we need the concept of a limiting event, which we define as follows· A sequence of even ts {E" , n 2= I} is said to be an increasing sequence if E" C E,,+1> n ;> 1 and is said to be decreasing if E" :J E,,+I, n ;>
1. If {E", n 2= I} is an increasing sequence of events, then we define a new event, denoted by lim"_,,, E" by
OIl
limE,,= U E; when E" C E,,+I, n 2= 1. ,,_00 ,"'I
Similarly if {E", n 2= I} is a decreasing sequence, then define lim,,_oo E" by
OIl
limE" = n E" when E" :J E,,+I, n 2= 1. ,,_00
We may now state the following:
PROPOSITION 1.1.1
If {E", n 2: 1} is either an increasing or decreasing sequence of events, then
(
F" = E" Y E, t = E"E~_\, n>l
That is, F" consists of those points in E" that are not in any of the earlier E,. i < n It is easy to venfy that the Fn are mutually exclusive events such that
'" "" U F, = U E, and U F, = U E, for all n 2: 1. ,=\ ,=1 i-I ,=\
PROBABILITY ; 3
n
= lim PeEn), n~"
which proves the result when {En' n ;:::: I} is increasing If {En. n ;:::: I} is a decreasing sequence, then {E~, n ;:::: I} is an increasing se­
quence, hence,
But, as U~ F;~ = (n~ En)', we see that
1- P (0 En) = ~~"2l1 - PeEn)],
or, equivalently,
which proves the result
EXAMPLE 1..1.(A) Consider a population consisting of individuals able to produce offspring of the same kind. The number of individuals
4 PRELIMINARIES
initially present, denoted by Xo. is called the size of the zeroth generation All offspring of the zeroth generation constitute the first generation and their number is denoted by Xl In general, let Xn denote the size of the nth generation
Since Xn = 0 implies that Xn+l = 0, it follows that P{Xn = O} is increasing and thus limn_ oc P{Xn = O} exists What does it represent? To answer this use Proposition 1.1.1 as foHows:
= P{the population ever dies out}.
That is, the limiting probability that the nth generation is void of individuals is equal to the probability of eventual extinction of the population.
Proposition 1.1.1 can also be used to prove the Borel-Cantelli lemma.
PROPOSITION 1...1.2
The Borel-Cantelli Lemma Let EI> E2 , denote a sequence of events If
"" L: P(E,) < 00, 1=]
P{an infinite number of the E, occur} = 0
., .,
lim sup E, = n U E, ,-'"
This follows since if an infinite number of the E, occur, then U::n E, occurs for each n and thus n:=] U::n E, occurs On the other hand, if n:=t U::n E, occurs, then U::n E, occurs for each n, and thus for each n at least one of the E, occurs where i .2:
n, and, hence, an infinite number of the E, occur
PROBABILITY .' 5
As U':n E" n 2: 1, is a decreasing sequence of events, it follows from Proposition 1 1 1 that
P (Q ~ E,) = P (~~ Y. E,)
IX>
=0.
ExAMPlE 1.1.(a) Let Xl, X 2 , •• be such that
P{Xn = O} = lIn2 = 1 - P{Xn = 1}, n~1.
If we let En = {Xn = O}, then, as ~: peEn} < 00, it follows from the Borel-Cantelli lemma that the probability that Xn equals 0 for an infinite number of n is equal to O. Hence, for all n sufficiently large, Xn must equal 1, and so we may conclude that, with probabil­ ity 1,
lim Xn = 1.
For a converse to the Borel-Cantelli lemma, independence is required.
PROPOSITION 1.1.3
Converse to the Borel-Cantelli Lemma are independent events such that
IX>
6 PRELIMINARIES
P{an infinite number ofthe En occur} == P {~~~ td E,}
== limp(U Ei) n_oo ~!!!.n
= exp ( - ~ P(E,») '"
Hence the result follows.
ExAMPLE 1.1(c) Let XI, X 2 , ••• be independent and such that
P{Xn = O} = lin = 1 - P{Xn = I}, n~1
If we let E" = {Xn = O}, then as L:~I P(E,,) = 00 it follows from Proposition 1.1.3 that En occurs infinitely often. Also, as L:~I P(E~) = 00 it also follows that E~ also occurs infinitely often. Hence, with probability 1, Xn will equal 0 infinitely often and will also equal 1 infinitely often. Hence, with probability I, X" will not approach a lImiting value as n -4 00.
RANDOM VARIABLES .7
1.2 RANDOM VARIABLES
Consider a random experiment having sample space S A random variable X is a function that assigns a real value to each outcome in S. For any set of real numbers A, the probability that X will assume a value that is contained in the set A is equal to the probability that the outcome of the experiment is contained in X-I(A). That is,
P{X E A} = P(X-I(A»,
where X-I(A) is the event consisting of all points s E S such that X(s) EA. The distribution function F of the random variable X is defined for any
real number x by
F(x) = P{X> x}
A random variable X is said to be discrete if its set of possible values is countable For discrete random variables,
F(x) 2: P{X = y} YSJ:
A random variable is called continuous if there exists a functionf(x), called the probability density function, such that
P{X is in B} = fB f(x) dx
for every set B. Since F(x) J~ ... f(x) dx, it follows that
d f(x) = dx F(x).
The joint distribution function F of two random variables X and Y is de· fined by
F(x, y) = P{X ::$; x, Y::$; y}.
The distribution functions of X and Y,
Fx(x) P{X s; x} and Fy(y) = P{Y <: y},
8 PRELIMIN ARIES
can be obtained from F(x, y) by making use of the continuity property of the probability operator. Specifically, let Yn, n 2: 1, denote an increasing sequence converging to 00 Then as the events {X:s; x, Y:s; Yn}, n ::> 1, are increasing and
'" lim{X :5; x, Y:5; Yn} U {X<:x, YS:Yn} {X <:x}, 11-+" 11=1
it follows from the continuity property that
lim P{X :s; x, Y <: YII} = P{X:5; x}, n_oo
or, equivalently,
Similarly,
Fy(y) lim F(x, y). X_OIl
The random variables X and Yare said to be independent if
F(x, y) = Fx(x)Fy(y)
for all x and y. The random variables X and Yare said to be jointly continuous if there
exists a function t(x, y), called the joint probability density function, such that
P{Xis inA, Yis in B} L t t(x,y) dy dx
for all sets A and B. The joint distribution of any collection XI, Xl> ", XII of random variables
is defined by
Furthermore, the n random vanables are said to be independent if
where
,"r
EXPECTED VALUE 9
1.3 EXPECTED VALUE
The expectation or mean of the random variable X, denoted by EtX], is defined by
(1.3.1) E[X] = J:oo x dF(x)
{I~ .. xf(x) dx
= ~xP{X=x}
if X is continuous
if X is discrete
provided the above integral exists Equation (1.3 1) also defines the expectation of any function of X, say
h(X). Since h(X) is itself a random variable, it follows from (1 3 1) that
E[h(X)] == J: ... x dFh(x).
where Fh is the distribution function of h(X) However, it can be shown that this is identical to J: .. h(x) dF(x). That is,
(1.32) E[h(X)] = J: ... h(x) dF(x)
The variance of the random variable X is defined by
Var X = E[(X - E[X])2]
= E[X2] - E2[X]
Two jointly distributed random variables X and Yare said to be uncorre­ lated if their covariance, defined by
Cov(X, Y) = E[(X - EX)(Y - EY)]
= E[XY] - E[X]E[Y]
is zero. It follows that independent random variables are uncorrelated How­ ever, the converse need not be true. (The reader should think of an example)
An important property of expectations is that the expectation of a sum of random variables is equal to the sum of the expectations
(1.3.3)
(134) Var [t. XI] II
2: Var(X,) + 2 2:2: Cov(X,}X) 1=1
EXAMPLE 1.3{A) The Matching Problem. At a party n people put their hats in the center of a room where the hats are mixed together Each person then randomly selects one We are interested in the mean and variance of X~the number that select their own hat
To solve, we use the representation
where
if the ith person selects his or her own hat
otherwise
Now, as the ith person is equally likely to select any of the n…