1
Contents Introduction: ........................................................................................................................................... 6
Assessments: ....................................................................................................................................... 6
Schedule: ............................................................................................................................................. 6
Stochastic Processes: .............................................................................................................................. 7
Basic concenpts: .................................................................................................................................. 8
Definition: Stochastic process ......................................................................................................... 8
Definition: Random walk ................................................................................................................. 9
Definition: Strong stationary ........................................................................................................... 9
Definition: State space of stochastic process ................................................................................. 9
Definition: Increments .................................................................................................................... 9
Markov Property: .............................................................................................................................. 10
Definition: Markov property ......................................................................................................... 11
Definition: filtration ...................................................................................................................... 12
Definition: Markov Process ........................................................................................................... 12
Markov Chains ...................................................................................................................................... 12
Basic concepts: .................................................................................................................................. 13
Notation: ....................................................................................................................................... 13
Gambler ruin problem: ..................................................................................................................... 13
Solution ......................................................................................................................................... 14
Classification of states: ..................................................................................................................... 15
𝑛 step transition probability notation: ......................................................................................... 15
Recurrence and transience ............................................................................................................... 19
Recurrence definition: .................................................................................................................. 19
Transient definition: ...................................................................................................................... 19
Green Function: ............................................................................................................................ 19
Period: ........................................................................................................................................... 22
Limiting theorems and Stationarity of Markov Chains ......................................................................... 25
Expected number of transitions from 𝑗 to 𝑗...................................................................................... 25
Definition: ..................................................................................................................................... 25
Number of visits to 𝑗 by the 𝑛𝑡ℎ step: .......................................................................................... 25
Stationary Distributions .................................................................................................................... 29
Discussion: .................................................................................................................................... 29
Set up: ........................................................................................................................................... 30
Theorem: irreducible aperiodic Markov chains and classes: ........................................................ 30
Poisson Processes ................................................................................................................................. 34
2
Introduction: ..................................................................................................................................... 34
Theory from notes: ....................................................................................................................... 35
Revision of Distributions ................................................................................................................... 36
Poisson distribution ...................................................................................................................... 36
Exponential Distribution ............................................................................................................... 37
Definition of Poisson Process: ........................................................................................................... 39
Counting process (definition) ........................................................................................................ 39
Poisson Process Definition: ........................................................................................................... 40
Interarrival and waiting time distributions: ...................................................................................... 41
Distribution of 𝐸𝑘: Interarrival time ............................................................................................. 41
Distribution of 𝑇𝑘: Waiting time ................................................................................................... 42
Conditional Distribution of the Arrival Times ............................................................................... 43
Superposition and Thinning of Poisson Processes ............................................................................ 46
Theorem: Superposition position of Poisson Processes: .............................................................. 46
Sampling (thinning) ....................................................................................................................... 47
Revision: ................................................................................................................................................ 49
Random Sums (STAT2911) ................................................................................................................ 49
Examples: ...................................................................................................................................... 49
STAT3911 Random sums: .............................................................................................................. 51
Moment Generating Function .......................................................................................................... 52
Definition: ..................................................................................................................................... 52
Calculating MGF: ........................................................................................................................... 52
Moment Generating function of a random Sum .......................................................................... 56
Branching Processes: ............................................................................................................................ 57
Set up: ............................................................................................................................................... 58
Class .............................................................................................................................................. 58
Notes: ............................................................................................................................................ 59
Formulation of branching process: ................................................................................................... 59
Expectation and variance of 𝑍𝑛 + 1 ............................................................................................. 60
Probability of dying out: ................................................................................................................ 61
Brownian motion .............................................................................................................................. 68
Definition 1.................................................................................................................................... 68
Definition 2.................................................................................................................................... 69
Time Series: ........................................................................................................................................... 69
Basic concepts: .................................................................................................................................. 71
Regular time seris: ........................................................................................................................ 71
3
Notation for TS data: ..................................................................................................................... 71
Graphs of Time Series Data ........................................................................................................... 73
Basic Terminology of TS analysis ................................................................................................... 80
Analysis of Components of time series: ............................................................................................ 82
Estimation and elimination of trend in absence of seasonality: ................................................... 82
Estimation and elimination of both trend and seasonal components of a TS .............................. 89
Stationary Processes and Time Series 1: ............................................................................................... 91
Autocovariance and Autocorrelation Functions ............................................................................... 92
Definition: autocovariance function 𝛾 .......................................................................................... 92
Definition: Autocorrelation (acf) 𝜌 ............................................................................................... 93
Estimation of 𝛾𝑘 and 𝜌𝑘 ............................................................................................................... 93
Sampling Properties of 𝑋𝑛; 𝐶𝑛, 𝑘; 𝑅𝑛, 𝑘 ........................................................................................... 95
Sampling properties of 𝑋𝑛 ............................................................................................................ 95
Sampling properties of 𝐶𝑛, 𝑘 ........................................................................................................ 96
Sampling properties for 𝑅𝑛, 𝑘 ....................................................................................................... 96
Sample Correlogram ..................................................................................................................... 97
Detection of Randomness, short term and long term correlations of a TS .................................. 98
Autocorrelation Plot as Diagnostic Tool ..................................................................................... 101
Partial Autocorrelation Function (PACF) ..................................................................................... 103
Stationary Time Series: ............................................................................................................... 104
Some Stochastic Models for Time Series ............................................................................................ 104
White Noise Process ....................................................................................................................... 104
Definition: ................................................................................................................................... 104
Statistical Properties of WN 𝑍𝑡 ................................................................................................... 105
Linear Combination of 𝑍𝑡 ............................................................................................................ 105
Some useful time series Models: .................................................................................................... 108
Moving average (MA) process .................................................................................................... 108
Autocorrelation function, acf of 𝑀𝐴(𝑞) process ........................................................................ 111
Simulating MA process in R ......................................................................................................... 111
Useful Operations in Time Series .................................................................................................... 112
Backshift Operator (Lag operator) .............................................................................................. 112
Differencing Operator: ................................................................................................................ 112
Seasonal Differencing Operator .................................................................................................. 113
Inevitability of MA Processes .............................................................................................................. 114
Invertible solution: .......................................................................................................................... 115
Theorem: Invertible 𝑀𝐴(1) process ........................................................................................... 115
4
Invertability of general 𝑀𝐴(𝑞) process: ......................................................................................... 115
Theorem: invertibility of 𝑀𝐴(𝑞) process.................................................................................... 116
PACF of invertible MA process: ................................................................................................... 118
Autoregressive Processes and the Properties: ................................................................................... 119
Autoregressive (𝐴𝑅) Processes: ..................................................................................................... 120
Definition: Autoregressive process of order 𝑝 ............................................................................ 120
Analysis of an 𝐴𝑅(1) Process: .................................................................................................... 122
Analysis of AR(2) process (2nd order AR) ..................................................................................... 128
Autoregressive Processes of Order 𝑝 (AR(p)) ..................................................................................... 129
Theorems ........................................................................................................................................ 129
Theorem 1: .................................................................................................................................. 130
Theorem 2: .................................................................................................................................. 130
Yule Walker Equations for stationary AR processes ....................................................................... 130
Yule Walker Equation: ................................................................................................................. 130
PACF of stationary AR(p) process: .............................................................................................. 136
Mixed autoregressive moving average (ARMA) process: ................................................................... 139
Notation: ......................................................................................................................................... 139
Note: ........................................................................................................................................... 139
Stationarity and invertibility of ARMA(p,q) Process ....................................................................... 139
Theorems .................................................................................................................................... 140
Special cases of ARMA(𝑝, 𝑞) ....................................................................................................... 140
Moments of Xt ∼ ARMA(p, q) ....................................................................................................... 142
Mean 𝐸𝑋𝑡 ................................................................................................................................... 142
Autocovariance function: 𝛾𝑘 ...................................................................................................... 143
Homogeneous Nonstationary Processes ........................................................................................ 145
Example data:.............................................................................................................................. 145
Modelling homogeneous nonstationary time series: ................................................................. 147
Autoregressive Integrated moving average ARIMA(p,d,q) ......................................................... 150
Identification and estimation:......................................................................................................... 151
ARMA/ARIMA Models:................................................................................................................ 151
Hypothesis testing of orders or ARMA(p,a) and estimation ............................................................... 157
Identification: .................................................................................................................................. 158
1: Test whether the series is a white noise of 𝐻0:𝑋𝑡 ∼ 𝐴𝑅𝑀𝐴(0,0) ........................................ 158
2: Test 𝑋𝑡 ∼ 𝐴𝑅𝑀𝐴(0, 𝑞) or 𝑋𝑡 ∼ 𝑀𝐴(𝑞) ................................................................................. 159
3: Test 𝑋𝑡 ∼ 𝐴𝑅𝑀𝐴𝑝, 0; or 𝑋𝑡 ∼ 𝐴𝑅(𝑝) .................................................................................... 160
Parameter estimation of ARMA models: ........................................................................................ 162
5
1: MA(1) ...................................................................................................................................... 162
Estimation of Parameters continued: ......................................................................................... 163
Diagnostic Checking: ........................................................................................................................... 164
Residual analysis ............................................................................................................................. 164
6
STAT3911 STOCHASTIC PROCESSES AND
TIME SERIES (ADV) COURSE NOTES
STANDARD LECTURES Lecture 1. Monday, 6 March 2017
Introduction:
Lecturer: Ray Kawai week 1-7 (stochastic processes)
Shelton Reiris week 8-13 (time series)
Qiuing Wang (ADV) week 1-7 (Stochastic processes)
Assessments:
Quizes
- Wednesday week 7 common (April 26) (Stochastic processes)
- Friday advanced week 7 (April 28) (Stochastic processes advanced material)
- Friday week 12 (June 2) (Time series)
Computer exam: week 13
Schedule:
1. Review of probability theory. Elements of stochastic processes and time series.
2. Markov chains.
3. Markov chains.
4. Markov chains.
7
5. The Poisson process.
6. The Poisson process.
7. The Poisson process.
8. Time series data, components of a time series. Filtering to remove trends and seasonal
components.
9. Stationarity time series. Sample autocorrelations and partial autocorrelations. Probability
models for stationary time series. Moving Average (MA) models and properties.
10. Invertibility of MA models. Autoregressive (AR) models and their properties. Stationarity of
AR models. Mixed Autoregressive Moving Average (ARMA) models and their properties.
11. Homogeneous nonstationary time series (HNTS). Simple models for HNTS. Autoregressive
Integrated Moving Average (ARIMA) models and related results. Review of theoretical
patterns of ACF and PACF for AR, MA and ARMA processes. Identification of possible AR,
MA, ARMA and ARIMA models for a set of time series data.
12. Estimation and fitting ARIMA models via MM and MLE methods. Hypothesis testing,
diagnostic checking and goodness-of-fit tests. AIC for ARIMA models. Forecating methods for
ARIMA models.
13. Minimum mean square error (mmse) forecasting and its properties. Derivation of l-step
ahead mmse forecast function. Forecast updates. Forecast errors, related results and
applications.
Stochastic Processes:
In probability theory and related fields, a stochastic or random process is a mathematical
object usually defined as a collection of random variables. Historically, the random variables
were associated with or indexed by a set of numbers, usually viewed as points in time, giving the
interpretation of a stochastic process representing numerical values of some
system randomly changing over time, such as the growth of a bacterial population, an electrical
current fluctuating due to thermal noise, or the movement of a gas molecule.[1][4][5] Stochastic
processes are widely used as mathematical models of systems and phenomena that appear to
vary in a random manner. They have applications in many disciplines including sciences such
as biology,[6] chemistry,[7] ecology,[8] neuroscience,[9] and physics[10] as well
as technology and engineering fields such as image processing, signal processing,[11] information
theory,[12] computer science,[13] cryptography[14] and telecommunications.[15] Furthermore,
8
seemingly random changes in financial markets have motivated the extensive use of stochastic
processes in finance.[16][17][18]
Basic concenpts:
- Randomness, indexed by time
{𝑋𝑡: 𝑡 ∈ 𝕋}
Where 𝑡 ∈ [0, 𝑇] for some endtime 𝑡.
- Can be either discrete or continuous
State Time
Continuous Height,
temperature
Continuous
time
discrete Dice,
coinflip,
number of
people
1 day, 1
second, 1
year
Definition: Stochastic process
A stochastic process is a model of time-dependent random phenomena. A single random variable
describes a static random phenomena; a stochastic process is a collection of random variables
{𝑋𝑡: 𝑡 ∈ 𝕋}, one for each time 𝑡 ∈ 𝕋.
- Can be either discrete or continuous
9
Definition: Random walk
We have a time set 𝕋 and state space 𝑆. We defininte the stochastic process {𝑋𝑡: 𝑡 ∈ 𝕋}. In principle,
we need to know the joint distribution of (𝑋𝑡1 , 𝑋𝑡2 , … , 𝑋𝑡𝑛), where 𝑡𝑘 ∈ 𝕋, 𝑛 ∈ ℕ. This is difficult.
We define a random walk as a stochastic process {𝑋𝑡: 𝑡 ∈ 𝕋}, where 𝕋 = ℕ in such a way that
𝑋𝑡 = 𝑋𝑡−1 + 𝑍𝑡; (𝑡 ∈ 𝕋)
Where {𝑍𝑘}𝑘∈ℕ is a sequence of iid RV with ℙ(𝑍𝑘 = 1) = 𝑝, ℙ(𝑍𝑘 = −1) = 1 − 𝑝 for some 𝑝 ∈
(0,1).
The equation 𝑋𝑡 = 𝑋𝑡−1 + 𝑍𝑡 is an example of a difference equation. It is an implicit definition of 𝑋𝑡,
since it is only given in terms of 𝑋𝑡−1. In continuous time, this becomes the differential equation
Definition: Strong stationary
A SP {𝑋𝑡: 𝑡 ∈ 𝕋} is said to be strong stationary, if the two joint distributions of (𝑋𝑡1 , 𝑋𝑡2 , … , 𝑋𝑡𝑛) and
(𝑋𝑘+𝑡1 , 𝑋𝑘+𝑡2 , … , 𝑋𝑘+𝑡𝑛) are identical ∀𝑡1, … , 𝑡𝑛; 𝑘 + 𝑡1, … , 𝑘 + 𝑡𝑛 ∈ 𝕋
Definition: State space of stochastic process
The set of values that the 𝑋𝑡 ‘s can take is called the state space, 𝑆, of the stochastic process
Definition: Increments
An increment of a stochastic process is the amount by which its value changes over a period of time,
for example 𝑋𝑡𝑘+1 − 𝑋𝑡𝑘, where 𝑡𝑘 < 𝑡𝑘+1 ∈ 𝕋
Definition: Stationary increments
The SP {𝑋𝑡: 𝑡 ∈ 𝕋} is said to have stationary increments if the distribution of the increment depends
only on the difference between the two time points.
- If, for 𝑡1 ≤ 𝑡2 and 𝑡3 ≤ 𝑡4 ⟹ 𝑡2 − 𝑡1 = 𝑡4 − 𝑡3
𝑋𝑡2 − 𝑋𝑡1ℒ
=𝑋𝑡4 − 𝑋𝑡3
(the increments have the same distribution)
Eg: temperature between today and tomorrow is distributed the same
10
Example: stock prices
Let {𝑆𝑡: 𝑡 ∈ ℝ+} denote the price of one share of a specific stock. It might be considered reasonable
to assume that the distribution of the return over a period of duraction Δ > 0
𝑆𝑡+Δ − 𝑆𝑡𝑆𝑡
=𝑆𝑡+Δ𝑆𝑡
− 1
Depends on Δ but not 𝑡. Generally, we assume that {𝑆𝑡+Δ
𝑆𝑡: 𝑡 ∈ ℝ+} is a stationary stochastic process.
Accordingly, the log-price processes 𝑋 ≔ ln 𝑆𝑡 would have stationary increments 𝑋𝑡+Δ − 𝑋𝑡 =
ln (𝑆𝑡+Δ
𝑆𝑡), even though the stochastic process {𝑋𝑡: 𝑡 ∈ ℝ+} might not be stochastic. In other words,
for fixed Δ, the stochastic process 𝑌𝑡Δ ≔ 𝑋𝑡+Δ − 𝑋𝑡 is stationary.
Definition: Independent increments
A stochastic process {𝑋𝑡: 𝑡 ∈ 𝕋} has independent increments, if ∀𝑡 ∈ 𝕋, and Δ > 0|𝑡 + Δ ∈ 𝕋, the
increment 𝑋𝑡+Δ − 𝑋𝑡 is independent of all the past {𝑋𝑠: 𝑠 ∈ 𝕋} of the SP.
- The increments at some time period are independent of previous events.
o The first half of this course will assume stationary and independent increments
Markov Property:
In probability theory and related fields, a Markov process (or Markoff process), named after
the Russian mathematician Andrey Markov, is a stochastic process that satisfies the Markov
property[1][2] (sometimes characterized as "memorylessness"). Loosely speaking, a process
satisfies the Markov property if one can make predictions for the future of the process based
11
solely on its present state just as well as one could knowing the process's full history;
i.e., conditional on the present state of the system, its future and past states are independent.
A Markov chain is a type of Markov process that has either discrete state space or discrete
index set (often representing time), but the precise definition of a Markov chain varies.[3] For
example, it is common to define a Markov chain as a Markov process in either discrete or
continuous time with a countable state space (thus regardless of the nature of time)[4][5][6][7], but it is
also common to define a Markov chain as having discrete time in either countable or continuous
state space (thus regardless of the state space).[8]
- Future events only depend upon current time, and not previous events
Definition: Markov property
A SP {𝑋𝑡: 𝑡 ∈ 𝕋} is said to have a Markov Property if
𝑃(𝑋𝑡𝑚+1 ∈ 𝐴𝑚+1|𝑋𝑡1 ∈ 𝐴1, … , 𝑋𝑡𝑚 ∈ 𝐴𝑚) = 𝑃(𝑋𝑡𝑚+1 ∈ 𝐴𝑚+1|𝑋𝑡𝑚 ∈ 𝐴𝑚)
Where 0 ≤ 𝑡1 ≤ 𝑡2 ≤ ⋯ ≤ 𝑡𝑚 ≤ 𝑡𝑚+1 and {𝐴𝑘}(𝑘∈ℕ} is a sequence of measurable sets in 𝑆
Corollary: Independent increments have the Markov property
A SP {𝑋𝑡: 𝑡 ∈ 𝕋} with independent increments has the Markov property.
Example: State space of natural numbers
Eg, let 𝑆 = ℕ, then
𝑃(𝑋𝑡𝑚+1 = 𝑥𝑚+1|𝑋𝑡1 ∈ 𝐴1, … , 𝑋𝑡𝑚−1 ∈ 𝐴𝑚−1, 𝑋𝑡𝑚 = 𝑥𝑚)
= 𝑃(𝑋𝑡𝑚+1 − 𝑋𝑡𝑚 = 𝑥𝑚+1 − 𝑥𝑚|𝑋𝑡1 ∈ 𝐴1, … , 𝑋𝑡𝑚−1 ∈ 𝐴𝑚−1, 𝑋𝑡𝑚 = 𝑥𝑚)
= 𝑃(𝑋𝑡𝑚+1 − 𝑋𝑡𝑚 = 𝑥𝑚+1 − 𝑥𝑚|𝑋𝑡𝑚 = 𝑥𝑚) (𝑏𝑦 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡 𝑖𝑛𝑐𝑟𝑒𝑚𝑒𝑛𝑡𝑠)
= 𝑃(𝑋𝑡𝑚+1 = 𝑥𝑚+1|𝑋𝑡𝑚 = 𝑥𝑚)
Note, independent increments imply markov property, but not the reverse (as what if 𝑋𝑛+1 = 𝑋𝑛 +
𝜖𝑛, where 𝜖_𝑛|𝑥0, . . , 𝑥𝑛 ∼ 𝑁(−𝑋𝑛, 1) for example)?. This brings in the concept of filtration, where
we need to model the flow of public information.
Lecture 2. Tuesday, 7 March 2017
12
Definition: filtration
Let (Ω, ℱ) be a measureable space, and 𝕋 ⊆ [0,∞).
1. Assume ∀𝑡 ∈ 𝕋 ∃ 𝜎 −field, ℱ𝑡 ⊆ ℱ. Assume that, for 𝑠 ≤ 𝑡 ⟹ ℱ𝑠 ⊆ ℱ𝑡. We call the
collection of 𝜎 fields (𝐹𝑡)𝑡∈𝕋 a filtration.
2. A SP {𝑋𝑡: 𝑡 ∈ 𝕋} is said to be (ℱ)𝑡∈𝕋 adapted if, ∀𝑡 ∈ 𝕋, the RV 𝑋𝑡 is ℱ𝑡 − measurable.
Remark: filtration generated by a stochastic process and information
If the filtration (ℱ𝑡)𝑡∈𝕋 is generated by a stochastic process {𝑋𝑡: 𝑡 ∈ 𝕋}, then ∀𝑡 ∈ 𝕋
ℱ𝑡 = 𝜎(𝑋𝑠: 𝑠 ∈ 𝕋, 𝑠 ≤ 𝑡)
In this case, the 𝜎 −field ℱ𝑠 contains all the information of the SP up till time 𝑠.
- The concept of filtration can then generalise the definition of the Markov property
Definition: Markov Process
Let (Ω, ℱ, ℙ) be a probability space, and (ℱ𝑡)𝑡∈𝕋 be a filtration. A (ℱ𝑡)𝑡∈𝕋 adapted SP {𝑋𝑡: 𝑡 ∈ 𝕋} is
called a Markov process, if, ∀𝐵 ∈ 𝜎(𝑋𝑠: 𝑠 ≥ 𝑡)
ℙ(𝐵|ℱ𝑡) = ℙ(𝐵|𝑋𝑡)
(note that 𝐵 depends only on {𝑋𝑠: 𝑠 ≥ 𝑡}).
Lecture 3. Wednesday, 8 March 2017
Markov Chains
In probability theory and related fields, a Markov process (or Markoff process), named after
the Russian mathematician Andrey Markov, is a stochastic process that satisfies the Markov
property[1][2] (sometimes characterized as "memorylessness"). Loosely speaking, a process
satisfies the Markov property if one can make predictions for the future of the process based
solely on its present state just as well as one could knowing the process's full history;
i.e., conditional on the present state of the system, its future and past states are independent.
A Markov chain is a type of Markov process that has either discrete state space or discrete
index set (often representing time), but the precise definition of a Markov chain varies.[3] For
example, it is common to define a Markov chain as a Markov process in either discrete or
13
continuous time with a countable state space (thus regardless of the nature of time)[4][5][6][7], but it is
also common to define a Markov chain as having discrete time in either countable or continuous
state space (thus regardless of the state space).[8]
- Discrete time and discrete space problem
o After we will look at continuous time and discrete space (Poisson processes)
- That is to say, they are indexed by 𝑡 ∈ 𝕋 = ℕ = {0,1,2,… }
Basic concepts: Independence of increments are replaced by the Markov property
- That the future and the past are independent given the present
Notation: Given 𝑛, 𝑟, 𝑘0, … , 𝑘𝑛+𝑟, with 𝑃(𝑋𝑛 = 𝑘𝑛) > 0 :
- we write the past event as
{𝑋0 = 𝑘0, 𝑋1 = 𝑘1, … , 𝑋𝑛−1 = 𝑘𝑛−1} ≔ 𝐴
- And all future events as
{𝑋𝑛+1 = 𝑘𝑛+1, 𝑋𝑛+2 = 𝑘𝑛+2, … , 𝑋𝑛+𝑟 = 𝑘𝑛+1} ≔ 𝐵
We then get the conditional probability that:
𝑃(𝐴 ∩ 𝐵|𝑋𝑛 = 𝑘𝑛) = 𝑃(𝐴|𝑋𝑛 = 𝑘𝑛)𝑃(𝐵|𝑋𝑛 = 𝑘𝑛)
- This is equivalent that ∀𝑛 ∈ ℕ, 𝑘0, 𝑘1, … , 𝑘𝑛+1 with 𝑃(𝑋0 = 𝑘0, 𝑋1 = 𝑘1, … , 𝑋𝑛−1 = 𝑘𝑛−1) >
0 that:
𝑃(𝑋𝑛+1 = 𝑘𝑛+1|𝑋0 = 𝑘0, … , 𝑋𝑛 = 𝑘𝑛) = 𝑃(𝑋𝑛+1 = 𝑘𝑛+1|𝑋𝑛 = 𝑘𝑛)
(by the Markov property)
Transiontion probability
The right hand side of the above equation is known as the transition probability. It does not depend
on time, but only on the states 𝑘𝑛 and 𝑘𝑛+1. We write that
𝑃(𝑋𝑛+1 = 𝑗 |𝑋𝑛 = 𝑘) ≔ 𝑝𝑘,𝑗
Gambler ruin problem:
- Start with an amount of money $𝑛, with probability 𝑝 you win $1, and 𝑞 = 1 − 𝑝 you lose
$1. You stop after you either:
o Lose all money
o Win up to a certain amount $𝑁
14
- What is the probability that we will succeed with $𝑁 without losing everything?
Solution - His fortune is a MC on {0,1,… ,𝑁} ∈ ℕ.
Definite: 𝐴𝑛 ={eventual success starting from $𝑛}; and 𝑎𝑛 = 𝑃(𝐴𝑛) [the eventual success when
starting from initial state 𝑛].
- We have that 𝑎0 = 0, 𝑎𝑁 = 1
𝑃(𝐴𝑛) = 𝑃(𝐴𝑛|{𝑤𝑖𝑛 𝑎𝑡 𝑛})𝑃({𝑤𝑖𝑛 𝑎𝑡 𝑛}) + 𝑃(𝐴𝑛|{𝑙𝑜𝑠𝑒 𝑎𝑡 𝑛})𝑃({𝑙𝑜𝑠𝑒 𝑎𝑡 𝑛})
= 𝑃(𝐴𝑛|{𝑤𝑖𝑛 𝑎𝑡 𝑛})𝑝 + 𝑃(𝐴𝑛|𝑙𝑜𝑠𝑒 𝑎𝑡 𝑛}) (1 − 𝑝)
= 𝑃(𝐴𝑛+1|{𝑤𝑖𝑛 𝑎𝑡 𝑛})𝑝 + 𝑃(𝐴𝑛−1|{𝑙𝑜𝑠𝑡 𝑎𝑡 𝑛})(1 − 𝑝)
(as, if we win at 𝑛, we go to game 𝑛 + 1, and if we lose at 𝑛 we go to game 𝑛 − 1)
= 𝑃(𝐴𝑛+1)𝑝 + 𝑃(𝐴𝑛−1)(1 − 𝑝)
(using the markov property)
∴ 𝑎𝑛 = 𝑎𝑛+1𝑝 + 𝑎𝑛−1𝑞
Which is difference equation, with 𝑃(𝐴0) = 𝑎0 = 0; 𝑃(𝐴𝑛) = 𝑎𝑛 = 1
The solution to this is:
𝑁
𝑛
0
𝑝
1 − 𝑝
𝑛 + 1
𝑛 − 1
$ (discrete)
Time (discrete)
15
𝑎𝑛 =
{
1 − (
1 − 𝑝𝑝 )
𝑛
1 − (1 − 𝑝𝑝
)𝑁 ; 𝑖𝑓 𝑝 ≠
1
2
𝑛
𝑁; 𝑖𝑓 𝑝 =
1
2
- Note that this expression is continuous in 𝑝; and that the 𝑛
𝑁 expression can be derived based
on the one for 𝑝 ≠1
2 using the asymptotic behavious 1 − (
𝑞
𝑝)𝑛= 4𝑛 (𝑝 −
1
2) +
𝑂 ((𝑝 −1
2)2) as 𝑝 →
1
2.
We get the same solution if we let 𝛽𝑛 be the probability of eventual ruin when starting at 𝑛, finding
that 𝛽𝑛 = 𝑝𝛽𝑛+1 + 𝑞𝛽𝑛−1, with 𝛽0 = 1 and 𝛽𝑁 = 0. This means that the gambler must either
succed or be ruined, and the gambler will not be able to converge to a steady state of some other
amount of money.
Also; observe that in the limit:
lim𝑁↑+∞
𝑎𝑛 =
{
1 − (𝑞
𝑝)𝑛
[> 0]; 𝑖𝑓 𝑝 >1
2
0; 𝑝 ≤1
2
So, taking 𝑁 ↑ +∞ means that in the limit, the gambler will only ever stop if ruined. In this situation,
if each gamble is in the player’s favour (𝑝 >1
2), then there is a positive probability that the gambler
will never get ruined, but instead become infinitely rich. If each gamble is out of favour, (𝑝 <1
2),
then the gambler will get ruined almost surely.
Classification of states:
𝑛 step transition probability notation: Let
𝑝𝑘0,𝑘𝑛(𝑛) ≔ ℙ𝑘0(𝑋𝑛 = 𝑘𝑛) = ℙ(𝑋𝑛 = 𝑘𝑛|𝑋0 = 𝑘0)
- The probability that we are at state 𝑘𝑛, given that we started at 𝑘0 𝑛 steps ago.
- For ease of notation, 𝑝𝑖,𝑗(1) = 𝑝𝑖,𝑗
16
Additional 𝑚 steps
- Where are we after an additional amount of steps?
𝑝𝑖,𝑗(𝑛+𝑚) = 𝑃(𝑋𝑛+𝑚 = 𝑗|𝑋0 = 𝑖)
=∑𝑃(𝑋𝑛+𝑚 = 𝑗, 𝑋𝑛 = 𝑘 |𝑋0 = 𝑖) [𝑖. 𝑒. −𝑔𝑜𝑖𝑛𝑔 𝑡ℎ𝑟𝑜𝑢𝑔ℎ 𝑠𝑡𝑎𝑡𝑒 𝑘]
=∑𝑃(𝑋𝑛+𝑚 = 𝑗|𝑋𝑛 = 𝑘, 𝑋𝑜 = 𝑖)𝑃(𝑋𝑛 = 𝑘|𝑋0 = 𝑖)
𝑘
[𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛𝑎𝑙 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦]
=∑𝑃(𝑋𝑛+𝑚|𝑋𝑛 = 𝑘)𝑃(𝑋𝑛 = 𝑘|𝑋𝑜 = 𝑖)
𝑘
[𝑏𝑦 𝑡ℎ𝑒 𝑀𝑎𝑟𝑘𝑜𝑣 𝑝𝑟𝑜𝑝𝑒𝑟𝑡𝑦]
=∑𝑝𝑘,𝑗(𝑚)𝑝𝑖,𝑘
(𝑛)
𝑘
[𝑢𝑠𝑖𝑛𝑔 𝑛𝑜𝑡𝑎𝑡𝑖𝑜𝑛]
𝑝𝑖,𝑗(𝑛+𝑚) =∑𝑝𝑘,𝑗
(𝑛)𝑝𝑖,𝑘(𝑚)
𝑘
- Which looks like the definition of matrix multiplication‼
𝑛 step transitional probability matrix
The above equation: 𝑝𝑖,𝑗(𝑛+𝑚) = ∑ 𝑝𝑘,𝑗
(𝑛)𝑝𝑖,𝑘(𝑚)
𝑘 is called the Chapman-Kolmogorov equation, and can be
written as the matrix equation:
𝑃(𝑛+𝑚) = 𝑃(𝑛)𝑃(𝑚)
- From which we see that
𝑖
𝑘
𝑗
𝑛 𝑛 +𝑚
17
𝑃(𝑛) = (𝑃(1))𝑛
In the matrix:
𝑜𝑟𝑖𝑔𝑖𝑛
𝐷𝑒𝑠𝑖𝑛𝑎𝑡𝑖𝑜𝑛
( )
Accessibility:
We say that state 𝑗 is accessible from state 𝑖, ∃ some number of steps 𝑛 ∈ ℕ+ |𝑝𝑖,𝑗(𝑛)
> 0
- i.e.: it is possible to get to state 𝑗 from state 𝑖 after some amount of states (as probability >
0)
- can write as 𝑖 → 𝑗
Communicate
If 𝑖 and 𝑗 are accessible from one another, they are said to communicate, written as 𝑖 ↔ 𝑗
Properties of communicating states:
1. reflexivity: 𝑖 ↔ 𝑖
2. symmetry: 𝑖 ↔ 𝑗 ⟺ 𝑗 ↔ 𝑖
3. transitivity: if 𝑖 ↔ 𝑗, and 𝑗 ↔ 𝑘; then 𝑖 ↔ 𝑘
Communicating classes:
Communicating states can be partitioned into communicating classes:
- largest set A of states such that all 𝑖, 𝑗 ∈ 𝐴 communicate.
- all states in a communicating class communicate with one another
o in the gamblers ruin problem; there are 3 communicating classes:
{𝑁}
{0}
{1,2,… ,𝑁 − 1}
Closed communicating classes
We say a communicating class 𝐶 is closed if no state outside of 𝐶 can be reached from any state in
𝐶.
i.e:
𝑝𝑖,𝑗 = 0 for 𝑖 ∈ 𝐶 and 𝑗 ∉ 𝐶
Irreducible:
If the markov chain consists of only 1 communicating class, then the MC is said to be irreducible.
Absorbing state
A state is said to be absorbing, if we cannot go anywhere after it.
- i.e; the set {𝑗} is a closed class, that is 𝑝𝑗,𝑗 = 1
18
o in the gambler’s guin problem: 𝑝0,0 = 1 and 𝑝𝑁,𝑁 = 1
Lecture 4. Monday, 13 March 2017
Example: Markov Chain
Consider the MC with states {1,2,3}
𝑃 =
(
1
2
1
20
1
2
1
4
1
4
01
3
2
3)
Eg: (probability of 1 → 3 is 0)
Accessible states:1,1; 1,2; 2,1; 2,2; 2,3; 3,2; 3,3
1 ↔ 2 ↔ 3
{1,2,3} is a communicating class
Example 2: markov chain
4 states with
𝑃 =
(
1
2
1
20 0
1
2
1
20 0
1
4
1
4
1
4
1
40 0 0 1)
4 is absorbing sate
1 and 2 communicate
1,2,3,4 is accessible from 3.
- This is not irreducible (3 communicating classes)
Example 3: weather
Probability of fine or rain (0,1). Chances of rain tomorrow depends on today’s conditions. If it is fine
today, probability of fine tomorrow is 0.7. if it is rainy today, probability of fine tomorrow is 0.4
𝑃 = (0.7 0.3. 4 0.6
)
- This is irreducible
3 step transition:
𝑃3 = (0.7 0.3. 4 0.6
)3
= (0.583 . 417. 556 . 441
)
19
Ie: 𝑝0,1(3)= 0.417
Recurrence and transience
Let 𝑓𝑖,𝑗(𝑛)
denote the probability that the first transition into 𝑗 takes place at time 𝑛, when the chain
starts at state 𝑖
𝑓𝑖,𝑗(𝑛) ≔ ℙ({𝑋𝑛 = 𝑗} ∩ {𝑋𝑛−1, 𝑋𝑁−2, … , 𝑋1 ≠ 𝑗})
- i.e. the “first” probability from 𝑖 → 𝑗 after 𝑛 steps, without having vistited 𝑗 in between.
- If 𝑓𝑖,𝑗(0)= 0 if 𝑖 ≠ 𝑗 and 𝑓𝑖,𝑗
(0)= 1 if 𝑖 = 𝑗. Then the quantity becomes:
𝑓𝑖,𝑗 ≔∑𝑓𝑖,𝑗(𝑛)
+∞
𝑛=1
= ℙ(𝑋𝑛 = 𝑗 for some 𝑛 ≥ 1|𝑋0 = 1)
Recurrence definition: Indicates the probability of ever making a transition into state 𝑗 when the chain starts at 𝑖. We call
state 𝑗 recurrent if 𝑓𝑗,𝑗 = 1 (i.e.: starting at 𝑗, the chain will almost surely return to itself in a finite
number of steps).
Transient definition: A non recurrent state is said to be transient.
- For example, if 𝑓𝑗,𝑗 < 1, then state 𝑗 is transient
- Not sure that we will return
Green Function: The Green function of the MC is the expected number of visits to 𝑗 for the chain starting at 𝑖.
𝐺(𝑖, 𝑗) ≔ ∑𝑝𝑖,𝑗(𝑛)
+∞
𝑛=0
= ∑𝔼(𝕀(𝑋𝑛 = 𝑗)|𝑋0 = 𝑖)
+∞
𝑛=0
= 𝔼∑(𝕀(𝑋𝑛 = 𝑗)|𝑋0 = 𝑖)
+∞
𝑛=0
- Is the expected number of visits to 𝑗 starting from state 𝑖.
Transience and green function:
State 𝑗 is transient iff 𝐺(𝑖, 𝑗) < ∞ (i.e. the expected number of moves before returning is finite)
𝑡𝑟𝑎𝑛𝑠𝑖𝑒𝑛𝑐𝑒 ⟺ 𝐺(𝑗, 𝑗) < ∞
20
Proof:
After visiting 𝑗, the probability of returning is 𝑓𝑗,𝑗 after some time. This happens every time we return
to state 𝑗. As we are counting the number of successes of visits to 𝑗, this means the distribution is
geometric; with 𝑝 = 𝑓𝑗,𝑗. ∼ 𝐺𝑒𝑜𝑚𝑒𝑡𝑟𝑖𝑐(𝑓𝑗,𝑗)
Mean of geometric is 𝑝
1−𝑝
𝐸[𝑋] =𝑓𝑗,𝑗
1 − 𝑓𝑗,𝑗
But we already know that 𝐺(𝑗, 𝑗) = 𝐸(𝑋), so
𝑓𝑘,𝑘 < 1 iff 𝐺(𝑗, 𝑗) < ∞
Lecture 5. Tuesday, 14 March 2017
Recurrence of communicating states
If 𝑖 ↔ 𝑗 and if state 𝑖 is recurrent, then state 𝑗 is recurrent.
Proof:
As 𝑖 ↔ 𝑗∃𝑚, 𝑛|𝑝𝑖,𝑗𝑚 , 𝑝𝑖,𝑗
𝑛 > 0. We fix such 𝑚, 𝑛, it then holds that ∀𝑠 ≥ 0
𝑝𝑗,𝑗𝑛+𝑠+𝑚 ≥ 𝑝𝑗,𝑖
𝑛 𝑝𝑖 , 𝑖𝑠𝑝𝑖, 𝑗
𝑚
Due to the chapman kolmorgov equation when:
∑𝑝𝑗,𝑗𝑛+𝑠+𝑚
𝑠
≥∑𝑝𝑗,𝑖𝑛 𝑝𝑖,𝑖
𝑠 𝑝𝑖,𝑗𝑚
𝑠
= 𝑝𝑗,𝑖𝑛 𝑝𝑖,𝑗
𝑚∑𝑝𝑖,𝑖𝑠
𝑠
= 𝑝𝑗,𝑖𝑛 𝑝𝑖,𝑗
𝑚𝐺(𝑖, 𝑖) = ∞
Recurrence and accessibility
If state 𝑖 is recurrent and state 𝑗 is accessible from 𝑖, then 𝑓𝑖,𝑗 = 1 and 𝑖 ↔ 𝑗
Proof:
Let 𝑋0 = 𝑖. As 𝑗 is accessible from 𝑖∃𝑛|𝑝𝑖,(𝑛) > 0. Fix such 𝑛, we define the following:
𝐴0 = {𝑋𝑛 = 𝑗}; 𝑇1 = min{𝑘 ≥ 𝑛:𝑋𝑘 = 𝑖}
𝐴1 = {𝑋𝑇1+𝑛 = 𝑗}; 𝑇2 = min{𝑘 ≥ 𝑇1 + 𝑛:𝑋𝑘 = 𝑖}
…
𝐴𝑟 = {𝑋𝑇𝑟+𝑛 = 𝑗}; 𝑇𝑟+1 = min{𝑘 ≥ 𝑇𝑟 + 𝑛: 𝑋𝑘 = 𝑖}
time
j j
21
Since 𝑖 is recurrent; 𝑇𝑘 are finite. Then, the MC {𝐴𝑛} are independent and have th 𝑝𝑖,𝑗𝑛 so one of
them occurs.
Remark:
- We have used the Strong Markov Property to show this. Which always holds for discrete
time markov chains, and usually for continuous ones. It says that: if 𝑁 is the stopping time of
a MC {𝑋𝑛}𝑛∈ℕ+ and we write 𝜓(𝑖, 𝐵) = 𝑃𝑖((𝑋0, 𝑋1, … ) ∈ 𝐵)
The Strong markov property says that:
𝑃((𝑋𝑁 , 𝑋𝑁+1, … ) ∈ 𝐵|𝑋0, 𝑋1, … , 𝑋𝑁) = 𝜓(𝑋𝑁 , 𝐵)
Example:
Consider the 4 staes with probability:
𝑃 =
(
0 0
1
2
1
21 0 0 00 1 0 00 0 0 0)
It is easy to see that all states communicate: 1 → 3 → 2 → 1 → 4 → 2 → 1
- All states must be recurrent (using the above property)
Example 2:
𝑃 =
(
1
2
1
20 0 0
1
2
1
20 0 0
0 01
2
1
20
0 01
2
1
20
1
4
1
40 0
1
2)
The chain consists of 3 classes: {1,2}, {3,4}, {5}.
22
The first two classes are recurrent, if the chin starts at 1 it will come back to 1 sometime a.s. however;
if it starts at 5, it may never come back, as if it goes to state {1,2} it can never return.
Period:
Definition
The period of state 𝑘 is the greatest common divisor of the number of steps to come back to 𝑖,
starting from 𝑖 {𝑛 ∈ ℕ: 𝑝𝑘,𝑘(𝑛) > 0}, often written as
𝑑(𝑘)
If 𝑑(𝑘) = 1 then the state is called aperiodic.
Lecture 6. Wednesday, 15 March 2017
Remark on periodicity:
The minimum number of steps required to return is purely irrelevant to the concept of periodicity,
eg: consider
𝑃 = (0
1
20
0 0 11 0 0
)
Observe that: starting from state 2; the chain returns to state 2 after 3,5,6 steps. The minimum
number of steps required to return to state 2 startin at 2 is 3. Nevertheless, the greatest common
divisor is 1, so the periodicity is 1.
Communicating states and periodicity
If 𝑖 ↔ 𝑗 then 𝑑(𝑖) = 𝑑(𝑗)
Proof:
If 𝑖 = 𝑗, result is trivial. Suppose 𝑖 ≠ 𝑗. We show that𝑖 ↔ 𝑗 means 𝑑(𝑖) divides 𝑑(𝑗). Find a positive
interer 𝑠, such that 𝑝𝑖,𝑖𝑠 > 0. By definition, this integer divides 𝑑(𝑖). Moreover, there exists positive
interges 𝑚, 𝑛|𝑝𝑖,𝑗𝑛 , 𝑝𝑗,𝑖
𝑛 > 0. Then we have that:
𝑝𝑗,𝑗𝑛+𝑚 ≥ 𝑝𝑗,𝑖
𝑛 𝑝𝑖,𝑗𝑚 > 0
So (𝑛 + 𝑚) divides 𝑑(𝑗). By visiting 𝑖 in the middle, we have:
𝑝𝑗,𝑗𝑛+𝑠+𝑚 ≥ 𝑝𝑗,𝑖
𝑛 𝑝𝑖,𝑖𝑠 𝑝𝑖,ℎ
𝑚 > 0
So (𝑛 + 𝑠 +𝑚) divides 𝑑(𝑗). ∴ 𝑠 divides 𝑑(𝑗) and 𝑑(𝑖). Meaning that 𝑑(𝑖) = 𝑑(𝑗) as they are the
greatest common divisor
23
Example: Random walk:
Consider the MC on ℤ such that, for a given 𝑝 ∈ [0,1]; 𝑘 ∈ ℤ
𝑝𝑘,𝑘+1 = 𝑝; 𝑝𝑘,𝑘−1 = 1 − 𝑝
(example of a MC with period 2).
We show that, for 𝑝 =1
2 it is recurrent. First, observe that the n step transition probability is
binomial
𝑝0,0(2𝑛)
= (2𝑛𝑛) (1
2)𝑛
(1
2)𝑛
Which we approximiate (using Stirlings approximation), as
𝑝0,0(2𝑛)
∼1
√𝜋𝑛
Hence, 𝐺(0,0) = ∞.
- One of a famous theorems in probability extends to higher dimensions (Polya theorem),
which says symmetric random walks on ℤ𝑘 are transient iff 𝑘 ≥ 3
- Computing the probability of this for a 1D random walk will ever return if 𝑝 ≠1
2: WLOF,
suppose we start at 0. Let 𝑌𝑘 be the 𝑘𝑡ℎ step of the walk, {𝑌𝑘}𝑘∈ℕ is a sequence of iid RV
with
𝑌𝑘 = {+1; 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑝
−1; 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 1 − 𝑝
𝑋𝑛 denotes the position of the walk after 𝑛 steps, with 𝑋0 = 0. So 𝑋𝑛 = ∑ 𝑌𝑘𝑛𝑘−1 .
Considering the first transition,
𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0)
= 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = 1)𝑃(𝑌1 = 1)
+ 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = −1)𝑃(𝑌1 = −1)
= 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = 1)𝑝 + 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = −1)(1 − 𝑝)
If 𝑝 >1
2, (walk tends in positive direction), observe by the law of large numbers that as 𝑛 ↑
+∞
1
𝑛∑𝑌𝑘
𝑛
𝑘=1
→ 𝐸(𝑌1) = 2𝑝 − 1 (> 0)𝑎. 𝑠.
Which implies that ∑ 𝑌𝑘𝑛𝑘−1 tends to infinity almost surely. If we investigate the transitional
probability 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1) conditioning on the second transition, we get
24
= 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1,𝑌2 = −1)𝑃(𝑌2 = −1)
+ 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1,𝑌2 = +1)𝑃(𝑌2 = +1)
= 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = 1, 𝑌1 = −1)(1 − 𝑝) + 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = 1, 𝑌2 = 1)𝑝
= 1 − 𝑝 + 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1,𝑌2 = +1)𝑝
= 1 − 𝑝 + 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑋0 = 0, 𝑋1 = 1, 𝑋2 = 2)𝑝
Which holds as the walk restrats after 2 steps. If the walk is at state 𝑋2 = 2, in order to ever return
to state 0, we must first ever enter state 1. The probability that the walk ever enters state 1 starting
from 2 is identical to 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛 𝑡𝑜 0|𝑋1 = 1). Similary, the probability that the walk ever
enters state 0 from 1 is 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛 𝑡𝑜 0|𝑋1 = 1). And so:
𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑋2 = 2)
= 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 1 𝑠𝑡𝑎𝑟𝑡𝑖𝑛𝑔 𝑓𝑟𝑜𝑚 2)𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 1|𝑠𝑡𝑎𝑟𝑡𝑖𝑛𝑔 𝑓𝑟𝑜𝑚 2)
+ 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑛𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 1 𝑠𝑡𝑎𝑟𝑡𝑖𝑛𝑔 𝑓𝑟𝑜𝑚 2)𝑃(𝑛𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 1|𝑠𝑡𝑎𝑟𝑡𝑖𝑛𝑔 𝑓𝑟𝑜𝑚 2)
= 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑠𝑡𝑎𝑟𝑡 𝑓𝑟𝑜𝑚 1)𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 1|𝑠𝑡𝑎𝑟𝑡 𝑓𝑟𝑜𝑚 2)
= 𝑃(𝑒𝑣𝑒𝑟 𝑒𝑛𝑡𝑒𝑟 0|𝑠𝑡𝑎𝑟𝑡 𝑓𝑟𝑜𝑚 1)2
We get that: 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1) = 1 − 𝑝 + 𝑝𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1)2
Givins us:
𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1) = 1 𝑜𝑟1 − 𝑝
𝑝
The probability =1 is impossible, as we know by transience that it is strictly less than 1. Giving us that
𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛|𝑋0 = 0, 𝑌1 = +1) =1 − 𝑝
𝑝
∴ 𝑃(𝐸𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛) =2
1 − 𝑝
For 𝑝 <1
2, we get 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛_ = 2𝑝
Generally thr probability of returning in a Random Walk is:
𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛) = 2min{𝑝, 1 − 𝑝}
For example, if 𝑝𝑘,𝑘+1 =2
3; 𝑝𝑘,𝑘−1 =
1
3, starting at 0, the walk will come back to 0 again with
probability 2
3. (note the above result inclues 𝑝 =
1
2→ 𝑃(𝑒𝑣𝑒𝑟 𝑟𝑒𝑡𝑢𝑟𝑛) = 1