Lecture 2: ARMA(p,q) models (part 1)frapetti/CorsoP/chapitre_21_IMEA_1.pdf · Lecture 2: ARMA(p,q) models (part 1) Florian Pelgrin University of Lausanne, Ecole des HEC Department

Lecture 2: ARMA(p,q) models(part 1)

Florian Pelgrin

University of Lausanne, Ecole des HECDepartment of mathematics (IMEA-Nice)

Sept. 2011 - Jan. 2012

Florian Pelgrin (HEC) Univariate time series Sept. 2011 - Jan. 2012 1 / 72

Introduction

Motivation

Characterize the main properties of AR(p) models.

Estimation of AR(p) models


Introduction

Road map

1 Introduction

2 AR(1) model

3 Application with an AR(1) model

4 Autoregressive model of order p, AR(p)

5 Appendix


AR(1) model

Autoregressive models2. Autoregressive model of order 1, AR(1)

Definition

A stochastic process (Xt)t∈Z is said to be an autoregressive process oforder 1 if it satisfies the following equation :

Xt = µ+ φXt−1 + εt ∀t

where φ 6= 0, µ is a constant term, (εt)t∈Z is a weak white noise processwith expectation zero and variance σ2

ε (εt ∼WN(0, σ2ε )).


AR(1) model

0 100 200 300-4

-2

0

2

4Simulation of an AR(1) with phi=-0.5

0 100 200 300-4

-2

0

2

4Simulation of an AR(1) with phi=0

0 100 200 300-4

-2

0

2

4Simulation of an AR(1) with phi=0.5

0 100 200 300-10

-5

0

5

10Simulation of an AR(1) with phi=0.9


AR(1) model

Remarks :

1. In lag notation, one has :

Φ(L)Xt ≡ (1− φL)Xt = µ+ εt

2. The previous process can be written in mean-deviation as follows :

Xt = φXt−1 + εt

where

Xt = Xt −µ

1− φ

since

Xt −µ

1− φ= φ

(Xt−1 −

µ

1− φ

)+ εt ⇔ Xt = µ+ φXt−1 + εt


AR(1) model

Remarks (cont’d) :

3. For each date, one has an equation relating the value of X for thatdate to its previous value and the current value of ε. Suppose thatX−1 is known, one gets :

X0 = µ+ φX−1 + ε0

X1 = µ+ φX0 + ε1

X2 = µ+ φX1 + ε2

...

Xt = µ+ φXt−1 + εt .

4. Such a specification is a stochastic difference equation of order one. itcan be solved by recursive substitution.


AR(1) model


5. Solution by recursive substitution yields :

Xt = µ+ φXt−1 + εt

= µ+ φ (µ+ φXt−2 + εt−1) + εt = (1 + φ)µ+ φ2Xt−2 + εt + φεt−1

= (1 + φ)µ+ φ2 (µ+ φXt−3 + εt−2) + εt + φεt−1

= (1 + φ+ φ2)µ+ φ2Xt−3 + εt + φεt−1 + φ2εt−2

...

= (1 + φ+ · · · + φt)µ+ φt+1X−1 + εt + φεt−1 + · · · + φtε0

=1 − φt+1

1 − φµ+ φt+1X−1 +

t∑k=0

φkεt−k

=1 − φt+1

1 − φµ+ φt+1X−1 +

t∑k=0

ψkεt−k

where ψk = φk .


AR(1) model


6. Alternatively, solving forward h periods from time t (omitting the constant term) :

Yt+h = φh+1Xt−1 +h∑

k=0

ψkεt+h−k .

The dynamic multiplier h-period ahead is :

dXt+h

dεt= φh = ψh.

The impulse-response function is obtained by plotting ψh versus h.The cumulative impact (up to horizon h) of a shock is :

h∑k=1

ψk

The long-run cumulative impact is :

∞∑k=1

ψk


AR(1) model

Stationarity and stability conditions

If |φ| < 1, then :

limk→∞

φk = limk→∞

φk = 0

and the stable and weak stationary solution (Wold’s representation)for the AR(1) becomes (the limiting behavior of the recursive solutionof the first-order stochastic difference equation) :

Xt =µ

1− φ+∞∑k=0

φkεt−k

and the long-run cumulative impact is 11−φ .

If |φ| = 1, then Xt is a random walk with drift and is not stationary.

If |φ| > 1, there exists a non-causal stationary solution that we ruleout.


AR(1) model

Stability conditions for AR(1) models

-20

-15

-10

-5

0

5

10

15

20

25

30

1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 321 341 361 381 401 421 441 461 481 501

-140000

-120000

-100000

-80000

-60000

-40000

-20000

0

Stable AR(1) Random walk Unstable AR(1) (right axis)


AR(1) model

Alternatively, if |φ| < 1, then (see Block 2, part I) :

(1− φL)−1 =∞∑k=0

φkLk

and

Xt = (1− φL)−1 (µ+ εt)

=µ

1− φ+∞∑k=0

φkεt−k

1 This is the infinite moving average representation of an AR(1) process.2 The AR(1) representation is then called the fundamental or causal

representation.3 (εt) can be then interpreted as the innovation process, i.e. the

unpredictable part of Xt given the information available at time t − 1,It−1 = Xt−1,Xt−2, · · · , .


AR(1) model

More formally...

Definition

The representation of the autoregressive process of order one defined by :

Xt = µ+ φXt−1 + εt ,

is said to be causal or fundamental—(εt) is the innovation process—if theroot of the characteristic equation Φ(λ) ≡ 1− φλ = 0 lies outside the unitcircle :

|λ| > 1⇔ |φ| < 1.

In particular,

Cov (Xt−j , εt−`) = 0 for all j < `.


AR(1) model

Remark :

One can also use the reverse characteristic equation to find thestability and stationary conditions :

zΦ

(1

z

)= 0⇔ z − φ = 0.

The condition writes (for an AR(1) process) :

|z | < 1⇔ |φ| < 1.


AR(1) model

Moments of stationary AR(1)

Definition

Suppose that the stationary process (Xt) satisfies an AR(1) dynamics :

Xt = µ+ φXt−1 + εt

where |φ| < 1 and εt ∼WN(0, σ2ε . Then :

E [Xt ] = (1− φ)−1µ ≡ m

V [Xt ] ≡ γX (0) =σ2ε

1− φ2

γX (h) = φhσ2ε

1− φ2

ρX (h) = φh.

Proof : See Appendix 1.


AR(1) model

Partial autocorrelations

Definition

The h-th order partial autocorrelation of Xt −m is the partial regressioncoefficient in the h-th order autoregression :

Xt −m = φ1h(Xt−1 −m) + · · ·+ φhh(Xt−h −m) + εt .

Therefore,

aX (1) ≡ φ11 = φ

aX (h) = 0 for all h > 1.

Remark : The theoretical partial autocorrelation of an AR(1) is exactlyzero for h > 1. This property is often used to identify the AR component.


AR(1) model

Simulation of an AR(1) model (with φ = 0.9 and µ = 0)

-8

-4

0

4

8

12

50 100 150 200 250 300 350 400 450 500


AR(1) model

Scatter plots : left panel (Xt-1 versus Xt ) and right panel (Xt-2 versus Xt )

-8

-4

0

4

8

12

-8 -4 0 4 8 12

X_t

X_(

t-1)

-8

-4

0

4

8

12

-8 -4 0 4 8 12

X_t

X_(

t-2

)


AR(1) model

Scatter plots: Left panel (Xt-10 versus Xt) and right panel (Xt-20 versus Xt)

-8

-4

0

4

8

12

-8 -4 0 4 8 12

X_t

X_(

t-10)

-8

-4

0

4

8

12

-8 -4 0 4 8 12

X_t

X_

(t-20

)


AR(1) model


AR(1) model

(Partial) autocorrelogram of an AR(1) process with φ = 0.5 (left-panel) and φ = 0.2 (right-panel)


AR(1) model

(Partial) autocorrelogram of an AR(1) process with φ = -0.9 (left-panel), φ = -0.5 and φ = -0.2 (right-panel)


AR(1) model

Estimation of an AR(1)

Different techniques :

1 Ordinary least squares method

2 Conditional or Exact maximum likelihood estimator

3 Generalized method of moments

4 Method of Yule and Walker

5 Etc.


AR(1) model

Ordinary least squares method

The objective function is to minimize the sum of squared residuals :

(µols , φols

)= argmin

(µ,φ)

T∑t=2

(xt − µ− φxt−1)2 .

The estimator of (µ, φ)′ is given by :

µols =1

T − 1

T∑t=2

xt − φols1

T − 1

T∑t=2

xt−1

φols =

∑Tt=2(xt − x)(xt−1 − x−1)∑T

t=2(xt−1 − x−1)2

where x = (T − 1)−1T∑t=2

xt and x−1 = (T − 1)−1T∑t=2

xt−1.


AR(1) model

Conditional maximum likelihood estimator

Suppose that (εt) is a Gaussian white noise.

Let It denote the available information at time t :

It = xt , xt−1, · · · , x1

Conditional on It−1 :

xt |It−1 ∼ N (µ+ φxt−1, σ2ε )

The conditional density f (xt |xt−1, µ, φ, σ2ε ) is then :

f (xt |xt−1, µ, φ, σ2ε ) =

(2πσ2

ε

)−1/2exp

(− 1

2σ2ε

(xt − µ− φxt−1)2

).


AR(1) model

The conditional log-likelihood is :

`(θ | x) ≡T∑t=2

logf (xt | xt−1, θ)

= −T − 1

2log(2π)− T − 1

2log(σ2

ε )

− 1

2σ2ε

T∑t=2

(xt − µ− φxt−1)2.

where θ = (µ, φ, σ2ε )′.

Solving the first-order conditions of the log-likelihood function withrespect to µ, φ, and σ2

ε yields :

µcmle = µols and φcmle = φols

σ2cmle =

1

T − 1

T∑t=2

(xt − µcmle − φcmlext−1

)2.


AR(1) model

Exact maximum likelihood estimator

The conditional maximum likelihood estimator does not exploit theinformation at time t = 1.

The marginal log-likelihood for the initial value x1 is :

logf (x1; θ) = −1

2log(2π)− 1

2log

(σ2ε

1− φ2

)− 1− φ2

2σ2ε

(x1 −

µ

1− φ

)2

.

The exact log-likelihood function is the sum of the marginal (t = 1)and conditional (t ≥ 2) log-likelihood functions :

`(θ | x) = −T

2log(2π)− 1

2log

(σ2ε

1− φ2

)− 1− φ2

2σ2ε

(x1 −

µ

1− φ

)2

−T − 1

2log(σ2

ε )− 1

2σ2ε

T∑t=2

(xt − µ− φxt−1)2


AR(1) model

In general, there is no closed form solution for the exact maximumlikelihood estimators.

The exact maximum likelihood estimates must be determined bynumerically maximizing the exact log-likelihood function

The estimates of the Hessian and Score (in order to find an estimateof the variance-covariance matrix) may be computed numerically(using numerical derivative routines) or they may be computedanalytically (if analytic derivatives are known).


AR(1) model

Generalized method of moments

An overview...

Using the canonical representation, one has the following momentconditions :

E [1× εt ] = 0

E [xt−h × εt ] = 0 for h ≥ 1.

Therefore, the moment conditions write :

E[z ′tεt

]= 0h×1

where zt = (1, xt−1, · · · , xt−(h−1)).

zt is a set of instruments, i.e. each instrument is (i) uncorrelated withthe error term, εt , and (ii) is correlated with the regressor, xt−1.


AR(1) model

The empirical counterpart is given by (dividing by T without loss ofgenerality) :

1

T

T∑t=2

g(zt , θ) ≡ 1

T

T∑t=2

z ′t(xt − µ− φxt−1) = 0h×1.

This is a system of h equations (moment conditions) with twounknowns, µ and φ

Three cases of interest :1 h < 2 : the statistical model is under-identified (this case is ruled by a

so-called rank condition)2 h = 2 : the statistical model is just-identified ;3 h > 2 : the statistical model is over-identified.

There is a bias-efficiency trade-off with the number of instruments. Inpractise, the over-identified case is the rule !


AR(1) model

An estimator of θ = (µ, φ)′ is obtained by solving the set of equationsas close as possible to the null vector...

In doing so, the following quadratic form can be minimized :

θ2s−gmm = arg minθ∈Θ

1

T

T∑t=2

g(zt , θ)′Ω−1T

1

T

T∑t=2

g(zt , θ)

where Ω−1T is a first-step consistent estimator of the inverse of the

variance-covariance matrix of the moments conditions and it accountsfor the dependence of moment conditions.

The corresponding estimator is called the two-step GMM estimator.


AR(1) model


Application with an AR(1) model

3. Application with an AR(1) model

Effective Fed fund rate : 1970 :01-2010 :01 (monthly observations)

Descriptive statistics :

Statistics 1970 :01-2010 :01 1990 :01-2010 :01

Mean 6.279 4.033Median 5.550 4.680Max 19.100 8.290Min 0.110 0.110Std. Dev. 3.554 2.054Skewness 0.923 -0.220Kurtosis 4.453 2.293



Effective Fed fund rate

0

4

8

12

16

20

1970 1975 1980 1985 1990 1995 2000 2005 20100

4

8

12

16

20

24

28

32

0.00 1.25 2.50 3.75 5.00 6.25 7.50



(Partial) correlation of the effective Fed fund rate



The autocorrelation function decreases at a slow rate : the stochasticprocess may be quite persistent...

The partial autocorrelation function is statistically significant for thefirst two coefficients : this may be the autocorrelation function of anAR(2) model

Estimation of AR(1) and AR(2) models confirms that the Fed fundrate is highly persistent : φ is close to one (respectively, φ1 + φ2 isclose to one) :



ML estimation of the effective Fed fund rateCoefficients Estimates Std. Error P-value

µ 4.810 2.549 0.051φ 0.987 0.008 0.000

µ 5.753 1.313 0.000φ1 1.395 0.042 0.000φ2 -0.414 0.042 0.000



Adjusted effective Fed fund rate

-8

-6

-4

-2

0

2

4

0

5

10

15

20

1970 1975 1980 1985 1990 1995 2000 2005

Residual Actual Fitted



Effective Fed fund rate---diagnostics of the estimated AR(1) model

0.2

0.4

0.6

0.8

1.0

5 10 15 20 25 30 35 40 45

Actual Theoretical

Aut

ocor

rela

tion

-0.4

0.0

0.4

0.8

1.2

5 10 15 20 25 30 35 40 45

Actual Theoretical

Par

tial au

toco

rrel

atio

n



Effective Fed fund rate---impulse response function of the estimated AR(1) model

-.2

.0

.2

.4

.6

.8

10 20 30 40 50 60 70 80 90 100

Impulse Response ± 2 S.E.

0

10

20

30

40

50

60

10 20 30 40 50 60 70 80 90 100

Accumulated Response ± 2 S.E.



An alternative is to use a first-difference transformation

∆rt = rt − rt−1

The representation and the properties of the transformed series arethen different.

In particular, the first-differenced effective Fed fund rate displays lesspersistence and the corresponding partial autocorrelation function canbe interpreted as the one of either an AR(1) or AR(2) process (on∆rt).



Effective Fed fund rate (first-difference)

-8

-6

-4

-2

0

2

4

1970 1975 1980 1985 1990 1995 2000 2005



(Partial) correlogram of the effective Fed fund rate (first-difference)



ML estimation of the effective Fed fund rate (in first-difference)

Coefficients Estimates Std. Error P-value

µ -0.007 0.0267 0.796φ 0.399 0.043 0.000

µ -0.011 0.037 0.762φ1 0.474 0.046 0.000φ2 -0.187 0.046 0.000



Issues :

Stationarity versus non-stationarity of the effective Fed fund rate

Model specification : AR(1) versus AR(2)

Other specifications ? ARMA(p,q) ?

Model selection : criteria ?

Nonlinear time series models (Markov switching models, etc) ?


Autoregressive model of order p, AR(p)

Autoregressive models4. Autoregressive model of order p, AR(p)

Definition

A stochastic process (Xt)t∈Z is said to be an autoregressive process oforder p if it satisfies the following equation :

Xt = µ+ φ1Xt−1 + · · ·+ φpXt−p + εt ∀t

where φp 6= 0, µ is a constant term, (εt)t∈Z is a weak white noise processwith expectation zero and variance σ2

ε (εt ∼WN(0, σ2ε )).



Remarks :

1. In lag notation, one has :

Φ(L)Xt ≡ (1− φ1L− · · · − φpLp)Xt = µ+ εt

2. The mean adjusted autoregressive model of order p can be written asfollows :

Xt = φ1Xt−1 + · · ·+ φpXt−p + εt

where

Xt = Xt −µ

1−∑p

k=1 φk.

3. The specification of an AR(p) model is a stochastic differenceequation of order p (previous results with an AR(1) can begeneralized).



4. Companion representation : the p-th order stochastic differenceequation can be written as a first-order vector stochastic differenceequation.

Xt

Xt−1

Xt−2...

Xt−p+1

︸︷︷︸

ξt

=

φ1 φ2 · · · · · · φp1 0 · · · · · · 0

0 1. . . · · · 0

......

. . . · · · . . .

0 0 · · · 1 0

︸︷︷︸

F

Xt−1

Xt−2

Xt−3...

Xt−p

︸︷︷︸

ξt−1

+

εt0000

︸︷︷︸

ut

.

5. Using insights from AR(1) :

ξt+h = F h+1ξt−1 + ut + Fut−1 + · · ·+ F hut−h.



Stationarity and stability conditions

An AR(p) process is stable and weakly stationary if all the roots ofthe characteristic equation Φ(λ) = 0 are outside the unit circle :

1− φ1λ− φ2λ2 − · · · − φpλp = 0⇔ |λi | > 1

for i = 1, · · · , p.

Using the companion representation of an AR(p) process, stabilityrequires (initial values have no impact...) that :

limh→∞

F h = 0

or that all of the eigenvalues of F have modulus less than one.



Definition

The representation of the autoregressive process of order p defined by :

Xt = µ+ φ1Xt−1 + · · ·+ φpXt−p + εt ,

is said to be causal or fundamental—(εt) is the innovation process—if allthe roots of the characteristic equation Φ(λ) ≡ 1− φ1λ− · · · − φpλp = 0lies outside the unit circle :

|λi | > 1.

In particular,

Cov (Xt−j , εt−`) = 0 for all j < `.



Moments of stationary AR(p)

Definition

Let (Xt) denote a stationary stochastic process that has a fundamentalAR(p) representation, Xt = µ+ φ1Xt−1 + · · ·+ φpXt−p + εt . Then, theautocovariance of order h (with |h| > p) satisfies the following differenceequation of order p :

γX (h) =

p∑k=1

φkγX (h − k)

i.e.

γX (h)− φ1γX (h − 1)− φ2γX (h − 2)− · · · − φpγX (h − p) = 0.

Proof : See Appendix 2.



ResolutionThe recurrence equation of order p can be solved by determining thecharacteristics roots of :

zpΦ

(1

z

)≡ zp − φ1z

p−1 − · · · − φp−1z − φp = 0

Since these roots zi , i = 1, · · · , p, are also zi = λ−1i , they are all of

modulus less than one.

Suppose that all the characteristics roots are real and distinct, thegeneral solution writes :

γX (h) =

p∑k=1

akzhk

where the constant terms can be determined from the first p initialconditions, γX (0), · · · , γX (p − 1).



Definition

Let (Xt) denote a stationary stochastic process that has a fundamentalAR(p) representation, Xt = µ+ φ1Xt−1 + · · ·+ φpXt−p + εt . Then, theautocorrelation of order h (with |h| > p) satisfies the following differenceequation of order p :

ρX (h)− φ1ρX (h − 1)− φ2ρX (h − 2)− · · · − φpρX (h − p) = 0



The first p autocorrelations (1 ≤ h ≤ p) satisfy :

ρX (h) =

p∑k=1

φkρX (|k − h|)

Solving these p equations yields the following p initial conditions :ρX (1), · · · , ρX (p).

Otherwise, the same resolution applies (see autocovariances).

All in all, the autocorrelation function exhibits exponential decaytowards zero : it does not cut off but gradually dies out as h increases(possibly, with damped oscillations).



(Partial) autocorrelogram of an AR(2) process: φ1 = 1.2 and φ2 = -0.3 (left panel), φ1 = 1.6 and φ2 = -0.7, φ = 0.3 and φ = 0.5 (right panel)



Partial autocorrelations

Definition

The h-th order partial autocorrelation of Xt −m is the partial regressioncoefficient in the h-th order autoregression :

Xt −m = φ1h(Xt−1 −m) + · · ·+ φhh(Xt−h −m) + εt .

Therefore,

aX (h) ≡ φhh for all 1 ≤ h ≤ p

aX (h) = 0 for all h > p.

Remark The theoretical partial autocorrelation of an AR(p) is exactly zerofor h > p.



Example : Let (Xt) be an autoregressive process of order 2

Xt −m = φ1(Xt−1 −m) + φ2(Xt−2 −m) + εt

The first partial autocorrelation, aX (1), is the partial regressioncoefficient, φ11, in the first order autoregression :

Xt −m = φ11 (Xt−1 −m) + εt

The second partial autocorrelation, aX (2), is the partial regressioncoefficient, φ12, in the second order autoregression :

Xt −m = φ21(Xt−1 −m) + φ22(Xt−2 −m) + εt .

Therefore,

aX (2) = φ22 = φ2.



Estimation of an AR(p)

Different techniques :

The methods developed in the case of an autoregressive process oforder 1 can be generalized in the present setting.In particular, the objective function of the ordinary least squaresestimator of θ = (µ, φ1, · · · , φp)′ is :

θols = argminθ

T∑t=p+1

(xt − µ−

p∑k=1

φkxt−k

)2

and

µols =

(1−

p∑k=1

φols

)x by stationarity

φols =(X ′p:T−1X p:T−1

)−1X ′p:T−1Xp+1:T

where X p:T−1 is the matrix of regressors andXp+1:T = (xt−p−1, · · · , xT )′.



The conditional maximum likelihood estimator

The joint density of all the observations is :

f (xT , · · · , x1; θ) =

T∏t=p+1

f (yt |It−1; θ)

f (xp, xp−1, · · · , x1; θ)

where It is the information available at time t, (xp, · · · , x1)′ is thevector of initial values, and θ = (µ, φ1, · · · , φp, σ2

ε )′.

The conditional log-likelihood function is :

`(θ|x) =T∑

t=p+1

logf (xt |It−1; θ)

The conditional maximum likelihood estimator of θ solves :

θcmle = argmaxθ

T∑t=p+1

logf (xt |It−1; θ)



The exact maximum likelihood estimator

The exact log-likelihood function is :

`(θ|x) =T∑

t=p+1

logf (xt |It−1; θ) + logf (xp, · · · , x1; θ)

The conditional maximum likelihood estimator of θ solves :

θmle = argmaxθ

T∑t=p+1

logf (xt |It−1; θ) + logf (xp, · · · , x1; θ)

For stationary models, θcmle and θmle have the same properties forlarge T . In finite samples, they are generally not equal and may differby a substantial amount in certain situations.



Yule-Walker estimation

Using sample autocorrelations and collecting the first p Yule-Walkerequations in matrix notation yields :

1 ρX (1) · · · ρX (p − 1)ρX (1) 1 ρX (p − 2)ρX (2) ρX (1) · · · ρX (p − 3)

.

.

.. . .

.

.

.ρX (p − 2) ρX (1)ρX (p − 1) ρX (p − 2) · · · ρX (1) 1

φ1φ2φ3

.

.

.φp−1φp

=

ρX (1)ρX (2)ρX (3)

.

.

.ρX (p − 1)ρX (p)

.

or, in short,

Γφ = ρp



The Yule-Walker estimator is given by :

φ = Γ−1ρp

where ρX (h) (h = 1, · · · , p) is estimated from the sampleautocorrelation function :

ρX (h) =

T∑t=h+1

(xt − µ)(xt−h − µ)

T∑t=1

(xt − µ)2

.

The Yule-Walker estimator is presented by solving exactly p equations(the system is just-identified). This estimator can also be presented inthe over-identified case.


Appendix

5. Appendix

1. Moments of an AR(1).

2. Autocovariances of an AR(p).


Appendix

1. Moments of an AR(1)

Mean of an AR(1)

E(Xt) = E (µ+ φXt−1 + εt)

= µ+ φE(Xt−1) + E(εt)

= µ+ φE(Xt)

since E(Xt) = E(Xt−j) for all j (stationarity property) and E(εt) = 0(white noise). Therefore,

E(Xt) =µ

1− φ≡ m.


Appendix

Variance of an AR(1)

γX (0) = V(Xt) = E[(Xt −m)2

]= E

[(φ(Xt−1 −m) + εt)

2]

using the mean adjusted form

= φ2E[(Xt−1 −m)2

]+ 2φE [(Xt−1 −m)εt ] + E

[ε2t

]= φ2E

[(Xt −m)2

]+ E

[ε2t

]= φ2γX (0) + σ2

ε

since E[(Xt−1 −m)2] = E

[(Xt −m)2] (by stationarity), E[ε2

t ] = σ2ε (by

definition), and E [(Xt−1 −m)εt ] (by definition of an innovation andE(m × εt) = mE [εt ] = 0). Therefore,

γX (0) =σ2ε

1 − φ2.

Using the Wold’s decomposition yields the same result :

γX (0) = V

(m +

∞∑k=0

φkεt−k

)= V

(∞∑k=0

φkεt−k

)= σ2

ε

∞∑k=0

φ2k =σ2ε

1 − φ2.


Appendix

AutocovariancesTrick : Take the mean adjusted form, multiply Xt −m by Xt−h −m,and take expectations :

γX (h) = E [(Xt −m)(Xt−h −m)]

= E [φ(Xt−1 −m)(Xt−h −m)] + E [εt(Xt−h −m)]

= φγX (h − 1)

since E [(Xt−1 −m)(Xt−h −m)] = E[(Xt −m)(Xt−(h−1) −m)

]=

γX (h − 1) (by stationarity) and E [εt(Xt−h −m)] = 0. Therefore,

γX (h) = φhγX (0) by recursive substitution

= φhσ2ε

1− φ2.

Remark : The equation γX (h) = φγX (h − 1) is called the Yule-Walkerequation of an AR(1).


Appendix

Autocorrelations

ρX (h) =γX (h)

γX (0)=φhγX (0)

γX (0)

= φh = ψh.

The autocorrelation function exhibits exponential decay towards zero :it does not cut off but gradually dies out as h increases.


Appendix

2. Autocovariances of an AR(p)

Mean of an AR(p)

E(Xt) = E (µ+ φ1Xt−1 + · · ·+ φpXt−p + εt)

= µ+ φ1E(Xt−1) + · · ·+ φpE(Xt−p) + E(εt)

= µ+ E(Xt)

p∑k=1

φk

since E(Xt) = E(Xt−j) for all j (stationarity property) and E(εt) = 0(white noise). Therefore,

E(Xt) =µ

1−∑p

k=1 φk≡ m.


Appendix

General methodology : the Yule-Walker equations1 Determine the variance, γX (0), and the autocovariances of order

1, · · · , p − 1 : γX (1), · · · , γX (p − 1).

2 Use these p equations to find γX (0), · · · , γX (p − 1).

3 Write the recurrence or difference equation for all the autocorrelationsof order |h| ≥ p.

4 Solve this recurrence equation using the first p initial conditions :γX (0), · · · , γX (p − 1).


Appendix

Step 1 : Determination for h = 0Take the mean adjusted form, multiply by Xt −m on both sides, andtake expectations :

γX (0) = E [(Xt −m)(Xt −m)]

=

p∑k=1

E [φk(Xt−k −m)(Xt −m)] + E [εt(Xt −m)]

=

p∑k=1

φkγX (k) + σ2ε

since E [(Xt−k −m)(Xt −m)] = γX (k) (by stationarity) andE [εt(Xt −m)] = σ2

ε .


Appendix

Step 2 : Determination for 1 < h < pTake the mean adjusted form, multiply Xt −m by Xt−h −m, and takeexpectations :

γX (h) = E [(Xt −m)(Xt−h −m)]

=

p∑k=1

E [φk(Xt −m)(Xt−h −m)] + E [εt(Xt−h −m)]

=

p∑k=1

φkγX (|k − h|)

since E [εt(Xt−h −m)] = 0 for h 6= 0.

Examples :

h = 1 : γX (1) = φ1γX (0) + φ2γX (1) + · · ·+ φpγX (p − 1)h = 2 : γX (2) = φ1γX (1) + φ2γX (0) + φ3γX (1) + · · ·+ φpγX (p − 2)


Appendix

Step 3 : Determination for h > p

γX (h) = E [(Xt −m)(Xt−h −m)]

=

p∑k=1

E [φk(Xt −m)(Xt−h −m)] + E [εt(Xt−h −m)]

=

p∑k=1

φkγX (h − k)

i.e.

γX (h)− φ1γX (h − 1)− φ2γX (h − 2)− · · · − φpγX (h − p) = 0

⇒ This is a recurrence or difference equation of order p.


Documents

Lecture 2: ARMA(p,q) models (part 1)frapetti/CorsoP/chapitre_21_IMEA_1.pdf · Lecture 2: ARMA(p,q) models (part 1) Florian Pelgrin University of Lausanne, Ecole des HEC Department