8/6/2019 stat420c3

1/33

Chapter 3: Some Time-Series Models

1

8/6/2019 stat420c3

2/33

Section 3.1 Stochastic Processes and Their

Properties

There are several terminologies in this sections:

Stochastic processes (random processes): xtfor t R, where xt is a random variable whent is given.

continuous: t (,).

discrete: t = 0,1,2, . Ensemble: the set of xt for all possible t.

Realization: the element of the ensemble.

We write X(t) or Xt if we treat time series

as random, and x(t) or xt if we treat it as

observations.

2

8/6/2019 stat420c3

3/33

LetXt

be the time series. Then,

the mean function is

(t) = E[X(t)];

the variance function is

2(t) = V ar[X(t)];

the autocovariance is(t1, t2) =Cov(X(t1), X(t2))

=E{[X(t1) (t1)][X(t2) (t2)]} the autocorrelation is

(t1, t2) =Cor(X(t1), X(t2))

=E{[X(t1) (t1)][X(t2) (t2)]}

(t1)(t2)

= (t1, t2)(t1)(t2)

.

Clearly, there is (t, t) = 2(t) and (t, t) = 1.

3

8/6/2019 stat420c3

4/33

3.2. Stationary Processes

Strictly stationary.

A time series is said strictly stationary if the joint

distribution ofX

(t1

), , X

(tk

) is the same as the

joint distribution of X(t1 + ), , X(tk + ) forany possible values of and k

If X(t) is strictly stationary, then (t) i s aconstant and (t1, t2) only depends on t1 t2.

Then, we can write

(t1, t2) = (|t1 t2|)

and

(t1, t2) = (|t1 t2|).

4

8/6/2019 stat420c3

5/33

Second-order stationary.

A time series is said second-order stationary if

(t) is independent oft, and (t1, t2) only depends

on |t1 t2|.

If the joint distribution of X(t1), , X(tk) isalways normal, then we call X(t) is a normal

time series (or just normal).

If X(t) is normal, then second-order stationaryand strictly stationary are equivalent.

Mostly, we simply call second-order stationary asstationary.

5

8/6/2019 stat420c3

6/33

Let X(t) be a time series. If the limit distribution

of X(t) exist, then this distribution is equilibrium

distribution.

If the conditional distribution of X(t + 1) givenX(t) is invariant, then there exists an equilibrium

distribution. This is also called a Markov Chain.

6

8/6/2019 stat420c3

7/33

3.3 Some Properties of Autocorrelation Function

Let X(t) be a (second-order) stationary time se-

ries. Then we can define the following: Autocorrelation function:

() =()

(0).

The correlation matrix of X(t1), , X(tn) is

1 (t1 t2) (t1 tn)(t2 t1) 1 (t2 tn)

... ... . . . ...

(tn t1) (tn t2) 1

,

which should be nonnegative-definite.

Therefore, an autocorrelation function ()satisfies

(i) |()| 1;

(ii) (0) = 1;

(iii) () is nonnegative-definite.

Nugget effect: if(0+) = lim

0 () < 1,

there there exist a nugget effect.7

8/6/2019 stat420c3

8/33

3.4. Some Useful Models

3.4.1 Purely random processes:

Let Zt be a discrete stationary time series. If Zt1and Zt2 are independent when t1

= t2, then Zt is

call a pure random processes or a white noise. It

has:

an autocovariance function as(k) =Cov(Zt, Zt+k)

=

2Z k = 00, k = 0.

an autocorrelation function as(k) =Corr(Zt, Zt+k)

=1 k = 00, k = 0.

Sometimes, we make the assumption weaker from

independent to uncorrelated. This is enough for

any inference of linear operations.

8

8/6/2019 stat420c3

9/33

3.4.2. Random walks:

Let Xt be a discrete time series and Zt be the

white noise. Then, Xt is called a random walk if

Xt = Xt1 + Zt and X0 = 0.

For a random walk, we have

Xt =t

i=1

Zi.

Suppose E(Zt) = and V(Zt) = 2Z. Then, we

have E(Xt) = t and V(Xt) = t2Z. By CLT, we

have

Xt tt

L N(0, 2Z).

9

8/6/2019 stat420c3

10/33

An example:

Assume Tom and Jerry are gambling. Tom has

m units of money and Jerry has n units of money.

Each time, either Tom or Jerry wins 1 unit of

money. Suppose the probability of Tom win is p.

Compute the probability that Tom wins all Jerrysmoney.

Solution: Let Zt be defined by

P(Zt = 1) = p; P(Zt = 1) = 1 p.Define X0 = 0 and

Xt =t

i=1

Zi.

Then, if Xt attains n first, Tom wins the game;

otherwise Jerry wins the game.

Let f(a) be the probability that Tom wins the

game if Tom has a units of money and Jerry has

m + n a units of money.

10

8/6/2019 stat420c3

11/33

Then, we have

(i) f(n) = 1;

(ii) f(

m) = 0;

(iii) ifm < a < n,

f(a) = pf(a + 1 ) + qf(a 1).

Thus, we have

f(a + 1) f(a) =qp

[f(a) f(a 1)]

=(q

p)2[f(a 1) f(a 2)]

=(q

p)a+m[f(m + 1) f(m)]

=(qp

)a+mf(m + 1).

11

8/6/2019 stat420c3

12/33

Assume p = q. Thus, we haven

a=m[f(a + 1) f(a)] =

nm

(q

p)a+mf(m + 1)

f(m + 1)(1 (q/p)m+n

1

(q/p)

) = 1

f(m + 1) = 1 (q/p)1 (q/p)m+n.

Therefore,

1

a=m

[f(a + 1) f(a)] =1

a=m

(q

p)a+mf(m + 1)

f(0) = f(m + 1)(1 (q/p)m

1 (q/p) )

f(0) = 1 (q/p)m

1 (q/p)m+n.

When p = q, we have

f(0) =m

m + n

by taking the limit pq 1.

12

8/6/2019 stat420c3

13/33

Thus, we have

f(m) =

1(q/p)m1(q/p)m+n when p = q

mm+n when p = q = 1/2

Clearly, we have when

limn f(m) =

1 (q/p)m when p > 1/20 when p 1/2

13

8/6/2019 stat420c3

14/33

3.4.3. Moving average (MA) processes

A moving average process xT has the form of

Xt = 0Zt + 1Zt1 + + qZtq, (1)where Zt is a white noise (purely random process)

with E(Zt) = 0 and V(Zt) = 2Z.

Usually, 0 is scaled to 0 = 1.

We write

Zt W N(0, 2Z)for such Zt and

Xt M A(q)for such Xt.

14

8/6/2019 stat420c3

15/33

Clearly, we have

E(Xt) = 0

V(Xt) = 2Z

qi=1

2i

(k) = (k)

=2Zqki=0 ii+k, when k = 0,

, q

0 when k > q.and

(k) = (k)

=

qk

i=0 ii+k

qi=0

2i

when k = 0, , q

0 when k > q.

15

8/6/2019 stat420c3

16/33

A MA process Xt expressed by above is invertible

if the innovation expression converges that is

Zt =

j=0

jXtj

with absolute convergence coefficient as

j=0

|j| < .

16

8/6/2019 stat420c3

17/33

The backward shift operator B is defined by

BXt = Xt1; B2Xt = Xt2; ,and in general there is

Bj

Xt = Xtj.

Then, equation (1) can be expressed as

Xt = (0 + 1B + + qBq)Zt = (B)Zt

where

(B) = 0 + 1B + + qBq.

If an MA(q) process is invertible if the roots of

the equation

(B) = 0

are all outside the unit circle in the complex plane.

17

8/6/2019 stat420c3

18/33

Example: Consider the following M A(1) model

(a) : Xt = Zt 0.5Zt1 = (1 0.5B)Zt.Then,

Zt =1

1 0.5BXt

=

k=0

0.5kBkXt

=Xt +1

2Xt1 +

1

4Xt2 +

1

8Xt3 + .

Then, based on Xt, we can define Zt according to

the above formula.

(b) : Xt = Zt 2Zt1 = (1 2B)Zt.Then,

Zt =1

1 2BXt

=Xt + 2Xt1 + 4Xt2 + 8Xt3 + .Then, based on Xt, we cannot define Zt according

to the above formula.

18

8/6/2019 stat420c3

19/33

The autocorrelation for (a) is

(1) =0.5

1 + 0.52= 0.4

The autocorrelation for (b) is

(1) = 2

1 + 22 = 0.4.Thus, MA(1) model in (a) and (b) are equivalent,

which implies (a) is more approrpiate.

19

8/6/2019 stat420c3

20/33

Example: check whether the following MA pro-

cesses are invertible

(a) Xt = (1 + B)Zt.

(b) Xt = (1 + 0.7B + 0.1B2)Zt.

20

8/6/2019 stat420c3

21/33

3.4.4. Autoregressive processes

Suppose Zt W N(0, 2Z). A process {Xt} is saidan autoregressive of order p (AR(p)) if

Xt = 1Xt1 + + pXtp + Zt. (2)This can also be expressed as

Zt = Xt (1Xt1 + + pXtp). (3)

Since we only observed Xt, equation (3) can be

used to derived the white noise Zt.

21

8/6/2019 stat420c3

22/33

First-order process

Xt = Xt1 + Zt.

This is equivalent to an infinite MA process

Xt =

j=0

jZt

j,

which is well defined when || < 1. For this pro-cess, we have

E(Xt) = 0

V(Xt) = 2Z

j=0

2j

(k) = 2Z

j=0

k+2j,

and

(k) = k

for k = 0, 1, .

22

8/6/2019 stat420c3

23/33

General-order process

Zt = (1 1B pBp)Xtor equivalently as

Xt =Zt

1

1

B

p

Bp= f(B)Zt,

where

f(B) =(1 1B pBp)1=(1 + 1B + 2B

2 + ).If

j=1

|j| <

then Xt is well defined which is also stationary.

It is also equivalent that if the roots of

(B) = (1 1B pBp) = 0are all outside of the unit circle on complex plane,

then Xt is well defined.

23

8/6/2019 stat420c3

24/33

Since finding 1, 2, usually is not easy, wesometimes use stationarity and derived the fol-

lowing formulae (Yule-Walker equations)

(k) = 1(k 1) + + p(k p)for k > 0. Note that (0) = 1 and (k) = (k).We can solve those by linear equations.

This has a general expression

(k) = A1|k| + + App|k|

where i are roots of the auxiliary equations

yp 1yp1 p = 0,

where A1, , Ap can be solved by the first p linearequations.

24

8/6/2019 stat420c3

25/33

Example: Consider AR(2) processXt = 1Xt1 + 2Xt2 + Zt.

Then, Xt is stationary if

|

1 21 + 42

2 |< 1.

This requires

1 + 2 < 1; 1 2 > 1; 2 > 1.

The roots are real if

1 + 42 0.

Suppose conditions are satisfied. Then, the Yule-

Walker equations are

(0) =1

(1) =1(0) + 2(1) = 1 + 2(1)(k) =1(k 1) + 2(k 2).

25

8/6/2019 stat420c3

26/33

Example 3.1. Consider the AR(2) process

Xt = Xt1 1

2Xt2 + Zt.

Then,

(B) = 1 B + 12

B2.

The roots are

1

1

2

2=

1

1

2=

1

2 1

2i.

Therefore, it is stationary. By Yule-Walker equa-

tions, we have

(1) =(0) 12

(1)

(1) =11

2(1)

(1) =23

.

For other (k), we can use

(k) = (k

1)

1

2

(k

2).

Another expression

(k) =A1(1

2+

1

2i)|k| + A2(

1

2 1

2i)|k|

=(1

2)|k|(cos k

4

+1

3

sink

4

).

26

8/6/2019 stat420c3

27/33

3.4.5. Mixed ARMA models

An ARMA(p,q) processes is

Xt = 1Xt1+ +pXtp+1Xt1+ +qZtq.It can also write as

(B)Xt = (B)Zt

where

(B) = 1 1B pBp

and

(B) = 1 + 1B + + qBq.The process is stationary if

(B) = 0

and

(B) = 0

are outside of the unit disc on complex plane.

27

8/6/2019 stat420c3

28/33

Let (B) = (B)/(B). Then, we have

Xt = (B)Ztwhich is a pure MA process.

Alternative, let (B) = 1/(B). Then, we have

(B)Xt = Zt

which is a pure AR process.

In general, the above expressions can be used in

theoretical inference and are rarely used in appli-

cations.

28

8/6/2019 stat420c3

29/33

Example 3.2: Find the and i weights for

ARMA(1,1) model given by

Xt = 0.5Xt1 + Zt 0.3Zt1.Solution: Let (B) = 1 0.5B and (B) = (1 0.3B). Then,

(B) =B

(B)

=(1 0.3B)(1 0.5B)

=(1

0.3B)

i=0

0.5iBi

=1 +

i=0

0.2 0.5i1Bi.

Thus,

i = 0.2

0.5i1

for i = 1, 2, . Similarly, we have

i = 0.2 0.3i1

for i = 1, 2, . We always have 0 = 0 = 1.

29

8/6/2019 stat420c3

30/33

3.4.6. Integrated ARMA (ARIMA) models

An ARIMA(p,d,q) model is defined by

(B)(1 B)dXt = (B)Ztwhere Zt W N(0, 2Z).

If we write Wt = (1 B)d, then we have(B)Wt = (B)Zt

and Wt is stationary under some conditions. How-

ever, Xt is not stationary.

30

8/6/2019 stat420c3

31/33

3.4.7. The general linear process

A general linear process is

Xt =

i=0

iZti.

If

i=0

|i| <

then Xt is stationary.

Clearly, MA(q), AR(p), ARMA(p,q)

and AMIMA(p,d,q) are all linear process.

31

8/6/2019 stat420c3

32/33

3.4.8. Continuous processes

Suppose xt is a continuous time series. Then,

() = Corr(Xt, Xt+)

is a function defined in (,).A continuous time series can be approximated by

a discrete time series.

32

8/6/2019 stat420c3

33/33

3.5. The Wold Decomposition Theorem

Consider the regression Xt on (Xtq, Xtq1, )and denote the residual variance by 2q .

If

limq q = V(Xt)

then we call Xt purely indeterministic.

If

limq q = 0

then we call Xt purely deterministic.

The Wold Decomposition Theorem says: any dis-crete time stationary series can be expressed as

the sum of two uncorrelated processes, one purely

deterministic and another purely indeterministic.