Linear Regression with One Predictor variable

KNNL – Chapter 1

Model – Error Distribution Unspecified

0 1

0 1

2 2

1,...,

Response on the trial, parameters (intercept and slope of line)

known constant, value of predictor variable on the trial

Random error term 0 ,

i i i

thi

thi

i i i i j

Y X i n

Y i

X i

E

0 1 0 1 0 1 0 1

2 2 2 20 1

0 1 0 1

*0 1 1 0 1

0

(Based on Rules in Appendix A)0

, , , 0

Alternative Form:

i i i i i i i

i i i i

i j i i j j i j

i i i i i

i j

E Y E X X E X X

Y X

Y Y X X i j

Y X X X X X

*0 0 1X

Least Squares Estimation - I

0 1

0 1

0 1

220 1

1 1

0 1 0 1

0 10

1,...,

function of unknown , and observed data

Goal: select values of , that minimize and label them as ,

: 2 1

i i i

i i i

n n

i i ii i

i ii

Y X i n

Y X

Q Y X

Q b bQi Y X

set

0 11 1 1

set2

0 1 0 11 1 1 11

0

: 2 0

n n n

i ii i

n n n n

i i i i i i ii i i i

Y nb b X

Qii Y X X X Y b X b X

Least Squares Estimation - II

1

22

11 1 1 1 1

2

121 11

1 1

1

Solving (by multiplying by and by and taking ) :n

ii

n n n n n

i i i i i ii i i i i

nn n

ii in nii i

i i ii i

i

i X ii n ii i

n X Y X Y b n X X

XX YX Y b X

n n

X Yb

1 1

1 12 2

1 11

2 1

1

0 11 1

1From :

n n

i in ni i

i i i n nii i XYi i inn i iXX XXiiin

ii

i

n nii i i

i iXX

X YX X Y Y X XSSn Y k Y

SS SSX XXX

n

X X Xi b Y b X Y lY

n SS

Fitted Values and Residuals 0 1 0 1

^

0 1

^

0 1

True Regression Function: (Unknown, since , parameters)

Estimated Regression Function (Fitted):

For the observation: 1,...,

Residuals:Differences between ob

thi i

E Y X

Y b b X

i Y b b X i n

^

0 1

1

1

^

0 1 0 11 1 1 1

served and fitted (predicted) values:

1,...,

Properties of Residuals:

0 (From LS eq )

0 (From LS eq )

0

ii i i i

n

ii

n

i ii

n n n n

i i i i i i ii i i i

e Y Y Y b b X i n

e i

X e ii

Y e b b X e b e b X e

Estimating Error Variance 2

2 22 2 2

0 1

^

0 1

2

2^2

2 1 1

0

unobservable since

We use residual to "estimate"

Obtain the "average" squared residual to estimate :

2 2 2

n n

ii ii i

E E E E

Y X

e

e Y Y Y b b X

e Y YSSEs M

n n n

SE

Normal Error Model

20 1

2

0 1

2/2 0 12

0 111

2^ ^ ^

0 1

1,..., ~ 0, (independent)

1 1exp 1,...,22

1, , 2 exp2

Goal: Choose values , , th

i i i i

i ii i

n nn i in

iii

Y X i n N

y Xf y f i n

y XL f

2

0 12

1

2

0 10 1

1

^ ^

0 10 1

2 2

at maximize (or equivalently ln( )) :

1ln 2 ln2 2 2

Note: maximizing wrt , is same as minimizing

,

1 112 2

ni i

i

ni i

i

L l L

y Xn nl

y Xl

b b

yl n

2^ ^22 0 12^

0 1 21 1221

20

n n

i i in seti i i i

i

y X eX n sn n n