Chapter 2 Ordinary Least Squares Copyright © 2011 Pearson Addison-Wesley. All rights reserved....

Chapter 2

Ordinary Least

Squares

Copyright © 2011 Pearson Addison-Wesley.All rights reserved.

Slides by Niels-Hugo BlunchWashington and Lee University

2-2© 2011 Pearson Addison-Wesley. All rights reserved.

Estimating Single-Independent-Variable Models with OLS

• Recall that the objective of regression analysis is to start from:

(2.1)

• And, through the use of data, to get to:

(2.2)

• Recall that equation 2.1 is purely theoretical, while equation (2.2) is it empirical counterpart

• How to move from (2.1) to (2.2)?

2-3© 2011 Pearson Addison-Wesley. All rights reserved.

Estimating Single-Independent-Variable Models with OLS (cont.)

• One of the most widely used methods is Ordinary Least Squares (OLS)

• OLS minimizes (i = 1, 2, …., N)(2.3)

• Or, the sum of squared deviations of the vertical distance between the residuals (i.e. the estimated error terms) and the estimated regression line

• We also denote this term the “Residual Sum of Squares” (RSS)

2-4© 2011 Pearson Addison-Wesley. All rights reserved.

Estimating Single-Independent-Variable Models with OLS (cont.)

• Similarly, OLS minimizes:

• Why use OLS?• Relatively easy to use

• The goal of minimizing RSS is intuitively / theoretically appealing

• This basically says we want the estimated regression equation to be as close as possible to the observed data

• OLS estimates have a number of useful characteristics

2-5© 2011 Pearson Addison-Wesley. All rights reserved.

Estimating Single-Independent-Variable Models with OLS (cont.)

• OLS estimates have at least two useful characteristics:

• The sum of the residuals is exactly zero

• OLS can be shown to be the “best” estimator when certain specific conditions hold (we’ll get back to this in Chapter 4)

– Ordinary Least Squares (OLS) is an estimator

– A given produced by OLS is an estimate

i i i

ˆe Y Y

n n2 2

i 1 i 1i i iˆe (Y Y)

1 i

n n2 2

i 1 i 1 0i iˆ Xˆe (Y )

i 0 1 i i

ˆ ˆY X e

Estimating Single-Independent-Variable Models with OLS (cont.)

How does OLS work?First recall from (2.3) that OLS minimizes the sum of the squared residuals

i 0 1 i iY X i 0 1 i i

ˆ ˆY X

2-7© 2011 Pearson Addison-Wesley. All rights reserved.

Estimating Single-Independent-Variable Models with OLS (cont.)

• Next, it can be shown (see Exercise 12) that the coefficients that ensure that for the case of just one independent variable are:

2-8© 2011 Pearson Addison-Wesley. All rights reserved.

Estimating Multivariate Regression Models with OLS

• In the “real world” one explanatory variable is not enough

• The general multivariate regression model with K independent variables is:

Yi = β0 + β1X1i + β2X2i + ... + βKXKi + εi (i = 1,2,…,N)

• Biggest difference with single-explanatory variable regression model is in the interpretation of the slope coefficients

– Now a slope coefficient indicates the change in the dependent variable associated with a one-unit increase in the explanatory variable holding the other explanatory variables constant

2-9© 2011 Pearson Addison-Wesley. All rights reserved.

Estimating Multivariate Regression Models with OLS (cont.)

• Omitted (and relevant!) variables are therefore not held constant

• The intercept term, β0, is the value of Y when all the Xs and the error term equal zero

• Nevertheless, the underlying principle of minimizing the summed squared residuals remains the same

2-10© 2011 Pearson Addison-Wesley. All rights reserved.

Example: financial aid awards at a liberal arts college

• Dependent variable:

• FINAIDi: financial aid (measured in dollars of grant) awarded to the ith applicant

2-11© 2011 Pearson Addison-Wesley. All rights reserved.

Example: financial aid awards at a liberal arts college

• Theoretical Model:

(2.9)

(2.10)

where:– PARENTi: The amount (in dollars) that the parents of the ith

student are judged able to contribute to college expenses

– HSRANKi: The ith student’s GPA rank in high school, measured as a percentage (i.e. between 0 and 100)

2-12© 2011 Pearson Addison-Wesley. All rights reserved.

Example: financial aid awards at a liberal arts college (cont.)

• Estimate model using the data in Table 2.2 to get:

(2.11)

• Interpretation of the slope coefficients?

– Graphical interpretation in Figures 2.1 and 2.2

2-13© 2011 Pearson Addison-Wesley. All rights reserved.

Figure 2.1 Financial Aid as a Function of Parents’ Ability to Pay

2-14© 2011 Pearson Addison-Wesley. All rights reserved.

Figure 2.2 Financial Aid as a Function of High School Rank

2-15© 2011 Pearson Addison-Wesley. All rights reserved.

Total, Explained, and Residual Sums of Squares

• (2.12)

• (2.13)

• TSS = ESS + RSS

• This is usually called the decomposition of variance

2-16© 2011 Pearson Addison-Wesley. All rights reserved.

Figure 2.3 Decomposition of the Variance in Y

Total, Explained, and Residual Sums of Squares

TSS ESS RSSTSS TSS TSS

ESS RSS1TSS TSS

2ESS

RTSS

2RSS

1 RTSS

2-18© 2011 Pearson Addison-Wesley. All rights reserved.

Describing the Overall Fit of the Estimated Model

• The simplest commonly used measure of overall fit is the coefficient of determination, R2:

(2.14)

• Since OLS selects the coefficient estimates that minimizes RSS, OLS provides the largest possible R2 (within the class of linear models)

2-19© 2011 Pearson Addison-Wesley. All rights reserved.

Figure 2.4 Illustration of Case Where R2 = 0

2-20© 2011 Pearson Addison-Wesley. All rights reserved.

Figure 2.5 Illustration of Case Where R2 = .95

2-21© 2011 Pearson Addison-Wesley. All rights reserved.

Figure 2.6 Illustration of Case Where R2 = 1

2-22© 2011 Pearson Addison-Wesley. All rights reserved.

The Simple Correlation Coefficient, r

• This is a measure related to R2

• r measures the strength and direction of the linear relationship between two variables:

– r = +1: the two variables are perfectly positively correlated

– r = –1: the two variables are perfectly negatively correlated

– r = 0: the two variables are totally uncorrelated

2-23© 2011 Pearson Addison-Wesley. All rights reserved.

The adjusted coefficient of determination

• A major problem with R2 is that it can never decrease if another independent variable is added

• An alternative to R2 that addresses this issue is the adjusted R2 or R2:

(2.15)

Where N – K – 1 = degrees of freedom

2-24© 2011 Pearson Addison-Wesley. All rights reserved.

The adjusted coefficient of determination (cont.)

• So, R2 measures the share of the variation of Y around its mean that is explained by the regression equation, adjusted for degrees of freedom

• R2 can be used to compare the fits of regressions with the same dependent variable and different numbers of independent variables

• As a result, most researchers automatically use instead of R2 when evaluating the fit of their estimated regressions equations

2-25© 2011 Pearson Addison-Wesley. All rights reserved.

Evaluating the Quality of a Regression Equation

Checkpoints here include the following:

1. Is the equation supported by sound theory?

2. How well does the estimated regression fit the data?

3. Is the data set reasonably large and accurate?

4. Is OLS the best estimator to be used for this equation?

5. How well do the estimated coefficients correspond to the expectations developed by the researcher before the data were collected?

6. Are all the obviously important variables included in the equation?

7. Has the most theoretically logical functional form been used?

8. Does the regression appear to be free of major econometric problems?

*These numbers roughly correspond to the relevant chapters in the book

2-26© 2011 Pearson Addison-Wesley. All rights reserved.

Table 2.1a The Calculation of Estimated Regression

Coefficients for the Weight/Height Example

2-27© 2011 Pearson Addison-Wesley. All rights reserved.

Table 2.1b The Calculation of Estimated Regression

Coefficients for the Weight/Height Example

2-28© 2011 Pearson Addison-Wesley. All rights reserved.

Table 2.2a Data for the Financial Aid Example

2-29© 2011 Pearson Addison-Wesley. All rights reserved.

Table 2.2b Data for the Financial Aid Example

2-30© 2011 Pearson Addison-Wesley. All rights reserved.

Table 2.2c Data for the Financial Aid Example

2-31© 2011 Pearson Addison-Wesley. All rights reserved.

Table 2.2d Data for the Financial Aid Example

2-32© 2011 Pearson Addison-Wesley. All rights reserved.

Key Terms from Chapter 2

• Ordinary Least Squares (OLS)

• Interpretation of a multivariate regression coefficient

• Total sums of squares

• Explained sums of squares

• Residual sums of squares

• Coefficient of determination, R2

• Simple correlation coefficient, r

• Degrees of freedom

• Adjusted coefficient of determination , R2