Chapter 2 Ordinary Least Squares Copyright © 2011 Pearson Addison-Wesley. All rights reserved. Slides by Niels-Hugo Blunch Washington and Lee University

Chapter 2

Ordinary Least

Squares

Copyright © 2011 Pearson Addison-Wesley.All rights reserved.

Slides by Niels-Hugo BlunchWashington and Lee University

2-2© 2011 Pearson Addison-Wesley. All rights reserved.

Estimating Single-Independent-Variable Models with OLS

• Recall that the objective of regression analysis is to start from:

(2.1)

• And, through the use of data, to get to:

(2.2)

• Recall that equation 2.1 is purely theoretical, while equation (2.2) is it empirical counterpart

• How to move from (2.1) to (2.2)?


Estimating Single-Independent-Variable Models with OLS (cont.)

• One of the most widely used methods is Ordinary Least Squares (OLS)

• OLS minimizes (i = 1, 2, …., N)(2.3)

• Or, the sum of squared deviations of the vertical distance between the residuals (i.e. the estimated error terms) and the estimated regression line

• We also denote this term the “Residual Sum of Squares” (RSS)



• Similarly, OLS minimizes:

• Why use OLS?• Relatively easy to use

• The goal of minimizing RSS is intuitively / theoretically appealing

• This basically says we want the estimated regression equation to be as close as possible to the observed data

• OLS estimates have a number of useful characteristics



• OLS estimates have at least two useful characteristics:

• The sum of the residuals is exactly zero

• OLS can be shown to be the “best” estimator when certain specific conditions hold (we’ll get back to this in Chapter 4)

– Ordinary Least Squares (OLS) is an estimator

– A given produced by OLS is an estimate

i i i

ê Y Y

n n2 2

i 1 i 1i i iê (Y Y)

1 i

n n2 2

i 1 i 1 0i iˆ Xê (Y )

i 0 1 i i

ˆ ˆY X e


How does OLS work?First recall from (2.3) that OLS minimizes the sum of the squared residuals

i 0 1 i iY X i 0 1 i i

ˆ ˆY X



• Next, it can be shown (see Exercise 12) that the coefficients that ensure that for the case of just one independent variable are:


Estimating Multivariate Regression Models with OLS

• In the “real world” one explanatory variable is not enough

• The general multivariate regression model with K independent variables is:

Yi = β0 + β1X1i + β2X2i + ... + βKXKi + εi (i = 1,2,…,N)

• Biggest difference with single-explanatory variable regression model is in the interpretation of the slope coefficients

– Now a slope coefficient indicates the change in the dependent variable associated with a one-unit increase in the explanatory variable holding the other explanatory variables constant


Estimating Multivariate Regression Models with OLS (cont.)

• Omitted (and relevant!) variables are therefore not held constant

• The intercept term, β0, is the value of Y when all the Xs and the error term equal zero

• Nevertheless, the underlying principle of minimizing the summed squared residuals remains the same


Example: financial aid awards at a liberal arts college

• Dependent variable:

• FINAIDi: financial aid (measured in dollars of grant) awarded to the ith applicant


Example: financial aid awards at a liberal arts college

• Theoretical Model:

(2.9)

(2.10)

where:– PARENTi: The amount (in dollars) that the parents of the ith

student are judged able to contribute to college expenses

– HSRANKi: The ith student’s GPA rank in high school, measured as a percentage (i.e. between 0 and 100)


Example: financial aid awards at a liberal arts college (cont.)

• Estimate model using the data in Table 2.2 to get:

(2.11)

• Interpretation of the slope coefficients?

– Graphical interpretation in Figures 2.1 and 2.2


Figure 2.1 Financial Aid as a Function of Parents’ Ability to Pay


Figure 2.2 Financial Aid as a Function of High School Rank


Total, Explained, and Residual Sums of Squares

• (2.12)

• (2.13)

• TSS = ESS + RSS

• This is usually called the decomposition of variance


Figure 2.3 Decomposition of the Variance in Y

Total, Explained, and Residual Sums of Squares

TSS ESS RSSTSS TSS TSS

ESS RSS1TSS TSS

2ESS

RTSS

2RSS

1 RTSS


Describing the Overall Fit of the Estimated Model

• The simplest commonly used measure of overall fit is the coefficient of determination, R2:

(2.14)

• Since OLS selects the coefficient estimates that minimizes RSS, OLS provides the largest possible R2 (within the class of linear models)


Figure 2.4 Illustration of Case Where R2 = 0


Figure 2.5 Illustration of Case Where R2 = .95


Figure 2.6 Illustration of Case Where R2 = 1


The Simple Correlation Coefficient, r

• This is a measure related to R2

• r measures the strength and direction of the linear relationship between two variables:

– r = +1: the two variables are perfectly positively correlated

– r = –1: the two variables are perfectly negatively correlated

– r = 0: the two variables are totally uncorrelated


The adjusted coefficient of determination

• A major problem with R2 is that it can never decrease if another independent variable is added

• An alternative to R2 that addresses this issue is the adjusted R2 or R2:

(2.15)

Where N – K – 1 = degrees of freedom


The adjusted coefficient of determination (cont.)

• So, R2 measures the share of the variation of Y around its mean that is explained by the regression equation, adjusted for degrees of freedom

• R2 can be used to compare the fits of regressions with the same dependent variable and different numbers of independent variables

• As a result, most researchers automatically use instead of R2 when evaluating the fit of their estimated regressions equations


Evaluating the Quality of a Regression Equation

Checkpoints here include the following:

1. Is the equation supported by sound theory?

2. How well does the estimated regression fit the data?

3. Is the data set reasonably large and accurate?

4. Is OLS the best estimator to be used for this equation?

5. How well do the estimated coefficients correspond to the expectations developed by the researcher before the data were collected?

6. Are all the obviously important variables included in the equation?

7. Has the most theoretically logical functional form been used?

8. Does the regression appear to be free of major econometric problems?

*These numbers roughly correspond to the relevant chapters in the book


Table 2.1a The Calculation of Estimated Regression

Coefficients for the Weight/Height Example


Table 2.1b The Calculation of Estimated Regression

Coefficients for the Weight/Height Example


Table 2.2a Data for the Financial Aid Example


Table 2.2b Data for the Financial Aid Example


Table 2.2c Data for the Financial Aid Example


Table 2.2d Data for the Financial Aid Example


Key Terms from Chapter 2

• Ordinary Least Squares (OLS)

• Interpretation of a multivariate regression coefficient

• Total sums of squares

• Explained sums of squares

• Residual sums of squares

• Coefficient of determination, R2

• Simple correlation coefficient, r

• Degrees of freedom

• Adjusted coefficient of determination , R2

Documents

Chapter 2 Ordinary Least Squares Copyright © 2011 Pearson Addison-Wesley. All rights reserved. Slides by Niels-Hugo Blunch Washington and Lee University