Chapter 13

Simple Linear Regression

13.1: Types of Regression Models

Lots of Terms:• Dependent or response variable

• Independent or predictor or explanatory variable

• Simple linear regression (SLR)

• Linear relationship

Formula 13.1, page 515

• Simple Linear Regression Model

• Beta0 Intercept

• Beta1 Slope

• Epsilon: noise, random error, stochastic error

• Figure 13.2, page 515 (linearity)

Relationship to Previous Work

• Where’s the mean?

• Where are the hypotheses?

13.2: Determining the Simple Linear Regression Equation

• We want the best fitting line.• We use the method of “Least-Squares.”

It guarantees the “best fitting” line.• Estimate Beta0 with b0.• Estimate Beta1 with b1.• The “b” values are called

“coefficients.”

Least Squares Results

• The slope is b1.

• Interpretation: for each 1 unit increase in X, the average or predicted value of Y changes by b1 units.

• Underlined terms should be defined in the context of the problem.

Least Squares Results

• The intercept is b0.• Interpretation: when X is 0 units, the

average or predicted value of Y is b0.• Very often, this value of the intercept is

outside the range of the data (ie the “relevant range.”)

• Interpretations of b0 should be made cautiously. Beware of extrapolation.

13.3: Measures of Variation

• “Usefulness” is an important concept.

• Two types of usefulness: statistical and practical.

• Statistical usefulness almost always requires managing the “sums of squares.”

• Practical usefulness can be assessed by managing the sums of squares or by obtaining an informed opinion.

Calculating Sums of Squares

The three most important sums of squares are shown on page 526:

• SST = SSR + SSE

• SST results from summing up squared deviations between actual values of Y and Y-bar.

• SSR = variation in Y accounted for by the regression.

• SSE = variation in Y NOT accounted for by the regression.

Using Sums of Squares

• Raw sums of squares are not that helpful.

• The Coefficient of Determination (r2) can be calculated with formula 13.7.

• Interpretation: r2 percent of the variation in variable y is explained by the variation in variable x in this data set.

• 0 r2 +1

Using Sums of Squares

• Standard Error of the Estimate• Measures the variability of the

predicted values of Y relative to the actual values of Y.

• Formula 13.13, page 530.• Interpretation: The general variability

of Y around the fitted line is “standard error of the estimate” units.

13.4: Assumptions13.5: Residual Analysis

There are 4 assumptions of Regression

• Normality of Error

• Homoscedasticity

• Independence of Errors

• Linearity.

Errors• What are the “errors?”• In Equation 13.1, errors are the epsilons.• We do not know the epsilons—they live in the

population.• We approximate the epsilons with sample data:

residual = y – yhat. See formula 13.14.• If the residuals meet the assumptions, then we

feel better about the usefulness of our analysis. • If the residuals do not meet the assumptions,

then we need a new analysis technique.

Linearity

• Examine the plot of X versus the residuals.

• Example: see Figure 13.12.

• There should not be pattern.

• A pattern means that the linear regression was not effective at explaining the variation in Y, ie the SST.

Notes on Residual Analysis

• There are a LOT of techniques that can be used to examine residuals.

• You are trying to assess the validity of assumptions.

• Each observation produces a residual.• The process of calculating Studentized

Residuals allows you to look each observation to see if it produced a “strange” residual.

Assumption of Normality

• For any value of X, the errors (and residuals) conform to a normal distribution.

• At this point, we subjectively assess with graphical means.

• Histogram of residuals.

• Normal probability plot.

Homoscedasticity

• For all X, the errors and residuals should have constant, or same, variance.

• Assess subjectively by looking at the graph of residuals versus predicted values (or studentized residuals versus X).

• Assess numerically by performing a test of equal variances (divide the set of residuals in half and test).

• Figure 13.16 shows a problem.

Independence

• Previous residuals are not correlated with current and future residuals.

• Assess by plotting the residuals in order of observation.

• A formal procedure called “Dubin-Watson test” exists.

• Usually only a problem with time-series data—data observed over time.

13.7: Inferences about the slope and correlation coefficient.

• There is a “t test” for the slope.

• There is an “F test” for the overall explanatory value of the regression line.

• There is a confidence interval estimate for the slope (skip).

• There is a “t test” for the significance of the correlation coefficient (skip).

“t-test for Slope”

• Test follows the standard hypothesis test pattern.

• Like all “t-tests,” the test statistic follows the usual format, shown in Equation 13.16, page 542.

• Like all analyses on large data sets, we look to the computer output for answers.

“F Test”

• The formula for the F Test is shown in Equation 13.17, page 544.

• This test should look familiar to you: it was developed in the ANOVA section.

• Even though the text discusses this test in terms of the slope, the more general form is the more useful.

13.8 Estimation of Mean Values and Prediction of Individual

Values

• CI for a mean value of Y (13.20)

• PI for a value of Y (13.21)

• Won’t have to calculate by hand but might need to interpret.

13.9: Pitfalls in Regression and Ethical Issues

Page 554 lists some difficulties.

• Lack of awareness of assumptions.

• Unable to evaluate assumptions.

• Unable to proceed if assumptions are violated (what are the alternatives?).

• No knowledge of subject area.

Moral of the Story

• Examine assumptions.

• ALWAYS Plot the data.– The 4 sets of data in Table 13.7 all

have the same regression results, BUT they look vastly different.

Some Objectives• Find the regression coefficients using software or

software output.• Interpret the slope, R2, and the standard error of the

estimate.• Perform a test of hypothesis on the slope.• Perform a test of hypothesis on “all of the slopes.”• Write confidence intervals and prediction intervals

for Y given X.• Evaluate assumptions when given computer output.• Suggest the type of output necessary to evaluate

assumptions.• Name some difficulties of using simple regression.