Multiple RegressionDavid A. Kenny

January 12, 2014

2

The Equation                Y = a + bX + cZ + E

   Y   criterion variable   X   predictor variable   a   intercept: the predicted value of Y when all the predictors are zero   b   regression coefficient: how much of a difference in Y results from a one unit difference in X   E   residual variable  

Y, X, Z, and E are variables and a, b, and c are coefficients.

 

3

Y Hat and the Multiple Correlation

The variable Y is the predicted Y given X and Z or equivalently a + bX + cZ, often called "Y hat.“ Note that

E = Y - Y

R is the multiple correlation: the correlation between

Y and Y.  Note also that R2 can be defined as the

variance of Y divided by the variance of Y.

4

Least SquaresThe coefficients (a, b, and c) are chosen

so that the sum of squared errors is minimized. The estimation technique is then called

least squares or ordinary least squares (OLS). Given the criterion of least squares, the

mean of the errors is zero and the errors correlate zero with each predictor.

5

Standardized VariablesIf the predictor and criterion variables are all

standardized, the regression coefficients are called beta weights.

A beta weight equals the correlation when there is a single predictor. If there are two or predictors, a beta weights can be larger than +1 or smaller than -1.

6

Order of Entry and Stepwise RegressionThe predictors in a regression equation have no

order and one cannot be said to enter before the other. Generally in interpreting a regression equation, it

makes no scientific sense to speak of the variance due to a given predictor. Measures of variance depend on the order of entry in step-wise regression and on the correlation between the predictors. Also the semi-partial correlation or unique variance has little interpretative utility.

7

AssumptionsFor significance testing the following assumptions

are made about the errors or Es:1) They have a normal distribution.2) The variance of the errors is constant and does

not depend on the level of any predictor.3) Errors are independent of each other, i.e., no clustering.

8

Significance TestingThe standard test of a regression coefficient

is to determine if the multiple correlation significantly declines when the predictor variable is removed from the equation and the other predictor variables remain. In most computer programs this is test is given by the t or F next to the coefficient.

9

MulticollinearityIf two predictors are highly correlated or if one

predictor has a large multiple correlation with the other predictors, there is said to be multicollinearity. With perfect multicollinearity (correlations of plus or minus one), estimation of regression coefficients is impossible.

Multicollinearity results in large standard errors for coefficient, and so a statistically significant regression coefficient is difficult (power is low).

10

MulticollinearityMulticollinearity for a given predictor is typically

measured by what is called tolerance which is defined as 1 - R2 where R2 is the multiple correlation where the predictor now becomes the criterion and the other predictors are the predictors.  Generally tolerance values below .20 are considered potentially problematic. 

Another measure is the variance inflation factor which is defined as 1/(1 - R2).  Values above 5 are considered to be potentially problematic.

11

SuppressionIt can occur that a predictor may have little or correlation with

the criterion, but have a moderate to large regression coefficient.  For this to happen, two conditions must co-occur:  1) the predictor must be co-linear with one or more other

predictor and 2) these predictors have non-trivial coefficients. 

With suppression, because the suppressor is correlated with a predictor that has large effect on the criterion, the suppressor should correlate with the criterion.  To explain this, the suppressor is assumed to have an effect that compensates for the lack of correlation.

12

Advanced Topics

• Rescaling• No intercept• Adjusted R2

• Bilinear Effects

13

No Intercept• It is possible to run a multiple regression

equation but fix the intercept to zero.• This is done for different reasons.

– There may be a reason to think the intercept is zero: criterion a change score.

– May want two intercepts, one for each level of a dichotomous predictor: two-intercept model.

14

RescalingImagine the following equation:

Y = a + bX + E

If Xʹ = cX + d, the new regression equation would be:

Y = a + dMX + (bc)Xʹ + E

where MX Is the mean of X.

15

Adjusted R2

The multiple correlation is biased, i.e. too large. We can adjust R2 for bias by

[R2 – k/(N – 1)][(N – 1)/(N – k -1)]where N is the number of cases and k the number of predictors. If the result is negative, the adjusted R2 is set to zero.

The adjustment is bigger if k is large relative to N.Normally, the adjustment is not made and the

regular R2 is reported.

16

Bilinear or Piecewise Regression

• Imagine you want the effect of X to change at a given value of X0.

• Create two variables• X1 = X when X ≤ X0, zero otherwise

• X2 = X when X > X0, zero otherwise

• Regress Y on X1 and X2.

17

ExampleConsider the hypothetical regression equation in which Age (in years) and Gender (1 = Male and –1 = Female) predict weight (in pounds):

Weight = 12 + 22(Gender) + 3(Age) + Error

18

InterpretationWe interpret the unstandardized coefficients as follows:           intercept: the predicted weight for people who are zero years of age and half way between male and female is 12 pounds           gender: a difference between men and women on the gender variable equals 2 and so there is a 44 (2 times 22) pound difference between the two groups           age: a difference of one year in age results in a difference of 3 poundsIt is advisable to center the Age variable.  To center Age, we would subtract the mean age from Age.  Doing so, would change the intercept to the predicted score for persons of average age in the study.

19

RescalingNote that if we recoded gender to be

1 = Male and 0 = Female, the new equation would be:

Weight =   -10 + 44(Gender) + 3(Age) + Error

       intercept: the predicted weight for women who are zero years of age and is ‑10 pounds       gender: men weigh on average  44 more pounds than women, controlling for age       age: a difference of one year in age results in a difference of 3 pounds