1

Logistic regression

2

Programme

• 2:05 pm Talk • 3:15 pm Coffee break • 3:45 pm A short exercise• 4:30 pm Finish

Revision

Correlation and regression

3

4

A study

• Does watching screened violence promote violent behaviour in children?

• In a study of the effects of media violence, some children were measured on their Actual violence and on their Exposure to screened violence.

• Here is a scatterplot of Actual violence against Exposure.

5

The scatterplot

• Each point in the plot represents one child.

• The coordinates of the point are the child’s scores on Exposure to and Actual violence.

• A strong statistical ASSOCIATION between Exposure to and Actual violence is evident from the shape of the cloud of points.

A linear association

• An elliptical scatterplot such as this indicates a LINEAR association between the variables.

6

7

The Pearson correlation

• Where the association is linear, its strength is measured by the Pearson correlation, the formula for which is:

Sums of squares and products

8

Alternative formula for the Pearson correlation

9

10

Regression

• Regression is a set of statistical techniques enabling the researcher to exploit an association among variables to PREDICT the values of one variable from those of others.

• From regression, you can also ascertain the extent to which the variance of a target variable can be EXPLAINED or accounted for in terms of the other variables.

11

Some key terms

• The variable we are trying to predict or account for is known variously as the DEPENDENT VARIABLE (DV), the CRITERION, or the TARGET VARIABLE.

• The predictors are known as the INDEPENDENT VARIABLES (IVs) or REGRESSORS.

• In our current example, the DV is Actual violence and the IV is Exposure to screen violence.

12

Simple regression and multiple regression

• In SIMPLE REGRESSION, there is just ONE IV or regressor.

• In MULTIPLE REGRESSION, there are TWO OR MORE IVs or regressors.

13

The regression line

• In simple regression, a line called the REGRESSION LINE is drawn through the points.

• The regression line is the line that fits the points most closely, according to the LEAST SQUARES CRITERION.

• The traditional approach to regression is therefore referred to as ORDINARY LEAST SQUARES (OLS) REGRESSION.

14

A linear equation

15

The equation of a straight line

16

The regression line

17

The regression equation

18

On the graph …

19

Interpretation of the slope or regression coefficient

• The slope or REGRESSION COEFFICIENT is the average number of units of change in the DV that result from a change of one unit on the IV.

• In our example, slope = .74 .• So, an increase of one unit in Exposure

produces, on average, an increase of .74 (rather less than one unit) in Actual violence.

20

The residual (e)

21

Joe’s residual score

• Joe scored 8 on Exposure and 9 on Actual.

• Joe’s predicted score from regression Y/ is 8.

• Joe’s residual score is : (9 – 8) = 1.

22

Least squares criterion for goodness-of-fit

• The values of the slope and intercept of the regression line are such that the sum of the squares of the residuals (SSerror) is a minimum.

23

A unique solution

• The values of b and c needed to achieve the least squares criterion are given by the formulae below.

• Clearly, the regression coefficient b is closely related to the Pearson correlation.

24

Relation between the regression coefficient and the correlation

coefficient

• The value of the regression coefficient is directly proportional to the value of the correlation coefficient.

25

Regression and correlation

• Regression and correlation are two sides of the same associative coin.

• The higher the correlation, the narrower the elliptical cloud of points in the scatterplot.

• For fixed values of the variances of X and Y, the higher the correlation, the greater will be the value of the regression coefficient.

26

The violence data

27

A negative correlation

• So far, I have considered only positive correlations. Here’s a negative one.

• Does the number of complaints made against GPs very inversely with the average length of their appointments?

• The following scatterplot supports this hypothesis.

28

A strong negative correlation

29

The signs of b and r

• The regression coefficient and the correlation always have the same sign.

• For the violence data, both are positive. • For the data on GPs’ appointments, both

are negative. • Note that a negative correlation of – 0.89

indicates an association as strong as one for which the correlation is +.89.

30

Complete independence

• I take two random samples, each of size 10,000 from a normal population with mean 100 and SD 25. (The Syntax for doing this is shown in the appendix.)

• Since there should be no association between the two samples, the correlation between them should be zero.

• The scatterplot will be CIRCULAR.• The regression line will be HORIZONTAL, that

is, with zero slope.

31

No association

The case of independence

32

33

Intercept-only regression

• The regression line is horizontal and passes through the value 100 on the y-axis. This is the MEAN VALUE OF THE DISTRIBUTION OF THE DEPENDENT VARIABLE.

• Here the intercept of the regression line is equal to the mean value of Y and its slope is zero.

• When X and Y are independent, you can only predict the mean value of Y, WHATEVER THE VALUE OF X.

• This is known as INTERCEPT- ONLY REGRESSION.

34

Model-building

• When testing the goodness-of-fit of regression models to the data, a useful baseline is provided by the INDEPENDENCE MODEL, which makes intercept-only predictions of the dependent variable by predicting the mean value of the DV whatever the value of the IV.

• In several computing procedures, this is labelled as STEP 0 in the analysis.

• A good regression model should be a big improvement upon the independence model.

35

The coefficient of determination (r2)

• The square of the Pearson correlation is known as the COEFFICIENT OF DETERMINATION.

• It is so-called because r2 is the proportion of the variance of Y that is accounted for by regression upon X.

36

Coefficient of determination

37

Cohen’s (1988) guidelines

38

Multiple regression model

• We could try to expand our regression model to include additional variables, such as level of parental violence and their levels of formal education.

• We should then have to determine the relative importance of the various IVs and whether we needed to include all of them in the regression model.

• These are problems in MULTIPLE REGRESSION.

39

Equations for simple and multiple regression

• In the multiple regression equation, b0 is the CONSTANT and b1, b2, …,bp are the PARTIAL REGRESSION COEFFICIENTS.

40

Partial regression coefficients

• In multiple regression, a PARTIAL REGRESSION COEFFICIENT is the estimated average change in the DV resulting from an increase of one unit in one particular IV with ALL THE OTHER IVs HELD CONSTANT.

41

The multiple correlation coefficient R

• The MULTIPLE CORRELATION COEFFICIENT (R) is the correlation between the target variable Y and the corresponding predictions Y/ of Y from regression upon the IVs.

42

Notation

• When it is necessary to specify which variables are involved in a multiple regression, a subscript notation is used.

• The multiple correlation between Y and X1, X2, …, Xp is

43

Properties of R

• R can never take a negative value, because the sign of the slope of the regression line is always the same as that of the correlation.

• Recall that the Pearson correlation varies within the range from –1 to +1, inclusive.

• The multiple correlation R, however, can only take values between zero and +1, inclusive.

44

The case of one IV

• The multiple correlation coefficient is defined even in simple regression, where there is only one IV.

• Here, remembering that R can never be negative, it takes the ABSOLUTE VALUE of the Pearson correlation (r ) between X and Y, even when r has a negative value.

• So in SPSS, R is included in the output for simple regression as well as in the output for multiple regression.

45

The coefficient of multiple determination R2

• In multiple regression, THE COEFFICIENT OF MULTIPLE DETERMINATION R2 is the proportion of variance of the dependent variable Y that is accounted for by regression upon the IVs.

46

A spatial representation of the coefficient of multiple determination

47

What if the DV is a set of categories?

• Simple and multiple OLS regression assume that all the variables are CONTINUOUS, that is, measures on an independent scale with units.

• But suppose we want to predict whether a person will suffer from a heart attack or contract a certain illness with known risk factors.

• Here, we are predicting not a VALUE, but CATEGORY MEMBERSHIP.

48

Regression with a categorical DV

The two most commonly used techniques are:

1.Logistic regression

2.Discriminant analysis

49

Discriminant analysis

• If all (or most) IVs are continuous, you might consider using DISCRIMINANT ANALYSIS (DA).

• But the DA model makes assumptions about the distributions of the IVs (such as multivariate normality) which research data often fail to satisfy.

• Moreover, DA doesn’t like qualitative IVs, such as sex or nationality.

• For these reasons, logistic regression is increasingly preferred to DA when the DV is categorical.

50

Categorical IVs

• Unlike DA, logistic regression is happy with qualitative IVs; in fact, logistic regression is happy even if ALL the IVs are qualitative.

51

A research question

• It is suspected that smoking and drinking are risk factors in the incidence of a pre-morbid blood condition.

• Records of the incidence of the condition in 100 patients are available, together with estimates of the amounts that they smoke and drink.

52

The data

53

How many of the patients have the antibody?

54

Use Frequencies

55

Frequencies dialog

56

Forty-four of the hundred patients have the antibody

57

The odds

• In an EXPERIMENT OF CHANCE (tossing a coin, rolling a die) the ODDS in favour of an event is the number of ways in which the event could occur, divided by the number of ways in which it could fail to occur.

• If a die is rolled, there is one way of getting a six and there are five ways of not getting a six.

• The odds in favour of a six are therefore 1/5.

58

Odds in favour of having the antibody

• We know that out of 100 patients, 44 have the antibody. We select a person at random from this group.

• There are 44 ways of selecting a person with the antibody; and there are 56 ways of selecting someone without it.

• The odds in favour of the person having the antibody are 44/56 = .79.

59

Probability

• A probability is a measure of likelihood ranging from 0 (an impossibility) to 1 (a certainty).

• The CLASSICAL DEFINITION of probability, like that of the odds, also arises in the context of an experiment of chance.

• The probability p of an event is the number of ways it can happen divided by the TOTAL number of possible outcomes.

• When a die is rolled, there are six possible outcomes. • There is one way of getting a six. • The probability of a six when a die is rolled is therefore

1/6.

60

Relationship between probability and the odds

• Probability and the odds are both measures of likelihood and have been defined in the same context – an experiment of chance.

• They are related according to the equation on the left.

61

Logarithms

• In a logarithmic system, numbers are expressed as powers (logarithms) of a constant called the BASE of the system.

• In COMMON LOGS, the base is 10.

• In NATURAL LOGS, the base is the mathematical constant e, where e is approximately 2.72 .

Definition of a logarithm

• The logarithm of a number is the POWER to which the base must be raised to give the number itself.

• So the log of 100 to the base ten is 2 because 102 = 100.

• The log of 1000 to the base ten is 3, because 103 = 1000.

62

Notation

63

64

Logs and antilogs

• Before the IT revolution, calculations involving large numbers were done by converting the numbers to logs, working with the logs (which are much smaller numbers), then reversing the log function with the ANTILOG FUNCTION to get back to the original number scale.

Logs of sums and products

65

• The log of the PRODUCT of two numbers is the SUM of their logs.

• The log of the QUOTIENT of two numbers is the DIFFERENCE between their logs.

Two advantages of logs

66

• You are working with SMALLER numbers.

• You are replacing the operations of MULTIPLICATION and DIVISION with the easier ones of ADDITION and SUBTRACTION.

Antilogs

• The ANTILOG function reverses the process of obtaining a logarithm.

67

The common antilog

68

The natural antilog

69

70

Antilogs (base 10)

71

Antilogs (base e)

The antilog of a sum

• The antilog of the SUM of two logs is the PRODUCT of the numbers themselves.

72

73

An asymmetrical measure

• The ODDS suffers from ASYMMETRY OF RANGE.

• Extremely unlikely events have odds confined between 0 and 1; whereas very likely events can have huge odds running into millions.

• Two very likely events could be separated by millions in terms of odds; two very unlikely events will be separated by minute fractions.

74

The log odds or logit

• The LOG ODDS (LOGIT) is the natural logarithm (log to the base e) of the odds.

75

Even Steven

• Suppose the odds were 50 to 50 (50/50 =1).

• The natural log of 1 is zero (e0 = 1).

• So for raw odds of 50 to 50, the logit (log odds) is zero.

76

Range of the logit

• The logit has a symmetrical range: a positive sign means the odds are in favour; a negative sign means the odds are against.

• Unlike the odds, which has a lower limit of zero, the logit has neither an upper nor a lower finite limit.

77

Example

• In our current example, the odds in favour of a case having the antibody are 44/56 = 11/14 = .79

• logit = ln(.79) = –.24 • The event is less likely than not, hence the

negative sign. • If the odds in favour had been 56/44, the logit

would have been ln(56/44) = ln(1.27) = +.24. • Notice the symmetry of the scale of magnitude

around the neutral point at 0.

78

Odds as antilogs

• A number such as the odds can be written as an ANTILOG, that is, the base e to the power of the natural log of the odds (the logit):

79

Probability and the logit

• We can therefore express the probability in terms of the logit, rather than the odds.

80

The logistic regression function

• We have arrived at the LOGISTIC REGRESSION FUNCTION:

81

Assumptions of logistic regression

• Either you have the antibody or you don’t. • As smoking and alcohol increase, however, the

probability of having the antibody is assumed to increase CONTINUOUSLY as a function of the IVs.

• In logistic regression, we estimate the probability of having the antibody with the LOGISTIC REGRESSION FUNCTION

• If the estimated probability exceeds a cut-off (usually set at 0.5), the case is classified by the program as a Yes, rather than a No.

82

A logistic regression function

83

Logistic regression and logit functions

• We have seen that the logistic regression function is non-linear.

• The logit function, however, is assumed to be linear.

84

The logit equation

• The logit function looks like an OLS linear regression equation, with a constant and partial regression coefficients.

85

Typical graph of the logit function

86

The decision rule

87

Interpretation of a logistic regression coefficient

• The partial regression coefficient is the increase in the LOG ODDS or LOGIT arising from an increase of one unit in the independent variable.

• If one unit is added, the logit becomes:

New odds

• To find the new odds you must MULTIPLY the old odds by exp(b).

89

Example

• In terms of the ODDS, an increase of one unit in the IV MULTIPLIES the original odds by the ANTILOG of b, that is, by eb, or exp(b).

• If b = 1.1, exp(1.1) = 3.0• So an increase of one smoking unit results in the

odds being MULTIPLIED by 3, that is, the antibody is THREE times as likely to be present in the blood of those who smoke a unit more.

90

The regression problem

• In the logit equation, we must find values of the constant and partial regression coefficients such that correct assignment to categories by the logistic regression function is maximised.

91

No mathematical solution

• In logistic regression, there is no equivalent of the formulae for the intercept and coefficients in OLS regression.

• A ‘brute force’ computing algorithm is used whereby, starting at arbitrary values of the coefficients, the values are progressively adjusted to try to arrive at a set which maximises the likelihood of obtaining the observed frequencies.

92

Iteration and ‘convergence’

• In a process known as ITERATION, estimates of the parameters are calculated again and again in the hope that they will ‘converge’ to stable values.

• IT DOESN’T ALWAYS HAPPEN!• We must therefore check that this

‘convergence’ really has been achieved by examining the ITERATION HISTORY in the SPSS output.

93

Potential difficulties

• The algorithm will not run successfully if the IVs are too highly correlated. This is the familiar MULTICOLLINEARITY PROBLEM sometimes encountered in OLS regression.

94

Centring

• As with OLS multiple regression, it is a good idea to CENTRE variables, by subtracting the mean from each score, so that the mean of the transformed scores is zero.

• CENTRING leaves the correlations among the variables unchanged.

• But centring makes the algorithm less likely to crash when there are substantial correlations among the variables.

95

Finding binary logistic regression

96

Covariates

In SPSS logistic regression dialogs, IVs that are scale or continuous variables are known as COVARIATES.

97

Always ask for the ITERATION HISTORY, so that you can check whether the algorithm was able to

arrive at a stable estimate.

98

Dire warning!

• Should the iteration history show failure to converge, the results of the analysis can be ridiculous!

• The effects of failure to converge are not limited to the IV concerned: they can mess up the whole analysis!

99

The logistic regression dialog

100

Fitting a model

• The goodness-of-fit of a model is measured by a log likelihood chi-square statistic.

• The SMALLER the value of chi-square, the BETTER the fit.

• The LARGER the p -value the better.

101

Step 0 in logistic regression

• We know that 44/100 people have the condition. • Armed only with this fact, and with no knowledge

of any associations there might be among the variables, we shall maximise our hit rate if we predict ABSENCE of the condition for ANY person selected at random.

• This is the equivalent, in logistic regression, of INTERCEPT-ONLY (no-regression) prediction in OLS regression: there you just guess MY, whatever the value of X; here you always guess that the condition is absent.

102

Here is the logistic regression output for Step 0

Iteration history at Step 0

103

104

Classification table at Step 0

105

Iteration history after applying regression model

106

Classification table at Step 1(after the regression model

has been applied)

107

The Nagelkerke R2 statistic

• The NAGELKERKE statistic is the counterpart of the coefficient of determination R2 in OLS multiple regression.

• It is a measure of the PROPORTION OF THE TOTAL VARIANCE in incidence of the antibody accounted for by regression.

108

The Nagelkerke R2 statistic

109

Hosmer and Lemeshow contingency table

110

Goodness-of-fit test

111

The Wald statistic

• The WALD STATISTIC tests a regression coefficient for significance.

• The null hypothesis is that, in the population, the coefficient is zero.

• The Wald statistic is distributed approximately as chi-square on one degree of freedom.

112

Some regression statistics

• The Wald statistic confirms that Smoking has an effect (p-value is very small) but Alcohol does not (the p-value is large).

113

The regression coefficient

114

The logit equation

115

The logistic regression equation

116

Graph of accuracy of prediction

117

Conclusion

• The incidence of the blood condition is indeed predictable from regression and raises the hit rate from 54% to 85%.

• Smoking contributes significantly to the model.

• Alcohol does not contribute significantly to the model.

118

The next step

• This session has been merely an introduction to the technique of logistic regression.

• The next step is to do some further reading.

119

Getting started

• There’s an elementary section on logistic regression in

– Kinnear, P., & Gray, C. (2011). IBM SPSS Statistics 18 made simple. Hove: Psychology Press. Chapter 14.

• This is mainly a practical, get-started guide; but there is an outline of the rationale of the technique as well.

120

• Dugard, P., Todman, J., & Staines, H. (2010) Approaching multivariate analysis: a practical introduction. (2nd ed.) London & New York: Routledge.

The next step

121

Sage paperbacks

• Menard, S. (2002). Applied logistic regression analysis (2nd ed.). London: Sage.

• Jaccard, J. (2001). Interaction effects in logistic regression. London: Sage.

122

• Tabachnik, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Allyn & Bacon. Chapter 10.

• Field, A. (2009). Discovering statistics using SPSS for Windows: Advanced techniques for the beginner (3rd ed.). London: Sage. Chapter 6.

123

Appendix

Using syntax to draw random samples from specified

populations

124

Drawing two samples from a normal distribution with mean 100 and SD 15