Factor Analysis

1

Factor Analysis

Factor analysis attempts to bring inter-correlated variables together under more general, underlyingvariables.

More specifically, the goal of factor analysis is to reduce “the dimensionality of the original space and to give an interpretation to the new space, spanned by a reduced number of new dimensions which are supposed to underlie the old ones” (Rietveld & Van Hout 1993:254).

Rietveld, T. & Van Hout, R. (1993). Statistical Techniques for the Study of Language andLanguage Behaviour. Berlin – New York: Mouton de Gruyter.Thursday 20 April 2023 04:58 PM

2

Factor Analysis

Or to explain the variance in the observed variables in terms of underlying latent factors” (Habing 2003).

Thus, factor analysis offers not only the possibility of gaining a clear view of the data, but also the possibility of using the output in subsequent analyses (Field 2000; Rietveld & Van Hout 1993).

Field, A. (2000). Discovering Statistics using SPSS for Windows. London – Thousand Oaks – New Delhi: Sage publications.

Rietveld, T. & Van Hout, R. (1993). Statistical Techniques for the Study of Language andLanguage Behaviour. Berlin – New York: Mouton de Gruyter.

Thursday 20 April 2023 04:58 PM

http://www.stat.sc.edu/~habing/courses/530EFA.pdf

3

Factor Analysis

The starting point of factor analysis is a correlation matrix, in which the inter-correlations between the studied variables are presented. The dimensionality of this matrix can be reduced by “looking for variables that correlate highly with a group of other variables, but correlate very badly with variables outside of that group” (Field 2000: 424). These variables with high inter-correlations could well measure one underlying variable, which is called a ‘factor’.

Field, A. (2000). Discovering Statistics using SPSS for Windows. London – Thousand Oaks – New Delhi: Sage publications.


4

Factor Analysis

Factor analysis is a method of dimension reduction.

It does this by seeking underlying unobservable (latent) variables that are reflected in the observed variables (manifest variables).


5

Factor Analysis

There are many different methods that can be used to conduct a factor analysis

There are many different types of rotations that can be done after the initial extraction of factors.

You also need to determine the number of factors that you want to extract.

6

Factor Analysis

Given the number of factor analytic techniques and options, it is not surprising that different analysts could reach very different results analysing the same data set.

7

Factor Analysis

However, all analysts are looking for a simple structure.

Simple structure is a pattern of results such that each variable loads highly onto one and only one factor.

8

Factor Analysis

Factor analysis is a technique that requires a large sample size.

Factor analysis is based on the correlation matrix of the variables involved, and correlations usually need a large sample size before they stabilize.

9

Factor AnalysisAs a rule of thumb, a bare minimum of 10 observations per variable is necessary to avoid computational difficulties.

Number of Cases Prospects

50 very poor

100 poor

200 fair

300 good

500 very good

1000 excellent

Comrey and Lee (1992) A First Course In Factor Analysis

10

Factor Analysis

In this example I have included many options, while you may not wish to use all of these options, I have included them here to aid in the explanation of the analysis.

11

Factor AnalysisIn this example we examine students assessment of academic courses. We restrict attention to 12 variables.

Item 13 INSTRUCTOR WELL PREPARED

Item 14 INSTRUCTOR SCHOLARLY GRASP

Item 15 INSTRUCTOR CONFIDENCE

Item 16 INSTRUCTOR FOCUS LECTURES

Item 17 INSTRUCTOR USES CLEAR RELEVANT EXAMPLES

Item 18 INSTRUCTOR SENSITIVE TO STUDENTS

Item 19 INSTRUCTOR ALLOWS ME TO ASK QUESTIONS

Item 20 INSTRUCTOR IS ACCESSIBLE TO STUDENTS OUTSIDE CLASS

Item 21 INSTRUCTOR AWARE OF STUDENTS UNDERSTANDING

Item 22 I AM SATISFIED WITH STUDENT PERFORMANCE EVALUATION

Item 23 COMPARED TO OTHER INSTRUCTORS, THIS INSTRUCTOR IS

Item 24 COMPARED TO OTHER COURSES THIS COURSE WAS

Scored on a five point Likert scale, seven is better.

https://www.staff.ncl.ac.uk/mike.cox/III/spss13.ppt

12

Factor AnalysisIn this example we examine students assessment of academic courses. We restrict attention to 12 variables.

Scored on a five point Likert scale.

13

Factor AnalysisAnalyze > Dimension Reduction > Factor

14

Factor AnalysisSelect variables 13-24 that is “instructor well prepared” to “compared to other courses this course was”. By using the arrow button.

Use the buttons at the side of the screen to set additional options.

15

Factor AnalysisUse the buttons at the side of the previous screen to set the Descriptives. Employ the Continue button to return to the main Factor Analysis screen.

Note the request for a determinant.

16

Factor AnalysisUse the buttons at the side of the main screen to set the Extraction. Employ the Continue button to return to the main Factor Analysis screen.

Note the request for Principal axis factoring, 3 factors and a scree plot.

17

Factor AnalysisUse the buttons at the side of the main screen to set the Rotation (Varimax). Employ the Continue button to return to the main Factor Analysis screen.

18

Factor AnalysisVarimax rotation tries to maximize the variance of each of the factors, so the total amount of variance accounted for is redistributed over the three extracted factors.

19

Factor AnalysisSelect the OK button to proceed with the analysis, or Paste to preserve the syntax.

Syntax for varimax and 3 factors, alternatives promax and 2

factor /variables item13 item14 item15 item16 item17 item18 item19 item20

item21 item22 item23 item24 /print initial det kmo repr extraction rotation fscore univaratiate /format blank(.30) /plot eigen rotation /criteria factors(3) /extraction paf /rotation varimax /method = correlation.

20

Factor Analysis

The descriptive statistics table is output because we used the univariate option.

Mean - These are the means of the variables used in the factor analysis.

Are they meaningful for a Likert scale!

Norman, G. (2010). Likert scales, levels of measurement and the “laws” of statistics. Advances in health sciences education, 15(5), 625-632.

Descriptive Statistics

4.46 .729 1365

4.53 .700 1365

4.45 .732 1365

4.28 .829 1365

4.17 .895 1365

3.93 1.035 1365

4.08 .964 1365

3.78 .909 1365

3.77 .984 1365

3.61 1.116 1365

3.81 .957 1365

3.67 .926 1365

INSTRUC WELLPREPARED

INSTRUC SCHOLARLYGRASP

INSTRUCTORCONFIDENCE

INSTRUCTOR FOCUSLECTURES

INSTRUCTOR USESCLEAR RELEVANTEXAMPLES

INSTRUCTOR SENSITIVETO STUDENTS

INSTRUCTOR ALLOWSME TO ASK QUESTIONS

INSTRUCTOR ISACCESSIBLE TOSTUDENTS OUTSIDECLASS

INSTRUCTOR AWARE OFSTUDENTSUNDERSTANDING

I AM SATISFIED WITHSTUDENTPERFORMANCEEVALUATION

COMPARED TO OTHERINSTRUCTORS, THISINSTRUCTOR IS

COMPARED TO OTHERCOURSES THISCOURSE WAS

Mean Std. Deviation Analysis N

21

Factor Analysis


Std. Deviation - These are the standard deviations of the variables used in the factor analysis.

Are they meaningful for a Likert scale!


4.46 .729 1365

4.53 .700 1365

4.45 .732 1365

4.28 .829 1365

4.17 .895 1365

3.93 1.035 1365

4.08 .964 1365

3.78 .909 1365

3.77 .984 1365

3.61 1.116 1365

3.81 .957 1365

3.67 .926 1365














22

Factor Analysis


Analysis N - This is the number of cases used in the factor analysis.

Note N is 1365.


4.46 .729 1365

4.53 .700 1365

4.45 .732 1365

4.28 .829 1365

4.17 .895 1365

3.93 1.035 1365

4.08 .964 1365

3.78 .909 1365

3.77 .984 1365

3.61 1.116 1365

3.81 .957 1365

3.67 .926 1365














23

Factor Analysis

The correlation matrix is included in the output because we used the determinant option.

All we want to see in this table is that the determinant is not 0.

If the determinant is 0, then there will be computational problems with the factor analysis, and SPSS may issue a warning message or be unable to complete the factor analysis.

Correlation Matrixa

Determinant = .002a.

24

Factor Analysis

Kaiser-Meyer-Olkin Measure of Sampling Adequacy This measure varies between 0 and 1, and values closer to 1 are better. A value of 0.6 is a suggested minimum.

KMO and Bartlett's Test

.934

8676.712

66

.000

Kaiser-Meyer-Olkin Measure of SamplingAdequacy.

Approx. Chi-Square

df

Sig.

Bartlett's Test ofSphericity

25

Factor Analysis

Bartlett's Test of Sphericity (see the ANOVA slides) - This tests the null hypothesis that the correlation matrix is an identity matrix. An identity matrix is matrix in which all of the diagonal elements are 1 and all off diagonal elements are 0 (indicates a lack of correlation). You want to reject this null hypothesis.


.934

8676.712

66

.000


Approx. Chi-Square

df

Sig.


26

Factor Analysis

Taken together, these tests provide a minimum standard, which should be passed before a factor analysis (or a principal components analysis) should be conducted.


.934

8676.712

66

.000


Approx. Chi-Square

df

Sig.


27

Factor Analysis

Communalities - This is the proportion of each variable's variance that can be explained by the factors (e.g., the underlying latent continua).

Communalities

.564 .676

.551 .619

.538 .592

.447 .468

.585 .623

.572 .679

.456 .576

.326 .369

.516 .549

.397 .444

.662 .791

.526 .632













Initial Extraction

Extraction Method: Principal Axis Factoring.

28

Factor Analysis

Initial - With principal factor axis factoring, the initial values on the diagonal of the correlation matrix are determined by the squared multiple correlation of the variable with the other variables. For example, if you regressed items 14 through 24 on item 13, the squared multiple correlation coefficient would be 0.564.

Communalities

.564 .676

.551 .619

.538 .592

.447 .468

.585 .623

.572 .679

.456 .576

.326 .369

.516 .549

.397 .444

.662 .791

.526 .632













Initial Extraction


29

Factor Analysis

Extraction - The values in this column indicate the proportion of each variable's variance that can be explained by the retained factors. Variables with high values are well represented in the common factor space, while variables with low values are not well represented. (In this example, we don't have any particularly low values.)

Communalities

.564 .676

.551 .619

.538 .592

.447 .468

.585 .623

.572 .679

.456 .576

.326 .369

.516 .549

.397 .444

.662 .791

.526 .632













Initial Extraction


30

Factor AnalysisFactor - The initial number of factors is the same as the number of variables used in the factor analysis. However, not all 12 factors will be retained. In this example, only the first three factors will be retained (as we requested).

Total Variance Explained

6.249 52.076 52.076 5.851 48.759 48.759 2.950 24.583 24.583

1.229 10.246 62.322 .806 6.719 55.478 2.655 22.127 46.710

.719 5.992 68.313 .360 3.000 58.478 1.412 11.769 58.478

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Factor1

2

3

4

5

6

7

8

9

10

11

12

Total % of Variance Cumulative % Total % of Variance Cumulative % Total % of Variance Cumulative %

Initial Eigenvalues Extraction Sums of Squared Loadings Rotation Sums of Squared Loadings


31

Factor AnalysisInitial Eigenvalues - Eigenvalues are the variances of the factors. Because we conducted our factor analysis on the correlation matrix, the variables are standardized, which means that the each variable has a variance of 1, and the total variance is equal to the number of variables used in the analysis, in this case, 12. Total Variance Explained

6.249 52.076 52.076 5.851 48.759 48.759 2.950 24.583 24.583

1.229 10.246 62.322 .806 6.719 55.478 2.655 22.127 46.710

.719 5.992 68.313 .360 3.000 58.478 1.412 11.769 58.478

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Factor1

2

3

4

5

6

7

8

9

10

11

12




32

Factor AnalysisInitial Eigenvalues - Total - This column contains the eigenvalues. The first factor will always account for the most variance (and hence have the highest eigenvalue), and the next factor will account for as much of the left over variance as it can, and so on. Hence, each successive factor will account for less and less variance. Total Variance Explained

6.249 52.076 52.076 5.851 48.759 48.759 2.950 24.583 24.583

1.229 10.246 62.322 .806 6.719 55.478 2.655 22.127 46.710

.719 5.992 68.313 .360 3.000 58.478 1.412 11.769 58.478

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Factor1

2

3

4

5

6

7

8

9

10

11

12




33

Factor AnalysisInitial Eigenvalues - % of Variance - This column contains the percent of total variance accounted for by each factor (6.249/12 = .52 or 52%).


6.249 52.076 52.076 5.851 48.759 48.759 2.950 24.583 24.583

1.229 10.246 62.322 .806 6.719 55.478 2.655 22.127 46.710

.719 5.992 68.313 .360 3.000 58.478 1.412 11.769 58.478

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Factor1

2

3

4

5

6

7

8

9

10

11

12




34

Factor AnalysisInitial Eigenvalues - Cumulative % - This column contains the cumulative percentage of variance accounted for by the current and all preceding factors. For example, the third row shows a value of 68.313. This means that the first three factors together account for 68.313% of the total variance.


6.249 52.076 52.076 5.851 48.759 48.759 2.950 24.583 24.583

1.229 10.246 62.322 .806 6.719 55.478 2.655 22.127 46.710

.719 5.992 68.313 .360 3.000 58.478 1.412 11.769 58.478

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Factor1

2

3

4

5

6

7

8

9

10

11

12




35

Factor AnalysisExtraction Sums of Squared Loadings - The number of rows in this panel of the table correspond to the number of factors retained. The values are based on the common variance (of the retained factors). The values in this panel of the table will always be lower than the values in the left panel of the table, because they are based on the common variance, which is always smaller than the total variance.


6.249 52.076 52.076 5.851 48.759 48.759 2.950 24.583 24.583

1.229 10.246 62.322 .806 6.719 55.478 2.655 22.127 46.710

.719 5.992 68.313 .360 3.000 58.478 1.412 11.769 58.478

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Factor1

2

3

4

5

6

7

8

9

10

11

12




36

Factor AnalysisRotation Sums of Squared Loadings - The values in this panel of the table represent the distribution of the variance after the varimax rotation. Varimax rotation tries to maximize the variance of each of the factors, so the total amount of variance accounted for is redistributed over the three extracted factors. Note the more even split.Total Variance Explained

6.249 52.076 52.076 5.851 48.759 48.759 2.950 24.583 24.583

1.229 10.246 62.322 .806 6.719 55.478 2.655 22.127 46.710

.719 5.992 68.313 .360 3.000 58.478 1.412 11.769 58.478

.613 5.109 73.423

.561 4.676 78.099

.503 4.192 82.291

.471 3.927 86.218

.389 3.240 89.458

.368 3.066 92.524

.328 2.735 95.259

.317 2.645 97.904

.252 2.096 100.000

Factor1

2

3

4

5

6

7

8

9

10

11

12




37

Factor AnalysisThe scree plot graphs the eigenvalue (variance) against the factor number. You can see these values in the first two columns of the variance explained table.

38

Factor AnalysisFrom the third factor on, you can see that the line is almost flat, meaning the each successive factor is accounting for smaller and smaller amounts of the total variance.

You need to locate this, so called, elbow!

In other words, when the drop ceases and the curve makes an elbow toward a less steep decline.

39

Factor Analysis

Factor Matrix - This table contains the unrotated factor loadings, which are the correlations between the variable and the factor. Because these are correlations, possible values range from -1 to +1. It is usual to not report any correlations that are less than |.3|. As shown.

Factor Matrixa

.713 -.398

.703 -.339

.721

.648

.783

.740 .345

.616 .415

.550

.732

.613

.819 -.345

.695 -.386













1 2 3

Factor


3 factors extracted. 7 iterations required.a.

40

Factor Analysis

Factor - The columns under this heading are the unrotated factors that have been extracted. As you can see by the footnote provided by SPSS, three factors were extracted (the three factors that we requested).

Factor Matrixa

.713 -.398

.703 -.339

.721

.648

.783

.740 .345

.616 .415

.550

.732

.613

.819 -.345

.695 -.386













1 2 3

Factor


3 factors extracted. 7 iterations required.a.

41

Factor Analysis

The plot shows the items (variables) in the rotated factor space. While this picture may not be particularly helpful, when you get this graph in the SPSS output, you can interactively rotate it.

42

Factor Analysis

Rotation may help you to see how the items (variables) are organized in the common factor space.

43

Factor Analysis

Another run of the factor analysis program is conducted with a promax rotation. It is included to show how different the rotated solutions can be, and to better illustrate what is meant by simple structure.

As you will see with an oblique rotation, such as a promax rotation, the factors are permitted to be correlated with one another. With an orthogonal rotation, such as the varimax shown above, the factors are not permitted to be correlated (they are orthogonal to one another).

Oblique rotations, such as promax, produce both factor pattern and factor structure matrices. For orthogonal rotations, such as varimax and equimax, the factor structure and the factor pattern matrices are the same.

44

Factor AnalysisUse the buttons at the bottom of the screen to set the alternate Rotation, employ the Continue button to return to the main Factor Analysis screen.

45

Factor AnalysisThe resulting plot with a “simple” structure is shown.

46

Factor AnalysisFor a recent review see Factor Analysis at 100. Historical Developments and Future Directions. By Robert Cudeck, and Robert C. MacCallum (Eds.). Lawrence Earlbaum Associates, Mahwah, NJ, 2007, xiii+381 pp., ISBN:978-0-8058-5347-6 (hardcover), and, ISBN 978-0-8058-6212-6 (paperback).

47

Factor AnalysisSummary

Factor Analysis like principal components is used to summarise the data covariance structure in a smaller number of dimensions. The emphasis is the identification of underlying “factors” that might explain the dimensions associated with large data variability.

A Beginner’s Guide to Factor Analysis: Focusing on Exploratory Factor Analysis An Gie Yong and Sean PearceTutorials in Quantitative Methods for Psychology 2013 9(2) 79-94

http://journaldatabase.org/articles/beginners_guide_factor_analysis.html

http://journaldatabase.org/articles/beginners_guide_factor_analysis.html

48

Factor AnalysisPrincipal Components Analysis and Factor Analysis share the search for a common structure characterized by few common components, usually known as “scores” that determine the observed variables contained in matrix X.

However, the two methods differ on the characterization of the scores as well as on the technique adopted for selecting their true number.

In Principal Components Analysis the scores are the orthogonalised principal components obtained through rotation, while in Factor Analysis the scores are latent variables determined by unobserved factors and loadings which involve idiosyncratic error terms.

The dimension reduction of matrix X implemented by each method produces a set of fewer homogenous variables – the true scores – which contain most of the model’s information.

49

Factor AnalysisSummary

Principal Components is used to help understand the covariance structure in the original variables and/or to create a smaller number of variables using this structure.

For Principal Components, see next weeks lecture.

50

Factor Analysis

Overview of the steps in a factor analysis. From: Rietveld & Van Hout (1993: 291).

Rietveld, T. & Van Hout, R. (1993). Statistical Techniques for the Study of Language andLanguage Behaviour. Berlin – New York: Mouton de Gruyter.

51

Factor AnalysisAfter having obtained the correlation matrix, it is time to decide which type of analysis to use: factor analysis or principal component analysis. The main difference between these types of analysis lies in the way the communalities are used. In principal component analysis it is assumed that the communalities are initially 1. In other words, principal component analysis assumes that the total variance of the variables can be accounted for by means of its components (or factors), and hence that there is no error variance. On the other hand, factor analysis does assume error variance. This is reflected in the fact that in factor analysis the communalities have to estimated, which makes factor analysis more complicated than principal component analysis, but also more conservative.

For further details see "Factor Analysis" Kootstra 2004

http://www.let.rug.nl/nerbonne/teach/rema-stats-meth-seminar/Factor-Analysis-Kootstra-04.PDF

52

SPSS Tips

Now you should go and try for yourself.

Each week our cluster (5.05) is booked for 2 hours after this session. This will enable you to come and go as you please.

Obviously other timetabled sessions for this module take precedence.

Documents

Factor Analysis