58
Lecture #11 - 11/2/2005 Slide 1 of 58 Introduction to Factor Analysis Lecture 11 November 2, 2005 Multivariate Analysis

Lecture 11 November 2, 2005 Multivariate Analysis

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 1 of 58

Introduction to Factor Analysis

Lecture 11November 2, 2005

Multivariate Analysis

Page 2: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

● Today’s Lecture

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 2 of 58

Today’s Lecture

■ Factor Analysis.

◆ Exploratory factor analysis (EFA).

◆ Confirmatory factor analysis (CFA).

■ How to do an EFA or CFA.

■ Comparisons to PCA.

■ Reminder: Homework #6 due this Friday.

■ Next homework (#7): Posted on Friday, due next Friday(November 11th).

Page 3: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

● History

● Intelligence

● Assumptions

● Common Factor Model

● EFA or CFA?

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 3 of 58

History of Factor Analysis

■ Factor analysis has a long history in psychology, dating backto the work of Charles Spearman and Karl Pearson in theearly 1900’s.

Spearman Guinness

Page 4: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

● History

● Intelligence

● Assumptions

● Common Factor Model

● EFA or CFA?

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 4 of 58

History of Factor Analysis

■ Spearman’s seminal 1904 paper has shaped psychology asa science.

◆ Spearman, C. (1904). “General intelligence” objectivelydetermined and measured. American Journal ofPsychology, 15, 201-293.

■ Notable quotations from this paper:

Most of those hostile to Experimental Psychology are inthe habit of reproaching its methods with insignificance,and even with triviality...they protest that such meanscan never shed any real light upon the human soul,unlock the eternal antinomy of Free Will, or reveal theinward nature of Time and Space. (p. 203)

The present article, therefore, advocates a’Correlational Psychology.’ (p. 205)

Page 5: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

● History

● Intelligence

● Assumptions

● Common Factor Model

● EFA or CFA?

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 5 of 58

Measurement of Intelligence

■ The idea that Spearman was pursuing with his work was away to pin down intelligence.

■ At the time, Psychologists had thought that intelligence couldbe defined by a single, all-encompassing unobservableentity, called “g” (for general intelligence).

■ In his paper, Spearman sought to describe the influence of gon examinee’s test scores on several domains:◆ Pitch◆ Light◆ Weight◆ Classics◆ French◆ English◆ Mathematics

■ In reality, g may or may not exist, but postulating g providesa mechanism to detect common correlations among suchvariables.

Page 6: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

● History

● Intelligence

● Assumptions

● Common Factor Model

● EFA or CFA?

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 6 of 58

Measurement of Intelligence

■ The model proposed by Spearman was very similar to alinear regression model:

Xi1 = µ1 + λ1gi + ǫi1

Xi2 = µ2 + λ2gi + ǫi2...

...Xip = µp + λpgi + ǫip

■ Here:◆ Xij is the score of examinee i (i = 1, . . . , n) on test

domain j (j = 1, . . . , p).

◆ µj is the mean of test domain j.

◆ gi is the value of the intelligence factor for person i.◆ λj is the loading of test domain j onto the general ability

factor g.

◆ ǫij is the random error term for person i and test domain j.

Page 7: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

● History

● Intelligence

● Assumptions

● Common Factor Model

● EFA or CFA?

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 7 of 58

Spearman Model Assumptions

■ The Spearman factor model has the following assumptions:

◆ E(g) = µg = 0

◆ Var(g) = 1

◆ E(ǫip) = µǫip= 0 for all i and p.

◆ Var(ǫ) = φ

■ Although the Spearman model is very similar to a regressionmodel (just replace g with an observed variable), estimationof the model cannot proceed like regression because g is notobserved (we will get to model estimation shortly).

■ Note that the Spearman model is still very much in effecttoday with it being the basis for Item Response Theory (IRT)- methods used to estimate ability from test scores (and howyou received your score on the GRE).

Page 8: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

● History

● Intelligence

● Assumptions

● Common Factor Model

● EFA or CFA?

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 8 of 58

Common Factor Model

■ As theories of intelligencebegan to change fromgeneralized ability tospecialized ability, theSpearman model with asingle latent constructbecame less popular.

■ In the 1930’s, L. L.Thurstone developed thecommon factor model.

■ The common factor modelposited that scores were afunction of multiple latentvariables, variables thatrepresented morespecialized abilities.

Louis Leon Thurstone

Page 9: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

● History

● Intelligence

● Assumptions

● Common Factor Model

● EFA or CFA?

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 9 of 58

Common Factor Model

■ The Common Factor Model was also very similar to a linearmultiple regression model:

Xi1 = µ1 + λ11fi1 + λ12fi2 + . . .+ λ1mfim + ǫi1

Xi2 = µ2 + λ21fi1 + λ22fi2 + . . .+ λ2mfim + ǫi2...

...Xip = µp + λp1fi1 + λp2fi2 + . . .+ λpmfim + ǫip

■ Where:

◆ Xij is the response of person i (i = 1, . . . , n) on variable j(j = 1, . . . , p).

◆ µj is the mean of variable j.

◆ fik is the factor score for person i on factor k(k = 1, . . . ,m)

◆ λjk is the loading of variable j onto factor k.

◆ ǫij is the random error term for person i and variable j.

Page 10: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

● History

● Intelligence

● Assumptions

● Common Factor Model

● EFA or CFA?

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 10 of 58

Common Factor Model

■ As you could probably guess, the Common Factor Modelcould be more succinctly put by matrices:

Xi = µ + Λ Fi + ǫi

(p× 1) (p× 1) (p×m) (m× 1) (p× 1)

■ Where:

◆ Xi is the response vector of person i (i = 1, . . . , n)containing variables j = 1, . . . , p.

◆ µ is the mean vector (containing the means of allvariables).

◆ Fi is the factor score vector for person i, containing factorscores k = 1, . . . ,m.

◆ Λ is the factor loading matrix (factor pattern matrix).

◆ ǫi is the random error matrix for person i containing errorsfor variables j = 1, . . . , p.

Page 11: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

● History

● Intelligence

● Assumptions

● Common Factor Model

● EFA or CFA?

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 11 of 58

Common Factor Model Assumptions

■ Depending upon the assumptions made for the commonfactor model, two types of factor analyses are defined:

◆ Exploratory factor analysis (EFA).

◆ Confirmatory factor analysis (CFA).

■ EFA seeks to determine:

◆ The number of factors that exist.

◆ The relationship between each variable and each factor.

■ CFA seeks to:

◆ Validate the factor structure presumed by the analysis.

◆ Measure the relationship between each factor.

■ CFA, and subsequently, Structural Equation Modeling (SEM)were theorized in Bock in 1960.

Page 12: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 12 of 58

Exploratory Factor Analysis Assumptions

■ Exploratory factor analysis makes some assumptions thatallows for estimation of all factor loadings for each requestedfactor.

■ Given the Common Factor Model:

Xi = µ + Λ Fi + ǫi

(p× 1) (p× 1) (p×m) (m× 1) (p× 1)

■ The assumptions are:◆ Fi and ǫi are independent.

◆ E(F) = 0.

◆ Cov(F) = I - key assumption in EFA - uncorrelated factors.

◆ E(ǫ) = 0.

◆ Cov(ǫ) = Ψ - where Ψ is a diagonal matrix.

Page 13: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 13 of 58

Implications

■ Due to the model parameterization and assumptions, theCommon Factor Model specifies the following covariancestructure for the observable data:

Cov(X) = Σ = ΛΛ′ + Ψ

■ To illustrate what this looks like:

◆ Var(Xi) = σii = λ2

i1 + . . .+ λ2

im + ψ

◆ Cov(Xi, Xk) = σij = λi1λk1 + . . .+ λimλkm

■ The model-specified covariance matrix, Σ, is something thatillustrates the background assumptions of the factor model:that variable intercorrelation is a function of the factors in themodel.

Page 14: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 14 of 58

More Implications

■ The Common Factor Model also specifies that the factorloadings give the covariance between the observablevariables and the unobserved factors:

Cov(X,F) = Λ

■ Another way of putting this statement is:

◆ Cov(Xi, Fj) = λij

Page 15: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 15 of 58

EFA Definitions

■ Because of how the EFA model is estimated, a couple ofdefinitions are needed:

■ From two slides ago, we noted that the model predictedvariance was defined as:

σii︸︷︷︸

= λ2

i1 + . . .+ λ2

im︸ ︷︷ ︸

+ ψ︸︷︷︸

Var(Xi) = communality + specific variance

■ The proportion of variance of the ith variable contributed bythe m common factors is called the ith communality.

hi = λ2

i1 + . . .+ λ2

im

■ The proportion of variance of the ith variable due to thespecific factor is often called the uniqueness, or specificvariance.

σii = h2

i + ψi

Page 16: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 16 of 58

Model Identification

■ The factor loadings found in EFA estimation are not unique.

■ Rather, factor loading matrices (Λ) can be rotated.

■ If T is an orthogonal (orthonormal) matrix (meaning T′T = I),then the following give the same factor representation:

Λ∗ = ΛT

and

Λ

■ Such rotations preserves the fit of the model, but allow foreasier interpretation of the meanings of the factors bychanging the loadings systematically.

Page 17: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 17 of 58

Model Estimation Methods

■ Because of the long history of factor analysis, manyestimation methods have been developed.

■ Before the 1950s, the bulk of estimation methods wereapproximating heuristics - sacrificing accuracy for “speedy”calculations.

■ Before computers became prominent, many graduatestudents spent months (if not years) on a single analysis.

■ Today, however, everything is done via computers, and ahandful of methods are performed without risk of carelesserrors.

■ Three estimation methods that we will briefly discuss are:

◆ Principal component method.

◆ Principal factor method.

◆ Maximum likelihood.

Page 18: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 18 of 58

Example #1

■ To demonstrate the estimation methods and results from EFA, let’s begin withan example.

■ In data from James Sidanius(http://www.ats.ucla.edu/stat/sas/output/factor.htm), the instructor evaluationsof 1428 students were obtained for an instructor.

■ Twelve items from the evaluations are used in the data set:

Page 19: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 19 of 58

Example #1

Page 20: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 20 of 58

Principal Component Method

■ The principal component method for EFA takes a routinePCA and rescales the eigenvalue weights to be factorloadings.

■ Recall that in PCA we created a set of new variables,Y1, . . . , Ym, that were called the principal components.

■ These variables had variances equal to the eigenvalues ofthe covariance matrix, for example Var(Y1) = λ

p1

(where λp1

represents the largest eigenvalue from a PCA).

■ Now, we must rescale the eigenvalue weights so that theyare now factor loadings (which correspond to factors thathave unit variances).

■ The estimated factor loadings are computed by:

λjk =√

λpkejk

Page 21: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 21 of 58

Principal Component Method Example

■ Additionally, the unique variances are found by:

ψi = sii − h2i

■ To run an EFA, we use proc factor from SAS.

■ Note that the user guide for proc factor can be found at:

http://www.id.unizh.ch/software/unix/statmath/sas/sasdoc/stat/chap26/index.htm

■ To run the analysis:

libname example ’C:\11_02\SAS Examples’;

proc factor data=example.ex1 nfactors=2;var item13-item24;run;

Page 22: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 22 of 58

Principal Component Method Example

Page 23: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 23 of 58

Principal Component Method Example

Page 24: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 24 of 58

Principal Component Method Example

Now, we will compare this result with a result we would obtain from a PCA:

proc princomp data=example.ex1 n=2;var item13-item24;run;

Notice how λjk =√

λpkejk.

Page 25: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 25 of 58

Principal Factor Method

■ An alternative approach to estimating the EFA model is thePrincipal Factor Method.

■ The Principal Factor Method uses an iterative procedure toarrive at the final solution of estimates.

■ To begin, the procedure picks a set of communality values(h2) and places these values along the diagonal of thecorrelation matrix Rc.

■ The method then iterates between the following two stepsuntil the change in communalities becomes negligible:

1. Using Rc, find the Principal Components methodestimates of communalities.

2. Replace the communalities in Rc with the currentestimates.

Page 26: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 26 of 58

Principal Factor Method

■ To run the analysis:

*SAS Example #2;proc factor data=example.ex1 nfactors=2

method=prinit priors=random;var item13-item24;run;

Page 27: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 27 of 58

Principal Factor Method Example

Page 28: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 28 of 58

Principal Factor Method Example

Page 29: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 29 of 58

Principal Factor Method Example

Page 30: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 30 of 58

Maximum Likelihood Estimation

■ Perhaps the most popular method for obtaining EFAestimates is Maximum Likelihood (ML).

■ Similar to the Principal Factor Method, ML proceedsiteratively.

■ The ML method uses a the density function for the normaldistribution as the function to optimize (find the parameterestimates that lead to the maximum value).

■ Recall that this was a function of:

◆ The data (X).

◆ The mean vector (µ).

◆ The covariance matrix (Σ).

■ Here, Σ is formed by the model predicted matrix equation:Σ = ΛΛ

′ + Ψ (although some uniqueness conditions arespecified).

Page 31: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 31 of 58

ML Example

■ To run the analysis:

*SAS Example #3;proc factor data=example.ex1 nfactors=2

method=ml;var item13-item24;run;

Page 32: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 32 of 58

Principal Factor Method Example

Page 33: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 33 of 58

Principal Factor Method Example

Page 34: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 34 of 58

Principal Factor Method Example

Page 35: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 35 of 58

Iterative Algorithm Caution

■ With iterative algorithms, sometimes a solution does notexist.

■ When this happens, it is typically caused by what is called a“Heywood” case - an instance where the unique variancebecomes less than or equal to zero (communalities aregreater than one).

■ To combat cases like these, SAS will allow you to set allcommunalities greater than one to one with the heywoodoption (placed on the proc line).

■ Some would advocate that fixing communalities is not goodpractice, because Heywood cases indicate more problemswith the analysis.

■ To not fix the communalities, SAS allows the statementultraheywood (placed on the proc line).

Page 36: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 36 of 58

Estimation Method Comparison

■ If you look back through the output, you will see subtledifferences in the solutions of the three methods.

■ What you may discover when fitting the PCA method and theML method is that the ML method factors sometimes accountfor less variances than the factors extracted through PCA.

■ This is because of the optimality criterion used for PCA,which attempts to maximize the variance accounted for byeach factor.

■ The ML, however, has an optimality criterion that minimizesthe differences between predicted and observed covariancematrices, so the extraction will better resemble the observeddata.

Page 37: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 37 of 58

Number of Factors

■ As with PCA, the number of factors to extract can besomewhat arbitrary.

■ Often, a scree plot is obtained to check for the number offactors.

■ In SAS, there is a scree plot option:

*SAS Example #4;proc factor data=example.ex1 nfactors=2

method=ml scree;var item13-item24;run;

■ Also, with the ML method, SAS prints out a likelihood-ratiotest for the number of factors extracted (see slide 33).

■ This test tends to lead to a great number of factors needingto be extracted.

Page 38: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 38 of 58

Scree Plot Example

Page 39: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 39 of 58

Factor Rotations

■ As mentioned previously, often times a rotation is made toaid in the interpretation of the extracted factors.

■ Orthogonal rotations are given by methods such as:◆ Varimax.

◆ Quartimax.

◆ Equamax.

■ Oblique (non-orthogonal) rotations allow for greaterinterpretation by allowing factors to become correlated.

■ Examples are:

◆ Proxmax.

◆ Procrustes (needs a target to rotate to).

◆ Harris-Kaiser.

Page 40: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 40 of 58

Orthogonal Rotation Example

■ To demonstrate an orthogonal rotation, consider thefollowing code that will produce a varimax transformation ofthe factor loadings found by ML:*SAS Example #5;proc factor data=example.ex1 nfactors=2

method=ml rotate=varimax;var item13-item24;run;

Page 41: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 41 of 58

Orthogonal Rotation Example

Page 42: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 42 of 58

Oblique Rotation Example

■ To demonstrate an oblique rotation, consider the followingcode that will produce a promax transformation of the factorloadings found by ML:*SAS Example #6;proc factor data=example.ex1 nfactors=2

method=ml rotate=promax;var item13-item24; run;

Page 43: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 43 of 58

Orthogonal Rotation Example

Page 44: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 44 of 58

Factor Scores

■ Recall that in PCA, the principal components (Y - the linearcombinations) were the focus.

■ In factor analysis, the factor scores are typically anafterthought.

■ In fact, because of the assumptions of the model, directcomputation of factor scores by linear combinations of theoriginal data is not possible.

■ Alternatives exist for estimation of factor scores, but even ifthe data fit the model perfectly, the factor scores obtained willnot reproduce the resulting parameters exactly.

■ In SAS, to receive factor scores, place an out=newdata in theproc line, the factor scores will be placed in the newdata dataset.

Page 45: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

● Model Implications

● EFA Definitions

● Identification

● Estimation

● Example #1

● Principal Component Method

● Principal Factor Method

● Maximum Likelihood

● Caution

● Method Comparison

● Number of Factors

● Factor Rotations

● Orthogonal Rotation

● Oblique Rotation

● Factor Scores

Confirmatory Factor Analysis

Wrapping Up

Lecture #11 - 11/2/2005 Slide 45 of 58

Factor Scores Example

*SAS Example #7;proc factor data=example.ex1 nfactors=2

method=ml rotate=promax out=example.newdata;var item13-item24;run;

Page 46: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

● CFA Example

Wrapping Up

Lecture #11 - 11/2/2005 Slide 46 of 58

Confirmatory Factor Analysis

■ Rather than trying to determine the number of factors, andsubsequently, what the factors mean (as in EFA), if youalready know the structure of your data, you can use aconfirmatory approach.

■ Confirmatory factor analysis (CFA) is a way to specify whichvariables load onto which factors.

■ The loadings of all variables not related to a given factor arethen set to zero.

■ For a reasonable number of parameters, the factorcorrelation can be estimated directly from the analysis(rotations are not needed).

Page 47: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

● CFA Example

Wrapping Up

Lecture #11 - 11/2/2005 Slide 47 of 58

Confirmatory Factor Analysis

■ As an example, consider the data given on p. 502 ofJohnson and Wichern:

Lawley and Maxwell present the sample correlationmatrix of examinee scores for six subject areas and 220male students.

■ The subject tests are:

◆ Gaelic.

◆ English.

◆ History.

◆ Arithmetic.

◆ Algebra.

◆ Geometry.

Page 48: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

● CFA Example

Wrapping Up

Lecture #11 - 11/2/2005 Slide 48 of 58

Confirmatory Factor Analysis

■ It seems plausible that these subjects should load onto oneof two types of ability: verbal and mathematical.

■ If we were to specify what the pattern of loadings would looklike, the Factor Loading Matrix might look like:

Λ =

λ11 0

λ21 0

λ31 0

0 λ42

0 λ52

0 λ62

↑ ↑

Verbal MathAbility Ability

← Gaelic← English← History← Arithmetic← Algebra← Geometry

Page 49: Lecture 11 November 2, 2005 Multivariate Analysis

Lecture #11 - 11/2/2005 Slide 49 of 58

Confirmatory Factor Analysis

■ The model-predicted covariance matrix would then be:

Σ = ΛΦΛ′ + Ψ

■ Where:◆ Φ is the factor correlation matrix (here it is size 2× 2).

◆ Ψ is a diagonal matrix of unique variances.

■ Specifically:

Σ =

λ2

11+ ψ1 λ11λ21 λ11λ31 λ11φ12λ42 λ11φ12λ52 λ11φ12λ62

λ11λ21 λ2

21+ ψ2 λ21λ31 λ21φ12λ42 λ21φ12λ52 λ21φ12λ62

λ11λ31 λ21λ31 λ2

31+ ψ3 λ31φ12λ42 λ31φ12λ52 λ31φ12λ62

λ11φ12λ42 λ21φ12λ42 λ31φ12λ42 λ2

42+ ψ4 λ42λ52 λ42λ62

λ11φ12λ52 λ21φ12λ52 λ31φ12λ52 λ42λ52 λ2

52+ ψ5 λ52λ62

λ11φ12λ62 λ21φ12λ62 λ31φ12λ62 λ42λ62 λ52λ62 λ2

62+ ψ6

Page 50: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

● CFA Example

Wrapping Up

Lecture #11 - 11/2/2005 Slide 50 of 58

Confirmatory Factor Analysis

■ Using an optimization routine (and some type of criterionfunction, such as ML), the parameter estimates thatminimize the function are found.

■ To assess the fit of the model, the predicted covariancematrix is subtracted from the observed covariance matrix,and the residuals are summarized into fit statistics.

■ Based on the goodness-of-fit of the model, the result is takenas-is, or modifications are made to the structure.

■ CFA is a measurement model - the factors are measured bythe data.

■ SEM is a model for the covariance between the factors.

Page 51: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

● CFA Example

Wrapping Up

Lecture #11 - 11/2/2005 Slide 51 of 58

Confirmatory Factor Analysis Example

Page 52: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

● CFA Example

Wrapping Up

Lecture #11 - 11/2/2005 Slide 52 of 58

Confirmatory Factor Analysis Example

Page 53: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

● CFA Example

Wrapping Up

Lecture #11 - 11/2/2005 Slide 53 of 58

Confirmatory Factor Analysis Example

Page 54: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

● CFA Example

Wrapping Up

Lecture #11 - 11/2/2005 Slide 54 of 58

Confirmatory Factor Analysis Example

Factor Correlation Matrix:

Page 55: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

● CFA Example

Wrapping Up

Lecture #11 - 11/2/2005 Slide 55 of 58

Confirmatory Factor Analysis Example

Factor Loading Matrix:

Page 56: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

● CFA Example

Wrapping Up

Lecture #11 - 11/2/2005 Slide 56 of 58

Confirmatory Factor Analysis Example

Uniqueness Matrix:

Page 57: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

● Final Thought

● Next Class

Lecture #11 - 11/2/2005 Slide 57 of 58

Final Thought

■ EFA shares many featureswith PCA, but is primarilyused to determine theintercorrelation of variablesrather than to develop newlinear combinations ofvariables.

■ CFA is concerned withassessing the plausibility ofa structural model forobserved data.

■ We have only scratched the surface of topics about EFA andCFA.

■ This class only introduced concepts in factor analysis. For amore thorough treatment of the subject (and othermeasurement topics), consider taking my Psychology 892(Measurement Methods) course next semester.

Page 58: Lecture 11 November 2, 2005 Multivariate Analysis

Overview

Introduction

Exploratory Factor Analysis

Confirmatory Factor Analysis

Wrapping Up

● Final Thought

● Next Class

Lecture #11 - 11/2/2005 Slide 58 of 58

Next Time

■ Midterm discussion.

■ Canonical Correlation.

■ More examples.