Reducing Response Bias with Ipsative Measurement in the ... · Reducing Response Bias with Ipsative...

Reducing Response Bias with Reducing Response Bias with IpsativeIpsativeMeasurement in the Context of Measurement in the Context of Confirmatory Factor AnalysisConfirmatory Factor Analysis

Mike W.L. CHEUNGMike W.L. CHEUNG

WaiWai CHANCHAN

Department of Psychology,Department of Psychology,

the Chinese University of Hong Kongthe Chinese University of Hong Kong

Paper presented at the International Association for Cross-Cultural Psychology XVth Congress, Pultusk 2000, Poland.A revised paper of the presentation was published asCheung, M. W. L., & Chan, W. (2002). Reducing uniform response bias with ipsative measurement in multiple group

confirmatory factor analysis. Structural Equation Modeling, 9, 55-77.

What is Response Bias?What is Response Bias?

� A systematictendency to respond to a range of questionnaire items on some basis other than the specific content (Paulhus, 1991).

� Generally, there are two categories (Paulhus, 1986; Lanyon, 1982):

– Response Style

– Response Set

Response StyleResponse Style

� The tendency to distort responses in a particular directionmore or less regardless of the content of the stimulus (Paulhus, 1986; Rorer, 1965)

� e.g., Uniform response bias, yeasayers and naysayers

� Methods to minimize:

– +ve and -ve wordings

Response SetResponse Set

� Conscious or unconscious desire on the part of the respondent to answer in such a way as to produce a certain picture of himself

� e.g., Social Desirability

� Methods to minimize:

– forced choices, anonymous responses or

– unthreatening questions before sensitive issues

– randomized response method

Examples of Response Bias in Examples of Response Bias in Different AreasDifferent Areas

� Sensitive issues research

– drug usage and sexual practices are underestimated (Bradburn, Sudman, Blair & Stocking, 1979)

– the reliabilities and validities of self-reported drinking behavior are questionable because of response bias (Embree & Whitehead, 1991)

� Cross-cultural studies

– Stronger tendency for extreme checking in Hispanic than non-Hispanic (Hui& Triandis, 1989)

– Different tendencies to use extreme scores in Japanese, British, American, Hong Kong, etc. managers (Stening & Everrett, 1984)

Estimated Accuracy of Estimated Accuracy of SelfSelf--report Informationreport Information

Adopted from Marquis, Marquis and Polich (1986)

Estimated Reliabilities of Estimated Reliabilities of SelfSelf--report Informationreport Information

Implications forImplications for Factor AnalysisFactor Analysis

� Factor analyses (EFA, CFA or SEM) are always used in validation study.

� Using simulation, Cheung (1997) showed that the statistical performance of CFA was poor when response bias was present.

� In responding to this problem , ipsativemeasurement was proposed to analyze data when response bias was present.

IpsativeIpsativeMeasurementMeasurement

� Definition: 1Tx = constant (within individual)

– That is x1+x2+…+xp= constant for every individual

– It is similar to within-subject centering

� Two ways to achieve ipsative data:

– i) ask participants response in relative values;

– ii) transform the raw data into ipsative data.

An ExampleAn Example

� Participant 1 withoutresponse style:

– Raw scores: 16, 19, 20, 22 and 23

– Average score: (16+19+…+23)/5 = 20

� Participant 1 withresponse style (+3):

– Raw scores: 19, 22, 23, 25 and 26

– Average score: (19+22+…+26)/5 = 23

An Example: CalculationsAn Example: Calculations

� Ipsative scores: – Par 1 withoutbias Par 1 withbias

– 16-20 = -4 19-23 = -4

– 19-20 = -1 22-23 = -1

– 20-20 = 0 23-23 = 0

– 22-20 = +2 25-23 = +2

– 23-20 = +3 26-23 = +3

An Example: RemarksAn Example: Remarks

� When there are response bias, the “true” scores and the observed scores are different.

� However, they are the same after ipsatization

� ⇒ remove uniform response bias

� Note: if we only have the ipsative scores, we can neverknow their original raw scores.

How can How can ipsativeipsativedata be collected?data be collected?

� For Response Style, e.g., uniform response bias:

� Steps:

– Collect data in raw scores

– Ipsatize the raw scores

• Uniform response bias is removed

– Analyze the ipsative scores

� For Response Set, e.g., Social Desirability:

� Steps:

– Collect data in the ipsative format, i.e., asking the relative values of respondants.

• Social desirability is minimized because the actual attitudes or frequencies are unknown

– Analyze the ipsative scores

Can Can ipsativeipsativedata be factorized?data be factorized?

� Mathematical and empirical evidences: NO!– (e.g., Chan & Bentler, 1993; Dunlap & Cornwell, 1994;

Jackson & Alwin, 1980)

� Remember that: x1+x2+…+xp= constant

– COV(xi, x1+x2+…+xp) = 0 for every i

– Then COV(xi, x1) +…+ COV(xi, xp) = 0

� ⇒ at leastone of the COV is negative even though they are positive in their raw scores

Problems in CFA (SEM)Problems in CFA (SEM)

� Sum of COV equal “0” in every row/column of the COV matrix

– i.e.,

� ⇒ the sample and population COV matrices are singular

� ⇒ EQS and LISREL cannot routinely be used

Proposed MethodProposed Method

� Chan & Bentler (1993, 1996 and 1998):

– methods to analyze ipsative data in SEM;

– statistical theory for these methods; and

– conditions for these methods work.

Proposed MethodProposed Method

� They showed that

– the proposed method was identified in general conditions

– the goodness-of-fit indices were good

– the “recovered” parameter estimates were near the true values

� Cheung (1997) showed that this method could be applied in analyzing data with response bias.

A Real Example: CPAIA Real Example: CPAI

� A Dataset of Chinese Personality Assessment Inventory (CPAI) was borrowed from Prof. Fanny Cheung, CUHK

� It consists of 4 personalities and 2 clinical factors (Cheung, Leung, Fan, Song, Zhang & Zhang, 1996)

� Two personalities factors were chosen here

CPAI: Interpersonal RelatednessCPAI: Interpersonal Relatedness(Chinese Tradition)(Chinese Tradition)

� Six scales:-

– HAR- Harmony

– FAC- Face

– REN- Ren Qing (Relationship Orientation)

– FLE- Flexibility

– MOD- Modernization

– T_E- Thrift-Extravagance

CPAI: IndividualismCPAI: Individualism

� Three scales:

– S_S- Self vs. Social Orientation

– L_A- Logical vs. Affective Orientation

– DEF- Defensiveness (Ah-Q Mentality)

CPAI: Method of AnalysisCPAI: Method of Analysis

� General steps (Chan & Bentler, 1993 for details):

– a) Propose the pre-ipsative factor structures I N T E R I N D

H A R

F A C

R E N

F L E

M O D

T _ E

S _ S

L _ A

D E F

0

0

0

0

0

0

αααααααααααααααααααααααα

ββββββββββββ

1

2

3

4

5

6

0 7

0 8

0 9

– b) Calculate the ipsative factor structuresI N T E R I N D

H A R

F A C

R E N

F L E

M O D

T _ E

S _ S

L _ A

D E F

αααα αααα ββββαααα αααα ββββαααα αααα ββββαααα αααα ββββαααα αααα ββββαααα αααα ββββ

αααα ββββ ββββαααα ββββ ββββαααα ββββ ββββ

1

2

3

4

5

6

7

8

9

−−−− −−−−−−−− −−−−−−−− −−−−−−−− −−−−−−−− −−−−−−−− −−−−

−−−− −−−−−−−− −−−−−−−− −−−−

αααα αααα αααα αααα αααα αααα αααα

ββββ ββββ ββββ ββββ

= + + + + + + + +

= + + + + + + + +

( ) /

( ) /

1 2 3 4 5 6 0 0 0 9

0 0 0 0 0 0 7 8 9 9

– c) Constrains some loadings to be the same• Before ipsatization: Zero loadings

• After ipsatization: Equal loadings

– d) Delete one variable from analysis

– e) Calculate the matrix of error terms A=I-p-111T since they are systematically

correlated after ipsatization

� Before ipsatized (A) After ipsatized (A)1 0 0 0 0 0 0 0 0

0 1 0 0 0 0 0 0 0

0 0 1 0 0 0 0 0 0

0 0 0 1 0 0 0 0 0

0 0 0 0 1 0 0 0 0

0 0 0 0 0 1 0 0 0

0 0 0 0 0 0 1 0 0

0 0 0 0 0 0 0 1 0

0 0 0 0 0 0 0 0 1

0 888 111 111 111 111 111 111 111 111

111 0 888 111 111 111 111 111 111 111

111 111 0 888 111 111 111 111 111 111

111 111 111 0 888 111 111 111 111 111

111 111 111 111 0 888 111 111 111 111

111 111 111

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . .

− − − − − − − −− − − − − − − −− − − − − − − −− − − − − − − −− − − − − − − −− − − − 111 111 0 888 111 111 111

111 111 111 111 111 111 0 888 111 111

111 111 111 111 111 111 111 0 888 111

111 111 111 111 111 111 111 111 0 888

− − − −− − − − − − − −− − − − − − − −− − − − − − − −

. . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

– g) Fit the proposed model

– h) Calculate or “recover” the original factor loadings, standard errors of parameter estimates

CPAI Results: CPAI Results: GoodnessGoodness--ofof--fit Summariesfit Summaries

� Common CFA ignoring response bias: � χχχχ2 (N=1585) = 783.62, df=26, p<.001– CFI=0.780, NNFI=0.695, RMSEA=0.136,

AIC=731.62

� Ipsative method (bias was treated as response set): � χχχχ2 (N=1585) = 269.12, df=17, p<.001– CFI=0.905, NNFI=0.844, RMSEA=0.097,

AIC=235.12

CPAI Results: Factor LoadingsCPAI Results: Factor Loadings

� Common Method: Ipsative Method:

−−−−−−−−

97.10

60.10

37.10

026.1

011.1

027.2

018.1

084.1

056.1

Def

L_A

S_S

T_E

Mod

Fle

Ren

Fac

HarINDINTER

−−−−−−−−

17.00

98.20

42.10

057.2

094.1

058.1

007.2

008.1

070.2

Def

L_A

S_S

T_E

Mod

Fle

Ren

Fac

HarINDINTER

CPAI Results: Factor Correlation CPAI Results: Factor Correlation and Error Variancesand Error Variances

� Common Method:– Factor correlation = .824– Error Variances =

• [3.45 8.60 2.59 3.68 5.86 7.07 6.05 6.06 5.54]

� Ipsative Method:– Factor correlation = .732– Error Variances =

• [2.21 8.06 2.62 4.49 6.30 6.60 6.06 2.60 3.08]

CPAI Results: ImplicationsCPAI Results: Implications

� The ipsative method fits the data better!

� The directions are consistent in both methods

� Ipsative method can give us a clearer picture of the factor structures.

ConclusionsConclusions

� When there is response bias, ipsative method can be used to minimize its effects.

� When there is NO response bias, ipsative method works as BUT not optimal.

� Does response bias be the causes for the non-replicated factor structures across cultures?

Questions Are Welcomed!

� Selected References� Chan, W., & Bentler, P.M. (1993). The covariance structure analysis of

ipsative data. Sociological Methods & Research, 22, 214-247.

� Cheung, M.W.L. (1997). Covariance structure analysis of data with response set bias: A comparison of the ipsative and ordinary approaches. Unpublished Thesis.

� Clemans, W.V. (1966). An analytical and empirical examination of some properties of ipsative measures. Psychmetric Monographs, 14, 1-56.

� Paulhus, D. L. (1991). Measures of personality and social psychological attitudes. In J. P. Robinson, & R. P. Shaver (Eds.), Measures of Social Psychological Attitudes Series(Vol 1, p.17-59). San Diego: Academic Press Inc.