Assessing Cross- Cultural Competence: How Good Are the Available Instruments? Institute for Cross- Cultural Management Florida Institute of Technology

Assessing Cross-

Cultural Competence:

How Good Are the Available Instruments?

Assessing Cross-

Cultural Competence:

How Good Are the Available Instruments?

Institute for Cross-Cultural Management

Florida Institute of Technology

Institute for Cross-Cultural Management

Florida Institute of Technology

2Florida Tech – Institute for Cross-Cultural Management

AuthorsAuthors

• William Gabrenya, PhD - Cross-Cultural Psychology (USA)

• Rana Moukarzel, MS - Industrial/Organizational Psychology (Lebanon)

• Marne Pomerance, MS – I/O (USA)• Richard Griffith, PhD – I/O (USA)

• William Gabrenya, PhD - Cross-Cultural Psychology (USA)

• Rana Moukarzel, MS - Industrial/Organizational Psychology (Lebanon)

• Marne Pomerance, MS – I/O (USA)• Richard Griffith, PhD – I/O (USA)


SupportSupport

• DEOMI Contract FA2521-10-T-0087 • Dan McDonald, PhD• Patrice Reid, PhD

• DEOMI Contract FA2521-10-T-0087 • Dan McDonald, PhD• Patrice Reid, PhD


Cross-Cultural Competence and the U.S. Military

Cross-Cultural Competence and the U.S. Military

• Cross-Cultural Psychology watches Iraq disintegrate– Case study in worst practices– Sidelined culture experts– How could this happen?

• Cross-Cultural Psychology watches Iraq disintegrate– Case study in worst practices– Sidelined culture experts– How could this happen?


Culture Expertise Rediscovered

Culture Expertise Rediscovered

• Defense Language Office (DLO)• DEOMI• Army Research Institute• Services• DLO project to try to keep track of it all…

• Defense Language Office (DLO)• DEOMI• Army Research Institute• Services• DLO project to try to keep track of it all…


Three Issues in Cross-Cultural Competence

Three Issues in Cross-Cultural Competence

1. Identify competencies2. Measure competencies3. Train competencies4. Utilize capabilities acquired at

considerable cost

1. Identify competencies2. Measure competencies3. Train competencies4. Utilize capabilities acquired at

considerable cost


DLO FrameworkDLO Framework

• “Framework for Cross-Cultural Competency”

• Competency model– Organized set of requirements for acceptable

performance– Higher level generality to lower level, hierarchy– Expressed behaviorally

• Two parts– 3C competencies– Antecedent variables (enablers)

• Revised

• “Framework for Cross-Cultural Competency”

• Competency model– Organized set of requirements for acceptable

performance– Higher level generality to lower level, hierarchy– Expressed behaviorally

• Two parts– 3C competencies– Antecedent variables (enablers)

• Revised


CompetenciesCompetencies

Name (# elements) Example

C1 Culture-General Concepts and Knowledge (3)

Comprehends and navigates intercultural dynamics

C3 Cultural Perspective-Taking (3)

Understands how one’s own group is viewed by members of another group

C4 Communication (2) Acquires and applies knowledge and concepts of intercultural communication skills

C5 Interpersonal Skills (2)

Builds relationships in support of mission performance

C6 Cultural Adaptability (2)

Understands the implications of one’s actions and adjusts approach to maintain relationships with other groups, or cultures


Antecdents (Enablers)Antecdents (Enablers)

Name (# elements) Example

E1 Cognitive Bias Resilience (3)

Accepts, or does not feel threatened by, ambiguous situations and uncertainty.

E2 Emotional Resilience (4)

Tolerates emotionally shocking, frustrating, or exhausting circumstances…

E3 Self-Identity Resilience (3)

Demonstrates ability to maintain personal values independent of situational factors

E4 Learning Motivation (4)

Is motivated to make sense of inconsistent information about social rules and norms…

E5 Social Interaction (4) Actively seeks out and explores unfamiliar cross-cultural interactions …


Validation* of the FrameworkValidation* of the Framework

• Content validity– The right competencies?

• Criterion validity– Are these competencies related to performance

and adjustment?

• *Analogously

• Content validity– The right competencies?

• Criterion validity– Are these competencies related to performance

and adjustment?

• *Analogously


Content Evaluation of the DLO Framework

Content Evaluation of the DLO Framework

• Developed using subject matter experts (SMEs):– Returned soldiers– Experts in expatriate assignments– Military psychologists

• We compared it to:– Military models– Civilian models– New SME research

• Short answer: content is good

• Developed using subject matter experts (SMEs):– Returned soldiers– Experts in expatriate assignments– Military psychologists

• We compared it to:– Military models– Civilian models– New SME research

• Short answer: content is good


Criterion EvaluationCriterion Evaluation

• Difficult…– The research is civilian– The measurement is poor

• Imprecise…– Mapping a competency model to common

constructs

• Short answer: Middling support

• Difficult…– The research is civilian– The measurement is poor

• Imprecise…– Mapping a competency model to common

constructs

• Short answer: Middling support


This Report: Evaluate the quality of instruments that

can assess Framework competencies

This Report: Evaluate the quality of instruments that

can assess Framework competencies

• Using “off the shelf” instruments• Using non-cultural instruments

– Personality, cognitive, etc.

• Using “off the shelf” instruments• Using non-cultural instruments

– Personality, cognitive, etc.


Overview of the ProcedureOverview of the Procedure

1. Deconstruct the competency model to elemental form

2. Map elements to commonly studied constructs

3. Find “all” plausibly useful instruments4. Evaluate the quality of the instruments5. Map instruments to Framework elements

1. First: All instruments2. Second: Only the good ones

6. Tally it up

1. Deconstruct the competency model to elemental form

2. Map elements to commonly studied constructs

3. Find “all” plausibly useful instruments4. Evaluate the quality of the instruments5. Map instruments to Framework elements

1. First: All instruments2. Second: Only the good ones

6. Tally it up


1. Deconstruct the Framework

1. Deconstruct the Framework

• Element• Aka “component” in competency modeling• C5: Interpersonal Skills• “[C5.1] Develops and maintains rapport.

Builds relationships in support of mission performance. // [C5.2] Manage and resolve conflict in support of mission objectives.”

• Element• Aka “component” in competency modeling• C5: Interpersonal Skills• “[C5.1] Develops and maintains rapport.

Builds relationships in support of mission performance. // [C5.2] Manage and resolve conflict in support of mission objectives.”


2. Map the elements to constructs

2. Map the elements to constructs

• What existing, researched construct does this competency seem to be related to?

• What existing, researched construct does this competency seem to be related to?

Competency 1

Competency 2

Enabler 2

Framework Constructs Measures

Competency element 1.1



Enabler element 2.1

Enabler element 2.2

Personality Construct 1

Personality Construct 2Competency element 1.2

Enabler 1

Enabler element 1.1

Enabler element 1.2

Attitude Construct 1


Cognition Construct 1


Abilities Construct 1


Instrument/Subscale 1








Competency 3: Cultural Perspective Taking

Competency element 3.1: Demonstrates an awareness of one’s own worldview… and how one’s own group is viewed…

Competency element 3.3: Takes the cultural context into consideration when interpreting situational cues.

• Knowledge of attributed stereotypes

• Self-insight• Perspective

taking

• Metacognition• Situational

Awareness

• MAKSS-Awareness

• ICC-Awareness• SEE-Empathic

perspective taking

• SEE-Empathic Awareness

• BEVI-Sociocultural Closure

• (Perspective taking)

• CQS-Metacognition


3. Find ALL Instruments3. Find ALL Instruments

• Hunting and gathering in a vast literature• Two kinds:

– Single-construct– Batteries

• Three business models (why this matters)– Open– Restricted– Proprietary

• Hunting and gathering in a vast literature• Two kinds:

– Single-construct– Batteries

• Three business models (why this matters)– Open– Restricted– Proprietary


Many InstrumentsMany Instruments

• Don’t trust the published lists• We unfairly divided scales into “primary”

and “secondary”

• Don’t trust the published lists• We unfairly divided scales into “primary”

and “secondary”


Primary InstrumentsPrimary Instruments

Name Subscales

CCAI Cross-Cultural Adaptability Inventory

Flexibility/Openness (FO); Emotional Resilience (ER); Perceptual Acuity (PAC) ； Personal Autonomy (PA)

CQS Cultural Intelligence Scale Metacognition ； Cognition ； Motivation ； Behavior

ICAPS Intercultural Adjustment Potential Scale

Emotion Regulation (ER); Openness (OP); Flexibility (FL); Creativity (CT)

IDI Intercultural Development Inventory

Denial/Defense (DD); Reversal [R]; Minimization (M); Acceptance/Adaptation (AA); Cultural Disengagement (CD)

MPQ Multicultural Personality Questionnaire

Cultural Empathy; Openmindedness; Social Initiative; Emotional Stability; Flexibility

ICC Intercultural Communicative Competence

Complex structure: 3 domains; 5 dimensions; 4 developmental levels

INCA Intercultural Competence Assessment 6 subscales


Name SubscalesCCAI Cross-Cultural

Adaptability InventoryFlexibility/Openness (FO); Emotional Resilience (ER); Perceptual Acuity (PAC) ； Personal Autonomy (PA)

IES Intercultural Effectiveness Scale

Continuous Learning; Interpersonal Engagement; Hardiness

ISS Intercultural Sensitivity Scale

Interaction Engagement; Respect for Cultural Differences; Interaction Confidence; Interaction Enjoyment; Interaction Attentiveness

SEE Scale of Ethnocultural Empathy

Empathic Feeling and Expression (EFE); Empathic Perspective Taking (EP); Acceptance of Cultural Differences (AC); Empathic Awareness (EA)


Secondary InstrumentsSecondary InstrumentsInstrument

ADS Adjustment Difficulties Subscale

AIC Beliefs, Events, and Values Inventory

ASSIS Acculturative Stress Scale for International Students

BASIC Behavioral Assessment Scale for Intercultural Communication Effectiveness

BEVI Beliefs, Events, and Values Inventory

CCSI Cross-Cultural Social Intelligence

CGAIC Culture-Generic Approach to Intercultural Competence

CWQ The Culture in the Workplace Questionnaire

EMMIC European Multidimensional Models of Intercultural Competence

GAP Test

Global Awareness Profile

ICSI Intercultural Sensitivity Inventory

Instrument

IRC Intercultural Readiness Checklist

IRI Interpersonal Reactivity Index

ISAS The Inventory of Student Adjustment Strain

MAKSS Multicultural Competence Scale

MASQUE

Munroe Multicultural Attitude Scale Questionnaire

MCI Multicultural Competence Scale

MCKAS Multicultural Competence Scale

PCAT Peterson Cultural Awareness Test

PCSI Peterson Cultural Style Indicator

SCS Social Connectedness Scale

WDS Workplace Diversity Survey


Validation of the InstrumentsValidation of the Instruments

• Primary– Face– Construct– Criterion

• Secondary– Criterion

• Primary– Face– Construct– Criterion

• Secondary– Criterion


Face ValidityFace Validity

• Good: Most instruments

• Bad: ICAPS (Intercultural Adjustment Potential Scale)

• Moderate: CQS (Cultural Intelligence Scale)

• Good: Most instruments

• Bad: ICAPS (Intercultural Adjustment Potential Scale)

• Moderate: CQS (Cultural Intelligence Scale)


Construct ValidityConstruct Validity

• Convergent validity– Similar to what it should be similar to?

• Divergent validity– Different than what it should be different than?

• Structure– If subscales: support for dimensionality?

• Convergent validity– Similar to what it should be similar to?

• Divergent validity– Different than what it should be different than?

• Structure– If subscales: support for dimensionality?


Construct Validity FindingsConstruct Validity Findings

• (Primary scales only)• Missing information• Diverse outcomes, moderate validity• Mushy• Method variance not accounted for

• (Primary scales only)• Missing information• Diverse outcomes, moderate validity• Mushy• Method variance not accounted for


Construct Validity FindingsConstruct Validity Findings

• Good: MPQ (Multicultural Personality Questionnaire)

• Mixed: most instruments• Bad: ICAPS, ISS (Intercultural Sensitivity

Scale)• Unknown: GCI, IES, INCA, ICC (mainly

proprietary)


• Mixed: most instruments• Bad: ICAPS, ISS (Intercultural Sensitivity

Scale)• Unknown: GCI, IES, INCA, ICC (mainly

proprietary)


Criterion ValidityCriterion Validity

• Does the scale predict something real?– Adjustment: psychological or sociocultural– Performance– Response to training in experiments

• Known-groups– Do groups differ on the scale the way they

should?

• Does the scale predict something real?– Adjustment: psychological or sociocultural– Performance– Response to training in experiments

• Known-groups– Do groups differ on the scale the way they

should?


Criterion Validity FindingsCriterion Validity Findings


• Moderate: CCAI, CQS, SEE• Mixed: ICAPS• Poor: ISS• Unknown: GCI, IES, INCA


• Moderate: CCAI, CQS, SEE• Mixed: ICAPS• Poor: ISS• Unknown: GCI, IES, INCA


Conclusion: ValidityConclusion: Validity

• Few scales with overall high validity• Many highly desirable scales have little

support

• Few scales with overall high validity• Many highly desirable scales have little

support


5. Map (good) instruments to Framework elements

5. Map (good) instruments to Framework elements

• So: can the Framework be assessed?

• Strategy: Estimate extent to which each element is assessed using high and moderate validity instruments

• So: can the Framework be assessed?

• Strategy: Estimate extent to which each element is assessed using high and moderate validity instruments

Competency 1

Competency 2

Enabler 2

Framework Constructs Measures




Enabler element 2.1

Enabler element 2.2

Personality Construct 1

Personality Construct 2Competency element 1.2

Enabler 1

Enabler element 1.1

Enabler element 1.2















Competency 3: Cultural Perspective Taking

Competency element 3.1: Demonstrates an awareness of one’s own worldview… and how one’s own group is viewed…

Competency element 3.3: Takes the cultural context into consideration when interpreting situational cues.

• Knowledge of attributed stereotypes

• Self-insight• Perspective

taking

• Metacognition• Situational

Awareness

• MAKSS-Awareness

• ICC-Awareness• SEE-Empathic

perspective taking

• SEE-Empathic Awareness

• BEVI-Sociocultural Closure

• (Perspective taking)

• CQS-Metacognition


Score Card: CompetenciesScore Card: Competencies

Competency Name (# elements) Rating

C1 Knowledge (3) *

C3 Perspective-Taking (3) ***

C4 Communication (2) **

C5 Interpersonal Skills (2) **

C6 Cultural Adaptability (2) ****


Score Card: EnablersScore Card: Enablers

Enabler Name Rating

E1.1 Ambiguity tolerance (****)

E1.2 Need for closure (****)

E1.3 Suspending judgment *

E1.4 Inclusiveness **

E2.1 Stress resilience **

E2.2 Emotion regulation ****

E3.1 Self-confidence (****)

E3.2 Self-identity ****

E3.3 Optimism (**)


More EnablersMore Enablers

Enabler Name (# elements) Rating

E4.1 Learning through observation (3)

0

E4.2 Inquisitiveness (*)

E5.1 Social flexibility (3) ****

E5.2 Willingness to engage ***


So How Do We Measure a Competency?

So How Do We Measure a Competency?

• Plan A: off the shelf self-report measures– Mixed value– Various problems with self-report measures

• Faking, response biases, cognitive/declarative knowledge, etc.

• Plan B: behavior-based assessment centers– Designed for each competency– Difficult, expensive, slow

• Plan A: off the shelf self-report measures– Mixed value– Various problems with self-report measures

• Faking, response biases, cognitive/declarative knowledge, etc.

• Plan B: behavior-based assessment centers– Designed for each competency– Difficult, expensive, slow


Another Way…Another Way…

• 1. Rewrite the Framework as a blended competency/causal model

• Competency model:– Just the right competencies for the job

• Causal model:– Related competencies and enablers in models– Add moderators; mediation

• 1. Rewrite the Framework as a blended competency/causal model

• Competency model:– Just the right competencies for the job

• Causal model:– Related competencies and enablers in models– Add moderators; mediation


• 2. Assess the model, not just the competencies

• For each competency, assess several variables in the model – “Triangulate” on the individual’s competency– Relieves the pressure on assessing the

competency itself

• 2. Assess the model, not just the competencies

• For each competency, assess several variables in the model – “Triangulate” on the individual’s competency– Relieves the pressure on assessing the

competency itself


Blended ModelBlended Model

Openness

Multicultural Attitudes

E4.2: Culture Inquisitiveness

E4.1.2: Learning Through

Observation

C1.1a: Culture-General

Knowledge

Need for Cognition C: Competencies

Requiring KnowledgeSituational &

External Antecedents


finfin

Documents

Assessing Cross- Cultural Competence: How Good Are the Available Instruments? Institute for Cross- Cultural Management Florida Institute of Technology