María José Rendón, B.A.

Stephanie Donnelly, M.Ed.

Billie Schwartz, M.A. Presenters

Guerda Nicolas, Ph.D. Chair

Janet Helms, Ph.D. Discussant

SYMPOSIUM OVERVIEW

• Studies of culturally and developmentally diverse groups have been critiqued for (mis)use of measures.

• This symposium examines measurement considerations in the assessment of culturally diverse children.

1. Equivalence and Bias Issues in Measurement

2. Developmental Considerations in Self-Report Data with Children

3. Adaptation of the Children’s Racial and Ethnic Socialization Scale (CRESS)

Using Measurement Instruments with Diverse Populations: A Review of Equivalence and Bias Issues in Measurement María José Rendón, B.A. University of Miami

OVERVIEW

• Equivalence, Bias, & Culturally Competent Assessment: Why Researchers Should Care

• Equivalence

Construct, Structure, Metric, Scalar

• Bias

Construct, Method, Item

• Minimizing Bias: Adaptation of Measures

• Testing for Bias: Psychometric Tools

• Measurement tools: Materializing the intangible…

• “Error” in measurement and our quest to minimize it.

EQUIVALENCE & BIAS: WHY BOTHER?

Construct

Instrument’s score

Error

Random error

Systematic error

•Measures of sociocultural identity cater to a specific population. •Disregarding equivalence and bias considerations threaten study credibility

AND is a disservice to your population of interest.

EQUIVALENCE & BIAS: WHY BOTHER?

• Cross-cultural psychology

– “Universal” vs. “Indigenous” perspectives

• Research with traditionally underrepresented groups in the U.S.

– “Universal” constructs imposed on ethnic / racial groups experiences

• Equivalence and bias relevant to minimize confounding factors in between-group differences.

Leong, Leung & Cheung (2010)

EQUIVALENCE

CONSTRUCT EQUIVALENCE •Does the construct exist in the population of interest? How is this construct operationalized?

STRUCTURAL EQUIVALENCE •Does the construct hold its relationships to related constructs? Does its factor structure hold?

METRIC EQUIVALENCE •Identical interval or ratio between units of measurement.

SCALAR EQUIVALENCE •Full scale equivalence: metric equivalence + origin equivalence.

Would racial socialization be culture-specific,

universal? Or partially equivalent?

Do the hypothesized dimensions for racial

socialization hold for the pop. of interest?

~For group comparison studies

~For group comparison studies

DO SCORES HAVE THE SAME MEANING ACROSS GROUPS?

(Van de Vijver & Leung, 2010; van de Vijver & Tanzer, 2004; Ægisdóttir et al., 2008)

CONSTRUCT BIAS •Incomplete overlap of construct operationalization across groups (though the idea is the same, the way it is expressed is culturally different).

•METHOD BIAS

•Sampling •Administration •Instrument

•ITEM BIAS •People with the same level of ability on the target trait have different scores

Does the expectation that racial socialization involves “promotion of cross-racial

relationships” hold for Caribbean immigrant

families? Is it relevant? Would lower scores

indicate this is low in the sample, or indicate that

the dimension is not relevant?

A consideration on construct bias

BIAS NUISANCE FACTORS INCREASING SYSTEMATIC ERROR

(Van de Vijver & Leung, 2010; van de Vijver & Tanzer, 2004)

SAMPLE BIAS •Differences in background variables (e.g., education, social desirability) beyond the grouping variable of interest (ethnicity). ADMINISTRATION BIAS •Differences in administration conditions (interview vs. survey method) •But consider: some groups will need interview method to be comparable in quality of responses to samples in which survey method was administered. INSTRUMENT BIAS •Different familiarity with response procedures.

Access to museums and resources in community

and school (beyond parental socialization practices at home).

Children needed interview aides to answer survey

format and explain concepts.

Scale modified for better understanding of

“magnitude”

BIAS NUISANCE FACTORS INCREASING SYSTEMATIC ERROR

ITEM BIAS •DIF: Differential item functioning •When people with the same level of ability on the target trait have different scores •Item does not capture the trait equally across groups.

Causes: •Poor item wording (ambiguity, low familiarity of item content in your sample, faulty translation) Consequences: •Respondents might misunderstand item or give responses that are not representative of their actual trait level

In measuring socialization into one’s group’s heritage, would this item be biased for children of Caribbean

immigrant families?

“In my family I am encouraged to watch

documentaries or movies on Black history”

BIAS NUISANCE FACTORS INCREASING SYSTEMATIC ERROR

MINIMIZING BIAS: ADAPTATION OF MEASURES

Concept Improving agreement of concepts across groups. E.g., Find the equivalent phrases to express the construct of racial identity across groups. Consult your local gatekeepers and your lit review.

Culture Adaptations to fit cultural norms and practices (e.g., behaviors unique to Caribbean groups in socializing children).

Language Applicable to translations (see van de Vijver & Leung, 2010), but also points to culture-specific conventions on the use of language (e.g., politeness)

Measurement Familiarity of group with assessment procedure (method bias).

A variety of adaptation models exist. We will point to Bernal & Roselló, 1995

(Van Vivjer & Leung, 2010)

TESTING FOR BIAS: PSYCHOMETRIC TOOLS

Testing for Construct & Structural Equivalence

Exploratory Factor Analyses Confirmatory Factor Analyses Path modeling (SEM)

Testing for Method Bias Include confounding variables into the design of the study. Conduct covariance or hierarchical regression analysis.

Testing for Item Bias Item Response Theory Mantel-Haenszel statistic SEM

(Sireci, Patsula & Hambleton, 2005; Corral & Landrine, 2010)

REFERENCES

• Ægisdóttir, S., Gerstein, L. H., & Çinarbaş, D. (2008). Methodological issues in cross-cultural counseling research: Equivalence, bias, and translations. The Counseling Psychologist, 36(2), 188-219. doi:10.1177/0011000007305384

• Brown, T. L., & Krishnakumar, A. (2007). Development and validation of the Adolescent Racial and Ethnic Socialization Scale (ARESS) in African American families. Journal of Youth and Adolescence, 36(8), 1072-1085. doi:10.1007/s10964-007-9197-z

• Leong, F. T. L., Leung, K., & Cheung, F. M. (2010). Integrating cross-cultural psychology research methods into ethnic minority psychology. Cultural Diversity and Ethnic Minority Psychology, 16 (4), 590–597.

• Sireci, S. G., Patsula, L. & Hambleton, R. K. (2005). Statistical methods for identifying flaws in the test adaptation process. In R. K. Hambleton, P. F. Merenda & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 93-115). Lawrence Erlbaum Associates, Mahwah, NJ

• Sue, S. (1996). Measurement, testing, and ethnic bias: Can solutions be found? In G. Sodowsky & J.C. Impara (Eds.), Multicultural assessment in counseling and clinical psychology (pp7-37). Lincoln, NB: Buros Institute of Mental Measurements.

• van de Vijver, F. R., & Leung, K. (2011). Equivalence and bias: A review of concepts, models, and data analytic procedures. In D. Matsumoto, F. R. van de Vijver, D. Matsumoto, F. R. van de Vijver (Eds.) , Cross-cultural research methods in psychology (pp. 17-45). New York, NY US: Cambridge University Press.

• van de Vijver, F., & Tanzer, N. K. (2004). Bias and equivalence in cross-cultural assessment: An overview. European Review of Applied Psychology/Revue Européenne de Psychologie Appliquée, 54(2), 119-135. doi:10.1016/j.erap.2003.12.004

LOGO

Obtaining Self-Report Data from Children: Developmental and Measurement Considerations

Stephanie Donnelly, M.Ed.

University of Miami

Children in Self-Report Research

What do we want to

know?

Experiences

Perceptions of Self

Perceptions of Others

Preferences

Etc.

Children in Self-Report Research

How do we decide the appropriate method to

get this information?

Presentation Overview

Trends in Self-Report Research with Children

Developmental Considerations

Theories of Self-Concept

Developmental Differences

Measurement Considerations

Format

Scaling

Administration

Initial research focused developmental difference between ages

Later, use of theories of cognitive development to understand

the structural organization of self

Focus on individual differences in self-concept limited due to

measurement difficulties

Recent efforts have focused on understanding identity

development but this is largely limited to adolescents

Trends in Self-Report Research with Children

Looking Glass Self (Cooley, 1902)

Social construction of self

As children age make more comparisons and more influenced by

evaluation of others

Global Self-Perception (Coopersmith, 1967)

Ex. Self-esteem

Multidimensional Self (Harter, 1996; March 1987)

Self evaluations in different areas of life

More complex, dimensional with age

Hierarchical Models (Epstein, 1973)

Organization of self concept

Overall self-concept influenced by underlying multidimensional evaluations

Developmental Considerations: Theories of Self-Concept

Very Early Childhood (3-4)

• Concrete, observable characteristics

• Lack of coherence

• Overly positive

Early Childhood (5-7)

• Focus on competencies

• All or none thinking

• Compares self with others

Late Childhood (8-11)

• Global evaluations

• Greater accuracy

• Importance of other’s opinions

Developmental Considerations: Developmental Differences (Harter, 1999, pg 36)

Four-year-old

“I live in a big house with my mother and father and sister, Lisa. I

have a kitty that is orange and a television in my own room. I

know all my A, B, C’s listen. . . . I can run faster than anyone! I

like pizza and have a nice teacher. I can count to 100.”

Eight-year-old

“I’m in the third grade this year and pretty popular at least with

the girls. That’s because I’m nice, helpful, and can keep

secrets.. . . . .I’m feeling pretty smart in certain subjects,

Language Arts and Social Studies. I got A’s in these subjects on

my last report card and was really proud of my myself. But I’m

feeling pretty dumb in Arithmetic and Science, particularly when

I see how well the other kids are doing.”

(Harter, 1996, pg 208)

Developmental Considerations: Examples

Measumrent Considerations

Measurement Considerations: Format

Measurement Considerations: Scaling

Picture Weights Line Marking

Draw a line on the

thermometer to

show how angry

you get when

another kid takes

your toy.

Measurement Considerations: Scaling

Picture Weights Bifurcated

Questions

Limit Choices

Measurement Considerations: Administration

Varies depending on age

Probe for understanding

Children often need

assistance to maintain

attention

Adaptations for a variety

of reading levels

Use Response Cards

Definition list

Use of advanced technology to collect data from children

Further efforts to validate data collection methodologies and

specific scales is needed

More consistency in use of strong measures

Comparison of self-report to other sources of information

Adaptation of measures for use with

ethnically diverse populations

Future Directions

Summary

Self-report research with children is possible!!

Consider developmental abilities when determining concepts to

measure and measurement methods.

Guide research choices- questions, measures, etc., on theory of

development.

Different methods may be needed to collect data from children

of different ages (even in the same sample).

Methods should keep children’s attention, test their

understanding, and be fun.

Cooley, C. H. (1902). Human nature and the social order. New York: Scribner’s.

Coopersmith, S. (1967). The antecedents of self-esteem. San Francisco, CA: Freeman.

Epstein, S. (1973). The self-concept revisited or a theory of a theory. American Psychologist,

28, 405-416.

Harter, S. (1996). Historical roots of contemporary issues involving self-concept. In B. A.

Bracken, B. A. Bracken (Eds.) , Handbook of self-concept: Developmental, social, and

clinical considerations (pp. 1-37). Oxford England: John Wiley & Sons.

Harter, S. (1996). Developmental changes in self-understanding across the 5 to 7 shift. In A.

J. Sameroff, M. M. Haith, A. J. Sameroff, M. M. Haith (Eds.) , The five to seven year shift: The

age of reason and responsibility (pp. 207-236). Chicago, IL US: University of Chicago Press.

Harter, S. (1998). The development of self-representations. In N. Eisenberg, W. Damon, N.

Eisenberg (Eds.) , Handbook of child psychology, 5th ed.: Vol 3. Social, emotional, and

personality development (pp. 553-617). Hoboken, NJ US: John Wiley & Sons Inc.

Harter, S. (1999). The construction of the self: A developmental perspective. New York, NY:

The Guilford Press.

Marsh, H. W. (1987). The hierarchical structure of self-concept and the application of

hierarchical confirmatory factor analysis. Journal of Educational Measurement, 24, 17-19.

References

The Children’s Racial and Ethnic Socialization Scale (CRESS): The Cultural and Developmental Adaptation of a Socialization Measure for Children Billie Schwartz, M.A.

University of Miami

Why study racial & ethnic socialization?

• Research has shown that Black children will face racism and discrimination that will impact their development

• The concept of identity in young children is currently debated—can identity be measured this young?

• Racial socialization is defined as messages taught or communicated regarding race and ethnicity

• The family is considered the primary socializing factor for kids

• Positive messages can lead to greater academic achievement and self-esteem and sense of identity

• Few researchers have attempted to combine these factors into a measure for children

Adolescent Racial and Ethnic Socialization Scale (ARESS) (Brown & Krishnakumar, 2007)

• Racial Socialization (RS) & Ethnic Socialization (ES)

• RS: 16items (i.e., racial barrier awareness, coping with racism and discrimination, the promotion of cross racial relationships)

• ES: 22 items (i.e., cultural values, cultural embeddedness, history, promotion of ethnic pride)

• Youth reported on a Likert scale of 1–4

• Questions asked about messages they’ve received from maternal/paternal caregivers

• To adapt the ARESS for use with for elementary-aged children

• Children’s Racial and Ethnic Socialization Scale (CRESS)

• By creating a scale for younger children:

• Identify problems at an earlier age

• Be better informed on children’s beliefs and attitudes regarding the self

• Help coping with and navigating racism and discrimination

• Create an effective assessment tool to better understand the messages that children receive about their racial and ethnic heritage

Purpose of the Study

Adaptation Procedures

• First group of young children given the ARESS without changes

• Children reside in the Miami-Dade County area.

• 2009-2010 academic school year

• Group served as a focus group providing feedback at four week intervals

• Preliminary analysis showed difficulties three distinct areas requiring further adaptation

• words, language and content

• the concept of a Likert scale

• administration method

Adaptation: Language & Content

•The Ecological Validity and Culturally Sensitive Framework - Bernal and RossellÓ (1995)

•Vocabulary level not developmentally appropriate for use with children

•Definition list for administrator

•3 Qualitative questions :

•Better understand who is talking to the kids

•The length was reduced: 35-items to 19-items (quantitative)

•Subscale of Cultural Embeddedness removed for content with young children

Bernal et al., 1995

Elements Culturally Sensitive Language Culturally appropriate, culturally

syntonic language Metaphors Symbols and concepts shared with

the populations Content Cultural knowledge: values,

traditions; uniqueness of groups Concepts Treatment concepts consonant

with culture and context Goals Support of adaptive values from

culture of origin Methods Development and cultural

adaptation of treatment methods Context Consideration of changing

contexts: acculturative stress, phase of migration, social supports and relationship to country of origin, etc.

Original Items ARESS Adapted Items CRESS

My maternal/paternal caregiver encourages me to have Black friends

How much are you told to have friends that have the same skin color as you?

My maternal/paternal caregiver teaches me the importance of family loyalty

How much does someone talk to you about the importance of sticking together as a family?

My maternal/paternal caregiver teaches me that my skin color is beautiful

How much are you told that your skin color is beautiful?

Examples of Language Changes

Adaptation: Scaling & Administration

Scaling

•Participants had difficulties with the Likert format and marked only the extremes

•Reduction from 4 options to 3

•Introduction of picture scale

Administration

•Changed from a self-report administration to a semi-structured interview

•Depending on reading level, either one-to-one or small group of 2-4 to one adult

Summary

• The CRESS is an assessment of racial and ethnic socialization messages received by elementary-aged Black children

• Changes in:

• wording and content

• scaling

• administration style

• CRESS helps to understand the messages kids receive about their racial and ethnic identity

• Not a measure of racial and/or ethnic identity

• Future studies will examine the psychometric properties

Bernal, G., Bonilla, J., & Bellido, C. (1995). Ecological validity and cultural sensitivity for outcome research: Issues for the cultural adaptation and

development of psychosocial treatments with Hispanics. Journal of Abnormal Child Psychology, 23, 67–87.

Brown, T. L., & Krishnakumar, A. (2007). Development and validation of the Adolescent Racial and Ethnic Socialization Scale (ARESS) in African American

families. Journal of Youth and Adolescence, 36(8), 1072–1085.

Brown, T., Linver, M., Evans, M. & DeGennaro, D. (2009). African-American parents’ racial and ethnic socialization and adolescent academic grades:

teasing out the role of gender. Journal of Youth Adolescence, 38, 214-227.

Clark, K. and Clark, M. (1939). The development of consciousness of self and the emergence of racial identification in Negro preschool children. Journal of

Social Psychology, 10, 591-599.

Clark, K. B. and Clark, M. K. (1947). Racial identification and preference. Readings in Social Psychology, 16-178.

Coard, S., Foy-Watson, S., Zimmer, C. & Wallace, A. (2007). Considering culturally relevant parenting practices in intervention development and

adaptation: a randomized controlled trial of the Black Parenting Strengths and Strategies (BPSS) Program. The Counseling Psychologist, 35, 797-820.

Coard, S. I., & Sellers, R. M. (2005). African American families as a context for racial socialization. In V. C. McLoyd, N. E. Hill, & K. A. Dodge (Eds.), African

American family life: Ecological and cultural diversity (pp. 264-284). New York: Guilford.

Coard, S. I., Wallace, S. A., Stevenson, H. C., & Brotman, L. M. (2004). Towards culturally relevant preventative interventions: The consideration of racial

socialization in parent training with African American families. Journal of Child and Family Studies, 13(3), 277–293.

Hughes, D. & Johnson, D. (2001). Correlates in children's experiences of parents' racial socialization behaviors. Journal of Marriage and Family, 63(4), 981-

995.

Hughes, D., Smith, E. P., Stevenson, H. C., Rodriquez, J., Johnson, D. J., & Spicer, P. (2006). Parents’ ethnic and racial socialization practices: A review of

research and directions for future study. Developmental Psychology,42(5), 747–770.

Lesane, C. L. (2002). Race socialization in Black families: A selective review of the literature. African American Research Perspectives, 8(1), 27–34.

Marshall, S. (1995). Ethnic socialization of African American children: Implications for parenting, identity development, and academic achievement. Journal

of Youth and Adolescence, 24(4), 377–396.

Phillips Smith, E., Walker, K., Fields, L., Brookins, C. & Seay, R. (1999). Ethnic identity and its relationship to self-esteem, perceived efficacy and prosocial

attitudes in early adolescence. Journal of Adolescence, 22, 867-880.

Phinney, J. S. (1990). Ethnic identity in adolescents and adults: Review of research. Psychological Bulletin, 108, 499-514.

Stevenson, H., Reed, J., Bodison, P., & Bishop A. (1997). Racism stress management: racial socialization beliefs and the experience of depression and

anger in African American youth. Youth Society, 29(2), 197-222.

Thomas, A. J., & Speight, S. L. (1999). Racial identity and racial socialization attitudes of African American parents. Journal of Black Psychology, 25(2), 152–

170.

References

Questions?

Thank you!