Testing of English- and Testing of English- and French-Speaking Students French-Speaking Students

in Canada /in Canada /

Évaluation des élèves Évaluation des élèves anglophones et des élèves anglophones et des élèves francophones au Canadafrancophones au Canada

Kadriye Ercikan, UBC, Marielle Simon, Kadriye Ercikan, UBC, Marielle Simon, uOttawa, Maria Elena Oliveri, UBC & uOttawa, Maria Elena Oliveri, UBC &

Normand Dufour, MÉLSNormand Dufour, MÉLS

Canadian Society for the Study of EducationCanadian Society for the Study of Education

Canadian Education Researchers AssociationCanadian Education Researchers Association

May-June 2010, Concordia University, MontrealMay-June 2010, Concordia University, Montreal

ObjectiveObjective

Describe the context, challenges, Describe the context, challenges, actual procedures, other solutions actual procedures, other solutions and implications for achieving and implications for achieving comparability of English and French comparability of English and French student test results on large-scale student test results on large-scale assessments (LSAs) in Canadaassessments (LSAs) in Canada

ContextContext

Outside of Quebec, Francophone students Outside of Quebec, Francophone students form between 1 to 34% of the population form between 1 to 34% of the population

In Quebec, Anglophone students make up In Quebec, Anglophone students make up 19% of the population19% of the population

Anglophone and Francophone minority Anglophone and Francophone minority groups learn and are tested in their groups learn and are tested in their respective language. respective language.

Francophone students are provided Francophone students are provided education in French in most jurisdictionseducation in French in most jurisdictions

Only New Brunswick officially adopted the Only New Brunswick officially adopted the two national languagestwo national languages

ContextContext

French Immersion programs are made widely French Immersion programs are made widely available in all the provincesavailable in all the provinces

Comparative comments often made: “As was the Comparative comments often made: “As was the case in PISA 200o and PISA 2003, students case in PISA 200o and PISA 2003, students enrolled in the French-language school systems in enrolled in the French-language school systems in Nova Scotia, New-Brunswick, Ontario and Nova Scotia, New-Brunswick, Ontario and Manitoba performed significantly lower in reading Manitoba performed significantly lower in reading than did students in the English-language systems than did students in the English-language systems in the same provinces” (Government of Canada, in the same provinces” (Government of Canada, 2008 web site).2008 web site).

Test scores are used for obtaining diploma, Test scores are used for obtaining diploma, scholarship decisions, university entrance scholarship decisions, university entrance selection, etc.selection, etc.

Context

7 out of 10 jurisdictions surveyed in 7 out of 10 jurisdictions surveyed in 2007 said comparability was intended 2007 said comparability was intended

1 jurisdiction’s tests are developed 1 jurisdiction’s tests are developed and administered independentlyand administered independently

Holistic scoring by teachers in 1 Holistic scoring by teachers in 1 jurisdiction needs validation jurisdiction needs validation committees to ensure comparabilitycommittees to ensure comparability

The two language versions of a test The two language versions of a test may be developed based on different may be developed based on different target curricula, content specifications target curricula, content specifications or teaching resourcesor teaching resources

Procedures

Comparability verification procedures Number of jurisdictions

Back translation 2

Review of language versions by experts

8

Review of language versions by independent groups

4

Pilot-testing with students 6

Comparing item difficulties 4

Using cultural sensitivity panels 4

Challenges

Ideal conditions for scores to be comparable: Ideal conditions for scores to be comparable: Equivalence of testsEquivalence of tests

construct definitionsconstruct definitions itemsitems

Equivalence of scoresEquivalence of scoresthrough parallel teststhrough parallel tests

Comparability of scores through linking Comparability of scores through linking proceduresprocedures

simultaneous calibrationsimultaneous calibrationanchor item linkinganchor item linking

ChallengesChallenges

Concern with comparability of scores is Concern with comparability of scores is relatively recent;relatively recent;

There is great variation in provincial There is great variation in provincial practices;practices;

There is preference for development and There is preference for development and validation committees over statistical validation committees over statistical procedures;procedures;

The small minority language sample sizes The small minority language sample sizes limit: limit: • the types of psychometric procedures to the types of psychometric procedures to

create accurate score scales for the create accurate score scales for the minority students; minority students;

• linking score scales; linking score scales; • demonstration of equivalence.demonstration of equivalence.

Potential Solutions

Adaptation solutions: Adaptation solutions: Use parallel, successive (Rogers et al, Use parallel, successive (Rogers et al,

2003), concurrent (Solano-Flores et al, 2003), concurrent (Solano-Flores et al, 2002) or simultaneous (Tanzer, 2005) 2002) or simultaneous (Tanzer, 2005) test development strategiestest development strategies

Expert reviewsExpert reviewsContent (Bolwes et al, 2008)Content (Bolwes et al, 2008)Linguistic (for equivalence of meaning, Linguistic (for equivalence of meaning,

cognitive requirements, difficulty of cognitive requirements, difficulty of vocabulary and expressions and cue vocabulary and expressions and cue given to answer items)given to answer items)

Potential Solutions

Psychometric evidence based onPsychometric evidence based onClassical test theory Classical test theory Unidimensionality analyses (Oliveri Unidimensionality analyses (Oliveri

et al, in press)et al, in press)Identification of DIF and its sourcesIdentification of DIF and its sources

Student cognitive processes using Student cognitive processes using think aloud processes (TAPs) (Ercikan think aloud processes (TAPs) (Ercikan et al, 2010)et al, 2010)

Final NoteFinal Note

We reviewed and summarized research, We reviewed and summarized research, practices and challenges and recommended practices and challenges and recommended methodologies for examining comparability methodologies for examining comparability of test items and tests given in multiple of test items and tests given in multiple languages. languages.

We want to highlight the importance of We want to highlight the importance of optimizing comparability in the development optimizing comparability in the development phase of assessments. phase of assessments.

We also want to encourage further research We also want to encourage further research to investigate the comparability of cognitive to investigate the comparability of cognitive processes of students from different processes of students from different language groups as integral components of language groups as integral components of validity investigations.validity investigations.

ReferencesReferences Bowles, M. S. & Stanfield, C. W. (2008). A Practical Guide to Bowles, M. S. & Stanfield, C. W. (2008). A Practical Guide to

Standards-Based Assessment in the Native Language. Retrieved Standards-Based Assessment in the Native Language. Retrieved on March 28, 2008 from on March 28, 2008 from .ncela.gwu.edu/spotlight/LEP/2008/bowles_stansfield.pdf

Ercikan, K., Arim, R.,G., Law, D. M., Lacroix, S., Gagnon, F., & Ercikan, K., Arim, R.,G., Law, D. M., Lacroix, S., Gagnon, F., & Domene, J. F. (2010). Application of think-aloud protocols in Domene, J. F. (2010). Application of think-aloud protocols in examining sources of differential item functioning. examining sources of differential item functioning. Educational Educational Measurement: Issues and PracticeMeasurement: Issues and Practice, , 2929, 24-35. , 24-35.

Rogers, W. T., Gierl, M .J., Tardif, C., Lin, J., & Rinaldi, C. M. Rogers, W. T., Gierl, M .J., Tardif, C., Lin, J., & Rinaldi, C. M. (2003). Differential validy and utility of successive and (2003). Differential validy and utility of successive and simultaneous approaches to the development of equivalent simultaneous approaches to the development of equivalent achievement tests in French and English. achievement tests in French and English. Alberta Journal of Alberta Journal of Educational Research, 49Educational Research, 49(3), 290-304.(3), 290-304.

Solano-Flores, G,, Trumbull, E., & Nelson-Barber, S. (2002). Solano-Flores, G,, Trumbull, E., & Nelson-Barber, S. (2002). Concurrent Development of Dual Language Assessments: An Concurrent Development of Dual Language Assessments: An Alternative to Translating Tests for Linguistic Minorities Alternative to Translating Tests for Linguistic Minorities International Journal of Testing , 2, 2, 107 – 129., 107 – 129.

Tanzer, N.K. (2005). Developing tests for use in multiple Tanzer, N.K. (2005). Developing tests for use in multiple languages and cultures: A plea for simultaneous development. languages and cultures: A plea for simultaneous development. In: R.K. Hambleton, P.F. Merenda & C.D. Spielberger (Eds.), In: R.K. Hambleton, P.F. Merenda & C.D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural Adapting educational and psychological tests for cross-cultural assessment. assessment. Mahwah, NJ: Erlbaum.Mahwah, NJ: Erlbaum.