USING THE CEFR SCALES TO ASSESS STUDENTS’ PROFICIENCY LEVELS IN A PREPARATORY YEAR CONTEXT OUTSIDE EUROPE
Ebtesam AbdulhaleemPhD Student
Supervised by:Prof. Dr. Claudia Harsch and Dr. Neil Murray
The Study Aim
To explore the potential of self- and tutors’ assessment in determining the proficiency levels of EFL medical and healthcare students in Saudi Preparatory Year Programme (PYP).
Part of a bigger PhD project in which the CEFR is used to perform needs analysis to identify potential gaps.
Overview
Background Information
Concerns
The study purpose
Participants
Methods
Data Analyses
Results
Conclusion
Background Information
Matriculated University Students:
Enrol at the Preparatory Year Programme (PYP)
Placed in three levels (elementary, intermediate and advanced)
Take a high-stakes standardised exit exam at the end of the year.
Join different colleges based on their GPA.
Background Information PYP ELSD Beginner Elementary Pre-
Intermediate
Pre-
Intermediate
Plus
Intermediate Intermediate
Plus
Advanced
CEFR A1 A1-A2 A2 A2-B1 B1 B2 C1
Track Levels
Sem
este
r1
Quarter 1 Quarter 2
Sem
este
r2
Quarter 3 Quarter 4
Me
dic
al/
He
alt
hca
re t
rack Elementary
Elementary
A1-A2
Pre-
Intermediate
Pre-
Intermediate
Plus
Intermediate
B1
Intermediate
Pre-
Intermediate
A2
Pre-
Intermediate
Plus
Intermediate Intermediate
Plus
B1-B2
Advanced Intermediate
B1
Intermediate
Plus
Upper
Intermediate
Advanced
C1
Noticed Concerns:
The students’ results in the final writing exam are of concern
Difficult to have a reliable test that can discriminate the students’ writing proficiency levels.
The Study Purposes:
Identify the students’ writing proficiency levels at the end of PYP as perceived by the students and their tutors
Explore and compare how PYP students and their tutors used the CEFR scales to identify the students’ proficiency levels
By using the CEFR can-do statements, potentially able to gain more detailed insight of students’ actual level of written proficiency, compared to when a standard grade awarded.
From the Literature
To the best of my knowledge, comparing CEFR self- and tutors’ assessment has not been explored empirically in the Saudi context
Using functional language (e.g. can do) increases the accuracy of self-assessment (Ross, 1998)
Advanced students tend to underestimate their levels compared to less proficient students (Sahragard & Mallahi, 2014)
Low proficient students are less accurate in their judgment of their language proficiency (Babaii, Taghaddomi & Pashmforoosh, 2016)
At the end of the academic year 2015
473 PYP students from the medical and healthcare track (All levels)
19 English tutors teaching these students
Participants
Methods
10 selected CEFR scales focusing on writing skills
The students were asked to self-assess themselves
They were given detailed instructions on how to use the scales in addition to an audio-video instructional guide
Meetings with the tutors to explain how the scales should be used
Tutors were asked to assess the same students using the same scales
1 2 3 4 5 6 7 8 9
Ov
erall
writ
ten
pro
du
cti
on
I can write simple isolated phrases and
sentences.
I can write a series of simple phrases and
sentences linked with
simple connectors like ‘and’, ‘but’ and
‘because’.
I can write straightforward
connected texts on a
range of familiar subjects within my field of
interest, by linking a
series of shorter discrete elements into a linear
sequence.
I can write clear, detailed texts on a variety of
subjects related to my
field of interest, synthesising and
evaluating information
and arguments from a number of sources.
I can write clear, well-structured texts of
complex subjects,
underlining the relevant salient issues, expanding
and supporting points of
view at some length with subsidiary points,
reasons and relevant
examples, and rounding off with an appropriate
conclusion.
I can write clear, smoothly flowing,
complex texts in an
appropriate and effective style and a logical
structure which helps the
reader to find significant points.
Ov
erall
writ
ten
in
teracti
on
I can ask for or pass on personal details in
written form.
I can write short, simple formulaic notes relating
to matters in areas of
immediate need.
I can convey information and ideas on abstract as
well as concrete topics,
check information and ask about or explain
problems with
reasonable precision. I can write personal
letters and notes asking
for or conveying simple information of
immediate relevance,
getting across the point I feel to be important.
I can express news and views effectively in
writing, and relate to
those of others.
.
I can express myself in writing with clarity and
precision, relating to the
addressee flexibly and effectively.
As C1
√ √√
√
Data Analyses
Cronbach Alpha
One-way-ANOVA
Students’ self- assessment across PYP levels
Tutors’ assessment (of the same students) across PYP Levels
Independent Sample T-Test
Students’ vs tutors’ Assessment within PYP Levels
Correlation Coefficient Analysis
Weighted Kappa
Results
High reliability was found in students’ self-assessment (α= .884) and tutors’ assessment (α=.951)
The ten scales will be treated as one multi-item scale
Only the average results will be considered for this presentation
Results One-way ANOVA across PYP Levels
F=95.2, P <.001, η 2=0.26 F=113.5, P<.001, η 2 =0.30
Ele
me
nta
ry
Inte
rme
dia
te
Ad
van
ced
Ele
me
nta
ry
Inte
rme
dia
te
Ad
van
ced
Independent Sample T-Test within PYP Levels
M= 4.3, SD=1.5
M= 3.9, SD=1.5
M= 6.6, SD=1.4
M= 5.1, SD=1.6
M= 4.8, SD=1.4
M= 6.59, SD=1.5
t(161)=1.94, P=.05Cohen’s d= 0.26
t(499)= -2.32, P=.02Cohen’s d= -0.14
t(365)=.249, P= .803Cohen’s d= 0.026
Average Scales
Correlation Kappa % Exact
agreement
% within one
adjacent CEFR-
level difference
% within two
adjacent CEFR-
level difference
0.388
P<0.001
0.389 19.0 52.4 79.9
Strength, Direction and Agreement
Proficiency Levels at the End of PYP:
Students’ self-assessment tutors’ assessment PYP Curriculum
Expectations
mean CEFR Levels mean CEFR levels Exit CEFR Levels
Elementary 4.39 B1 3.91 A2+-B1 B1
Intermediate 4.83 B1-B1+ 5.15 B1+ B1-B2
Advanced 6.63 B2-B2+ 6.59 B2-B2+ C1
Conclusion
There is an acceptable agreement between students and tutors which will probably increase with more training (AlFallay, 2004 and Sahragard & Mallahi, 2014)
Using CEFR in a similar controlled way could assist participants in their assessment
Advanced students are more accurate in their assessment compared to lower-level students (Babaii, Taghaddomi & Pashmforoosh, 2016 and Engelhardt & Pfingsthorn, 2013)
Elementary level students tend to overestimate their proficiency levels;
Self and tutor assessment using the CEFR can potentially be used to obtain better insight to the varied aspects of students’ writing proficiency which may be lost through the current practice of exit testing at the PYP.
References AlFallay, I. (2004). The role of some selected psychological and personality traits of the rater
in the accuracy of self-and peer-assessment. System, 32(3), 407-425.
Babaii, E., Taghaddomi, S., & Pashmforoosh, R. (2016). Speaking self-assessment: Mismatches
between learners’ and teachers’ criteria. Language Testing, 33(3), 411-437
Engelhardt, M., & Pfingsthorn, J. (2013). Self-assessment and placement tests–a worthwhile
combination?. Language Learning in Higher Education, 2(1), 75-89.
Ross, S. (1998). Self-assessment in second language testing: A meta-analysis and analysis od experiental factors.” Language Testing 15 (1): 1-20.
Sahragard, R., & Mallahi, O. (2014). Relationship between Iranian EFL learners’ language learning styles, writing proficiency and self-assessment. Procedia-Social and BehavioralSciences, 98, 1611-1620.
Thank you