Embed Size (px)
Citation preview
Briefing Session on the 2012 HKDSE English Language
Assessment Framework
March 2008
2
Background
Third consultation conducted in late 2006 New public assessment of HKCEE English
Language first implemented in 2007 Further consultation in the light of the 2007
HKCEE experience Questionnaire survey conducted in
January 2008
3
2007 HKCEE Experience
4
SBA Implementation ScheduleYear Options for School2007 1. Submit SBA marks for feedback; marks included as 15% of
the subject result
2. Submit SBA marks for feedback only; marks not included in subject result
3. Not submit SBA marks; exam results to contribute 100% of subject result
2008 1. Submit SBA marks for feedback; marks included as 15% of the subject result
2. Submit SBA marks for feedback only; marks not included in subject result
2009 1. Submit SBA marks for feedback; marks included as 15% of the subject result
5
Implementation of SBA in 2007No. of
schoolsPercentage
(%)No. of
candidatesPercentage
(%)
Option 1 (Yes)
199 34 31,876 43
Option 2 (Trial)
125 22 20,945 28
Option 3 (No)
254 44 21,388 29
Total 578 100 74,209 100
6
Moderation Results of Option 1 Schools
The mean of the SBA scores is … No. of Schools
Percentage (%)
within the range 144 72.4
slightly higher than expected 29 14.6
higher than expected 2 1.0
much higher than expected 0 0
slightly lower than expected 21 10.6
lower than expected 3 1.5
much lower than expected 0 0
7
Moderation Results of Option 1 Schools
The standard deviation of the SBA scores is …
No of Schools Percentage
(%)
as expected 179 89.9
slightly wider than expected 0 0
wider than expected 0 0
slightly narrower than expected 10 5.9
narrower than expected 9 4.5
much wider than expected 0 0
much narrower than expected 1 0.5
8
Moderation Effect on Candidates
Mark Adjustment
(% of subject)
No. of candidates
Percentage of candidates
(%)
0 (0) 5365 17
1-3 (<1) 19881 62
4-6 (<2) 6237 20
7-9 (<3) 392 1
31,875 100
9
Implication of Moderation Results
Mean and spread of the SBA marks submitted by the majority of schools fall within the expected range
Most teachers have a good understanding of the assessment criteria and can assess their students accurately
Reliability of SBA not an issue Fairness ensured by statistical moderation
10
Absentee Rate of School Candidates in 1996 CE English Language Speaking Exam
English Language
(Syllabus A)
No. Sat 6,623
No. of Absentees 3,058
Absentee Rate 31.6 %
English Language
(Syllabus B)
No. Sat 88,201
No. of Absentees 19,236
Absentee Rate 17.9%
11
Absentee Rate of School Candidates in 2007 CE English Language Speaking Exam
No. Sat No. of Absentees
Absentee Rate (%)
Option 1 (Yes) 27,935 3,398 10.8
Option 2 (Trial) 19,307 1,466 7.1
Options 1 & 2
(Yes + Trial)
47,242 4,864 9.3
Option 3 (No) 19,298 3,681 16.0
All Schools 66,540 8,545 11.4
12
Possible Explanations 1. Students in “Yes” and “Trial” schools had more
speaking practice in school because of the SBA
2. They became more confident with the public exam format, which involves similar speaking tasks – group discussion and individual response
3. Positive backwash effect of the SBA on the ability to speak English and to interact with peers in different contexts.
13
Effect of SBA on Speaking Exam
No. SatSpeaking Exam
Mean Moderated
SBA Mean
Option 1 (Yes) 27,804 25.38 27.81
Option 2 (Trial) 19,381 26.00 -
Option 3 (No) 21,293 23.51 -
All Schools
68,478
24.97 -
14
Effect of SBA on Speaking Exam
Students in “Yes” schools got higher marks in the SBA than in the speaking exam
Students in “Yes” and “Trial” schools got higher marks in the public speaking exam than those in “No” schools
BUT “Yes” and “Trial” schools may just be better schools
15
Effect of SBA on Speaking Exam
No. SatActual – Predicted
Mean Scorest-value
Option 1 (Yes) 27,804 0.25 -1.97
Option 2 (Trial) 19,381 0.03 -0.21
Option 3 (No) 21,293 - 0.35 2.34*
All Schools
68,478
- -
16
Effect of SBA on Speaking Exam
Students in “Yes” schools did better than expected based on prediction from written papers (+0.25)
Students in “Trial” schools did slightly better than expected (+0.03)
Students in “No” schools did significantly worse than expected in the speaking exam (-0.35)
17
Congeneric Measures Model Analysis
Strong evidence that the various components are measuring a single underlying ability
Reliability of components .882 to .702 R > L&IS > W > SBA > S SBA (.753) more reliable than speaking ex
am (.702)
18
Inter-paper correlationsCorrelation Matrix Correlation Matrix (by listwise case exclusion, N=28,253)(by listwise case exclusion, N=28,253)
R W L & IS S SBA
R 1.000 .858 .887 .776 .803
W .858 1.000 .852 .767 .797
L & IS .887 .852 1.000 .764 .796
S .776 .767 .764 1.000 .787
SBA .803 .797 .796 .787 1.000
19
Survey Results
20
Main focus of questionnaireMain focus of questionnaire
Assessment of the modules in the Elective Part
Replacing the speaking exam with SBA The need for strategic implementation of
SBA in 2012 HKDSE Teachers’ satisfaction with the level of
support provided by the HKEAA and the EDB
21
Results: Response RateResults: Response Rate
All Schools
Part.
Schools
Option 1 Schools
Option 2 Schools
Option 3 Schools
717 640 199 125 254
422 411 165 109 130
59% 64% 83% 87% 51%
22
ResultsResults
Responses measured on 6-point scale 3.5 adopted as minimum level of acceptabi
lity
23
Assessment of the Modules in the Elective Part
Question
All schools
Option 1 schools
Option 2 schools
Option 3 schools
1. Assessing the modules in the Elective Part of the curriculum through the writing paper is appropriate.
3.7 3.7 3.7 3.7
2. The proposed Paper 2 structure is appropriate.
3.8 3.8 3.8 3.7
3. The proposed weighting of Part A (10%) and Part B (20%) is appropriate.
4.0 4.0 4.0 4.0
4. The question types for the modules in the Elective Part are appropriate.
3.7 3.7 3.7 3.7
5. Assessing the modules in the Elective Part of the curriculum through Part B of the SBA is appropriate.
3.9 4.0 4.0 3.7
24
Assessment of the Modules in the Elective Part
Question
All schools
Option 1 schools
Option 2 schools
Option 3 schools
6. The proposed weighting of Part B (5%) is appropriate.
3.6 3.7 3.6 3.5
7. The amount and type of effort required of teachers to implement the SBA in HKDSE is reasonable.
2.9 3.1 3.0 2.7
8. The amount and type of effort required of students to complete the SBA in HKDSE is reasonable.
3.4 3.5 3.5 3.3
9. Students have benefited from the introduction of SBA.
3.9 4.1 3.9 3.6
10. The effort spent on implementing SBA has been worthwhile.
3.5 3.7 3.5 3.3
25
Replacing the Speaking Exam with SBA
Question
All schools
Option 1 schools
Option 2 schools
Option 3 schools
12. School candidates need not take the public speaking examination as their speaking skills can be assessed by means of school-based assessment.
2.7 2.7 2.6 2.7
26
Teachers’ satisfaction with level of support Teachers’ satisfaction with level of support provided by HKEAA and EDB (1)provided by HKEAA and EDB (1)
13. Resources and support provided by the HKEAA All schools
01. Handbook provided for the HKCEE English Language School-based Component
4.3
02. 2007 HKCE English Language Examination Introduction to the School-based Assessment Component (with DVD)
4.3
03. Training CDRs 1 and 2 4.1
04. HKCEE Chinese Language and English Language School-based Assessment (DVD distributed to parents and schools in October 2006)
3.8
05. HKCEE Chinese Language and English Language Information for Parents on School-based Assessment (leaflet distributed to parents in October 2006)
3.7
06. Statistical moderation of School-based Assessment scores (booklet distributed to schools in June 2007)
3.8
07. Group Coordinators System 3.8
27
Teachers’ satisfaction with level of Teachers’ satisfaction with level of support provided by HKEAA and EDB (2)support provided by HKEAA and EDB (2)
08. Teachers’ seminars on the implementation of HKCEE English Language SBA component held in September/October
3.9
09. Assessment training sessions held in November (chaired by SBA group coordinators)
4.0
10. Inter-school sharing sessions (chaired by SBA group coordinators)
3.8
11. Professional development courses (conducted by HKU and HKIED)
3.9
12. SBA Corner on the HKEAA website (including General Information, List of Recommended Texts and FAQs)
4.0
13. The Platform for SBA of CE English Language Exam on the English Campus of HKedCity
4.3
14. The HKU website on SBA Projects 3.9
28
Teachers’ satisfaction with level of Teachers’ satisfaction with level of support provided by HKEAA and EDB (3)support provided by HKEAA and EDB (3)
14. Resources and support provided by the EDB All schools01. Resource materials 4.0
02. Professional development programmes 4.0
03. Grants 4.3
04. School-based Support Services 3.8
05. Collaborative research and and development (“seed”) projects
3.5
29
Need for Strategic Implementation of SBA in 2012 HKDSE
Question
All schools
Option 1 schools
Option 2 schools
Option 3 schools
16. My school’s preparation plan for implementing the SBA for English Language in 2012 HKDSE is on track.
3.5 3.7 3.7 3.3
17. The proposed SBA framework for English Language is desirable for implementation in the 2012 HKDSE.
3.6 3.7 3.7 3.5
30
Respondents’ comments (1)Respondents’ comments (1)
6. The proposed weighting of Part B (5%) is appropriate.
Part B should have a heavier weighting.Both Parts A and B should each account for 10%Both Parts A and B should each account for 5%
31
Respondents’ comments (2)Respondents’ comments (2)
7. The amount and type of effort required of teachers to implement SBA in the HKDSE is reasonable.
Workload for teachers too heavyToo many assessments to doMore resources needed, especially technical supportMore sample tasks provided
32
Respondents’ comments (3)Respondents’ comments (3)
8. The amount and type of effort required of students to complete SBA in the HKDSE is reasonable.
Reading six texts in three years and having to do four assessments in two years is too demanding.Hard to monitor time spent by candidates on SBA reading/viewing programme
33
Respondents’ comments (4)Respondents’ comments (4)
18. State the kind of modification(s) to the English Language framework you think would be desirable for the 2012 HKDSE.
Provide sample tasksImprove standardizationSBA weighting should be increased/reducedUse outside examiners for SBAKeep Speaking examReduce the number of assessments
34
Proposed/Revised Assessment Frameworks
Current proposed framework
Revised proposed
framework Components Weighting Requirements Weighting Requirements
Paper 1 Reading 20% 20%
Paper 2 Writing 30% 25%
Part A 10% Word limit: 200 10% Word limit: 200
Part B 20% Word limit: 400 15% Word limit: 400
35
Proposed/Revised Assessment Frameworks
Components Weighting Requirements Weighting Requirements
Paper 3 Listening & Integrated Skills
30% 30%
Paper 4 Speaking 20% For private candidates only
10% For all candidates
36
Proposed/Revised Assessment FrameworksComponents Weighting Requirements Weighting Requirements
SBA 20% For school candidates only
15% For school candidates only
Part A 15% Three assessments; group interaction and individual presentation; report three marks
10% Two assessments; group interaction and individual presentation; report two marks
Part B 5% One assessment based on the modules in the Elective Part of the curriculum; report one mark
5% One assessment based on the modules in the Elective Part of the curriculum; report one mark
37
Rationale for the Revised Framework Teachers need to deal with HKCEE, HKALE an
d HKDSE from 2009 to 2012 Speaking exam retained for school candidates
as the majority of teachers do not favour its replacement by SBA
Weighting of SBA component reduced from 20% to 15% and weighting of Part B of Writing paper reduced from 20% to 15% so that 10% can be allocated to the Speaking exam
The total number of SBA marks to be submitted reduced from 4 to 3 to address teachers’ concerns about workload.
Weighting of SBA component reduced to 15%, which is the same as the 2007 HKCEE
38
Schedule for HKDSE DevelopmentStandards-referenced Reporting (SRR) Production of sample papers and marking
schemes Pilot test of sample papers conducted Draft of level descriptors and annotated
examples completed Briefing sessions conducted for schools &
SRR requirements finalised by Dec 08 SRR booklets with sample papers
disseminated to schools by Mar 09
39
Schedule for HKDSE Development
School-based Assessment (SBA) Assessment criteria, specifications and
sample tasks developed Briefing sessions conducted for schools;
SBA requirements finalised by Dec 08 SBA handbook disseminated to schools
by May 09
40
Thank You Thank You
Questions?
