View
5
Download
0
Category
Preview:
Citation preview
1
DEPARTMENT FOR LANGUAGE TEACHING AND INTERNATIONAL STUDENTS
ALTE Member
Saint Kliment Ohridski University of Sofia
2
A Developmental PerspectiveonStandardized Language Testing
Andonyan M., Borissova A., Yontcheva G., Todorinova J.
3
This presentation covers some important points regarding the examination through the Standardized Tests of Bulgarian:
4
Points of interest:
Test development Test takersTest resultsImpact of the ALTE audit 2007
5
The BeginningThe design and development of Bulgarian
standardized tests gradually evolved from 2000 onwards.
Forty-five years lasting experience of the Department in teaching Bulgarian to foreign students and citizens and the sharp increase in the interest towards its studying and certifying, have been the main factors that affected the process.
6
The aim of the tests is: to meet the needs of the international
audience for a test of Bulgarian according to the standards of ALTE and corresponding to the Common European Framework (CEFR)
to serve as an educational or job entryrequirement
to serve as a standard outcome for the international students taught at the Department
to be a part of the procedure for acquiring Bulgarian citizenship.
7
Proficiency LevelsBulgarian standardized tests cover 5 levels of language
proficiency: A2, B1, B2, C1 and C2, which are strictly aligned with the Common European Framework (CEFR) and ALTE Framework.
BASIC LEVEL – broadly corresponds to the Common European Framework of Reference for Languages, level A2
INTERMEDIATE LEVEL – B1 and B2 CEFR levels
ADVANCED LEVEL – C1 and C2 CEFR levels
8
Test Construction
Test construct, task types and format that measure four essential receptive, productive and interactive skills are based on the model of communicative competence for Bulgarian languageknown as Bachman-Tacheva’s model.
9
Test Developers A team of experienced Bulgarian
language teachers from the Department has been established to develop the design and content of the exam.
The team members are teachers and lecturers with scientific interests in the field of language testing. Most of them have academic degrees and are authors of textbooks, handbooks and manuals for language testing.
10
Test ContentTest content has been collected from
a great number of authentic sources: newspapersmagazines reference or history books brochures, booklets fiction, etc. Key principles for text selection are given
in the guidelines for item writers.
11
Content Selection CriteriaThe main selection criteria are: external context and the language use
for different levels according to CEFR the level of skills to be tested typical subject or topic areas for different
levels socio-cultural context vocabulary range grammar and structures specific difficulties and obstacles for
foreign learners of Bulgarian
12
Pre-testing
Due to some objective factors, related to test population and administration, real-time pre-testing has been difficult to be conducted.
The quality and validity of the tasks and items are evaluated by in-class testing and expert analyses, organized by the Examination Board.
13
New Materials
Prior to using, the new examination materials are checked and approved by two working committees:
The Content revising committeechecks and approves the linguistic and theme content.
The Ethics committee monitors for avoiding the insensitive language usage.
14
Training the Personnel
The Examination Board holds seminars, training workshopsand sessions on a regular basis addressed to:
test constructors and item writers examiners and supervisors experts, involved in the language
testing process.
15
Examination CentresTwo operating centres administer the test: Testing Centre at the Department Foreign Examination Centre at IMXA
Institute – Thessaloniki (Greece)We have also contracts for establishing new
centres in: Pizza (Italy) and Nantes (France)but they have not operated yet.
16
MarkingMarking is carried out according to very
clear and detailed assessment instructions both for the oral and written parts of the exam.
These instructions describe in detail: procedures of marking assessment criteria cut-off scores for different grades grading scale.
17
Scoring
Multiple choice items are scored by a prescribed number of points, given for the right answer only. No score reduction is made for
either the wrong or missing answers.
18
Scoring
Speaking is assessed and assigned a grading mark according to a special assessment instruction for the oral part of the examination only.
This part of the examination is fully audio-recorded.
19
AccuracyAll components are marked immediately
after the examination. To assure the accuracy and quality of the
marking, only professional examiners are assigned to conduct the examinations. They have been selected among the most qualified full-time language teachers at the Department.
The team of trained examiners is annually approved by the Examination Board.
20
Training the examiners
During the marking process examiners have to follow strictly defined rating criteria. These criteria and grading procedures are clarified in special training sessions, which take place at least once a year.
21
Examination supervisor
The accuracy of marking is monitored by an examination supervisor, who has the right to confirm or disprove the marks assigned by the examiners.
In case of disapproval, the supervisor is authorized to correct the assigned marks and to report the case to the Examination Board.
22
The Overall Grade
Speaking25%Writing
38%
Reading15%
Listening22%
The overall grade of the examination is formed by the results from different parts as follows:
23
Test timing and cut-off score
The test has to be completed within 3 hours.
The cut-off score is set to 60% of the total.
Candidates are provided with leaflets describing the scope of the examination and a user guide how to apply and how to interpret the results.
24
Post-test studies
Several post-test bias analyses are carried out to make sure the results are not influenced by any extraneous factors.
These studies also reveal test-takers’ profile and draw conclusions based on the data, collected from both the pre-test and post-test questionnaires.
25
QuestionnairesResponses to the questionnaires enable
the Examination Board to gather information about:
test-taker’s educational background the purpose and use of the exam pre-testing study and preparation for
the exam immediate impression and degree of
satisfaction from the examination
26
Distribution of the tested population over the period 2000-2007
Total = 419
43 5062
90
174
0
20
40
60
80
100
120
140
160
180
2000/2003 2004 2005 2006 20070
20
40
60
80
100
120
140
160
180
200
27
Purpose of the examination
General3%
Work34%
Citizenship15%
Education48%
The examination is mostly used for education, workand citizenship.
28
B2 level is mostly selected for education
A2
B1
B2
C1
C2
workgeneraleducationcitizenship
29
B1 level is mostly selected for citizenship
A2
B1
B2
C1
C2
workgeneraleducationcitizenship
30
B1, B2, C1 and C2 levelsare selected for work
A2
B1
B2
C1
C2
workgeneraleducationcitizenship
31
Purpose of the examination distributed by age
2126
3540
education general work citizenship
32
Purpose of the examination distributed by gender
7
53
4
4332
75
4
4639
128
8
89
0
20
40
60
80
100
120
140
citizenship education general work
female male totalSignificantly more men than women apply for citizenship
33
Purpose of the examination distributed by nationality
Turkish, 39%
Greek, 64%
Greek, 35% Other, 38%
Other, 26%
Education Work
34
Tested population distributed by age groups
26 - 4035%
- 188%
40 +17%
19 - 2540%
35
Tested population distributed by gender
male53%
female47%
36
Which are the most popular test information sources?
Advertissment
Test Book
Friends
IMHA Institute
Internet
None
Office
Staff
40+ years26-40 years19-25 years-18 years
37
Is preliminary test information sufficient for the test-takers?
YES50%NO
19%
No answer18%
Neither13%
38
Which factors best supporttest preparation?
0 10 20 30 40 50 60
-18 years
19-25 years
26-40 years
40+ years
TeacherSelf-practiceText book
Teacher is the main test preparation factor.
39
How do the test-takers rate the effectiveness of their test preparation?
-18 years old
19-25 yearsold
26-40 yearsold
40+ years old
positivenegative
Young people demonstrate negative attitude towards test preparation.
40
What is the average grade for different testing levels?
4.75
4.72
4.60
5.18
5.43
4.00 4.50 5.00 5.50 6.00
A2
B1
B2
C1
C2
41
Statistics of the overall grade by age groups
5.1
4.6
4.99
4.93
4.84
5.38
4.75
5
5
5
4 4.5 5 5.5 6
-18
year
s19
-25
year
s26
-40
year
s40
+ye
ars
All
Mean Median
Teenagers demonstrate higher results
42
Statistics of the overall grade by gender
5.01
5
4.68
4.75
2 3 4 5 6
Mea
nM
edia
n
MaleFemale
43
Average grade and L1
3.00 3.50 4.00 4.50 5.00 5.50 6.00
Turkish
Asian
Greek
Hebrew
Arabic
French
German
English
Italian
Spanish
Slavic
44
How do the test-takers qualify their examination result?
19%
15%
54%
12%
no answer
worse
as expected
better
45
Self-rating of the results distributed by levels
0% 20% 40% 60% 80% 100%
A2
B1
B2
C1
C2
worseas expectedbetter
27% of C1 level candidates qualify their result as worse than expected
46
How do the test-takers rate the quality of examination?
no answer17%
very good39%
good8%
satisfactory1%
unsatisfactory1%
excellent34%
81% of the test-takers appreciate the quality of examination
47
ALTE audit 2007
B2 level test was audited in June 2007.
This ALTE audit revealed some drawbacks and also inspired immediate actions for improving the quality and administration of the Standardized Tests of Bulgarian.
48
Increasing the number of candidates The Council of the Department decided
to allow the international students free of charge examination as a regular outcome of their studies at the end of the academic year 2006/2007.
90 students took advantage of this proposal (the sponsorship cost the Department nearly € 13 500).
49
Improving pre-testing
Pre-testing in real-examination conditions is to be carried out parallel with in-class pre-testing from 2008.
Preliminary test is to be introduced to help pre-testing and avoid misleading the candidates by identifying their approximate level.
50
Reliability
Insertion of anchor itemsacross different variants to handle the reliability of the test.
Increasing the numberof test-takers.
51
Calibration
To continue the process of developing calibrated examples for evaluation of various models for the different levels of the test.
52
Thank you for your attention
Recommended