Upload
richard-cooper
View
220
Download
1
Tags:
Embed Size (px)
Citation preview
Advisory Group Meeting, 6th November 2009
Student Performance in National Examinations (SPINE): the
dynamics of language in school achievement
(ESRC/DfID RES-167-25-0263)
www.bristol.ac.uk/spine
Research Team Changes
• Bristol: – Dr Neil Ingram (Biology)– Dr Federica Olivero (Maths)
• Zanzibar: – Dr Haji Mwevura (overseas: post-doctoral
attachment); Shumbana Said (overseas: doctoral studies)
– Mohamed Abeid Mbarouk (Maths & Science)
2
• Study 2 : System wide examination study
• Study 4: Classroom study: focus on Teacher Assessment Issues
• Study 5: Examination achievement: focus on the learners
• Update on data collection for Process study (5.2) and Product study (5.3)
• Dissemination: Conferences, Publications
RESEARCH UPDATE AND FINDINGS
4 STUDY 2: ENG and MATHTotal: 64.717, school=29.18%,
pupil=70.82%
ENGLISH explains (90.4-
64.717)/90.4=28.41% of the
maths total variance
Total=90.4, School=30.75%,
Pupil=69.25%
5 STUDY 2: ENG and BIO5
Total=65.646, school=18.17%,
pupil=81.83%
ENGLISH alone explains
(114.968-65.646)/114.968=42.90%
of the total variance in BIOLOGY
Total=114.968, school=18.79%,
pupil=81.21%
6 STUDY 2: English and CHEMTotal=158.281, school=23.06%,
pupil=76.94%
ENGLISH explains (275.993-
158.281)/275.993=42.65% of the
total CHEM variance
Total=275.993, school=22.05%,
pupil=77.95%
7 STUDY 2: Summary of multilevel models
• ENGLISH is a significant and substantial predictor of student performance in MATH, BIO & CHEM.
• School-level variances – all models (using ENGLISH) as the single explanatory variable demonstrated a substantial proportion of the variance attributable to school factors
Study 4: Focus on Teacher Assessment
In Form II, Teacher Assessment (TA) was implemented
based on the following assessment procedures:
• Class work
• Oral questions
• Homework
• Weekly tests
• Terminal tests
•
8
Study 4: Focus on Teacher Assessment
• Understanding of what assessment is to be done
• Awareness of guidelines
• Percentage of marks that go to the final examination
• Sharing what teachers know and practice about
assessment guidelines among teachers within and
across schools
• Level of monitoring of Teacher Assessment by head
teachers.
Study 4: Focus on Teacher Assessment
However: significant variability among teachers:
• the types of specific activities that constitute teacher
assessment
• the number of activities from which they take marks for
each month
• decisions on how to handle student absenteeism
• how to get 10% of marks for each month; some picked
the one that students did better on, some picked
randomly, and some put the average of all marks
Study 2
Study 5.1: What are the four things that man could resemble whales?
45 students took this item:
• 35.6% = no answer
• 26.7% = wrong answer
• 28.9% = partially correct answer
• 8.8% = correct answer
11
Study 5.1: Interview: D1 who didn’t answer Q3 explains
D1: “because I did not understand by this this … resemble” (lines 115-117)
Int: “If I tell you that resemble means ‘to look like’ … can you do the question now?
D1: “Yes”Int: OK so what’s the answer?D1: “Man … is warm blooded … and whales
also … whales have lungs and man also have lungs …” (122-133)
12
13 Study 5 - Biology: responses to original item
• Give a brief explanation on what will happen on the following: A
locust not suffocating when its head is immersed in water while the
remaining part of the body is outside.
• No answer = 67.4%
• Wrong answer = 21.7%
• Partially correct answer = 6.5%
• Correct answer = 4.4%
14 Biology: on locusts
15
Biology: on locusts
16 Biology: on locusts
Questions:
a) In which picture do you think the locust will/may
die?
b) Why do you think it will/may die?
17 Original item modified:
• Greater contextualisation
• Simplification of instruction
• 2 structured parts: A & B
• Visual clues to support information retrieval
• Rephrasing of the item
• Altering item layout
18 Results
• Original item– Only 32.6% of students wrote an answer – Just under 11% gave a partially correct or
correct answer
• Modified item– 100% responded to this item– 42% gave a partially correct answer to Part A– 53% gave a partially correct answer to Part B
19Changes in student response
OR
Sc
Written responses on Modified Item
MOD
Score
Comment
(Neil Ingram: biologist)
H3 0 In picture A the locust may/will die
I think it will/may dies because the locust get its breathing by using its body
More than 0
Understands that locusts breathe using the body
C2 1 I think it will/may die because the boy is dipping the locust in the water to all the bodies with its trachea that used to respiration as a respiratory surface of a locust
more than 0
Getting very close to a very complex answer “I think a very able pupil indeed”. The right answer but with great difficulty in expressing this in English
Study 5: Language Factors in Exam Performance
• Considerable challenges faced by students in understanding English lexis/vocabulary
• Piloted an ‘off the shelf’ vocabulary test (receptive)• Developed a measure of vocabulary knowledge based on
corpora (an innovation in the design) developed from all past examination papers & current textbooks (Maths, Chemistry, and Biology)
• Piloted (Sept09), revised administered (Nov09)• To our knowledge this is the first time a special purpose
receptive vocabulary measure has been developed for English L2 school-age learners in sub-Saharan Africa.
20
Study 5.2: Focus on Process – CH & MT
• Aims: to capture in-depth insights on how
students:– process and produce their answers to
examination questions – use both English and Kiswahili in so doing– use English and Kiswahili in class
• Stimulus– video clips from Study 4– exemplar examination items
21
Study 5.3: Focus on Product (achievement)
• Aim: to probe further the complex relationships between
the use and effects of learning and assessment through
English.
• developed exemplar examination items for Maths,
Chemistry and Biology
• 3 different forms of the examination for c. 1000 Form II
learners: – English only– Kiswahili only– bilingual: Kiswahili and English.
22
Study 5.3:Focus on Product (achievement)
• Receptive vocabulary measure
• Student questionnaire (capture variables e.g.
English language exposure, SES data)
• Head Teacher questionnaire (capture variables
e.g. teacher qualifications matched with subjects
they teach)
23
Progress of SPINE
• anticipated all data collection activities will be
completed on schedule as well as the bulk of the
analyses – end July 2010
• refinement of analyses, writing & publication, &
ongoing dissemination of research findings will be
ongoing post July 2010.
24
Next Steps
• Data analysis ongoing & preparation of project
outputs
• Dissemination in Zanzibar & regional engagement:
practitioners, academics, NGO, policy makers
• SPINE International Symposium, Zanzibar 2011
• SPINE ‘mark 2’: what do we - SUZA/MoEVT - want
to do next?
25