Promoting the wellbeing of Africans through policy-relevant research on population and health 1
Impact evaluation of the East African Quality in Early Learning (EAQEL)
Initiative
Presenter: Moses Ngware Contributors: ERP team
African Population and Health Research Center (APHRC)
Supported by Hewlett/QEDC May 2010
Promoting the wellbeing of Africans through policy-relevant research on population and health 2
Content
Background
Research questions
Design & methods
Findings
Policy implications
Promoting the wellbeing of Africans through policy-relevant research on population and health 3
Background
An intervention study being implemented by AKF- Reading to Learn (RtL/scaffolding).
Role of APHRC: Impact assessment of the proposed RtL
Promoting the wellbeing of Africans through policy-relevant research on population and health
Are children in lower primary grades (1, 2 and 3) able to read and do mathematics calculations more proficiently as a result of the RtL approach?
Are there differences in proficiency for children exposed to parental involvement in the RtL Approach (core model plus) compared:• to those exposed to the RtL Approach with no parental involvement
(core model), • to control schools?
What are the key contributing factors to these improvements in numeracy and literacy in grades 1, 2 and 3?
Research Questions
Promoting the wellbeing of Africans through policy-relevant research on population and health
Design - EAQEL
Randomised Control Trial (RCT)
Four districts – 2 Kenya & 2 Uganda
Treatment and control schools in each district
120 & 109 primary schools in Kenya & Uganda, respectively
229 HT, over 1000 teachers, 14,000 pupils, 7000 parents are targeted.
Intervention: i. Core module in 1 district & ii. Core module plus in the other district.
Control Treatment
District No % No %
Kinango 33 45.
83 39 54.17
Kwale 30 62.50 18 37.50
Amolatar 25 51.
02 24 48.98
Dokolo 32 53.33 28 46.67
Total 120 52.40 109 47.60
Promoting the wellbeing of Africans through policy-relevant research on population and health
Sample determination, effect size & power calculations
Assumptions (used OD software): Significance level = 0.05 Cell size (n) or # of pupils = 20 Effect size = 0.22 (informed by literature) Correlation coeff. = 0.10
For a power of 80%, 100 schools needed.
For a power of 90%, 128 schools needed.
Promoting the wellbeing of Africans through policy-relevant research on population and health
Data Analysis Given that the causal effect for a single unit
u cannot be observed, we aim to identify the average causal effect for the entire sample
The ATE of t (relative to c) over U (or any sub-population) is given by:
ATE =E [Y1(u) – Y0(u)] = E [Y1(u)] – E [Y0(u)]
Promoting the wellbeing of Africans through policy-relevant research on population and health
Data Analysis
TimeTreatment
Outcome
Treatment Group
Control Group
Average Treatment Effect on the Treated
Promoting the wellbeing of Africans through policy-relevant research on population and health
Design Issues - Kenya
EAQEL
S I P
YES NO
YES G1 G2
NO G3 G4
Promoting the wellbeing of Africans through policy-relevant research on population and health
Methods - instruments
Assessment tools:Single numeracy test for grades G1 – 3Single literacy test in Kiswahili or Lang’o G1-3Test items range from simple to difficultCurriculum & assessment experts, practitioners and researchers
were involved – for content validity.All instruments were pre-tested.
Survey tools:head teacher’s questionnaireteacher questionnairepupil questionnairehousehold questionnaire targeted parents/guardian
Promoting the wellbeing of Africans through policy-relevant research on population and health
Sampling Process - Kenya
Kenya’s districts: Kwale & Kinango
Kinango
Treatment Control
Non-Kensip
Kensip
Non-Kensip
Kensip
Kwale
Treatment Control
Promoting the wellbeing of Africans through policy-relevant research on population and health
Sampling process - Uganda
Uganda 2 Districts
District 1: Core Model District 2: Core Model Plus
Cluster of school zones
Control Treatment Treatment Control
Promoting the wellbeing of Africans through policy-relevant research on population and health
BS Findings- Literacy written
No. of pupils Mean st. dev No. of pupils Mean st. dev
Overall (Both KE & UG) 4382 18.07 22.90 4365 9.08 6.61
Kenya 2223 30.69 24.50 2213 11.11 7.45
Uganda 2159 5.08 10.64 2152 6.99 4.80
Treatment 2206 16.72 21.78 2199 8.68 6.47
Control 2176 19.45 23.90 2166 9.48 6.73
Boys 2155 18.09 23.06 2158 9.17 6.66Girls 2227 18.06 22.74 2207 8.99 6.56
Grade 1 Grade 2
Literacy written (several aspects tested) - Administered orally to the pupils and were to write answers on a provided booklet (mean is out of 100%)
Promoting the wellbeing of Africans through policy-relevant research on population and health
Literacy – 101 (Oral) Oral literacy (several aspects)- Pupils were required to
answer to a set of questions administered orally.
No. of pupils Mean st. dev No. of pupils Mean st. dev
Overall (Both KE & UG) 4318 55.27 16.56 4302 58.71 17.78
Kenya 2227 59.55 16.53 2199 65.09 18.81
Uganda 2091 50.70 15.33 2103 52.04 13.77
Treatment 2173 53.98 16.44 2162 57.40 17.36
Control 2145 56.57 16.58 2140 60.04 18.10
Boys 2114 55.37 16.48 2122 59.14 17.88Girls 2204 55.16 16.64 2180 58.30 17.68
Grade 1 Grade 2
Promoting the wellbeing of Africans through policy-relevant research on population and health
Literacy written – Distribution of score - Kenya
050
100
150
200
250
300
350
400
Freq
uenc
y
-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.
0 20 40 60 80 100LIT WRITTEN
Source: EAQEL Baseline 1, 2009
Grade 1 Kenya
050
100
150
200
250
300
350
400
Freq
uenc
y
-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.
0 10 20 30LIT WRITTEN
Source: EAQEL Baseline 1, 2009
Grade 2 Kenya
Poor performance in written literacy in Kenya- Both grades- Average % score <40%
Greater variability
Promoting the wellbeing of Africans through policy-relevant research on population and health
Literacy written – Distribution of score -Uganda
0.0
5.1
.15
.2D
ensity
-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.
0 20 40 60 80LIT WRITTEN
Source: EAQEL Baseline 1, 2009
Grade 1 Uganda
0.0
5.1
.15
Den
sity
-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.
0 10 20 30LIT WRITTEN
Source: EAQEL Baseline 1, 2009
Grade 2 Uganda
Poor performance in written literacy in Kenya- Both grades- Average % score <10%
Promoting the wellbeing of Africans through policy-relevant research on population and health
Literacy written – Distribution of score - Kenya
050
100
150
200
Freq
uenc
y
-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.
0 20 40 60 80 100Oral Literacy
Source: EAQEL Baseline 1, 2009
Grade 1 Kenya
050
100
150
200
Freq
uenc
y
-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.
0 20 40 60 80 100Oral Literacy
Source: EAQEL Baseline 1, 2009
Grade 2 Kenya
Better performance on oral literacy than written in Kenya
Promoting the wellbeing of Africans through policy-relevant research on population and health
Literacy written – Distribution of score -Uganda
Better performance on oral literacy than written in Kenya
050
100
150
200
Freq
uenc
y
-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.
0 20 40 60 80 100Oral Literacy
Source: EAQEL Baseline 1, 2009
Grade 1 Uganda
050
100
150
200
Freq
uenc
y
-2 s.d. -1 s.d. Mean +1 s.d. +2 s.d.
0 20 40 60 80 100Oral Literacy
Source: EAQEL Baseline 1, 2009
Grade 2 Uganda
Promoting the wellbeing of Africans through policy-relevant research on population and health
Oral Literacy competencies: Listening
Grade 1: High rating on Listening skills by country
40
50
60
70
80
90
100
Identification of Objects Identifying set of instructions Sound discrimanation Listening comprehension
Perc
ent
GRADE 1: LISTENING BY COUNTRY
Kenya Uganda
Promoting the wellbeing of Africans through policy-relevant research on population and health
Oral Literacy competencies: Listening
Grade 2: High rating on Listening skills by country
GRADE 2: LISTENING SKILLS BY COUNTRY
70
75
80
85
90
95
100
Identification ofObjects
Identifying set ofinstructions
Sound discrimanation Listeningcomprehension
Perc
enta
ge
Kenya Uganda
Promoting the wellbeing of Africans through policy-relevant research on population and health
Oral Literacy competencies: speaking
Grade 1: High rating in Kenya than Uganda
GRADE 1: SPEAKING SKILLS BY COUNTRY
20
30
40
50
60
70
80
Describing objects Telling direction Story composition
Perc
enta
ges
Kenya Uganda
Promoting the wellbeing of Africans through policy-relevant research on population and health
Oral Literacy competencies: speakingGRADE 2: SPEAKING SKILLS BY COUNTRY
20
30
40
50
60
70
80
90
Describing objects Telling direction Story composition describing process
Perc
enta
ges
Kenya Uganda
Grade 2: High rating in Kenya than Uganda
Promoting the wellbeing of Africans through policy-relevant research on population and health
GRADE 1: READING SKILLS BY COUNTRY
0
10
20
30
40
50
60
70
80
90
Letters: vowels Letters: Consonants Words
Perc
enta
ges
Kenya Uganda
Grade 1: High rating in Kenya than Uganda Reading words scored lowly
Oral Literacy competencies: Reading
Promoting the wellbeing of Africans through policy-relevant research on population and health
GRADE 2: READING SKILLS BY COUNTRY
0
10
20
30
40
50
60
70
80
90
100
Letters: vowels Letters: Consonants Words Simple sentences
Perc
enta
ges
Kenya Uganda
Grade 2: High rating in Kenya than Uganda
Oral Literacy competencies: Reading
Promoting the wellbeing of Africans through policy-relevant research on population and health
Class size and literacy score- Grade 130
4050
6070
80(m
ean)
sco
re_p
erce
nt
0 50 100 150classize
bandwidth = .8
Class size Vs pupil score: Kenya Grade 1
2030
4050
6070
(mea
n) s
core
_per
cent
50 100 150 200 250 300classize
bandwidth = .8
Class size Vs pupil score: Uganda Grade 1
Correlation Kenya: -0.0256 Correlation Uganda: 0.2369
Promoting the wellbeing of Africans through policy-relevant research on population and health
Correlation Kenya: -0.0918 Correlation Uganda: -0.0477
Class size and literacy score- Grade 230
4050
6070
80(m
ean)
sco
re_p
erce
nt
0 50 100 150classize
bandwidth = .8
Class size Vs pupil score: Kenya Grade 2
2040
6080
(mea
n) s
core
_per
cent
50 100 150 200 250classize
bandwidth = .8
Class size Vs pupil score: Uganda Grade 2
Promoting the wellbeing of Africans through policy-relevant research on population and health
Availability of reading materials at school: - Story books
Literacy (Oral) – Availability of story books Grade 1
Story book: Available in class Coef P CI Available not in class -1.34 0.443 [-4.77 ; 2.09] Not available -3.95 0.027 [-7.44 ; -0.46] Uganda -24.42 0 [-27.3 ; -21.54] Constant 31.77 0 [29.3 ; 34.25]
Grade 2 Coef. P CI Story book: Available in class Available not in class -0.54 0.408 [-1.81 ; 0.74] Not available -0.65 0.329 [-1.96 ; 0.66] Uganda -3.93 0 [-5 ; -2.86] Constant 11.39 0 [10.44 ; 12.34]
Promoting the wellbeing of Africans through policy-relevant research on population and health
Household characteristics 43.96 of the household heads have primary education,
with 34.81 with no education
46% of the pupils are not give homework from school.
Among those given homework, 29.23% are not assisted in doing the homework- this varies by country (Kenya- 55.10% and Uganda – 20.79%)
69% whose heads have no education, do not help their pupils in doing homework compared with 19% whose household heads have tertiary education
Promoting the wellbeing of Africans through policy-relevant research on population and health
Literacy 101: Regression coefficientsRegression on Literacy (ORAL) performance by some selected HH characteristics
KENYA (n=2164) UGANDA (n=871) Coefficient CI Coefficient CI Hmwk: Always Helped (Ref) Hmwk: Sometimes Helped -2.03 [-04.66 ; 00.59] 0.22 [-02.64 ; 03.08] Homework: Not helped -2.20* [-04.59 ; 00.20] 0.94 [-02.21 ; 04.08] No Homework -3.85** [-06.22 ; -01.49] -1.01 [-02.89 ; 00.87] Grade 1 (Ref) Grade 2 -16.88** [-18.24 ; -15.53] -0.95 [-02.31 ; 00.42] HH edu: No education (Ref) HH edu: Primary 0.63 [-00.92 ; 02.17] 0.01 [-01.84 ; 01.86] HH edu: Secondary 5.49** [02.86 ; 08.12] 2.48** [00.24 ; 04.73] HH edu: Tertiary 4.44* [-00.16 ; 09.04] 1.44 [-02.42 ; 05.31] HH edu: Not known 1.53 [-02.09 ; 05.15] -0.38 [-02.61 ; 01.86] HH: Father (Ref) HH: Mother 1.31 [-00.92 ; 03.55] -1.13 [-03.38 ; 01.13] HH: Male Guardian 1.03 [-01.49 ; 03.55] -2.31 [-05.27 ;0 0.65] HH: Female Guardian 0.34 [-03.38 ; 04.06] -0.33 [-04.63 ; 03.97] Pup sex: Female (Ref) Pup sex: Male 0.57 [-00.78 ; 01.92] 0.98 [-00.38 ; 02.35] Constant 27.76 [25.13 ; 30.39] 7.31 [04.86 ; 09.75]
Promoting the wellbeing of Africans through policy-relevant research on population and health
Regression results [2]
Pupils who come home with homework and helped within the household score better- in Kenya
HH education: Pupils from households headed by heads with secondary and tertiary education score high- Kenya
Uganda only secondary education matters