Upload
ruth-phelps
View
214
Download
1
Tags:
Embed Size (px)
Citation preview
Approaches to evidence based policy making in education
Professor Geeta Kingdon
Congratulations CBSE
CAER commitment to evidence based decision-making
Centre will help to improve learning by promoting reliable and valid assessments
also help to create capacity do robust evaluations
Thus bring assessment-led reform in Indian education
Splendid example of a PPP
Talk is on evidence based policy, focus on macro
Current landscape
Pitiably poor learning levels (PISA, EI, NAS, ASER, NFHS, SchoolTELLS; NAS showed that <50% of class V kids could answer “how much greater is 555 than 198”? )
Measuring learning can be disheartening, when much effort & large resources expended (SSA reforms, RTE, NCF, 6 Pay Comm, TET, NCFTE, NCSL) & still outcomes are poor; denial
Encouraging that Planning Comm’s 12th FYP (GOI, 2013, p49) “will place the greatest emphasis on improving learning outcomes at all levels”
Steps for evidence based decision-making
Obtain the evidence (e.g. measure learning levels)
Evidence has to be credible (e.g. measure learning with validity and reliability, with proper sampling)
Embrace & analyse results to reach diagnoses
Formulating policies to take remedial action
Pilot test new policies, to see their impacts on learning
Scale up policies that have strong impact at lowest cost
Evidence from learning assessments – can empower parents
An imp potential use of evidence from A – is parental info
Has potential to increase school accountability
World university rankings, but not school rankings
Fear they may reflect socio-econ background but there are ways of reducing that
In India a debate needed
Evidence from assessment of teachers
One potential reason for poor learning - teachers themselves lack competence
Teachers rarely tested in large scale way
Evidence from T tests in India showed poor competence
Led to the decision to bring in the TET
SchoolTELLS survey
The teacher tests were graded by SCERT Bihar staff
Assessment tasks for teachers aligned with standard teaching tasks that teachers in primary school would be required to do in the classroom routinely.
Language tool helped to understand teacher’s ability in the following:
Do you know: e.g. meaning of difficult words in a grade 4 level text Can you explain : e.g. explain difficult words in simple language or
summarize a Std 4 story text effectively Can you spot common mistakes: e.g spelling and grammar mistakes
Maths tool also helped to understand the teacher’s ability in the following :
Do you know: e.g. solve problems Std 4 or 5 level Can you explain: e.g. explain problem solving in simple steps Can you spot and analyze common mistakes : e.g in arithmetic
operations
TASK 1: VOCABULARY RELATED TASKS
Grading done on 3 criteria:
Was the word meaning “meaningful” ? Was language used “easy to
understand” ? Were there any spelling mistakes ?
From Std 3 onwards, the vocabulary in language textbooks becomes difficult. So, teachers need to be able to explain difficult words in simple language.
TASK 2 : SUMMARIZING TEXTS
TASK 3: SPOTTING SPELLING & GRAMMATICAL MISTAKES
Deficits in teachers literacy skills
43% of word meanings correct; 57% wrong
45% of summaries were meaningful; 55% wrong
40% teachers did not have spelling mistakes (in a 2-sentence write up)
35% had 1-2 mistakes; 25% had >=3 mistakes
Only 50% of teachers could spot >3 mistakes in a write up in which we had deliberately introduced 6 mistakes.
TASK 6 : SPOTTING & UNDERSTANDING COMMON MISTAKES IN ARITHMETIC
Samples of children’s work shown. Teachers asked to choose…
Deficits in teachers’ numeracy skills
78% could spot correct one when presented a sample of three simple division sums
24.5% could do a percentage sum; 75% not
27.9% could do an area sum; 72% not
About 20% said they never had problems in addressing the maths queries of their pupils
How this evidence and TET evidence helps
School-TELLS (2008) & ‘Inside Classrooms’ study (2011) highlighted deficits in T knowledge & ability to teach
This evidence contributed to decision to introduce the TET
TET objective to vet applicants, ensure competence Abysmal pass rate - 0.4 to 3.7% pass rate
Evidence in this T assessment is extremely valuable
The evidence in this T assessment is extremely valuable It helps to identify the training needs of teachers.
It can inform policy makers who decide training curricula
But have states used this evidence in this way? States need the desire to analyse; make use of this evidence; a transparent approach
Congratulate the CAER for analysing C-TET – this is evidence based policy making (the policy maker CBSE sought evidence – gave data)
Is this evidence relevant only for govt and rural private schools?
Is this problem of low cognitive skills of Ts confined to government primary teachers ?
Clearly more generic problem – Even highly paid govt T have major deficits in skills TET evidence
Private schools cannot be complacent on this
Testing T can help to assess the training needs of each T
The importance of evidence
There are many initiatives to improve education
NFE (1982) OBB (1986) TLC (1988) MDM (1982) SK (1987) LJ (1988) DPEP (1993) SSA (2003)
Aadhar, ABL, MLE, Nallikali, Nai Disha, Read India, RTE
Have these programs had impact? – little evidence to judge
Efforts to improve more successful, when based on evidence
Culture of seeking evidence Why base decisions on evidence?
While poor quality schooling does not threaten lives, it seriously affects people’s quality of life, and even longevity.
In medicine, its unthinkable without thorough testing by experts, & the use of most robust, expensive randomised control trials; But in Education, Ministers freely make policies without consulting evidence / experts
In good educ systems, and ideally, policy will be not made on supposition, ideology or political expediency. Govt seeks evidence
UK policy makers’ use of evidence fromimpact evaluations
Early intervention - quality & effectiveness of pre-school experience in securing better long term outcomes was used to justify more investment (Sammons et al, 2006)
Class size - evidence of no stat difference was used to justify not extending the policy into later years (Blatchford et al, 2002)
Formative assessment - effect sizes in attainment (alongside pupil & teacher perspectives) led to inclusion in national policy (Wiliam et al, 2004)
Education Maintenance Allowance - evidence from pilot study of post 16 retention led to national roll out, though long-term sustainability was worse in pilot areas (Middleton et al, 2005)
Policies could be better, if evidence based
The narrative in the 6th Pay Comm for across-the-board doubling teacher salaries (without increased accountability) was : it will motivate teachers. Was this effective? Did it raise teacher effort? No one checked. [next slide]
A state govt recently announced it would regularise 176,000 para teachers, in the name of quality of education; it did not look at the relative effectiveness of regular and para teachers (3 papers)
Under RTE it has been made mandatory for teachers to have B.Ed. Certification, reduced PTR to 30, and many inputs mandatory; where is the evidence for this? no pilot testing
If there were garnering of evidence on the impact of policies, then ineffective policies could be weeded out
UP
Reg. Para Priv.
Salary/month, 2008(today after 6th Pay Comm.)
12,017(27,000)
3,000(3,500)
940(1400)
Absence rate 24.6 12.0 17.4
% time teaching 75.3 83.3 89.0
SchoolTELLS survey (2008)
Higher resources, lower effort
Structure of accountability matters more than resources
What kind of evidence?
What kind of evidence is useful / acceptable?
Distinguishing correlation and causation
The importance of methodology
Using the force of the federal chequebook to nudge researchers to use robust methods capable of yielding causal inferences.
Interest in evidence of impact Field of medicine long interested in evidence of impact
Recent upsurge of interest in impact evaluation in many fields nutrition, labour, governance, rural development, education, poverty
‘International Initiative for Impact Evaluation’ (3ie) established 2008
Network of Networks on Impact Evaluation (NONIE), comprised of: the OECD/DAC Evaluation Network (DACEN), the UN Evaluation Group (UNEG), the Evaluation Cooperation Group (ECG), and the International Organization for Cooperation in Evaluation (IOCE)
International Development Evaluations Association (IDEAS)
Development IMpact Evaluation (DIME)
Spanish Impact Evaluation Fund (SIEF)
What is impact evaluation?
Measuring outcomes (eg learning) is difficult
But measuring outcomes is NOT impact evaluation
measurement of net effects of a program on the outcomes of interest (e.g. on learning)
In IE, it is important to appreciate the difference between correlation and causation
Correlation vs Causation
As ice cream sales increase, number of drowning deaths increases sharply Therefore, ice cream causes drowning
Sleeping with one's shoes on is strongly correlated with waking up with a headache Therefore, sleeping with one's shoes on causes headache.
Taller people have higher earnings Therefore, higher height causes higher earnings
Salary and Height
Salary and Height?
Spurious Relationships This relationship is said to be “spurious” When we did the bivariate relationship, we said,
In reality, things look like this:
SalaryHeight
SalaryHeight
Gender
Think of
Sarva Shiksha Abhiyan
and its impact
Correlation vs Causation In achievement equation with
class-size, the coeff on class-size var is -0.46, i.e. an increase in class-size by 10 is associated with a 4.6 point lower ach mark
Can we say from this that: class-size is negatively related
to achievement? reducing class-size will lead
student achievement to rise? what is the diff between these?
what else determines ach? where is that included in the
way the reg. equation is written
Class size graph
y = -0.4635x + 76.437
01020304050607080
0 20 40 60 80
class size
mar
k
Quantitative impact evaluation methods
OLS production function (not causal relation) Instrumental Variables Panel data Randomised experiment Quasi-experimental
Propensity score matching method Each method has strengths & drawbacks RCT, PSM, Panel, IV
Evidence on impact of policy interventions
Performance related pay Duflo and Hanna, RCT Muralidharan/Sundararaman, RCT
Contract teachers Atherton & Kingdon, panel data Muralidharan et al, RCT Goyal & Pandey, OLS, school FE
DPEP Jalan and Glinskaya, PSM Schmid, IV
Private schools Desai et al, panel French/Kingdon, panel
Mid Day Meal Afridi, 2010, panel
Union membership Kingdon Teal, panel
Greater emphasis on evidence
NCERT evaluated impact of programs in 4 states, under TCF
Data becoming available – EI, ASER, DISE, SEMIS, NAS
What’s imp is the quality of the data / studies, i.e. robustness of methods/ designs; degree to which they tease out causal effect
Long term investment needed in capacity dev
MHRD / JRM approval for estb of National Assessment & Evaluation Centre
Some important considerations
Even evidence showing reliably what works is insufficient
Policy makers need to: seek, read such evidence discern good/bad evidence act on evidence, i.e. make evidence-based policy
Freedom from political interference
Independence from funding body, hence PPP better
Good evidence does not always permit good decisions
Even when we have good evidence a policy has capacity to substantially improve outcomes, there can be powerful political economy barriers to the implementation of policies.
E.g. Duflo and Hanna (Rajasthan) say : “Although this study suggests that a system of automatic monitoring with enforcement by physically remote agents who are prepared to enforce the rules is technically feasible and indeed provides better incentives for teachers, a later effort to introduce this system with higher-skilled, higher-status, and more politically powerful health-care workers ran into strong political obstacles (Banerjee et al. 2007b).” Another e.g. para T in UP
We need to understand the political economy constraints, and how can they be eased
Thanks
TASK WITH DIFFICULT WORDS
Bihar govt. schools
UP govt. schools
Bihar/
ALLUP
Reg.Para
05Para
06 Reg. ParaPrivat
e All
Word meaning
Not attempted 8.8 10.8 10.2 5.0 8.3 12.5 9.4
Wrong meaning 35.5 33.4 35.4 27.2 29.3 32.0 32.2
Partial meaning 12.4 14.8 17.3 16.9 13.5 15.1 14.9
Full meaning 42.4 41.0 37.1 50.8 48.9 40.4 43.4
Four difficult words are given. Please write their meaning using simple words
VOCABULARY TASKS : DO TEACHERS KNOW WORD MEANINGS ?
SUMMARIZING TEXTS : EXAMPLES FROM TEACHERS
TASK WITH PASSAGE
Bihar govt. schools
UP govt. schools
Bihar/
ALLUP
Reg.Para
05Para
06 Reg. Para Private All
Gave meaningful summary?
Not attempted 3.5 2.4 5.6 3.3 2.6 8.6 4.4
Irrelevant/wrong 25.4 27.0 40.5 28.6 40.2 37.9 33.0
Partially meaningful 25.4 15.9 22.5 9.9 16.2 16.4 17.9
Fully meaningful 45.6 54.8 31.5 58.2 41.0 37.1 44.7
SUMMARIZING TEXTS : CAN TEACHERS SUMMARIZE ?
TASK WITH PASSAGE
Bihar govt. schools
UP govt. schools
Bihar/
ALLUP
Reg.Para
05Para
06 Reg. Para Private All
Summary easy to understand
Easy to Understand 71.2 69.6 78.6 79.3 81.3 73.8 75.0
Are there any spelling errors?
No spelling error 32.0 29.1 28.2 54.7 48.2 46.7 39.6
1-2 Spelling errors 44.3 38.5 25.6 30.2 35.5 34.4 35.3
>=3 spelling errors 23.7 32.5 46.2 15.1 16.4 18.9 25.1
SUMMARIZING TEXTS : CAN TEACHER SUMMARIZE
USING SIMPLE LANGUAGE
Of those writing meaningful summary
Bihar UP All
TASK: PERCENTAGE WORD PROBLEM
Reg. Para 05
Para 06
Priv. Reg. Para Priv.
Not attempted 14.4 12.0 26.4 37.0 16.7 23.5 28.6 20.6
Incomplete 32.7 48.8 46.2 25.9 40.0 40.0 54.6 42.6
Wrong steps 5.8 6.4 5.5 11.1 10.0 3.5 1.3 5.7
Correct steps wrong answer 3.9 6.4 3.3 3.7 4.4 7.0 1.3 4.6
Correct answer no steps 0.0 1.6 3.3 0.0 1.1 4.4 2.6 2.1
Solved correctly
43.3 24.8 15.4 22.2 27.8 21.7 11.7 24.5
TASK 4 : PERCENTAGE PROBLEM : Findings
Bihar UP All
AREA PROBLEMTASK Reg.
Para05
Para06-07 Priv. Reg. Para Priv.
No attempt 27.9 28.8 38.5 51.9 30.0 48.7 41.6 36.6
Incomplete 19.2 25.6 26.4 7.4 18.9 19.1 26.0 21.8
Wrong steps & A 5.8 4.0 1.1 3.7 7.8 3.5 2.6 4.1
Correct steps, wrong A 3.9 3.2 8.8 0.0 4.4 1.7 5.2 4.1
Only correct A, no steps 4.8 5.6 3.3 0.0 8.9 4.4 9.1 5.5
Solved correctly 38.5 32.8 22.0 37.0 30.0 22.6 15.6 27.9
TASK 5 : ARITHMETIC : AREA PROBLEM
BIHAR UP
Fully agree
Parti ally
agree
Somewhat agree
Dis agree
Fully agree
Parti ally
agree
Somewhat agree
Dis agree
Govt. school teachers
24.5 11.0 46.8 17.7 15.2 18.3 43.1 22.3
Private school teachers
16.7 12.5 45.8 25.0 16.9 18.5 36.9 27.7
% teachers who agree with the statement“Sometime I have difficulties in addressing mathematical queries and
problems of my students”
80% primary school teachers have difficulties in teaching maths
Only about 20% of govt. school teachers believe they don’t face problems. About 80% admit to have difficulties sometimes. This suggests possible interest in in-service training to upgrade maths skills