1

Review of Toolkit

Amber GoveRTI International

Session 3.4

2

Potential Purpose(s)

1. How is the system doing? Raise government/public awareness of a national or regional problem

2. How is the project doing? Evaluate discrete interventions

3. How are students doing? Help teachers track student progress in the lowest primary levels, before students have problems and drop out

In all cases, the end goal is to provide resources for interventions that clearly address the problem.

3

2. RemediateUse evidence-based

instructional approachesand support to teachers

to improve student learning.

3. MonitorDevelop progress

monitoring tools and foster accountability

for meeting goals

1. DiagnoseUse EGRA to identify instructional needs, raise awareness,

and set system goals

The continuous cycle of improving student learning.EGRA and EGRA-based assessments can be used to diagnose, remediate and monitor progress toward improving student learning outcomes.

2. RemediateUse evidence-based

instructional approachesand support to teachers

to improve student learning.

3. MonitorDevelop progress

monitoring tools and foster accountability

for meeting goals

1. DiagnoseUse EGRA to identify instructional needs, raise awareness,

and set system goals

2. RemediateUse evidence-based

instructional approachesand support to teachers

to improve student learning.

3. MonitorDevelop progress

monitoring tools and foster accountability

for meeting goals

1. DiagnoseUse EGRA to identify instructional needs, raise awareness,

and set system goals

The continuous cycle of improving student learning.EGRA and EGRA-based assessments can be used to diagnose, remediate and monitor progress toward improving student learning outcomes.

4

Lessons from Pilots: Phases of EGRA

1. Planning and Development

2. Adaptation and Training

3. Field Work

4. Data Entry and Analysis

5. Dissemination and Reporting

5

Planning and Development

Review purpose with funder and Ministry

Involve/consult with teachers, ministry staff, parents, survey team, curriculum experts

Develop a detailed workplan, clarify roles and responsibilities

Determine use of results and dissemination plan

Carefully develop sample to fit purpose, budget

6

ACTIVITIES Responsible Other actors AG SLT

Adaptation of EGRA instrumentsPrepare materials and presentations Gove Linan-Thompson 3 1 JumeMeet with Honduran specialists to review objectives

World BankGove;Linan-Thompson, Honduran specialists

2 1 June

Adapt existing instruments to Honduran Gove Honduran specialists TBD 2 2 JunePre-test of instruments in 2 schools, ~30 children, revise

Honduran specialistsGove;Linan-Thompson, Honduran specialists

1 2 June

Identification of pilot and baseline samples Honduran specialists Gove (review) 1 SeptPilot fieldwork arrangements (transportation, equipment, contact w/ pilot sites)

Honduran fieldwork team TBD

Gove (review) 1 Sept

Pilot with EGRA Instrument, Training and Training of enumerators Gove Gove; Honduran team 3 SeptPiloting of instruments in 6 schools with ~ 180 children

Honduran team Gove 2 Sept

Pilot study data entry and management Gove Honduran team 1 SeptAnalysis of results and modification of instruments as needed

Gove Gove; Honduran team 1 Sept

Validation of final version of instruments Gove Honduran team 1 SeptIdentification of baseline sample of 60 schools

Honduran specialists Gove 1 Sept

Trip report Gove 1 Oct

DateLOE

7

ACTIVITIES Responsible Other actors AG SLT

Baseline assessment of EGRABaseline fieldwork arrangements (transportation, equipment, sites)

Honduran team Oct

Fieldwork in 60 schools with ~ 1800 children (30 children / school)

Honduran teamGove (long distance trouble shooting)

2 Oct

Baseline assessment analysis and Data entry and management Honduran team Gove (review) 1 NovAnalysis of results, including instrument psychometrics

GoveLinan-Thompson; Honduran specialists

4 Nov

Preparation of draft Ministry report (~10 pages)

GoveLinan-Thompson; Honduran specialists

5 3 Nov

Preparation of draft World Bank report (~15 pages)

GoveLinan-Thompson; Honduran specialists

7 3 Nov

Report review and feedback World Bank Honduran specialists Nov

Preparation of revised, final reports GoveLinan-Thompson; Honduran specialists

5 5 Nov

Production of final report (editing, RTI Gove (review) 0.5 NovWorld Bank presentation World Bank Gove; Honduran specialists 0.5 Nov

Total LOE : 45 17

DateLOE

8

Adaptation and Training

Start with complete, but draft version

Adapt with teachers, survey specialists, curriculum experts (bring textbooks, word lists, sample stories)

Balance economy and desire to include everything

Pre-test and Pilots serve both training and instrument improvement opportunities (stopwatch practice; 0 vs. missing)

Changes must be evidence-based

Assess Inter-Rater Reliability using audio recordings

9

Monday Tuesday Wednesday Thursday Friday

IntroductionsLogisticsReview EGRA development processReview Research underlying EGRA

Review draft instrument and student questionnaireRevise draft instrument

Application training and practice

Pre-test in 2-6 schoolsEnter data as a practice exerciseConduct simple analyses using excel

Discuss resultsDiscuss instructional implicationsRevise draft instrument based on results

10

Monday Tuesday Wednesday Thursday Friday

Review of underlying principlesReview draft instrumentReview draft teacher questionnaireApplication training and practice

Application training and practiceReview roles and responsibilities of supervisors and enumerators

Pilot in 3 to 6 schools (determined by the number of teams being trained) Enter dataAnalyze results and modify instrument

Print final version of the instrumentApplication training and practice

Prepare and pack materialsFinalize and review logistics

11

Field Work

1 Supervisor plus 3 to 4 Enumerators

Timely transport is critical: arrive before school starts and meet, greet, thank director and teachers

Avoid false negatives: be nice to the children

Sample across all classrooms using teacher lists; start with youngest (1st grade) students

Prior to leaving the school: verify completeness and avoid missing data (waste of everyone’s time)

12

Data Entry and Analysis

Involve supervisors in data entry if possible

Develop and test data entry program in advance

Clarify difference between 0 and missing

If using excel, ensure consistency of entry

Use analysis as opportunity to work with ministry staff

Develop strategies for analysis in excel

13

Variable name Variable description Coding notes Clread Correct letters read Blank means that the task

was not continued Clseconds Time in seconds to correctly read

letters Blank means that the task was not continued

Clpm Fluency in correct letters read per minute (created variable)

Zero means that no words were read as the task was not continued

Cwctread Correct words in connected text read

Blank means that the task was not continued

Cwctseconds Time in seconds to read connected text

Blank means that the task was not continued

Cwpmct Correct words per minute in connected text (created variable)

Zero means that no words were read as the task was not continued

14

Grade Variable 1 2 3 Correct letters per minute 12.8 25.4 36.1 Correct letters per minute, excluding non-readers 19.8 28.6 37.7 Correct familiar words per minute 1.2 2.3 4.3 Correct familiar words per minute, excluding non-readers 6.2 8.3 9.2 Correct words per minute, connected text 2.2 4.0 9.2 Correct words per minute, connected text, excluding non-readers 11.0 11.6 17.3

15

Grade 1 2 3 Count 419 389 392 Average correct letters per minute 12.8 25.4 36.1 Standard deviation 18.2 21.9 23.2 Standard error 0.89 1.11 1.17

16

Dissemination and Reporting

Develop different reports for different audiences (parents, teachers, ministry staff, testing specialists)

Recall purpose is to improve, not blame; use results accordingly

Discuss next steps, teacher training

Provide solutions to the challenges identified

Use results to mobilize for improvement

17

1 Not even tracked or benchmarked past kindergarten, on the assumption that this skill is dominated in Kindergarten. Note the low level of the average, for Grade 2, for the case in the table.

Example of possible benchmarking exercise

Average across children in a given country, middle of

year, Grade 2

Maximum school-level average in the

same country, Grade 2

Developed-country

benchmarks for comparison purposes

Letter recognition fluency 22.7 41 40 by end of

year kindergarten1

Nonsense word fluency 7.5 25 50 in middle of

Grade 1

Connected text fluency 11.4 36 20 in middle of

Grade 1 Comprehension score 0.4 2 NA