Trialogue ConferenceMay 2011

• Introducing the Zenex Foundation

• The Approach to Monitoring and Evaluation

• Processes

• Lessons and Challenges

The Presentation

Introducing the Zenex Foundation• Independent Private Foundation dedicated to funding Maths, Science &

English education

• Funded through an Endowment Fund that has its historical roots with

ESSO Oil and Zenex Oil

• Operating since 1995 and invested over R300 million in education projects

• Strong Focus on Evaluation and has commissioned over 40 evaluations

•.

Developing schools for

Maths, Science and Language

Excellence

OutcomeImproved learner performance using a pipeline approach across primary and high schools.

The Zenex Foundation in the donor sector is the funder, innovator and think tank

for improving learner performance in mathematics, science and language through

interventions, research and evaluation in schools

Teacher Development Programmem

e

OutcomeIncreasing numbers of professionally qualified teachers.

Learner Development Programme

OutcomeIncreasing numbers of learners with quality passes

Research and Development

OutcomeImpact on education policy and classroom practice

Cross cutting Programme: Capacity Building and Evaluation

Maths, Science and Language

Impact? Grade R Learners

Foundation Phase Learners

FET Phase Learners

Principals Teachers

Approach to M & E It has been a general weakness in South Africa donor practices

• Seen as a unnecessary additional expense that could have gone directly to development

• Too little is spent on poorly designed evaluation studies that can tell you very little on the

impact of the project

Purpose of M & E in Zenex

• Accountability: addresses the question of impact of the project

• Improve and inform project delivery

• Contributing to Zenex Strategy Deveopment

• Learning and developing best practice

Approach to M & E• M&E is built into the entire project management cycle and involves all role-players in the

project. The approach is ‘as far as possible’ consultative and participatory.

• We use the Logic Model in designing evaluations and this is reflected in project design

through to evaluation design. It is important to unpack the implicit Theory of Change

underpinning the project intervention

• We mainly use quasi-experimental design with a combination of both qualitative and

quantative methods

• Guide 10% of the project costs and this is relative to the size of the project. Often

this costs have exceeded the 10% benchmark

ProcessesChoosing Evaluators

• Zenex does not go the “tender route” and this is based on the rationale that there is a small

pool of evaluation specialists working in the education sector.

• Worked with both individuals and agencies: Criteria include

Independence

Range of experience in various methods

Contextual/Sector knowledge

Technical knowledge in evaluation design

• Committed to developing a cadre of Black evaluators and we budget and plan for this with

coaching and mentoring.

Teacher ProgrammeProgramme Evaluation Design and Methods

TEACHER• Formative • Review of Models• Impact

TeachersInterviewsClassroom Observations of Teacher Practice Review of Portfolios prepared by teachers Review of Exam resutls

LearnersTesting in Content Knowledge: pre & post Focus Groups

Project Partner: Secondary Analysis of documentsObservations of Academic Training and classroom supportInterviews

Time Span: During the proejct intervention and one year tracking

Learner Programme Programme Evaluation findings

LEARNER• Formative• Review of the models• Impact

LearnersInterviewsTesting in content

SchoolInterveiws with teachers and principalsClassroom obsrvations

Project Aprtner: Project Activiteis reveiwed: Mentor meetings, camps, meetings, teacher training reveiwed

Time Span Follows proejct implementation and tracks learners post school for two eyars

School Programme Programme Evaluation findings

School • Formative• Review of the models• Impact

Who• Evalauotors• Service Providers• Zenex Team School Results

LearnersTesting in content Learner Book Reviews

Teachers InterviewsClassroom Observations

SchoolInterveiws with principalsSchool Audit

Project Aprtner Service Provider Interventions Workshops, in-class support.

Time Span Follows proejct implementation

Lessons & ChallengesWhat is working

• Evaluations shaped Zenex’s strategy

• Negotiate/consult with partners from position of documented research

• Early identification of problem areas, allows for problem solving and time for reflection

• Led to capacity building programmes identified during evaluation process

• Allows for realistic setting of results and adjustment of project results

What have we learnt

• Alignment of Roles with so many role-players involved in the proejct all engaged in some form of M & E

A classic example is the multiple testing of learners by different role players.

• Relying one External sources of data in evaluation studies which have not always been reliable – for

example Grade 11 examinations

• Realizing the need to investment in feasibility research before project implementation

• Compromising on evaluation design due to budgetary constraints and reducing the value of the study.

• Understanding the lag effect education interventions extending the length of the evaluation post the

project interventions

Lessons & ChallengesWhat is working

• Evaluations shaped Zenex’s strategy

• Negotiate/consult with partners from position of documented research

• Early identification of problem areas, allows for problem solving and time for reflection

• Led to capacity building programmes identified during evaluation process

• Allows for realistic setting of results and adjustment of project results

What have we learnt

• Alignment of Roles with so many role-players involved in the proejct all engaged in some form of M &

E

A classic example is the multiple testing of learners by different role players.

• Relying one External sources of data in evaluation studies which have not always been reliable – for

example Grade 11 examinations

• Realizing the need to investment in feasibility research before project implementation

• Compromising on evaluation design due to budgetary constraints and reducing the value of the study.

• Understanding the lag effect education interventions extending the length of the evaluation post the

project interventions

Lessons and ChallengesChallenges to Address

• The Zenex grant making cycle only allows for engagement with potential evaluators after

the project proposal has been approved often leading to delays in commissioning evaluations

• Continuous use of the same team of evaluators and researchers: Question of familiarity?

Quality Assurance? Should Zenex look at appointing technical advisors?

• Development Debate: the cost of the evaluation and the outcome: if a project is poorly

designed the evaluation may not be able to reveal very useful information – how evaluable

is the project?

Thank You