South African Extraordinary Schools Coalition

Monitoring and Evaluation 7 Feb 2013-The Framework -Planning to Measure Impact- What we know about costs

Today’s Presentation» Purpose

˃ Shortly share the evaluation framework again to ensure everyone knows why we are spending time on this

˃ Discuss some ideas about how we will be measuring Impacts (By the May Meeting)

˃ Share some of the preliminary evaluation findings relating to the budget / cost analysis and get inputs on a membership typology in terms of cost

The DRAFT M&E Framework

The Coalition already decided that M&E is…» Key to its sustainability, long-term growth and

continued relevance to the learning community served by the Coalition.

» Will essentially address three key questions:

How is the Coalition

performing?

How can the Coalition be

strengthened?How is the Coalition planning for sustainability?

Purpose of the Coalition’s M&E» Learning: It is anticipated that the M&E will be used by

the coalition’s project manager and key decision makers to inform its future work, and improve its practice.

» Accountability: To prove to its donors that the resources provided to the organization is spent well and is delivering good results

» Building the Knowledge base: To supplement the management of Bridge’s knowledge base about how communities of practice like this one can function optimally.

How is the Coalition performing?

How can the Coalition be strengthened?

How is the Coalition planning for sustainability?

The Coalition’s Evaluation Criteria» It is important to consider the relevance,

effectiveness, impact, budget and timeline compliance, and sustainability of the Coalition

(Organization for Economic Cooperation and Development – Development Assistance Committee Criteria)

» If the coalition is successful we would expect˃ People to react positively ˃ People to learn something (Skills, knowledge, values &

attitudes)˃ People to change their individual behaviours˃ Organizations to start transforming

Donald Kirkpatrick, the Strategic Training Model

How is the Coalition performing?

How can the Coalition be strengthened?

How is the Coalition planning for sustainability?

The Coalition’s Evaluation Questions

Relevance

Sustainability

Budget and Timeline Compliance

Effectiveness

Impact

The Coalition’s Relevance QuestionsRelevance is the extent to which the coalition and its activities are suited to the priorities and policies of the target group, recipient and donor.

How has the coalition’s objectives and activities changed since inception, and why?

To what extent are the objectives of the coalition still valid? Are the key activities and outputs of the coalition still relevant to the

overall goals of the coalition? Are the key activities and outputs of the coalition still relevant to the

intended o impacts and effects influencing policy and practice, (finding a

solution to the education crisis)?o impacts in schools?

POSSIBLE EVALAUTION METHODS: Interviews / survey with key stakeholders and thematic analysis. Review of documented objective statements over time.

The Coalition’s Effectiveness Questions

Effectiveness is a measure of the extent to which the coalition attains its objectives.

To what degree did the coalition support its membership base to achieve and maintain the coalition’s values / minimum criteria?

To what extent were the planned objectives achieved / are likely to be achieved?

What were the major factors influencing the achievement or non-achievement of the objectives?

POSSIBLE EVALUATION METHODS: Outlined in relation to each of the aspects contained in the logical framework

ACTIVITIES OUTPUTS OUTOMES IMPACTS

4 x National workshops per year where each of the five work streams meet

Seventeen participating schools attend 4 x national workshops (30 – 50 participants)

2011 – 2012 All

participating schools send at least one participant to each of the meetings

2013: All participating

schools send at least two participants (Leader and an instructional practitioner) to all national workshops

By August 2012: Involvement of education department stakeholders, and education agencies in national workshop.

Satisfaction: Members report that the coalition provides a safe space for difficult conversations (trust) (Members Survey) Attitudes: Members feel that the coalition members have a common purpose (Analyse meeting notes, Members survey) Knowledge: Individual participants and Member schools as a whole shared and learnt new working practices from one another(Analyse meeting notes for evidence of sharing, survey asking for concrete examples rated on a rubric) Behaviour: New working practices adopted by participating individuals and schools as a whole (Survey asking for concrete examples rated on a rubric)

More coalition schools can show strong evidence of maintaining criteria for membership and implementation of principles of excellent practice(Annual reports from schools on criteria, Stories / Case studies for principles) Schools improve performance on reported peer review results over time (approximately 3 year cycle)Peer review reports where at least two are available for each school)

The Coalition’s Impact QuestionsThe positive and negative changes produced by the coalition, directly or indirectly, intended or unintended.

• What were the intended / unintended changes, that the coalition contributed to at different levels?

• How have individuals and member organizations used the gains from the coalition meetings?

• How many people have been affected?o Participation in meetings / peer review / social mediao Number of children in Coalition schools, number of success cases in

Coalition schools

POSSIBLE EVALUATION METHODS: Monitoring records on participation and school enrolment figuresSocial media monitoring (Twitter, Facebook, Bridge online)Interviews / Survey questions

Questions about Budget and Timeline Compliance

A consideration of the actual costs required to run the coalition in addition to the budgeted grant amounts, and an indication of the extent to which the coalition met timelines

What financial and non-financial contributions did the coalition

leverage for its work over time? (e.g. core donor funding versus inputs from schools vs. funding from other donors)

In which ways did the coalition contribute towards using the financial and other contributions optimally (i.e. did the cost of participation go up or down over time?)

Did the coalition meet key project milestones timeously and if not,

why not? Were the meeting outputs made available to all participants

timeously, and if not, why not?

POSSIBLE EVALUATION METHODS: Cost analysis, Verification of milestone analysis

The Coalition’s Sustainability Questions

Sustainability is concerned with measuring whether the benefits the coalition are likely to continue after donor funding has been withdrawn.

• Is the coalition still valuable enough to keep participants involved?• Are the networks that were established of significance to sustain

future maintenance of the coalition?• Does the coalition have a concrete plan to sustain its activities

should current funding cease? (E.g. Are Network champions developed?)

• Is the model of the coalition scalable / replicable? • What are the risks to sustainability? POSSIBLE EVALUATION METHODS: Stories / Survey, Key informant interviews

Pause and Reflect» Questions or Inputs?

How to measure Impact - We need your input

The Coalition’s Impact QuestionsThe positive and negative changes produced by the coalition, directly or indirectly, intended or unintended.

• What were the intended / unintended changes, that the coalition contributed to at different levels?

• How have individuals and member organizations used the gains from the coalition meetings?

• How many people have been affected?o Participation in meetings / peer review / social mediao Number of children in Coalition schools, number of success cases in

Coalition schools

POSSIBLE EVALUATION METHODS: Monitoring records on participation and school enrolment figuresSocial media monitoring (Twitter, Facebook, Bridge online)Interviews / Survey questions

The Impact Story Template» At the previous meeting, John asked for some

examples of changes / impacts that the coalition members have noticed as a result of participating in the Coalition

Some Stories…• The HODs that participated in the school peer review have come back with a direct leadership impact in instructional practice • Looking at the young leaders and young interns – what is possible in the future –intern programme and gift of transgenerational sharing

• Follow-up on instructional leadership and personal management by two of the school leaders following the SPR • Learnt from the timetable structuring at Lebone (SPR)

•Talking about classroom practice – practice of classroom writing – first 10 minutes in the classroom• Comment on student uniforms – looking at this as a change• Staff development

To make sense of these stories, we need to know a few more things!

The Impact Story Template

Example Story» Curriculum change – we are embarking in 2013 on an

integrated curriculum approach in gr 8. A team has met and redesigned our offering of 6 subjects. This is a direct result of seeing this operative at Lebone 11.

Example Story» Curriculum change – we are embarking in 2013 on an

integrated curriculum approach in gr 8. A team has met and redesigned our offering of 6 subjects. This is a direct result of seeing this operative at Lebone 11.

Example Story» Curriculum change – we are embarking in 2013 on an

integrated curriculum approach in gr 8. A team has met and redesigned our offering of 6 subjects. This is a direct result of seeing this operative at Lebone 11.

Example Story» Curriculum change – we are embarking in 2013 on an

integrated curriculum approach in gr 8. A team has met and redesigned our offering of 6 subjects. This is a direct result of seeing this operative at Lebone 11.

Example Story

Example Coding» Impact: 1-5 Individuals, 6 – 30 Individuals, 31 – 100

individuals; Organizational; System Wide» Impact on Learners: Significant, Tenuous, Not Applicable» Impact on Teachers: Significant, Tenuous, Not Applicable» Change: Value; Attitude; Skill; Knowledge, Individual

Behaviour, Organizational Behaviour» Verified: Yes / Partly / No / Not Yet

Pause and Reflect» Does the template work?» Any suggestions?

Do Now! – More Story Leads» Can you think of any evidence that relates to you,

your colleagues or your organization that could help us to document examples of how the Coalition helped to: ˃ Build and improve the reflective practice capacity of individuals, schools and

organizations˃ Foster the adoption of new working practices by participating individuals,

schools and organizations? ˃ Reduce the duplication of effort?˃ Establish effective partnerships?˃ Expand reach?˃ Influence broad policy change?˃ Support Coalition schools to maintain criteria for membership?˃ Support Coalition schools to implementation of principles of excellent

practice ?

The Budget / Cost Analysis

Questions about Budget and Timeline Compliance

A consideration of the actual costs required to run the coalition in addition to the budgeted grant amounts, and an indication of the extent to which the coalition met timelines

What financial and non-financial contributions did

the coalition leverage for its work over time? (e.g. core donor funding versus inputs from schools vs. funding from other donors)

In which ways did the coalition contribute towards using the financial and other contributions optimally (i.e. did the cost of participation go up or down over time?)

Did the coalition meet key project milestones timeously and if not,

why not? Were the meeting outputs made available to all participants

timeously, and if not, why not?

POSSIBLE EVALUATION METHODS: Cost analysis, Verification of milestone analysis

Budget and Cost Analysis» Two Sources of Data – ˃ From the Coalition’s Financial Records

+ Good for calculating actual financial contributions / expenses

+ Not good for calculating in-kind contributions / expenses

˃ From the Coalition’s self-reported information + May be difficult to reconstruct for each event

after the fact+ Might be possible to come up with a model of

“typical” cost contributions

Cost Estimates Claims (MSDF Contribution)

In Kind Contribution

Other Donor Contributions (In Kind)

Meeting Expenses R 85 800 R 30 645 R 55 155 Venue R 30 000 Catering R 55 800 Office Supplies R - R 523 R -

Travel R - R 341 300 R 121 174 R 220 126 Accommodation R 39 900 Flights R 180 000

Ground Transport R 121 400

Project Manager & Bridge Staff R - R 863 679 R - Other Professional Fees R - R 266 470 Overheads R - R 98 005 R 266 470 Special Projects R - R 320 000 SUBTOTAL R - R 1 114 026 R 541 751 R 320 000 For every One Rand the coalition contributed R 2.06 R 1.00 R 0.59 For Every One Rand the MSDF contributed R 1 R 0.49 R 0.29

Cost Metrics» Leverage (per year)» Cost per workshop participant per day (per

year)

Costs & Contributions to Account for

» National Meetings˃ Venue˃ Facilitation˃ Catering (x number of participants attending)˃ Participant Accommodation (x number of staff attending)˃ Participant Flights & Ground Travel (x number of staff attending)

» Peer Review˃ Reviewers Fees˃ Catering˃ Reviewer Accommodation˃ Reviewer Flights & Ground Travel

» Steering Committee˃ Participant Fees

Costing Typology» Typology

˃ Travel and accommodation at National Meetings (Might differ for meetings in own province / other province)

+ Level 1: Claims for most costs for one person, do not send more staff at own costs+ Level 2: Claims for most costs for one person, sends one or more staff at own costs+ Level 3: Do not claim any costs, one participant+ Level 4: Do not claim any costs, sends 2 – 3 participants+ Level 5: Do not claim any costs, sends more than 3 participants

˃ Hosts a meeting (National Meeting / Peer Review) and provides for most of the catering costs+ Level 1: Provides Venue free of charge+ Level 2: Provides Catering and Venue free of charge+ Level 3: Provides Catering, Venue and Accommodation

˃ Member that actively contributes time to facilitate National Meetings / Participate as reviewers / lead reviewer

+ Level 1: Spends time facilitating at least one session at a National meeting + Level 2: Spends time participating in steering committee meetings and decision making+ Level 3: Spends time participating as a reviewer in at least one peer review+ Level 4: Spends time participating as a lead reviewer in at least one peer review+ Level 5: Multiple engagement in various of the aforementioned activities

Pause and Reflect» Is this a credible approach?» Any suggestions?

Next steps – What do you propose?

1 •FRAMEWORK

2 •IMPACT STORIES

3 •COST DATA

4 •OTHER?

Thank you for your participation!

Contact Details:Benita WilliamsFeedback Research & Analytics1102 Burnette Str. (Corner of Hilda), Hatfield, PretoriaFax/Phone: (012) 430 2009E-mail: bwilliams@feedbackra.co.zawww.feedbackra.co.za