Impact evaluation: External and internal stakes

Impact evaluation: External and internal stakes

Impact evaluation seminar - 2 to 6 December, Phnom Penh

International context and stakes

Uncertainty on aid, Paris Declaration 2005, achievement of MDG (Millennium Development Goals) and after-MDG

Highlights the Results-Oriented Management (ROM) Focuses on results/effects/impact Insists on participative approach

Since the 90s, increasing initiatives to improving quality and impact of humanitarian action

Sphere, HAP Standards in Accountability and Quality, Code of behaviour, Don not harm, etc.

Increasing concern for understanding changes induced by humanitarian aid


• Stake of accountability: Stakeholders have a moral obligation to allocate necessary means to the purpose of their mission and to prove the changes made by their action;

• Stake of visibility: Need for demonstrating one’s added-value among an increasing number of stakeholders in the humanitarian sector and international solidarity;

• Stake of financial funding: donors prescriptions to prove impact and/or changes along with the development of new tools and approaches (Value for money, Theory of change, etc. )

International context and stakes


How we keep busy How we make a difference

Outputs Outcome ImpactInputs

Results Chain

Crossing the red line....

Définitions Outputs/Extrants/Produits: The immediate results of the organisation’s activities – the processes, goods and services that it produces. The organisation controls its outputs.→ 40 health professionals are trained on inclusive approach, reception and follow-up of people with disabilities

Outcomes/Effets: The likely or achieved short-term and medium-term effects of an intervention’s outputs. They contribute to the improvement of people’s life. The organisation only influences outcomes, besides external factors.→ Improved access to health services for persons with disabilities → Improved health conditions of persons with disabilities

Impact:Long-term, sustainable changes in the conditions of people and the state of the environment that structurally reduce poverty, improve human well-being. Your organisation contributes partially and indirectly to these enduring results in society or the environment.→ Improved quality of life of persons with disabilities


1950/60s 1980/90s1970/80s

Impact analysis ex ante to anticipate the economic effects of

interventions

Taking account of unexpected and negative effects, one-dimensional models (Cost-profit analysis, social

impact measurement, environmental impact measurement)

Evolution towards a multidimensional comprehension

of impact, development of ex-post evaluations

Linear approach of change, target populations as objects of studies and not actors of change

Rural assessment participative approach, populations as actors of

change

A short story of impact

2000 →

Elaboration of new approaches; theory of

change, outcome mapping

Two main approaches

Usual methods of impact evaluation: different types of studies and evaluation, sometimes requiring important resources;

Approaches measuring effects by being integrated within the intervention with the use of tools monitoring changes at different levels (short and mid-term effects, sometimes impact)E.g.: The planning, Monitoring and Evaluation tool of ACTIONAID, highlights learning and ownership (ALPS for Accountability, learning and planning system)


Why measuring impact?

CREDIBILITY

LEARNING

ACCOUNTABILITY

ADVOCACY IMPACT

What has changed?

For whom?

How significant was it?

Will it last?

In what ways did we contribute to these changes?

Key questions on impact measurement


Responsibility of impact


« The systematic analysis of lasting or significant change –positive or negative,

intended or not – in people’s lives brought about by an action or a series of actions »

(Roche, OXFAM)« Measuring the changes taking place as the

result of an emergency project (The Good Enough Guide Impact Measurement and

Accountability in Emergencies)

One or several definitions of impact?


Common elements of definition: • concept of change/effects,• changes are positive or negative,• changes are expected/desired or not

Elements of definition adapted to each intervention type and environment:

• the object of change,• the sustainable, significant or structuring aspects of change

Towards an open definition?


Impact measurement challenges

When initiating an impact evaluation?

Which approach? What level of investigation?

What resources?

What participation of the different stakeholders?

What responsibility to changes?


“Handicap International reaffirms the importance of taking effective action, generating positive and

measurable effects and introducing changes that bring about tangible improvements for the beneficiaries, and for people with disabilities in particular. For Handicap

International, the development of services and facilitating access to these services are essential to achieving a

positive impact on the living conditions of beneficiaries».

2011-2015 Federal Strategy Chapter 2 - An approach centred on

achieving a measurable impact

HI and partners’stakes in Impact measurement

Sharing common references and terminology

Proposing and implementing rigorous methods and tools adapted to projects and intervention contexts

Asking the good questions to each effect or impact measurement initiative: What, when, how, why? Integrating effect or impact measurement within or Monitoring-Evaluation tools and systems

Supporting a balance between the quantitative and qualitative analysis

Analysing and taking into account negative effects of projects

Séminaire Evaluation d'impact du 2 au 6 décembre, Phnom Penh

Seminar objectives Identify and analyse HI and partners’ practices on effect and impact measurement, share experiences, draw recommendations for improving tools and methods;

Identify, by technical domain and by intervention context, the areas of change expected, the different levels of change and the related indicators;

Evaluate the possibility of integrating and implementing new approaches of change or impact measurement; criteria of choice and identification of resources;

Identify the ways of integrating change follow-up and measurement into Monitoring-Evaluation tools

Séminaire Evaluation d'impact du 2 au 6 décembre, Phnom Penh

LET’S GO TO WORK!


Documents

Impact evaluation: External and internal stakes