Designing and managing projects for impact at scaleDr Andy Hall • Research Group Leader - Global Agriculture Innovation Dynamics

12 November 2015

Objectives of today

• Understand Project Design and MEL are part

of the same process

• Understand how to develop the activities,

output, outcome, impact logic of projects

• Understand the range of tools used for

different aspects of monitoring evaluation and

learning.

• Understand how to select relevant tools for

different MEL tasks

Designing and managing projects for impact at scale • Dr Andy Hall2 •

Recapping key terms

• Project purpose: The development challenge that the project is contributing to

resolving

• Project outcome: Changes in behavior of farmers, traders, policy makers and

other actors related to the project purpose that result from the activities and

outputs of the project

• Project output: New information, ideas, technologies created by the project.

• Project activity: Actions implemented by the project, including research.

Designing and managing projects for impact at scale • Dr Andy Hall5 •

Worked example from AAPP

• Growth and development of food systems in east Africa

• Researchers, farmers, private sector collaborate to develop and apply new

knowledge

Designing and managing projects for impact at scale • Dr Andy Hall3 •

AAPP ground rules

• All projects had to identify a development purpose that they were contributing

to: e.g. consumers have reduced exposure to aflatoxin.

• All projects had to identify 2 major types of outcome that would contribute to

the purpose (end of project deliverables)

• Adoption of technology/policy/standards

• Researchers, farmers, private sector, NGO, etc. working in new ways that deliver

solutions.

Designing and managing projects for impact at scale • Dr Andy Hall6 •

Design process• Build on earlier research and research teams

• Extensive collaborator and stakeholder workshops and consultations

• Development of problem trees (identifying a interconnected set of problems)

• Development of project outcomes, outputs and activities needed to address those

problems.

• Development of a theory of change

• Development of an MEL systems to track progress on outcomes

Tip: Don’t start with research you would like to do. Start with the problem you

want to solve. 6 • Designing and managing projects for impact at scale • Dr Andy Hall

Identify the development challenge

7 •

Outputs & Outcomes to Achieve Impact

8 •

A Theory of Change

9 • Designing and managing projects for impact at scale • Dr Andy Hall

• Technology adoption and impacts:

– selecting indicators and data collection methods

• Tracking effectiveness of collaboration:

– Quality or quantity? Scope, Relevance?

• How stakeholders and systems respond:

– Unpredictable and unexpected. But project needs this information to keep on

track.

MEL challenges

10 • Designing and managing projects for impact at scale • Dr Andy Hall

11 •

Morning coffee break

Designing and managing projects for impact at scale • Dr Andy Hall

12 •

Current State: what are we doing at the moment

Designing and managing projects for impact at scale • Dr Andy Hall

You have 5 minutes to think about the following questions. Use your note pads to capture your

thoughts. After your 5 minutes of individual reflection we are going to discuss what you currently

do in terms of MEL as a group.

Question 1: If you do MEL why do you do it?

Question 2: Do you have any particular approaches or tools you use?

Question 3: Have you found success with any particular approach or tool used, if so why?

Question 4: What do you use the information collected and analysed as part of your MEL for?

Question 5: Are there any limitations to these tools?

Exercise 1: What do you currently use?

13 | Designing and managing projects for impact at scale • Dr Andy Hall

14 •

What is a MEL System

Designing and managing projects for impact at scale • Dr Andy Hall

What is the MEL System about?

15 •

What: Describe the change

Why: Explain the change

Is it good enough: Evaluative part

DATA

DIALOGUE

Designing and managing projects for impact at scale • Dr Andy Hall

Why M&E&?• Project design: Contributing to the project design process by developing a logic that links project

activities to a stated project outcomes and purpose.

• Project learning performance management/ monitoring: Organises the collection of information

during the life of the project to make judgments about the progress, relevance and effectiveness

of the project activities towards the achievement of a project outcomes and purpose.

• Project accountability/ evaluation: Organises the collection of information about the levels of

outcomes and purpose (impact) achieved to report to donor and other stakeholders.

• Program learning: Organises the collection of information about generic lessons from the

implementation of projects/ clusters of projects that can inform project and program practice in

the future.Designing and managing projects for impact at scale • Dr Andy Hall16 •

17 •

Different MEL Tools & approaches for different purposes

Designing and managing projects for impact at scale • Dr Andy Hall

Indicators

18 |

INDICATOR ASSUMPTIONSMEANS OF

VERIFICATION

Designing and managing projects for impact at scale • Dr Andy Hall

Activity plans and milestones

19 • Designing and managing projects for impact at scale • Dr Andy Hall

Internal activity monitoring

Presentation title • Presenter name20 •

Technology adoption/ impact tracking

Presentation title • Presenter name21 •

• Base line surveys and end of project / post project resurveys

• Rapid appraisals and ad hoc studies – valuable for keeping an eye

on progress and getting feeds back on weather the project

activities are leading to expected results.

Rubrics

22 | Designing and managing projects for impact at scale • Dr Andy Hall

Outcome rating report

Presentation title • Presenter name23 •

Implementation and output performance

statements RATING Commentary, evidence and prospects

Learning events are relevant, timely, appropriately designed; involve Australian and/or in-country partners and their networks; and are perceived as worthwhile; GOOD

A series of training events on Gender and value chain development, DCED standards and impact investing have been delivered and well received. An ambitious series of events are planned that respond to DFAT priorities. See www.foodsystemsinnovation.org.au/events Reflection events with DFAT in country programmes have been slow to materialise, although concrete plans are in place to conduct 2 events in the next quarter

Knowledge products are relevant, practice-based and practice-oriented, and aligned with current and emerging (agriculture food, and nutrition) international development thinking, practices, and needs within Australia and overseas; collaboratively-created, reader-friendly and audience-appropriate, and produced and delivered in a timely manner;

ADEQUATE

A series reports synthesising international perspectives on aid for trade have been well received and are frequently cited. Inputs have been made into DFAT strategy and operational guidelines. A dossier on improving nutrition through agriculture linkages has been published. A practice note series has been slow to take off. A renewed effort to develop practice note collection addressing FSI priorities is being progressed by bringing in additional expertise and external commissioning. A recently commenced line of research on inclusive agribusiness innovation will deliver high profile knowledge products by the end of the project. See www.foodsystemsinnovation.org.au/resources

An expanded range of relevant experts and international development practitioners who actively contribute to FSI Australian and in-country partners’ and their networks’ discussions, designs, and practices in ways that are perceived as collaborative, salient, credible and useful.

ADEQUATE

External expertise has been used in the areas of M&E, partnership development and impact investing. Regional expertise will be used to service research and networking activities associated with an inclusive agribusiness innovation round table. This outcome has been slow to develop as tasks to engage external expertise have only just started to be come apparent.

FSI creates opportunities for in-country partners to participate in reflection and learning on food systems innovation EXCELLENT

A good start was made with the FSI symposium. FSI has built strong learning alliances with partner projects in East Africa, South Asia, Indonesia and Timor Leste. A major initiative in the remainder of the project will support the Grow Asia platform with an inclusive agribusiness innovation round table supported with research and analysis.

Most Significant Change (MSC)

24 | Designing and managing projects for impact at scale • Dr Andy Hall

Institutional History (IH)

25 •

• A narrative that records key points about how institutional arrangements –

new ways of working – have evolved over time and have created and

contributed to more effective ways to achieve project or programme goals.

• An IH is generated and recorded in a collaborative way by scientists, farmers

and other stakeholders.

• A key intention behind institutional histories is to introduce institutional

factors into the legitimate narrative of success and failure in research

organizations.

Designing and managing projects for impact at scale • Dr Andy Hall

Reviews and external evaluations

26 |

Relevance: The extent to which the aid activity is suited to the priorities and policies of the

target group, recipient and donor.

Effectiveness: A measure of the extent to which an aid activity attains its objectives.

Efficiency: Efficiency measures the outputs - qualitative and quantitative - in relation to the

inputs. This generally requires comparing alternative approaches to achieving the same

outputs, to see whether the most efficient process has been adopted.

Impact: The positive and negative changes produced by a development intervention, directly

or indirectly, intended or unintended. This involves the main impacts and effects resulting

from the activity on the local social, economic, environmental and other development

indicators.

Sustainability: Sustainability is concerned with measuring whether the benefits of an

activity are likely to continue after donor funding has been withdrawn.

Designing and managing projects for impact at scale • Dr Andy Hall

So in summary what do we know …

27 |

There is no silver bullet

and

design and MEL are not

separate functions … they

inform each other

Designing and managing projects for impact at scale • Dr Andy Hall

28 •

Finding your fit for purpose approach

Designing and managing projects for impact at scale • Dr Andy Hall

Resources

• Online resource portals for MEL practitioners• Better Evaluation http://betterevaluation.org

• Online community of practices• Eval Partners http://www.mymande.org/evalpartners

• S.Stone-Jovichi (2015) The journey to fit for purpose: The development of a MEL system for a complex project. Practice Note Series by Food Systems Innovation • www.foodsysteminnovation.org.au

29 | Designing and managing projects for impact at scale • Dr Andy Hall

30 •

Lunch break

Designing and managing projects for impact at scale • Dr Andy Hall

31 •

Applying MEL concepts and lessons to an example

Designing and managing projects for impact at scale • Dr Andy Hall

GOAL:

Technologies and services by

public and private sectors

help Pastoralists

have SRL

Companies crowd in and

adopt technology and

practice innovations(Private Sector)

Improved policy & public

investments targeting

pastoralists(National Public

Sector)

Incubation & refinement through experimentation

/ problem solving

Technical backstopping

Market & community intelligence

Improved development investments

targeting pastoralists

(International public sector)

SYSTEM & BEHAVIOURAL CHANGESPROVING CONCEPT

Strengthened organisations for

capacity, facilitation &

advisory services

Filling market gapCapacity building

Feedback

FeedbackFeedback

Science Capability & Networks :

- Science of innovation - International researchers- Economic and finance - ILRI researchers- Remote sensing programing - etc

- Impact Assessments (ex-anti & ex-post)- Systems diagnosis- Tracking system changes

Implementation Capability & Networks :

- Practice of implementation innovation - private sector- Community consultations - Government - Capacity development capability - NGOs- Qualitative / Ad hoc studies - Donors- Quality Assurance / trusted advisor - etc- Partnership management & brokering

Feedback

Feedback

Panel data collection & impact assessment

Feedback

Insurance Example

33 • Presentation title • Presenter name

Plans for scaling the theory of change / operations model

Thinking about the example you just saw break into 4 small groups and discuss the

following questions.

Question 1: What sort of change needs to be tracked?

Question 2: What sort of data or evidence will be needed?

Question 3: What sort of tools will they need?

Question 4: When should they do it?

Question 5: who should do it?

Each group will then report back to the larger group.

Exercise 1: What do you currently use?

34 | Designing and managing projects for impact at scale • Dr Andy Hall

35 •

Questions

Designing and managing projects for impact at scale • Dr Andy Hall

Level 1 Ecosystem Sciences Black Mountain Laboratories, Acton ACT 2602 AustraliaTEL. +61 2 6246 4771Mob +61 477 735 348Email: andrew.hall@csiro.auwww.foodsystemsinnovation.org.au

• ADD BUSINESS UNIT/FLAGSHIP NAME

Thank you