2013 Roper Making MEL Powerful

8/13/2019 2013 Roper Making MEL Powerful

1/19

Making MEL Powerful

December 2013

Laura Roper, Ph.D.

Roper LYV Consulting

([email protected])


2/19

Obstacles to MEL

Poor program planningaspirations far exceed capacities; sloppythinking (lack of theory of change; theory of action); MEL notresourced

Overwhelmed and under-resourcedlack of time biggest foe togood MEL

Reluctance to be held accountable or to hold others accountable MEL imposed from above or outside; MEL gets caught up in

politics;

MEL design inappropriate to programtoo heavy; measures thewrong things; culturally inappropriate; data not used [Somerville

housing example] No incentives to do MEL (not in work plan, not part of

performance review)


3/19


4/19

The Power of Good MEL

Closing the Loop

Building off tested instruments (e.g. survey questions into

culturally adapted focus group questions); then focus group

questions/responses informed survey design.

Surveys (frequently) and focus groups repeatedlongitudinal data

to test their hypothesis about changing needs over time. Mixed methodsFGD, survey, content analysis of text messages

Generated actionable information (e.g. around need for and

effectiveness of cholera prevention messaging; most trusted

media outlets; information needs [invalidating hypothesis aboutmen and women having different needs], etc. etc.)


5/19

The Power of Good MEL

Closing the Loop

Well presented in a visually compelling wayMost Important Issue


6/19

Closing the LoopLearn A Little, Do a Little

Diagnostic/Ne

edsAssessment

MediaProgram

Content

UserSatisfaction,

Knowledge

Adjusting Radio

Programming

AND

Aid Delivery

UserSatisfaction,

Knowledge &

Actions

Dialogic

At inception, 3 month, 6month, 1 year


7/19

A Review of the Basics

The Basic Evaluation Questions

How do we do what we do better?

What difference does it make?

Does it make more difference than other investments?


8/19

Self-Interested MEL

Proactive on what are fair terms for judging your work, dont

let the donor dictate

Use MEL reporting to tell a compelling narrative about yourwork (why it is important; how you confront challenges; that

you are strategically nimble; that you are always building up

your foundation of alliances/political capital)

Use MEL to educate your colleagues, make the more effective

stakeholders


9/19

Laying Out Program Cycle

Problem Identification

and Diagnosis

Stakeholder

Analysis

Theory of ChangeActors

Motivations

How Power is

Deployed and

other assumptions

Causal If-Then to

Get to Solution

(Theory of Action)

Strategy

GoalsObjectives

Strategic Elements

- ActivitiesImplementation

(Operational Plans/Log Frame)

-Activities completed (outputs)

MEL

(contin-

uous and

iterative)

-- First Order Outcomes

-- 2nd, 3rd, etc. Order Outcomes

-- Impacts

G

O

A

L


10/19

Addressing Gun ViolenceTheory of Change:

All the major factors and actors that affect gun-

related morbidity and mortality

Theory of ActionGun Legislation

(i.e. what your

organization can

address)

Theory of Action

Gun Violence as

Public Health

problem

Theory of Action

Drugs and Gang

Violence

Theory of ActionMedia and Video

Violence

LOG FRAME: passing gunlegislation

activities outputs outcomes impacts

Obj 1

Obj 2

Obj 3

Getting from Broad Theory of Change to

Organizational ActionA Three Step Process


11/19

Assessment Levels of MEL and

what they can tell you

Theory of Change

Actors

Motivations

How Power is Deployed

and other assumptions

Causal If-Then to

Get to Solution

Strategy

GoalsObjectives

Strategic Elements

- Activities

Implementation



-- 2ndOrder Outcomes

-- 3rdOrder Outcomes

-- Impacts

Monitoring: Did you

do what you said you

would and did you do

it well?Evaluation: Did you

get the results you

expected; why or

why not?

Whats the

problem?

Theory,

Strategy,

Execution?

Learning: What are

we going to dodifferently in this

instance and in

similar or future

cases?


12/19

Need clarity on program goals/objectives (end state)

AND point of departure (baseline)

Problem Identification

and Diagnosis

Power Analysis

Theory of Change

Actors

Motivations

How Power is

Deployed and

other assumptions

Causal If-Then to

Get to Solution

Strategy

GoalsObjectives

Strategic Elements

- ActivitiesImplementation



-- 2ndOrder Outcomes

- 3rd Order Outcomes

-- Impacts

Baseline

End State-Accountable

for delivering(aspirational vs

realistic; process vs

specific)

Intermediate

markers

Benchmarks

G

O

A

L


13/19

Finding the Breadth/Depth BalanceBroad, but Light

Routine Monitoring,

Self-Assessment and reportingacross projects/programs

Targeted

Evaluation

Problem Solving

Best Practice

Cost-

effectiveness

Capacities built

Summative Evaluation

AccountabilityBroader

Learning;

More strategic, less

expensive if monitoring anddocumentation has been

strongLearning

Create space (time andincentives)

Create

Framework/Strategy forlearning

Create Platforms

Knowledge

Management - Tools

Formative

Mid-term or

critical juncture


14/19


15/19

What type of evaluator are you?


16/19


17/19

Regardless of what type of evaluator you are,

there are some basic principles to follow (AEA

Principles for evaluators:

A. Systematic Inquiry

B. Competence

C. Integrity/Honesty

D. Respect for People

E. Responsibilities for General Public/WelfareAEA Principles for Evaluators


18/19

Ethical Principles in Data Collection in

Humanitarian Response - InterAction

Broad PrinciplesRespect, Do No Harm, Non-

Discrimination

Broad Principles Operationalized:

Risk Benefit Analysis

Informed Consent

Confidentiality

SecurityFairness

Dignity (subjects, not objects of evaluation)


19/19

Mindful Practice

Know you strengths

Know your biases

Be clear what values youre bringing to the process

Seek information from multiple sources; ideally use

mixed methodsMake an extra effort to identify and reach stakeholders

who may not typically have a voice

Listen to what is said and what is going unsaid

Dont just document; seek to understandBe familiar with resources in your field; seek out expert

advice when you need it.

Documents

2013 Roper Making MEL Powerful