Embed Size (px)
Citation preview
Global Youth Economic Opportunities Conference
Making Cents International
Washington, DC
Wednesday, September 12, 2012
Comparing Evaluation Methods for
Financial Education Interventions in Africa
Rationale for this Session
Citi Foundation’s perspective & results-oriented framework
Overview of evaluation methodologies
The Financial Education Fund (FEF)
Selecting indicators
Data collection and tools
Breakout group discussions and feedback
Overview: Philanthropy & Evaluation
Trends
Power of Data
– Assess program or intervention effectiveness
– Measure impact upon participants
– Build knowledge in the field
Translation/Making the Case
– Benefits?
– Imperative?
– Helpful details for funders?
5
Citi Foundation’s Impact Measurement Approach
Establish investment guidelines and clearly define measurable outcomes with practical benchmarks
Integrate a social ROI focus upfront and during the due diligence process
Shifted our investment approach and communications to talk about what has been accomplished
through our grant support vs. what activities our funds supported
6
Example of Prior Statements
About 15,000 youth will attend and complete the financial education workshop
series by the end of the grant period.
Example of Current Statements
Out of 15,000 young people, 12,000 (or
80%) will reduce their debt and/or increase
their monthly savings by 20% for at least 9
consecutive months to achieve their defined
financial goals.
What will be the measurable outcome from the proposed grant?
How likely is it that the outcome can be achieved?
How can we best leverage grant funds and other resources to maximize results and impact?
Activities & Outputs Outcomes & Results
6
Express interest in
postsecondary education or
entrepreneurship
Enroll in program and
identify necessary
skills
Acquire necessary skills
Access financial or employment
resources needed to succeed
Use new skills and resources to move
towards enrollment in postsecondary
education, employment, or start a
business
Demonstrate new skills
obtained through training
and support
Microfinance (US & Int’l) the supply and use of asset building
financial products
Enterprise Development
(US & Int’l)
micro or small enterprises and job
creation
College Success (US) college graduates/completion
Neighborhood
Revitalization (US)
small businesses, affordable
housing units and community
facilities
Financial Capability and
Asset Building (US & Int’l)
positive financial behaviors and
financial assets
Youth Education &
Livelihoods (Int’l)
youth (ages 13-25) that are employed
in a living wage job, start their own
business, or obtain postsecondary
education
We seek increases in… In each of focus area…
Citi’s Results-Oriented Framework: Defining and Tracking Results in Every Grant
Youth Education & Livelihoods
Results Path
We track results in terms of quantity of pre-defined desired outcomes achieved
and the quality of the expected outcomes based on our “results path”
Changing
attitudes through
exposure and
training
Modeling skills
and accessing
new opportunity
• Project metrics, indicators, and assessment described
• Verified during grant investigation and review
• Data coded in grants management system
• Data collected in applications from grantees
• Verified against outcomes outlined in reporting
Impact Evaluation
Rigour of Impact Evaluation Methodologies
Cross sectional study
Time Series
Cohort Study
Nonrandomised w/ historical
controls
Non randomised with contemporaneous
controls RCT
Scientific Rigour
High Level Pointers for Rigour in Impact Studies
• Randomised studies are stronger at proving causation than non-
randomised studies.
• Prospective studies are stronger than retrospective studies (baseline and
endline).
• Large studies are stronger than small studies (sample sizes).
• Contemporaneous controls are stronger than historical ones.
• External controls (multiple-group designs) are stronger than self-controls
(one-group designs).
High Level Pointers for Rigour in Impact Studies
Process Evaluation
Process Evaluation
• Carried out alongside an impact evaluation
• Gather information to determine how, why, who with regards to an
intervention
• Independently to assess the implementation procedures
• Provide contextual data against which impact evaluation results can be
judged
• Understand the programme’s resources and how they are used
• Gather understanding of the participants’ experiences of the
programme/intervention
Why Undertake a Process Evaluation?
• Two broad groups of methods
– Monitoring and operational research
– Social research
• Qualitative research
• Depth interviews (with providers and users)
• Focus/discussion groups
• Case studies
• Observational studies, diaries
Process Evaluation Methodology
Financial Education Fund (FEF)
Objectives of the FEF & Implementation
• UK DFID funded GBP 4 million for the establishment of the FEF including
implementation and management
• Grant matching component mobilised a further GBP 1 million
• 15 grantees in 8 African countries were funded across a range of target
audiences, delivery mechanisms, and financial education messaging
• Unique emphasis on impact evaluation as a core operational component of
each project
Financial Education Fund
Evaluation parameter “Gold” “Silver” “Bronze”
Survey administered before and after the
intervention (Y/N) Y Y Y
Control group, comparison group, national
baseline survey data only
Control
group
Comparison
group
Treatment
Group Only
Conducted by grantee or professional third
party (Grantee/ Prof 3rd)
Prof 3rd
Prof 3rd Grantee
Additional follow-up survey after defined
period (e.g. 3 or 6 months) (Y/N) Y N N
Quantitative and/or qualitative method used Both Both Quantitative
only
Reasons for financial behavior beyond FE
intervention probed further (Yes/No) Y Y Y
Verification of usage of financial services
verified with financial service provider
Y, where
relevant
Y, where
relevant N
Evaluation Parameters
Randomised Control Trial
Nakekeli Imali – Financial Education for Mineworkers
– Population sample of 11, 719 (5,013 trained), formal wage
earners with access to wide array of in/formal financial
products
– Only 61.6% of the mineworkers assigned to treatment
actually attended the financial literacy workshops – challenge
in terms of achieving the desired reach * Trade off for
donors/funders
– This type of evaluation only
measures IMPACT
– Approximately 27% of budget
Cross-sectional Time-series Evaluation
Camfed – FE For Young Women in Rural Zambia
– 10,701 young women in rural areas of 8 districts
– Sampled two ‘comparable’ villages and two intervention villages
– Selected non-randomised because of the ethics of randomly assigning
beneficiaries within the same village
– While the village demographics appeared the same – baseline survey
identified that there were differing characteristics
– Intended to do a longitudinal
study but the researchers
couldn’t trace the comparison
group
– Incorporated both impact and
process evaluation
– Approximately 40% of budget
(40% of which was surveying)
Time-series Evaluation with Entire Population
• Imali Matters – One on One Advisory Clinic
– Due to the nature of the walk-in clinic, was not possible to identify a
control or comparison group (voluntary)
– Entire population was evaluated (just under 400)
– Evaluation entailed qualitative research, survey and credit check
• Survey was divided into 2 sections:
– Pre-test: measures client’s financial knowledge
– Health check: looks at the client’s financial situation
• ICT Credit Check (“black listing”)
• Focus Group Discussions
– Process evaluation was critical to understand the relevance, use and
appropriateness of the one-on-one service
– Only 9% of budget (questionnaire built in to service)
Determining What to Measure
Identifying and Defining Objectives for Evaluation (1)
• Evaluations are carried out against
objectives, so…
– Objectives need to be clear
– Avoid unrealistic objectives
– Avoid ones that are vague or
cannot be measured
• Objectives need to be set for both
the outputs and outcomes
Identifying and Defining Objectives for Evaluation (2)
• To promote more responsible money management in
the community
• To reduce the number of participants who regularly rely
on borrowing to make ends meet
Action
Fin
ancia
l
Capabili
ty
Time/consumer behaviour
Triggering
Equipping
Change of mind
Knowledge:
I know about the
services, its benefits
and where to access
it
Skills:
I know how to use it
and have access to
the channel/
technology
Attitudes:
I trust the provider/
channel and am
committed to using
the service
Behaviour:
I am using the
service
Measuring Financial Capability
• The choice of indicators will depend on the objectives
• Indicators need to be:
– Relevant to the programmes
– Feasible to collect
– Easy to interpret
– Able to track changes over time
• Cost of data collection increases as indicators shift from input through output and outcome and impact
• Projects often set TOO MANY indicators and gather data that is never
used - FEF Grantees were asked for 8 – 10 outcome/impact based
indicators
Selecting Indicators (1)
Key indicators
Thematic
Areas
Financial
Capability
Aspect
Area of Measurement
Knowledge, Skills, Attitude, Behaviour
Budgeting
Knowledge Respondents who know the key
components of a household budget
Skills Respondents developing a household
budgeting plan (Income vs. Expenditures)
Behaviour Respondents that follow a budget on a
regular basis
Savings
Knowledge Respondents who know key aspects of
savings
Attitude Respondents that believe they can save
Behaviour Respondents with increased net savings
Selecting Indicators (2)
Data Collection & Tools
Key indicators Evaluation Stages
Use
Data Analysis
Data Capturing
Data Collection and Editing
Tool Design & Research Planning
Establish Evaluation Methodology and Indicators
Everything needs to
be planned!
• Data collection is extracting
source information using tools
such as
– Questionnaires
– Interview Guides
– Observation Forms
– Checklists/rating scales
• Well designed tools are easy to
use and thus produce accurate
results
Key indicators Selecting Appropriate Tools (1)
• Relevant and culturally sensitive to the
population being evaluated?
• How different groups prefer to share
information (e.g., orally, in writing, one-
on-one, in groups, through the arts)?
• Consideration for potential language
barriers that may inhibit some people
from understanding the evaluation
questions
• Are multiple methods being used, so
that information can be analysed in a
variety of ways?
Selecting Appropriate Tools (2)
Group Breakout Session
• Option One: Village Savings and Loan Group for farmers
• Option Two: Community radio programme and newspaper inserts
• Option Three: Afterschool programme for 13 & 14 year olds
• Option Four: Example from one of the group members
Group Examples
• What are the objectives of the evaluation given the particular intervention?
• What are some of the indicators to measure the achievements?
• Which evaluation activities are most appropriate for the intervention?
• Identification of control/comparison as appropriate and sample.
• Which data collection methodologies will provide the information required?
• Timeframe
Discussion Points
• Quality evaluation is a combination of:
– Proper planning
– Clearly defined indicators
– Representativeness of the sample
– Accuracy and precision of measurements
– Data management and processing
Key Points on Quality Evaluation
Contact: Alyna Wyatt
Genesis Analytics
Jasmine Thomas
Citi Foundation
