Upload
joayal
View
221
Download
0
Embed Size (px)
Citation preview
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 1/36
Developing a High
Quality Baseline
Salimah Samji & Mona Sur World Bank, New Delhi
June 21, 2006
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 2/36
Overview
What is a baseline?
Why you should care
The phases of conducting a baseline
The errors to avoid in each phase
How to manage the common errors
Elements of a baseline survey (TOR)
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 3/36
What is a baseline?
Fixing the time at the base – a benchmarkfrom which you measure progress
Snapshot of indicators at a time
Instrument used to:
Test hypotheses of project (assess results)
Planning (refine targeting, indicators tomonitor)
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 4/36
Why you should care …
To identify whether there were any benefits for the investments made Were objectives met?
What factors explain the result?
How can the program be improved?
Compare alternative models to get the biggest
bang for your buck To inform next generation projects
Evidence-based policy making – demonstrationeffect for government
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 5/36
The phases of conducting a baseline
Design
Implementation – actual survey
Data entry and analysis Report writing
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 6/36
Phase 1: Design Phase
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 7/36
P1: Check-list
Clear objectives (what is the problem?)
Clear idea of how you will achieve the
objectives (causal chain or hypotheses)
Clear and measurable indicators
Clear (precise and unambiguous)
Relevant (to objectives)
Monitorable
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 8/36
Example of a causal chain
Impact
Outcomes
Outputs
Activities
Inputs Facilitators, Revolving fund (credit)
Forming, federating and organizing SHGs
Number of SHGs, decreased input prices
Increased use of credit for income generation, Loan
repayment rates
Higher income levels
Example: APDPIP
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 9/36
P1: Check-list (cont‘d)
Design survey instrument – keep it simple andrelated to the objectives and hypotheses youwant to test
Link surveys to GIS – use consistent units Select controls/counterfactuals to attribute
change (causality)
Timing of baseline
Before project (what if project never materializes)? 2 years into the project (intervention has begun)?
Other factors (seasonality)
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 10/36
P1: Check-list (cont‘d)
Sampling strategy
Random allows inferences
about a population Random
Stratified random (include groups
which could be excluded)
Non-random – use a group
smaller than the population Introduces selection bias. Note:
every stratification introduces a
level of bias.
Population
Size
Sample
Size
10 10
50 44
100 80
500 217
1,000 278
3,000 341
50,000 381
1,00,000 385
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 11/36
P1: Check-list (cont‘d)
Over sample for attrition
Design the database system for data entry
Translate the questionnaire Back translate to verify
Provide adequate training on
The objectives and importance of the study How the sections are linked and what the questions
mean
Attend the training if possible (stay involved)
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 12/36
P1: Check-list (cont‘d)
Test the questionnaire
Ask yourself, can you answer these
questions? Are they relevant to the outcomes?
Will they be understood?
Field test: To test both how the surveyor
administers the instrument AND how therespondent understands the question.
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 13/36
P1: Why it is important to field test (i)
When asking a question about the level of awareness, the surveyor used a word that couldmean awareness or knowledge – therespondent understood it to mean education. :
The question was: ―Ram/ Gita knows about everything that happens(vikas) in the village. For instance, they know [the name of the
sarpanch, when and where the Gram Panchayat meets, nature and
type of development work in village, etc.]‖
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 14/36
P1: Why it is important to field test (ii)
CASE STUDY: Domestic Violence Study in India
―In studying domestic violence, a question in the survey instrument asked if
female respondents had ever beenbeaten
by their husbands in thecourse of their marriage. Only 22 per cent of the women respondedpositively to this question – a domestic violence rate much lower thanstudies in Britain and the US had shown. In probing the issue with in-depth interviews we discovered that the women had interpreted the word‗beating’ to mean extremely severe beating – when they had lostconsciousness or were bleeding profusely and needed to be taken to the
hospital. Hair pulling, ear twisting, etc, which were thought to be moreeveryday occurrences, did not qualify as beating. Reponses to a broader version of the abuse question, comparable to the questions asked in theUS and UK surveys, elicited a 70 per cent positive response.‖
Source: Vijayendra Rao (1998) – ―Wife-Abuse, Its Causes and Its Impact
on Intra-Household Resource Allocation in Rural Karnataka‖
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 15/36
Phase 2: Implementation
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 16/36
What would you do if you …
Were a city person who didn‘t speak the local
language very well
Had to travel to several villages and spendhours asking people questions that have no
relevance to you.
Were paid a small sum per questionnaire
Not monitored by supervisors
This is not your full time job
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 17/36
The answer is simple
Sit at home or in a bar and fillout the questionnaires!!
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 18/36
P2: Providing incentives and motivation
Sub-contracting surveyors from the statewho speak the language
Include women surveyors
Include a supervisor who conducts datascrutiny
If possible pay reasonable wages
Randomly verify questionnaires to reducethe likelihood of false responses (informthem beforehand - during the training)
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 19/36
Phase 3: Data Entry and Analysis
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 20/36
P3: Check-list
Data Entry
Make the data entry system as fool proof as
possible - has unique identifiers to link both
household, village and GIS data
Ensure database allows for merging of data
Do not change/erase data on questionnaires
Raw data should always be input as is, changes
can then be made in the database software
(programatically) with documentation
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 21/36
P3: Check-list (cont‘d)
Often data entry is contracted out. Name variables corresponding to the question and section in the
questionnaire – include a dictionary
Code descriptive answers (to facilitate analysis)
All fields should be filled (NA or NR)
Units should be uniform by district
Totals calculated by formula not from summary column
Consistency checks – check for missing entries, wrong
entries, sample statistics, patterns (queries should beinbuilt)
Validity checks – similar questions in different places onthe questionnaire (RCH example)
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 22/36
P3: Check-list (cont‘d)
Data analysis
Common mistakes in interpreting data
No analysis! No correlations, crosstabs, statistical significance levels or
regressions
Over generalizing the results
Mis-reporting statistics Using % when the numbers are small
Attributing causality when it is not demonstrated
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 23/36
Phase 4: Report Writing
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 24/36
P4: What the report should be …
Simple, Clear and Relevant
State limitations (attribution, causality)
Major findings should be upfront
Focus on quality rather than quantity Technical details in an appendix
Should always include the questionnaire in the appendix
ask for electronic copy of data Request copies of filled out surveys
Essential if you change consultants at midtermor want to conduct internal analysis to comparemodes of delivery (data lost example).
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 25/36
How to manage the common errors
Phase 1: Design Clear objectives and hypotheses – know what you
want to test
Identify a person in your unit who will manage this
process Write a good TOR, remember the baseline determines
the quality of your panel
You can add questions as project evolves but cannotchange questionnaire – threat to internal validity
Identify consultants Procurement – focus on quality not the cheapest bid ―if you
throw peanuts you‘ll attract monkeys‖
Ideally you should have a black-list of organizations
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 26/36
How to manage the common errors
Phase 2: Implementation
Organize an impact evaluation workshop if
necessary
Randomly verify questionnaires to reduce the
likelihood of false responses (no filling it in a
bar)
Pay reasonable wages to surveyors (if possible)
Show the client and firm that you care
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 27/36
How to manage the common errors
Phase 3: Data entry and analysis
Double-data entry (2 separate organizations and
verify. Payment based on quality of data entry)
Select 15 questionnaires at random and check dataentry – person in your unit managing
Check data quality (consistency and validity checks)
Hold an IE workshop to build data analysis capacity (if
necessary)
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 28/36
How to manage the common errors
Phase 4: Report writing
Agree on an outline beforehand
Dedicate a chapter on indicators you aretracking
Focus on quality not quantity
Think ―Big Picture‖
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 29/36
Elements of a Baseline Survey
Terms of References 1. Background: Project objectives and components
2. Survey design: Consult a sampling expert!!!
3. Survey instruments
4. Guidance on survey implementation
5. Data processing and analysis
6. Staffing
7. Duration and time schedule
8. Submission of reports and datasets
9. Support to the firm
10. Budget & Payment Schedule
11. Annexes: Draft questionnaires, Results Framework
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 30/36
Baseline Survey Design:
Typical Tasks for Consultants Recommend the methodology for sampling
Calculate the optimal sample size
Develop the sample frame and select thesample
The final sample and details of the statisticalmethodology used to select the sample need to
be cleared by the project Construct the sample weights and provide
documentation on the methodology used toconstruct the weights
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 31/36
Survey Instruments:
Questionnaires Design or refinement and adaptation of the data
collection instruments
Specify levels of data collection
Length of questionnaires Prepare all support documentation including
coding guides, interviewer and supervisor manuals and the data entry manual
Translation and back-translation Skip patterns, coding open ended questions
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 32/36
Guidance on Survey
Implementation Implementation plan
Selection and training of field workers: specifyminimum duration of training
Pilot testing should be explicitly specified in ToR
Responsibility for all field operations, includinglogistical arrangements for data collection and
obtaining household consent lies withConsultants.
Ask for field-work progress reports (bi-weekly/monthly)
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 33/36
Staffing
Sampling expert/statistician
Technical specialists as relevant
Economist Sociologist.
Core survey staff: the survey manager, the
field manager, the data manager Enumerators, supervisors and data entry
staff
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 34/36
Baseline Report & Data
Explicitly request final electronic
datasets—with complete documentation.
Agree on outline of baseline report up-front.
7/30/2019 Developing High Quality Baseline
http://slidepdf.com/reader/full/developing-high-quality-baseline 35/36
Managing a Baseline Survey
Consult the experts—survey specialist and
sampling specialist and develop the ToR in
consultation. Selection committee should include a survey
expert and social scientists in addition to
technical experts.
You can never over-supervise!!! Hire third-partysupervision consultant if needed.
Question the data and the findings.