Data Collection Data Collection MethodologiesMethodologies

Module 7Module 7

22

OverviewOverview

Definition of data collection Data sources Characteristics of data collection

strategy Questionnaires, focus groups,

interviews: pros and cons

33

Data CollectionData Collection

Observable, measurable units of analysis

Relevant to the purpose of the evaluation

Specific to the unit of analysis Appropriate procedures and tools Collected, recorded, analyzed, to draw conclusions

44

Data Sources Data Sources

People: program managers ,administrators, beneficiaries, donors, program staff, government officials, etc.

Documents: program strategy, annual plans, progress reports, financial reports,

Infrastructure observation: building and grounds, laboratories, program sites,

Observation of group dynamics: nature of meetings, Board meetings, etc.

55

Tools for Data CollectionTools for Data Collection

Questionnaires Interviews Focus groups Observation Performance tests

66

ExperimentationExperimentation

To ensure an optimal validity of data collected, am ideal model would the experimentation approach.

Given resources constraints in many evaluations experimentation is not always feasible nor accepted

More sophisticated techniques exist to ensure validity and reliability of data

77

Data Collection StrategyData Collection Strategy Consider the following:

• Size, geographical dispersion, literacy level of target population

• Resources available for data collection (time, people, money, technology)

• Sensitivity of issues to evaluate• Reliability and validity sought• Needs of the evaluation• Developing an adequate strategy is

key!

88

Questionnaires and SurveysQuestionnaires and Surveys

Printed or electronic list of questions Distributed to a predetermined group Completed and returned to evaluator

99

When to use a questionnaireWhen to use a questionnaire

Target population is literate, and large (over 200) or dispersed

Need for categorical data Need for quantitative data and

statistical analysis You have access to people who can

process and analyze this type of data Need to examine responses by sub-

groups

1010

6 Steps To Effective 6 Steps To Effective QuestionnairesQuestionnaires

1. Develop the purpose

2. Draft the items

3. Sequence the items

4. Design the questionnaire

5. Pilot the questionnaire

6. Develop a strategy for data collection and analysis

1111

1. Develop the Purpose1. Develop the Purpose

Specific to the information you need Add value Relate to the evaluation Refined to relevant questions only

1212

2. Draft the Items2. Draft the Items

Formulate into the following item types:

Fill in the blank• Question requiring one word answer

Multiple Choice• Question with 4-8 defined answers

Comment-On• Open ended question, written response

1313

2. Draft the Items 2. Draft the Items (cont`d)(cont`d)

List• Requesting list of 3 to 5 short responses• Can be in order of importance

Likert Scales• Statement to be ranked for agreement

Rank• List of items to be ranked by importance

1414

3. Sequence the Items3. Sequence the Items

Group into themes, or by question type

Rewrite questions, eliminate redundancies

Where necessary, use filter questions • guide respondents to different

branches Sequence with logical progression

1515

4. Design the Questionnaire4. Design the Questionnaire

Group similar themes together Organize into titled sections Format into booklet Use space effectively Decide on the need for pre-coding Include a title and introductory

explanation

1616

5. Pilot the Questionnaire5. Pilot the Questionnaire

Test questionnaire with small group Clarify wording Identify ambiguity Revise Retest if necessary

1717

6. Develop the Strategy6. Develop the Strategy

Select the sample Determine distribution plan Develop a cover letter Prepare distribution package, and send Monitor responses, follow-up Enter data Analyze data

1818

Effective QuestionnairesEffective Questionnaires

Gather valid information Provide logical and organized data Facilitate process of data entry and

analysis Simple, timely and cost efficient

1919

InterviewsInterviews

Dynamic process of gathering information

Personal interaction between two individuals

Follow a guide or protocol Require recording of responses

2020

When to use interviewsWhen to use interviews

Need to have the views of key informants

Target population is small Need for depth of information rather

than breadth You have reason to believe that people

will not respond to a questionnaire Informants are willing and available to

meet

2121

7 Steps to an Effective 7 Steps to an Effective InterviewInterview

1. Define interview purpose

2. Draft interview questions

3. Sequence the questions

4. Consider process needs

5. Prepare introduction and closing

6. Prepare to record responses

7. Pilot test questionnaire

2222

1. Define the Interview 1. Define the Interview PurposePurpose

Define objectives for the interview Link objectives to the evaluation Identify the data needed State the purpose for each part of

interview Focus on content rather than process

2323

2. Draft Interview Questions2. Draft Interview Questions

Carefully drafted and worded Minimize ambiguity Combine open-ended with close-ended

2424

3. Sequence the Questions3. Sequence the Questions

Vary question types• Open-ended (What do you think

about…)• Close-ended (Did the program begin

in 1998?) Organize questions into themes or

sections

2525

4. Consider Process Needs4. Consider Process Needs

To manage the interview process, use:• transitions to move to each theme• reminders to paraphrase or

summarize• lists of standard probes

2626

5. Prepare Introduction and 5. Prepare Introduction and ClosingClosing

Introduction to set tone and rapport• state purpose of interview• explain who the interviewer is• confirm confidentiality and use of data

Closing to signify the end of interview• reinforce usefulness of interview• thank respondent • confirm arrangements for follow-up

2727

6. Prepare to Record 6. Prepare to Record ResponsesResponses

For key informant interviews:• tape record (with permission)• take brief notes on notebook• detail notes immediately after

interview For normative interviews:

• check boxes• space for efficient data recording

2828

7. Pilot-test Interview7. Pilot-test Interview

Validate the interview:• protocol• content• flow• arrangement to record responses

If more than one interviewer, validate:• ability to use protocol

2929

Sample Protocol - Program Sample Protocol - Program Evaluation InterviewEvaluation Interview

Introduction: “We have been asked by.. To conduct a .. We would like to discuss the ...”

Theme 1 - Program relevance Theme 2 - Program success Theme 3 - Program cost effectiveness Conclusion: “In conclusion, what are

the main strengths and weaknesses of this program?”

3030

Effective Interviews Are:Effective Interviews Are:

Planned sufficiently• clear objectives

Focused on the process• control over context and process• two-way communication• awareness of non-verbal

communication Conducive to establishing trust Dependant on a cooperative interviewee

3131

Effective Interviewing Skills Effective Interviewing Skills Include:Include:

Active Listening• relaxed, eye contact, verbal feedback

Openness and Empathy• accepting what is said without

judgement Paraphrasing/Summarizing

• rewording to clarify response Controlling the Process

• setting context, shaping responses

3232

Focus GroupsFocus Groups

Group discussions of a predetermined issue

Members share common characteristics

Moderated by facilitator Responses are recorded

3333

When to Conduct Focus GroupsWhen to Conduct Focus Groups

You need rich description to understand client needs

Group synergy is necessary to uncover underlying feelings

You have access to a skilled facilitator Informants are willing to speak in

groups

3434

5 Steps to Effective Focus 5 Steps to Effective Focus GroupsGroups

1. Define the purpose

2. Select an appropriate sample

3. Determine date and location

4. Develop Focus Group Guide

5. Plan to record responses

3535

1. Define the Purpose1. Define the Purpose

Based on the objectives of the evaluation

Identify information needed and why Develop the purpose of the focus

group

3636

2. Select an Appropriate 2. Select an Appropriate SampleSample

Individuals who are:• informed about the topic of discussion• in the best position to share

information• represent all target populations

Pre-screen participants to ensure they meet the required characteristics

Select 6 - 12 participants

3737

3. Determine Date and 3. Determine Date and LocationLocation

Make clear arrangements regarding• purpose• starting time• how long it will last• neutral convenient location• participants feeling at ease

3838

4. Develop Focus Group Guide4. Develop Focus Group Guide

Construct effective questions • content related to evaluation purpose • open-ended to facilitate discussion• limit to five or six

Sequence questions • ensure natural flow• facilitate transition from one to the

next• lead logically through variety of topics

3939

5. Plan to Record Responses5. Plan to Record Responses

Taping gives full record of discussion• inform and attain consent in advance

Detailed notes are indispensable• taken by assistant to moderator• organize data in predefined categories• document relevant observations• avoid cues about value of responses• underline or highlight reference points

4040

Effective Moderation IncludesEffective Moderation Includes

Active listening Facilitation of discussion Awareness of group dynamics Suspension of personal bias Alertness to time schedule Probing for elaboration Affirmation of responses Respect for different perspectives

4141

Observation: when to useObservation: when to use

When the setting is important for the evaluation

When the interaction amongst people is a key factor of the evaluation

Before developing a questionnaire, in order to get a feeling for some of the issues

To understand cross-cultural issues

4242

Steps in ObservationSteps in Observation

Define the population or setting to observe

Select representative places to visit/observe

Develop an observation schedule with the dimensions to observe

If there are many observers, train observers for consistency

4343

During ObservationDuring Observation

Prepare for an entry phase so the observer does not impact the setting too much

Record data according to the observation dimensions selected: interaction, environment, light, smell, infrastructure, etc.

Record qualitative (how good, how bad) as well as quantitative (how many) data

Exit from the setting and record data immediately

4444

Performance Tests: When to Performance Tests: When to useuse

When there is a need to define aptitudes or abilities

When the study requires to select good performers (as opposed to people to train for good performance)

When there is a need to diagnose performance problems

4545

Categories of performance Categories of performance teststests

Aptitude tests Attitude scales Psychological tests Performance tests

4646

Choosing a performance testChoosing a performance test

First criteria is to match the test to the purpose

Second is to match the resources (time, money, technology), context and culture to the purpose

If the purpose is to screen out the best individuals, aptitude tests make sense

If the purpose is to assess progress, pre- and post-activity tests make sense

4747

Scoring performance testsScoring performance tests

Criterion referenced (Can a person perform according to a given standard?)

Normative (Who is the best in a group?)

4848

Summary of Methodologies Summary of Methodologies

Type Strength WeaknessQuestionnaire Efficient

Large #respondents

Extensive planning Low response

Interview In-depth info High response

Extensive planning Time consuming Analysis difficult

Focus Group Group synergy Diverse

perspectives

Extensive planning Analysis difficult Logistics