MEPA Qualitative Tools "Not everything that counts can be counted."

MEPA Qualitative Tools

"Not everything that counts can be counted."

SESSION OBJECTIVES• At the end of the session, the participants

are expected to:a.Discuss the basics and nuances of using

qualitative tools in MEPA;b.Explain how to prepare and use appropriately

tools such as: Interview (KII & IDI), Most Significance Change (MSC) Story Technique MSC, Participatory Rapid Appraisal (PRA), and Focus Group Discussion (FGD);

c. Develop deeper understanding on the importance of .both quantitative and qualitative tools in preparing Completed Staff Work (CSW) for MEPA in order to provide story to quantitative data.

ACTIVITY: Self-Assessment

No.

Qualitative ToolsI have adequate knowledge and have already used:

Pre-test Post-test

1 Key Informant Interview (KII) 1 2 3 1 2 3

2 In-depth Interview (IDI)

3 Most Significance Change (MSC)

4 Focus Group Discussion (FGD)

5 Participatory Rapid Appraisal (PRA)

Total Score

Level of Competence

Directions: Assess how knowledgeable and competent you are in using qualitative tools in the preparation of CSW for MEPA, using the following rating scale:

1 – Developing 3 – Advanced2 - Maturing

How is your pre-test result?

Directions: 1.Sum-up the total score and divide it by 5.2. Determine your level of competence based on the table below:

Range Descriptive Rating

Interpretation

1.0-1.49 Developing You are still in the process of acquiring the skill; hence, you need Technical Assistance.

1.50-2.49 Maturing You have acquired the skill to a larger extent, but you need to enhance it further.

2.50-3.0 Advanced Congratulations! You have fully acquired the skill to a largest extent, you are very much ready to prepare CSW for MEPA.

ANALYSIS

1. How did you find the activity?2. What is your level of competence in

terms of using qualitative tools?3. What facilitating/hindering factors

contributed for having such level?4. Which specific tool do you need

more TA?5. Why do you need to have high level

of competence in all these qualitative tools?

Qualitative Monitoring

• Data-gathering techniques that

are focused on the significance

of observations made in a study

rather than the raw numbers

themselves.

Quantitative: numbers breadth generalizabilityQualitative: words depth specific

Remember, "Not everything that counts can be counted."

4 (Quantity) Happy (Quality) Kids

Quantitative methods vs Qualitative methods

Quantitative Qualitative

Surveys Focus groups

Tests Unstructured interviews and observations

Questionnaires Rapid Appraisal

Mixing Quantitative Methods & Qualitative Methods

•Both quantitative and qualitative methods have their strengths and weaknesses. •What qualitative methods lose on reliability they gain in terms of validity. They provide a more in depth and rich description of change and learning. • Quantitative methods are reliable and can be projected to a large population but are weak in terms of describing actual experience.• Combining quantitative and qualitative research provides a more holistic picture of outcomes and impacts achieved.


Quantitative Monitoring Mid Term Results

Region A

Physical Outputs Accomplishment 30%

Financial Funds Utilization: 18%

Funds Liquidated: 5%

Qualitative Monitoring Mid Term ResultsRegion ADuring the Orientation on DO 62 I was full of questions. Why do we have to give so much emphasis on IP education when they do not show interest in going to school. Going to the immersion in Mindoro was an eye opener, here the IPs were both teacher and student. They learned mainstream concepts together with their own indigenous knowledge. The river, land and air were all part of the learning environment. After the immersion, the Region decided to conduct a second round of orientations on DO 62. With the benefit of our experience we understood the principles behind the order. We also decided to repeat the proposal preparation training to emphasize community participation. As a result, although late the second round of proposals submissions were much better than the first.


Use QUANTITATIVE Research To:

•find whether there is consensus on a particular issue

•projects results to a larger population

•test specific hypotheses and examine specific relationships

Use QUALITATIVE Research To:• look for a range of ideas and feelings about something•understand different perspectives between groups and categories of people•uncover underlying motivations and factors that influence decision making and opinions

Qualitative Data Gathering

• Qualitative research takes place in the real world, as opposed to the laboratory, and deals with how people give meaning to their own experience.

• These often involve face-to-face interactions between researcher and participant

• The researchers need to be flexible and sensitive to the needs of the social context within which the data is obtained.

Qualitative Data Analysis

• Then it is followed by an attempt to interpret the behaviour and the meanings that people have given to their experience.

• The data is then analyzed and interpreted. To look for themes rather than than trying to confirm a hypothesis.

• The objective of qualitative research is to describe and possibly explain events and experiences.

Participatory Rapid Appraisal (PRA)

Source: Cana & Tucit (2012). Participatory Rapid Appraisal. Retrieved on Dec. 7, 2014 from http://www.slideshare.net/ronelcana/participatory-rapid-appraisal

http://www.slideshare.net/ronelcana/participatory-rapid-appraisal




















THE KEYS TO KEY INFORMANT INTERVIEW (KII)

• Who is a Key Informant? - Someone who can ‘unlock’ key

information for you. • What is KII?- Discussion between an individual

respondent and an interviewer which can be done via telephone or face-to-face ranging from informal to highly structured chats.

Source: Performance M&E, USAID Center for Dev’t Info. And Eval; http://pdf.dec.org/pdf _docs/PNABS541.PDF


• Purpose of KII?- Collect qualitative, in-depth information

from a wide range of people about a pressing issue to understand beliefs, motivations and sensitive topics

• Types of KII:- Informal, conversational interview- General, guided interview- Standardized, open-ended interview- Closed, fixed-response interview



• Key to Success?- Diversity is important.- If saturation is reached, it’s time to stop.- Interviewer skills- Location: private, no distractions and non-

threatening• Tips for Writing Questions:- Open-ended, neutral, one at a time, clear,

use probing questions (can you give me examples, what changes have you noticed)



• Analyzing the Data:- Create interview summary sheet- Descriptive codes: themes, concepts,

questions and/or ideas

• KII is perfect when:- Descriptive information is needed- Subject matter is complex.- Respondents are of high status.- Subject matter is highly sensitive.


IN-DEPTH INTERVIEW (IDI)

• What is IDI?• Questions are based upon the

responses collected• Objectives of IDI:- Discover the interviewee’s own

framework of meanings- Obtain rich, contextualized, in-depth

information

Source: Crystal Graphics, Inc. (2014) Retrieved on Dec. 7, 2014 from http://www.powershow.com/view3/50df42-MTE1M/Observations_and_in-

depth_interviews_powerpoint_ppt_presentation


• IDI Guide Questionnaire- List of core questions: Topics- Variations: Order, terminology (adapt to

the terminology of the respondent)- Additional questions may be introduced

• Types of Questions for IDI:- Behavior, opinion/belief, feelings,

knowledge, sensory, background

Source: Crystal Graphics, Inc. (2014) Retrieved on Dec. 7, 2014 from http://www.powershow.com/view3/50df42-MTE1M/Observations_and_in-depth_interviews_powerpoint_ppt_presentation


• Qualities of Questions in IDI- Open-ended, neutral, sensitive, clear to the

interviewee

• Practical Aspects of IDI:- Number of interviewees, status of informant,

recording process, dynamics of interview, leading role of informant, questions, sequential interviews, probing, relationship

Source: Crystal Graphics, Inc. (2014) Retrieved on Dec. 7, 2014 from http://www.powershow.com/view3/50df42-MTE1M/Observations_and_in-depth_interviews_powerpoint_ppt_presentation

FOCUS GROUP DISCUSSION

• What is FGD?- Involves a whole group answering

questions together (Bailey, 1994: 192).- ‘Guided group discussions’ designed to

provide information on a certain topic from a certain population (Ward et al., 1991: 267).

- Are groups of unrelated individuals that are formed by a researcher and then led in group discussion of a topic (Schutt, 1999: 308).


• When to use FGD?- To familiarize a research topic- To polish research objectives- To deepen understanding of survey

data- To validate survey data- When interested in capturing diverse

and ‘detailed’ views and/or opinions


• Characteristics of FGD- Most focus groups involve 7 to 10

people- May be semi-structured or

unstructured- Discussion is guided/facilitated by an

FGD facilitator/leader- FGD proceedings are

recorded/documented by an FGD documenter


• Pre-FGD Activities- Defining the objectives- Construction of the FGD guide especially for

semi-structured FGD- Familiarizing the FGD guide/questions - Preparation of recording devices/materials- Identifying and inviting FGD participants- Confirming the attendance of FGD

participants


• Stages in the conduct of FGD

I. Orientation of the FGD participants—setting discussion rules

II.Getting to know (especially if participants are unrelated)

III.Discussion properIV.Synthesizing and closing the discussionV.Distribution of tokens/snacks


• Post-FGD Activities- Checking the

documentation/recorded data- Transcribing the recorded

discussion- Encoding the finalized

transcriptions- Data Analysis

MOST SIGNIFICANCE CHANGE (MSC)

• Most Significant Change (MSC) involves the collection and systematic participatory interpretation of stories of significant change.

• Unlike conventional approaches to monitoring, MSC does not employ quantitative indicators, but is a qualitative approach.

MOST SIGNIFICANCE CHANGE (MSC)

• It is a form of participatory monitoring and evaluation.

• It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analyzing the data.

HISTORY OF MSC

• The MSC approach was originally developed by Rick Davies through his work with a participatory rural development project in Bangladesh in 1994.

• It has since been adapted and widely promoted in many developing countries.

HISTORY OF MSC

• It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program.

• It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.

MSC AS M&E TOOL

• It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program.

• It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.

MSC PROCESS

• Essentially, the process involves:

• the collection of significant change (SC) stories emanating from the field level, and

• the systematic selection of the most significant of these stories by panels of designated stakeholders or staff.

MSC QUESTION

• SC stories are collected from those most directly involved, such as participants and field staff. The stories are collected by asking a simple question such as:

‘During the last quarter, in your opinion, what was the most significant change that took place for participants in the program?’

MSC PROCESS• The designated staff and stakeholders are

initially involved by ‘searching’ for project benefits.

• Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes.

• When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact.

WHY STORIES?

• Storytelling is the vivid description of ideals, beliefs, personal experiences, and life-lessons through stories or narratives that evoke powerful emotions and insights.

It enables expression of emotional aspects, factual content, and tacit knowledge.

It increases the potential of meaningful knowledge-sharing.

It augments the likelihood that learning will take place and be passed on.

Stories are the creative conversion of life itself into a more powerful, clearer, more meaningful experience. They are the currency of human contact. — Robert McKee

Everyone is necessarily

the hero of his own life story.

JOHN BARTH

If you don’t know the trees you may be lost in the forest, but if you don’t

know the stories you may be lost in life.

— Siberian Elder

There is no agony like

bearing an untold story inside of you.

- Maya Angelou

STEPS IN IMPLEMENTING MSC TECHNIQUE

1. Raise interest in MSC2. Defining the domains of change 3. Determine the process and

responsibilities4. Collect significant change stories5. Select the most significant of the stories6. Verify stories7. Analyze the top stories for program

learning

Sample of a Promising Story

I had my first PRIME experience when I attended the immersion at the IP Center on February 10, 2012.

In my encounter with the IP students, teachers, folks and the community, I had developed in myself pride of one’s culture, love and care of one’s identity with rich tradition and diverse cultural heritage. It was an eye-opener to me that no one could certainly preserve one’s culture other than ourselves.

Sample MSC Questionnaire

Sample 1: MSC Form for Teachers

Sample 2: MSC Form on Drop-outs for School Head

Sample 3: MSC Template on New Systems, Technology

MSC SCATTERGRAMDimensions Stages of M & E

InstitutionalizationPolicy Formulation * *Managing & Monitoring

* * * * * * * * *

Initial Gain: Improving Teacher’s Competence

* * * * * * * *

Technical Assistance * * * * * * * *Innovation * * * * * * *Implementation of the Program

* * * * * * * * * * * *

New Process/ System/ Technology Experience or Encountered

* * * * * * * * * *

Awareness * * * * * * * * * ** *

ProgressM & E

Initial Gain Evaluation

Outcome Evaluation

Impact Evaluatio

nNote: * This Indicator shows that such program component is embedded explicitly / implicitly in the MSC story of the respondents.

APPLICATION: MSC

1. Tell us about your involvement in program/projects addressing dropout in your school/district/division.

2. What is the most significant change/experience of the school/community as a result of the dropout reduction program/project?

3. Can you cite observable changes among the children who have availed of the dropout reduction programs/schemes? Relate a personal account of a particular student/s.

4. How were the parents & community leaders involved in bringing about these changes?

5. Why is this story significant to you? What lessons can we learned from this experience and what could be improved?

Answer the following questions based on your experience:

Documents

MEPA Qualitative Tools "Not everything that counts can be counted."