Information note:

“RBM KS in IFAD Rural Development Context” Bangkok, 3-7 December

Cecilia Ruberto,

M&E consultant, IFAD Hq Policy and Technical

Advisory division, Rome, Italy. Mail:

c.ruberto@ifad.org and

ceciliaruberto@gmail.com Skype: ceci1982lia

Nrusingh Charan Pradhan,

Project Coordinator, International

Development Enterprises India, Plot No. Local

Shopping Centre, Sector-12, Dwarka, New

Delhi-110075 (India)

nrusingh.p@ide-india.org

ncp.bbsr@gmail.com

BKK Information Note – Scampis MSC experience 1

Contents

Abstract ................................................................................................................................................ 2

1.What is Most significant Change? ...................................................................................................... 2

2. The SCAMPIS MSC technique ............................................................................................................ 3

Stories collections ............................................................................................................................. 3

The analysis implemented at various levels ...................................................................................... 4

Lessons Learned ................................................................................................................................ 5

3. Experience of Most Significant Change (MSC) Methodology under the SCAMPIS Project in Odisha. 5

Staff Training & Kids Training ............................................................................................................ 5

Kids/ Youth Training .......................................................................................................................... 6

Stories collection: methods and results ............................................................................................ 6

Second level analysis: methods and results (how it has been done, main results and lessons

learned about the methodology ....................................................................................................... 7

Overall Lessons learned MSC Scampis India ..................................................................................... 7

Web references .................................................................................................................................... 8

Bibliography .......................................................................................................................................... 8

List of Acronyms

SCAMPIS Scaling up of Micro Irrigation Systems

MIS Micro Irrigation Systems

LOF Liquid Organic Fertilizers

STP Surface Treadle Pump

BK Bucket Kit

DK Drum Kit

IFAD International Fund for Agriculture Development

MSC Most Significant Change

NGO Non Government Organisation

RBM Resource Based Management

KM Knowledge management

KS Knowledge Sharing

M&E Monitoring & Evaluation

OTELP Odisha Tribal Empowerment and Livelihoods programme

BKK Information Note – Scampis MSC experience 2

Abstract In 2012 the Scampis project1 introduced an innovative M&E methodology to complement the

quantitative data collection. The project, implemented in India, Guatemala and Madagascar

experimented in the first 2 countries the Most Significant Change methodology.

The main objective in using MSC was to maximize the possibilities to learn from the project, from the

beneficiaries and the staff involved and to discover hidden factors/dynamics that are the cause of

the project success or not. Factors invisible to quantitative, pre-established M&E tools.

Many are the lessons learned, problems solved and problems still open in the adaptation and

application of the MSC method, but overall we can say that in both India and Guatemala it has been

appreciated very much and also from the Hq. point of view it has been very important to solve sense

making issues and create connection among different (and far) stakeholders.

The experience has been documented in an informal blog: www.scampisblog.blogspot.com . Some

work should still be done to capitalize the experience, for sure the same learning from it for further

innovation and use in new context is part of the capitalization process, as something will not be

really part of our background until we do not experiment it practically. For the advantage of the

Bangkok write-shop event a synthetic overview of what has been the Scampis MSC experience with

the exemplification of the India case study is offered. The Guatemala experience is available here.

1.What is Most significant Change? The most significant change (MSC) technique is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded (collection of significant change (SC) stories) and in analysing the data. It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program.

It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole. And it contributes in strengthening organization learning and communication through the analysis process, when the staff comes together to discuss about the perceived MSC or the not perceived (Answering to the question: “what went wrong?”)

Why collecting stories?

• People tell stories naturally - indigenous • Stories can deal with complexity and context • People remember stories • Stories can carry hard messages /undiscussables • But stories not known for accuracy/truth

Why the analysis process

• Analysis for discussions/communication

• For learning: elaborate information and transform it in practical lessons

• For improving our work/our organization work: increase efficetiveness in our activities

• To create bridges among far places and different cultures/experiences

• To understand numbers

• To discover new important element difficoultly perceived by normal M&E tools (especially if

qualitative)

1 http://www.ifad.org/english/water/scampis/

BKK Information Note – Scampis MSC experience 3

Why MSC is relevant for us?

To complete qualitatively the quantitative M&E analysis

• To improve planning (having a better understanding of what is going on in the

field)

• To help explain a point to a farmer (can be used as KM material)

• To recruit new participants

• To help explain a point to another member of staff

2. The SCAMPIS MSC technique In our case, the SCAMPIS project, the technique was extremely relevant to complete the

participatory M&E process started in 20102. The methodology was adopted by SCAMPIS in

Guatemala and India, developed in a participatory way with IFAD Scampis M&E responsible. The tool

consisted in the implementation of 4 steps:

• Stories collection

• Field level analysis and selection / second field level analysis and selection

• National level Analysis (Hq) (only analysis no more selection)

• Cross-countries analysis

The methodology have been reshaped and modified to better match: specific needs and context,

limited availability of resources (money, time).

The main goal for Scampis MSC were: i) understanding M&E numbers, ii) knowledge and learn about

(unexpected) farmers perception of the project/technology, iii) fill the gaps between field reality and

higher top management or donor, iv) involve all staff in a coherent process of discussion and

reflection of the work conducted.

Stories collections3

The methodology has been reshaped from the original MSC (Davis and Dart) in order to adapt to

specific goal and context.

Collected by kids (age between 13-16 years) who will contemporaneously, as real reporters, film,

photograph and ask questions to the project beneficiaries. A training for the kids was provided at the

beginning of the day to break the ice, practice the use of the technologies and understand the

reason why of the activity as well the questions for the adults.

Why Kids or young people?? They are curious, they can easily learn how to use the technology, they

will let the interviewed feel comfortable (elder or senior people are not advisable as they would

influence to much the interview). They can at the best preserve the “ethic” of the MSC stories

collection that is: DO NOT INFLUENCE THE ANSWERS, LET COME OUT THE REAL STORY OF THE

INTERVIEWED. 4 Finally the see how important they are for their community!

2 Video on Scampis M&E and MSC http://www.youtube.com/watch?v=LbjJ4If9yUI 3 Guatemala photos here: http://www.youtube.com/playlist?list=PLA547A54E873204EB&feature=addto ; and videos:

https://picasaweb.google.com/CeciliaRuberto/ScampisGuatemala2012

4 Additional information on how kids have been trained and involved in the stories collection, the central role they played,

opportunities, pitfalls and lessons learned will be shared separately.

BKK Information Note – Scampis MSC experience 4

Figure 1 Stories collection India Figure 2 Stories collection Guatemala

The analysis implemented at various levels The analysis at various level was conducted using brain storming modality reflecting on each story

form 4 different point of view:

A. Who is the interviewed and what I understood of his specific life experience (general)

B. MSC perceived and shared

C. Difficulties and risks perceived

D. New ideas (hidden stories)

After each analysis an overview of the findings have been discussed within the group. During the

first stories analysis, the staff selected the “best”10 MSC stories, and during the second analysis

among this 10 stories 6 have been chosen as most representatives. The videos were translated and

subtitled.5 After the national analysis a cross-country analysis have been implemented (Guatemala

national staff analysed the Indian video and vice versa).

Finally the various analysis plus the stories collected have been collected at IFAD Hq. level for final

evaluation and connection to M&E data. Below a representation of the main learning from MSC.

Figure 3 Scampis MSC - findings

5 Guatemala MSC video playlist: http://www.youtube.com/playlist?list=PLmkP-_uSr1KqUMlmTRPu6crS5FDBsWAGx India MSC video playlist: http://www.youtube.com/playlist?list=PL166815510C378424

BKK Information Note – Scampis MSC experience 5

Lessons Learned Limits of the analysis process were mostly due to time constrains and end of the activities in the

field, no longer staff employed and involved in the project):

- Cross-country analysis only at national level and not local level (which would have been

extremely useful).

- IFAD Hq analysis has not been implemented yet (not sure if will be done..)

- Storage of information: even if the MSC community suggested various different tools for

appropriate storage and elaboration of the info accumulated (text, photos, videos) there has

not been time to learn to use them, so has been used a simple excel programme.

The strength of the tool adopted showed strengthens on the major areas of monitoring, evaluation,

learning and communicating

- Organization development/learning: improve dialogue, trust and democratic discussion.

Moreover it can fill the gaps between top management and field reality

- Project implementation: Activities implementation & Strategy adaptation: if expected results

are not satisfied, the staff will understand it quick and can modify the approach

- M&E: Understanding and contextualize quantitative M&E (it is compatible)

- Effectiveness: Involve project beneficiaries in improving project effectiveness.

- Communication: Improve internal and external communication

- Sustainability: it can be easily implemented and managed by communities and staff (low

costs).

- Evaluation: It can be used for ex-post evaluation

3. Experience of Most Significant Change (MSC) Methodology under

the SCAMPIS Project in Odisha.6

Staff Training & Kids Training The project facilitators like 2 Senior Marketing Officers, 5 Business Associates, Project Coordinator

and Chief General Manager participated in a two day state level training programme on

“Introduction of the new concept of Most Significant Change (MSC). The trainings was attended by

all the stakeholders namely, NGOs, Government agencies under the SCAMPIS programme in May-

2012 at Bhubaneswar

The Scampis IFAD M&E responsible implemented a

training to develop skills and knowledge on the

concept during the training. Basically we learnt

about the technique which was completely new and

innovative in nature. The process of adoption was

purely participatory where the local youths/kids and

users would be the main stakeholders in executing

the process in the operational project villages.

During training programme, the IFAD officials

discussed with all the participants on M&E and MSC

similarities and differences, the principles of MSC, importance of getting the stories and also did the

6 See authors references in the bibliography section. Additional info on the Scampis India MSC activities and

results: http://www.scampisblog.blogspot.com/p/scampis-india.html .

Training event info: http://scampisblog.blogspot.com/2012/11/most-significant-change-training.html

Figure 4 Youths training, India

BKK Information Note – Scampis MSC experience 6

practical exercise on story collection, practical exercise in analysing stories and the role of various

stakeholders.

Accordingly we identified two blocks namely Mohana & Raigad of Gajapati district & three blocks

Pottangi, Nandpur & Dasmanthpur of Koraput district for execution of the MSC tools.

Kids/ Youth Training After completion of the training we proceeded to the identified operational blocks and in the first

step we organised the village meetings to identify user farmers and youth/kids who could be trained

to conduct the interviews. Their identification & training was critical as they were to take the lead in

executing Most Significant Change (MSC) activities. Subsequently 5 training programmes for the

village youths/kids was conducted by the trained Senior Marketing Officers, Business Associates of

the project and also trained staff of local NGOs in 5 blocks. There are few questionnaires developed

by the facilitators and put up during interaction in order to support the narration of the story of the

users and understand the individual Most Significant Change:

I) Could you tell me your name?

II) Where are you from?

III) How many are in your family? How many sons and daughters do you have?

IV) When did you start using the SCAMPIS technology? Where did you see it for the first

time? What did you think of it at the beginning? What technology do you use?

V) Do you cultivate and take care of the plot alone? Someone helps you? Who?

VI) What did you harvested since you started/ How much?

VII) Do you used to cultivate vegetables before having technologies? If answer yes- Do

you see any difference? If answer No-Why?

VIII) Is the main source of income for your family? Have you sold part of the vegetables

produced with MIS? What did you do with the amount obtained from the selling?

IX) How was before the project and now, which are the most significant change you

perceive? Why those changes are important?

The team conducted interactions with the user farmers and shared about the MSC and its purpose.

As a result many farmers volunteered to participate and share their leanings, degree of mobilisation,

success in adopting the technologies and the benefits they accrued. Based on this, plan for the kids

was made.

The youths/Kids were trained in different aspects including conducting interviews, making videos,

taking still photography, developing story writing & also in observing and documenting the process.

At the end of training, each youth/Kid was asked if they had understood their roles & responsibilities

well & any doubts were clarified. In all 24 youth were trained and out of these 18 were boys & 6

were girls. Those Youths/Kids learned about:

Interaction with project beneficiaries Questionnaires Observation Field Visits Focus Group Discussion

Videography Still Photography Documentation of success stories Analysis of the stories

Stories collection: methods and results How it has been done, how many interviews and the results of the interviews. Highlight difficulties

and ideas for improving it.

BKK Information Note – Scampis MSC experience 7

After the youth were trained they started conducting the interviews. A total of 24 kids- 18 boys and

6 girls were trained and they conducted the interview.

Based on the farmer feedback and numbers distributed across the identified locations, the

youths/kids were spread out. 24 kids covered 13 villages in Koraput and Gajapati district from 18-25

May’ 2012 and they met with and interviewed 25 user farmers (Male-23 & Female-2). This was

followed by organising a one day farmers meeting & training for the youths/Kids in each of the

selected villages.

Feedback from the participants (MSC Facilitators) on MSC: “I was very much excited about the

training & facilitating interviews with the user farmers. I learnt many things out of the training and

was enthusiastic how to carry out my responsibilities like making of videos, taking still photographs

and interaction with the users etc. After training, I started to think myself how I will do it…what I will

say etc. But when I actually interacted with the farmers I was very surprised to hear them talk so

much. I am personally inspired by the task entrusted to me carried out up to my satisfaction. This

training helped me to learn more about M&E. It helped me in understanding the communities and

impact of the development interventions. I was also delighted to hear with the farmers who have

been benefited out of the project their increased income and how they feel a sense of social dignity.

Once all the interviews were completed and videos recorded, these were sent for translation into

English. In the meantime, during IDEI’s monthly SCAMPIS staff meeting, the team invited people

from other NGOs, Odisha Tribal Empowerment and Livelihoods Programme (OTELP), Business

Associates (IDEI) and Senior Marketing Officers of SCAMPIS (IDEI) who had attended state level

training at Bhubaneswar to analyze all stories and to select 10 best stories (1st level) on MSC

collected by the youths/Kids. This exercise was successfully completed and 10 stories of the total 25

were selected for 2nd level analysis.

Second level analysis: methods and results (how it has been done, main

results and lessons learned about the methodology Soon after the 2nd level analysis was undertaken. In the second level analysis senior representatives

from NGOs, Odisha Tribal Empowerment and Livelihoods Programme (OTELP), Senior Marketing

Officers and Project Coordinator of SCAMPIS were invited to analyze selected 10 stories and to

select best 6 stories on MSC which would be again analyzed and to select best 3 stories at state level

by the next level representatives of IDEI, and IFAD Delhi team.

Keeping the guidelines in mind, the second level analysis involved watching the videos together (all

representatives present). Everybody was handed over the MSC analysis sheet and each filled their

comments on it. At the end of watching all the videos, six from the ten were selected and six analysis

sheets (one for each video) was finalized and submitted.

Everybody liked the idea of involving the local youth, since it is there habitation they understand the

local environment better and they would not miss the nuances so this reduces the loss of

information.

Overall Lessons learned MSC Scampis India Pitfalls, opportunities, potentiality of the MSC tool for monitoring, for evaluation (at various level) for

strengthening the organization, limitations of the tool.

The overall experience of participating in the MSC training and conducting it was very encouraging

for all involved. First key learning, which we now look as an opportunity is engaging with the local

youth. At first when this was shared we were bit sceptical. We had not involved the local youth in an

BKK Information Note – Scampis MSC experience 8

evaluation process before like this so the anticipation. However, as we met with the local youth and

shared with them about the training and the following work they need to undertake, needless to say

they were very excited and were keen learners.

Secondly, MSC is a good technique that has helped us capture the qualitative aspects/ impact of the

work. It has helped us go beyond numbers and also actually helped us better analyse the numbers

and make cross analysis. Example- if a certain number was achieved (as reported) and in the MSC

farmers shared how they used the income it validated the impact. It helped us get a better insight

into the ‘real’ impact of our work.

While it has many strengths, however I feel that it would be useful to always use it as in addition to a

tool that measures quantitative value of programme success.

Web references Scampis blog www.scampisblog.blogspot.com

Youtube (video material) www.youtube.com/ceciliaruberto

PicasA https://picasaweb.google.com/CeciliaRuberto

Scampis IFAD webpage http://www.ifad.org/english/water/scampis/

M&E news (Rick Davies’ Blog) http://mande.co.uk/special-issues/most-significant-change-msc/

Bibliography The “Most Significant Change”(MSC) Technique – A Guide to Its Use by Rick Davies and Jess Dart

(April 2005). Funded by CARE International, UK

The Most Significant Change Technique (MSC)Power Point document - Dr Jessica Dart, Clear Horizon

The Scampis “Most Significant Change”(MSC) Technique (May 2012) – Cecilia Ruberto, IFAD

MSC Experience in India, Information Note, November 2012.

• Sri Nrusingh Charan Pradhan, Designation: Branch Manager, IDE-India, Mailing Address: A/7,

Nilakantha Nagar, Nayapalli, Unit-8, Bhubaneswar-751012, Odisha, India, Email Id:

nrusingh.p@ide-india.org

• Sri Debendra Kumar Samal, Senior Marketing Officer, IDE-India, mail: debendra@ide-

india.org

• Sri Tapan Kumar Pattanayak, Chief General Manager, IDE-India, mail: tapan@ide-india.org