Listening to the community. Using information and communication

technologies for program monitoring

Zoltán Ferenczibetterplace labgut.org gAGBerlin, 10405 GermanyE-Mail: zfe@betterplace.org

Susanna Krügerbetterplace labgut.org gAGBerlin, 10405 GermanyE-Mail: skr@betterplace.org

Paper submitted to the International Social Innovation Research Conference, London, 2011

1.Introduction

1. Not enough timely data; limited reporting capacities at district levels

2. What is “effective” is contested; “multiple realities” of different stakeholders

Need for more direct, real-time, community level data about social services in DCs

Developing countries face information related challenges in social program monitoring (healthcare, water supply, etc.)

-72 percent of all mobile subscribers located in DCs (ITU 2010)

-“Frogleap development” in telecommunications; no alternative to mobile phones in DCs

1. Introduction (2)

Relevance of mobile phones for the developing world

1.Introduction (3)

-Case studies: one pilot for healthcare in Kenya, and two pilots for water services in Guatemala and Tanzania

-Questions:To what extent has the application of ICTs enabled service providers to remotely gather timely feedback data from program beneficiaries?Were providers able to actually make use of the resulting information to improve their services?

-Not framed as rigorous research: Empirical datasets are yet problematic

The aim of the paper: to explore the potential of ICTs, especially mobile phones

2. Literature

1. Sourcing of information for monitoring purposes (“remote sensing”, “geographical mapping”, “crowdsourcing”)

2. Existing mechanisms of citizen reporting (Citizen Report Cards)

3. Participatory Monitoring & Evaluation (e.g. Guba and Lincoln 1989)

4. Real-time monitoring and evaluation

5. Research about mobile phones and data collection

Five core themes

3. Framework

Concept ValuesDescriptive indicator / quantitative

measure

“Remoteness” “Yes” / “No”

Description of the ICT used in the case

(for example, simple mobile phone with

SMS capabilities)

“Degree of inclusiveness”

“Traditional information sourcing” / “Key

informants” (bounded crowdsourcing) /

“Beneficiary involvement”

(c rowdsourcing)

Description of people involved in

reporting for monitoring (e.g., citizens,

community health workers, local civil

society members, trained employees of

NGOs/IOs, researchers, external

evaluators)

“Immediacy”“Non real time” / “Moderate” / “Real-

time”

Frequency of data transmission (for

example, 3 times a day, daily, weekly,

monthly, etc.)

“Size of feedback data set” Quantitative measureNumber of transmitted reports in relation

to the target group

“Error proneness” Quantitative measureNumber of error entries relative to all

entries*

“Evidence on change”“None” / “Service provision” / “Strategic

use”/ “Both”

Existence of policy document or rhetoric

demonstrating use of data

4. Data

-ChildCount is an integrative technological platform

-Use of SMS text messages from mobile phones by 108 community health workers (CHWs) for patient registration and health reporting

-Central web-based interface containing aggregated health data from the community

Case 1: Health care pilot in Kenya - Involvement, remoteness, immediacy

4. Data (2)

-Over 20,000 SMS-based health reports reaching the web-based interface (patient registrations, nutrition screening reports, vaccination registration)

-Daily data supply

-10 per cent of all messages supplied in an improperly structured format, resulting in their rejection by the system

-Analysis of health-related macrodata of the community such as birth rates, nutritional trends and seasonal variability of malaria rates; performance monitoring of CHWs

Case 1: Health care pilot in Kenya - Resulting data set and action on data

4. Data (3)

-Field Level Operations Watch (FLOW), a cloud-based data collection system to replace paper-based surveys

-Use of electronic surveys installed on Adroid-phones by Water For People staff, field workers of local partner NGOs and community members

-Web-based system for mapping, data analysis and aggregation

Case 2: Water point monitoring pilot in Guatemala - Involvement, remoteness, immediacy

4. Data (4)

-Over 1000 data points: 116 water points 112 school water facilities, 1000 household surveys

-Data on one water point is provided at least once a year (or more frequently)

-Zero errors at data entry

-Data provides the basis for regional meetings among Water For People staff, local partner organisations and the local government; data is regularly included into reports

Case 2: Water point monitoring pilot in Guatemala - Resulting data set and action on data

4. Data (5)

-Approach to empower rural people to become change agents themselves and demand accountability

-Use of text messages and simple mobile phones by citizens

-Central web-based interface

Case 3: Water point monitoring pilot in Tanzania- Involvement, remoteness, immediacy

4. Data (6)

-800 text messages from beneficiaries

-Approx. 2.5 reports a day

-All reports were fed into the system; however, many messages were inappropriately formatted or contained incomplete information

-Use of data for political purposes; 200 messages were passed on to the local government in order to start repairing broken water points

Case 3: Water point monitoring pilot in Tanzania- Resulting data set and action on data

5. Analysis and summary

Case “Remote-ness”

“Inclusiveness” “Immediacy” “Size of feedback data

set”

“Error proneness”

“Evidence based on data”

ChildCount, Kenya

Simple mobile phones (SMS)

Local CHWs as key informants (bounded

crowdsourcing)

real-time(3 reports by CHWs daily on average)

Over 20000 SMS by CHWs

10% of all messages were

rejected

Both service provision

(treatments ) and strategic use (health

programming)

Water for People,

Guatemala

Android capable phones

Local volunteers and community

members (bounded crowdsourcing)

moderate (every water

point regularly updated, at least once a

year)

Ca. 200 data points, each

updated several times= 1000+ feedback set

Zero rejection Both service provision (repairing water points) and

strategic use (programmatic

decisions)

Maji Matone, Tanzania

Simple mobile phones (SMS)

Beneficiaries(Crowdsourcing)

real-time (approx. 2.5 reports a day during pilot)

800 SMS by Tanzanian citizens

Zero rejection. Many messages

were inappropriately

formatted or contained incomplete information

Both service provision (at least 12 water points repaired) and strategic use

(organizational learning)

6. Discussion

-Treat results with caution; success may be context-specific

-Under specific conditions: great potential in the field of international development

However; important to establish the “specific conditions” under which ICT pilots are successful

Need for quantitative studies with large-N comparisons

Implementing NGOs should be aware of the “dark side” of mobile phones; conduct thorough multidisciplinary analysis of the target area prior to pilot launch

More research is needed!

7. Suggestions for future research

-Outreach and promotional activities

-Training

-Setting the right incentives

-Ownership/commitment

-Data validation

Areas of institutional design that may influence outcome:

9. References

Amin, S., J. Das, et al. (2007). Are You Being Served? New Tools for Measuring Service Delivery. Washington, DC:, The World Bank.

Berg, M., J. Wariero, et al. (2009). Every Child Counts - The use of SMS in Kenya to support the community based management of acute malnutritition and malaria in children under five, Millenium Villages Project, Earth Institute at Columbia University.

Bonbright, D. (2006). "Not learning from beneficiaries." Alliance, 11. (2).

Cars, M. (2006). Project Evaluation in Development Cooperation: A Meta-Evaluative Case Study in Tanzania, Stockholm University, Institute of International Education

Caseley, J. (2003). Blocked Drains and Open Minds: Multiple Accountability Relationships and Improved Service Delivery Performance in an Indian City. IDS Working Paper. Brighton, UK, Institute of Development Studies.

Chambers, R. (1994). "Participatory rural appraisal (PRA): Analysis of experience." World Development, 22. (7).

Chambers, R. (1997). Whose reality counts? Putting the first last. London, Intermediate Technology.

Cornwall, A. and V. S. P. Coelho (2007). Spaces for Change? The Politics of Participation in New Democratic Arenas.

Cornwall, A., V. Schattan, et al. (2004). New Democratic Spaces?

Coyle, D. and P. Meier (2009). New Technologies in Emergencies and Conflicts: The Role of Information and Social Networks. Washington, D.C. and London, UK, UN Foundation-Vodafone Foundation Partnership.

Cracknell, B. E. (2000). Evaluating Development Aid. Issues, Problems and Solutions. New Delhi, Thousand Oaks, London, SAGE Publications.

Daraja (2009a). Raising the Water Pressure - Programme Strategy Paper. Harnessing citizens’ agency to promote accountability, equity and sustainability in rural water supply.

Daraja (2009b). Raising the Water Pressure. A Concept Note. Harnessing new technology, the power of information and citizens’ agency to promote equity and functionality in rural water supply.

Donner, J., K. Verclas, et al. (2008). Reflections on MobileActive08 and the M4D Landscape. In Perspective. Proceedings of 1st International Conference on M4D 2008.

Eagle, N. and A. S. Pentland (2009). "Eigenbehaviors: identifying structure in routine." Behavioral Ecology and Sociobiology, (63): 1057–1066.

Fals-Borda, and Rahman, M. A. Eds. (1991). Action and Knowledge: Breaking the Monopoly with Participatory Action-Research. New York, Apex Press.

9. References

Forss, K. and J. Carlsson (1997). "The Quest for Quality - Or Can Evaluation Findings Be Trusted." Evaluation, (3).

Freire, P. (1972). Pedagogy of the Oppressed. New York, Herder & Herder.

Goetz, A.-M. and J. Gaventa (2001). Bringing citizen voice and client focus into service delivery. IDS Working Paper 138. Brighton, Institute of Development Studies.

Guba, E. G. and Y. S. Lincoln (1989). Fourth Generation Evaluation. London, Sage Publications.

Hellström, J. (2008). Mobile phones for good governance – challenges and way forward. http://www.w3.org/2008/10/MW4D_WS/papers/hellstrom_gov.pdf,

Holland, J. and J. E. Blackburn, Eds. (1998). Whose Voice? Participatory Research and Policy Change. London, Intermediate Technology Publications.

Howe, J. (2006). "The Rise of Crowdsourcing." Wired, (14.06.).

ITU (2010). World Telecommunication/ICT Development Report 2010. Monitoring the WSIS targets. A mid-term review.

Kothari, U. (2001). Power, Knowledge and Social Control in Participatory Development. Participation. The New Tyranny? B. Cooke and U. Kothari. London, Zed Boks: 139-152.

Krüger, S. and S. Teggemann (2008). Institutional Leadership in a Multistakeholder International Development Setting. Leadership as a Vocation. Houben/Rusche, Nomos Verlag.

Lundberg, M. (2008). Client Satisfaction and the Perceived Quality of Primary Health Care in Uganda Mattias Lundberg. Are you being served? New Tools for Measuring Service Delivery: 323.

Martin, C. (2009). Put up a billboard and ask the community: Using mobile tech for program monitoring and evaluation. MobileActive.org. 2009, October 31. http://mobileactive.org/put-billboard-and-ask-community-using-mobile-tech-program-monitoring-and-evaluation, 15.03.2011

McGee, R. and J. Gaventa (2010). Review of Impact and Effectiveness of Transparency and Accountability Initiatives. Sythesis report. Transparency and Accountability Initiative Workshop, Institute of Development Studies.

Mohan, G. (2001). Beyond Participation: Strategies for Deeper Empowerment. Participation. The New Tyranny? U. Cooke Bill; Kothari. London, Zed Books: 163-167.

Munyua, A. W. and M. Mureithi (2008). "Harnessing the power  of the cell phone by women entrepreneurs: new frontiers in the gender equation in kenya. grace project research report."

9. References

Norheim-Hagtun, I. and P. Meier (2010). "Crowdsourcing for Crisis Mapping in Haiti." Innovations: Technology, Governance, Globalization 5 (4): 81-89.

Patnaik, S., E. Brunskill, et al. (2008). Evaluating the Accuracy of Data Collection on Mobile Phones: A Study of Forms, SMS, and Voice.

Patton, M. (2011). Developmental Evaluation. Applying Complexity Concepts to Enhance Innovation and Use. New York, London, The Guilford Press.

Paulos, E., R. J. Honicky, et al. (2009). Citizen Science: Enabling Participatory Urbanism. Handbook of Research on Urban Informatics: The Practice and Promise of the Real-Time City. M. Foth. Hershey, New York, Information Science Reference, IGI Global.

Power, M. (1997). The Audit Society, Rituals of Verification. New York, Oxford University Press.

Ravindra (2004). An Assessment of the Impact of Bangalore Citizen Report Cards on the Performance of Public Agencies, ECD Working Paper Series

Rebien, C. (1996). Evaluating Development Assistance in Theory and Practice. Avebury, Aldershot

Rudqvist, A. and P. Woodford-Berger (1996). Evaluation and Participation. Sida Studies in Evaluation 96/1. Stockholm, Department for Evaluation and Internal Audit, Sida

Schuster, C. and C. Perez-Brito (2011). "Cutting costs, boosting quality and collecting data real-time. Lessons from a Cell Phone-Based Beneficiary Survey to Strengthen Guatemala’s Conditional Cash Transfer Program." En Breve, World Bank LAC, (166).

Sutton, P., D. Roberts, et al. (1997). "A Comparison of Nighttime Satellite Imagery and Population Density for the Continental United States." Photogrammetric Engineering and Remote Sensing, (63): 1303-1313.

Turner, M. (2011). Mobilizing Development. Washington, D.C., Berkshire, UK, The UN Foundation and Vodafone Foundation Technology Partnership.

Ulbricht (2011). MobileActive.org, http://www.mobileactive.org/darker-side-mobiles-women-part-two-potential-dangers, 06.06.2011

UNDP (1997). Who Are the Question-Makers? A Participatory Evaluation Handbook.

Water for People (2007). Water for People - Guatemala Country Strategy.

Weiss, C. (1997). "How can theory-based evaluation make greater headway?" Evaluation, 21. 501-524.

WHO (2006). The World Health Report 2006 - Working Together for Health. Geneva, World Health Organization

WHO (2008). Safer Water, Better Health.

World Bank (2004). World Development Report. Making Services Work for Poor People.

9. Interviews

-Interview with Ben Taylor, Executive Director of Daraja, Tanzania

-Interview with Keri Kugler, programmatic data manager at Water for People, Denver

-Interview with Dr. James O. Wariero, ChildCount, Kenya, Nairobi

betterplace lab

Schlesische Strasse 2610997 BerlinTel +49 30 76 76 44 88-0Fax +49 30 76 76 44 88-40lab@betterplace.org

Vielen Dank.