Embed Size (px)
Citation preview
Evaluation of the Development Account Project
“Strengthening the Capacity of African Countries to Use Mobile
Technologies to Collect Data for Effective Policy and Decision
Making”
Midterm evaluation report
Submitted by: Alexandre Diouf
September 2016
i
Disclaimer: This report is prepared for the United Nations Economic Commission for Africa (ECA). It was written by Alexandre Diouf. The
author’s views expressed in this report do not necessarily reflect those of the ECA.
DA Project MTE
Report i
TABLE OF CONTENTS
Abbreviations .............................................................................................................................................. ii
executive summary ..................................................................................................................................... iii
PROJECT BACKGROUND............................................................................................................................ III
THE EVALUATION ..................................................................................................................................... III
RESULTS ................................................................................................................................................... III
RECOMMENDATIONS ................................................................................................................................ IV
1. Introduction ............................................................................................................................................ 1
1.1 PILOT PROJECT OVERVIEW .............................................................................................................. 1
1.2 GEOGRAPHIC SCOPE AND SELECTION CRITERIA ............................................................................. 2
1.3 KEY PERFORMANCE INDICATORS .................................................................................................... 3
1.4 EVALUATION FRAMEWORK ............................................................................................................. 4
1.5 EVALUATION CRITERIA ................................................................................................................... 5
2. Methodology ........................................................................................................................................... 5
2.1 PHASE 1: PREPARATION ................................................................................................................... 5
2.2 PHASE 2: COUNTRY DATA COLLECTION AND SYNTHESIS ............................................................... 6
2.3 PHASE 3: DRAFT AND FINAL REPORTING ........................................................................................ 7
3. Results and discussions .......................................................................................................................... 7
3.1 RELEVANCE...................................................................................................................................... 7
3.2 EFFECTIVENESS .............................................................................................................................. 10
3.3 EFFICIENCY .................................................................................................................................... 13
3.4 PROGRESS TOWARDS IMPACT ........................................................................................................ 17
3.5 SUSTAINABILITY ............................................................................................................................ 21
4. Major issues and problems at the country level ................................................................................ 24
5. Lessons learned .................................................................................................................................... 25
6. Recommendations ................................................................................................................................ 26
Annexes ........................................................................................................................................................ 1
1. DATA COLLECTION TOOLS .................................................................................................................... 1
2. EVALUATION TIMELINE ........................................................................................................................ 5
3. PEOPLE INTERVIEWED AND CONTACTS ................................................................................................. 6
4. EVALUATION TOR ................................................................................................................................ 9
DA Project MTE
Report ii
ABBREVIATIONS
ACS African Centre for Statistics
CAPI Computer-assisted personnel interview
CEA Commission Économique des Nations Unies pour l’Afrique
CPI Consumer Price Index
CSA Central Statistical Agency
CSPro Census and Survey Processing
DA Development Account
ECA Economic Commission for Africa
ESA Ethiopian Statistical Association
GBoS The Gambia Bureau of Statistics
GPS Global Positioning System
ICT Information and communication technology
INS Institut National de la Statistique
KNBS Kenya National Bureau of Statistics
M&E Monitoring and evaluation
MoHCC Ministry of Health and Child Care
MTBDC Mobile technology-based data collection
NGO Non-governmental organisation
NSO National Statistic Office
NSI National Statistical Institute
PDA Personal digital assistant
SOM School of Mathematics
TRI Training and Research Institute
UoN University of Nairobi
ZIMSTAT Zimbabwe National Statistics Agency
DA Project MTE
Report iii
EXECUTIVE SUMMARY
PROJECT BACKGROUND
This is the Final Report of a Midterm evaluation of the Development Account Project “Strengthening the
Capacity of African Countries to Use Mobile Technologies to Collect Data for Effective Policy and
Decision Making” (DA project). The project was implemented by the United Nations Economic
Commission for Africa (ECA). The Development Account project was launched in 2013 with the aim of
enabling the National Statistical Offices (NSOs) to develop technological bases for the mobile data
collection in a partnership with Training and Research Institutes (TRIs) within the country. Cameroon,
Gambia, Kenya, Zimbabwe, Ethiopia and Tunisia were selected as pilot countries where NSO and TRI of
each country established the implementation program in a collaboration.
THE EVALUATION
The rationale for the evaluation is threefold:
Assess the achievement of the DA project to date from the standpoint of its overall
performance, coverage and outreach approach, relevance, efficiency, effectiveness, impact
and sustainability.
Verify and ground-truth, to the extent feasible, DA results reported by country teams.
Generate data and information to allow ECA to make informed decisions concerning the
best use of DA resources for the second phase of the project.
The evaluation employed the following data collection methods:
Document review: Soon after being awarded the contract, the team of consultants conducted a
literature review focused on the performance of the DA project. Documents reviewed included
project proposal, performance reports provided by ECA, project design documents, and other
relevant documents.
Primary Research. A national consultant was contracted in each of the beneficiary countries. The
work of the national consultants was coordinated by an international team leader. The team leader
developed the evaluation tools and procedures and coached the national consultants during the
evaluation process. National consultants were in charge of data collection, processing and reporting
as indicated by the team lead at the national level.
Focus Group Discussions and Key Informant Interviews: Focus Group Discussions (FGDs) were
organized and key informant interviews were organized in each country and at ECA level. National
consultants were able to interview the ECA focal points, the staff of the NSOs and TRIs involved
in the development of the software as well as the enumerators who were the ultimate users of the
software.
RESULTS
Regarding relevance, all participating countries had their long term development plans that required timely
provision of more statistics with better quality. For example, in the case of The Gambia, the project was
particularly relevant as the government aims to boost the economic development by incorporating IT
technologies. The project was also found very pertinent in Zimbabwe where in-house staff capacity already
existed but needed support in providing training and purchasing equipment.
DA Project MTE
Report iv
About Effectiveness, across all six countries, in terms of reporting, the ECA coordination team had
developed an M&E mechanism to guide the implementation of the DA project. As a result, there is evidence
to support the rate of implementation. Reporting can range from essential short-/midterm reports (e.g.
national situation analysis, national workshop reports, hardware procurement, etc.) to ECA focal points
reporting to DA project headquarters every 1–2 months on project status in the countries according to the
work plan. In all but one country, the NSOs were able to develop and pilot a geo-enabled mobile data
collection system for the CPS, successfully enhancing the reliability of the data as they can track the
authenticity of the data collected. But since it was a pilot project with a limited budget, the NSOs were not
always able to test the software at national level. In Tunisia particularly, however, respondents felt at times
that the NSO chose the ESSAI as its TRI more to meet an administrative requirement of the agreement with
the ECA than to develop a real technological partnership. And indeed, this TRI’s lack of participation at
the design phase of the project, and the absence of a specific budget assigned to the ESSAI, weakened its
commitment to the project. Under the effectiveness criteria, then, the project is deemed satisfactory because
it achieved most of its intended results by the end of its first phase.
For efficiency, in general, the project was highly efficient across the six countries. The project was highly
valued by the interviewees because it facilitated the processes used by the NSO to collect, process and
report on price information. The project built the capacity of both the TRIs and the NSOs, which was a key
achievement. The software seemed to have integrated geo-referencing information and were said to be very
user-friendly and are actually being used to collect and report on price information.
With regard to impact, the DA project, although piloted in six countries, had several impacts at country
level. The most important of those impacts are: the reduced data entry errors, the reduced workload, the
real time data collection and transmission and the easy way to detect anomalies. Most importantly the time
needed to generate critical information related to prices was cut by about 20 to 40% depending on the
country. The software made it possible to detect at an early stage any error, typo that happen during the
process of data collection, entry and processing which was found to be instrumental for it widespread
adoption by key actors.
For sustainability, at the time of the midterm evaluation, the TRIs and NSOs which have been involved
in the implementation of the project had the capacity to continue to use the software and generate useful
price-related information using mobile technologies. Critical capacity has been generated and is available
at country level, should similar project need to be replicated. No plan seemed to be put in place for the
project to continue critical activities beyond its lifetime though. Going forward it will be necessary to think
about that to maintain the project achievements and perpetuate its impacts.
RECOMMENDATIONS
At the end of this evaluation, the following recommendations are made to help sort out the problems
encountered by the DA project and facilitate design and implementation of similar projects in the future:
It is important for ECA’s future projects that the NSO receives assistance in the TRI
selection methodology, through the provision of a TRI assessment grid prior to signing the
agreement between the ECA and NSO. At the same time, the agreement must clearly define
DA Project MTE
Report v
the TRI as well as the TRI’s expected activities and its budget for those activities.
Otherwise, there is a risk that the NSO will work alone in order to retain the entire budget.
The ECA should take into account the time it needs to complete its administrative processes
before the funds can be accessed by their partners. A proper planning will reduce delays
suffered by similar projects in the future.
It is important to have local or regional cloud servers to guarantee the security of data
collected via the newly-developed platforms.
The choice of a TRI and the theme of the DA project needs to be consultative and
exhaustive in order to ensure project success.
The CSA should carry out further pilot testing with large geographic coverage of urban
and rural areas of the country. This would promote better appreciation for the efficiency
and effectiveness of the MTBDC system for price data collection, including the presence
or absence of power, internet connectivity, and other essential factors.
DA Project MTE
Report 1
1. INTRODUCTION
Statistics increasingly play a crucial role in the design of monitoring and evaluation (M&E)
systems for development policies, by drawing out and clarifying priorities and objectives to be
attained. This finding, confirmed during the 2012 UN Conference on Sustainable Development
(Rio+20), also highlighted the need to make statistics available and accessible to all users—at
local, national, and international levels. In some African countries, however, statistical data are
not always collected, nor are their quality, reliability, and timeliness always assured.
The Economic Commission for Africa (ECA) has cited several reasons why statistics in some parts
of Africa are not being collected, compiled, and made available to national and global users.
Financial constraints, lack of appreciation for how data can be used for evidence-based decision
making and monitoring of implementation, low awareness of the advantage data being shared with
the international community are a few of the obstacles noted. But inadequate human capacity and
low technology base are the obstacles deemed the most critical to national statistical systems
preparing to shift from manual, paper-based data collection methods to those that increasingly rely
on information and communication technologies (ICTs), mobile technology-based data collection
(MTBDC) systems in particular.
ICT initiatives have the ability to support unprecedented changes in leveraging limited resources
for those committed to overcoming such core challenges as low levels of ‘digital literacy’ and
internet connectivity. They offer an opportunity to intervene and narrow the digital divide. The use
of mobile technologies to collect data and disseminate information would reduce the cost and time
associated with these tasks and lead to the following additional benefits:
Higher quality and more complete data availability
Use of real- or near real-time trend spotting with visualisation tools. Real-time data enable
better decision-making, adaptive management, and improved allocation of limited
resources.
Greater data security and archiving, which are especially important for ensuring data
transparency and conducting data audits.
The African region is witnessing one of the strongest increases in mobile data use in the world. It
is predicted that mobile internet traffic across Africa is expected to increase dramatically by 2018,
and will possibly see a 20-fold increase by the end of the decade (Lange et al. 2014).1
1.1 PILOT PROJECT OVERVIEW
In many African countries, several factors prevent statistics from being collected, compiled, and
made available to national and global users. These include financial constraints, poor appreciation
for the use of data for evidence-based decision-making and monitoring of implementation, and
1 Africa mobile broadband market. Lange et al. 2014. (Available at: http://www.budde.com.au/Research/2014-Africa-
Mobile-Broadband-Market.html?r=51)
DA Project MTE
Report 2
low awareness of the advantage of broadening the international community’s access to the data.
But of these factors, it is low human capacity and a low technology base that most limit the full
potential of ICT tools—mobile devices in particular—to improve how statistical data are being
collected, disseminated, and used.
To target the problems associated with data collection, particularly the technology and human
capacity issues, the ECA launched the Development Account (DA) project, “Strengthening the
Capacity of African Countries to Use Mobile Technologies to Collect Data for Effective Policy
and Decision Making”. Begun in 2013, this two-phase, two-year pilot project aimed to strengthen
data collection through the use of mobile technologies. It focused on building capacities of national
statistical offices (NSOs) and government departments in a number of African countries for
effective policy and decision-making. The expected outcomes centred on increased capacity in
two critical areas—namely, NSOs would partner with training and research institutions (TRIs) to
develop methodologies and systems for mobile data collection, and project countries would
develop geo-enabled data collection systems running on mobile devices.
1.2 GEOGRAPHIC SCOPE AND SELECTION CRITERIA
For phase 1 of the DA project, five pilot beneficiary countries were selected based on several criteria,
including:
Level of commitment and readiness to implement the DA project
Geographical placement, language, and manageable size of the country
Level of development of ICT tools/mobile devices and willingness to use them for data
collection
Security level/sustainable institutions in the country
Relevance to the work of the NSO.
As part of the situation analysis, a consultant was engaged to review potential countries based on
these criteria, and put forward those that met them, to inform the selection. The consultant’s
proposal was presented to a joint meeting of the technical and steering committees, which was
held during a regional workshop in Praia, Cape Verde, in March 2014. Cameroon (Central Africa)
and Kenya (East Africa) were selected based on the consultant’s recommendation in the situation
analysis report; both countries confirmed their willingness and readiness to participate. The
Gambia and Tunisia were selected from West and North Africa based on presentations by their
respective NSOs during the workshop and meeting. Zimbabwe was selected after presentations by
Research and Information Services (which later became the TRI for the Zimbabwe pilot project).
The presentations demonstrated both the suitability of the countries for the project as well as their
strong commitment to the project’s implementation. The DA project subsequently expanded to
include a sixth country, Ethiopia, after the Embassy of Ireland in Ethiopia expressed interest in
supporting the project.
DA Project MTE
Report 3
1.3 KEY PERFORMANCE INDICATORS
To measure progress towards the achievement of the project objectives, five key performance
indicators were developed, all based on the number of:
1. Geo-enabled mobile data collection systems adapted or developed through partnerships
with training and research institutions established and functional
2. Countries that have adapted project methodology and work-flow for data collection and
processing
3. Pilot countries using mobile devices to collect and analyse data
4. Data collection campaigns undertaken as part of the project
5. Publications and reports attributing their data sources in the pilot countries to national and
regional data collected by the project.
Brief, country-specific highlights of the DA project are given below.
Cameroon
Implementation of the DA project related to data collection aimed at designing consumer price
indices is a routine operation of the National Statistical Institute (NSI). It is articulated around four
main initiatives: the organisation by the NSI of a national training workshop for the data collection
on price; using mobile devices to collect data on producer prices at 3,702 points of sale across the
country; the calculation of price indices; and conducting a satisfaction survey.
Ethiopia
Only in Ethiopia was the project funded by the Government of Ireland. The Central Statistical
Agency (CSA), in partnership with the Ethiopian Statistical Association (ESA), implemented the
pilot project. Field testing and pilot mobile data collection were carried out for the monthly retail
and producer prices in the Tigray region of northern Ethiopia, in order to build synergies with other
programmes that the Irish government is funding in that part of Ethiopia.
The Gambia
The project was implemented by The Gambia Bureau of Statistics (GBoS) as the NSO, in
collaboration with the University of The Gambia as the TRI. Implementation of the DA project,
which sought to collect and disseminate data on price indices, ended in March 2015.
Kenya
The project was implemented by the Kenya National Bureau of Statistics (KNBS), in partnership
with the School of Mathematics (SOM)–University of Nairobi (UoN). Following a presentation
by the KNBS on areas where it collects, the Consumer Price Index (CPI) was selected for the pilot.
Tunisia
DA Project MTE
Report 4
The project was planned to be implemented by the Institut National de la Statistique (INS), in
collaboration with the Ecole Supérieure de la Statistique et de l'Analyse de l'Information (ESSAI)
as the TRI.
Zimbabwe
The Zimbabwe National Statistics Agency (ZIMSTAT), in partnership with Research and
Information Services, initially implemented a DA pilot project. It focused on the revival and
maintenance of electronic village/area registers as a tool for collecting, using, and disseminating
data for development. The project, however, was not completed as the TRI ran into problems with
the government over its registration. As a result, and with the approval of the ECA, ZIMSTAT
implemented an alternative project, the Consumer Price Survey (CPS), to achieve the objectives
of the DA pilot project in Zimbabwe without a TRI.2
1.4 EVALUATION FRAMEWORK
An independent midterm evaluation (MTE) of the DA project was carried out in the six target
countries from June to July 2016. The MTE will enable the ECA to assess the validity of the
project’s design and assumptions, and its performance. The findings will specifically inform the
design and implementation of upscaling the intervention into additional countries across the
African continent. The specific objectives of the assignment are to:
Make an overall independent assessment of the first phase of the DA project’s
performance, paying particular attention to the relevance, effectiveness, efficiency,
sustainability, and impact of project actions against results and objectives.
Identify key lessons learned and propose practical recommendations that guide the design
of the project’s second phase.
The MTE was based on an iterative design aimed at capturing the perspectives of all stakeholders
and assessing the DA project’s overall performance against ECA’s evaluation criteria. This
approach enables a ‘360 degree evaluation’, incorporating the views and perspectives of the ECA
and its country-level partners (NSOs, TRIs) as well as enumerators who have been using the
software developed in each participating country. These multiple lines of evidence provided the
considerable length and breadth of data needed to assess the capacity-building, software quality,
and the data quality improvement components as well as the potential sustainability of the project.
These data will also inform possible programmatic decisions for the DA’s second phase for both
the ECA and partners.
2 Even though there was some progress and expenditure related to the uncompleted project, including some outputs in
terms of reports, the midterm evaluation focuses on the latter project implemented by ZIMSTAT on the CPS.
DA Project MTE
Report 5
1.5 EVALUATION CRITERIA
The MTE formulated a set of broad strategic questions, with relevant sub-questions, to provide
information on the extent to which the project has been implemented. Organised around five
components (i.e. ‘project implementation dimensions’), the evaluation sought to determine:
Relevance: The extent to which the objectives pursued by the DA project are consistent
with the country context, needs, and priorities of the member States and institutions in
collecting, analysing, and reporting on macro-level statistics.
Effectiveness: The extent to which the DA project is on track to attain its intended targets,
and whether these targets have been transformed into the results anticipated at the project’s
design stage. More specifically, the MTE assessed the extent to which the project has
enhanced the capacity of mobile data collection at NSO and TRI levels, the quality of the
software, and the extent to which it is being used by NSOs and TRIs.
Efficiency: The extent to which the DA project, implemented at country level, transformed
the available resources into the expected results in terms of quantity, quality, and
timeliness. The evaluation assessed the extent to which the software developed during the
project is user-friendly and complete. The evaluation has identified the factors that explain
(or contribute to) the level of use of the software. Under this component, the evaluation
also looked at the capacity of the software to integrate geo-referenced information.
Impact: The extent to which the results achieved through the DA project have contributed
(or will contribute) to increasing the quality and reducing the time needed to collect and
report country-level statistics. It also assessed the extent to which the project has improved
the quality of data in NSOs.
Sustainability: The extent to which the positive results of the DA project will continue
after it is completed. The evaluation specifically looked at the capacity of the NSOs and
TRIs to continue to use and improve the software. It examined both the working
arrangements and plans that have been put in place so as to continue critical activities of
the project beyond its lifetime, as well as the financial plan that has been set up to support
the implementation of those activities.
2. METHODOLOGY
Operationally, the MTE was conducted in three complementary phases as discussed below.
2.1 PHASE 1: PREPARATION
DA Project MTE
Report 6
Phase 1 began following contract award. It included extensive communication between the
consultant and ECA’s focal point from the African Centre for Statistics (ACS) and its Evaluation
Section. During this phase, all the DA documents were shared with the consultant, who reviewed
them and then developed a work plan for the evaluation. Also during phase 1, the consultant held
Skype conversations with the ACS focal point and the Evaluation Section. The purpose of the
communication was to discuss and clarify ECA’s expectations from the exercise and its results in
order to achieve consensus and develop a common understanding of the MTE’s overall schedule
and approach. The consultant submitted an inception report detailing the methodology and the
evaluation tools. At the end of phase 1, he had a detailed evaluation methodology that included the
revised list of evaluation questions and assumptions to be tested, refined data collection and
analysis tools, and an agreed timeline (see Annexes 1 and 2).
2.2 PHASE 2: COUNTRY DATA COLLECTION AND SYNTHESIS
The development of the inception report followed an initial document review of the project
proposal, including documents related to the project in each country, project monitoring reports,
and country situation analysis reports. National data collectors were contracted in each of the six
countries; they led the enumeration and analysis work at country levels. The lead consultant held
one-on-one Skype meetings with each of the national data collectors to explain the methodology
and expectations regarding data collection, analysis, and information reporting. The data collectors
then interviewed representatives of the NSOs, TRIs, and enumerators in their respective countries,
using the methodology and data collection tools in the inception report.
At the NSO level, the data collector identified and interviewed at least three people who have been
involved in the implementation/management of the DA project, which represents between 20 and
50% of the people who have been involved in the implementation of the project. Similarly, at least
three people were interviewed at the TRI level and at least five enumerators who have used the
software. A set of questions that specifically shed light on the evaluation questions has been
developed for each group of interviewees. Annex 3 presents a list of those interviewed and their
contact information.
To ensure that the process used for the evaluation was as thorough and reliable as possible, the
different data collection tools to be employed (semi-structured interview guides, focus group
discussion guides, key informant interview guides, etc.) were approved by the ECA, along with
the other tools and procedures used during this evaluation, prior to the fieldwork phase.
An important element of the MTE was the breadth of data sources used and the multiplicity of
stakeholder perspectives sought. Data collected from each country was analysed separately and
reported in a specific document. The synthesis stage of the project evaluation involves an
overarching analysis of all the different types of data collected on thematic and country bases,
using the evaluation criteria set forth in the terms of reference (Annex 4). This analysis compares,
contrasts, and synthesises perspectives and experiences across the different data sources and
stakeholders.
DA Project MTE
Report 7
2.3 PHASE 3: DRAFT AND FINAL REPORTING
After collecting the data in phase 2, country-level data collectors submitted a draft of the country
reports to the lead consultant and the ECA for their review. The reports were reviewed in
compliance with the guidelines laid down in the inception report. Following the submission of
revised reports by the national data collectors, the lead consultant drafted the consolidated the
present MTE report.
3. RESULTS AND DISCUSSIONS
3.1 RELEVANCE
The main area of evaluation under this section focused on the relevance and applicability of the
DA project activities as well as the anticipated benefits and problems that the project could help
to solve. It examined whether the project approach and objectives were relevant and appropriate
to the achievement of expected outcomes, given national and regional priorities.
Cameroon
For Cameroon, the activities were relevant and consistent with the national development plan in
relation to statistics. The DA project helped to build the capacities not only of NSI staff but also
those in the national statistics system (i.e. Ministère de l’Agriculture et du Développement Rural
and Ministère de l’Élevage, des Pèches et des Industries Animale) to use MTBDC methods. In
June 2015, the NSI carried out a diagnostic study that identified needs in relation to this.
Ethiopia
In Ethiopia, in-depth discussions with relevant representatives from each of the CSA and ESA
show that the project approach and objectives are highly relevant to national and regional priorities.
Representatives of the CSA (formerly the Central Statistical Authority) and ESA enumerators both
agree that the logical framework of the DA project has been very helpful in the implementation of
project activities. Similarly, the ECA reported that the key performance indicators in the logical
framework were also appropriate and of sufficient quality to assess performance.
The CSA and ESA considered project activities at country level to be well designed and to fit the
framework of national and regional strategies for development in the area of intervention. In this
respect, the strategic plan of the CSA (i.e. the National Statistics System) emphasises use of
technology to improve supply of data and quality of data, as opposed to the traditional paper-based
system.
The mandate of the CSA is twofold: (1) to collect, process, analyse, and disseminate statistical
data, and (2) to provide technical guidance and assistance to government agencies and institutions
in building administrative systems and registers. This includes building capacity and providing
directives for database creation and proper management of administrative records. The current
second 5-year national development plan of Ethiopia, the Growth and Transformation Plan II, also
DA Project MTE
Report 8
requires massive data for M&E of the national growth and development process, including demand
for statistical data in areas or issues which have not been covered by formal surveys/census
activities of the CSA. Regarding the gender dimension of the DA project, proper data collection,
dissemination, and use and by policymakers at macro- and micro-levels do help to address gender
gaps and challenges in national development. Designing data collection forms in a gender-
disaggregated manner is central for development policy decision-making. So too are collecting
and disseminating gender-disaggregated data and analysed information. The use of MTBDC
technologies can better facilitate and improve the quality, quantity, and timely generation of
gender-disaggregated data, further catalysing gender-friendly and gender-inclusive development
activities at national, regional, and local levels.
The data collected through the project’s MTBDC systems have all been fully geo-referenced. The
enumerators already had Global Positioning System (GPS) software loaded on their tablets. They
also had a Garmin (external GPS) coordinate reader to help ensure that they took proper geo-
referencing of the retail and producer price market places. The respondents for this evaluation from
the CSA and ESA stated that the project’s objectives remained valid and relevant throughout its
implementation.
The Gambia
In The Gambia, almost every stakeholder met during the evaluation rated the project as very
relevant, especially in the context of demand for timely provision of statistical data for informed
policy decisions. Furthermore, respondents expressed that the project’s targets are well defined
and have strong linkages with the Gambia’s VISION 2020 (and by extension, its Medium
Development Plan known as the ‘Programme for Accelerated Growth and Employment’).
The GBoS plays a crucial role in the country’s overall development and policymaking process by
providing the evidence for setting policy objectives, targets, and priorities for the government and
the international community. The manual form of data collection used by the GBoS remains a
challenge as it is neither a speedy nor efficient way to process data. To overcome this problem, it
is crucial to introduce emerging technologies if the GBoS is to have an efficient and cost-effective
method of collecting data for use by NSOs elsewhere in Africa. The DA project is quite relevant
and will continue to be so in The Gambia. The government aims to make the country’s ICT
infrastructure ‘e-ready’ and so boost economic development by focusing on telecommunications,
information technology, and media. The project, as revealed during the interviews, was perfectly
designed for this as it facilitated a collaborative approach between the GBoS as an implementing
partner and the School of Information, Technology and Communication of the University of The
Gambia as a TRI. This is a mutually reinforcing partnership; each party has a comparative
advantage during project implementation.
Kenya
For Kenya, the CPI and corresponding inflation statistics are compiled and disseminated every
month as one of the key statistical outputs of the KNBS. As noted during the evaluation, KNBS
DA Project MTE
Report 9
staff were satisfied with the approach used during implementation of the DA project. To them, the
selection of the CPI for the pilot was affirmed following a presentation on areas where it collects
data. Both the KNBS and SOM-UoN felt that the project’s activities are quite relevant. They
observed that the collection, transmission, validation, analysis, dissemination, and storage of
survey data needed to be more efficient and cost-effective, while utilising scalable and suitable
technologies.
Retail prices to be used in the compilation of the CPI is currently collected every month in 25 data
collection zones in 13 urban centres. Each of these 25 zones is manned by a price collector and
supervised by a county statistical officer. Nationally, more than 12,000 price observations are
made every month from a sample of more than 4,000 outlets. According to the interviewees, the
CPI is used in a number of ways. These include deflation of monetary values; as an indicator of
macroeconomic performance; to determine supplier/debtor price variations; to determine
employer/ employee wage negotiations; and to index pension benefits. All these have to do with
the value of money. Hence, it is imperative that the KNBS collects, compiles, and disseminates
CPI data timely and with a high degree of accuracy so that it remains a credible source of this
important statistic. Besides, quality CPI data are crucial for informing policymakers, investors,
and the general public on inflation trends in Kenya. Prior to February 2009, prices collected from
the outlets were written on paper and posted to the CPI office, at the KNBS headquarters. This
process was discontinued because it cost money for courier services and every month it was common
for some field returns to be late.
The logical framework indicators were also considered appropriate and of sufficient quality to
assess the performance of CPI data collection. What is more, the DA project activities in Kenya
were well designed and fit into the framework of national and regional strategies.
Tunisia
According to INS officials, at the beginning of the DA project Tunisia had no available expertise
in mobile technology for data collection, neither at universities nor in private sector (telecoms or
information technology development companies). The INS therefore needed to develop in-house
skills to be able to quickly implement technical solutions, partly due to the urgent need to carry out
the CPI survey. INS officials are satisfied with the activities formulated during the project, which
have enabled them to achieve their objectives.
Zimbabwe
In Zimbabwe, the DA project was relevant for a number of reasons. Although the Agency’s staff
had capacity to implement mobile systems to collect data, they needed to build relevant, supporting
capacities in training (how to implement Android-based systems) and equipment (server and
tablets provided by the project). The project was also relevant because it showed the NSO and its
stakeholders that it is possible to improve data quality and cut costs when mobile data collection
systems are implemented. As well, the conventional paper-based CPS uses too much paper and
generates processes that make having data available for decision-making unnecessarily
DA Project MTE
Report 10
complicated and delayed. The survey managed to use geo-referencing, thus reducing the likelihood
that enumerators and their supervisors would manipulate data since the application could confirm
whether enumerators had actually collected the data from the field.
The project was piloted in only 3 of Zimbabwe’s 10 provinces. Evidence from the pilot results
suggests that implementation of MTBDC systems in all 10 provinces would have reduced the
administrative burden of printing questionnaires, the cost of transporting the questionnaires to and
from the field, and the rigorous process of data entry and cleaning for analysis. It has been
estimated that the use of mobile data collection would reduce the time for a given survey by about
45 days. Furthermore, had the project been implemented in all the provinces, data quality would
be very high as the likelihood of mistakes was reduced at the point when data were collected. The
manual paper system was less efficient, as the information takes a long time to reach the head
office, where it must then be entered. Any errors would merely serve to prolong this process. With
the DA project, the information would be delivered instantly for analysis after a quick supervisory
check. The project was relevant to ZIMSTAT as it also helped to build capacity of its staff and
enhanced its ability to handle statistical data within the government. As a result of the project,
many government departments are seeking support for a similar system for their work. ZIMSTAT
staff felt that the logical framework indicators were appropriate and of sufficient quality to assess
project performance.
Conclusion
For many of those consulted, the project was highly relevant to the needs of their respective
country. All the NSOs were handling a lot of data, and the support the project provided was
instrumental in demonstrating that they could improve the way they handled and processed their
datasets.
3.2 EFFECTIVENESS
To measure the DA project’s effectiveness, the MTE focused on determining project targets at
proposal level and how and whether they transformed into expected results. Further, the evaluation
looked at whether the project has enhanced the capacity for mobile data collection at NSO and
TRI levels, as well as the generation of knowledge to be shared across the network of NSOs and
TRIs.
Cameroon
In Cameroon, the DA project was able to strengthen NSI’s existing capacity in MTBDC as it has
been using this technology since 2010. At the launch of the project, the NSI had already carried
out four statistical operations using MTBDC methods, but none of them had used tablets to collect
price data. The project conducted a training workshop from 29 July to 1 August 2015, to build
NSI’s capacities in the use of tablets. It also enabled the NSI to take ownership of the development
of data collection applications using tablets, and to experiment with the transfer of data directly
DA Project MTE
Report 11
onto a server. Under the project, the NSI acquired equipment (i.e. 60 electronic tablets with wallets;
10 power-banks; six laptop computers, one server, one 3-kva inverter, and Windows 2008 business
software) for 200 posts.
Ethiopia
In Ethiopia, the implementation tools and mechanisms applied were found to be very appropriate
and helpful at ensuring the effectiveness of the DA project. This included, importantly, the NSI
arrangement mechanisms that cover the project advisory board (consisting of ESA board chairman
and ESA president, head of Addis Ababa University statistics department, and two deputy director
generals of the CSA) and a technical working group of five from the CSA and ESA.
The CSA and ESA respondents also cited the other valuable implementation tools and mechanisms
that were critical to project performance. These included the initial project design and planning
approach, strategies, the agreement letters and memorandum of association signed between and
among implementing partners, as well as the logical framework. Respondents also mentioned as
critical the project work plan, its timeline, and budgeting; fund transfers (85%) from the ECA to
CSA and then ESA; expertise/human resources staffing and management arrangements;
stakeholder participation; and the monitoring and reporting system put in place to enable better
follow-up among implementing partners.
The CSA, ESA, and the enumerators’ response unanimously shows that in Ethiopia, the project
attained its intended targets (e.g. national and regional workshops and trainings, software
development, and field testing). MTBDC systems were used on retail and producer prices in North
Ethiopia (Tigray region), and internal generation of a mobile price collection report that meets
CSA’s statistical data quality. A lag in procuring the tablets caused the CSA and ESA to borrow
some from the government’s Agricultural Transformation Agency. The respondents also noted
that the project enhanced the capacity for MTBDC at CSA and ESA levels. During the project’s
mobile price data collection in Ethiopia’s Tigray region, the field assistants who were deployed
with the enumerators experienced first-hand how useful both personal data assistants (PDAs) and
mobile phones are in collecting data, especially compared with paper-based methods.
The Gambia
The development of the software by the TRI in The Gambia was timely, and enumerators were
trained for five days on how to use and operate it. After the training, pre-testing was conducted at
the field; the outcome was very satisfactory. Officers of the GBoS, enumerators, and system
developers all rated the performance of the software as very good and described it as user-friendly.
The Android application developed by the TRI is used primarily to collect and tabulate data. The
GBoS uses the application to collect both the CPI and producer price index in conjunction with an
SQL database for analysis. But the Android application installed in a tablet can also be used as a
sole programme for processing the collected information if the cloud or server functionalities are
not available for any given period. For example, GBoS enumerators can collect market price
DA Project MTE
Report 12
indices using handheld tablet devices, and store the information with both the GPS and time stamp
of where the data are collected. The information can be further synchronised to the server in the
GBoS cloud platform when network connectivity is available.
Kenya
In Kenya, the KNBS, in partnership with SON-UoN, have been able to develop and pilot a geo-
enabled mobile data collection system for CPI data collection (at present it is not being
implemented). Drawing upon the DA project methodology, the KNBS is using MTBDC to conduct
the Kenya Integrated Household Budget Survey (KIHBS) and the Micro and Small Medium
Enterprises (MSME) survey (completed). It is worth noting that even though the KIHBS and
MSME are independent surveys, their use of MTBDC methods was informed by the DA CPI pilot
project. Overall, Kenyan respondents noted an improvement in the capacity of the KNBS and
SOM-UoN to use MTBDC tools because of their participation in the project. They observed that
the project also enhance the role of the KNBS to provide national statistics. Development of the
software and the collection of data benefit students at the SOM-UoN, because an agreement with
the KNBS permits data collected for CPI and other surveys to be used as training materials by the
university.
Tunisia
In Tunisia, the project exceeded its objective to strengthen national capacity to develop geo-active
data collection systems using mobile technologies). The INS now has the resources to improve the
software application developed through the project, as well as the capability to develop other
applications for the variety of surveys carried out by the NSO. These resources facilitated the
acquisition of new skills within the project, which today develops systems to disseminate surveys,
as well as the implementation of a private cloud that can potentially spread these technologies
within Tunisia and abroad. Since the beginning of January 2016, 100% of the CPI data is collected
solely through tablets, and the application is now used by all enumerators in Tunisia.
Zimbabwe
As noted earlier, the DA project was implemented in 10 provinces in Zimbabwe; financial
constraints prevented it from being rolled out to the other 7. The main objective of project was to
recommend the use of mobile devices in conducting the CPS and other surveys carried out by the
NSO. This was achieved, as the project not only managed to show the utility of tablets, but also
built the capacity of NSO staff and increased their efficiency to process the collected data in a
timely manner. Some branches of the government, such as the Ministry of Lands and Rural
Resettlement and the Ministry of Health and Child Care (MoHCC), have started to rely on the
expertise of ZIMSTAT in collecting data using mobile devices after the success of the project. The
ministries approached ZIMSTAT about conducting their own surveys using the electronic data
collection system that was piloted and established during the project, for several reasons.
DA Project MTE
Report 13
ZIMSTAT has the necessary hardware (tablets); the technical expertise to develop data collection
software; and the capacity to design the studies, manage the data collection process, and provide
technical backstopping. The MoHCC has already been carrying out a Malaria Indicator Survey,
and the Ministry of Lands and Rural Resettlement has proposed to do a land audit which is waiting
to be launched. And although the present surveys the Agency is doing for other line ministries are
one-off, the MoHCC has expressed interest in a long-term partnership because it does conduct
different surveys at regular intervals. At present, ZIMSTAT can do more to conduct its own
surveys using the electronic system. But the Agency cannot purchase the technology it needs to
carry out national surveys electronically, as the government is unable to finance the technology.
Conclusion
Across all six countries, in terms of reporting, the ECA coordination team had developed an M&E
mechanism to guide the implementation of the DA project. As a result, there is evidence to support
the rate of implementation. Reporting can range from essential short-/midterm reports (e.g.
national situation analysis, national workshop reports, hardware procurement, etc.) to ECA focal
points reporting to DA project headquarters every 1–2 months on project status in the countries
according to the work plan. From experience, however, such reports mostly focus on measuring
the process of project implementation, with emphasis on upward financial accountability.
Admittedly, this monitoring of project activities is an important management function, and the
information is certainly useful in attributing impact to a given intervention. Yet such monitoring
data rarely reveal much about a project’s real impact.
In all but one country, the NSOs were able to develop and pilot a geo-enabled mobile data
collection system for the CPS, successfully enhancing the reliability of the data as they can track
the authenticity of the data collected. But since it was a pilot project with a limited budget, the
NSOs were not always able to test the software at national level. Under the effectiveness criteria,
then, the project is deemed satisfactory because it achieved most of its intended results by the end
of its first phase.
3.3 EFFICIENCY
Under the efficiency component, the MTE sought to determine the extent to which the available
resources were transformed into results. Key evaluation aspects included:
The value added by the project to the role of NSOs and TRIs
The extent to which the resources allocated enabled the project to achieve results in terms
of quantity, quality, and timeliness
User-friendliness of the software developed compared with other software currently being
used in data collection and reporting
DA Project MTE
Report 14
The extent to which management, decision-making, and relationship structures of the
project support the successful implementation of the project
The level of use of the software, with an explanation of the key drivers and the extent to
which the software is able to integrate geo-referenced information.
Cameroon
For Cameroon, the satisfaction survey was carried out by the NSI amongst 23 users about the
application developed by the DA project. The survey revealed that 82.6% had never used an
application before the project, so this was their first experience. Some 73.9% of the users stated
that they were satisfied with the application. In terms of difficulty, 17.4% of respondents
highlighted problems with the application throughout the data collection phase; 56.5%
encountered malfunctioning of the application at the beginning of data collection; and 26.1%
experienced no difficulties using the application. That nearly 40% of the respondents had never
used a tablet before could explain some of the difficulties they encountered. The level of use of
this application for surveys in Cameroon is still relatively low, however, and it needs to be made
more user-friendly to facilitate its dissemination. Many surveys are still carried out on paper, and
there are financial and logistical constraints in the purchase of tablets.
Ethiopia
In Ethiopia, the DA project’s use of MTBDC significantly reduced time across all major activity
chains: data collection, transfer, coding, analysis, and reporting. Respondents reported that the
software developed was very user-friendly and the devices easy to use, especially compared with
other software currently being used to collect and report data. Mobile devices also greatly reduced
the logistical arrangements from field sites to branch offices in the different regions. The tablets
prompt enumerators to save data onto tablets, and periodically to upload and synchronize daily
collected data onto the server at the CSA. Consequently, collected data do not need to be emailed
to CSA’s head office, and the search for specific items for data collection is easier. Tremendous
savings in time and resources are realised as MTBDC technologies replace paper-based
questionnaires, eliminating or streamlining many of the laborious and costly steps associated with
manual systems. The effect of such time reductions has broader implications for the CSA as it
collects, analyses, and reports other data sought by national development, or to expand the scope
and coverage of specific statistical data surveys. The CSA is now well positioned to carry out in-
depth data collection on price, and can expand to other types of demand-driven surveys/ censuses.
It can provide highly relevant, well-analysed, and informative data to the public/private sectors
and regional and international institutions. Cumulative reductions in time to collect and process
data, and parallel increases in the efficiency and cost-effectiveness of processing and sharing the
data, enable the scope and coverage of specific statistical data surveys to expand, not only for the
CSA but also for other stakeholders who collect data for their organisational activities.
MTBDC methods, along with a PDAs/CAPI (computer-assisted personnel interview) system,
helped to reduce the time and increase the quality for mobile price data collection, analysis, and
DA Project MTE
Report 15
reporting. Field testing of the software, and the feedback received, was quite helpful in improving
the user-friendliness of the software (it has both English and Amharic features). The enumerators
identified additional significant increases in the volume of data collected in a relatively short time
(a few days). They remarked how they were able to imagine that mobile technologies, if
mainstreamed, could significantly transform data quality, quantity, and timeliness in order to
produce and disseminate statistically rich publications to different users. These project
interventions, along with critical capacity and knowledge created and utilised in MTBDC systems
among implementing partners and stakeholders, are helping to increase data quality, security, and
reliability.
In all instances, the software developed was able to fully integrate geo-referenced information.
Initial problems faced by enumerators in taking GPS coordinates after entering the market were
quickly resolved. In this regard, enumerators used both the external Garmin device and the GPS
incorporated into the software in the tablet device they use for mobile data collection.
The Gambia
In The Gambia, because all the project activities were implemented within four months, targets
were met far sooner than anticipated. During implementation, TRI’s collaboration with the GBoS
allowed for knowledge-sharing and the smooth transition of project development phases. All key
deliverables and milestones have been successfully completed on time and within budget. While
the software uses a simple graphical interface, the Android application also contains a
sophisticated programming language that can be used to create highly customized applications.
The application can also use simple quality control checks, and advanced users can use the
enhanced features using SQL query. The programme’s user manual also contains rich
documentation describing the language, the Android features, and the step-by-step user functions.
Kenya
For Kenya, all respondents considered the main areas where the DA project added value were data
collection, transmission, and validation. In the short time that the KNBS has used MTBDC for
CPI, several advantages have been noted (e.g. integration of specification aspects in the data
collection software and geo-referencing, minimising errors in calculations; real-time transmission
of data; and elimination of data entry, thus saving time). CPI data collection using manual systems
used to take about three weeks after the end of the month before compilation and reporting. When
MTBDC systems are fully implemented, this process should be reduced by up to one week.
KNBS staff noted that the project added value to its provision of national statistics. Compared with
other software currently being used to collect and report data, E-Survey CPI software is user-
friendly, efficient, and does not need third-party software to operate. (It was customised from
software already developed by the KNBS.) It has a user manual detailing all the steps, and each
interface has captions properly labelled with clear instructions.
The KNBS and SOM-UoN tried different measures to ensure cost-effectiveness during the
project’s implementation. As targeted, 40 Techno tablets were procured and installed with the
DA Project MTE
Report 16
software. And although a Samsung tablet, when compared with a Techno device, proved more
user-friendly, it is far more expensive ($600) than a Techno’s tablet ($300). Piloting was carried
out over three months, and the issues identified were shared with the software developer during
the training and review workshop. According to the KNBS, the money allocated was sufficient
only for the data collection component and not for compilation and analysis, which are equally
important. As a result, during piloting, only relevant personnel were involved to ensure continuity
and achieve more for less cost.
The E-Survey CPI software is not being fully utilised at present, likely because the trained KNBS
CPI enumerators are few and overwhelmed by the various data collection exercises. As such, the
KNBS sorely needs capacity in this respect, coupled with continuous capacity building in order to
continue the project objectives. To improve its quality, the software will be continuously modified,
tested, and reviewed for 12 months, after August 2016. As noted during the evaluation, Kenya’s
internet connectivity is stable, and geo-referencing is able to confirm whether actual observations
were made. Respondents noted that geo-referencing, coupled with a record of actual time periods
when prices are observed, is very instrumental in instilling professionalism in retail price
collection. The E-Survey CPI software geo-referencing capability was appreciated.
Tunisia
The project achieved the first objective which allowed the NSO to effectively start data collection
with mobile technologies for the CPI survey. The soon-to-come employment survey will also be
done with the tablets. However the other objective (development of a partnership between the
NSO and TRI) was not achieved. This partnership was however important because it was meant
to support the development of skills and expertise within the TRI so that the development of mobile
technologies for data collection becomes sustainable either through the adaptation of these
technologies for other surveys to be done by the NSO or the replication of the tool by other actors.
Part of this has to do with the fact that the start of the project was delayed due to administrative
issues on ECA side. The budget for the project in Tunisia was 212,518.20 USD. This budget was
enough to build the capacity of the NSO and TRI to develop the tool that was used in the CPI
survey. To date, all CPI surveys are using the mobile data collection platform which is said to be
very user-friendly and has helped the NSO to reduce the time to collect and report price
information by 50% of the time it needed before the adoption of this system.
Zimbabwe
Zimbabwe was a singular case, given the issues of deregistration of the TRI and delays in starting
the alternative project (i.e. CPS) that affected the entire project’s timeliness and cost-effectiveness.
Resources were spent on the earlier project, which had to be abandoned. But there are some
positives that were taken from the new project. For example, ZIMSTAT was able to receive
support in linking their server to mobile networks, which is an important issue when conducting
data collection using mobile devices.
DA Project MTE
Report 17
Value for money was evident in that the project collected data more efficiently by eliminating data
entry processes, the transportation of papers, and timely submission after data collection. The
initial cost of setting up the systems is high; but once systems are in place it is cheaper. Although
the project was only piloted in three provinces, it has put ZIMSTAT in the limelight as it
demonstrated that the Agency can keep up with the times and lead the paperless revolution in the
country’s collection of vital statistics. Other departments have since been using the same software
on different projects.
The software used for the DA project was user-friendly and clearly labelled. The screen transition
was very smooth, although some enumerators complained about some of the machines freezing
whilst in use. Even though there are too many categories as required in this type of survey, CSPro
was appropriate for the pilot project. ZIMSTAT needed to invest more time in training the
enumerators, but this was limited by the availability of funds. In terms of cost effectiveness, the
NSO bought Android tablets that were fit for the job and not fancy. The alternative, CPI project
managed to achieve its set targets in a short time (Aug.–Dec. 2015), adding value for money as the
project managed to promote the use of MTBDC in surveys. Other ministries, too, have come to
appreciate the role of the NSO due to this short-term project. The collection and quality of data
would have benefitted even more had the project been implemented over a longer period.
Nonetheless, the project was cost-effective, considering the time and resources used to implement
it. In the end, the NSO managed to develop and implement a geo-enabled mobile data collection
system (including software and hand-held devices, such as techno tablets) for collecting CPI data.
Conclusion
In general, the project was highly efficient across the six countries. The project was highly valued
by the interviewees because it facilitated the processes used by the NSO to collect, process and
report on price information. The project built the capacity of both the TRIs and the NSOs, which
was a key achievement. The software seemed to have integrated geo-referencing information and
were said to be very user-friendly and are actually being used to collect and report on price
information.
3.4 PROGRESS TOWARDS IMPACT
The fourth dimension of the evaluation looked at the extent to which the results achieved through
the DA project helped to reduce the time needed to collect and report statistics across the six
countries and improve their quality in the NSOs. The MTE also tried to assess any progress
towards impact that the project may yield in the foreseeable future.
Cameroon
DA Project MTE
Report 18
For Cameroon, the DA project set up more efficient complementary modules for real-time central
processing. The project has also made data collected available to the NSI on a daily basis, which
limits the risk of loss of data and reduces delays in data processing. Furthermore, the DA project
has reduced data collection times, and set up a platform for collaboration with the École Nationale
Supérieure Polytechnique for the management and transfer of data. Nonetheless, the mode of
collaboration between these two institutions needs to be improved. The project has improved real-
time quality control of data collected in the field. One improvement is the fact that data are
transferred directly to the server.
Ethiopia
In addition, other important outcomes of the DA project for Ethiopia are the partnership,
knowledge, and capacity created at national level; and the institutional arrangement and
partnerships created and strengthened during the project period, which will serve as a reliable basis
for magnification and scale-up of the project’s approaches and systems. The exemplary partnership
between the CSA and ESA, beside boosting both institutions’ capacity, knowledge, and experience
in MTBDC project management and implementation, has created the opportunity for the CSA to
identify and access senior academicians (such as statisticians) for their professional consultancy
and advisory services. The favourable enabling environment created as a result of the project
enhanced capacity of the CSA on MTBDC, will enable the CSA to better discharge its legally set
duties and responsibilities (as enshrined in its establishment proclamation), and spearhead/lead the
National Statistical System towards utilisation of MTBDC systems. Yet, subsequent to the project,
an overriding outcome has been the inspiration created by the project on mobile data collection on
the activities of the CSA, ESA, and potentially other stakeholder institutions.
The introduction and use of MTBDC in other institutions in the country are expected to be another
outcome of the project. (The respondents did indicate that some international and a few national
institutions do use smart phones/tablets for some of their data collection needs.) Moreover, with
the increased capacity of the CSA to use MTBDC systems also comes its capacity to support and
guide mobile data collection activities nationally. This will result in multiplier effects of the project
to be effectively utilised.
All respondents from the CSA, ESA, and the enumerators expressed their confidence that the
project has improved the quality of data in the NSO. This is because the MTBDC activity has
significantly reduced human errors that could occur along the data collection–entry–editing
process. It has also enabled more data to be collected reliably and in a very short time than actually
needed for doing the same work using paper-based or PDAs methods. (Noteworthy, too, is the
significant reduction in the number of enumerators, supervisors, budget, logistics, transport, paper,
etc. that could fairly be expected from adopting MTBDC systems). Comparative/experimental
research on quality, quantity, and timeliness of the different methods of data collection (e.g.
MTBDC, PDAs, and CAPI, paper-based) has yet to be done by either the CSA or ESA.
Representatives of both institutions commented that such research be done first as soon as possible
before any mobile data collection activity.
DA Project MTE
Report 19
The Gambia
The results of project implementation have been positive despite some challenges. The TRI noted
a number of indicators of achievement. For example, MTBDC ensures that prices are verified as
they are entered directly from the collection point and transferred to the server, reducing both the
use of manual data collection and the time it takes. Besides network connectivity issues
experienced in some parts of the country, the data collection survey has been conducted
successfully across the country. All supervisors of the respective teams were able to synchronise
price data with the tablet devices to the central server from the selected markets nationwide.
The use of MTBDC ensures that data are readily available upon collection and up-load. There were
no problems or data entry errors; and access to content by staff of the ‘GBoS Price Statistics Unit’
has been problem free. GBoS research analysts very much appreciated the analytical tool they used
for its ability to generate the index with a click of a button. GBoS price statisticians thoroughly
observed the process and have been satisfied with the results.
Kenya
In Kenya, through MTBDC, the quality of data has improved tremendously, not only for CPI but
also for the KIHBS. Hence, the KNBS, as the sole producer of national statistics, will not have to
use additional resources to correct data; improved quality will enhance stakeholder confidence and
support. Overall, respondents maintained that there is increased adoption of MTBDC methods
since DA support in Kenya. The SOM-UoN, for example, has been involved in other research/data
collection exercises using MTBDC, such as a recent survey in partnership with the Kenya
Meteorological Department. Currently, the KNBS is collecting data for the KIHBS and the Micro
and Small Medium Enterprises (MSME) Survey (completed). Both surveys had a component on
data collection using mobile technology. However, even though the KIHBS and MSME are
independent surveys, their use of MTBDC techniques was informed by the DA CPI pilot project.
Tunisia
For Tunisia, the data collected by enumerators in the field are uploaded daily by email to the
internet platform, compared with 10 days or so by fax. As a result, the NIS is now able to reduce
the time it takes to publish its monthly report, from M+4 to M+1 (1 day after the end of the month
instead of 4 days after the end of the month). The quality of data collected has greatly improved,
although officials were unable to quantify this. This improvement is due to features of the
application that bring to the attention of the enumerator whenever there is a significant price
difference compared with the previous month. The amount of missing data also is reduced, as the
platform manager can track near real-time data collection, and can identify points of sale that have
not yet been visited and thus have personalised follow-up with each enumerator in each
governorate of the country. There is now the possibility to identify products by photo and make
DA Project MTE
Report 20
audio recordings in order to obtain information in situations where it is not possible to use the
tablet (e.g. in case a vendor refuses, or because of risk of theft).
Zimbabwe
In Zimbabwe, the CPS is a very important exercise for the government. It is carried out by
ZIMSTAT as part of its mandate, and benefitted from the piloting of mobile technology during its
implementation. The pilot project demonstrated that mobile technology can enhance the data
collection process, data quality, and the timely publishing of results from the survey. Some of the
impacts of the pilot project include the following:
Reduction of costs currently being spent on printing and transporting of questionnaires
Reduction of costs and time for data entry
Real-time data collection and transmission
Timely production of statistics
Improvement in data quality
Overall reduction in the cost of producing statistics
Saving of storage space as questionnaires need to be stored
Solved the problem of enumerators running out of questionnaire
Improved data quality as they were able to prove the authenticity of data collected as the
tablets collected geo-referenced data
Lowered resistance within ZIMSTAT to move from paper to mobile system, and afforded
people within the NSO to see the usefulness of the system and helped to pre-test both the
software and tablets
Helped build the capacity of ZIMSTAT to deploy mobile devices in data collection, which
are now a viable option for doing business
Capacity building of ZIMSTAT staff in terms of training on the use of CSPro on Android
devices using the tablets in data collection and the purchase of the server
Positioned ZIMSTAT in the limelight because other line ministries appreciate its role in
collecting statistics.
Overall, the project has demonstrated the capacity of ZIMSTAT to collect statistics using mobile
technology in Zimbabwe. This has been shown by line ministries (e.g. Ministry of Health)
approaching the NSO to design electronic data collection systems for their own surveys. The only
missing link is funding for the procurement of more tablets for use in large national surveys.
Conclusion
The DA project, although piloted in six countries, had several impacts at country level. The most
important of those impacts are: the reduced data entry errors, the reduced workload, the real time
DA Project MTE
Report 21
data collection and transmission and the easy way to detect anomalies. Most importantly the time
needed to generate critical information related to prices was cut by about 20 to 40% depending on
the country. The software made it possible to detect at an early stage any error, typo that happen
during the process of data collection, entry and processing which was found to be instrumental for
its widespread adoption by key actors.
3.5 SUSTAINABILITY
The sustainability component dealt with the critical measures that needed to be put in place to
ascertain whether the results of the DA project would continue after its completion. The evaluation
looked at whether the TRIs and NSOs have the capacity to continue to use and improve the
software. It considered whether there are appropriate working arrangements and plans in order to
continue the project’s critical activities beyond its lifetime. The MTE also examined whether a
financial plan had been set up to support the implementation of those activities, as well as whether
the project led to policy reforms at national or regional level.
Cameroon
For Cameroon, this application was developed with CSPro, a software that the NSI has used for
more than 20 years. Moreover, the Institute played a major role in the development of this
application. As part of its partnership with the World Bank for its social safety net projects, the
NSI uses MTBDC systems. Several operations have already been carried out, including the Proxy
Mean Test , the field test, data collection and the final evaluation for the social safety net project.
But there is no clear funding plan to finance the planned activities, however, and the NSI is
working on action plans within available funding models. It plans to carry out production of a final
household CPI based on data collected through mobile technologies.
Ethiopia
In Ethiopia, there are emerging trends in the CSA in using the knowledge and capacity created
through the DA project on mobile data collection towards sustainability. Importantly, the major
inspiration and immediate impact of the project will be on the upcoming National Population and
Housing Census of Ethiopia in 2017. In this respect, respondents indicated that questionnaires for
collecting Census data with mobile-based technologies have been developed using CSPro. At
present, the questionnaires are being field-tested in one of the regions in Ethiopia (Oromia region).
This Census is estimated to require about 140,000 enumerators, hence a similar number or more
of tablets are needed, including reserves. These tablets, or other resources (in-kind or financing),
are expected to be secured through cooperation with international partners, such as the United
Nations Population Fund.
Ultimately, the CSA is making major efforts to raise awareness on and promote the idea of mobile
data collection among government offices, and is finalising the Census tools and instruments. It is
working to secure the government’s endorsement to train mobile data collection enumerators,
DA Project MTE
Report 22
supervisors, and others, with goal of launching the first-ever use of MTBDC to gather national
Census data in 2017. The CSA is currently carrying out preparatory activities and experimentation
on introducing MTBDC methods for industrial producer prices and quarterly medium and large
industries surveys. Through a network created under the ECA and ESA, the CSA hopes to
coordinate and share experiences on the use of MTBDC, with other African countries or sub-
regional and regional institutions. The partnership between the CSA and the US Census Bureau
will also continue to be key to further improvement and utilisation of the software for MTBDC
systems.
Since the DA project was completed, there has been no major intervention or plan to engage in
further MTBDC activities, or in committing funds and financial resources for such an intervention.
The Gambia
The application developed by the DA project was fully embraced by project users eager to test the
capability of a system, network, or process to handle a growing amount of work, or its potential to
be enlarged in order to accommodate growth. Such high regard should not be underestimated, as
it is a reliable predictor for early adoption of MTBDC methods and the commitment to build and
strengthen human capacity for their continued use. After application development and
maintenance, new features might be requested or bugs might be detected. The ability to quickly
add new features, fix bugs, and deploy changes is also very important.
To determine the best priced index, it is imperative that features that are strictly essential for any
price data collection be considered for deployment. Given the relative strengths and weaknesses
identified in digital and manual platforms, decisions as to which platform to use will depend solely
on user preference and data collection requirements. MTBDC systems save costs and increase
speed, accuracy, functionality, and security. In this sense, they are the future.
Kenya
To sustain DA project activities in Kenya, 100% of respondents thought that it was crucial to
continue to encourage buy-in by different government officials and sharpen the focus on
developing capacity and skills within the KNBS, SOM-UoN, and other research institutions.
Respondents underscored the importance of data validity and reliability, and the benefits that
accrue, stressing the need for continued operationalisation of the E-Survey CPI software. They
highlighted the need to inculcate the culture of knowledge generation and sharing for the benefit
of the entire country. During the evaluation, it was established that the KNBS has re-engaged an
earlier consultant (see Section 3.3) to incorporate the current software issues identified and to train
staff for internal support and continuity. Other sustainability measures include plans by the KNBS
and SOM-UoN to finalise the implementation report by the end of June 2016. This would enable
them to receive the remaining 15% of the project’s funds meant for procuring hardware for safe
storage of CPI data. The KNBS also plans to recruit and train more enumerators to bridge the
current capacity gaps. As noted during the evaluation, most of the enumerators are aging (50 years
DA Project MTE
Report 23
and older); hence the pressing need for continuity. Additionally, the KIHBS and MSME’ 400
Samsung tablets will revert to the KNBS once the surveys are completed.
As to funding, the KNBS, through the Medium Term Expenditure Framework, has earmarked
resources to continue project activities and its core functions. Overall, the pilot project has not
resulted in any policy reforms at national or regional level. Regionally, the KNBS and other NSOs
have only harmonised concepts and compilation methods—not data collection techniques. The
main successes have been the project’s contribution to raising awareness about mobile data
collection in Kenya. By making strategic use of this baseline information and activity appraisals,
the KNBS and SOM-UoN should be able to develop targeted strategies for implementation to
further the ECA’s aspirations. Interviews with respondents showed that the project’s approach was
widely applicable, and that there was a potential for wide learning across/between countries and
organisations, locally and internationally.
Tunisia
For Tunisia, the sustainability of the project within the NSO is guaranteed to the extent that
the use of tablets is now part of its standard working procedure. The CPI has since abandoned
paper-based data collection, including the 2017 Employment survey, as the NSO plans to gradually
introduce digital data collection methods for all surveys. The NSO is now positioned to further
develop the application as shown by the new development currently on-going.The TRI, however,
did not develop capacities to use or develop the application because of its limited involvement in
the project.
Zimbabwe
ZIMSTAT in Zimbabwe now has the capacity to continue with the project and use MBTDC
systems. A huge bonus too is that the Agency can develop its own needed software, eliminating
the high cost of hiring outside software developers. Although the DA project was initiated,
implemented, and completed in just 5 months (Aug.–Dec. 2015), ZIMSTAT has shown that it has
the capacity to continue using the skills gained from the project by setting up other governmental
line ministries (i.e. MoHCC, Ministry of Lands) to carry out their own surveys. There has been
buy-in within ZIMSTAT as the use of MBTDC is now an option as part of its mandate. Initially,
there was resistance on whether migration from paper to the use of mobile devices was viable, but
the project managed to dispel those doubts. Overcoming these early misgivings is crucial to
consolidating the gains realised by the project. Moreover, collecting statistics on mobile devices
enhanced data validity and reliability by improving the quality of data and ensuring that CPS data
is geo-referenced.
And yet, despite the project’s many positives, the government cannot afford to purchase and
maintain new equipment, or to train enumerators in order to expand the process beyond the three
provinces where the tablets were piloted. At the moment, ZIMSTAT has not achieved the threshold
needed for collection of statistics at the national level with the same mobile devices, and has no
financial plan for acquiring more MTBDC equipment. But development partners have expressed
DA Project MTE
Report 24
interest in supporting the government, although nothing has been confirmed to date. An important
measure that needs to be in place to ensure the sustainability of the project is the allocation of
funds needed to maintain mobile devices/ servers, update/develop software, and train staff. (The
initial cost of purchasing mobile devices and servers, which is usually the most burdensome factor
for NSOs, was covered by the DA project.) No doubt, all of these measures that have direct
influences are important. Equally so are the indirect factors, such as commitment at senior level in
the organisation and awareness of the use of new MTBDC systems. These form the basis of support
for introducing direct measures and fostering innovation.
Conclusion
At the time of the midterm evaluation, the TRIs and NSOs which have been involved in the
implementation of the project had the capacity to continue to use the software and generate useful
price-related information using mobile technologies. Critical capacity has been generated and is
available at country level, should similar project need to be replicated. No plan seemed to be put
in place for the project to continue critical activities beyond its lifetime though. Going forward it
will be necessary to think about that to maintain the project achievements and perpetuate its
impacts.
4. MAJOR ISSUES AND PROBLEMS AT THE COUNTRY LEVEL
Across all six countries, evidence shows that the implementation of project activities was
substantially complete, though some challenges remained unresolved.
For example, in Zimbabwe, apart from issues related to the deregistration of the TRI, there were
critical issues around the inability of the government to provide funds to support the expansion of
the project beyond the three provinces where it had been piloted. There were also issues relating
to inadequate time for testing of the system and training of the enumerators, but these were
resolved by close follow-up and technical support during the data collection phase.
In Kenya, project implementation was hampered by lack of expertise within the KNBS and SOM-
UoN in software development, which forced them to bring in external consultants to support the
process. ‘Teething’ problems were also noted as both institutions had never worked together prior
to the DA project. They are different entities altogether—SOM-UoN is a research institution, the
KNBS a semi-autonomous government agency—with entirely different modus operandi.
In Ethiopia and Tunisia, the enumerators at times could not receive the GPS coordinates easily.
Ultimately, they did overcome this by making sure that they first took proper GPS readings before
starting to collect their price data at marketplaces. Enumerators also faced a problem with network
connectivity when trying to upload their data to the servers.
In Tunisia, having the ESSAI be the designated TRI may indeed not have been the most
appropriate choice, as it had not been selected objectively. The project would have benefitted from
a more thorough assessment of the ESSAI and its existing capacities and limitations before the
project got started. This would have made a public or private school or university a better choice,
DA Project MTE
Report 25
especially one that specialised in information technology, particularly in mobile applications. To
meet ECA’s requirement, the NIS instead opted for the easiest solution: a TRI it already knew and
had collaborated with (e.g. the recruitment of ESSAI graduates, mentoring of students during final
year projects, etc.). The better alternative, one that would have enabled the two to develop a real
technological partnership, would have been to choose a TRI with real capabilities in MTBDC
systems.
In Cameroon, the process was less smooth. Implementation was impeded by the delay to receive
the funds from ECA at the start of the project, the lack of clarity on the characteristics of the
software to be developed by the TRI (ENSP) and the insufficient participation in the process of
key factors such as INS/prix, MINEPIA, and MINADER.
In the Gambia, several other problems came up: first, the tablets used in the survey were high-
status items in The Gambia, making them a theft risk. Keeping the tablets secured at day time and
hidden while walking to interviews was a priority for the enumerators. Second, safe data storage
depends on uploading the day’s interviews via the Internet, and where WiFi of GSM signals are
not available at the end of each day, the risks go up of a lost or damaged of the tablet compromising
the survey sample. Third, during the interviews, respondents often sat side-by-side with the
enumerators so they could see how the tablet worked. This could influence the answers.
Respondents tend to choose the last answer given if they are hearing the questions or the first
answer given if they are reading the question (the primacy effect).
5. LESSONS LEARNED
Several lessons learned have emerged from the evaluation of the DA project:
Pilot projects are necessary to demonstrate the utility of systems where there is resistance
to changing data collection systems from paper to digital.
Mobile data collection systems can both reduce the cost of producing statistics and improve
data quality.
Integration of specification aspects of data collection software such as geo-referencing
minimises bias and enhances the reliability and validity of data.
Building the capacity of NSOs is key to the financial sustainability of MTBDC surveys, as
shown by the ability of ZIMSTAT to develop its own software—a costly component when
outsourced.
The DA project has sparked a veritable technological revolution, including in the NSOs
approach to its work. It demonstrated that it is possible to collect data in a user-friendly
and much more rapid way, while also ensuring better quality data. Enumerators are more
motivated to carry out their work by raising their profile and recognition, and the reputation
of the NIS as a progressive institution that produces high-quality, reliable products is
DA Project MTE
Report 26
enhanced. The involvement of NSO staff in the project, either at management level or
through the training of enumerators, was essential to the project’s success.
Although NIS’s development of a tablet application was an undeniable success, the
cultivation of the partnership between the NSO and the TRI—in Tunisia particularly—was
key to guaranteeing the dissemination of mobile technologies and to ensuring that these
reach the largest possible number of organisations that carry out field surveys. It is
essential, then, that the two parties work to develop a real partnership for the benefit of the
NSO as well as for the whole country.
Proper collaboration between the NSOs and TRIs is beneficial in the long run as it
generates knowledge that can be shared across the network, resulting in improved validity
and reliability of country data. The TRI, however, needs to develop a track record in the
area that it is supporting to ensure that the mandate and skills gained through the project
can continue beyond the pilot phase.
It is important to clearly identify the characteristics of the software when partnering the
NSO with the TRI in order to be certain the final product includes all the features needed
to achieve its purpose.
In general such project are very relevant to the needs of the NSOs at country level hence
the high level of utilization. It is nonetheless important to both start the contracting and
administrative process well ahead of time to prevent delays and involve all the actors in
the design process to fully benefit from their participation.
6. RECOMMENDATIONS
At the end of this evaluation, the following recommendations are made to help sort out the
problems encountered by the DA project and facilitate design and implementation of similar
projects in the future:
It is important for ECA’s future projects that the NSO receives assistance in the TRI
selection methodology, through the provision of a TRI assessment grid prior to signing the
agreement between the ECA and NSO. At the same time, the agreement must clearly define
the TRI as well as the TRI’s expected activities and its budget for those activities.
Otherwise, there is a risk that the NSO will work alone in order to retain the entire budget.
The ECA should take into account the time it needs to complete its administrative processes
before the funds can be accessed by their partners. A proper planning will reduce delays
suffered by similar projects in the future.
A feature of the CSPro software is that it can capture the geo-code of the enumeration area
in combination with outlet codes. This can provide quite detailed geographic information
that may be linked to/analysed with price information. In addition, the software has a
function for GPS reading (longitude/latitude), although this largely depends on network
availability. It would appear from the interviews that this geo-information was not
DA Project MTE
Report 27
thoroughly explored across all the six countries. But having this functionality would enable
the NSO not only to conduct spatial analysis of the statistical data it collects with mobile
devices, but also to link these data with other socio-economic information that is beyond
the scope of DA project.
The choice of a TRI and the theme of the DA project needs to be consultative and
exhaustive in order to ensure project success.
Outsourcing software development work is expensive. These pilot projects should
therefore build the capacity of the NSOs’ own staff in order to set up mobile data collection
systems themselves.
The NSOs should carry out further pilot testing with large geographic coverage of urban
and rural areas of their countries. This would promote better appreciation for the efficiency
and effectiveness of the MTBDC system for price data collection, including the presence
or absence of power, internet connectivity, and other essential factors.
DA Project MTE Report A-
1
ANNEXES
1. DATA COLLECTION TOOLS
Interview guide with NSO and TRI staff
Date:
Country:
Institution:
Respondent:
Title of the respondent:
Contact of the respondent:
1. Are the project approach and objectives relevant to the achievement of expected outcomes, given
national and regional priorities?
2. What process did you follow to develop the software?
3. Did you receive all the support you needed to finalise the development of your software? If not
what was lacking?
4. How many times did you get to use the software since it was finalised?
5. How did you find it?
6. If you have not used it since its finalisation, what are the reasons behind?
7. Was the logical framework indicators appropriate and of sufficient quality to assess performance?
8. Were the project activities at country level well designed and do they fit the framework of national
and regional strategies for development in the area of intervention?
9. Has a ‘gender approach’ been considered in the project implementation?
10. Have the objectives remained valid and relevant throughout implementation?
11. What are the implementation tools and mechanisms? Are they appropriate for the smooth and
timely implementation of key outputs?
12. Has the DA project attained its intended targets as set forth in the project proposal?
13. Has the DA enhanced the capacity of mobile data collection at NSO and TRI levels?
14. To what extent did the DA project generate knowledge that has been shared across the network of
NSOs and TRIs?
15. To what extent did the DA project achieve its expected results, in terms of quantity, quality, and
timeliness at country level?
DA Project MTE Report A-
2
16. How user-friendly is the software developed during the DA project as opposed to the other software
currently being used in data collection and reporting?
17. To what extent is the software able to integrate geo-referenced information?
18. To what extent do the results achieved through the DA project contribute to increasing the quality
and reducing the time needed to collect and report country level statistics?
19. To what extent has the DA project improved the quality of data in NSOs?
20. What is the value added of the DA project with regard to the role of the NSOs?
21. What are the critical measures that need to be put in place to ascertain that the results of the DA
project continue after its completion?
22. Do the NSO and TRIs have the capacity to continue to use and improve the software?
23. Have appropriate working arrangements and plans been put in place to continue critical activities
of the project beyond its lifetime?
24. Has a financial plan been set up to support the implementation of those activities is there clear
commitment to fund it?
Interview guide with DA project focal persons in ECA
Date:
Department:
Respondent:
Title of the respondent:
Contact of the respondent:
1. Are the project approach and objectives relevant to the achievement of expected outcomes, given
national and regional priorities?
2. How did you identify the beneficiary countries? And the beneficiary institutions at country level?
3. Was gender a focus area during the development of the DA project? If yes what was done to ensure
that gender was mainstreamed? If not why?
4. Were the key performance indicators in the logical framework appropriate and of sufficient quality
to assess performance?
5. Were the project activities at country level well designed and do they fit the framework of national
and regional strategies and priorities for development in the area of intervention?
6. Have the objectives remained valid and relevant throughout implementation?
7. What are the implementation tools and mechanisms? Are they appropriate for the smooth and
timely implementation of key outputs?
DA Project MTE Report A-
3
8. Has the DA project attained its intended targets as set forth in the project proposal?
9. To what extent did the DA project generate knowledge that has been shared across the network of
NSOs and TRIs?
10. To what extent did the DA project achieve its expected results, in terms of quantity, quality, and
timeliness at country level?
11. How can the level of utilisation be explained? What are the key drivers of that utilisation?
12. How did you undertake monitoring and reporting regarding the project activities in your country
of responsibility?
13. What were the main issues/challenges that you had to face during project implementation?
14. To what extent is the software able to integrate geo-referenced information?
15. What are the critical measures that need to be put in place to ascertain that the results of the DA
project continue after its completion?
16. Do the NSO and TRIs have the capacity to continue to use and improve the software?
17. Have appropriate working arrangements and plans been put in place to continue critical activities
of the project beyond its lifetime?
18. Has a financial plan been set up to support the implementation of those activities?
Interview questions with enumerators
Date:
Country:
Institution:
Respondent:
Title of the respondent:
Contact of the respondent:
1. How many times did you use the software that was developed with the DA project to collect data?
2. How was the training that you had to take to be able to use the software?
3. After your experience using the software, what are the top three issues/challenges that you had with
the software?
4. How were they addressed?
5. Has the DA enhanced the capacity of mobile data collection at NSO and TRI levels?
6. How user-friendly is the software developed during the DA project as opposed to the other software
currently being used in data collection and reporting?
DA Project MTE Report A-
4
7. What is the level of utilisation of the software at country level?
8. How can the level of utilisation be explained? What are the key drivers of that utilisation?
9. To what extent is the software able to integrate geo-referenced information?
10. To what extent do the results achieved through the DA project contribute to increasing the quality
and reducing the time needed to collect and report country level statistics?
11. To what extent has the DA project improved the quality of data in NSOs?
12. What is the value added of the DA project with regard to the role of the NSOs?
13. What are the critical measures that need to be put in place to ascertain that the results of the DA
project continue after its completion?
DA Project MTE Report A-
5
2. EVALUATION TIMELINE
The following timeline was proposed for the evaluation.
# Activity From To
1 Development of the evaluation inception report which will include
the methodology, tools, and procedures that will be used by the
national data collectors and lead evaluator
April 18th May 6th
2 Finalise the inception report based on the comments from the ECA May 8th May
17th
3 Skype teleconference with each of the national data collectors to
explain the methodology and the tools that will be used
May 18th May
21th
4 Data collection at the national level May 22th May
30th
5 Report writing at the national level May 30 June 5th
6 Interviews with additional stakeholders June 5th June
10th
Country reports received from national data collectors June 15th
7 Improve the reports at the national levels through discussions with
the national data collectors
June 15th June 18th
8 Summarise the data and write up the MTE report June 18th June 25th
9 Submit draft of the MTE report June 27th
10 Comments of the first draft by the ECA June 30
11 Revise the draft and submit a final MTE report by incorporating
comments received from the ECA
June 30th July 2nd
12 Submit final report July 5th
DA Project MTE Report A-
6
3. PEOPLE INTERVIEWED AND CONTACTS
S/No. Name Institution Email
Zimbabwe
1. Mr Rodgers Sango ZIMSTAT [email protected]
2. Mr Wish Chipiro ZIMSTAT [email protected]
3. Ms Esnat Mutumbure ZIMSTAT -
4. Mr Elias Fushai ZIMSTAT [email protected]
5. Mr Shepherd Mandi ZIMSTAT [email protected]
m
6. Mr Emmanuel Gwenzi ZIMSTAT [email protected]
7. Mr Aubrita Gweshe ZIMSTAT [email protected]
8. Mr Salisio Budakuvaka ZIMSTAT [email protected]
m
9. Mr Z. Mabire ZIMSTAT [email protected]
10. Ms Inkyung Choi ECA [email protected]
Kenya
11. Mr Cleophas Kiio KNBS [email protected]
12. Mr Robert Nderitu KNBS [email protected]
13. Mr Simon Gaitho KNBS [email protected]
14. Mr Evans Munene KNBS [email protected]
15. Prof. Moses Manene SOM-UoN [email protected]
16. Prof. Patrick Weke SOM-UoN [email protected]
17. Mr Ndubi Arun KNBS -
18. Mr Silas Mulwa KNBS -
Ethiopia
19. Mr Kifle Gebre CSA 251-911409817
20. Mr Abdulaziz Shiffa
Yimam
CSA/ESA 251-912199550
21. Mr Zelalem Destaw ESA 251-912035205
22. Mr Didimos Ayele CSA 251-911418341
23. Mr Ermyas Arega CSA/ESA 251-911303646
24. Mr Abel Tesfaye CSA 251-911070790
25. Mr Yabebal Ayalew Addis Ababa University 251-913559036
26. Mr Gezahegn Getahun Private consultant 251-921794541
27. Mr Bekalu Mehari CSA 251-911027207
28. Mr Taddesse Kasahun Addis Ababa University 251-911835082
29. Mr Temesgen Abera Addis Ababa University 251-921467143
Cameroon
30. Léandre Ngongang CEA [email protected]
31. Martin Mba INS [email protected]
32. Guy Ndeffo INS [email protected]
33. Eric Jazet Kengap INS [email protected]
DA Project MTE Report A-
7
34. Jean Alogo Samba INS [email protected]
35. Clément Eyem Georges INS [email protected]
36. Romain Tchakoute INS [email protected]
37. Thomas B. Bouetou École Nationale Supérieure
Polytechnique
38. Peggy Kuinze K. Alvine INS [email protected]
39. Brice-Muriel Tientcheu INS [email protected]
40. Joel Nkouagnou Calin INS [email protected]
41. Marthe Ngo Lihep Ministère de l’Élevage, des Pèches et
des Industries Animales
42. Jules Cheumetou Ministère de l’Élevage, des Pèches et
des Industries Animales
The Gambia
43. Mr Léandre Ngogang
Wandji
44. Mr Nyakassi M.B.
Sanyang
The GBoS (NSO) Tel: +(220) 9969821
45. Mr Ousman Dibba The GBoS (NSO) Tel: +220 9969981
46. Dr Momodou Jain University of The Gambia (TRI) [email protected]
47. Mbemba Hydra University of The Gambia (TRI) [email protected]
48. Pa Safiong Kebbeh University of The Gambia (TRI) [email protected]
49. Mr Molamin Fadia The GBoS (NSO) [email protected]
50. Mr Morro Sanyang The GBoS (NSO) [email protected]
51. Mr Sainey Jallow The GBoS (NSO) [email protected]
52. Mr Tumbulou Drammeh The GBoS (NSO) Tel:- +(220) 6575535
53. Mr Ebrima Keita The GBoS (NSO) [email protected]
Tunisia
54. Mme Mouna Zgoulli INS [email protected]
55. M. Néjib Khlifi INS [email protected]
56. M. Hatem Sedghiani INS [email protected]
57. Mme Samira Chihi INS
58. Mme Saloua Khemiri INS
59. Mme Wissem Sellam INS
60. Mme Rim Lahmandi
Ayed
ESSAI [email protected]
om
61. Ms Meriem Aït Ouyahia ECA [email protected]
DA Project MTE Report A-
8
DA Project MTE Report A-
9
4. EVALUATION TOR
Midterm Evaluation of the Development Account Project “Strengthening the Capacity of
African Countries to Use Mobile Technologies to Collect Data for Effective Policy and
Decision Making”
1. Background and Context
Project number 2920
Project title Strengthening the Capacity of African Countries to Use Mobile
Technologies to Collect Data for Effective Policy and Decision
Making (Phase I)
Duration 2013–2015
Location Inter-regional project with 6 pilot countries (Cameroon, Ethiopia,
The Gambia, Kenya, Tunisia, and Zimbabwe)
Executing agency
(implementing entities)
Economic Commission for Africa (ECA)
Department of Economic and Social Affairs (DESA)—United
Nations Statistics Division (UNSD)
Partner Organizations National Statistics Offices (NSOs) and Training and Research
Institutes (TRIs) in 6 pilot countries
Total approved budget US $1,845,000
Donors Development Account fund
The Government of Ireland for implementing a pilot project in
Ethiopia
Division/IDEP/SRO/Section African Centre for Statistics (ACS)/Data Technology Section
(DTS)
SRO - Central Africa
SRO – Eastern Africa
SRO – North Africa
SRO – Southern Africa
SRO – West Africa
Programme/Project
Manager
ACS/DTS
Statistics play a crucial role in the overall development policy making process of countries by providing
the evidence for setting objectives, targets and priorities. However, the statistics on which such evidence
would be based are not always available in a timely manner in many African countries and their reliability
is not always assured. Most African national statistical systems often use manual, paper-based data
collection methods whose lengthy process not only delays the production of data for decision making, but
also requires a lot of personnel for data collection, thereby exacerbating the financial constraints. Computer
Assisted Interviewing (CAI) methods are increasingly replacing pen-and-paper methods of survey data
DA Project MTE Report A-
10
collection but due to the low technology base, CAI methods are not yet as widely used in Africa as in other
parts of the world.
The Development Account (DA) project on “Strengthening the Capacity of African Countries to Use
Mobile Technologies to Collect Data for Effective Policy and Decision Making” (DA project hereafter)
was launched in 2013 with aim of enabling the National Statistical Offices (NSOs) to develop technological
bases for the mobile data collection in a partnership with Training and Research Institutes (TRIs) within
the country. Cameroon, Gambia, Kenya, Zimbabwe, Ethiopia and Tunisia were selected as pilot countries
where NSO and TRI of each country established the implementation program in a collaboration.
2. Disbursement History
Total Approved Budget
(US$) 2013–2015
Expenditure (US$)
(2013–2015)
Expenditure
(%)
(2013–2015)
Remarks
1,845,000.00 1,724,200.15 93.45% Main DA (five countries)
163,224.00
(Exclusive of programme
support cost)
139,361.04 85.38% Parallel pilot project
(Ethiopia) funded by the Irish
Government
3. Purpose of the Evaluation
The evaluation will assess the quality and viability of the mobile data collection system established in the
region through DA project.
The global evaluation for the DA project as a whole will be carried out by an international evaluator (see
also Section 6. Evaluation Methodology) who will be recruited separately. The national evaluator will
collect detailed data for the international evaluator to collate into the comprehensive mid-term evaluation
report for the project.
The evaluation result will also be used for designing Phase 2 of DA project.
4. Scope of the Evaluation
Type of evaluation Midterm evaluation for DA project
Time period covered by the
evaluation
Entire project implementation period
Geographical coverage of the
evaluation
Regional with focus on 6 pilot countries
5. Evaluation Criteria And Key Evaluation Questions
Relevance: Were the objectives as stated in the result frameworks consistent with member state-specific
policies, strategies, and plans?
DA Project MTE Report A-
11
Sample questions:
Objective/ rationale / logical framework: are they relevant to the achievement of expected
outcomes, given national and regional priorities? Was the logical framework indicators appropriate
and of sufficient quality to assess performance?
Was the programmes well designed and do they fit the framework of national and regional
strategies for development in the area of intervention?
Has a “gender approach” been considered in the programme design?
Have the objectives remained valid and relevant throughout implementation?
What are the implementation tools and mechanisms? Are they appropriate for the smooth and
timely implementation of key outputs?
Effectiveness: How effective have the interventions carried out been in terms of achieving the targeted
results?
Sample questions:
To what extent have the objectives been achieved?
What have been the (quantitative and qualitative) effects of the intervention?
To what extent do the observed effects correspond to the objectives?
To what extent can these changes/effects be credited to the intervention?
What factors influenced the achievements observed?
To what extent did different factors influence the achievements observed?
How user-friendly and easy the developed software is judged by consultant himself/herself
How user-friendly and easy the developed software is judged by enumerators and self-enumerators,
if applicable
Whether user manual of the software is available
Whether developer manual of the software is available
Whether the developed software have a functionality to record and send geospatial information of
the data
Efficiency: How efficient has the overall effort been in terms of management of resources, time committed
vis – a- vis results achieved?
Sample questions:
Were activities cost-efficient?
Were objectives achieved at the least cost?
Were the interventions implemented in the most efficient way compared to alternative ways?
Impact: To the extent possible, assess how the impact of the interventions funded under can reasonably be
attributed to or be associated with the support of the Commission. While ECA aims for specific social and
economic impacts, a range of intermediate changes will be instrumental to achieving these impacts. The
DA Project MTE Report A-
12
evaluation will attempt to measure such changes, which include influence, leverage, and learning, as
evidence of progress towards impacts.
Sustainability (potential for scalability): How sustainable has been the overall efforts, how well suited
was the initiative to the members’ states respective structures and to the overall goal and strategies of the
program at national and regional level? How sustainable will be the results if the support comes to an end?
- How much viable the working arrangements are to incorporate the developed program into the NSO’s
regular work (if applicable) How much viable the working arrangements are for the self-enumerators to
continue reporting data?
Sample questions:
To what extent and in what ways have national capacities been enhanced in the government
To what extent is there potential for replication and scaling up of innovative or catalytic approaches
Have complementarities, collaboration and / or synergies fostered by ECA contributed to greater
sustainability of results at the country/regional level
6. Evaluation Methodology
To allow more comprehensive analysis in the pilot countries, six national evaluators will be recruited for
the pilot countries who will collect data based on following activities:
Review of reports and documents prepared by NSO and TRI regarding the project conducted in the
respective country
in-person interviews with NSO, TRI and enumerator
hands-on experience with mobile application developed by NSO and TRI
and write a national evaluation report for the consolidated midterm report.
The consultants will design the evaluation methodology. They are also expected to propose refined and
specific evaluation questions to be included in the Inception Report. The evaluation should go beyond
questioning whether the indicators as set out in the original log frame have been achieved and should aim
to question the contribution of the programme to the outcomes and to assess the unplanned and unintended
results of the programme (positive and negative), and to learn how and why the change happened. The lead
consultant is expected to propose a suitable evaluation design and methodology for addressing the
evaluation questions, with justification.
Desk Review/secondary data collection will involve A thorough review of the Logframe, progress
reports, final reports and assessments.
Document review including analysis of previous reviews and evaluations and key reports and
reference documents.
Field work/Primary data collection: This will be conducted to the national data collectors, with
substantive support and oversight from the lead consultant
Selecting respondents. Criteria for selecting respondents will include, but not limited to:
Focal points from selected countries – ministries and institutions.
Develop Data collection protocols
Define data collection techniques to be used for different respondents.
DA Project MTE Report A-
13
7. Timeframe and Deliverables
The evaluator will be working for one month in total which would be divided into two parts:
1. Approximate 1~2 week(s) before national evaluation begins to provide guideline for national
evaluators and communicate it with them and
2. The remaining period after national evaluations end to compile reports, evaluate the DA project
and write the final report.
During the consultancy period, the evaluator should deliver:
a work plan within one week of the beginning of contract
an evaluation guideline/template for the national evaluators within two weeks of the beginning of
contract
an evaluation report by the end of contract
8. Evaluation Team Composition
Following people will be involved in the evaluation process:
National evaluator, in charge of collecting data from the pilot country and writing a national
evaluation report for the country
International evaluator (team leader; will be recruited with a separate TOR), in charge of
coordinating the six national evaluations, providing general guidelines for national evaluators and
writing the consolidated evaluation report for the project as a whole.
Management of Evaluation Process
The evaluation section will provide the norms, tools and templates for the different stages of the evaluation
process. It will advise on evaluation matters and quality review the deliverables
Evaluation section in consultation with ACS will clear the final TORs, the inception report and the draft
evaluation report and the final report. The section will support the process of issuing a management
response, and post the report and/or key findings over evaluation section. The section will provide technical
back stopping to the whole evaluation process and will be a part of the committee to select the service
provider
ACS will ensure that all necessary documentation, including project details and data are timely available to
the evaluators, arrange in-house appointments and with key stakeholders that are to be interviewed.
