World Bank: Evaluating Digital Citizen Engagement

8/15/2019 World Bank: Evaluating Digital Citizen Engagement

1/172


2/172


3/172


4/172


5/172


6/172


7/172


8/172


9/172


10/172


11/172


12/172


13/172


14/172


15/172


16/172


17/172


18/172


19/172


20/172


21/172


22/172


23/172


24/172


25/172


26/172


27/172


28/172


29/172


30/172


31/172


32/172


33/172


34/172


35/172


36/172


37/172


38/172


39/172


40/172


41/172


42/172


43/172


44/172


45/172


46/172


47/172


48/172


49/172


50/172


51/172


52/172


53/172


54/172


55/172


56/172


57/172


58/172


59/172


60/172


61/172


62/172


63/172


64/172


65/172


66/172


67/172


68/172

60

4.2.2. D si nin v lu tion qu stionsA key part of the Design stage is the formulation of the evaluation questions. Al -though certain lenses may become more or less important, it is recommended thatevery lens is considered, as all ve lenses have a direct bearing on both the type andthe breadth of the questions asked. The number of evaluation questions per lens mayalso be signi cant. A perspective that carries more importance will often require a

wider range of questions or greater probing to uncover ndings at a deeper level.The evaluation questions can also carry di erent degrees of importance dependingon the de ned goal and objectives, the relative importance of the each lens, and theexpectations of the audience and other stakeholders. This hierarchy also points to -wards the type of data needed and its relative importance, and may point to furtherwork being required at the Scoping stage (illustrating the importance of the iterativeapproach in this regard).

Once agreed, the evaluation questions take over from the lenses as the principal

source of focus and guidance for the following stages of the evaluation—the analysisof existing data and the collection of primary data. Given this, another importantaspect to consider is the representation of perspectives, e.g. the perspective of citi -zens, civil society, government agents and funders, in the questions. When makingan evaluation single or multiple perspectives about the objectives and impacts ofDCE processes should be accounted for. Therefore, it is important to consider howthe evaluation questions re ect (or require in their answers) di erent perspectivesand, crucially, to be clear on which perspectives are being left out or not considered.

Table 11 shows examples of the sort of evaluation questions that can emerge fromusing the ve lenses at the Design and Scoping stages. Whilst some of these arequestions restricted to this stage, many of them are also carried over to form thebasis of nal research questions. Careful scoping and design can often form part ofthe actual evaluation work, so e ort is seldom wasted at this stage.

TABLE 11. SAMPLE EVALUATION QUESTIONS ACCORDING TO THE 5 LENSES APPROACH.

Lens Sample of typical evaluation questions

Objective • What are the goals and objectives of the DCE?

• Do the goals appear reasonable, practical, sensible?• Is there a clear objective in the project linking, for example, activities,

objectives and goals?

• What is the external reality of the program and how does this impact on theprogram?


69/172

61

Lens Sample of typical evaluation questions

Control What actors de ne/de ned the project goals and objectives?

To what degree are citizens, stakeholders, bene ciaries, end-users… engaged inthe initial decisions, the design of the DCE process, the design of the technicalplatform, the delivery, the monitoring and evaluation..?

Who participates at each stage of the DCE?

Are there vigilance mechanisms in place and suitable levels of transparency toprotect against corruption?

Participation What characterizes the target audience in terms of availability, environmental/societal in uences, access to the engagement technology, desire to participate?

Who is engaging? Are those who engage in DCE di erent from those whoengage in CE?

How are they engaging in DCE?

Which interests and groups in society participants claim to represent?

Technology How successful is the DCE? How is this measured?

What are the weaknesses and fractures of the DCE process? What is thepotential for abusing/manipulating/co-opting the DCE process? What can beimproved upon?

How does the program handle privacy issues resulting from citizen data beingkept on a technical platform?

E ects Has the DCE resulted in a change of government/citizen behaviour?

Do the methods used have both internal and external validity?

What indicators do we use to measure “E ects”?

These kinds of scoping and design research questions were used extensively duringthe eld evaluations that inform this framework. Looking at the speci c questionsused in those studies can help clarify how and where this in uence and framingagainst the lenses has been most useful. Speci c examples from those four eldevaluations are in Table 12 below and a longer list of potential evaluation questionscan be found in Toolkit 1 .

Note that before moving on to consider what types of data are needed to answer theevaluation questions, it is worth taking a step back to ensure that the focus, goalsand objectives of the evaluation are clear and appropriate; that the questions for -mulated can realistically be answered (given the ndings that emerged during theScoping stage and the focus and constraints of the evaluation) and that the ques -tions will likely generate answers that will enable the evaluation goals and objec -tives to be achieved.


70/172

62

TABLE 12. CASE STUDY EVALUATION QUESTIONS (USING THE 5 LENSES APPROACH)

Evaluation questions

BRAZIL CAMEROON UGANDA KENYA

Lens | Objective

Are the goals of theSistema clear andappropriate?

How e ective is theSMS platform inachieving the DCEobjectives of theprogram?

What is the purpose ofthe U-Report?

Does the MajiVoiceplatform e ectivelyhandle customercomplaints andfeedback?

Lens | Control

Which actors controlthe budget beingallocated to theparticipatory process?

Which actors controlthe participatorybudgeting process?

Which actors drivetheU-Report and duringwhat stages?

Which actors controlthe platform use?

Lens | Participation

Does online votinga ect the level ofturnout?

Do online and o inevoters have di erentdemographics?

Do online and o inevoters engage with theparticipatory processin di erent ways?

Who does the SMSplatform engage with?

How do they compareto the generalpopulation, how dothey compare with thepeople that engage inthe budgeting processwithout engagementvia SMS?

Who are theU-Reporters and howdo they compareto the Ugandanpopulation?

If not representative,what could be doneto obtain a morerepresentative sampleof the population?

To what extent havethe di erent digitalfeedback mechanismshave been used, bywhom and for whatpurpose?

Lens | Technology

What opportunities forabuse exist in online /o ine processes?

What transparencyand oversight docitizens have of theSistema and of theimplementation of the

results?

How e ective ande cient is SMS asan engagement toolcompared to othermedia?

What bene ts/drawbacks does itprovide in its current

form?

What are thelimitations andopportunities forexpression andrepresentationsupported by theplatforms?

How does the data

collected throughU-Report compareto those obtained bytraditional means?

What is the impact ofthe digital feedbackmechanisms on thepropensity of peopleto provide feedback?


71/172

63

Evaluation questions

BRAZIL CAMEROON UGANDA KENYA

Lens | E ects

Do online and o inevoters vote di erentlyand how does thisa ect governmentspending?

What opportunities forindividual citizenshipand empowerment areavailable to online ando ine voters?

To what extent doesthe SMS engagementstrategy increaseparticipation in theprocess?

Does it changethe nature ofparticipation?

What types of changedoes U-report bringabout?

Does U-Report a ectthe decisions ofgovernment o cials?

What is the e ectof the digitalfeedback processon participants’attitudes, perceptionsand performance(providers of feedbackand receiversof feedback–organizational andindividual level)?

4.2.3. Wh t t p s of d t r n d d to nsw r th qu stions?Once the evaluation questions have been agreed, the approach itself can be de -signed. This involves assessing what data is needed (remembering the importanceof obtaining di erent perspectives), identifying and assessing whether any existingsystem data is su cient, deciding whether new data must be collected, and decidingthe broad approach to be taken in collecting the data. An evaluation matrix or simpledecision tree can be extremely useful in this process.

Example: Creating an Evaluation Matrix

When you have identi ed options that might be suitable for answering key evaluation questions, create

a matrix of the questions and potential options for data which may help to answer them. This matrix canhelp you check that the planned data collection will cover all the questions su ciently, and enable you tobetter see if there is su cient triangulation between di erent data sources ( Better Evaluation 2014c) .

Participantquestionnaire

Key informantinterviews

Projectrecords

Observation of programimplementation

KEQ1 What is the qualityof implementation

KEQ2 To what extentwere the programobjectives met?

KEQ3 What other impactsdid the program have?

KEQ4 How could theprogram be improved?


72/172

64

As with all evaluations, the nature of the evaluation questions determines whetherthe data needed is quantitative and/or qualitative (and often it will be both). In DCE,quantitative data could include participant demographics, self-reported percep -tions and number of exchanges (such as online corruption complaints or freedomof information requests). Qualitative data could be from, e.g., interviews to get thegovernment or citizen perspective and perceptions of ‘success’.

In addition to considering the broad nature of the data required, decisions aroundwhat data is needed will also be informed by decisions on the indicators to be used 1.In doing so, it is also worth considering choosing universally applicable (standard -ized) indicators in order to gain greater consistency across DCE project evaluations.The Strategic Framework for Mainstreaming Citizen Engagement in World BankGroup Operations proposed standardized results indicators for Citizen Engagementat both intermediate and nal outcome stage (World Bank, 2014b). Some examplesinclude: percentage of bene ciaries that feel project investments are re ecting/re -

ected their needs; percentage of bene ciaries satis ed with speci ed dimensions,e.g. access, delivery time, responsiveness to needs, transaction costs, operation -al e ciency, bribery experience, sta attitude, quality of facilities; reports/allega -tions of abuse/waste/corruption; investigations resulting in remedial actions and/orsanctions. The full draft list can be found in Appendix C .

A digital element would measure both these indicators and the impact and e ectivenessof the ICT medium, e.g. number of SMS sent/percentages responded to etc. Whicheverindicators are selected, where possible, these should be tested at small-scale or with asample group, and adjusted if necessary, before full deployment commences.

4.2.4. Is n w d t n c ss r ?Once it is clear what data is needed, the decision needs to be made as to whether theexisting and available data identi ed in the Scoping stage is su cient to answer theevaluation questions, or whether new data needs to be collected.

At this stage, an initial, super cial sweep of existing data will indicate how usefulit will be. In reality, there is rarely perfect data—a key variable might be missing,it doesn’t cover the full period in question, data quality is an issue, etc.—but thequestion should ask ‘is the data su cient for the purposes of the evaluation?’ ratherthan ‘is the data perfect?’.

Once data gaps have been identi ed, a decision can be made as to what (if any) new

1–This guide uses the de nition of an indicator as the measurement of “an objective to be met, a resource mobilized, an e ect obtained,a gauge of quality or a context variable. An indicator produces quanti able information with a view to helping actors concerned with publicinterventions to communicate, negotiate or make decisions” (Tavistock Institute, 2003). For indicators to be most e ective and informa-tive then they bene t from being SMART (Speci c, Measurable, Achievable, Realistic and Time Bound).


73/172

65

data will be needed and the time/budget available to collect this. It is also worthconsidering how any new data will be validated, stored, published and shared. Anextended evaluation matrix or decision tree can be useful in mapping data needs,sources and collection requirements.

Thinking about the whole data lifecycle from the beginning will help ensure that theright type of data is being collected, potentially saving a lot of work later.

4.2.5. D cidin on m thod for coll ctin n w d tChoosing the right methodological approach for an evaluation is a critical decisionand involves a high degree of technical knowledge. Although a detailed discussionof the di erent methods is beyond the scope of this guide, this brief introduction isprovided to compare the most common and relevant methods to facilitate discus -sion between commissioners of evaluation and experienced evaluators. More ad -vanced guides to research methods are included in the Further Reading at the end ofthis section. Practitioners with less experience in some methods may nd the ‘keyfactors’ boxes at the end of each method more helpful than the broader discussions.

This section reviews eight common evaluation methods (adapted from Nabatchi,2012, Shadish et al., 2002, and Shen et al., 2012). Each method has strengths andweaknesses, often depending on the nature of the data required to examine if andhow the objective of the intervention connects with key outputs to outcomes. Notall of these methods are necessarily of particular relevance to DCE, but they are im -portant for all evaluations, and all of them bene t from data collection via digitaltools such as SMS, multimedia platforms and online questionnaires. The rst four

sit on a spectrum of how causality can be established they are1

(true experiments;eld experiments; ex-post-facto design; quasi-experimental: non-experimental),while the following four are types of non-experimental studies with a more qualita -tive nature (theory-based; participatory; ethnographic; case study). However, all ofthese methods can, in fact, include both quantitative and qualitative analysis.

For each method a brief description is provided, followed by suggestions of when itis more or less suitable for use, and key factors to consider when used for evaluatingDCE programs.

4.2.5.1. RANDOMIZED CONTROL TRIALSRandomized trials (or randomized control trials–when a control group is present)are often seen the most rigorous methodological approach to study impact. Ran -

1–A true experiment has two main characteristics: (1) randomly assigned groups–every participant has an equal chance of being inthe experimental or control group(s); (2) manipulation of a variable where di erent groups are subject to di erent treatments orabsence of treatment (control group). Natural experiments or quasi-natural experiments are studies where subjects were groupedby nature or by other factors outside the control of the investigators.


74/172

66

domized trials involve randomly assigning interventions (treatments) to subjects,for instance di erent persons get di erent outreach messages for a particular citi -zen engagement project. Randomized trials that take place in the real world as partof an ongoing intervention with real participants are called eld experiments. How -ever, randomized trials can also take place in more controlled or laboratorial set -tings, involving small-scale treatments with recruited participants.

Experiments can help us understand how e ective an intervention really is by cre -ating the conditions for comparing two or more scenarios (treatments or levels ofindependent variable). Du o et al. (2012), for example, conducted an experimentwhereby teachers in randomly selected schools were monitored daily using cameras.Attendance was rewarded with a salary increase. The researchers compared the at -tendance rate of this (treatment) group with teachers from schools that went abouttheir business as usual (control group) and discovered that the attendance in thetreatment group increased by 21%. The random selection of schools makes it possi -ble to directly attribute this increase to the treatment being evaluated.

RCTs are regarded by many as the most rigorous appraisal tools to establish cause-ef -fects relationships. This is based on the view that without random assignment ofcases (e.g. individuals or schools) into di erent groups and direct manipulationof independent variable(s), one can never be certain that the observed e ects arethe result of the intervention and not pre-existing di erences between the groups(classi catory factors) or situational or environmental variables (pseudofactors).Quasi-experiments share with true experiments the manipulation of independentvariables, but fail at assigning participants at random to the treatment groups (they

use natural occurring groups). For instance: without random assignment, how canone ensure that participants in a DCE initiative feel empowered because of the in -tervention or because the project had features that attracted well-educated citizenscon dent about their rights and voice? It is very di cult to answer this questioncon dently after the intervention has taken place.

Experimental designs can also be useful to test particular aspects of a DCE interven -tion. For example, an RCT study in Uganda (Grossman et al., 2014) varied the priceof using an SMS system that allowed citizens to reach Members of Parliament to testif and how the cost a ected usage.


75/172

67

KEY FACTORS WHEN CONSIDERING USING AN RCT

Despite the positive aspects of RCTs, there are important potential drawbacks. RCTs that test thee ectiveness of the intervention in its entirety (rather than certain aspects of it or design features)can be expensive. However, the digitally mediated character of DCE interventions opens up newopportunities for testing and improving upon key design features quickly and cheaply and meansdigital random experiments can sometimes be a highly cost-e ective option.

Furthermore, RCTs work better when: (Stern et al., 2012:38–39):

• There is only one primary cause and one primary e ect. This might not be the case in morecomplex interventions;

• The control group is not ‘contaminated’ by the intervention (i.e. the ‘non-treated’ individuals needto be unaware of the treatment and with no contact with ‘treated’ individuals) so comparisonsbetween treatment and control groups is valid;

• The focus lies on the success of a particular intervention. Generalisation to other individuals,times, contexts or interventions are not feasible (the problem of external validity);

• Understanding what works is more important than understanding why it works.

• The group sizes support statistical analysis (at least 30 cases per group)


76/172

68

R cruitin xp rim nt p rtn rsfor R ndomis d Control Tri lTo do an RCT, a partner usually needs to be recruited, often a small organization orvoluntary group. Researchers rely on the organization agreeing to the experimentand understanding what is needed. It can be costly if the partner starts to doubt theexperiment once it has started. The partner might also be wondering whether theyare working with the right researchers. Partnerships can be like a dating game—earlycontacts are important in determining whether the partners like each other and where

exit strategies are available if the liaison is not going to work.

How best to start? Do you go in at the top, say at the political level, and write a lettersaying ‘I would like to do an experiment, please help me’? Such a letter might workor might not be answered or be answered negatively. In other cases, it is better toapproach personnel lower down the organization who deliver services. If they becomeenthused about the experiment they can seek higher-level authorization when thetime is right. Informal contacts also play a role if the researchers already know thepeople involved by going to meetings and social gatherings, which is where ideas can be

discussed and followed up.

The rst planning meeting between researchers and partners is very important. Tohave got that far is often a good sign that internal conversations have taken place. Butsuch meetings do not necessary go well as the organization can realize the costs of theintervention. But sometimes the worst meetings are where it goes too well: everyoneis excited by the trial and there is a warm glow all round. The cost of the RCT onlybecomes apparent much later on. It is a good idea to balance out creating enthusiasmwith conveying what is involved. It is important to go through each stage of the trial

and work out what the researcher and the partner need to do.

Professor Peter JohnProfessor of Political Science and Public Policy, University College London

www.ucl.ac.uk/spp/people/peter-john

http://www.ucl.ac.uk/spp/people/peter-johnhttp://www.ucl.ac.uk/spp/people/peter-john


77/172

69

4.2.5.2. ‘EX POST FACTO’In ex post facto design, investigators begin their study after the intervention hastaken place without manipulating independent variables or assignment participantsat random (Silva Carlos, 2010). This type of design allows evaluators to weakly iden -tify cause and e ect relationships. In this type of strategy the researcher takes inthe e ect (dependent variable) and looks back in time to investigate possible causes,

relationships and associations between natural occurring groups.In this type of design the investigators compare groups who were exposed to acertain intervention (cause or independent variable) in terms of an outcome of in -terest. For example, comparing the voter turnout in provinces where people haveaccess or have not access to online voting in participatory budget initiatives. It isalso possible to use the reverse strategy to look back in time to investigate possiblecauses of the di erences between two groups. For example, users and non-users ofa complaint platform could be compared in terms of satisfaction with the serviceprovided or other characteristic (e.g. level of education). Along these lines, sub - jects who di er on the dependent variable can be the starting point, and inspectdi erence on an independent variable (age, education, occupation). It may also benecessary to understand why an intervention a ected the participants in a certainway (Cohen et al., 2011). Because the ex post facto design is aimed at inspectingdynamics that occurred in the past, the most common method for data gatheringis structured questionnaires about past events, personal values or motivations orsocio-demographical characteristics.

KEY FACTORS WHEN CONSIDERING EX POST FACTO EVALUATIONS

Ex post facto designs are useful when:

A more rigorous experimental approach is not possible for ethical or practical reasons

• Studying conditions

• Weak cause-and-e ect relationships are being explored to be tested later through a RCT

• Studying the e ectiveness of an intervention on naturally in existing groups

One of the main weaknesses of ex post facto designs is that they often cannot help establish thedirection of causality (what is the cause and what is the e ect) and rule out other explanations forthe outcome that may have co-varied with the intervention, leading to potential confounding ofcauses (third variable problem). In addition, it is prone to self-selection bias as users and non-users

may di er a priori on a number of characteristics.


78/172

70

4.2.5.3. NON-EXPERIMENTAL DESIGNS (CORRELATIONAL DESIGNS)Unlike the designs presented thus far, non-experimental or correlational designs arenot set up to examine cause and e ect but only associations between events or char -acteristics of participants. These type of methods do not rely on manipulation of in -dependent variables, but on measurement of potential independent and dependentvariables to understand how they are associated. Surveys are the prime method for

data collection in correlational designs to explore associations between di erent vari -ables (e.g. age and frequency of participation). Their key strength lies in that they canbe used to explore a broader range of hypotheses than experimental designs.

Surveys are commonly used to measure the characteristics, knowledge, attitudesopinions and practices of a given population. In a DCE context, a population canbe de ned broadly (to refer, for example, to all users of a platform) or narrowly (toinclude for instance only high-frequency users). A signi cant correlation betweentwo variables might hint to a cause but is not in itself enough to establish a causeand e ect relationship. There are also statistical strategies (e.g., partial correlationor multiple regression analysis) to control for the e ects of other variables (mea -sured and included in the analysis) when an association between two variables isfound. Correlational designs can, however, be taken a step further to examine howdi erent variables contribute to an outcome. For instance, what factors can betterpredict whether a participant will contribute more? Is it mostly gender, or is it aninteraction of gender and education? However, it is important to think in advanceabout what kind of analysis might be required for a particular evaluation since thishas a bearing on how variables are measured.

An important aspect of a correlational design is sampling. Sampling is the objectiveaccording to which the evaluator selects who will respond to our survey is selectedto maximize the validity of the results. Although in the context of a DCE initiative, itmight be possible to conduct a survey of all registered participants (called a census)this might want to be avoided. It will be too costly, time-consuming and could leadto bias, if for example, only high-frequency users responded. Sampling strategiescan vary depending on the time and human resources and whether a lists for thepopulation (sampling frame) or accurate population gures exist.

In the context of DCE survey designs, it should also take into account importantcontextual factors that might render some of the variables meaningless. For exam -ple, when conducting a study in an African country it is not advisable to use the em -ployment categories that are used in surveys in western countries. This is becausevery few poor people are waged employees and make do by doing a little bit of every -thing-having a small stall at the market, working occasionally in construction, etc.


79/172

71

In the context of the DCE, where surveys can be conducted relatively cheaply over SMSthe Internet and with automated voice systems, this temptation becomes even moreappealing. Another good strategy to follow when designing a questionnaire for a DCEinitiative is borrow questions and adapt from other sources in areas relevant for theevaluation (some examples are included in Further Reading at the end of the section).

KEY FACTORS WHEN CONSIDERING USING CORRELATIONAL DESIGNS

• Correlational designs are not suited to examine cause and e ect relationships, but they are idealfor examining a broad range of hypotheses.

• The generalizability of ndings will largely depend upon the nature of the sample. A rigorouslyconstructed and sizeable sample is therefore an important aspect of this research design.

• In designing their questionnaires evaluators need also think about the types of analyses theywould do as this a ects how indicators are measured.

• Survey questions need to be informed by an understanding of the context of the evaluation.

• ICTs create new channels for delivering their questionnaires (SMS, face-to-face, telephone).However its option should be considered carefully as it will introduce its own biases in the datacollection process.


80/172

72

4.2.5.4. THEORY-BASED DESIGNSIn this approach, the intervention is considered as a conjunction of ‘causes’ thatfollow a sequence like the pieces of a domino. The ‘objective’ lens is a variant of thisresearch strategy (section 3.4.1). This approach develops an understanding not onlywhether an intervention works but also what are the conditions that make it work.Theory based methodologies are making a come-back in evaluative inquiry. This is

because being able to say that x caused y often does not allow us to understand whysome things work and do not work. Theory-based designs allow us to explain whyand how exactly an intervention led to the desired change.

There are weak and strong versions of theory-based designs (Stern et al., 2012).Weak theory-based or program theories are usually no more than objective modelsthat depict, usually diagrammatically, how an intervention is understood to con -tribute to intermediate outcomes and long-term impacts. Richer objective modelsmight include the views of di erent stakeholders. Stronger theory-based designs gointo more detail to identify not just the steps that lead from goals to outcomes andimpact but the exact mechanisms that make things happen, that can be highly con -textual. A program’s theory usually combines a theory of change, an understanding ofthe process through change is seen to come about and a theory of action, the way inwhich an intervention is implemented to realise the theory of change.

Theory-based designs form an integral part of most evaluations because of theirusefulness in enabling experienced evaluators to develop quickly some understand -ing of the main strengths and weaknesses of an initiative and to adapt the evalua -tion according. Stronger theoretical designs, especially when they are informed by

relevant literature, can be helpful in generating hypotheses for the evaluation andhelping to better understand cause and e ect relationships.

KEY FACTORS WHEN CONSIDERING USING A THEORY-BASED DESIGN

Theory based designs are to a lesser or greater degree part of any evaluative strategy as they helpevaluators appreciate how a program is understood to translate goals into intermediate outcomesand long-term impacts. Whereas RCTs help answer the question of whether an intervention workedor not, a theory-based design, through the development of nuanced hypotheses can help explainwhy it worked and why it didn’t.

Theory based designs are particularly relevant for DCE where it is often assumed that the bene ts

of using digital technologies will ow automatically.


81/172

73

A Not on Ethno r phic proj ct v lu tionEthnography is both scienti c (to understand the world through the empiricalsenses) and expressive (to share evidence from this engagement throughevocative writing and multi-media techniques techniques). The methodcontributes to evaluating research in two important ways: to understandgrounded knowledge embedded in the praxis of social life; to provide a credibleaccount of context and its relationship to lived experiences of individuals.

Ethnography can illumine the relationship between social context and digital

content from the perspective of the digital citizen. An example of unravelingdigital citizen engagement would be the following: Ethnography can contributein the understanding of urban slum youth’s engagement with social media,particularly Facebook. With extended eld immersions and thick descriptionsof the slum context and youth social media activity, insights on a number ofsocial relationships can be established: how do slum youth make meaning out ofFacebook and the ensuing digital a ordability at once unique and challenging?How is Facebook con gured by youth in the context of extreme lack of privacy in

an urban slum context while allowing a certain control over ones representationand expression? How is Facebook a gateway to many unattainable expressionsof one’s personal and social compass? Is Facebook the rst digital engagementwith global experiences?

Ethnographic methods can help explore and answer deeper questions that moretraditional methods o er little insight into. Results from such an ethnographicengagement can in uence developmental engagements with non-elite sectionsof society, guide policy to better serve these groups.

Nimmi Rangaswamy Xerox Research Centre India, Indian Institute of Technology Hyderabad

www.iith.ac.in


82/172

74

4.2.5.5. ETHNOGRAPHIC DESIGNSHere the emphasis is placed on understanding the context in which the interventiontakes place and the perceptions, views and ideas of stakeholders and the spaces thatshape it. Ethnographic research prioritizes understanding the context of people’severyday lives, their shared social norms, priorities and assumptions.

Compared to experiments and survey-based designs which often adopt narrowde nition of why, what and who, ethnographic data provides rich and holistic in -sights into people’s views, habits and actions and the spaces that they inhabit.Ethnography relies more on individual and focus groups interviews, observations,and presupposes the immersion of the researcher into the setting where partici -pants are located, often for long periods of time. Although this may be impractical,the spirit of ethnographic research can be adopted and adapted to the quick turn-around time of evaluations. Ethnography makes researchers an integral part ofthe study, inviting them to re ect on their own biases and assumptions and thosemade by the project. Key challenges of ethnography include that ndings are con -text speci c and may not be readily generalizable, and it can be time intensive bothin terms of data collection and analysis and in many cases, especially when Englishis not spoken widely, requires the use of translators. In the context of DCE, ethno -graphic research can be invaluable in supplementing or informing the analysis ofbig datasets generated through the digital platforms by helping evaluators developa sense of how the data is being generated and used, what practices and ideas driveparticipation and data use.

KEY FACTORS WHEN CONSIDERING USING AN ETHNOGRAPHIC DESIGN

• Ethnographic designs and methods are invaluable in developing a sense of the context of theevaluation and in informing the design of other methodologies of data collection and analysis.

• Ethnography need not be an all or nothing proposition. Used strategically, ethnographic methodssuch as observations / interviews can help re ne the design of other evaluative tools and analyses.

• The selection of interviewees is important in ethnographic designs as responses, reactions, and theexpressions of views can be in uenced by power dynamics between interviewees and the interviewers.

• The use of experienced local researchers can be help to mitigate some of the costs in datacollection and analysis and help overcome language problems.


83/172

75

4.2.5.6. PARTICIPATORY EVALUATIONParticipatory evaluations involve the participation of project participants from thestart to de ne the goals, agendas, measures against which a program will be evalu -ated. It often incorporates tools such as story-telling, scoring, social mapping, trendand change diagramming. The basis of causal inference here lies on the validation ofa programs outcomes and impact by participants. By having program participants as

partners in the research and privileging their ideas and opinions, it helps clarify thecontext and challenges that the initiative seeks to address, improving a DCE’s rele -vance and increasing ownership. Participatory designs involve a panoply of methodsthat include the collaborative creation of matrices, maps, owcharts and timelines,or questionnaires to review existing information, such as assessing program goalsand outcomes, to plan and assign roles and responsibilities (Chambers, 2008). Digi -tal storytelling can be an especially powerful tool for expression and learning.

Some aspects of participatory evaluation may be blended with other research strat -egies. Participants might be asked to de ne, for example, what success means in thecontext of the initiative and their de nitions might inform a questionnaire design.However, it can also be costly, requiring a signi cant degree of commitment on thepart of the participants.

KEY FACTORS WHEN CONSIDERING USING PARTICIPATORY EVALUATION

In addition to helping de ne locally relevant evaluation questions and indicators, participatoryresearch can increase participants’ sense of ownership of the project. However, it also requires asigni cant degree of commitment on the part of the participants and the evaluators in the processto ensure that the views and opinions raised in the process are taken into account.

The use of technology which involves the communication of highly complex terms and processesto participants could be a challenging but also potentially rewarding exercise. What do participantsmake of the new data ows? How do they address issues around anonymity?

Similarly to ethnographic designs, participatory designs are sensitive to power dynamics and elitecapture in particular.


84/172

76

4.2.5.7. CASE STUDY DESIGNThe term ‘case study’ is often used as an umbrella term for a collection of researchmethods that are combined to examine an ‘instance’ of a project, group, organiza -tion, event. This ‘instance’ in research terminology is often referred to as a ‘unit’. Inthe context of the DCE a ‘unit’ can be de ned in a number of ways. One can chose tode ne an entire project as a unit (holistic design). Such a design would treat the proj -

ect as a whole in terms of the ve lenses. Alternatively the project can be treated asconsisting of a number of di erent components, di erent organizations (includingpartner organizations) di erent participant groups, di erent processes. This em -bedded research design can serve to build a richer and fuller picture of an initiativeor di erent initiatives by incorporating di erent perspectives and levels of analysis.

A case study research design is by far the more pluralistic of the designs presentedthus far in terms of tools for data collection. Although it usually involves some typeof ethnographic work (e.g. interviews and observations) it can also incorporate sur -veys. A key aspect of this type of design which distinguishes it from ethnography isthe importance of theory development. This can be in the form a simple hypothesisat the beginning of the evaluation which is informed by relevant literature on whatworks and does not that can be tested further.

Although case studies are often considered as less scienti cally credible due to theirperceived limitations for drawing causal inferences and generalising ndings, pro -ponents of case studies have argued that, although the case study ndings mightnot be statistically generalizable, they can be analytically generalizable “ by havingthe theory developed for the study compared against the empirical ndings” (Yin,

2003:32). New methods for the systematic causal analysis of cases such as Quali -tative Comparative Analysis (Stern et al., 2012) are attracting more attention as avalid alternative to experimental research designs. Comparative case studies are ane ective tool, for example, when one’s own case is too small to engage in quanti -tative analysis, when other methods are unavailable or inappropriate, and/or whenthe variables are di cult to disentangle from each other.

In a DCE context, a well thought out case study research design combines the advan -tages of ethnographic work in terms of its ability to yield rich contextual data withsome of the key elements of theory-based designs.

KEY FACTORS WHEN CONSIDERING USING A CASE STUDY DESIGN

A case study need not be a study of the intervention as a whole. This design can may be useful inidentifying components of the program whose closer examination might bring generate importantinsights for the evaluators, especially when it is theoretically grounded. Combined with newmethods for systematic causal analysis, theoretically sophisticated case study design can be apowerful tool for the evaluators.


85/172

77

4.2.5.8. CHOOSING AND COMBINING METHODSThe boxes at the end of each discussion above should help in identifying how suit -able the di erent methods may be when designing a DCE evaluation. Although,traditionally, some of these methods have been considered as more scienti callyrigorous than others, especially when it comes to the examination of cause and ef -fect relationships, current debates suggest that if there is a ‘gold standard’ to be

achieved, it is through methodological pluralism (mixed methods), blending ele -ments of di erent research designs depending on what is appropriate for meetingthe requirements of a particular evaluation and for increasing the evaluation’s use -fulness for funders, program designers and implementers and participants.

Here the order in which the methods are used ( sequencing) is something that needs tobe considered. “Mixed methods” is not only about using multiple methods to high -light di erent perspectives, it is about using the results from one method to informdesigns of another. Triangulation , the process through which one method is used tocheck the results of the other can also be valuable in a mixed methods approach.

For DCE, such an innovative, mixed-methods approach is supported by the digitalcharacter of the intervention. The inherent characteristics of DCE also mean thatcollection of new data via both experimental and non-experimental methods is of -ten a realistic option in terms of cost and timescale. The ability to reach partici -pants through SMS, for example, may lower the costs for conducting representativesurveys, at the expense of exclusion of low-income groups or those with no digitalaccess. Similarly, digital tools allow for low-cost experiments that would have beenimpractical without large time and budgets using traditional methods.


86/172

78

Low-cost xp rim nt tion usin soci l m diCan lower-cost ICT channels increase participation among those who already have economicand political capacity to participate? Over the past year MIT Governance Lab (MIT GOV/LAB–web.mit.edu/polisci/research/govlab.html ), mySociety, and Mzalendo (info.mzalendo.com)have collaborated on a research program exploring ways to engage Kenyan citizens and togalvanize political action online. The researchers have developed an ‘iterated experimentation’approach–relatively short and small-scale investigations using rigorous social sciencetechniques that build on ndings from previous rounds to test which operational and designchoices are most e ective in particular contexts.

From conversations with citizens and reading news articles, we believed that the information

Kenyans receive about government is generally negative and threatening. We anticipated thatKenyans may experience ‘threat fatigue’ and instead of being motivated to act when there is aperceived threat, may be more likely to engage when there is a perceived opportunity or theyreceive information about the engagement of others.

We recruited Kenyan citizens interested in nding out about county government managementof public funds using Facebook ads. We bought four di erent ads targeting men and womenover and under 30 years old in order to explore how treatments a ected subpopulationsdi erently. Each person who clicked on the advertisement was randomly assigned to oneof the treatment pages, which presented participants with an article about how countygovernments have misspent taxpayer dollars. Di erent treatment pages either framed theinformation in terms of the threat of lost resources, the opportunity of gaining resources, orneutrally. Additionally, we included a ‘bandwagoning’ interaction with each presentation thatincluded information about how many other people had also taken action. At the end of thearticle, we provided four actions viewers could take: share the article on Facebook, share it ontwitter, sign a petition, and send a comment to the Senate Majority leader. We compared thedi erences in number and types of actions taken by viewers who saw each treatment page.

Preliminary ndings suggest that groups who have higher socioeconomic status–speci cally

older men – are more likely to take action via this online platform, and that people are morelikely to take action when they receive information about how many others are participating.Findings from this iteration will inform the next round of experiments by testing hypothesesthat expand on these results.

Lily L. Tsai and Leah RosenzweigMIT Governance Lab (MIT GOV/LAB)

www.web.mit.edu/polisci/research/govlab.html

http://web.mit.edu/polisci/research/govlab.htmlhttp://web.mit.edu/polisci/research/govlab.htmlhttp://web.mit.edu/polisci/research/govlab.htmlhttp://web.mit.edu/polisci/research/govlab.html


87/172

79

Key lessons learned from the eld evaluations- Designing -

Try to design the evaluation at the same time as designing the engagement

From Cameroon: “There was great interest in the possibilities of a more detailedevaluation following earlier preliminary evaluation work by the World Bank, as there isa large body of citizen engagement data and information available going back severalyears. Unfortunately much of this data are not easily accessible. Other than the actualdigital engagement platform system data (how many people have been sent SMS messagesabout the vote for example), it’s all largely non-digital, on paper and generally not ofgreat quality. So for example, data are being collected on paper when people are going toparticipatory budget meetings—in 2014, we were looking at 4 communes, 15 meetings

per month each, 7 months, average of 30 participants per meeting = 12,600 participantdetail records but all this is being collected on paper forms. If the program had thecapacity to collect it in digital form and used digital tools to control the quality of thedata collected from early on, then that would have made the analysis, quality-checkingand evaluation much quicker and easier. There had been plans to do just this but due toresource constraints within the local program, these were not implemented. That wasdisappointing.” (Martin Belcher, Aptivate)

Di erent approaches and tools can be used in the same evaluation and within di erentbudgets to reach a range of audiences

From Brazil: “We decided early on we wanted to focus on three groups of people—those whovoted online; those who voted using traditional voting booths and those who didn’t voteat all. We used SurveyMonkey which popped up on the voting website, asking them if theycould answer a few questions. For the face-to-face survey, we had around 50 enumeratorsaround the various polling booths. For those who didn’t vote, we did an automated random-digit-dial IVR survey which was an a ordable compromise from a full door-to-doorhousehold survey—which we did want to do but would have been much more expensive.”(Matt Haikin, Aptivate).

Local partners are an invaluable resource for testing and improving surveys

From Kenya: “There was a customer complainant survey—the local water company advisedon the questions and gave some useful feedback, for example we wanted to ask about incomeand they advised that people would be sensitive around that and refuse to answer and soreduce completion rates signi cantly. So we decided to use a couple of proxy indicators thatwe could infer income from (rent levels and geographic zoning of complainants—whichwater company o ce the complaint would be directed to, regardless of how the complaintcame in) to help with our understanding in this regard.” (Martin Belcher, Aptivate)


88/172


89/172

81

Selected readings and online resources on designing an evaluation

Accenture Digital Engagement Pulse Survey (2012) – http://www.accenture.com/us-en/ Pages/insight-digital-government-digital-citizens-ready-willing-waiting.aspx

Afrobarometer economic, social and political opinions/characteristics of Africans acrossseveral countries – www.afrobarometer.org

Civic engagement in the digital age – http://www.pewinternet.org/2013/04/25/civic-engagement-in-the-digital-age

Civic Plus Digital Citizen Engagement survey – http://go.civicplus.com/l/9522/2012-10-26/8t542

Conducting quality impact evaluations under budget, time and data constraints – http:// www.oecd.org/derec/worldbankgroup/37010607.pdf

Mixed-Method Impact Evaluation for a Mobile Phone Application for Nutrition ServiceDelivery in Indonesia – http://www.ids.ac.uk/publication/a-mixed-method-impact-evaluation-design-of-a-mobile-phone-application-for-nutrition-service-delivery-in-indonesia

Decide which evaluation method to use – http://betterevaluation.org/start_here/decide_which_method

Fools’ gold: the widely touted methodological “gold standard” is neither golden nor astandard – http://betterevaluation.org/blog/fools_gold_widely_touted_methodological_gold_standard

How useful are RCTs in evaluating transparency and accountability projects? – http:// www.makingallvoicescount.org/news/how-useful-are-rcts-in-evaluating-transparency-accountability-projects/

Overview of current Advocacy Evaluation Practice – http://www.innonet.org/client_docs/File/ center_pubs/overview_current_eval_practice.pdf

Participatory Approaches – http://www.participatorymethods.org/sites/participatorymethods.org/ les/Participatory_Approaches_ENG%20Irene%20Guijt.pdf

Participatory Evaluation – http://betterevaluation.org/plan/approach/participatory_evaluation

Participatory Monitoring and Evaluation: Learning from Change (1998) – http://www.ids.ac.uk/publication/participatory-monitoring-and-evaluation-learning-from-change

Qualitative Evaluation Checklist – http://www.wmich.edu/evalctr/archive_checklists/qec.pdf

Randomized Evaluations: Methodology Overview and Resources – http://www.povertyactionlab.org/methodology

http://www.accenture.com/us-en/Pages/insight-digital-government-digital-citizens-ready-willing-waiting.aspxhttp://www.accenture.com/us-en/Pages/insight-digital-government-digital-citizens-ready-willing-waiting.aspxhttp://www.afrobarometer.org/http://www.pewinternet.org/2013/04/25/civic-engagement-in-the-digital-agehttp://www.pewinternet.org/2013/04/25/civic-engagement-in-the-digital-agehttp://go.civicplus.com/l/9522/2012-10-26/8t542http://go.civicplus.com/l/9522/2012-10-26/8t542http://www.oecd.org/derec/worldbankgroup/37010607.pdfhttp://www.oecd.org/derec/worldbankgroup/37010607.pdfhttp://www.ids.ac.uk/publication/a-mixed-method-impact-evaluation-design-of-a-mobile-phone-application-for-nutrition-service-delivery-in-indonesiahttp://www.ids.ac.uk/publication/a-mixed-method-impact-evaluation-design-of-a-mobile-phone-application-for-nutrition-service-delivery-in-indonesiahttp://betterevaluation.org/start_here/decide_which_methodhttp://betterevaluation.org/start_here/decide_which_methodhttp://betterevaluation.org/blog/fools_gold_widely_touted_methodological_gold_standardhttp://betterevaluation.org/blog/fools_gold_widely_touted_methodological_gold_standardhttp://www.makingallvoicescount.org/news/how-useful-are-rcts-in-evaluating-transparency-accountability-projects/http://www.makingallvoicescount.org/news/how-useful-are-rcts-in-evaluating-transparency-accountability-projects/http://www.makingallvoicescount.org/news/how-useful-are-rcts-in-evaluating-transparency-accountability-projects/http://www.innonet.org/client_docs/File/center_pubs/overview_current_eval_practice.pdfhttp://www.innonet.org/client_docs/File/center_pubs/overview_current_eval_practice.pdfhttp://www.participatorymethods.org/sites/participatorymethods.org/files/Participatory_Approaches_ENG%20Irene%20Guijt.pdfhttp://www.participatorymethods.org/sites/participatorymethods.org/files/Participatory_Approaches_ENG%20Irene%20Guijt.pdfhttp://betterevaluation.org/plan/approach/participatory_evaluationhttp://www.ids.ac.uk/publication/participatory-monitoring-and-evaluation-learning-from-changehttp://www.ids.ac.uk/publication/participatory-monitoring-and-evaluation-learning-from-changehttp://www.wmich.edu/evalctr/archive_checklists/qec.pdfhttp://www.povertyactionlab.org/methodologyhttp://www.povertyactionlab.org/methodologyhttp://www.povertyactionlab.org/methodologyhttp://www.povertyactionlab.org/methodologyhttp://www.wmich.edu/evalctr/archive_checklists/qec.pdfhttp://www.ids.ac.uk/publication/participatory-monitoring-and-evaluation-learning-from-changehttp://www.ids.ac.uk/publication/participatory-monitoring-and-evaluation-learning-from-changehttp://betterevaluation.org/plan/approach/participatory_evaluationhttp://www.participatorymethods.org/sites/participatorymethods.org/files/Participatory_Approaches_ENG%20Irene%20Guijt.pdfhttp://www.participatorymethods.org/sites/participatorymethods.org/files/Participatory_Approaches_ENG%20Irene%20Guijt.pdfhttp://www.innonet.org/client_docs/File/center_pubs/overview_current_eval_practice.pdfhttp://www.innonet.org/client_docs/File/center_pubs/overview_current_eval_practice.pdfhttp://www.makingallvoicescount.org/news/how-useful-are-rcts-in-evaluating-transparency-accountability-projects/http://www.makingallvoicescount.org/news/how-useful-are-rcts-in-evaluating-transparency-accountability-projects/http://www.makingallvoicescount.org/news/how-useful-are-rcts-in-evaluating-transparency-accountability-projects/http://betterevaluation.org/blog/fools_gold_widely_touted_methodological_gold_standardhttp://betterevaluation.org/blog/fools_gold_widely_touted_methodological_gold_standardhttp://betterevaluation.org/start_here/decide_which_methodhttp://betterevaluation.org/start_here/decide_which_methodhttp://www.ids.ac.uk/publication/a-mixed-method-impact-evaluation-design-of-a-mobile-phone-application-for-nutrition-service-delivery-in-indonesiahttp://www.ids.ac.uk/publication/a-mixed-method-impact-evaluation-design-of-a-mobile-phone-application-for-nutrition-service-delivery-in-indonesiahttp://www.oecd.org/derec/worldbankgroup/37010607.pdfhttp://www.oecd.org/derec/worldbankgroup/37010607.pdfhttp://go.civicplus.com/l/9522/2012-10-26/8t542http://go.civicplus.com/l/9522/2012-10-26/8t542http://www.pewinternet.org/2013/04/25/civic-engagement-in-the-digital-agehttp://www.pewinternet.org/2013/04/25/civic-engagement-in-the-digital-agehttp://www.afrobarometer.org/http://www.accenture.com/us-en/Pages/insight-digital-government-digital-citizens-ready-willing-waiting.aspxhttp://www.accenture.com/us-en/Pages/insight-digital-government-digital-citizens-ready-willing-waiting.aspx


90/172

82

This section describes how the design process nowmoves to a more detailed level to decide whattools to use within the broad method for collectingnew data, whether or not to use digital tools tocollect new data, and how data collection can beimplemented. Implementation of a DCE evaluationis broadly the same as for any evaluation so thisis not covered in depth, but some speci c tips areincluded that are of speci c relevance to technologyand citizen engagement.

Scopin

D si nin

Pl nnin & Impl m ntin

An l in

Sc nnin , R ctin & L rnin

Coll ctin n w d tTo us Di it l Tools?

Choosin th ri ht tools

4.3


91/172


92/172

84

lation groups. When dealing with non-probabilistic sampling techniques, itis important to bear in mind that digital tools for data collection, e.g., mobilephones or internet, are more prone to produce biased samples as their uptakevaries across gender, age or education levels (as demonstrated in the UgandaU-Report’s evaluation).

Match variables if secondary data is used to complement primary research, forexample, age or income bands in a survey should match with the bands used incensus data.

Re-use best-practice survey questions that have been designed and testedin other studies with similar populations or found in searchable in questionsbanks, e.g. Survey Questions Bank by UK Data Service (http://discover.ukdata -service.ac.uk/variables ).

Pilot the method, e.g. survey or interview guide, with the exact audience inwhich it will be undertaken in order to detect problems in questions or answer -ing options, spontaneous reactions of respondents, refusal rates and timingresponse rates/drop-o points.

Make sure all the data needed is being collected including any data relatedto the control group. It is easier to get this right rst time than to go back andcollect more data later.

Use of intermediaries when conducting the DCE evaluation means they maybring in their own agenda/prejudices and become more than just a channel,introducing their own biases and dynamics with the respondents.

Unintended bias in interviews as interviewees may frame answers in a partic -ular way for a variety of reasons (hence the need for triangulation). There maybe di erent e ects in responses between one-to-one interviews and groupscenarios—such as focus groups—where respondents may be in uenced byothers in the group, either openly or tacitly. Other in uences can be whetherthe interviews are recorded or not, the language they are conducted in and, if aninterpreter is involved, the biases they bring in.

Negative impact on participants as those invited to participate in DCE are of -ten time poor—especially if they are women—and may have historical reasonsto expect little responsiveness from their governments. It is important to bearin mind that these factors may in uence their response.

Consider the e ect of using technology and how it may a ect the response,for example, when a recording device is switched o , an interviewee tends to

http://discover.ukdataservice.ac.uk/variableshttp://discover.ukdataservice.ac.uk/variableshttp://discover.ukdataservice.ac.uk/variableshttp://discover.ukdataservice.ac.uk/variableshttp://discover.ukdataservice.ac.uk/variables


93/172

85

speak more comfortably; if a tablet is used to collect responses in the eld itcould be that the interest and questions from respondents are about the tech -nology itself; the respondent may even take the artefact and start interactingwith it. In some cases, people may not feel comfortable enough to admit beingunable to use or understand the device.

New tools and technologies appear constantly, and some are valuable andsome are less so. It is important not to be fooled by the marketing of the tech -nology vendors or by whether the technology is open source or proprietary. Thetools need to be researched properly, the people using them need to be spokento and there needs to be an understanding of similar tools and the bene ts andpotential pitfalls they might bring before deciding on their use.


94/172


95/172

87

Di it l innov tion supportincross-countr citi n surv sRIWI’s Random Domain Intercept Technology (RDIT) o ersan innovative way of surveying global citizens by randomlyintercepting web users in every country and territory in the world(and on all web-enabled devices, from smartphones to desktops), itis enabling evaluators to capture the attitudes of citizens in hard-to-reach regions of the world.

This o ers the potential to provide a new voice for global citizensthat can otherwise often be left out of important discussions anddecision making.

RDIT is particularly useful for evaluations requiring large-scalecross-country surveys—previous examples include innovative globalpandemic work, in conjunction with Massey College at the University

of Toronto, a 450,000 person Global Corruption Index with theInternational Association of Prosecutors; democratic engagement inIndonesia with the International Foundation for Electoral Systems;real-time tracking of citizen attitudes in West Africa towards Ebolawith BioDiaspora; and the World Bank’s Open Government Project in63 countries (www.openinggovernment.com).

Eric MeerkamperPresident, The RIWI Corporation


96/172

88

4.3.3. W i hin th us of di it l tools in th v lu tion of DCEIt is important to consider carefully the pros and cons of digital vs non-digital methods,given the inherent trade-o s associated with deciding between competing approaches.There should be no assumption that one is better than the other. Table 13 below elabo-rates on some of the bene ts and challenges related to using digital approaches.

TABLE 13. CONSIDERATIONS ON USING DIGITAL DATA COLLECTION.

Bene ts Challenges

• Lowers cost considerably, e.g. usingdi erent wordings for target groups anddi erent user interfaces (Raftree andBamberger, 2014)

• Collects real-time data

• Triangulation through date stamps/IPaddress/GPS tracking/location throughmobile number if applicable

• Greatly facilitates data collection (Raftreeand Bamberger, 2014)

• Can be conducted remotely

• Video/audio recordings, for understandingbody language including participatory video(Lunch and Lunch, 2006)

• Accurate transcriptions of interviews;in-depth qualitative approaches suchas narrative/linguistic analysis can beconducted (i.e. to understand what thespeaker puts emphasis on)

• Cost-reduction may not be the maindeterminant of the evaluation, e.g. in qualitativeresearch, better rapport may be built in askingquestions face-to-face through a local evaluator.

• Infrastructure challenges, e.g. power cuts(Farmer and Boots 2014)

• Selectivity bias, excluding those who do nothave access to the technology (Raftree, 2013)

• While there is nothing inherently wrong withtechnology transfer per se, challenges canarise when insu cient attention is paid toconsequences of transferring technology intoa new context, or where the technology itselfrather than the underlying need becomes themain driver

• Technology may be viewed suspiciously and as abarrier to rapport building

• Loss of privacy and increased levels of risk,

especially once data enters the public domain• While technology is often seen to lower barriers

of access, it can similarly lend itself to higher‘drop-o points’

• Total costs of ownership to introduce thetechnology to the target group in the long term

A key argument in favour of collecting evaluation data digitally is cost saving, forexample, “ one program in Zimbabwe recorded a $10,000 saving by switching to tablets

to survey a sample of 5000 people, as compared with using a 25 page paper questionnaire”(Raftree and Bamberger 2014, p23). The experience from the four evaluations thataccompanied the development of this guide leads to a similar conclusion, demon -strating that using digital means can reach more people more quickly at less costthan using non-digital techniques (recognizing that bias may come in other ways,and that cost is often not the only consideration).


97/172

89

At the same time, there is a clear preference for non-digital methods (interviews,household surveys, etc.) where more detailed, nuanced information is required.More detail on the speci c data collection methods and costs for these eld evalua -tions can be found in Appendix D .

4.3.4. Choosin ppropri t di it l toolsIt is tempting, and all too common, to group all digital tools as one and judge themaccordingly. Table 14 below shows some of the digital tools available and how theiruses clearly vary depending on the context.


98/172


99/172

A d v a n t a g e s

D i s a d v a n t a g e s

E x a m p l e s

D i g i t a l d a t a c o l l e c t i o n a p p s ( s m a r t p h o n e s

, t a b l e t s , n e t b o o k s )

• M u l t i p l e d a t a e n t r y ( e

. g . ,

t e x t , m

u l t i p l e c h o i c e

,

m u l t i m e d i a )

• A u t o m a t i c g e o s p a t i a l a n d t i m e

d a t a

• E a s y d a t a c o l l e c t i o n i n s m a r t p h o n e s

• R e a l - t i m e d a t a c o l l e c t i o n

• A u t o m a t i c v i s u a l i z a t i o n o f c h a r t s a n d m a p s

• G e n e r a t e r e p o r t s a u t o m a t i c a l l y ( e

. g . ,

p d f )

• A l l o w s d o w n l o a d s o f a g g r e g a t e d a t a

• T e x t - t

o - s

p e e c h f u n c t i o n a l i t i e s a v a i l a b l e

• R u n i n s e v e r a l o p e r a t i n g s y s t e m s

• S o m e a l l o w i n t e g r a t i o n w i t h S M

S p l a t f o r m s

• U p f r o n t e q u i p m e n t c o s t s ( g o o d i n v e s t m e n t i n

t h e l o n g t e r m

f o r o r g a n i z a t i o n s t h a t r e g u l a r l y

u s e d a t a )

• T r a i n i n g t i m e ( m o s t a r e u s e r - f r i e n d l y )

• S a m p l e b i a s e d ( l i m i t e d t o s m a r t p h o n e u s e r s )

C o m m c a r e

E p i C o l l e c t

F i e l d a t a

i F o r m B u i l d e r

K o b o T o o l b o x

M a g p i

N o k i a D a t a

G a t h e r i n g

O p e n D a t a K i t

O p e n X D a t a

P e n d r a g o n F o r m s

P o i M a p p e r

R I W I

T a r o W o r k z

V i e w W o r l d

O n l i n e s u r v e y s

• R e l a t i v e l y c h e a p

• E a s y t o d e s i g n

• E a s y t o i m p l e m e n t a t s h o r t n o t i c e

• F a s t d a t a c o l l e c t i o n

• I m m e d i a t e a c c e s s t o d a t a

• A l l o w s s t a n d a r d i z e d a n d f r e e t e x t a n s w e r s

• E x t r e m e l y u s e f u l w h e n n o n - r e p

r e s e n t a t i v e s u r v e y s a r e

r e q u i r e d ( e

. g . ,

t o g a t h e r i n f o r m a t i o n o n p a r t i c i p a n t s

,

c h e c k s t a k e h o l d e r p e r c e p t i o n s

o f p r o g r a m

g o a l s )

• H i g h l e v e l o f n o n - r

e s p o n s e

• E a s y f o r r e s p o n d e n t s t o d r o p o u t

• N o n - c

o v e r a g e o f g r o u p s w i t h n o i n t e r n e t

a c c e s s

• S a m p l e o b v i o u s l y b i a s e d ( l i m i t e d t o l i t e r a t e ,

i n t e r n e t u s e r s )

F l u i d S u r v e y s

F o r m S i t e

G o o g l e F o r m s

K e y S u r v e y

L i m e S u r v e y

P o l l D a d d y

Q u e s t i o n P r o

Q u a l t r i c s

S n a p S u r v e y s

S t a t P a c

S u r v e y G i z m o

S u r v e y G o l d

S u r v e y P r o

S u r v e y M o n k e y

Z o o m e r a n g


100/172

A d v a n t a g e s

D i s a d v a n t a g e s

E x a m p l e s

S o c i a l M e d i a

• U s e f u l t o s h a r e a n d g a t h e r f e e d b a c k f r o m

a l a r g e

a u d i e n c e

• L e v e r a g e s o c i a l n e t w o r k s w i t h t r u s t t i e s

• P r o v i d e s r e a l t i m e a n a l y s i s o f q u a l i t a t i v e d a t a ( a n a l y t i c a l

d a s h b o a r d s a v a i l a b l e f o r u s e r s )

• E a s y t o t r a c k t o p i c s ( b a s e d o n p o p u l a r i t y a n d h a s h t a g s )

• U s e f u l t o a n a l y z e f e e l i n g s a n d t h o u g h t s o f t h e t a r g e t

p o p u l a t i o n

• P r o v i d e s a v a r i e t y o f i n f o r m a t i o n a b o u t t h e u s e r s

( l o c a t i o n

, u s e r h i s t o r y

, g e n d e r , a g e , I P a d d r e s s )

• T h e p l a t f o r m s a r e p u b l i c b u t w i t h p r o p r i e t a r y

r e s t r i c t i o n s

• P o

s e e t h i c a l i s s u e s ( e

. g . ,

i n f o r m e d c o n s e n t o f

t a r g e t g r o u p s )

• U s e r s ’ o p i n i o n s a e c t e d b y s e l f - p r e s e n t a t i o n

b i a s

B l o g s

F a c e b o o k

M x i t ( N i g e r i a a n d S o u t h A f r i c a )

T w i t t e r

Y o u T u b e

W e A g e n t ( R u s s i a )

W e C h a t ( C h i n a )

G e o s p a t i a l d a t a t o o l s

• U s e f u l f o r t r a c k i n g i n f o r m a t i o n

, a n a l y z i n g d a t a a n d

p r e s e n t i n g u p d a t e s i n r e a l - t

i m e

• I n e x p e n s i v e

• U s e r - f r i e n d l y i n t e r f a c e s

• A l l o w s e x p l o r a t i o n t h r o u g h w e b i n s t a t i c o r i n t e r a c t i v e

f o r m a t

• A l l o w t o i d e n t i f y a r e a s u n d e r s e r v e d b y t h e p r o g r a m

a n d

u n e v e n d i s t r i b u t i o n o f r e s o u r c e s

• A l l o w s t o l i n k s u r v e y a n d q u a l i t a t i v e d a t a w i t h

g e o g r a p h i c a l i n f o r m a t i o n

• S h a r e a b l e f o r m

o f i n f o r m a t i o n

• B i a s e d a c c e s s ( i n f o r m a t i o n s u b m i t t e d t h r o u g h

s m

a r t p h o n e s o r t a b l e t s )

• P a r t i c i p a n t s ’ l o c a t i o n i s i d e n t i a b l e m a k i n g

t h e m

v u l n e r a b l e ( e

. g . ,

t o g o v e r n m e n t ,

o r g a n i z a t i o n s , o t h e r i n d i v i d u a l s )

• M a p d a t a m a y n o t b e c r o s s - c

o m p a r a b l e a c r o s s

d i e r e n t t o o l s ( e

. g . ,

m i s s p e l l i n g s o f c i t y n a m e s )

C r o w d m a p

F i r s t M i l e G e o

O p e n S t r e e t M a p

P o i m a p p e r

Q u a n t u m

G I S

R e s o u r c e M a p

U s h a h i d i


101/172

93

As can be seen from this table, bias is an important issue to consider when usingdigital tools. If samples are employed in a quantitative approach, probabilistic tech -niques that rely on random selection are more rigorous (Groves, 2009). They pro -duce representative samples whose results can be generalized directly to the origi -nal population from which the sample was drawn, e.g., identifying all users from thesystem data (sampling frame), and randomly selecting a group of these to be con -

tacted. As such, random sampling requires more resources to invite selected par -ticipants and technical expertise (e.g. multi-stage sampling). It also runs the risk,however, of introducing bias through high non-response rates, unless greater e ortis made to reach non-respondents, but this can be resource-intensive and beyondmany budgets.


102/172

94

G ttin b tt r r sults from di it l v lu tionnd n m nt tools

SMS-based surveys should be designed carefully to ensure they inspire the respondent’s interest and keepsthem engaged through topics they care about. It is di cult to logistically make SMS free for users so at timesairtime can be given to compensate for the cost of participation, or be sure to use toll free numbers. There arealso concerns about data reduction to simple yes/no answers losing nuance, so they can be supplemented withother methods. If possible put the most important questions rst to deal with drop-o rates. Where possiblerun tests before hand to evaluate best wording of the content and other factors like time of day.

IVR (Interactive Voice Response) is usually free to engage and provide answers through button pressesto give their opinion. Using voice enables interaction with those speaking minority languages or lackingliteracy skills. To maximize participation it is important to use local languages, make interactions short(e.g., 5 minutes), keep questions clear and simple and ensure the topic is of interest to the user.

Participants can be motivated when they get feedback through the same IVR channel where they gave input. After engaging with a group a follow up message can be sent to summarize the ndings or share the impact oftheir input. This is especially important for regular IVR interactions such as quarterly citizen priority surveys.

It is important to test IVR surveys to ensure they work. Factors to test include gender of the voice, time oday of message, use of incentives, wording and order of questions and phrasing of introduction.

Using hybrid tools: Radio has massive reach but is often one way with no opportunity for citizen input.Through beep to vote, SMS and IVR surveys radio can be used to encourage participation. Farm RadioInternational often uses mobile technologies to run surveys during radio shows, sometimes gettingthousands of calls ins during a single show. This creates interactive radio content and allows for the

conversation to evolve based on listener input. It also provides a great channel for promoting citizenengagement and can be targeted at speci c groups by show (e.g., on a women’s health program askabout women’s priorities for local government investment).

Running tests before launch: It can be hard to know what approach for soliciting input will work best.When you’re unsure try an A/B test where you randomly split a piece of your target audience in half anduse a di erent approach for each. For example o er a nancial incentive to participate in giving input toone group and no incentive to the other. After running your test with a small sample, choose the bettermethod to continue with. Advanced use of this will successively iterate on a number of dimensions of your engagement (e.g., incentive, tone, length, structure) to truly optimize.

Create a forum for on-demand citizen input: It is very simple to set up and promote a hotline wherecitizens can call free of charge and provide input on key policy or programs questions. This structureputs the power in the hands of the citizen for when and how they provide input. The hotline can be openended where citizens provide broad input or it can be focused with speci c questions to be answeredthat will inform tangible decisions. The phone-based input can be instantly shared online so thereis transparency on what kind of feedback is coming in. An example cou

Documents

World Bank: Evaluating Digital Citizen Engagement