Measuring and visualizing cost- effectiveness for ... – cost... · the capacity-development approaches of our AuthorAID project. What seemed very compelling at first sight, however,

Measuring and visualizing cost-effectiveness for development interventions

Non-governmental organizations working in international development are keen to “[…] maximise the impact of each pound spent to improve poor people’s lives”.1 Often they need to provide evidence of this to donors. A cost-effectiveness analysis can give insight into some aspects of the value for money (VfM) of development programmes. Cost-effectiveness is an overarching feature of DFID’s approach to VfM comparing inputs directly with effects for a given activity. In contrast to a cost-benefit analysis, effects in a cost-effectiveness analysis are not monetarized.

In order to conduct such analysis, there are two important considerations: First, data on costs and effectiveness per activity needs to be available. The types of costs to include and a definition of effectiveness need to be clear from the beginning. Second, the activities of interest need to be reasonably comparable to each other in their goals and approaches.

In the following we will describe the approach taken to analyse the cost-effectiveness for INASP’s AuthorAID programme. Firstly, a short overview about AuthorAID and its different activities is provided. Secondly, the methods for measuring costs and effectiveness are described.

Thirdly, the results are visualized and discussed. The fourth and last step critically reflects on the shortcomings of the method employed.

AuthorAID in brief INASP’s AuthorAID is a wide-

Cost-effectiveness analysis is useful but may not tell the whole story

At INASP we want to demonstrate the value for money of our programmes to donors. To this end, at the beginning of 2016 we conducted a cost-effectiveness analysis of the capacity-development approaches of our AuthorAID project. What seemed very compelling at first sight, however, proved to be an inadequate way to reflect the complexities and realities of how AuthorAID operates on a daily basis. For example, the sustainability aspects of the AuthorAID approaches are difficult to factor into current costs, and the organizational support provided to institutions may translate into measurable effects only in the mid- and long term. In addition, different approaches have their own particular advantages that play out in specific contexts only.

Learning, Reflections & Innovation @ INASPProject evaluation October 2016

@INASPinfowww.inasp.info

There is no linear relationship between invested money and effects for development programmes. Many other factors have to be taken into account

Sto

ck p

hoto

Cop

yrig

ht :

Wee

rapa

t Kia

tdum

rong

ranging programme supporting Southern researchers to gain the confidence, knowledge and skills needed for publishing and communicating their research:

The AuthorAID website provides guidance and information resources to a wide audience. The site includes discussion forums and downloadable training resources. Through the website, users can join a mentoring system linking early-career with more experienced researchers.

The learning platform Moodle is used to provide online training in different aspects of research writing to small-group intensive online courses and large-group open online courses from across geographical borders.

AuthorAID awards travel grants to southern researchers to attend international

conferences, to present papers, and to learn from others in their field. There are also workshop grants for researchers to run local face-to-face training for other researchers in a cascade fashion.

For universities and research institutions, INASP’s AuthorAID provides organizational level support to independently run research training courses and to embed them in post-graduate curricula or professional development programmes of research staff.2

Data collection on costs and effectivenessData on programme costs

Information about direct expenses and staff time was collected for each of the above-mentioned

activities (the AuthorAID website, mentoring system, open online courses, intensive online courses, travel grants, face-to-face workshop grants, and embedding face–to-face courses in institutions).

Direct expenses were gathered through contracts and from the INASP finance team. The staff time invested, however, proved challenging to collect. Programme staff monitor their time only for AuthorAID as a whole and not separately for each of the capacity building approaches. However, if a particular activity occurred during the period when the study was conducted, staff were asked to monitor their time for this particular activity. This was the case for the time spent on the selection process for the travel and workshop grants. Otherwise, staff time invested on each activity was estimated based on funding proposals and by the programme

[email protected]


New features to the online mentoring system mean that mentors and mentees can be awarded electronic badges for mentoring and training tasks completed

staff themselves. Invested staff time was then monetarized based on INASP internal day rates.

The costs were usually calculated per workshop or time interval, i.e. for one workshop or for one month. Then, costs were divided by the number of beneficiaries per workshop or per month. Thereby, monitoring data from the last few years was used in order to account for variances over time. For example, rather than using the data from the most recent training activity, with, say, 23 participants in the denominator, the number of training participants were averaged across all training activities in the last two years.

Due to the various ways of estimating costs, different measurement errors apply. It might be that the costs estimated for the grants are much more reliable than costs estimated for the online courses because the former were monitored directly, while the latter were based on staff estimates of time spent. And while there is an understanding about which costs are likely to be overestimated and which ones are perhaps underestimated, further differentiation was stopped at a point where it seemed appropriate. It was found that the cost-differences of the approaches were quite substantial so that changes in the input side would not affect the final data much (see Table 1 on page 7).

Some capacity-building approaches may occur in future without additional costs to INASP’s AuthorAID. For example, embedding research-writing training courses in university curricula or staff development

programmes means that these courses will be run in the future without financial support from INASP, and that current costs would need to be discounted accordingly. But in this cost-effectiveness analysis, the assumption was made that courses would run only for the time agreed and paid for in the memorandum of understanding with the institutions. As the embedding process is relatively new (most institutions are still within the initial two-year period of INASP support), there is no data available yet about courses run beyond the embedding agreement (see the results section for more information).

Data on programme effectiveness

With its different activities, AuthorAID wants to increase the capacity and confidence of early-career researchers to publish and communicate their work. This goal underlies all activities even though there may be other goals as well. From this common goal, four different effectiveness categories were derived: direct publications, confidence in research writing, networking and audience reach. These categories were assessed for each activity in two steps.

First, an online survey was conducted with all direct beneficiaries of AuthorAID in 2015. In this survey, a total of 830 respondents (30% response rate) participated, reflecting well the overall characteristics of the AuthorAID community. Survey respondents were asked about how they participated in AuthorAID and what the benefits were. While the first question helped

to categorize respondents into the different activities (i.e. online course, face-to-face training, etc.), the second open question was used to determine the effects (e.g. “Because I took the course, I am much more confident in writing a paper”). It was then counted how often different effectiveness categories were mentioned and a relative value was determined based on the number of respondents per activity in the survey.

It was important that the effectiveness data originated from one source in order to keep measurement errors at the same levels for all of the activities. For example, we know that online surveys are subject to selection biases: people are more likely to participate in an online survey if they use the internet in their day-to-day work. In that way, the survey data is biased by not taking into account the effects of AuthorAID for people who are not using internet very often. The assumption was that the activities considered for this cost-effectiveness analysis suffer from this selection and other biases to a similar extent. This is why data is comparable among the different activities.

However, it was known that some activities had a very small number of beneficiaries and hence were underrepresented in the survey, which might skew results. In order to have commensurate data for these activities as well, a second step was taken.

In this second step, survey results and additional key evidence were reviewed with the AuthorAID programme team. Each activity

3 @INASPinfowww.inasp.info


Costs per activity per month or per training

Number of beneficiaries per month or per training = Costs per beneficiary

was rated for each effectiveness category on a scale from 1 to 5 (low = 1, low-medium = 2, medium = 3, medium-high = 4, high = 5). The least and the most effective activities (=benchmark) receiving a “low” or “high” effectiveness rating were identified. The remaining activities were then distributed along that continuum depending on the data and their relative position to the benchmark, i.e. most or least effective activity (see Table 1).

Arguably, this second step was not the most rigorous way of synthesizing the evidence because it was not led by a formal and explicit process but was consensus oriented. However, this allowed the inclusion of all activities in the analysis (which was important) and the calculation of an average effectiveness measure for each activity.

Visualization and results of the cost-effectiveness analysisFor each activity there were costs per beneficiary and an average effectiveness value available. Both variables were plotted in a simple

diagram (see Figure 1).

Figure 1 is quite compelling. It shows the most effective activity in relation to input costs. There are, however, different shortcomings of this visualization. The first one is that effectiveness is a synthesis of different values, and some activities may be more effective than others depending on which effectiveness category one looks at. This problem is easily solved by plotting a cost-effectiveness diagram for each effectiveness category separately (see Figure 2-5).

And indeed, the resulting diagrams show that the activities vary according to which effectiveness measure is looked at. It is important to know which effectiveness measure to consider and with which weight, i.e. is networking more important than research writing confidence? Should networking be considered at all?

First results

Each capacity-building approach has its own strengths and constraints that play out differently depending on the

context. While disregarding specific benefits of each approach (see Table 1), more general findings of the cost-effectiveness analysis are reported in the following:

Face-to-face (workshop grants) and online courses (intensive and open online courses) increase research and proposal writing confidence levels. According to the AuthorAID Survey 2015 data, research and proposal writing confidence levels are significantly increased when respondents participated in online or face-to-face courses. There may be several reasons for this finding, for instance that the courses provide a ‘safe’ space for researchers to ask questions and to share ideas. This is similarly true for the mentoring support.

Generally, the more effective capacity-building approaches are also more expensive. However, some approaches seem more cost-effective than others. As such, travel grants are more than twice as expensive as the second most expensive approach (mentoring) yet they are not twice as effective in terms of direct publications or networking.

[email protected]


Figure 1: Cost-effectiveness of AuthorAID capacity-building approaches. Costs were calculated per beneficiary or training course completer. Effectiveness consists of ratings regarding direct publications, research and proposal writing, confidence levels, networking, and audience reach. These ratings are not absolute but have to be seen in relation to the other capacity-building approaches. The difference between Low and Low-Medium may not be the same as the difference between Low-Medium and Medium.

Effectiveness synthesis

Cost

s pe

r be

nefi

cary

Low Low-medium Medium Medium-high High

£1,200

£1,000

£800

£600

£400

£200

£0

F2F (

embe

ddin

g)

Ope

n on

line c

ours

e

AA W

ebsit

e

F2F (

gran

ts)

Onl

ine c

ours

e (in

tens

ive)

Men

torin

g

Trav

el gr

ants

The reason is that costs for travel grants increase almost proportionally to the number of beneficiaries. In contrast, the costs of running the AuthorAID mentoring system are not affected by the number of beneficiaries.

On the other side of the effectiveness spectrum, face-to-face courses run by embedding partner institutions appear more expensive but are less or only equally effective in terms of direct publications and writing confidence than face-to-face courses run by grantees or open

online courses. The reason for this may be that the embedding work is in its early stages at the time of this analysis and that hence effects have not yet materialised or been measured in the survey.

The more expensive capacity-building approaches benefit a very small number of individuals only. Currently, all three most ‘expensive’ approaches (travel grants, mentoring system, and intensive online courses) benefit only a very small number of individuals. However, looking forward, mentoring has the

prospect of benefitting a larger audience to increasingly lower costs whereas travel grants and intensive online courses do not. The costs of the latter approaches will roughly stay the same no matter the number of beneficiaries. This is because the travel grant sum makes up for most of the associated costs. Similarly, for intensive online courses the greatest costs and effects regarding direct publications derive from the close supervision by the course facilitators. The more participants there are on an intensive online course, the more paid facilitators would be needed to keep up course effectiveness.

In general, the more effective (and more expensive) capacity-building approaches involve a higher facilitator to beneficiary ratio than the less expensive initiatives. For example, there is one travel grant for one grantee, one mentor for one mentee, and one facilitator for five intensive course participants, but there is one website for thousands of visitors, one open online course facilitator for hundreds of participants, etc. Hence, these latter approaches are more effective in terms of audience reach.

Online courses can be more cost-effective than face-to-face courses. Firstly, one has to view the open online course and the intensive online course separately. Open online courses provide a very low cost training to many participants (many more than 500) at the same time and are more effective in terms of writing confidence, audience reach and direct publications (but not networking) than face–to-face courses as run by embedding partner institutions.3

If the intensive online course were to be run face-to-face, the




Effectiveness networking

Cost

s pe

r be

nefi

cary


£1,200

£1,000

£800

£600

£400

£200

£0

F2F (

embe

ddin

g)

Ope

n on

line c

ours

e

AA W

ebsit

e

F2F (

gran

ts)

Onl

ine c

ours

e (in

tens

ive)

Men

torin

g

Trav

el gr

ants

Figure 2-3: Separate visualizations of costs per effectiveness category

Effectiveness direct publications

Cost

s pe

r be

nefi

cary

£1,200

£1,000

£800

£600

£400

£200

£0 O

pen

onlin

e cou

rse

AA W

ebsit

e

F2F (

gran

ts)

Men

torin

g

F2F (

embe

ddin

g)

Onl

ine c

ours

e

(in

tens

ive)

Trav

el gr

ants

costs would be much higher due to the need for the international travel costs of the participants and facilitators, full board and facilities in addition to the existing facilitator and administration costs. The workshop grants offer better effectiveness in terms of networking but do not contribute to writing confidence at higher costs per training completed than the intensive online courses by AuthorAID.

The AuthorAID website, with its materials and the discussion group, is a very inexpensive yet a very important complement to all other capacity-building approaches. The provided materials and forum supplement well the more direct support provided by AuthorAID. They provide networking opportunities, a learning platform, and facilitate knock-on effects such as improved teaching materials.

The face-to-face embedded training courses would have to run for an additional two-year period in order to be financially competitive with the next cheaper approach. Due to the focus on sustainability for the face-to-face embedding courses, data was extrapolated into the future. This allows an assessment about how often the embedded courses would have to be run within the partner institution, i.e. beyond the initial two-year programme duration, in order to be competitive in terms of costs with face-to-face courses by workshop grants. If the courses were run for an additional two-year period by the institutions, i.e. four additional training courses, then the costs per beneficiary would be the same as face-to-face training activities financed through the workshop grants. This assumes that two courses are run each year with 20 participants respectively.

Reflections on cost-effectiveness analysisThere are some problems with this cost-effectiveness analysis and the way it is visualized.

Comparing apples with pears can lead to premature decisions

One has to ask how useful it is two analyse very distinct approaches on the basis of costs and effectiveness only. A website with information

resources is fundamentally different from a capacity-building approach that targets an organization or an online course. The cost-effectiveness analysis considers only the most common denominator, i.e. costs and effectiveness in terms of publications or writing confidence, etc., but disregards important qualitative and contextual aspects. This may lead to false decisions. For instance, one may come to the conclusion that the work around embedding face-to-face training in institutions is not effective enough for its cost (see Figure 1).

[email protected]


Effectiveness writing confidence

Cost

s pe

r be

nefi

cary


£1,200

£1,000

£800

£600

£400

£200

£0

F2F (

embe

ddin

g)

Ope

n on

line c

ours

e

AA W

ebsit

e

F2F (

gran

ts) O

nlin

e cou

rse

(in

tens

ive)

Men

torin

g

Effectiveness audience reach

Cost

s pe

r be

nefi

cary


£1,200

£1,000

£800

£600

£400

£200

£0 O

pen

onlin

e cou

rse

AA W

ebsit

e

F2F (

gran

ts) O

nlin

e cou

rse (

inte

nsive

)

Men

torin

g

F2F (

embe

ddin

g)

Trav

el gr

ants

Figure 4-5: Separate visualizations of costs per effectiveness category



Table 1: Costs and effectiveness of different AuthorAID (AA) capacity-building approaches

Open online Online Face-to-face Face-to-face Mentoring AA Website/ Travel course course courses courses Online community grants (direct AA) (intensive course) (grants) (embedding) & resources

Effectiveness Low-Medium High Low Low High Low High direct publications

Effectiveness Medium-High High High Low-Medium Medium-High Low NA writing confidence

Effectiveness Low Low-Medium Medium Medium High Medium High networking

Effectiveness Medium-High Low Medium Low-Medium Low-Medium High Low audience reach4

Costs per £47 £453 £127 £252 £514 £0.4 £1,159 beneficiary

Facilitation 1:250 1:7 1:20 1:20 1:1 NA 1:1 / beneficiary ratio

Contextual advantages

Contextual disadvantages

Conducted online saving logistical costs advantages

Is open to participants with different backgrounds, in different time zones and geographic location

Caters to low-bandwidth needs

Targets a very broad audience

Conducted online saving logistical costs

Thematically targeted

Direct link to higher tier / quality journals in which to publish

Papers are worked on during the course

Links local facilitators to a familiar audience

Course adaptation to local needs (sometimes)

Links local facilitators to a familiar audience

Course adaptation to local needs creating ownership (sometimes)

Builds institutional capacities

Targets staff capacity development in research institutions

Potential for greater sustainability

Conducted online saving logistical costs

Provides a flexible volunteer model

Time of the mentor is voluntary and not reimbursed

Current costs for improving the functionality of the mentoring system developing costs are likely to reduce and pay off over time

Small maintenance costs

Complementing all other support types in their effectiveness

Provides opportunity for making visible southern research

Participants may not be at stage of writing a paper when they take the course

Targets explicitly participants who are in the process of writing a paper

Less control over quality of training

Institutional capacity building can takes a long time

Not every institution has sufficient demand for the course

The extent to which the courses will be run sustainably in each of the embedding institutions is not yet known

Number of finished tasks is currently not proportionate to developing and maintenance costs of the online system

Benefits only few selected individuals


However, that would disregard the high potential for sustainability that this approach has – although sustainability was not assessed in this analysis.

Synergy costs and effects are hard to untangle

AuthorAID is a programme with many interlinked activities that mutually reinforce each other. For example, through the mentoring system there is a growing pool of researchers available that is drawn upon for the facilitation of AuthorAID’s open online courses. A greater number of online course facilitators provides a greater depth of knowledge and expertise available for the participants, allows online discussion across time zones and creates networking opportunities. This simple interlinkage between the mentoring system and the online course suggests that costs and

effects cannot easily separated for the analysis.

Another example comes from AuthorAID’s embedding work, which intends to build institutional capacities. The goal is to incorporate the research-writing training course into staff development programmes or postgraduate curricula and to have a pool of institutional trainers to run the course on a regular basis. This institutional-level goal is closely linked to goals at an individual level for the academic staff and students, i.e. resulting in direct publications; networking; and writing confidence. While both types of goals at an institutional and individual level reinforce each other, costs were not separated per goal. All costs that are linked to the first goal, i.e. institutional capacity building, have been included in the analysis as well, even though only the second

goal was looked at in terms of effectiveness. In that way, this cost-effectiveness analysis did not take into account all intended goals and effects (such as institutional-level goals) of an activity but only its full costs. The reason for this was that the costs per goal are hard to differentiate and because goals are mutually interlinked.

Costs have not been discounted for sustainability aspects

AuthorAID begins the face-to-face course embedding work with universities and research institutions with a grant to cover the main facilitation and logistical costs for an initial two-year period. It is intended that the courses will be adopted into curricula for PhD and Masters’ students or for professional development schemes of research staff and

[email protected]

DFID’s 3Es (2011) and INASP’s approach to value for money:

Economy: Are we or our agents buying inputs of the appropriate quality at the right price? (Inputs are things such as staff, consultants, raw materials and capital that are used to produce outputs).

Efficiency: How well do we or our agents convert inputs into outputs? (Outputs are results delivered by us or our agents to an external party. We or our agents exercise strong control over the quality and quantity of outputs).

Effectiveness: How well are the outputs from an intervention achieving the desired outcome on poverty reduction? (Note that in contrast to outputs, we or our agents do not exercise direct control over outcomes).

Cost-effectiveness: How much impact on poverty reduction does an intervention achieve relative to the inputs that we or our agents invest in it?

In addition to the above, INASP considers a fourth ‘E’ and takes a proactive approach to value for money (VfM).

Equity: being concerned about who benefits from our work and ensuring that relatively excluded groups benefit.

INASP ensures that all our services and outputs offer the best possible VfM to all our funders. While we aim to achieve lower costs – because this helps to keep activities affordable for partners in the future and hence supports sustainability – we do not seek low costs at the expense of quality, and in some cases are willing to cover higher costs when we believe this will enable us to achieve more.

9

lecturers. In this way the course can be offered on a long-term basis. Despite this focus on sustainability, costs were only considered for the activities conducted in the initial two-year period. This is because there is not yet any information available yet about the sustainability of the courses.

It is important to take into account mid- and long-term effects

AuthorAID builds institutional capacities to embed and run research-writing courses. Institutional capacities are built through training of trainers in pedagogy skills or support to the initiation of mentoring programmes. These institutional capacities are likely to translate into the benefits as measured in this cost-effectiveness analysis only in the mid- and long-term. For instance, the right organizational environment for supporting publications of researchers has to be built up gradually over time. Since AuthorAID’s embedding

work is still in its early stages, these longer-term effects are not reflected in the survey data from 2015 used for effectiveness in the analysis.

Relative versus absolute measures

The costs per beneficiary provide absolute information about an activity if measurement errors are neglected. In that way, they can be compared to any other costs per beneficiary. However, this is not the case for effectiveness measures. The effectiveness measures of this analysis provide only relative and not absolute insights. This means that activities which are rated to be ‘low’ in effectiveness in this study may be very effective when compared to other activities that are not included in the analysis.

If genuinely different effectiveness categories are supposed to be synthesized into one, they would need a common scale. For instance, one can measure writing confidence of online training participants as well as

the number of people they met (networking category), but how can you reasonably combine writing confidence levels and networking measures? Only, if they are scaled in the same way, e.g. low-high or 1-5, can they be summed or averaged. For example, a low (1 point) on networking effectiveness and a medium-high (4 points) on direct publications results in a medium (3 points) average effectiveness measure. But ‘medium effectiveness’ of an activity is an artificial assessment; it does not mean anything if considered by itself alone. It becomes meaningful only if viewed in relation to other activities that underwent the same assessment procedure.

ConclusionsIn conclusion, cost-effectiveness analyses can be a very powerful tool. It provides quick information about the general picture. However, this picture may disregard important aspects, particularly if different measures are combined into one. In that case, critical reflection is required to determine whether the data that fed into the overall pictures and diagrams sufficiently reflects the different qualitative and contextual benefits of a capacity-building activity.

@INASPinfowww.inasp.info


INASP TeamJan Liebnitzky, Monitoring and Evaluation Officer

Ruth Bottomley, Senior Programme Manager, Research Development and Support

Andy Nobes, Programme Officer, Research Development and Support

Julie Walker, former Programme Manager, AuthorAID

1. DFID’s Approach to Value for Money, July 2011.

2. For a discussion about the pros and cons of the online courses, mentoring system and the embedding work, please see: www.inasp.info/en/publications/details/222.

3. Please note that this cost-effectiveness analysis did not take into account important mid- and long-term effects of the AuthorAID embedding work. See page 5 for a discussion on this point.

4. This is based on number of direct beneficiaries.

The AuthorAID online research-writing courses are based on INASP’s Moodle platform and attract over 1000 participants

Documents

Measuring and visualizing cost- effectiveness for ... – cost... · the capacity-development approaches of our AuthorAID project. What seemed very compelling at first sight, however,