ALINe working paper  

1

   

Working Paper 7 Evaluation of capacity enhancement fellowships for agricultural development: A case study of the Rothamsted International African Fellows Programme (AFP) Judy Mann Chaussalet

May 2012

ALINe working paper  

2

   

About ALINe

The Agricultural Learning and Impacts Network (ALINe) is an initiative between the Institute of Development Studies, UK and other partners, with funding from the Bill & Melinda Gates Foundation (BMGF). ALINe combines practical experience and technical

expertise with cutting-edge academic research in order to learn from farmer experience, share good practice and pilot new approaches to monitoring and evaluation. ALINe utilises a people-centred performance measurement that is responsive to poor farmers, offering the potential to re-incentivise and re-align the aid system. ALINe has multiple clients in the public, private and third sectors and a portfolio currently in 9 countries with emphasis in Sub-Saharan Africa and South Asia. Visit our website: www.aline.org.uk

About IDS

The Institute of Development Studies is one of the world's leading charities for research, teaching and communications on international development. Founded in 1966, the Institute enjoys an international reputation based on the quality of its work and the rigour with which it applies academic skills to real

world challenges. Its purpose is to understand and explain the world, and to try to change it – to influence as well as to inform. IDS hosts five dynamic research programmes, five popular postgraduate courses, and a family of world-class web-based knowledge services. These three spheres are integrated in a unique combination – as a development knowledge hub, IDS is connected into and is a convenor of networks throughout the world. The Institute is home to approximately 80 researchers, 50 knowledge services staff, 50 support staff and about 150 students at any one time. But the IDS community extends far beyond, encompassing an extensive network of partners, former staff and students across the development community worldwide. Visit our website: www.ids.ac.uk

1

Evaluation  of  capacity  enhancement  fellowships  for  agricultural  development:  A  case  study  of  the  Rothamsted  International  African  Fellows  Programme  (AFP)  Judy  Mann  Chaussalet    ALINe  Working  Paper  All  rights  reserved.  Reproduction,  copy,  transmission,  or  translation  of  any  part  of  this  publication  may  be  made  only  under  the  following  conditions:  •    with  the  prior  permission  of  the  publisher;  or  •    with  a  licence  from  the  Copyright  Licensing  Agency  Ltd.,  90  Tottenham  Court  Road,  London  W1P  9HE,  UK,  or  from  another  national  licensing  agency;  or  •    under  the  terms  set  out  below.    This  publication  is  copyright,  but  may  be  reproduced  by  any  method  without  fee  for  teaching  or  nonprofit  purposes,  but  not  for  resale.  Formal  permission  is  required  for  all  such  uses,  but  normally  will  be  granted  immediately.  For  copying  in  any  other  circumstances,  or  for  reuse  in  other  publications,  or  for  translation  or  adaptation,  prior  written  permission  must  be  obtained  from  the  publisher  and  a  fee  may  be  payable.  

2

         Available  from:  Communications  Unit,  Institute  of  Development  Studies,  Brighton  BN1  9RE,  UK  Tel:   +44  (0)  1273  915637    Fax:   +44  (0)  1273  621202  E-­‐mail:   bookshop@ids.ac.uk  Web:   www.ids.ac.uk/ids/bookshop    IDS  is  a  charitable  company  limited  by  guarantee  and  registered  in  England  (No.  877338)  

ALINe working paper  

3

Evaluation of capacity enhancement fellowships for agricultural development: A case study of the Rothamsted International African Fellows Programme (AFP)

Judy Mann Chaussalet

Judy Mann Chaussalet is an Independent Consultant, previously working at Rothamsted Research and managing the Rothamsted International team. She has a biological science background and considerable experience in managing individual approaches to capacity development.

Santiago Ripoll Lorenzo is an independent consultant. Originally trained as a vet, he did a PhD in anthropology and has a broad range of experience in food security and project evaluations.

Johanna Lindström is a political scientist with research interests in the areas of monitoring and evaluation; agricultural development, food security and nutrition; gender and public awareness of aid and international development. Having worked as a Research Officer for five years with IDS and ALINe, she moved to work as an Evaluator with the Swedish Agency of Development Evaluation (SADEV), and is currently working at the Swedish Institute for Public Administration (SIPU) International. She has an MA in Environment, Development and Policy.

ALINe working paper  

4

   

Andre Ling is a Research Officer with ALINe with more than six years of experience working in India with a wide range of approaches to M&E, including traditional, participatory and multi-stakeholder approaches and methods, using both quantitative and qualitative data.

Yvonne Pinto is the Director of ALINe, Institute of Development Studies (IDS). She delivered the first Global Conference Agricultural Research for Development (GCARD) in March 2010, for the Global Forum on Agricultural Research (GFAR) and the CGIAR. Yvonne has spent 13 years in global philanthropy as a Program Officer in the Bill and Melinda Gates Foundation’s Agricultural Development team and as Executive for the Developing Countries portfolio for the Gatsby Charitable Foundation in London. She has a background in fundamental science, programme management and institutional development and change.

Acknowledgements The authors gratefully acknowledge the Gatsby Charitable Foundation who sponsored the Rothamsted International African Fellows Programme, and Rothamsted Research who hosted the implementation of the programme. We are particularly grateful for their willingness in allowing the programme to be evaluated in this study.

ALINe working paper  

5

 Acronyms AFP Rothamsted International African Fellows Programme

CD Capacity development

GCF The Gatsby Charitable Foundation

KII Key Informant Interviews

M&E Monitoring and Evaluation

RI Rothamsted International

SSA Sub-Saharan Africa

ToC Theory of Change

   Figures Figure 1: illustration of the AFP theory of change.

Figure 2: Cohort 1 (2012) – training

Figure 3: Cohort 2 – training

Figure 4: Cohort 1 (2012) – overall rating of programme

Figure 5: Cohort 2 – overall rating of programme

Figure 6: Cohort 1 (2012) – recognition

Figure 7: Cohort 2 (2012) – recognition

Figure 8: Cohort 1 (2012) – value of time with research supervisor

Figure 9: Cohort 2 – value of time with research supervisor

Figure 10: Cohort 1 (2009) – contact with European institute

Figure 11: Cohort 1 (2012) – contact with European institute

Figure 12: Cohort 2 – contact with European institute

Figure 13: Cohort 1 (2012) – problems at home institute/university

Figure 14: Cohort 2 – problems at home institute/university

Figure 15: Cohort 1 (2012) – end-users of research

Figure 16: Cohort 2 – end-users of research

Figure 17: Cohort 1 – activities to publicise fellowship work

Figure 18: Cohort 2 – activities to publicise fellowship work

 

List of Acronyms and Figures

ALINe working paper  

6

Contents

Author notes 3 Acknowledgements 4 List of Acronyms and Figures 5 1 Background and General Introduction 7 2 Capacity Development: An Introduction 9 3 The RI African Fellows Programme: An Introduction 11 4 The Challenge of Evaluating the Impact of Capacity Development 13 Programmes: Fellowships 5 Methodology of the Study 17 6 Key Findings 22 7 Lessons Learnt 34 8 Conclusions 39 9 Appendices 41 A. AFP Fellows 41 B. Key Informant Interview Questions 47 10 References 49

ALINe working paper  

7

1. Background and General Introduction Capacity strengthening1 is gaining an increasingly prominent position in international development allowing sub-Saharan Africa (SSA) to compete with growing global technological and economic development advancements. This has far-reaching implications in all sectors. In agriculture, Africa has to be in a position to achieve food and nutritional security by overcoming the technology gap. It can achieve this through strengthening regional and national innovation systems to develop and adapt products appropriate for Africa’s use (Conway et al., 2010). Despite a noticeable withdrawal of donor investment in many aspects of tertiary education across SSA over the past two decades, the need to strengthen innovation systems has provided the opportunity for numerous examples of capacity strengthening programmes, employing a variety of models of intervention. Many of them have been both popular and successful. However, evaluation over the longer term has proved difficult, with donors being particularly challenged by gauging the return on their investments. So the renewed interest in capacity strengthening for international development has brought with it a debate about how best to monitor and evaluate capacity strengthening, efficiently and effectively. This study aims to contribute to this debate, looking particularly at individual approaches to capacity development. Individual approaches to capacity development have traditionally been Scholarship and Fellowship interventions, involving the provision of direct training. Long-standing programmes with a particular ‘Higher Education’ focus such as the UK Commonwealth Scholarships, the Australian Development Scholarships and the European Commission Marie Curie Fellowships, are well known and have a considerable reputation in terms of their quality provision of training and contributions to the development of many individuals. However, it is not clear what they contribute to the wider community in terms of international development. While conventional impact evaluations capture the more immediate and practical outcomes of Higher Education investment, there are societal benefits to Higher Education interventions which relate to social awareness and change, and which act over long time periods (LIDC, 2012). There is an increasing awareness that even though the individual is the direct focus of the intervention, long-term and sustainable capacity development is a complex process dependent on a range of different stakeholders and influencing players, requiring changes in systems and shifts in power relationships. To ensure sustainable capacity development, these factors must be taken into account when designing and planning programmes, building on the lessons learnt from past and on-going interventions. Through examining the contributions of these initiatives to change, successful practices can be identified and fed into a broader understanding of capacity development. The Agricultural Learning and Impacts Network (ALINe), at the Institute of Development Studies (IDS) initiated this study into the evaluation of methods for capacity development activities in agricultural research and development. ALINe’s interest stems from a drive to understand better how such programmes are evaluated in order to support the cluster of capacity initiatives in which it is involved. The aim of the study is to draw lessons for learning on how best to design and plan person-focused capacity development programmes for impact. The first part of this study focused on reviewing the current literature available on person-focused capacity development in SSA and to examine how these programmes have been evaluated, allowing an understanding of current thinking; the second stage was to develop a theory of change on a specific case study; and, the third part was to conduct an evaluation to test the applicability of this theory.

a) Literature Review

                                                                                                               1  Also known as capacity building or capacity development; see Section 3.  

ALINe working paper  

8

A literature review2 was conducted to give an overview of the major literature on individual approaches to capacity development, within agriculture, but more broadly including health and economics. Lessons learnt from the agricultural research and development literature were explored in terms of assessing the impact of capacity building schemes in research. The review frames individual approaches to capacity such as fellowship and other tertiary education schemes within the capacity development literature, and outlines existing evaluation tools used to assess fellowships by different donors and research organisations. It reports on methods used by different initiatives (mostly research fellowships) to assess the impact of their activities on fellows, on their organisations, and on society more broadly. Summaries are made of the current debates in capacity development and where individual approaches fall within these debates, noting the focus of programmes on hard versus soft skills development. Also included are the different elements that have been used in previous evaluations and some specific challenges raised in assessing the impact of fellowships, such as the complex and non-linear nature of interventions. The full review is not included in this report but a summary is presented in Section 3 below. A lack of appropriate and effective monitoring and evaluation has often hampered the ability to demonstrate the effectiveness of CD programmes, and opens up the question in donors’ minds of value for money. Much of the difficulty in demonstrating effectiveness is linked to the complex nature of the systems involved.

b) Creating the Theory of Change (ToC)

In 1995, after being faced with difficulties in evaluating complex community interventions, Carole Weiss advanced the idea of basing evaluation on the ‘theories of change’ that underlie an initiative. Since then, theory-based evaluation has been found to be extremely useful. The concept is built on the idea that a programme of intervention is based on explicit or implicit theories about how and why the programme will work, and once these have been mapped out in detail, the programme can be assessed against the extent to which these theories hold.

Based on this a second part of the current work involved the development of a Theory of Change in a specific study. Although a ToC would normally be developed prior to the commencement of the programme, here it was developed retrospectively after the programme was finished, to explore its use as an evaluation tool. The case study chosen was a moderately sized agricultural development intervention, the Rothamsted International African Fellows Programme (AFP). AFP was a capacity strengthening initiative in the scientific sector, focused on mid-career individuals.

c) Online Survey and Key Informant Interview Evaluation

The third part of this work highlights undertaking an evaluation through online surveys and Key Informant Interviews (KII) to test whether the assumptions outlined in the ToC actually hold. The survey built on monitoring data of the original programme, information drawn out from the literature review and the detailed assumptions outlined through the ToC. Results of the evaluation have been outlined and discussed in terms of the benefits of using a ToC in the evaluation process.

                                                                                                               2  This review will be available as a separate paper.  

ALINe working paper  

9

2. Capacity Development: An Introduction  What do we mean by capacity development? Often the terms ‘capacity building’, ‘strengthening capacity’ and ‘developing capacity’ are used with different meanings. In the late 1980s and early 1990s capacity development emerged recognising that the failure of previous approaches (‘technical assistance’, ‘technical cooperation’) ignored existing capacities. Donors and organisations began to understand that capacity development is an internal process that involves the main actors taking primary responsibility for change processes (UNDP 2009; Simister and Smith 2010). Yet this internal emphasis does not mean that external support is not useful or desirable, but that external support should catalyse and facilitate internal processes of change. A useful definition for capacity is that used by the OECD: ‘the capability of people, organisations and society as a whole to manage their affairs successfully’; and capacity development is a ‘process whereby people, organisations and society as whole, unleash, strengthen, create, adapt and maintain capacity over time’ (OECD 2006: 12). This definition illustrates that capacity development occurs in at least three levels: the individual, the organisation and broader society. Different development organisations place varying emphases on different levels. Individual approaches to capacity are well documented, focusing on capacity primarily as a ‘human resource issue related to skills development and training’ (Simister and Smith 2010: 4-5) at the individual level. The organisational level is most widely used as the locus of international development intervention, yet the majority of the actual ‘inputs’ involve training at the individual level (Taylor and Clarke 2007). Finally there is the broader society level, which encompasses both the enabling environment that sets the overall scope for capacity development and the ultimate target of development interventions to influence ‘good change’. Few explore how the links between the organisational level and individual level work, and it is important not to assume that training and personal learning automatically leads to organisational learning. While different, these three levels influence each other in a fluid way and therefore it is particularly valuable to view them in an integrated ‘systems’ approach. Thus an important question is:

Do schemes like fellowships acknowledge and integrate into their programmes the relational nature of individuals, organisations and society or do they leave organisations and society out of the equation?

Or perhaps for this particular study:

Did the individual approaches (acquired or developed knowledge by individuals) become institutionalised in some way?

There are two main models for individual approaches to CD based on their differential emphasis on the development of the capacity for ‘hard skills of science’ versus the development of other ‘softer’ capacities. ‘Hard’ capacities may include aspects of infrastructure, technology, and finance, while ‘soft’ capacities focus on more social aspects such as the capacity to manage knowledge or develop organisational procedures (Horton 2003). The most extreme differences in approach are seen in areas of ‘high technological research’, including science-focused fellowships such as the Borlaug Fellowships, Marie Curie Fellowships, OECD Fellowships, and Wellcome Fellowships where the aims emphasise the learning or use of technologies, and advancement of scientific research. Evaluation indicators for such programmes include ‘research outputs’, ‘research achievements’ and ‘publications’. Other fellowships with an influence from social sciences have a much stronger focus on personal development, where evaluation indicators include aspects of confidence, esteem, leadership and management (CGIAR AWARD Programme), and social justice (Ford Foundation). Others focus on

ALINe working paper  

10

assessing impact on the three levels given above with varying degrees of success (International Foundation for Science, UK Commonwealth Scholarships). While scientists tend to focus by definition on hard capacity development, there is a considerable amount of evidence suggesting that investment in hard capacities rarely leads to improvements in organisational performance unless soft capacities such as strategic leadership, programme and process management and networks are nurtured as well (ECPDM 2003). Individual approaches to CD such as science research fellowships need to develop not only hard science skills related to scientific research, teaching the use of new technologies and methods, but also develop soft science skills that are equally important: the capacity to network and build relationships with others, the strengthening of leadership skills, the fostering of confidence and resilience, (Ofir, Van Wyk et al. 2008). The success of individual approaches to CD depends on the ability that programmes have in providing training or research exchange opportunities that are part of a multi-level approach to organisational capacity development. They should be tailored for the specific context and social dynamics, and must incorporate incentives to apply the new skills, articulate personnel development, and link this with team performance and overall organisational efficiency. Another aspect to be considered is the extent to which capacity development is influenced by hierarchical or ‘power’ interactions. These need to be taken into account where possible and can occur at all three levels: individual, organisational, wider society. For example, those who hold the funds, or the organisations who are gatekeepers of these funds, wield power that might shape the partnerships being developed. This can lead to long-term processes that deliver intangible results that focus solely on accountability to donors rather than achieving developmental change.3 Such influences can be manifest in terms of unequal access to information, opportunities to publish or author articles. Alternatively, northern partners may have a disproportionate influence on decision-making in terms of the research agenda, and project negotiation in general. In parallel to the promotion of equitable partnerships and consortia, the promotion of networks and relationships is crucial in creating capacity. They depend on the construction and expansion of personal contacts, and rely highly on trust and informality. Despite the mounting evidence that networks are a fundamental channel to exchange knowledge and ideas, development programmes often do not assign them formal recognition (DFID 2006). However, the review of literature has shown that this is not the case for individual approaches to CD such as fellowships. CD in Agricultural Research and Development (ARD) has for some time understood the power of professional (and personal) networks and many ARD fellowships include specific activities aimed at strengthening alliances between professionals, such as exchanges, conferences, workshops, and informal events. After reflecting on the individual, organisational and societal components of capacity building, the evaluation looked specifically at the AFP as a case study.

                                                                                                               3  LIDC Report Measuring Impact of Higher Education for Development 2012.  

ALINe working paper  

11

3. The RI African Fellows Programme: An Introduction  The AFP was commissioned by the Gatsby Charitable Foundation (GCF) in 2004 and operated until the end of 2010. The concept behind the programme was that agricultural challenges are dynamic, and Sub-Saharan Africa needs to have the scientific capacity to be in a position to respond. The focus for AFP was strengthening Africa’s capacity in agricultural research to address problems blocked by lack of scientific capability. It was designed to ensure that excellence in UK and European agricultural science was made available for the direct benefit of small farmers, and that African scientists engaged in the programme were able to address specific constraints which may have required the facilities of an advanced research laboratory. AFP was viewed as an innovative mechanism to facilitate this transfer of knowledge. A distinctive feature of the programme was that it focused on advanced sciences, providing access to cutting edge technologies through placements at centres of excellence in Europe with a view to giving these scientists access to science of the highest quality. Box 2 below shows the original aims and objectives of the programme. The programme was made up of nine Rounds of Fellowship funding. The application process was two-stage with the initial stage entailing the submission of short Concept Notes from African scientists without the requirement of a European partner. Over 900 pre-proposal applications were submitted. The second stage involved a joint submission of a Full Proposal in partnership with a European partner. Where applicants did not already have a partner, the Rothamsted International management team linked them with an appropriate European partner. A total of 49 African scientists were offered placements at European laboratories for periods varying from of five to 12 months. Forty-four appointments were made with five withdrawals from 49 selected. The majority of placements were in the UK (28), but 16 were placed across the rest of Europe. The total cost of the programme was approximately £1.5 million. On-going monitoring of the programme included: 1) six and 12 monthly reports; 2) visits with the Fellow, (one at their placement location and in some cases one at Rothamsted Research); 3) end of Fellowship presentation and report; 4) feedback forms at the end of the Fellowship from both Fellow and European partner; 5) requested feedback from the African partner one year after the end of the Fellowship. In 2008 GCF requested an internal review of the programme focusing in particular on assessing how the programme had influenced agricultural production of small farmers in SSA. In addition a workshop of alumni was held in Cape Town in 2009 to look in more detail at career development and application of knowledge for fellows who had participated in the programme since 2004.

Box 1 Placements: UK: England (22); Scotland (3), Wales (3) Rest of Europe: Austria (1), Belgium (1), Denmark (1), France (2), Germany (3), Ireland (1), Italy (1), Netherlands (2), Norway (1), Sweden (2), Switzerland (1). Countries of Origin: Benin (2), Burkina (1), Cameroon (2), Ethiopia (9), Gambia (1), Ghana (2), Ivory Coast (2), Kenya (2), Malawi (1), Nigeria (5), Senegal (1), South Africa (4), Sudan (1), Tanzania (3), Togo (1), Uganda (7).

ALINe working paper  

12

Box 2 (text taken from 2005 AFP Report) Original Aims of the Programme The Rothamsted International African Fellows Programme aims to provide problem-focused training for mid-career Africans in skills and technologies relevant to the agricultural needs and aspirations of their countries which can be effectively implemented in Africa. Through this programme we endeavour to empower these countries to find their own sustainable solutions to the very real problems of food security in sub-Saharan Africa. The development of effective partnerships where all partners contribute, and all partners gain, building mutual recognition and respect, is fundamental to ensuring success for the longer term. The Fellowship Programme is managed on this basis. Objectives:

• Provide support for eight to 12 Fellowships per year in agricultural research for mid-career African scientists, for four to 12 months dependant on the project.

• Support problem-focused agricultural projects, not general training in a certain area, nor contribute to MSc. or PhD. programmes.

• Support projects of clear benefit to African agriculture, with the unmistakable potential for development impact in-country, via clear dissemination and/or technology transfer routes.

• Promote both high scientific quality as judged by the clarity and achievability of objectives, and innovative science that will provide an opportunity for publication in internationally refereed journals.

• Catalyse long-term ‘smart’ North-South partnerships through Fellowship allocation.

 

ALINe working paper  

13

4. The Challenge of Evaluating the Impact of Capacity Development Programmes: Fellowships

A literature review on the evaluation of capacity development initiatives helped inform the decision about an appropriate evaluation methodology for the AFP. The purpose of the review was to give an overview of the major literature on individual approaches to capacity. It also set out to outline and review methods used by different initiatives to assess the impact of their activities. It also helped set the expectations for an assessment of the impact of activities on fellows, on their organisations, and on society more broadly. The review also helped draw lessons for future evaluations of the impact of capacity building schemes in research, specifically the AFP, as well as within wider sectors such as agriculture, health and economics. In terms of a general approach to evaluating a research fellowship programme, the review suggested that it should use a systems approach that engages with stakeholders and explores power relations and equality in partnerships – an approach that acknowledges the relevance of hard science and soft skills. With regards to specific indicators that could be used to evaluate a fellowship programme such as the AFP, the literature noted that evaluations of individual approaches to capacity development are scarce, but that there is enough literature to allow identification of the main elements that should be taken into consideration. These aspects are summarised below.

A. Programme Design An evaluation of a grant or fellowship programme often requires a reflection on the contribution of the design to its goals. Was the programme plan well articulated and achievable? Were the activities clearly linked to well defined results? An assessment of the programme components (e.g. training courses, participation in conferences, contacts with other researchers, supervisor and mentor support) is also important: they can be rated and checked as to whether they reinforce each other. Criteria for the selection of fellows should be considered, as should the fairness and objectivity of recruitment, and the strictness of the process that determines the willingness of host institutions to take in fellows, based on an assumption about the quality of successful applicants. Similarly past literature makes it clear that a fellowship programme must have a relevant and appropriate placement of fellows in host institutions. This can be evaluated through feedback: from supervisors as to whether they have received the right kind of fellow and from fellows concerning the suitability of the institution where they were hosted. Is the researcher going to carry out his or her research in his or her research centre, or will he/she carry out research in another (normally more developed) research centre? The former supposition is based on a view that organisational learning can occur more effectively if the fellow remains at home, whereas placements in other organisations consider the home research centre benefiting upon the fellows’ return after the programme. The duration of the fellowship also needs to be considered. Longer term exchanges of approximately two years seem to be favoured over short term placements of up to six months. However, more senior scientists tend to prefer shorter periods. Duration also depends on the primary objectives of the programme, as tailored two-week skill development requires a significantly different approach to that of a longer programme ranging from months to years and which aims at a significant and more general transformation of knowledge and behaviour in the fellow. Assessments need to acknowledge these different objectives and how they relate to the duration of the programme. When evaluating the design of a fellowship programme, the engagement with stakeholders should be assessed. This is relevant for all fields, but especially so in agricultural research for development. Further, we cannot assume

ALINe working paper  

14

that the category ‘farmer’ is a homogeneous one: a farmers’ association does not represent necessarily all farmers, and technology will influence different kinds of farmers differently depending on context (small or large scale farmers).

B. Programme Implementation and Effectiveness In terms of implementation, how did the running of the programme affect the results? Did all activities take place and were the results achieved? What worked and what didn’t work? A critical variable is the degree of quality, motivation and time and resources available for the programme managing team. The evaluation needs to gauge a measure of satisfaction of those involved, a measure of learning, and a measure of behavioural change. Did the placements work well, was the mentoring process good and were there opportunities for networking? Most individual capacity development programmes in scientific research include an important component of networking – creating and participating in spaces both formal (conferences, workshops, etc.) and informal (events, gatherings, professional memberships, etc.), for grantees to interact with other scientific researchers and other stakeholders (policymakers, private sectors, farmers’ associations, etc.). Evaluations must gather grantees’ and supervisors’ views on the impact of individual capacity development schemes on the development of transnational networks, as well as the importance assigned to these new networks in their future career development. Other elements of the scheme have to be explored: appropriateness of remuneration, convenience of mobility, the quality of the host centres and acquisition of soft skills such as confidence, leadership and autonomy. An evaluation must inquire if this personal development component exists and, if so, assess its success. Although sometimes difficult to obtain, feedback from the fellows’ research institutes of origin is important. An evaluation should assess if and how these home centres benefited from the programme and what components they considered useful. Interviews can provide useful insights regarding the effects on research capacities when the fellow returns home. Individual approaches to capacity development have been criticised for taking into account only one element of the ‘capacity development system’: the individual. However it is useful to look at integrating the three levels of capacity development, individuals, organisations and wider society, in order to examine whether a focus on the individual also has spill-over effects on the home and host institutions. These ‘ripple effects’ could include broader organisational and societal benefits in terms of the creation of new partnerships, the development of new projects or new research agendas. What is the specific contributory role of individual capacity building schemes in the development of partnerships and how can this be evaluated? Most fellowships – due to their individual-oriented nature – do not assign a central importance to partnerships. However, individuals can act as catalysts for partnership building. To assess this, it might be important to look at whether there was ‘buy-in’ (leading to material and systems support) from the heads of the organisations involved and to what extent the proposal was north- or south-driven. Finally, do fellows actually return to their home institutions, to the research sector and remain in their country of origin? This has been a key concern in the management of grants and fellowships, concerned about the risks of investments in individual capacity building being lost to dominant northern research institutions and countries. An evaluation needs to map the ‘geography’ of flows: nationalities of grantees, destinations chosen for fellowship, and eventually which countries fellows choose to reside in after their fellowship. Assessing the economic efficiency of a capacity-building programme, and within that, the cost-effectiveness of an individual CD scheme is challenging, if not impossible. Connections and direct attribution to impacts on poverty

ALINe working paper  

15

and food security are unclear, and CD schemes have effects that are difficult to quantify, including behavioural changes such as empowerment, confidence, leadership, etc. (Ofir, Van Wyk et al. 2008).

C. Programme Impact Assessing the impact of individual CD schemes on direct beneficiaries (i.e. grantees or research institutions) is an easier task than assessing the impact on broader society. Consequently, the most common and developed impact assessment in all fellowship schemes determines the direct impacts on fellows’ careers in terms of commitment to stay in research, promotion, publications in peer-reviewed journals and success in funding applications (Fanzo and Gallin 2006; Ionescu-Pioggia and Pion 2006; Isaacs 2006; McNeely and O'Brien 2006). In terms of looking at impact on wider society, the issue of complexity has already been raised, as has the non-linearity of achieving technological change because of the multiple factors and actors involved. Hence it becomes more a question of attribution and accountability: CD schemes that work with individuals or organisations have difficulties proving to others (particularly donors) their role in achieving certain outcomes. As a result most organisations have ended up by:

(i) ‘deepening’ into mostly quantitative experimental economic analyses that simplify the complexity in their modelling;

(ii) adopting a qualitative approach integrating complexity such as Impact Pathway Analysis; or (iii) adopting a collaborative approach to impact assessment that sees knowledge as co-constructed

by stakeholders.

These points are explored here:

i. Experimental economic impact assessments

These impact assessments are often heavily quantitative. They are generally carried out by agricultural economists, and are mostly based on the projected monetary economic impact on poverty. The CGIAR and the World Bank have traditionally been favourable to this type of impact assessment.

ii. Theory-based evaluation approaches

Theory of change approaches to evaluation, variously referred to as programme theory (Bickman 1990), programme logic (Funnell 1997), theory-based evaluation or theory of change (Weiss 1995, 1998), theory-driven evaluation (Chen 1990), intervention logic (Nagarajan and Vanheukelen 1997), impact pathway analysis (Douthwaite et al. 2008), and programme theory-driven evaluation science (Donaldson 2005) refers to a variety of ways of developing causal models that depict and describe change processes. Although they have been practised for more than 20 years, these approaches have grown in popularity in the evaluation community in recent years and are now considered very useful for supporting processes of monitoring, evaluation and learning. Such approaches lay out the logical links from research inputs to economic, environmental and social changes, inquiring about external processes that may influence different stages in the pathway of change and the assumptions behind these processes. It breaks the pathway down into smaller sequences of events, adding intermediate steps. The aim is to build collectively a plausible argument, including why and how the research output feeds into the decisions of and changes made by the research clients. Thus it builds a programme theory or theory of change against which the programme can be assessed, using both qualitative and quantitative techniques, such as cost-benefit analyses and economic impact assessments as well as other participatory approaches such as outcome mapping (Walker et al. 2008).

iii. Collaborative participatory approaches

These methods are based on an understanding of knowledge as co-constructed, and engage researchers, users and other stakeholders in designing, monitoring and evaluating programmes. They include Outcome Mapping,

ALINe working paper  

16

Most Significant Change, Peer Ethnographic Evaluation and Research (PEER), Appreciative Inquiry, Horizontal Evaluation, Social Return on Investment (SROI).

ALINe working paper  

17

5. Methodology of the Study Based on the review of literature, the current study decided to use a theory-based approach to evaluating the AFP as a case study. We wanted to gain a deeper understanding not just of the extent to which the project worked, but also to understand how it worked or did not work. The ToC approach allowed us to compare what actually happened with what we expected to happen: it allowed us to test our own assumptions. This is very important in a context where the purpose of the evaluation is not to prove the success or failure of a project but to learn so that future programmes could be improved on the basis of the knowledge acquired. Capacity development has many layers of complexity and to evaluate a CD programme this complexity must be incorporated. This emphasises the need to gather systematic qualitative information to explain more quantitative data. Thus, drawing on the literature review, the evaluation team developed, ex-post, a theory of change of the AFP. The theory of change was then tested using an electronic survey of fellows and Key Informant Interviews with fellows, as well as representatives of host and home institutions. These focused on assessing the programme in terms of programme design, effectiveness and impact, and were focused on assessing the following question from the literature review: did the individual approaches (acquired or developed knowledge by individuals) become institutionalised in some way?

A. Creating a Theory of Change for AFP All interventions are based on a set of assumptions about the social-political-economic context in which they operate, and the way in which development will occur. Often programme staff and other stakeholders fail to make this underlying ‘theory’ explicit. The Theory of Change approach is about making explicit this implied theory of change. It is followed by a process of discussing and reaching agreement with stakeholders on the underlying logic behind an intervention, and its context. Finally it is about using evaluation and research to test, validate and refine the understanding of the ToC. Figure 1: illustration of the AFP theory of change.

Sector  Capacity

Organisational  Capacity

Individual  Capacity Time

GOALSpecific  constraints  

affecting  small-­‐holder  farmers  in  agricultural  

production  are  addressed

Control

Interest

ALINe working paper  

18

Hence a theory of change is a set of assumptions about how change can be triggered in order to address problems or issues that affect individuals, groups and systems. Typically, a theory of change defines the problem to be solved, the targets for change (including programme stakeholders, how they will act and possible conflicts of interest), the strategic levers required to solve the problem, and the expected results. A theory of change is both a product and a process. The product is a narrative, which includes an illustration of the change process in diagrammatic form, and the process is the approach to M&E that working with theories of change involve. The AFP contribution was to strengthen scientific capacity in SSA, such that scientists are in a position to respond to the dynamic nature of agricultural challenges. The programme provided scientists with the opportunity to develop their scientific skills, allowing them to overcome a researchable constraint to a practical agricultural problem. At the same time it intended to catalyse a partnership between European and African scientists that would extend and grow long after the end of the fellowship placement, allowing long-term capacity development. Finally it intended to facilitate the application of knowledge on the fellows’ return to Africa. In the AFP ToC diagram (Figure 1), the three horizontal bars at the end of the prism represent capacity at the individual (the fellow), organisational (the African institution) and sector levels (the wider network of actors in the African country who would need to be mobilised in order to achieve the solutions in the agriculture sector). The side of the prism represents time, and outcomes are shown as wide green arrows. The three dimensional aspect of the prism shows how the achievement of a particular outcome is related to capacities at each level, and that the outcomes are time dependent, with the second and third outcomes requiring increased participation of organisational and sector capacity to be achieved. Thus, while Outcome 1 is essentially about individual capacity, Outcome 2 (the development and quality of a partnership) is about organisational capacity and individual capacity. Outcome 3 involves capacity at the wider sector capacity development. The ToC identifies opportunities for placing a fellow at a centre of excellence in Europe as a primary intervention to achieve the AFP goal. This requires appropriate programme publicity, and assumes that scientists are available with sufficient scientific and agricultural knowledge to allow them to identify a problem that needs to be addressed through the application of scientific excellence. It also assumes that they are able to secure a European partner to work with, either through their own previous contacts or through facilitation by the Rothamsted International team. The other influencing factors at this level relate to the Selection Panel. There is the assumption that the experts on the Panel have the knowledge, and judgement, to identify targeted and feasible projects and that the criteria drawn up for them by the Rothamsted Team are appropriate. Indicators of success for the Theory of Change were drawn up including the following: Outcome 1: Fellow gains skills to address specific agricultural problem (This outcome lies within the sphere of control of the programme. Parties of influence are: European partners, Rothamsted International Team, the fellow.)

• Practical arrangements for fellows are well managed by management team as measured by number of fellows successfully placed, reports, feedback from surveys.

• Fellow completes research proposal and gains new knowledge and new techniques as measured by reports, scientific publications, and presentations.

• Fellow’s standing as a scientist increases as measured by promotions, positions of responsibility.

• European partner successfully hosts fellow as measured by completion of project, publications, mentoring relationship.

ALINe working paper  

19

Outcome 2: Fellow catalyses African-EU partnership This outcome lies within the sphere of influence of the programme. Parties of influence are: African partner, European partner, the fellow.

• Collaboration established between EU and African organisations as measured by joint proposals, exchange visits, conferences, mentoring relationship.

Outcome 3: Output from fellow’s project is applied to the agricultural context particularly at the institutional level and beyond This outcome lies outside the sphere of control and only very marginally in the sphere of influence of the programme. Parties of influence: African organisation, agricultural stakeholders.

• Output from the fellow’s project is applied to the agricultural context as measured by production of publications for end-users, talks and presentations given to end-users, policy changes, and uptake of methods by extension, industry, and other end-users.

Outcome 4: Long-term scientific capacity strengthening. This was not originally seen as an outcome of the programme but added following the literature review as a part of the current evaluation. This outcome lies outside the sphere of influence of the programme particularly in relation to creating changes at the organisational and sectoral levels. Parties of influence: the fellow, African organisation, European organisation, wider scientific community.

• Brain drain is prevented as measured by fellows returning to their home country.

• Fellow has improved skills to apply knowledge for sectoral capacity though not through EU-African partnership as measured by scientific publications; publications for stakeholders; involvement in workshops and conferences, communication with different stakeholders, publicising work through different media.

B. Electronic Survey of Fellows The case study looked at the fellows in two cohorts. The first cohort comprised 21 AFP fellows placed in Europe over a three-year period between 2005 and 2008. This group of fellows previously gave detailed feedback in 2009 for an Alumni Workshop through an online questionnaire, thus providing an opportunity to compare 2009 data with that collected in the current study. The second cohort comprised 22 fellows placed in Europe between 2008 and 2010. The study used first an online questionnaire drawn up using SurveyMonkey Software. Separate questionnaires were used for Cohorts 1 and 2. The questions in the 2009 survey answered by Cohort 1 were used as the basis for the survey subsequently sent out to Cohort 24.

C. Key Informant Interviews The second method used to evaluate the programme was a series of Key Informant Interviews (KII). To gain a more detailed understanding of the issues, a selection of fellows, European partners and representatives from the African partner organisations were invited for interview. These were selected in order to get a range of candidates from different geographical regions, across different subject areas, covering both cohorts and both genders. The

                                                                                                               4  In addition to these original questions a series of new questions were added which built on the findings of the literature review and the development of a Theory of Change. In the online survey sent out to Cohort 1 most of the original 2009 questions were removed but the new questions were posed. Hence there was the possibility of comparing all questions between cohorts, and certain questions both across time (for Cohort 1 only) and between cohorts. A total of 43 fellows were approached for feedback using the online survey, and a period of five weeks was allowed for responses. Results have been portrayed in tabular and graphic form, with supporting text, but the sample size did not allow for any statistical analysis.  

ALINe working paper  

20

interviews were 30 minutes to one hour in length, were in more depth than the questionnaire and semi-structured, assessing the key aspects of fellows’ careers and the dynamics of programme implementation. This allowed more detailed information on some of the issues arising from the survey. It was important in allowing an understanding of motivation and the differing perspectives of the players, and allowed the possibility of exploring ideas of those interviewed. Findings from the KIIs were particularly useful in supporting and explaining findings from the surveys, and in identifying ideas for mechanisms that might be useful in future programmes. In both the online surveys and the KIIs (see Appendices for details) the evaluation sought to examine the design of the programme, its effectiveness and implementation, and its influence on local agriculture. To do this, five areas were explored as follows:

1. Practical aspects of the programme

2. The project itself (or the research project they had been working on)

3. Influence on the individual

4. Application of knowledge

5. Long-term partnerships.

The questions targeted indicators of success as drawn up through the development of the ToC. So for Outcome 1 (fellow gains skills to address specific agricultural problem):

a) A series of questions firstly looked at whether the practical arrangements for the fellows had been appropriate and well managed in terms of: • appropriateness of programme duration; • appropriateness of programme location; • management of the programme; and • degree to which home organisations were involved in the design of the project.

b) Secondly information was sought on knowledge and skills gained in terms of:

• degree of learning acquired by fellows as measured by associated publications; • subsequent changes in behaviour such as presentational skills; • development of soft skills and personal development including self-awareness, leadership and

management skills, confidence and motivation, capacity to become role models; and • effects on networking.

c) Thirdly, information was sought on changes to a fellow’s standing as a scientist as measured through:

• successful publications; • promotions and increasing responsibility; and • projects taken up by the fellow after the fellowship.

d) Fourthly, questions probed the success of the European placement in terms of the quality and impact of

mentor relations.

For Outcome 2 (fellow catalyses African-EU partnership) questions revolved around: the interaction between the fellow and the European partner, whether contact had been maintained and at what level, from sporadic contact to working on joint initiatives; how the mentoring relationships worked; and the contribution of the programme to organisational capacity (in both host and home organisations).

ALINe working paper  

21

For Outcome 3 (output from the fellow’s project is applied to the agricultural context), fellows were asked about the degree of involvement with stakeholders; the extent to which they had implemented activities and results obtained through the AFP; their perceptions of the influence of their work on technology development, technology adoption and changes in agricultural production; their levels of communication with different players in the agriculture sector. Outcome 4 (long-term scientific capacity strengthening) was not initially a planned outcome of the programme, but through the process of assessing the evaluation literature and developing a Theory of Change for AFP, it became apparent that an aspect of influence not being accounted for was the contribution to scientific capacity at the wider national or regional level. Hence these aspects were investigated through questions focusing on brain drain, mobility of the fellows after their placements over the medium term and where they ended up, publication levels and career progression.

ALINe working paper  

22

6. Key Findings

General Findings

Out of 44 Alumni fellows invited to take part in this study, a total of 36 (84 per cent) responded to the online survey. Of these, one person did not complete the survey. For Cohort 1, 17 out of 21 responded (81 per cent) and for Cohort 2, 19 out of 22 responded (86 per cent). This is a very high response to a survey, and is interesting considering that all Cohort 1 finished their placements more than three years ago and some six years ago. It reflects the quality of the relationship that was developed with fellows, as well as the relatively small size of the programme allowing the possibility of approaching people more personally. Of the 16 fellows invited to be interviewed, key informant interviews were held with 11 individuals, and of the five African and EU partners invited, four African and four EU partners were interviewed.5 The aims and specific objectives of the AFP are noticeable in their generality and highlights the need that all of those involved in the programme (donors, managers, participants, stakeholders) have the same interpretation. Developing a ToC allows the achievement of a consensus and ensures that all of those involved in the project are aligned and pulling in the same direction. The inclusiveness of the ToC process also has advantages. Once approached, past fellows and partners showed great enthusiasm to be involved and provide feedback. Representatives of African partner organisations highlighted this inclusiveness as being very valuable, particularly where feedback may have the possibility of feeding into, and improving, other programmes. Thus a process whereby participation is invited early at the planning stage of the project, and where communication is subsequently maintained, can be very useful in ensuring successful delivery of the targets. Individual Level Overall the fellows were very positive about the programme and the opportunities it had given them. Fellows felt that the programme had been well managed, and this was still the case when probed during the key informant interviews. Positive feedback included: the opportunity to identify projects focusing on their own needs as identified by the home institute; the genuine demand for training of personnel in African scientific organisations; the flexibility of the programme; and, the monitoring of placements through regular reports and visits. Suggestions were made regarding increased length of placements and opportunities for fieldwork in Africa, which will be discussed below, and some reference was made to stipends (‘scale them according to the cost of living in the country of placement’). A small number of fellows also highlighted the problem of being away from family for long periods of time and the potential need to return during the fellowship to deal with urgent matters. As mentioned above, aspects of programme design which particularly affected the fellows included location and time of placement. All fellows responded that the locations of their placements were appropriate to their projects, though in interviews one fellow did question whether a different location would have been more appropriate in order to allow work to be on the African crop as opposed to a model system. Two interviewees suggested that a second scientific mentor allowing coverage of subject matter outside the expertise of the European partner would have been helpful. With regard to the length of placements, 72 per cent of fellows were happy about the length of their placement, and of the 11 who were not, ten would have preferred to have more time. This issue was probed during the

                                                                                                               5  For confidentiality reasons, names and details will not be published.      

ALINe working paper  

23

interviews, and all fellows questioned had the perception that longer fellowships, or extensions of placements, would not pose a problem to the African organisations as cover was normally organised prior to departure and the value of the training to the organisation would outweigh any problems with extensions. However, representatives of the African organisations were less enthusiastic about longer placements or extensions, with only one person saying that it would not have caused a problem. One said that AFP placements had been a little long and shorter periods of training would be better; one said that longer placements and extensions cause real problems to management; and one from an international organisation said that it was not a problem for them but would be for national programmes. In terms of learning, the responses were quite consistent regarding of technical knowledge or hard skills gained: of all fellows who responded, 95 per cent had learnt to use new scientific equipment, 97 per cent had learnt new laboratory techniques, and 97 per cent had learnt new scientific methods. Fellows rated the training as follows: Figure 2: Cohort 1 (2012) – training 1=Poor 2=Satisfactory 3=Good 4=Very Good 5=Outstanding Figure 3: Cohort 2 – training

1=Poor 2=Satisfactory 3=Good 4=Very Good 5=Outstanding

0%

10%

20%

30%

40%

50%

60%

70%

1 2 3 4 5

Please rate the training gained through the project:

1

2

3

4

5

 

ALINe working paper  

24

Fellows stated that the fellowship had been important to their research, but the percentage of Cohort 1 claiming that it had contributed ‘a significant change to their understanding’ (as opposed to being ‘one of many influences’) decreased from 62 per cent in 2009 to 47 per cent in 2012, probably reflecting the fact that over this time period they have had a number of other influences on their career. This and other aspects below, highlight the importance of the timescale in the ToC. Networking was emphasised by all fellows as an important part of the programme and many had particular examples where their networking opportunities had resulted in them being offered further awards or opportunities for PhD placements. Many seemed to realise the importance of networking for the first time. As mentioned in Section 4, networking is an extremely important skill for scientists to develop as it provides opportunities to brainstorm with scientists from a variety of backgrounds, to present work, or form collaborations and partnerships, and to identify opportunities to progress their careers through further visits or awards. Many African scientists have few opportunities for this sort of interaction, so it is of particular value to them, though somewhat difficult to measure through evaluations. Building on these opportunities to network, 71 per cent of fellows felt that the fellowship had provided opportunities to collaborate with new partners, a key focus for the programme. The overall ratings for the fellowship programme as a whole were good, though slightly higher for Cohort 2. Figure 4: Cohort 1 (2012) – overall rating of programme

 

ALINe working paper  

25

Figure 5: Cohort 2 – overall rating of programme

1=Poor 2=Satisfactory 3=Good 4=Very Good 5=Outstanding When questioned about less technical or soft skills gained, 91 per cent of fellows had learnt new ways of thinking during their placements, and 94 per cent had developed their personal skills. When questioned in the KIIs about this, many (five out of 11) felt they had gained leadership skills, one was unsure and the another referred to ways in which they had taken on leadership type activities, although they did not directly attribute this to developing leadership skills during the programme. Of the four African partners interviewed, three were very positive about the placements having increased the leadership skills of the fellows, while the fourth expressed that the focus of the placement had really been scientific and technical, rather than to build leadership. In terms of other soft skills, three out of 11 fellows participating in the interviews stated that they had gained independence, five that it had improved their ability to integrate, interact or communicate with other people, four had gained confidence, three said it improved efficiency and organisation, two mentioned discipline/focus and two delegation skills. All but three of the fellows responding to the online survey had received recognition as a result of their fellowships. The most frequent form of recognition was added responsibility, followed by promotion, although interestingly neither appeared to be associated with increased salary! This is worth noting as some programmes use ‘increased salary’ as an indicator of impact in their evaluations. Cohort 1 fellows had a much higher level (70 per cent) of promotion than did fellows from Cohort 2 (20 per cent). This might be expected given that Cohort 2 had only recently completed their fellowships as opposed to Cohort 1 who completed theirs four to six years before. Time prior to promotion was not measured in the study. Cohort 1 also had a slightly higher rate (10 per cent) of being offered another fellowship or award. The fact that this difference was not greater (when compared for example with ‘promotions’) suggests that if fellows are going to be offered another award, it tends to happen soon after the end of a placement. This might be because networks and connections providing opportunities will be most ‘fresh’ at that point. The importance of the placements in improving their profiles as a way of promoting their careers was particularly emphasised by three individuals in the interviews.

0%

10%

20%

30%

40%

50%

60%

70%

1 2 3 4 5

Please rate the Fellowship Programme as a whole.

1

2

3

4

5

ALINe working paper  

26

Figure 6: Cohort 1 (2012) – recognition

Figure 7: Cohort 2 – recognition The ratings for ‘mentorship’ and the ‘value of the time spent with European partners’ suggested that experiences had been more mixed in this respect. Responses were similar within cohorts suggesting that these aspects might have been linked, and were probably dependant on the placement. Of the 11 fellows interviewed, nine had acted as mentors themselves after the end of their placement thus becoming role models themselves. The mentorship was somewhat more positively rated than time spent with the European partner. Key informant interviews suggested that this may have been because the European partners in some cases spent less time with the fellows than did the mentors, and delegated some of the training of the fellows to other staff.

0%

10%

20%

30%

40%

50%

60%

70%

80%

Yes, moreresponsibility

Yes, promotion Yes, highersalary

Yes, anotherfellowship or

award

No

Have you received any recognition as a result of the Fellowship?

 

ALINe working paper  

27

Figure 8: Cohort 1 (2012) – value of time with research supervisor

Figure 9: Cohort 2 – value of time with research supervisor

1=Poor 2=Satisfactory 3=Good 4=Very Good 5=Outstanding Institutional Level This section looks at how the individual interacted with the two partner institution that is the African and European partner organisations. Placed in Europe, the fellow would be in a position to catalyse an institutional partnership, but this required input from an increasing number of players which were outside the sphere of control of the programme.

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

1 2 3 4 5

Please rate the value of the time spent with your research supervisor:

1

2

3

4

5

 

ALINe working paper  

28

Another aspect to be explored is how the donors or managers of the programme might have shaped the programme and the inter-institutional relationship. Key to the facilitation of long-term partnerships are engagement and involvement of the African partner who encourages local ‘buy-in’, and mutual commitment and trust. To gauge this, fellows were asked whether ideas for the AFP projects arose from the African or European side and what support the fellows had received from their home organisations. Of 35 respondents to this particular question, 34 stated that the ideas for the project originated from them and their home organisation; and 28 had received scientific, financial or logistical support from their home organisation allowing them to take up their placements. How the placement contributed to partnership building was also explored. Measures for this were taken in terms of on-going contact with the European partners, and follow-up work in partnership. The graphs below are interesting as they show the differences over time for Cohort 1 as well as a comparative snapshot between Cohorts 1 and 2 taken at approximately the same time period (i.e. Cohort 1 in 2009 vs. Cohort 2 in 2012). Figure 10: Cohort 1 (2009) – contact with European institute Figure 11: Cohort 1 (2012) – contact with European institute

 

 

ALINe working paper  

29

Figure 12: Cohort 2 – contact with European institute

Contact between fellows and European partners maintained for Cohort 1 in 2009 was higher than that of Cohort 2 in 2012. But even after four to six years only 12 per cent of Cohort 1 no longer maintained contact with their European partners. This is quite impressive and suggests that at least this aspect of the capacity-building has been institutionalised in some way. Interestingly the percentage of Cohort 1 who had written joint publications decreased between 2009 and 2012. One possible explanation for this might be that there is a high rate of publication rejections. In further questions in the questionnaire, all of Cohort 2, and 77 per cent of Cohort 1, claimed to be planning follow-up work or continued collaboration with their European partners. This suggests that partnerships have indeed been catalysed, and that they last over the medium term. In terms of contributing to the capacity of African organisations, 82 per cent of fellows felt that their placements had benefited their home organisations with new knowledge; 77 per cent felt they had gained access to facilities unavailable in their home organisation; and 74 per cent had developed new contacts with European scientists. Further, of all fellows responding to the current surveys, 83 per cent reported that they had been able to find solutions to scientific problems that they would not have been able to solve before, and 94 per cent had been able to apply scientific methods learnt during their fellowship. Assessing in real terms what this meant to the capacity of the African organisations was more difficult. Key informant interviews with African partners suggested that from an institutional point of view, the main benefit was the hard and soft skill development of the individuals, but clear information on application and uptake of knowledge was much more difficult to collect. So the real question is to what extent ‘acquired or developed knowledge by individuals becomes institutionalised in some way’ in the African institution and how can this be measured most efficiently. Many fellows felt that lack of facilities and poor management had hindered their work back at their home organisations as seen below. This is a recurring theme in evaluations of CD programmes. Mechanisms allowing some sort of continuity when they return home are really very important. It highlights the importance of selecting projects that yield outputs that can be taken up and advanced on return home.

26.3% 26.3%

15.8%

31.6%

0.0%

Yes, informalcontact

Yes, maintained amentoring

relationship

Yes, applied forjoint funding

Yes, written jointpublications

No

Have you kept in contact with scientists from the European institute?

ALINe working paper  

30

Figure 13: Cohort 1 (2012) – problems at home institute/university

Figure 14: Cohort 2 – problems at home institute/university

Wider Societal Level This section covers aspects of the evaluation addressing Outcomes 3 and 4, which have their main potential to impact on the wider agricultural and scientific communities. Influence here is clearly beyond the sphere of control of the programme; often the sphere of influence is weak. However, it is still important to measure this in order to test whether the theory of change of the programmes is borne out in practice.

0%10%20%30%40%50%60%70%80%90%

100%

Lack of interestin your research

Lack ofunderstanding

of your research

Lack ofcommitment to

applying theresearch

Lack of facilities Poormanagement

Have you experienced any of the following problems in carrying out scientific research in your home institute or university?

0%10%20%30%40%50%60%70%80%90%

100%

Lack of interestin your research

Lack ofunderstanding

of your research

Lack ofcommitment to

applying theresearch

Lack of facilities Poormanagement

Have you experienced any of the following problems in carrying out scientific research in your home institute or university?

ALINe working paper  

31

Application of knowledge gained during the AFP placement Most fellows saw farmers as the ultimate end-users of their research, although when probed in key informant interviews, they often did not see them as the immediate users of the outputs of the projects and suggested obstacles in delivery of outputs to farmers. Some did not have the required links, or the authority to advance the work as they would have liked, due to institutional structures. Others did not have the funding required to test outputs under field conditions which was the next obvious step in the pipeline. Yet others came from university environments where their role in passing on knowledge is through teaching rather than through actual implementation. This latter point raises the issue of ensuring correct project selection to achieve the targets of the programme: even though a Selection Panel may do its best to select projects that target the aims and aspirations of a programme, it is often difficult to assess the organisational structures of partner institutions and how they might impact on project uptake. Figure 15: Cohort 1 (2012) – end-users of research

Figure 16: Cohort 2 – end-users of research

0%

10%

20%

30%

40%

50%

60%

70%

80%

Farmers Extension Industry Other scientists

Who do you see as the end-users of your research:

Farmers

Extension

Industry

Other scientists

0%

10%

20%

30%

40%

50%

60%

70%

80%

Farmers Extension Industry Other scientists

Who do you see as the end-users of your research:

Farmers

Extension

Industry

Other scientists

ALINe working paper  

32

In total 91 per cent said that small farmers rather than large farmers would benefit from their research. Some fellows (22 per cent) said that they had experienced serious problems in applying research, and 32 per cent had experienced slight problems. Of those expressing serious or slight problems most (68 per cent) blamed the lack of facilities, equipment or consumables, and many (37 per cent) referred to funding limitations. This is similar to problems that hindered their research in general in their home organisations (see figures 13 and 14). Communication with stakeholders is an important step towards applying scientific knowledge and it was encouraged through the programme although was well outside its sphere of control. This aspect was measured by asking fellows to assess the frequency with which they communicated with different stakeholders before and after the programme. For both cohorts, increased communication after their placements was most notable with scientists from other institutions in their home country and with scientists in the EU partner organisation. Perhaps to a lesser extent but still noticeable was the increase in communication with scientists in their own departments, which is interesting as it suggests either increased confidence or improved status on returning home. Communication also increased but perhaps to a lesser extent with other international scientists, extension staff, and the wider agricultural community (defined as agro-dealers, traders and groups engaging with farmers). In terms of applying knowledge gained during their fellowships to the wider scientific community, the number of scientific papers linked to AFP work that had been published by fellows from Cohort 2 varied from zero (four fellows) to a maximum of three, and the average number of publications was 0.92. For Cohort 1, who had three more years to get their papers through to publication, all had published a scientific paper linked to AFP work; the maximum number by any one fellow was six papers, and the average number per fellow was two. For both cohorts, 80 per cent of fellows had also made presentations on their work to other scientists. Many had also given talks to non-science groups linked with agriculture and written articles for wider society uptake: Figure 17: Cohort 1 – activities to publicise fellowship work

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

None Given talks toscientists

Given talks tofarmers groups,

NGO's, agro-dealers or thoseengaging withfarmers e.g.extension

Producedscientific

publications

Written articlesin newspapersand magazines

What activities have you carried out to publicise the work from your Fellowship?

ALINe working paper  

33

Figure 18: Cohort 2 – activities to publicise fellowship work

Longer Term Science Capacity Building An aspect of the programme highlighted by this study, but not accounted for through the monitoring of the operational programme, is that there was a great deal of learning that may not have directly impacted on either partner organisation, nor substantially on the agriculture sector. However that learning is of considerable value nonetheless in terms of long-term science capacity building. Here we are referring to the increased ability to write and publish peer-reviewed science articles in international journals, the development of networking skills, the application of scientific methods to new problems, and problem solving in general. Perhaps even more importantly, keeping people in science in order to maintain a trained science workforce is critical to ensuring that SSA countries are in a position to respond to the agricultural challenges facing them. Hence it was interesting to measure whether fellows stayed in science after their placements or moved into related sectors in order to achieve greater salaries or status. This study found that of the 36 responding fellows all but one are still working in science, and this individual is now working in an international organisation in a research-related position. Seven fellows had been offered employment abroad after their fellowship (three in the USA, two in Germany, one in Kenya, and one in Liberia). All took up the offers aside from the post in Liberia. Of these six, three have now returned to their country of origin, one has remained in the USA, and two have remained in Germany but all still work in science. Several fellows had also been offered further short-term awards of fellowships in Europe or North America thus increasing their mobility. But interestingly, aside from those three candidates, all fellows returned to SSA, though not necessarily to the original African organisation. This mobility allows scientists to grow, and if the motivations and incentives to return to SSA can be maintained, then it will be very important in building agricultural science capacity for the region over the long term.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

None Given talks toscientists

Given talks tofarmers groups,

NGO's, agro-dealers or thoseengaging withfarmers e.g.extension

Producedscientific

publications

Written articlesin newspapersand magazines

What activities have you carried out to publicise the work from your Fellowship?

ALINe working paper  

34

7. Lessons Learnt  This case study built on current information and thinking on individual approaches to CD as set out through the literature review. This information helped to develop a ToC for the AFP, thinking through the context of the programme, the assumptions required to achieve the desired outcomes, the players involved and the roles they needed to play. The results outlined above allow us to formulate a series of lessons: but what are the lessons for and how is this process different to the previous assessment of the programme? This current evaluation of the AFP as a case study built on the on-going monitoring6 of the programme that took place while it was being implemented. Although a baseline study was not taken at the beginning of the programme, nor at the beginning of each fellowship against which to measure developments, projects were however closely monitored, and feedback obtained through reports, presentations and feedback questionnaires from the fellows and European partners on each project. An internal evaluation of the first phase of the programme was made in 2008 and further information and feedback gathered at an Alumni Workshop held in 2009. This level of detailed monitoring of a programme can provide very good information on how it is working and in many cases, where monitoring is planned and targeted correctly, it provides sufficient information for the reporting required by donors of the programme. However, the more detailed evaluation of AFP using a Theory of Change has also been very useful. This type of approach allows us to capture where programme targeting has changed; where information is needed on the processes that result in change; or when trying to think through new programme design or scaling up of existing programmes. Below are some of the lessons that have been gained through the process of searching background literature, developing a ToC, and then testing the ToC, that were not clearly apparent through programme monitoring alone.

• Length of placements was raised as an issue by many of the fellows on the programme, with most expressing a need to have slightly longer or extended projects, due to unexpected problems in the research, or because they had found exciting results that needed to be probed. The cost of longer or extended projects is obviously a major issue here but in thinking about the design of a programme, it might be worth considering an option for keeping a pot of money that past or current fellows could bid into for the extension of on-going projects. This would allow for genuine situations where it would be cost-effective in the long term to ensure a piece of work is seen through to a useful end-point rather than stopping in the middle.

Another aspect to this discussion was the perspective of the African partner organisations where longer or extended fellowship placements can cause real problems in terms of resource management for situations where skilled personnel are in short supply. It is important to consider the implications of fellowship length on them. While circumstances do vary between organisations, the programme design should take this into account, allowing for flexibility in placements including a variety of models such as longer placements, shorter placements or a series of short visits with return to the home organisation in between. While travel costs might be higher in the latter instance, this may result in better outputs in the long term.

                                                                                                               6  ‘Monitoring and Evaluation’ is often used as a general term referring to the assessment of a project or programme. However it is worth noting that these are two different processes: ‘monitoring’ refers to observing and checking the progress or quality of a project over a period of time, in other words keeping it under systematic review. ‘Evaluation’ is more about appraising a project – ascertaining its value or worth – and tends to be done after the project (or a phase of it) has been completed.  

ALINe working paper  

35

• Location of fellowships was viewed very positively, with all fellows answering that the location of the fellowship placement had been appropriate. However, during the key informant interviews, one fellow did suggest that it might have been better if she had been working on an African crop rather than a model system and this issue was also brought up by an African partner not associated with the fellowship, which suggests that it is a matter to be noted.

One unusual aspect of the AFP was that African scientists could apply at the Concept Note stage without having identified a European partner. The assumption here was that there are good African scientists with innovative ideas who do not have an opportunity to follow up on ideas because they do not have the appropriate northern links. Thus the Rothamsted International management team provided a partnering system for these candidates. From the evaluation it appears that in most cases this worked well, but there were some instances where it could have been improved. It is worth noting that this was very resource-heavy for the management team and that for larger initiatives different models for facilitating partnering need to be considered. An interesting idea raised by a fellow during an interview was that such programmes should consider a design whereby once fellows have successfully completed a project and returned to their home organisations, future fellows might be placed with them for projects rather than sending them to Europe. Variations on this idea might be a very interesting way of catalysing the ripple effect as discussed previously and hence the long-term impact of a programme.

• Split time with Africa: The importance of having an option for a period of the placement spent in Africa at the home organisation was really highlighted in the key informant interviews: programme structure should be tailored according to desired impact and if the target is to achieve application of knowledge in SSA, a period of support in SSA should be an option, if not a compulsory component. There were widely differing views as to the model for this in terms of the timing during the project for a period of time spent in Africa. This suggested the need for flexibility in programme design, allowing the most appropriate options for a wide variety of projects.

• Soft versus hard skills: There was considerable variation in response to questions about development of soft skills such as leadership, confidence, independence, public speaking, and communication in general, with some fellows acknowledging great progress and importance to this attribute of the programme. Others were less sure. This was similarly reflected in responses during key informant interviews of African and European partners where some felt that these skills were essential, while others felt that the focus and outputs of the programme had really been in terms of hard skills. Despite this, an option that might be considered, as suggested by one fellow, is that it would be very valuable if programmes provide an assortment of tailored training options available to participants and that this might include soft skills as well as technical and scientific skills.

• African partner input: Project areas were overwhelmingly felt to be driven by the African partners rather than the European partners and during the interviews this aspect was applauded by African partners. This was an important aspect of the targeting.

Implementation

• Learning: Fellows were very positive about hard scientific skills gained in terms of knowledge, research techniques and methodologies, and publications. Their overall rating for the programme was high so clearly this aspect of the programme was a success. Many expressed changes in their behaviours in terms of applying new methods and solving new problems. Much of this information confirmed what had been found through project monitoring.

ALINe working paper  

36

• Mentoring: When questioned about this, particularly during the key informant interviews, it became clear that ‘mentoring’ had different meanings for different fellows. Rather than the mentoring being attributed to one person, many fellows felt that they had received mentoring support from a range of people including their non-scientific mentors (where appointed), by the European partners and by the RI team who maintained communication with them for the duration of their fellowships. Fellows in general felt that they were very well supported during their placements.

Perceptions of value of the time spent with their European supervisors were good, but a little lower than ratings given for the programme as a whole. This was interesting and highlighted a point that had not been raised at the monitoring stage. It may have reflected that often the European partners were stretched and had delegated some of the training aspects of the projects to others, who often were more appropriate trainers. However it does also highlight another issue that came up during interviews where some fellows and European partners felt that it would have been useful to acknowledge in some way the time and effort given by the European partners, whose time was not costed into the programme in any way. Is it possible that monetary compensation would have enabled improved quality time spent between fellow and European partner by freeing up pressure on them?

• Partnerships: Despite the last comment, strong partnerships were built with European partners, and an interesting finding from the study is that a high majority still maintained good contact and were working on joint publications even a considerable time since leaving Europe (up to six years). This shows the value of this form of CD in terms of strengthening capacity through partnership building and networking – a critical aspect of science and innovation – and supports the assumption that fellowship placements can support long-term partnering. Again this could not easily be shown through on-going programme monitoring.

• Networking was an aspect of the fellowship that all fellows seem to have highlighted and particularly valued. In some cases, fellows said that they realised the importance of networking for the first time, and almost all were able to provide examples where networking had led them to new opportunities.

• Training of individuals: This was highly valued by the individuals involved, with fellows acknowledging the hard and soft skills developed, the wealth of career opportunities provided and the changes that had resulted. However perhaps more interestingly from an evaluator’s perspective was the value assigned to it by the organisations from which they came. The great need for training of personnel was stressed as being an essential and very valued component of capacity building and one they cannot do without.

• Power dynamics: It was hard to find generalities on power dynamics coming out of this study where real insights could be gained. This does not mean of course that power dynamics were not influencing factors. But there may have been reticence or a perceived lack of freedom to express views about them. A small number referred to lack of flexibility, or career development support, from African supervisors and there were references also made to underlying discrimination. However, the most common power dynamic seen as a hindrance was between fellows and African supervisors related to mobility opportunities and on-going support.

ALINe working paper  

37

Application

• End-users: By far the majority of fellows (75 per cent) considered farmers to be end-users of their research, but when questioned more closely most felt that they were not in a position to apply their work directly in a context that may impact farmers immediately.

For some this was because more work was necessary either by other scientists or agronomists, while others needed to do field tests for which they did not have the facilities or resources. Others explained that the research and development structures within their country are set up in such a way that scientists do not have direct access to application of results and must pass information to, and rely on, other players such as extension agents, or NGOs, who are responsible for the application process. However, there were some fellows who returned to home organisations where extension and research are closely linked, and they were very positive about the clear routes whereby results from their projects had been fed directly into the agricultural situation. A critical assumption for Outcome 3 of the programme was that scientists working on a project targeted at overcoming a problem in agricultural production would be in a position to apply that knowledge in the agricultural context on return to their country. It appears that this assumption was not entirely correct. It is clear that any programme with such aims needs to put specific strategies in place to support activities that relate to increased application for end-users.

• Facilities: The lack of facilities in Africa was seen as a major hindrance to the application of knowledge gained and to the continuation of work on return to Africa. Similarly and perhaps linked to this, was the lack of funding. While any single programme cannot deal with such major issues, to prevent these aspects from being major barriers they need to be considered in programme design.

For example, real care needs to be taken in ensuring that fellows are trained in areas where they can continue to work on returning home. While this was one of the criteria considered by the Assessment Panel, it appeared to be difficult to judge through proposals as many candidates are eager to receive high-tech training to progress their careers regardless of whether their home organisation can support them on their return. Transparency of this nature is not always explicitly requested in proposals. Regarding funding, as it is increasingly important for scientists to be able to compete for funding, it would be extremely valuable to include a proposal-writing course for all participants in CD programmes for agricultural research.

• Multiplier effects: Although this was one part of the programme that the study was particularly interested in learning about, responses to questions in this area were very vague, with most fellows not seeing this aspect clearly, finding it difficult to assign their role or not feeling in a position to judge. European and African partners similarly felt that the complexity of issues and players determining a ripple effect were too great for them to really place attribution to any one thing or programme. While these multiplier effects are clearly of interest, the complex nature suggests that more focused consideration of how they work and how they can be measured needs to be looked at as a separate case. This could be an interesting and valuable component of a larger study.

• Engagement with African organisations: In previous studies this was seen as potentially a weakness of the AFP programme and was therefore highlighted in the key informant interviews. Interestingly, fellows were of the view that this was not really the responsibility of the programme management, and they found it difficult to respond when asked what could have been done to improve engagement with their home organisations. But they felt that this was more the responsibility of the fellow and how they interacted with

ALINe working paper  

38

the African organisation. However African partners did come up with some useful suggestions that might be considered in designing future programmes:

a. Fellows return with a document from the programme management for the attention of the director of the home organisation, detailing how the project has gone and proposing next steps in terms of application of knowledge;

b. Travel money is allocated for the European partners to make a visit to their fellow in the home organisation after the end of the fellowship to discuss application of knowledge and proposed next steps;

c. A small amount of programme funding is allocated to a separate pot of money to apply work from programme projects that can be bid for on completion of a placement.

• Brain drain and mobility: While increased mobility of fellows after their fellowship was not uncommon, there were very high rates of individuals returning to SSA within five years of the fellowship placement and all but one are known to have remained in science. Maintaining a trained and motivated workforce in agricultural science is critical to empowering countries in SSA to be in a position to respond to the challenges facing them. Programmes focusing on CD that incorporate fellowships and scholarships are often linked with brain drain and criticised for their role in it. Results from this evaluation suggest that with the correct approach and incentives, the risk of brain drain can be mitigated, and facilitation of mobility has an important role to play in maintaining and motivating a scientific workforce in SSA.

ALINe working paper  

39

8. Conclusions Capacity development schemes that work with individuals or organisations find it challenging proving to others (particularly donors) what is their role in achieving certain outcomes. In addition to this, donor pressure to make CD programmes accountable can have the undesired effect of decreasing the potential of CD to promote organisational learning, as impact assessments and evaluations become directed at gathering information for fundraising and satisfying donor requirements. At the beginning of this report the study focused on contributing evidence to the debate looking at how best to monitor and evaluate capacity strengthening efficiently and effectively, with particular reference to individual approaches to capacity development. The results from the study help identify components that are important in the design of individually focused capacity development programmes and their impact. In considering methods for evaluation, this study built on:

1. The findings of a comprehensive literature review ‘Evaluating individual Approaches to Capacity Development’ by Santiago Ripoll Lorenzo. This was essential in setting the context for evaluation and in ensuring that the study built on learning contributed by others.

2. A Theory of Change was developed for a specific case study: the Rothamsted International African Fellows Programme. While this would ideally be done before a project starts, it was an important part of the current study, as it accentuated assumptions and processes needed to achieve the outcomes of the programme. It highlighted the importance of the timeframe in evaluating outcomes, and uncovered an outcome not previously addressed: influence on the wider science community.

3. The case study programme was evaluated against the framework of the ToC. This three stage process provided a rigorous, yet feasible approach to evaluation.

The vision behind the AFP programme was that developing the capacity of individual researchers in a focused way would eventually improve farmers’ livelihoods. As illustrated in the ToC, there are a series of assumptions to support this logical thought. Multiple stakeholders involved in achieving change contribute a level of greater complexity in measuring change. Further, while recognising that there are spheres of control for the programme and spheres where the programme can only influence in an indirect way, the evaluation looked at both of these areas in an attempt to understand this complexity, and assess the fate of the individuals involved during the project lifetime and some time beyond its life. The current study built on programme monitoring but allowed a detailed opportunity to investigate the processes of change and the influence of the programme over the longer term. As stated by Santi Ripoll in the literature review it allowed us to consider the following question: did the individual approaches (acquired or developed knowledge by individuals) become institutionalised in some way? A number of lessons learnt have been outlined above but a brief summary is given below:

• Information on knowledge gained, skill development and career progression is fairly straightforward to collect, and much of the information here built on an understanding gained through monitoring the project during implementation.

• The process of developing long-term partnerships is less easy to measure but information from the study showed that that almost all fellows maintained some form of working relationship with their European

ALINe working paper  

40

partners, and this was measured up to six years after the end of their fellowships. We suggest that this is good evidence of this aspect of the programme ‘becoming institutionalised’.

• It was much more difficult to gauge the application of knowledge in the African organisations. Although all fellows participated in a variety of routes to dissemination such as training workshops, presentation of knowledge to colleagues and agricultural stakeholders, it was not possible to gauge uptake. If knowledge uptake within the African organisation is a focus for a CD intervention, then the need for mechanisms to assess this aspect of the programme must be taken into consideration (LIDC, 2012).

• Further, African partners really highlighted the importance of allowing for continuity of the work when participants return to their African organisations. Some organisations such as the International Foundation for Science have addressed this aspect of CD.

• Not all of the assumptions on which the ToC was based were found to be correct. For example while scientists viewed farmers as their ultimate end-users, on questioning they explained that in general they were not in a position to apply their knowledge in the agricultural context. Clearly it would have been better for this to have been acknowledged right at the beginning of the AFP programme and for mechanisms to be put in place to address this issue.

• Brain drain is always of concern when aspects of international training and CD are involved in any programme, and this concern is well-founded as poaching talent is an explicit policy of certain large funders. Most CD interventions focusing on development are now very aware of this and the current study shows clearly that with the correct structure and motivation, capacity and skills can be strengthened, and mobility encouraged without causing national brain drain. This study has shown that individuals may well move to different organisations within a country, or take up a series of mobility opportunities but within the medium term, they tend to return to their country of origin.

• In terms of achieving outcomes at the wider agriculture sector level, including aspects of the multiplier or ripple effect, the study emphasised the difficulties of influencing at this level due to the number of players involved. It also highlighted the impossible nature of trying to measure this in terms of any form of attribution.

• Where contributions to a particular outcome come from many potential sources and a number of different interventions, attribution of a particular output to one programme is almost impossible.

In evaluating CD interventions it is more useful to consider how a programme contributes to a particular outcome rather than attributing an outcome to one particular intervention. This would help to overcome the dilemma for funders in trying to assess return on investment. As stated in the report Measuring Impact Of Higher Education For Development, from a London International Development Centre conference, 19 to 20 March 2012, ‘a key issue for further investigation is how we move evaluation at the outcome level from the individual to the collective, and this seems relevant to both scholarship and research capacity building interventions. How do we capture the effect and value of interventions on the individual, organisational, institutional and societal level?’ Further work looking into the issue of ‘multiplier effects’ linked with the question of contribution versus attribution is important. If approached as the basis for developing an evaluation model that may be used by a range of CD interventions, then it would be very useful.

ALINe working paper  

41

9. Appendices  

A. AFP Fellows  Rounds 1 to 3 (2004–6)

ARITUA, Valentine Nationality: Uganda Sex: Male Age: 34 years Home Institute Kawanda Agricultural Research Institute (KARI) Partner Institute CSL – Central Science Laboratory Project Partner Dr Julian Smith Project Title Initial studies towards understanding the epidemiology of 'Xanthomonas campestris pv musasearum', the casual organism of Banana Bacterial Wilt.

ASSOGBADJO, Achille Nationality: Benin Sex: Male Age: 31 years Home Institute Université d'Abomey-Calavi Partner Institute University of Ghent Project Partner Prof. Patrick Van Damme, Dept Head Project Title Genetic diversity assessment of the multipurpose baobab tree (Adansonia digitata L) for its sustainable utilisation and management in parkland agroforestry system of Benin (West Africa).

DIALLO, Hortense Nationality: Cote D’Ivoire Sex: Female Age: 39 years Home Institute Université d'Abobo-Adjamé Partner Institute Rothamsted Research Project Partner Prof. Philip Jones Project Title Viruses of papaya in Cote d'Ivoire: biological and molecular characterization.

DIOP, Ndack Ndeye Nationality: Senegal Sex: Female Age: 35 years Home Institute CERAAS (Centre d’Etude Régional pour l’Amélioration de l’Adaptation à la Sécheresse) Partner Institute Université de Paris Xii Project Partner Dr Anne Repellin Project Title Isolation and characterization of cowpea cystatin genes, involved in drought resistance.

EKUE, Marius Nationality: Benin Sex: Male Age: 33 years Home Institute Université d'Abomey-Calavi Partner Institute Georg-August University of Göttingen Project Partner Prof. Dr. Reiner Finkeldey Project Title Exploring the genetic diversity of the endogenous agroforestry species ackee (Blighia sapida Konig) in Benin.

FABURAY, Bonto Nationality: Gambia Sex: Male Age: 39 years Home Institute International Trypanotolerance Centre (ITC) Partner Institute Utrecht University Project Partner Prof. Frans Jongejan Project Title Attenuated 'Ehrlichia Ruminantium' vaccine to control heartwater in smallholder livestock production systems in The Gambia.

KADU, Caroline A. Nationality: Kenya Sex: Female Age: 38 years Home Institute The World Agroforestry Centre (ICRAF) Partner Institute Scottish Crop Research Institute, Dundee Project Partner Dr Joanne Russell Project Title 'Allanblackia' in agroforestry systems: developing the tools to manage a new tree crop for small-scale farmers in Africa.

ALINe working paper  

42

LEKE, Walter Nationality: Cameroon Sex: Male Age: 34 years Home Institute Institute of Agricultural Research for Development (IRAD) Partner Institute Swedish University of Agricultural Science Project Partner Dr Anders Kvarnheden Project Title Molecular epidemiology of emerging Begomoviruses that infect Okra and Watermelon vegetable crops and the whitefly haplotypes in Cameroon.

MATOLE, Nompumelelo (Mpumi) Nationality: South Africa Sex: Female Age: 31 years Home Institute ARC-Roodeplaat Vegetable and Ornamental Plant Institute Partner Institute IGER (Institute of Grassland & Environmental Research) Project Partner Dr Mike Humphreys Project Title Characterisation of drought tolerance in Cowpea [Vigna unguiculata (L.) Walp] mutant lines using molecular markers for use in marker assisted plant breeding.

NAMAGANDA, Josephine Nationality: Uganda Sex: Female Age:48 years

Home Institute Kawanda Agricultural Research Institute (KARI) Partner Institute University of Leeds Project Partner Prof. Howard Atkinson Project Title Environmental biosafety of and banana improvement for subsistence growers in Uganda.

TIMM, Alicia Nationality: South Africa Sex: Female Age: 27 years Home Institute Stellenbosch University Partner Institute Agroscope Changins-Wädenswil Research Station ACW Project Partner Dr Juerg E. Frey Project Title Molecular diagnosis and genetic diversity of economically important thrips species present in southern Africa.

YEBOAH, Edward Nationality: Ghana Sex: Male Age: 43 years Home Institute Soil Research Institute Partner Institute Rothamsted Research Project Partner Dr Saran SOHI Project Title Storage of organic carbon in aggregate and density fractions: the influence of organic resource quality. Rounds 4 and 5 (2006–7)

AYELET, Gelagay Nationality: Ethiopia Sex: Male Age: 34 years Home Institute National Veterinary Institute Partner Institute Institute for Animal Health, Pirbright Project Partner Dr Madhuchhanda Mahapatra Project Title A study on molecular epidemiology of foot and mouth disease (FMD) virus circulating in Ethiopia.

BUA, Bosco Nationality: Uganda Sex: Male Age: 39 years Home Institute Kyambogo University Partner Institute University of Bristol Project Partner Prof. Gary Foster Project Title Assessing the genetic diversity of cassava brown streak virus and resistance to cassava brown streak disease from Uganda.

GEBEYEHU, Seyoum Nationality: Ethiopia Sex: Male Age: 35 years Home Institute Ethiopian Agricultural Research Organisation Partner Institute Institut de Recherche pour le Développement (IRD) Project Partner Dr Philippe Lashermes Project Title Innovative mass characterisation of coffee genetic resources ('Coffea arabica' L) collected in Ethiopia.

ALINe working paper  

43

KARIMURIBO, Esron Nationality: Tanzania Sex: Male Age: 41 years Home Institute Sokoine University of Agriculture Partner Institute Moredun Research Institute Project Partner Dr Keith Ballingall Project Title A molecular genetic investigation of mastitis resistance mechanisms in 'Bos indicus' Maasai and Boran cattle in Tanzania. KOUASSI, Koffi II Nazaire Nationality: Cote D’Ivoire Sex: Male Age: 44 years Home Institute Laboratoire Central de Biotechnologies (CNRA) Partner Institute Central Science Laboratory (CSL) Project Partner Dr Julian James Smith Project Title Development of diagnostic tools and analytical processes for the detection of banana virus diseases in Cote d'Ivoire.

KOUTOU, Mamadou Nationality: Burkina Faso Sex: Male Age: 33 years Home Institute Environment & Agriculture Research Institute (INERA) Partner Institute Swedish University of Agricultural Sciences (SLU) Project Partner Dr Anders Kvarnheden Project Title Molecular characterization of begomoviruses causing Tomato Yellow Leaf Curl Disease (TYLCD) in Burkina Faso.

SSEKYEWA, Charles Nationality: Uganda Sex: Male Age: 45 years Home Institute Martyrs University Partner Institute University of Bangor Project Partner Dr Katherine A. Steele Project Title Molecular Characterization of Tomato leaf curl disease causing viruses in Uganda.

YIMAMU, Fassil Nationality: Ethiopia Sex: Male Age: 44 years Home Institute Mekelle University Partner Institute University of Wales Bangor Project Partner Dr Philip Andrew Hollington Project Title Screening of lentils for salinity and boron tolerance.

Rounds 6 (2007–8)

AMOSA, Esayas Nationality: Ethiopian Sex: Male Age: 36 years Home Institute Jimma Agricultural Research Centre Partner Institute Rothamsted Research Project Partner Dr Toby Bruce Project Title Development of semiochemical-based management methods for the coffee berry borer, 'Hypothenemus hampei' (Ferrari) (Coleoptera: Scolytidae).

DAWO BARI, Fufa Nationality: Ethiopian Sex: Male Age: 35 years Home Institute Addis Ababa University Partner Institute Friedrich-Loeffler-Institut (FLI) Project Partner Prof. Christian Grund Project Title Molecular characterisation of Newcastle Disease (ND) virus isolates in Ethiopia.

KAJOBE, Robert Nationality: Ugandan Sex: Male Age: 38 years Home Institute National Agricultural Research Organisation (NARO) Partner Institute Central Science Laboratory (CSL) Project Partner Dr Gay Marris Project Title Development of appropriate surveillance systems for honeybee pests and diseases for improved production of honey and other bee products in Uganda.

OLUWAFEMI, Sunday Nationality: Nigerian Sex: Male Age: 48 years Home Institute Bowen University Partner Institute Rothamsted Research Project Partner Dr Michael Birkett Project Title The control of leafhopper pests of maize by induced or constitutive plant defence.

ALINe working paper  

44

OWUSU-ANSAH, Frank Nationality: Ghanaian Sex: Male Age: 31 years Home Institute Cocoa Research Institute of Ghana [CRIG] Partner Institute University of Reading Project Partner Prof. Robert Curnow Project Title The statistical basis for cocoa clone selection - supporting the breeding programme in Ghana.

TEDIHOU, Ekanao Nationality: Togolese Sex: Male Age: 35 years Home Institute International Institute of Tropical Agriculture (IITA) Partner Institute University of Aarhus Project Partner Dr Niels Holst Project Title The factors that influence 'Aspergillus flavus' strains and toxin expression in maize in different agro-ecozones of Benin: Forecast model.

Round 7 (2008)

ADEBOLA, Patrick Nationality: Nigerian Sex: Male Age: 40 years Home Institute ARC-Vegetable and Ornamental Plant Institute Partner Institute Central Science Laboratory Project Partner Dr Julian Smith Project Title Molecular characterisation of Alternaria species of Sweet Potato and development of a host resistance screening protocol.

FANKEM, Henri Nationality: Cameroon Sex: Male Age: 40years Home Institute University of Douala Partner Institute Scottish Crop Research Institute (SCRI) Project Partner Dr Timothy Simon George Project Title Selecting indigenous phosphate solubilizing fungi (Apergillus, Penicillium) for crop improvement in nutrient deficient soils of Cameroon.

GBADEGESIN, Michael Nationality: Nigerian Sex: Male Age: 43 years Home Institute University of Ibadan Partner Institute University of Bath Project Partner Dr John R. Beeching Project Title Characterisation of root-specific promoters for stacking desirable traits in important African cassava varieties.

NYONI, Benjamin Nationality: Malawian Sex: Male Age: 25 years Home Institute Malawi-Liverpool Wellcome Trust Partner Institute Rothamsted Research Project Partner Prof. Linda Field Project Title Investigating the molecular mechanisms of insecticide resistance in red spider mite and cotton bollworm.

ALINe working paper  

45

Round 8 (2008–9)

ADEJUYIGBE, Christopher Nationality:Nigerian Sex: Male Age: 44 years Home Institute University of Agriculture (Nigeria) Partner Institute Coventry University Project Partner Prof. Phil Harris Project Title Characterization and calibration of organic residues and composts for sustainable organic maize production.

BEKELE FIYSSA, Berhanu Nationality: Ethiopian Sex: Male Age: 41 years Home Institute Plant Protection Research Centre (PPRC) Partner Institute Nottingham University Project Partner Dr Matthew John Dickinson Project Title Molecular methods for detection and characterisation of phytoplasmas and viruses associated with citrus and papaya in Ethiopia.

BYAMUNGU, Mechtilda Nationality: Tanzanian Sex: Female Age: 52 years Home Institute Tsetse & Trypanosomiasis Research Institute Partner Institute The International Atomic Energy Agency Project Partner Mamadou Lelenta Project Title Use of molecular diagnostic techniques to determine epidemiological status of animal trypanosomiases in North-Eastern Tanzania.

GUGSA, Likyelesh Nationality: Ethiopian Sex: Female Age: 39 years Home Institute Ethiopian Institute of Agricultural Research (EIAR) Partner Institute University of Bern Project Partner Prof. Cris Kuhlemeier and Dr Zerihun Tadele Project Title Tackling the major yield limiting factor in TEF using conventional and biotechnological improvement techniques.

JOANNIS, Tony Nationality: Nigerian Sex: Male Age: 54 years Home Institute National Veterinary Research Institute, VOM Partner Institute Istituto Zooprofilattico Sperimentale delle Venezie (IZSVe) Project Partner Dr Giovanni Cattoli Project Title Experimental studies on the pathogenicity and environmental resistance of distinct H5N1 avian influenza virus genotypes circulating in Nigeria.

RAMGAREEB, Sumita Nationality: South African Sex: Female Age: 37 years Home Institute South African Sugarcane Research Institute Partner Institute Natural Resources Institute Project Partner Prof. David Hall Project Title Identification of volatiles from push-pull plants and 'Fusarium' species that serve as repellents or attractants to the cane borer 'Eldana saccharina'

Round 9 (2010)

ANYANGA, Walter Nationality: Ugandan Sex: Male Age: 50 years Home Institute National Semi-Arid Resources Research Institute Partner Institute University of Hohenheim Project Partner Dr Volker Hahn Project Title Using SSR markers as a marker assisted selection for resistance to Sclerotinia sclerotiorum in sunflower (Helianthus annuus L.).

BERRY, Shaun Nationality: South African Sex: Male Age: 34 years Home Institute South African Sugarcane Research Institute Partner Institute Rothamsted Research Project Partner Dr Rosane Curtis, PPM with Dr Mike Birkett, BCH Project Title Reducing nematode damage of sugarcane for small-scale farmers in South Africa using induced resistance strategies.

ALINe working paper  

46

DOUDO, Mohammed Nationality: Sudanese Sex: Male Age: 31 years Home Institute Central Veterinary Research Laboratories (CVRL) Partner Institute Institute for Animal Health, Pirbright Project Partner Dr Satya Parida Project Title Development and evaluation of FMDV non-structural protein (3D) antibody assay to detect and control FMD infection in endemic countries.

MWAURA MUTUA, Peter Nationality: Kenyan Sex: Male Age: 32 years Home Institute Jomo Kenyatta University of Agriculture & Technology Partner Institute Wageningen University, Netherlands Project Partner Dr Cees Waalwijk Project Title Molecular characterisation of Endophytic 'Fusarium oxysporum' isolates from Kenya: Towards a novel strategy of banana Parasitic nematodes management in Kenya.

NOWAKUNDA, Kephas Nationality: Ugandan Sex: Male Age: 44 years Home Institute Kawanda Agricultural Research Institute Partner Institute University College, Cork Project Partner Prof. Jorge Oliveira Project Title Profiling metabolites of East African highland bananas to enhance their improvement and utilisation.

NZALAWAHE, Jahashi Nationality: Tanzanian Sex: Male Age: 31 years Home Institute Tsetse & Trypanosomiasis Research & Control Centre Partner Institute University of Bristol Project Partner Prof. Wendy Gibson Project Title Development of 'T.vivax' LAMP for Bovine Trypanosomiasis studies in Kigoma region: Western Tanzania.

TAMIRU-ABAMO, Amanuel Nationality: Ethiopian Sex: Male Age: 33 years Home Institute Habitat Management Programme, ICIPE Partner Institute Rothamsted Research Project Partner Dr Toby Bruce Project Title Stemborer Oviposition-induced defense volatiles in maize and their effect on parasitoids and subsequent oviposition.

TEFERA, Takele Abayneh Nationality: Ethiopian Sex: Male Age: 40 years Home Institute Addis Ababa University Partner Institute Norwegian School of Veterinary Science Project Partner Prof. Henning Sørum Project Title Isolation and characterization of 'Edwardsiella tarda' species from fresh water fish species of Ethiopia.

ALINe working paper  

47

B. Key Informant Interview Questions Note on CONFIDENTIALITY Explain why the study is taking place – in order to learn from the AFP and feed lessons into other programmes that are being considered by funders. Preliminary question:

• What are you doing now? Are you still doing what you did when you took up the fellowship?

Five areas to explore: 1. Practical aspects of the programme

(Trying to find out view on the framework of the programme: was the structure of visits and support correctly targeted; was it managed well; management; targeting etc.)

a. Some fellows have said they would have liked their placements to be for a longer period of time. How would your organisation have covered if you had been away for longer?

b. It has been suggested that it would be good to have an option for a period of time spent in Africa either to validate results or to do fieldwork. What are your views? And how might that work?

c. Were there any particular aspects regarding the practical arrangements that you would like to raise?

2. The project

(Gain views on the actual time in Europe including mentoring, knowledge gained, building relationships) a. Would your project have benefited from other features e.g. specifically tailored training?

b. Did you have a non-scientific mentor (advisor)? How did the mentoring work and are there examples of you becoming a mentor yourself?

3. Influence on individual over longer term

(How did the fellowship affect career advancement; brain drain; your professional effectiveness? How do power plays affect your effectiveness: individual relationships; institutional cultures; organisational politics?)

a. How did the fellowship help you in terms of gaining leadership skills?

b. What has particularly helped or hindered you in terms of being effective: here I am looking for ways that you work with other people either one-to-one or at an organisational level or maybe with people outside your organisation.

c. How has the fellowship affected your career direction (your job) and your motivation (enthusiasm)?

4. Application of knowledge

(On return home what happened? Was the knowledge disseminated; was it taken up; was there any measurable benefit?)

a. What was it like returning to your African organisation after being on your AFP placement? How did your work go when you got back?

ALINe working paper  

48

b. Who are the end-users of your research: farmers, extension agents, agribusinesses, other scientists etc.? How did you work with these users of your research prior to your placement in order to target your research? Do you work more with them now?

c. Ripple effect: How have you influenced your home organisation / partner organisations / community groups / farmers as a result of your placement?

d. Networking: How did the fellowship help you to network with people or help you to build new networks with Europe or Africa?

5. Long-term partnership

(Establishment of a longer-term partnership with follow-up work? Benefits to African organisation?) a. What could the programme have done to make sure that your home (African) organisation

supported you and was more involved in applying your work when you returned home?

b. Thinking about building the capacity of your organisation: would it be more useful to spend money on fellowships or are there better ways to use money to build the capacity of your home institute?

ALINe working paper  

49

10. References Bickman, L. (1990) New Directions for Program Evaluation. Special Issue: Advances in Program Theory 47: 1–4 Conway, G., T. Arnold, H. Carsalade, L. Fresco, P. Hazell, N. Ngongi, J. von Braun, L. M. Sibanda, R. Tabo, D. Radcliffe and P. Pingali. (2010) 'The Montpellier Panel Report: African and Europe: Partnerships for agricultural development.' The Imperial College Report. Chen, H. (1990) Theory-Driven Evaluations. Newbury Park, CA: Sage DFID (2006). An Integrated Approach to Capacity Development, Renewable Natural Resources Research Strategy, London: Department for International Development (DFID). Donaldson, S. I. and M.W. Lipsey (2005) ‘Roles for Theory in Contemporary Evaluation Practice: Developing Practical Knowledge’ Handbook of Evaluation: 56-75 Douthwaite, B., S. Alvarez, et al. (2008) 'Participatory Impact Pathways Analysis: A practical method for project planning and evaluation.' Institutional Learning and Change Initiative (ILAC) Brief 17. ECPDM (2003) 'Learning about capacity development through its evaluation: experiences from Africa, Asia, Europe and the Americas' Capacity.org 17. Fanzo, J. C. and E. K. Gallin (2006) 'The Doris Duke Clinical Scientist Development Award', Enhancing Philanthropy's Support of Biomedical Scientist: Proceedings of a Workshop on Evaluation, G. R. Reinhart. Washington D.C.: The National Academies Press: 11-20. Funnell S. (1997): ‘An Adaptable Tool for Designing and Evaluating Programs’, Evaluation News and Comment 6(1): 5-17 Horton, D., (ed) (2011) 'Evaluating Capacity Development' in Capacity.org, issue 43, September 2011.. Horton, D. and R. Mackay (2003). 'Using Evaluation to Enhance Institutional Learning and Change: Recent experiences with agricultural research and development', Agricultural Systems 78: 127-142. Ionescu-Pioggia, M. and G. Pion (2006) 'Burroughs Wellcome Fund Evaluation Strategy' Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation, G. R. Reinhart. Washington D.C.: The National Academies Press. Isaacs, K. (2006) 'The Lucille P. Markey Charitable Trust Scholars Program' Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. G. R. Reinhart. Washington D.C.: The National Academies Press: 1-10. LIDC (2012) Report on a conference ‘Measuring Impacts of Higher Education for Development’, London, 19-20 March 2012 Mann, J. (2008) 'Making a difference', Rothamsted International African Fellows Programme Evaluation Report 2008, Rothamsted International African Fellows Programme.

ALINe working paper  

50

McNeely, C. L. and O'Brien, C. (2006) 'Exploring Program Effects on Life Sciences Faculty Diversity: Assessing the Ford Foundation Postdoctoral Fellowships for Minorities' Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation G. R. Reinhart. Washington D.C.: The National Academies Press. Nagarajan, N. and M. Vanheukelen (1997) ‘Evaluating EU expenditure programmes: a guide to intermediate and ex-post evaluation’ Luxembourg: Office for Official Publications of the EuropeanCommunities. OECD (2006) 'The Challenge of Capacity Development: Working towards good practice.' Paris: Organisation for Economic Development and Cooperation. OECD (2011) Research Fellowships and Conference Sponsorship - Co-operative Research Programme from www.oecd.org/document/40/0,3746,en_2649_33903_42629992_1_1_1_1,00.html. Ofir, Z., Van Wyk, B., et al. (2008) 'Comparative Evaluation of the G&D-Rockefeller and Borlaugh Women in Science Fellowship Programs', Working Paper 49. RUFORUM (2011) Monitoring and Evaluation Strategy: Tracking performance, progress and pathways to impactful capacity building in agriculture, Kampala: Regional Universities Forum for Capacity Building in Agriculture (RUFORUM). Simister, N. and Smith, R. (2010) 'Monitoring and Evaluating Capacity Building: Is it really that difficult?' INTRAC Praxis Paper 23. Taylor, P. and Clarke, P. (2007) Capacity for a change. Document based on outcomes of the ‘Capacity Collective’ workshop. Capacity Collective Workshop, Dunford House. UNDP (2009) Capacity Development: a UNDP primer, New York: UNDP. Walker, T., Maredia, M., et al. (2008) Strategic Guidance for Ex Post Impact Assessment of Agricultural Research. S. P. o. I. Assessment, Rome: GGIAR Science Council. Weiss, C. H. (1995) ‘Nothing as Practical as Good Theory:Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families’, New Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts: 65-93 Eds J. l. Connell, A. C. Kubisch, L. B. Schorr, C. H. Weiss. The Aspen Institute. Weiss, C. H. (1998) ‘Evaluation: methods for studying programs and policies’. Prentice Hall. 372pp

   

ALINe working paper  

1

 

ALINe (Agricultural Learning and Impacts Network) 3 Dean Trench Street London SW1P 3HB T: +44 207 148 0914 E: info@aline.org.uk W: www.aline.org.uk