1

Evaluation of the Commonwealth of Learning 2009-2012 Plan

Report 13 December 2011

Patrick Spaven Brighton, England

2

Contents Page

Abbreviations and acronyms 3

Acknowledgements 5

Executive summary 6

Introduction 16

Findings 21

Conclusions and Lessons 74

Recommendations 84

Annexes

1 Terms of reference 88

2 Evaluation framework 91

3 COL Stakeholder Survey 2010 – Conclusions 96

4 Board member and advisor topic guide 97

5 Principal COL documents consulted 99

6 People consulted 100

3

Abbreviations and Acronyms AC Additional contribution ACELO Advanced Certificate in Education and Life Orientation CAPA Commonwealth Association of Polytechnics in Africa CCTI Comm CEE Centre for Environment Education CEMBA/PA Commonwealth Executive Master of Business Administration/Public Administration CEMCA Commonwealth Educational Media Centre for Asia CEO Chief Executive Officer CFS Child-Friendly Schools CFT Competency Framework for Teachers CIDA Canadian International Development Agency CLN Community Learning Network COL Commonwealth of Learning COL RIM COL Review and Improvement Model COMOSA Commonwealth Association of Open Schools ComSec Commonwealth Secretariat CR Community radio DFID Department for International Development DHO District Health Office eLIO e-Learning for International Organisations ES Education Specialist FSD Flexible Skills Development HE Higher education HEI Higher education institution ICT Information and Communications Technology ICT4E ICT for education KM Knowledge Management L3F Lifelong Learning for Farmers Logframe Logical framework MDG Millennium Development Goal M&E Monitoring and evaluation MOU Memorandum of Understanding MTC Media and Training Centre for Health (South Africa) NABARD National Bank for Agriculture and Rural Development (India) NGO Non-government organization NOUN Nigerian National Open University NTI National Teachers Institute ODA Official development assistance ODL Open and distance learning/education OECD Organization for Economic Cooperation and Development OER Open educational resources PA Programme Assistant PCF Pan Commonwealth Forum on Open Learning PI Performance indicator QA Quality assurance QATTE Quality assurance toolkit for teacher education RBM Results based management RETRIDAL Regional Training and Research Institute for Open and Distance Learning SADC Southern African Development Community SADC-CDE Southern African Development Community Centre for Distance Education SAIDE South African Institute for Distance Education SHG Self-help group TC Teachers’ college TE Teacher education

4

TESSA Teacher Education in Sub-Saharan Africa TQF Transnational Qualifications Framework ToC Theory of Change ToRs Terms of reference TVET Technical/Vocational Education and Training TYP Three Year Plan UNESCO United Nations Educational, Scientific and Cultural Organization UNHCR United Nations High Commissioner for Refugees UNICEF United Nations Children’s Emergency Fund VC Vice Chancellor VUSSC Virtual University for the Small States of the Commonwealth WFHF William and Flora Hewlett Foundation WHO World Health Organization

5

Acknowledgments Evaluating COL in 2011 was challenging but less so than in 2008-09 and 2005-06. Nevertheless the cooperation and support of many people helped to make it a feasible and interesting exercise. COL’s Vice-President, Asha Kanwar created the vision for the evaluation and ensured that it stayed on track and had the resources it needed. Lydia Meister was meticulous and responsive in the administration of the evaluation. Other COL staff gave generously of their time to complete questionnaires, provide monitoring information, take part in one-to-one interviews and group discussions and review the evaluation draft report. The independent evaluators were all cooperative in agreeing to have their methodologies, findings and conclusions rigorously probed in lengthy phone conversations. Most Board members and advisers either tackled a lengthy questionnaire or engaged in real-time discussion with great commitment. COL’s strategic and programme partners consulted provided valuable insights into their experience of working with COL and the changes that had led to.

6

Executive Summary The Commonwealth of Learning 1. The Commonwealth of Learning (COL) is an intergovernmental organisation created

by Commonwealth Heads of Government to encourage the development and sharing of open learning/distance education (ODL) knowledge, resources and technologies. COL’s mission is to help governments and institutions to expand the scope, scale and quality of learning. It is mandated to be in the vanguard of technological change in education and training.

2. COL’s assistance targets developing Commonwealth countries. Its work is currently

grouped into two “sectors”: Education and Livelihoods and Health.

3. Each Sector encompasses four programmes or “Initiatives” which are constant throughout the Three Year Plan (TYP). In Education the Initiatives are:

o Open Schooling o Teacher Education o Higher Education o Virtual University for Small States of the Commonwealth (VUSSC)

In Livelihoods and Health they are:

o Skills Development o Learning for Farmers (best known as L3F) o Healthy Communities o eLearning

4. Each Initiative contains a small number of “Activities” which are permitted to

change from year to year, but tend to stay the same within a TYP period.

5. COL also commits itself to three “cross-cutting” dimensions in its work: Quality in ODL Gender equality Use of appropriate technologies.

The evaluation 6. COL has commissioned this evaluation of its work to date under the 2009-2012

TYP. The main purpose of the evaluation is to ascertain what difference COL has made in this period of roughly 28 months.

7. The four key questions for the evaluation are: To what extent has the Commonwealth of Learning made a difference in the

life of its current TYP? What lessons can COL, its Board of Governors and partners, take from the

operation of the plan into the next TYP? How appropriate are the two strategic sectors, Education and Livelihoods and

Health, in realising the aims of the current TYP? What has been the impact of COL’s work since the 2003-2006 TYP?

8. In the evaluation, each Initiative – and CEMCA, COL’s regional agency for Asia – is

assessed primarily through a sample project or other coherent set of activities within the Initiative. These samples were chosen by COL at or near the beginning of

7

the triennium, each submitted to an evaluation – referred to as a longitudinal study - by an external consultant.

9. The TYP evaluation was commissioned in early August 2011. The most intensive

period of work was 11-21 October when the evaluator visited COL’s office in Vancouver.

10. The principal sources of data were:

Customised reports on progress against the corporate performance indicators. Independent evaluations of 10 areas of COL’s work. Telephone interviews with the authors of these evaluations. Telephone interviews or email dialogue with a sample of COL partners and

clients. Consultations with COL Board members and advisors, in most cases through

telephone or face-to-face interviews, or electronically through a list of qualitative questions.

Interviews and email dialogue with COL and CEMCA staff. Facilitated group discussions with COL staff. Other documents such as Initiative and Activity logical frameworks and Board

papers. Limitations

11. The evaluation work did not include direct engagement with COL’s work in the field.

The external evaluations therefore were relied on heavily. A few of these evaluations were penetrating and very helpful. Unfortunately some were perfunctory and of limited value. Telephone conversations with the evaluators, partners and clients helped to fill some, but not all, of the gaps.

12. The evaluation is required to assess the impact of COL’s work since the 2003-06 triennium. This has proved to be extremely difficult for two main reasons. Few of COL’s current areas of work have continuity back to 2003-06; and of

these only three – VUSSC, L3F and the Regional Training and Research Institute for Open and Distance Learning (RETRIDAL) - were submitted to external evaluations.

With the exception of the one for L3F, the external evaluations were not designed as impact assessments. Furthermore, the L3F evaluation focused on only one project – Vidiyal – which did not begin in earnest until 2009.

Findings and conclusions Open Schooling 13. The Initiative promotes open schooling as a means of providing increased access to

learning opportunities at secondary level and supports countries to plan and implement it. In this triennium COL has undertaken an ambitious portfolio of work including multi-faceted support to new and aspiring open schools; the first institutionalised networking of open schools through COMOSA and an on-line community; and a challenging project to build capacity for OER production in six countries, which attracted funding from the William and Flora Hewlett Foundation.

14. In common with other COL Initiatives, not all the open schooling interventions have borne fruit within the expected time-frames, and some have stalled for a variety of reasons. COL has learnt from the difficulties it has faced, and has responded flexibly.

8

15. Set in the context of this relatively new terrain, COL is undoubtedly a very important and valued partner in the open schooling sector in the developing Commonwealth.

Teacher education 16. COL’s Teacher Education Initiative promotes and supports the use of ODL to

increase access to, and the quality of, teacher education. COL’s work is currently an eclectic mix. It is not difficult for COL to find footholds; but it has not always found it easy to obtain leverage in this subsector, particularly with strategic partners.

17. Its flagship work has been a project in partnership with UNICEF to integrate the concept of “Child Friendly Schools” in several countries, mostly in Africa. The project has been well conceived and well executed.

18. COL’s other work in this Initiative has been valuable, if unspectacular. COL needs to

take a hard look, before the end of the next triennium, at how to optimise its contribution in this important subsector.

Higher Education

19. COL’s Higher Education (HE) Initiative is broad-based, designed to support higher

education institutions to have policies, systems and staff competencies that support the use of ODL for increased learner access, completion rates and enhanced content.

20. Its two most prominent areas of work in the last 2-3 years have been the revision of materials for the Commonwealth Executive Master of Business/Public Administration (CEMBA/PA); and the development - and assistance with implementation in several HEIs – of a low-cost internal quality audit tool for HE known as the Review and Improvement Model (COL RIM).

21. The Initiative also includes oversight of RETRIDAL. An external evaluation of

RETRIDAL, based in the National Open University of Nigeria (NOUN), found that the institute had undertaken very little work for about two years, principally because of internal management issues.

22. The HE sector is much larger and more diverse – and the distinction between

distance and face-to-face education less clear-cut - than in the 1990s when COL was a major player in promoting ODL for the HE subsector. Nowadays it is increasingly difficult for COL to find a niche.

23. Besides this, the performance of COL’s HE work in the recent past attracted critical

comment from several stakeholders. 24. A big question for COL in the coming triennium is: should it strive to look for that

niche through a continuing commitment to a free-standing HE Initiative? Or should COL contribute to the subsector more obliquely – through its championing of OERs for example - and only where clear-cut opportunities arise?

Virtual University for Small States of the Commonwealth (VUSSC) 25. The Virtual University for Small States of the Commonwealth (VUSSC) is a group of

institutions from small states – currently 33 - brought together through COL since 2005, to collaboratively develop free content resources for use in an educational context.

9

26. VUSSC’s geographical coverage, and the apparent commitment of member countries, are impressive. Participants say they feel part of a grand enterprise. To have maintained this momentum over 6-7 years is a considerable achievement. Few organisations have the staying power to have managed this process in the way that COL has done.

27. At outcome level however the results are less impressive. Progress in the

production of courses in the member countries has been slower than anticipated. The main stumbling block has been the translation of material produced in the VUSSC training and course development workshops (or “bootcamps”) into actual course modules.

28. To overcome this problem, VUSSC is now concentrating on producing complete

courses at the bootcamps. This is already leading to take-up, so it seems like the right move for now.

29. VUSSC has also completed the construction of the Transnational Qualifications

Framework which is designed to promote educational and labour mobility. Skills Development 29. COL’s Skills Development Initiative supports ODL for skills development particularly

through new approaches that emphasise technology-enhanced, flexible and blended (FaB) teaching and learning.

30. There is a particular emphasis on skills for the informal sector which is the most

neglected in many countries and yet is increasingly likely to be the destination of school leavers without higher secondary qualifications.

31. COL is making good progress with this work. Its flagship project – Flexible Skills

Development for members of the Commonwealth Association of Polytechnics in Africa – is making headway, particularly in Kenya where four institutions are participating. It needs to consolidate, monitor, reflect and consult before attempting to branch out.

Learning for Farming 32. The basic aim of the innovative Learning for Farming Initiative (commonly known as

L3F) programme is, through a variety of ODL and ICT methodologies, to build capacity among the rural poor which leads to more productive and sustainable farming and more secure livelihoods. Convincing banks to lend to subsistence farmers on the strength of their learning activities is the linchpin in the standard L3F model.

33. COL has successfully tested this model in South Asia. It has shown that it can – in

certain circumstances - make a significance difference to people’s livelihoods. Growing empowerment among the predominantly women farmers also appears to be a tangible outcome.

34. COL is testing variations on the model in other regions. It is aware that time is

running out for the micro field testing of L3F and that it needs to demonstrate that it can rapidly scale up without the intensive care. COL needs to work hard at finding advocacy platforms for L3F that engage new types of partners for up-scaling and diversifying.

10

Healthy Communities 35. Healthy Communities is another of COL’s three highly focused – predominantly

single model - Initiatives. It promotes and facilitates participatory learning – predominantly about health, and through the medium of community radio (CR).

36. COL is supporting activity in an impressive number of locations. The wide reach

comes from an arms-length approach, working with intermediaries. The numbers reached at present are relatively few in any location. This is typical of the modelling phase of a COL Initiative.

37. Part of the value of COL’s work in Healthy Communities is in the refinement of the

model and its testing in a number of different social, cultural, economic and political contexts. As with other models, COL needs to validate it, refine it, and if appropriate diversify it, through the capture and use of evidence.

Integrating eLearning 38. Integrating eLearning is another of COL’s broad-brief Initiatives, but is not sector-

based. It is concerned with promoting and supporting the use of ICT in teaching and learning in both formal and non-formal contexts and at all levels.

39. Following the transfer of responsibility of WikiEducator – that had dominated the

Initiative in the previous triennium - it took time for COL to identify a new direction to follow. Once this was done, however, the work gathered momentum very quickly. This was helped by two factors: A strategic partnership with Commonwealth Secretariat (ComSec) and

Microsoft came together and took off very quickly with ICT for education work in the Caribbean.

The use of pre-existing models and materials that needed adapting and some re-purposing but not re-inventing.

40. The ICT4E work in the Caribbean is the largest and most coherent set of

interventions in this Initiative. It has already led to some solid early outcomes in three countries, influencing policy and supporting the countries in the adoption of a competency framework for integrating ICT in school teaching.

41. COL’s partners want to push on in the Caribbean and begin similar work in the

Pacific. The latter may be premature without consolidation and research on outcomes. The apparent effectiveness of the partnership is an important development for COL.

CEMCA 42. The Commonwealth Educational Media Centre for Asia (CEMCA) is COL’s regional

agency for Asia. It is the only unit outside Vancouver that COL manages directly. Its mission is to promote “the meaningful, relevant, and appropriate use of information and communication technologies to serve the educational and training needs of Commonwealth member states in Asia”. It received a programme budget of CDN$350,000 in 2011. It has a diverse range of outputs, the most prominent of which is support services for CR, mainly in India but also with small steps from a standing start in Bangladesh.

43. CEMCA, in 3-4 years, has become a valued strategic partner with the Government

of India in the promotion of, and support for, CR. CEMCA’s direct work with CR start-ups has equipped around 100 organisations to apply for licences and begin to

11

broadcast. CEMCA and the Government have big plans for the future of their cooperation in CR which would see CEMCA moving to an even more strategic role. This is a remarkable achievement, although understandable when the background of CEMCA’s current director is taken into account.

44. COL needs to ensure CEMCA continues this effective work after the retirement of

the current director in 2012 and integrate its planning and monitoring more closely with COL’s RBM system.

eLearning for International Organisations (eLIO)

45. eLIO is a full-cost recovery service provided by COL which develops and delivers

technology-mediated distance learning courses for the staff of international organizations like WHO, ComSec and UNHCR.

46. eLIO was not within the scope of the evaluation. However it is important to point out that the existence of the service and the way it operates has not been formally challenged in this triennium, unlike in the past.

Cross -cutting themes: gender, quality and appropriate technology

Gender 47. COL aims to be “a model organisation which mainstreams gender in all its

processes and at all levels”. 48. This is an ambitious aim. In pursing it, COL has invested a considerable amount of

time and money in the last decade in creating frameworks and tools for gender mainstreaming in the organisation and its programmes. COL also provides gender-related resources for its programme partners, and other external stakeholders and members of ODL communities.

49. At the centre of the COL’s gender action plan are two commitments:

At least one gender-related activity will be implemented in each Initiative. The longitudinal study for each Initiative will provide evidence whether COL’s

interventions have advanced gender equality or not with appropriate data from the field.

50. The former commitment appears to have been achieved. In terms of the provision

of evidence, the second commitment has not been fulfilled. Several of the evaluations failed to address the question of gender advancement and merely reported on gender balance in monitored activity such as workshops. This is not to say that COL has failed to advance gender equality through its work. It simply has not provided enough evidence except in its L3F Initiative.

51. COL should refresh the use of its Gender Keys – a simple tool for gender

mainstreaming in planning, implementation, monitoring and evaluation. It also needs to choose, and possibly refine, a gender outcome statement and 3-4 indicators from the long list in its Gender Action Plan, include them in the Logic Model for the next TYP, and purposefully work towards them.

Quality and appropriate technology

52. COL has not actively monitored these two cross cutting dimensions in the

triennium. There is no evidence however that it has failed to promote quality and appropriate technology in any aspect of its work. Neither of these will feature explicitly as cross-cutting dimensions in the next TYP.

12

Knowledge Management

53. COL ‘s Knowledge Management (KM) unit provides knowledge services to external

ODL communities and to the organisation itself. 54. The KM unit has recently re-strategised and is planning an ambitious and diverse

array of new services including some that reach out to customers rather than reside on its website. In line with COL’s commitment to OERs, it is collaborating with ComSec to create portal access to the rapidly expanding stock of OERs produced in developing countries.

55. To optimise KM’s potential for internal collaboration with COL’s programme work,

there needs to be regular and systematic dialogue between KM and the ESs. Results-Based Management 56. Results-Based Management (RBM) is a strategic approach to management that

translates principles and components of the programme/project cycle to the level of the organisation.

57. COL adopted the RBM approach with its 2003-06 TYP. Since then it has been

regularly developing and improving its approach. 58. The evaluation scrutinises in detail COL’s RBM system and practices. COL has made

good progress with its RBM so far. The evaluation makes a number of recommendations for improvement. That is to be expected given the relative immaturity of COL’s experience with RBM. Institutionalising RBM and making it work is a long haul project.

General conclusions 59. To this evaluator, COL looks and feels like a different organisation from the one first

encountered in 2005. In all the teams there is a palpable striving to be relevant and to create deep and long-term effects. There was evidence of this in the 2003-06 triennium, but it was certainly not universal. Progress was made in the 2006-09 and more since then. This does not always make COL a comfortable place to work, but it is transforming the way COL identifies, and carries out its work.

60. In particular COL understands the need to build sustainability into its interventions.

Sustainability was referenced in 2005/06 but was not a dominant concern. 61. COL’s RBM system has played a role in this. The sheer discipline of RBM and the

constant reinforcement of messages about outcomes and sustainability have probably been the main drivers. COL has not yet found a way to fully reap the potential benefits from RBM of strategic learning, but this is not unusual for the state of maturity of COL’s system.

62. Apart from RBM, the change has also been brought about through a complete

renewal of ES staff. These staff were recruited to an organisation with expectations of its programmes that are radically different from 2003.

63. It is important to stress that, with this transition, COL has not lost its disposition

and ability to be needs-driven and to listen to what this means for particular clients. There is always a risk - with pre-set strategic goals, more projectised working, more strategic partnerships and a drive to innovate – of becoming insensitive to

13

local needs and inflexible. Getting the balance right is very difficult but COL seems currently to be achieving this.

64. Probably the clearest manifestation of COL’s transition to more outcomes-oriented

approaches is its concentration on the development and application of innovative models for ODL.

65. In its most complete form, COL’s application of models would be to:

Apply its own thinking and experience – and occasionally research - in preparing the concept or approach for trialling.

Trial it in more than one institutional and geographical context – often in two or more countries or even regions.

Monitor, and assess or evaluate the trial and recycle the learning from this into a new version of the model.

Develop a strategy for replication or up-scaling.

66. In practice, COL is not always able to deliver all of these elements. The process is often messier, iterative, and incomplete. But COL is clearer in this triennium that the development and mobilisation of models are a core competence.

67. To fully realise the potential of models, COL needs to make the following

improvements It needs to build intensive outcome monitoring and evaluation as standard

into its planning for interventions with models. This will require resources. It needs to put more time into researching, discussing, designing and

implementing strategies for replication and up-scaling. 68. Taking into account the balance of stakeholder opinion, the logic of the extension of

COL’s use of models, and the learning about what works that will come from improvements in outcome monitoring and evaluation, this evaluation recommends that COL focuses even more on a small number of substantial medium to long-term interventions. These interventions should be transnational, ensuring that across its interventions all regions get a reasonable share of benefits.

69. COL is not always comfortable working with large-scale projects with pre-determined outputs and outcomes and tight deadlines. It is used to more flexible ways of operating, less full-on and less linear interventions, and emergent definitions of outputs and outcomes. COL does not have a large pool of programme funds, but this is often not a problem as COL’s expertise, ideas and brokerage are what gives it its edge. Large funding packages can work against local commitment, ownership, and sustainability.

70. However COL should also not shy away from external funding opportunities,

particularly where it is ready to take a model to a bigger stage. COL has shown that it can operate effectively in an external funding environment, although it should seriously consider appointing project managers in these circumstances if the scale justifies it.

71. A focus on 4-6 major interventions suggests that COL should reduce the number of

Initiatives. There is no intrinsic reason why COL should remain wedded to the notion of 7-8 separate Initiatives. Although it has tended to retain specialist staff, each with responsibility for a different subsector, this also is not necessarily a “given” for COL. In particular this evaluation is recommending that it keeps under review its commitment to an Initiative devoted entirely to higher education.

72. A strong message from a wide range of informants in this evaluation is to move

more resources to skills development. There a consensus about the unmet need in

14

this area in the developing Commonwealth – and in most countries around the world for that matter. COL is seen to be at the cutting edge of the mobilisation of ODL for skills development.

73. An expansion of work in skills development would have implications for several of

COL’s existing Initiatives. This may require a rethinking of the way COL configures its Initiatives. It is very important that COL’s work in this area benefits from coordination and collaboration across the organisation.

74. There is a strong suggestion from some stakeholders that COL should identify and

focus on “root” solutions: work which – with minimal adaptation – can support outcomes in more than one subsector. Quality assurance (QA) is a well-established root solution; and COL should consider pooling its resources and experience of QA for potential synergies across educational subsectors and beyond into informal learning.

75. OERs are the obvious emerging root area. COL is working with OERs in several

Initiatives and in KM. It is not immediately clear how much collaboration there is within COL on OERs, but there are likely to be advantages in this. It is not inconceivable that in three years’ time COL has a team that works with OERs across the current Initiative boundaries.

76. Policy is another root solution. Most of COL’s interventions include a policy

promotion component for sustainability. While this needs to be tailor-made at the level of institutions, national policy advocacy for ODL is a slower, more detached process. It may be better pursued outside the boundaries of specific Initiatives.

Principal recommendations (not already explicit in the Conclusions) 77. COL should keep its work in HE under close review. Unless it identifies and delivers

interventions in the next 2-3 years which are widely recognised as making a major contribution to the subsector, the most sensible course may be to wind down HE as a separate Initiative and contribute in other ways to the subsector where a clear rationale presents itself.

78. If COL’s efforts to revive RETRIDAL begin to feel consistently like pushing water uphill, it needs to devise an exit strategy.

79. COL needs to stay with VUSSC but accelerate the transfer of ownership while

reviewing – on the basis of better evidence – the mechanisms, almost certainly introducing more diversity to the model. It also urgently needs to explore sustainable funding mechanisms with its VUSSC partners.

80. COL needs to demonstrate that it can rapidly scale up L3F without the intensive

care provided by the development partners. COL needs to work hard at finding advocacy platforms for L3F that don’t just broadcast its messages but also engage new types of partners for up-scaling and diversifying. There is a need to demonstrate with clarity what the advantages are. This involves evidence through evaluation but also simpler communications.

81. Outcome statements in COL’s TYP Logic Model need to be less generic than in 2009-

12. They need to be pitched at medium-term results. 82. The TYP Performance Indicators need to be more precisely defined with

accompanying technical notes explaining among other things how the indicators will be monitored.

15

83. COL should develop short strategic plans for each Initiative. These should be constructed for the life of the TYP – or even longer in some cases. They should be built around a logic model or theory of change (ToC) that links key actions and outputs for each “Activity” to the Initiative outcome. There would be a strong case for including intermediate outcomes in the models as well. The construction of the logic models/ToCs would best be done collaboratively both within COL and with partners.

84. COL should build Activity logframes around coherent packages of work – or projects

– rather than generic types of work – such as COL’s “strategies”. 85. The development of outcome monitoring, in collaboration with partners, should be

the main item on COL’s agenda for improving its RBM system in the next TYP. 86. COL should institutionalise M&E planning with the implementation of the new TYP. 87. COL needs to consistently raise the level of its capability for commissioning

evaluations. This implies Being a pro-active partner in evaluation design through terms of reference

and dialogue at the inception stage. Improving the recruitment of evaluators by widening the pool and having

sound selection criteria and processes. 88. COL should look more strategically at evaluations; and decisions about this should

be shared. There is often greater value in thematic, cross-cutting evaluations which tend to continue to have relevance after specific interventions have been completed.

89. When an ES leaves, COL should have a clear medium-term framework for the area

to which it intends to recruit a successor. This implies having a clear profile of the person it should be looking for; and providing an effective induction into this framework for the new ES.

16

Introduction The Commonwealth of Learning

90. The Commonwealth of Learning (COL) is an intergovernmental organisation created

by Commonwealth Heads of Government to encourage the development and sharing of open learning/distance education (ODL) knowledge, resources and technologies. COL’s mission is to help governments and institutions to expand the scope, scale and quality of learning. It is mandated to be in the vanguard of technological change in education and training.

91. COL’s assistance targets developing Commonwealth countries. Its work is currently

grouped into two “Sectors”: Education and Livelihoods and Health. Each Sector has its own impact statement. These are: Education Target countries improve the accessibility and quality of their formal education systems at all levels through the use of ODL tools and strategies. Livelihoods and Health The income, livelihoods and quality of life of communities and their members are improved through new knowledge, skills and economic opportunities gained by means of ODL tools and strategies.

92. Each Sector encompasses four generic programmes or “Initiatives” which are

constant throughout the Three Year Plan (TYP). In Education the Initiatives are: Open Schooling Teacher Education Higher Education Virtual University for Small States of the Commonwealth (VUSSC).

In Livelihoods and Health they are: Skills Development Learning for Farmers (L3F) Healthy Communities eLearning.

93. Each Initiative contains a small number of “Activities” which are permitted to

change from year to year, but tend to stay the same within a TYP period. Most of them are generic – such as “strategic planning and policy development” in Skills Development - and relate to several different contexts and locations. A small minority are constructed around more tightly-focused interventions which resemble projects, such as Open Education Resources for Open Schools.

94. COL refers to five “strategies” that it uses to pursue its mission:

Partnerships Models Policies Capacity Materials

95. These strategies – with the exception, in most contexts, of partnerships – are in

fact the intended early outcomes of COL’s work. For example, its work is designed to construct models of effective practice for a diverse range of contexts; and COL supports the development of policies, capacity and materials for ODL. Partnerships

17

can be an outcome – where COL brokers relationships between organisations. Partnership, for COL, however is more often an enabler or enhancer of value creation in its own work.

96. COL also commits itself to three “cross-cutting” dimensions in its work:

Quality in ODL Gender equality Use of appropriate technologies.

97. COL has been operational since 1989 from its headquarters in Vancouver, Canada.

It is financially supported by Commonwealth governments on a voluntary basis. In 2011 the number of countries providing voluntary contributions had risen to 44. The annual budget allocated from COL’s core funding to sector programmes averaged CDN$3.55 p.a. in 2009-2011. COL receives additional contributions from international agencies, foundations and national governments for specific projects and services. These averaged CDN$1.45 million p.a. in the same period.

98. COL directly manages the Commonwealth Educational Media Centre for Asia

(CEMCA) based in Delhi, India. CEMCA’s work is defined in terms of five of the COL Initiatives – all except Teacher Education, VUSSC and Learning for Farmers.

99. COL helped to establish and supports – but does not directly manage - two other

regional centres for ODL: the Regional Training and Research Institute for Open and Distance Learning for West Africa (RETRIDAL) based in Lagos, Nigeria; and the Southern African Development Community Centre for Distance Education (SADC-CDE) based in Gaborone, Botswana.

100. The Commonwealth of Learning commenced its current TYP, Learning for

Development, in July 2009. It will be completed on June 30, 2012. The plan is set within a Results Based Management (RBM) framework. The framework is based on a Logic Model containing a statement of outcome for each Initiative as well as the two impacts referred to in paragraph 91. There are corporate level performance indicators for each of the eight Initiatives and for the three cross-cutting commitments.

The evaluation 101. COL has commissioned this evaluation of its work to date under the 2009-2012

TYP. The overall purpose of the evaluation is to ascertain what difference COL has made in this period. The evaluation is guided by the OECD Development Assistance Committee’s criteria of relevance, effectiveness, efficiency, impact, and sustainability. The four key questions are:

To what extent has the Commonwealth of Learning made a difference in the life of its current TYP?

What lessons can COL, its Board of Governors and partners, take from the operation of the plan into the next TYP?

How appropriate are the two strategic sectors, Education and Livelihoods and Health, in realising the aims of the current TYP?

What has been the impact of COL’s work since the 2003-06 TYP? 102. The main clients for the evaluation are COL and its Board of Governors. Interested

stakeholders are the 54 Commonwealth countries and those with whom COL works. 103. The evaluation’s full terms of reference can be found at Annex 1. 104. An evaluation framework (Annex 2) was agreed in the inception period which

expands on the four key evaluation questions.

18

105. The evaluation of the 2003-06 TYP was COL’s first comprehensive evaluation. It had

wide scope and extensive resourcing, covering most aspects of COL and its work. It resulted in 52 recommendations. This evaluation is shorter and focuses mainly on COL’s programmes.

106. A Stakeholder Survey was conducted by the current evaluator in November 2010. It

looked broadly at COL’s roles, its ways of working and the value it adds. This included COL’s programme support processes. That ground will not generally be re-covered in this evaluation, although it will be necessary to address some of COL’s processes to fully assess its programme effectiveness. In particular there will be an analysis of COL’s RBM system.

107. The conclusions of the 2010 Stakeholder Survey are at Annex 3. 108. Each Initiative – and CEMCA – is assessed primarily through a sample - a project or

some other coherent set of activities within the Initiative. These samples were chosen by COL at or near the beginning of the triennium and submitted to an external evaluation, or what COL called a “longitudinal study”.

109. In each Initiative, a small number of other interventions are also referred to, to

further illustrate the work COL pursues. These other interventions are chosen because they seem significant in scope and relevant to COL’s objectives. However they are not fully triangulated and this evaluation cannot vouch for their performance.

110. COL tends to use the word “partner” for people and organisations with whom it

works. These broadly fall into two categories, although there is sometimes overlap or ambiguity in the roles played.

Organisations that enable (through funding, other resources, access, ideas, or some other complementary asset) COL to create or deliver products and services. This evaluation refers to these as strategic partners. Examples include the Commonwealth Secretariat (ComSec), UNESCO and universities that collaborate in the production of resources, such as toolkits, for others’ use.

Organisations and institutions – including government departments and agencies - that are primarily COL’s clients: beneficiaries of COL’s products and services. These are referred to as programme partners. Examples are organisations whose staff receive capacity-building.

Methodology 111. The evaluation was commissioned in early August 2011. The inception period,

during which the evaluation framework was agreed and work was planned, ran from mid August to late September when interviewing began. The most intensive period of work was 11-21 October when the evaluator visited COL’s office in Vancouver.

112. A draft report was issued on 21 November and circulated among COL – including

CEMCA - staff. This final report was completed in the light of comments received during that process.

113. The principal sources of data were:

Reports on progress against the corporate performance indicators. Independent evaluations of 10 areas of COL’s work. Telephone interviews with the authors of these evaluations. Telephone interviews or email dialogue with a sample of COL partners and

clients.

19

Consultations with COL Board members and advisors, in most cases through telephone or face-to-face interviews, or electronically through a list of qualitative questions (Annex 4).

Interviews and email dialogue with COL and CEMCA staff. Facilitated group discussions with COL staff. Other documents such as Initiative and Activity logical frameworks and Board

papers. A full list can be found at Annex 5. 114. It was agreed to adopt a fuller case study approach than last time. COL’s TYP

outcomes and indicators are somewhat generic. For greater understanding and learning – both COL’s and its stakeholders – there is a need for a more detailed narrative. As with all case study-based evaluations, there are risks in extrapolating from a small number. But with as many as nine studies, patterns usually emerge and it is safer to generalise.

115. A list of people consulted is at Annex 6. Limitations 116. Unlike the evaluation of the 2003-06 Plan, the work did not include direct

engagement with COL’s work. The external evaluations therefore were relied on heavily. A few of these evaluations were penetrating and very helpful. Unfortunately some were perfunctory and of limited value. Telephone conversations with the evaluators, partners and clients helped to fill some, but not all, of the gaps.

117. The choice of areas of work to evaluate was made by COL. In most cases they

represent substantial interventions, but do not necessarily represent a balanced picture of COL’s effectiveness. It has not been possible to fully triangulate areas of work not subjected to external evaluation.

118. COL is a difficult organisation to evaluate. Its work is diverse, dispersed and

somewhat fragmented – although less so than in previous TYP periods. Some Educational Specialists (ES) are working in more than 20 different geographical and institutional contexts, even if their work has fewer overall strands than in the past. Much of COL’s work is exploratory and does not always lead to substantial interventions. Its work tends to be contributory or facilitative – process-oriented. That does not mean the interventions cannot be decisive or pivotal - it is simply less easy to attribute outcomes to COL’s work. This is compounded by COL’s inexperience in applying logical frameworks to much of its work.

119. Monitoring – particularly of outcomes - by COL and its partners has been sporadic.

COL’s reporting against corporate indicators, backed by discussions with the staff concerned, has provided a valuable orientation around the scale of COL’s work. But as in the 2009 evaluation, for insights into the depth, quality and sustainability of the work, there is heavy reliance on the impressions of stakeholders expressed qualitatively.

120. This evaluation is required to assess the impact of COL’s work since the 2003-06

triennium. This has proved to be extremely difficult for two main reasons. Few of COL’s current areas of work have continuity back to 2003-06; and of

these only three – VUSSC, L3F and the Regional Training and Research Institute for Open and Distance Learning (RETRIDAL) - were submitted to external evaluations.

With the exception of the one for L3F, the external evaluations were not designed as impact assessments. Even the L3F evaluation focused on only one project – Vidiyal – which did not begin in earnest until 2009.

20

121. Despite these limitations, the evaluation has a distinct advantage. The same consultant was used for the last three TYP evaluators. He has the advantage, if not of a real-time evaluator’s perspective, at least of three different snapshots. It is thus possible to trace certain changes over this period.

21

Findings Open Schooling 122. This Initiative promotes open schooling as a means of providing increased access to

learning opportunities at secondary level and supports countries to plan and implement it.

123. There are five Activities:

Develop capacity amongst policymakers and ODL practitioners to ensure sustainable open schools.

Establish and develop new models of open schools Facilitate collaboration to develop quality OERs Integrate technology in secondary education for equitable access for boys and

girls Research open schools and develop resources to ensure sustainable open

schools 124. COL works on a broad front in this Initiative, interacting with most if not all

institutions offering, or planning to offer, open schooling institutions in the Commonwealth. In 2009 it networked 22 of these institutions for the first time by setting up the Commonwealth Association of Open Schools (COMOSA). The intention is that meetings of COMOSA will have a clear capacity-building and collaborative purpose and be not just a talking shop. In 2011 COL also launched a web-based community platform - OpenSchoolingConnect - to assist open schooling stakeholders to access information and interact.

125. In this triennium, amongst other things, COL has:

Supported several actual or would-be start-ups with the multiple issues around the establishment of an open school or the introduction of open schooling in an existing institution.

Supported institutional development for a number of existing open schools in areas like quality assurance and M&E; and targeted capacity building in ICT strategy, radio programming, curriculum development, instructional design, copyright, and the integration of multi-media in materials.

Created a number of freely available resources to support the planning and operation of open schooling. Before 2009, there were few models, tools or comparative research for open schools. COL and its partners are filling these gaps.

126. The Initiative is led by the former director of one of Africa’s open schools who

brings institutional and political experience to bear on the work. Open Education Resources for Open Schools 127. COL’s largest open schooling intervention this triennium has been a project in OERs

with US$750,000 funding from the William and Flora Hewlett Foundation (WFHF). The initiative was COL’s. They conceptualised and designed the project and presented it to the WFHF as a proposal for funding.

128. The project’s aim was: To broaden access to secondary education and increase student achievement

through the development and deployment of sets of high quality ODL materials in 20 selected subjects.

22

129. The materials were created by groups of teachers in six countries. They are to be made available as Open Education Resources (OERs). A “set” of material refers to the complete syllabus for one subject at a Grade 10 or Grade 12 level in each of the countries with the possibility of adapting it to the curriculum of any other country. The material is intended to be suitable for use in both open and conventional schools.

130. A key parallel objective is the professional development of a group of “Master

Teachers” in materials design, using online collaborative methods. 100 teachers were selected to participate in the programme. The intention is that these teachers will become competent to train groups of other teachers in their countries and support online materials development projects beyond this project.

131. A further project objective is the establishment of an international network of OER-

competent open schooling professionals in developing countries. 132. Six countries have participated: Botswana, Lesotho, Namibia, Seychelles, Trinidad &

Tobago and Zambia (a late joiner). The project began in 2008 and was originally scheduled to finish in 2010.

133. The main activities have been:

Face-to-face workshops to train the Master Teachers. Facilitation of collaborative, online materials development using Basecamp

software and the COL instructional design template. 134. Materials were originally intended for WIKI, but COL introduced Moodle as they felt

this facilitated the online process better and empowered the teachers. Teachers were also introduced to Poodle, an offline version of Moodle.

135. COL did not appoint a full-time project manager, but consultants retained by COL

continued their online support to the teachers after the workshops. This was complemented by hands-on attention from COL staff.

136. A rigorous mid-term evaluation was completed by SAIDE in April 2010, focusing

mainly on the materials that were being produced. It concluded that “many of the basics were in place”, but more work was needed to strengthen design and pedagogic components “before they will be suitable for self study purposes in an open schooling context”. It advised that more Moodle training was required. SAIDE offered a number of resources and tools they felt would be helpful in the strengthening process.

137. Although COL contested some of the detailed points made by the SAIDE evaluators,

it scheduled further project training and collaborative review of the materials, requiring an extension of the project time - by 12 months - and budget by CDN$40,000.

138. By October 2011:

16 sets of complete courses materials had been developed. Five of these had multi‐media components. Nine out of the 16 had been converted to online courses with the use of Moodle. 14 has been placed on COLWIKI, one had been produced in Braille and one in DAISY for users with disabilities.

A guide to learner support had been produced as an open resource. 86 (of the original 100) teachers were still engaged with the project despite

the extension of the time frame. International collaboration had not started.

139. A final evaluation was completed by SAIDE in October 2011 which:

23

Assessed teacher attitudes, perceptions and confidence levels through a sample post survey questionnaire.

Analysed the results of post project assessment tasks administered to a random sample of teachers.

Assessed twelve selected units from materials developed in five countries. Assessed seven units that have been converted from print form to Moodle,

and analysed the platform used by the teachers to develop the materials. 140. The main conclusions, based principally on a review of the materials and a sample

survey of 29% of the teachers, were presented in the following terms: Writing teams significantly improved the materials since the mid-term

evaluation. While the materials do not as yet constitute high quality, a very firm base has been established on which to make the final recommended revisions.

The project outcome of professionally developing 100 master teachers has not yet been realised. Additional professional development support was needed to build the teachers confidence and technical ability to integrate the use of technology in classrooms.

As these teachers report that they are not confident themselves, it is unlikely that they would be in a position to train and support other teachers to use and integrate various technological applications in their classroom.

There was evidence of great improvement in using Moodle. However some further improvements still need to be made to maximize the features of this virtual learning environment.

141. The evaluators and programme partners consulted agree on three challenges the

project had faced which undermined its ability fully to achieve its objectives. Despite partnership agreements with the Ministries of Education containing related conditions:

Many of the teachers lacked materials development experience. As a key informant put it: most were starting from zero – no experience of OER content creation nor collaborative working.

There were difficulties in releasing the teachers for training and collaborative work; and customary incentives were generally not provided.

Many did not have ready access to computers and reliable internet. 142. Many teachers – especially those from conventional schools - struggled with the

project work both conceptually and logistically. The ICT constraints limited the use of Basecamp for team collaboration and support from the consultants.

143. These factors led to slow progress, and the need for more capacity support and

alternative, less satisfactory, collaborative arrangements such as teleconferences. Analysis 144. The SAIDE evaluators maintain that there should have been a greater emphasis on

instructional design in the first round of workshops, given the lack of experience of the majority of the teachers. The workshops, they felt, were too narrowly focused on technical aspects of materials such as the delivery platforms and tools. COL agrees with this contention but point to the fact that this lack of experience was contrary to the agreements with the government. Baseline surveys which identified this lack of experience were conducted both by SAIDE and COL, but the findings were either not delivered in time to influence the first round of workshops, or were not properly taken on board by consultants.

145. The project would clearly have benefited if the government partners had been able

to deliver on their commitments. The fact that most countries struggled to find

24

teachers with the right experience, to release and incentivise them sufficiently for the training and collaborative work, and to ensure they had consistent access to ICT, suggests that the commitments were over-ambitious on both sides and that the project would have benefited from a more thorough risk analysis at the planning stage. The re-scheduling and re-financing of the project after the mid-term evaluation was not the easiest of processes for the partners.

146. Difficulties COL experienced with control e.g. over the timing of SAIDE’s baseline

survey and the work of the consultants point to the need for a full-time manager in this project.

147. COL has already acknowledged and internalised the difficulties and learning with

this project. COL was agile in adjusting aspects of the project delivery – such as modes of support – in response to difficulties faced. The SAIDE evaluators give it great credit for that.

148. The two programme partners consulted also acknowledge the difficulties the project

faced with the participating teachers; and that there is still work to do before the materials and master trainers are in the shape envisaged by the project objectives. However they are broadly satisfied with where they are now. They say that the project has broken new ground and given them a clear understanding of how to do this type of work. The only substantive criticism they have of COL’s role in the project is that it was too ambitious in its design - both in the level of its objectives, and in the time-frame for achieving them. There is clearly still a strong commitment among the institutional programme partners, and there have been remarkably few drop-outs among the teachers.

149. COL intends to provide further support in the coming months to ensure the

materials are improved. It is not unusual for materials like these to go through several iterations. OER development can be seen as a journey without an end.

Conclusions 150. The Open Schooling Initiative outcome is: Countries plan and implement sustainable open schooling as a means of

increasing access to learning opportunities at the secondary level. 151. This is clearly being achieved and COL’s contribution in the last three years appears

to have been substantial. The outcome statement gives no sense of scale and does not define sustainability. The TYP Performance Indicators however do offer some help.

25

PIs Progress (by November 2011)

1 2 countries establish new open schools, and 10 existing open schools significantly extended.

Open schools initiated in Tonga and Kiribati. COL believes that 3 more will open before July 2012: in Bahamas, Ghana, and Solomon Islands. Various aspects of institutional capacity developed in over 10 existing schools.

2 Open schools in 6 countries offer new high quality courses in 20 subjects.

Courses in 16 subjects are completed in 5 countries. COL is confident that it will meet the target of 20 by June 2012. The quality of the courses is subject to continuing improvement.

3 10 secondary schools adopt open education resources as a central part of their learning strategies.

No schools have yet fully adopted OERs, although all 6 countries in the Hewlett funded project have implementation strategies.

4 5 countries have developed and integrated digital content in 10 courses.

Digital content developed in 4 countries (Namibia, Swaziland, Botswana, Zambia) and in development in Tanzania. Integration is underway in at least 3 countries.

152. The first PI is ambitious. A new open school is still a rare event; although the

movement is gathering momentum. Two new schools – and 3-4 in the pipeline – is an impressive tally. Since July 2009, open schooling has taken root in the Pacific, reached out beyond its first base in the Caribbean (Trinidad and Tobago) and made further progress in Africa. There is ample evidence that COL has been both a catalyst and a support in this process.

153. In the second part of PI 1 it is not immediately clear what “extended” means.

However, COL has clearly supported institutional development in a number of existing schools. “Strengthened” therefore would seem to be a more accurate term for what has been achieved.

154. This appears to be a substantial amount of work in little over two years. COL is helping open schools to become a community. As yet there is little networking independent of COL and it is too early to tell whether this will be self-sustaining, or will be dependent on COL’s active facilitation and mediation. If COL decides to take up a recommendation of this evaluation and research ODL networks, the open school context should be included.

155. PI 2 refers to the OERs for Open Schools project that has been addressed in detail

above. 156. PI 3 has not yet been achieved and is unlikely to in this triennium. It is about the

institutionalisation of OERs in schools, which is turning out to be a slower process than COL expected in 2009. Some institutions are jealous of their material and are not convinced that they should commit themselves to an OER policy. Although functioning examples do not yet exist as models, COL has developed guidelines on OERs for open schooling. The February 2012 meeting of COMOSA will focus on the institutionalisation of OERs.

157. COL expects to achieve the PI 4 target by June 2012. 158. In common with other COL Initiatives, not all the open schooling interventions have

borne fruit within the expected time-frames, and some have stalled for a variety of reasons. Despite these reservations, and set in the context of this relatively new terrain, COL is undoubtedly a very important and valued partner in the open schooling sector in the developing Commonwealth.

26

Teacher Education 159. COL’s Teacher Education (TE) Initiative promotes and supports the use of ODL to

increase access to, and the quality of, teacher education. 160. There are four Activities:

Quality Assurance in teacher development Training of teachers and teacher educators Materials development in Teacher Education COL-UNICEF Activity

161. COL’s work in this Initiative is currently an eclectic mix. Some examples are given

below. 162. COL has helped the Freetown Teachers College Sierra Leone to offer its Higher

Teachers’ Certificate Course (Primary) in ODL mode alongside the face-to face option. COL consultants trained 35 course writers and worked with the writers to develop 76 modules for the course.

163. COL continues to promote the Quality Assurance Toolkit for Teacher Education

(QATTE) developed in the mid 2000s in partnership with the National Assessment and Accreditation Council of India. Like COL’s toolkit for higher education (COL RIM), the QATTE is applicable to all TE institutions, not only those offering TE through ODL.

164. Nigeria has been the main focus of QATTE support this triennium. A proposal to

work with UNESCO in other African countries has not taken off. COL wants to build a model institutional framework for quality assurance to complement the toolkit, but has not yet found a partner institution to help develop and trial it.

165. COL is more optimistic about its work with CARICOM where a set of draft quality

standards for TE has been agreed and there is potential for a multiplier effect across the region if resources can be found for the implementation of the standards.

166. COL continues to support the Green Teacher programme - a one-year distance

mode Diploma in Environmental Education for teachers and educators developed by India’s Centre for Environment Education (CEE). It has taken CEE longer than expected to put the course on-line where it has potential to reach many more teachers, including those outside India.

167. COL is a partner with the Teacher Education in Sub-Saharan Africa (TESSA)

programme which develops multilingual open education resources (OERs) for teacher training. In this triennium COL is helping promote the dissemination and effective use of TESSA resources by primary school teachers and teacher educators in Uganda and Zambia. This intervention is working well in Uganda where the local partner institution has attracted the interest of three others. TESSA however often struggles to meet its full potential because of problems of internet access and inconsistent institutional support.

Child Friendly Schools 168. The Child-Friendly Schools (CFS) approach is based on the concept that quality

education involves placing the total needs of the child as the central focus of education decisions. It has been taken up and developed widely by UNICEF.

27

169. UNICEF describes the concept in these terms: Quality goes beyond good teaching methods and learning outcomes to also

include considerations of health and nutrition status of learners; adequacy of available facilities, services and supplies; as well as levels of safety and protection afforded by the learning environment. These are important not simply as means of supporting good teaching and promoting learning achievement, but also as goals in their own right and valid indicators of quality education from a child rights perspective.

170. The model may differ from country to country, but the common denominator across

cultures is a focus on child-centred education in a safe, healthy and holistic environment.

171. UNICEF and COL signed a Partnership Cooperation Agreement in 2008 with the aim

of mainstreaming the model into the pre-service and in-service TE programmes for primary schooling in selected countries. This has led to a project, funded1 largely by UNICEF and managed by COL. It is designed to promote CFS approaches at national level; and develop capacity for CFS among teachers and other education actors such as school heads, administrators, supervisors, curriculum planners, inspectors, teacher resource centre managers, and local communities in the form of school committees and parent-teacher associations.

172. The ten countries involved are: Botswana, Lesotho, Malawi, Nigeria, Rwanda, South

Africa, Sri Lanka, Swaziland, Trinidad & Tobago and Zambia. 173. The principal activities conducted are described as:

Sensitisation, advocacy and vision building. Review of pre-service and in-service teacher education curricula. Development of locally-contextualised trainer-training manuals (adapting

UNICEF’s CFS Manual). Training of teacher educators, resource centre managers, head teachers and

other key personnel. 174. The project incorporates the principle of country ownership. Each country has a

focal person either in the Ministry of Education or another key institution. Countries appointed their own consultants who worked with local teams to plan the interventions and to develop materials. Countries also helped to fund and organise workshops. COL’s role was chiefly coordination and facilitation with very little direct funding.

175. The aim was that by the end of the project, all countries would have as a minimum:

A work plan for mainstreaming CFS. Contextualised manuals for both pre-service and in-service training. Groups of trainers with the capacity to train others to integrate CFS principles

and practices in both pre-service and in-service training. 176. It was hoped that by the end of the project, Governments would also have

embraced and taken ownership of the CFS approach and begun integrating it into education policy; and that the training was already reaching school level and having an effect on classroom practice.

177. At the start of the project CFS was not entirely new to any of the countries.

Countries were chosen because of their prior experience. UNICEF country offices had been promoting the approach for some time. Swaziland in particular had a well-

1 UNICEF funding is CDN$1.9 million

28

developed strategy for CFS, although in 2008 it was said by key informants to be lacking momentum.

178. UNICEF offices were also able to identify the most appropriate institutional partners

for the project; and they conducted scans to map the penetration of CFS, and thus identify areas for further development for the project to focus on.

179. Consultative meetings organised by COL were held in each country, leading to the

development of country work plans, tailored to their circumstances and priorities. One of the main differentiators for these plans has been the type of lead institution. Government lead institutions – such as in Botswana - placed the emphasis on policy in the initial stage, while teaching institutions such as in South Africa focused on curriculum.

180. Progress with the implementation of these plans has varied. An external evaluation

of the project was conducted in July 2011 in four countries. These countries were chosen because they were among the most advanced. The evaluation found that Swaziland and Botswana had already completed the core outputs of the project and were already cascading the training to school level, using additional resources provided by their Ministries of Education. They were also fostering public awareness about CFS through advocacy meetings and workshops.

181. In Swaziland, there was a deliberate attempt to link the CFS project to the existing

national policy. Plans were also underway to integrate CFS into the general curriculum of the country’s Colleges of Education and the Ministry of Education; as was the case in Botswana.

182. An intended result of the project is the mainstreaming of the CFS principles and

methods in both the pre-service and in-service teacher education curricula – potentially a powerful multiplier. For most countries this is seen as an outcome: work undertaken by wider groups of teacher educators trained by those who had engaged directly with the project. In Botswana and Swaziland this phase was well underway at the time of the evaluation. In Lesotho it was just beginning.

183. In South Africa, the project has unfolded differently. It has been led from the

University of Limpopo and focused first on the curriculum. The team aimed to create a CFS framework to support the implementation of the curriculum for an Advanced Certificate in Education and Life Orientation (ACELO). This has been a slow process and was beset by delays which were attributed to several organisational factors.

184. The curriculum review and materials development for the ACELO course is now

complete; but the University, at the time of the evaluation, had yet to develop the training manuals and conduct training of trainer workshops for teacher educators from five universities that have expressed interest in the ACELO course. The University has plans to sustain the CFS intervention through its new 3 year project for the development of a B.Ed Foundation course being funded by the European Union.

185. At first sight the CFS experience in South Africa seems like a painstaking process,

unlike the strides taken in Swaziland and Botswana. But of course the difference in scale and complexity of the countries is marked, and with that qualification, the external evaluator felt that progress in South Africa was satisfactory.

186. The evaluation identified more fundamental institutional factors behind the slower

progress in Lesotho, but these seem to have now resolved themselves.

29

187. Of the project countries not included in the evaluation: Nigeria was advanced with its work plan. Three other countries had experienced considerable delays and weak

momentum. Two countries dropped out before implementation began.

188. In total around 900 personnel have been trained so far, the majority of whom were

female. 189. A discrete component of the CFS project is a focus on nomadic education in Nigeria

which is often challenged by poor teacher quality. The project is creating an interactive radio learning programme for the continuing professional development of nomadic primary school teachers in CFS approaches. COL facilitated planning and script-writing workshops earlier this year. The partners are now developing radio broadcasts and supporting print materials. Broadcasts are expected to begin in early 2012, targeting around 450 nomadic teachers in Adamawa State. The plan is to replicate the programme in other Nigerian States and other countries with a need for nomadic teacher training.

Analysis 190. It seems that the main differentiating factor in progress in the project has been

leadership at country level. Where the task group had strategically-positioned, committed, leadership with sufficient time to devote to the work, the project progressed well. Commitment at high level in government education agencies is critical whether or not these agencies have led the project. Turnover among leaders, distraction and lack of political leverage have been obstacles to progress.

191. The demonstrated successes in Swaziland, Botswana, and reasonable progress in 2-

3 other countries, suggest that the CFS concept is relevant and coherent and that the project was well designed. There appear to be high levels of satisfaction with the project in the four evaluated countries.

192. About COL’s contribution, the external evaluation concluded: COL’s work in bringing considerable technical expertise and financial support to

the project teams in the four countries had contributed in very significant ways to build the capacity of the project officers to work effectively with the consultants...It is also evidenced by the growing number of partners and partnerships on CFS in the four countries.

193. The most important partnerships to be forged are those between Ministries of

Education and teacher education institutions, not always easy bed-fellows. 194. The evaluation stresses the importance of buy-in and training at school level. “In

the long run, the success of the CFS project would require a focus on schools and classroom teachers... extending the CFS training to classroom teachers and head teachers was more likely to lead to positive outcomes than limiting it to teacher educators and other education officers...The committed involvement of head teachers ... is required for any systemic change.” This was not a requirement of the project and it did not figure in most of the countries’ plans. This may have been a weakness in the project concept.

195. The project has eight months to run and there is still work to do, especially for the

countries that have made less progress. The project outputs and early outcomes themselves will, in most countries, need to be followed by substantial further capacity building and curriculum work. UNICEF local offices will continue to offer light-touch support, but further progress will primarily be a test of the strength of

30

country ownership. Networking may help. It has not happened yet and there is probably a role for COL in this.

196. Several countries want to extend CFS mainstreaming into secondary education; and

COL and UNICEF have discussed the possibility of extending the project’s approach to other countries. It would be sensible to conduct an outcome-level evaluation of the current project before any new investment. That should ideally take place in 2-3 years time to allow for outcomes to mature and sustainability to be tested.

197. Dialogue between COL and UNICEF country offices was an important enabler in the

inception phase of the project. However there was a considerable delay – around nine months – before that dialogue began. This was the first time COL and UNICEF have been partners in a project on this scale and there was clearly a mismatch of assumptions about the way the project would be initiated. This however has not derailed the project and the loss of time has been recovered through a no-cost extension of the project’s life-cycle to 30th June 2012.

198. This is the only current example of COL’s work where they are employing a full-time

project manager. COL does not generally manage large implementation projects on in partnership with international or bi-lateral development agencies. The purpose and design of this project however are not dissimilar to those COL’s normally works with. The main differences are scale, and having to work to a greater extent to another agency’s agenda. Once there was mutual understanding about the methodologies, the project moved forward successfully.

199. This is a complex project, mainly because of the diverse paths that have been

followed. The diversity was a core project concept: encouraging ownership by obliging country teams to create their own road maps for the mainstreaming of CFS. It is too early to draw firm conclusions about its success, but on the whole, the results are promising.

Conclusions 200. The Teacher Education TYP outcome is: An increase in teacher education and training institutions using ODL

methodologies to train and upgrade teachers. 201. As with most of COL’s TYP outcomes, this is not on the face of it a very challenging

outcome and it has clearly been achieved. A sense of scale and quality is provided by the TYP Performance Indicators.

PIs Progress (by November 2011)

1 4 major institutions in at least 3 Commonwealth regions begin to train/upgrade teachers, or significantly increase the number trained or upgraded, through ODL.

Freetown Teachers College has become dual mode; NTI Nigeria has extended its reach through a major new in-service ODL course in maths and science; and COL expects CEE Ahmedabad to have extended the reach of its Green Teachers course by June 2012 with help from COL.

2 Teacher education institutions in 4 countries offer significantly improved curriculum content.

New materials in use in the 3 institutions in the PI 1 progress report, plus one in Zambia, one in South Africa and 5 primary TCs in Uganda.

3 Institutions in 6 countries adopt the Child Friendly Schools approach.

This has been achieved in 5 countries, although in at least one, CFS was already in policy and practice.

31

202. PI 2 has been exceeded although it is not a very challenging target. PI 1 is more challenging and will not be fully met, but the progress at Freetown TC has been valuable.

203. The CFS PI target may be fully met by the end of the triennium and represents a

solid achievement for the project. 204. The demand for effective school teaching in the developing Commonwealth far

outstrips supply. COL is addressing this deficit through its Open Schooling and TE Initiatives. Open Schooling works mainly with single mode ODL institutions; in teacher education, these institutions are few and far between and COL’s work is somewhat more oblique.

205. COL is open about the difficulties it has faced with its partners in this Initiative.

There are few clear-cut outcomes; but those at Freetown Teachers College and the CFS front-runners are solid. COL should not let setbacks deter it from pursuing partnerships, with UNESCO for example. If not, COL may become less relevant as a player in what will remain a highly relevant area.

206. There is a consensus that the focus of teacher education in most parts of the

developing Commonwealth must shift to providing recurrent in-service programmes of professional learning. COL is working to help teacher education institutions offer quality ODL programmes that will enable teachers to upgrade their skills and qualifications. Increasing the number and quality of teachers is an essential element in the quest to achieve Universal Primary Education by 2015 and coping with the knock-on effects for post-primary teaching and learning. Ambitious new thinking is required by COL and its partners.

Higher Education 207. COL’s Higher Education (HE) Initiative is broad-based, designed to support higher

education institutions (HEI) to have policies, systems and staff competencies that support the use of ODL for increased learner access, completion rates and enhanced content.

208. There are three Activities:

Capacity building Course and resource material The Regional Training and Research Institute for Open and Distance Learning

(RETRIDAL) 209. Surprisingly, given the wording of the outcome, there is no explicit mention of

policies or systems in the Activities. 210. COL’s HE work over the last two triennia has been the least monitored Initiative. It

appears that the two areas with the highest level of activity have been: the Commonwealth Executive Master of Business/Public Administration (CEMBA/PA), the HE Review and Improvement Model (COL RIM). The Initiative also includes oversight of RETRIDAL.

211. CEMBA/PA is a distance programme developed through a collaboration between

COL and four Asian open universities. The aim is to provide a quality masters programme – with private or public sector options - relevant to the needs of developing Commonwealth countries and accessible to more people. It is designed for part-time study and has the flexibility of several accredited exit points. It was launched in 2002. Delivery of CEMBA/CEMPA Programme is expanding through

32

partnership with universities – nine at present with more in the pipeline - in four regions.

212. COL began managing CEMBA/PA’s first major revision in the previous triennium.

Much of the material was seriously out of date. The revision process has been long drawn out and has been criticised by stakeholders. The process has been expedited in the last few months and is expected to be complete by the end of 2011.

213. COL acknowledges that it needs to make three fundamental changes to the way the

programme is managed: Put in place a process for more regular – even continuous - updating. Make more use of OERs Devolve more responsibility and ownership to the participating institution.

214. COL RIM is a low-cost, self-administered, quality audit tool designed for use by any

post-secondary education or training institution. A team of consultants act as external verifiers, but the results are owned by the institution. Unlike external quality audits, COL RIM is primarily designed to lead to institutional strengthening and not to be an accountability mechanism.

215. COL RIM was developed in the previous TYP period. In this period trial and full

audits have been conducted at several HEIs, including two funded by Commonwealth Secretariat (ComSec): at the National Institute of Health and Social Sciences in Seychelles and the University of Guyana. Improvements to COL RIM have been made in the light of feedback from the trial audits.

216. COL intends to shift its support from application of the quality instruments to the

development of institutional quality systems and policy. RETRIDAL 217. COL established RETRIDAL in 2003 in response to a proposal by the National Open

University of Nigeria (NOUN). Its purpose is to support ODL capacity building and research in ODL in the West Africa region. Accommodation, staffing and running costs are provided by NOUN. In 2003, NOUN was the only single-mode Open University in Commonwealth West Africa.

218. Until recent months, the Vice Chancellor (VC) of NOUN had sole authority to release funds for activity.

219. COL provides funding of up to CDN$100,000 per annum for programme work. The

COL funds are released when work is undertaken. COL reviews its commitment to RETRIDAL every three years.

220. The evaluation of COL’s 2003-06 TYP found that RETRIDAL had, in its first three

years, conducted little activity outside NOUN itself. NOUN was experiencing funding difficulties and RETRIDAL’s base was looking fragile. In the 2006-09 period however it was more productive with some capacity building activity outside Nigeria. NOUN itself was also on a firmer footing in this period. Little or no attention however was paid by RETRIDAL to the gender dimension and M&E was non-existent except at the activity level.

221. For much of the current triennium, RETRIDAL has not fulfilled its mandate.

Difficulties with its host institution led to a brake on almost all activity. 222. The specific difficulties seem now to have been removed and activity resumed in

early 2011. This included:

33

Leadership training workshop for 42 women academics and staff in West Africa.

Two eLearning workshops conducted in Ghana and in Sierra Leone for a total of around 60 academics.

The first edition of an ODL journal for West Africa is being prepared for publication.

223. Importantly, RETRIDAL’s regional Advisory Board has been reconvened and a work

plan for the year to June 2012 approved. It has secured an interim arrangement for the RETRIDAL Director to approve funding for activity in the absence of the VC of NOUN.

224. An external evaluation of RETRIDAL conducted in 2010 proposed a severing of links

with NOUN. Although it proposed a funding model based on subscriptions from institutions in the region, it would have involved COL in underwriting the organisation. COL does not see this as its remit.

225. The overall RETRIDAL concept - to create a capacity-building and knowledge centre

for ODL in HE in a region where these services are in great demand – is still valid. But to fulfil this remit, the centre must not only have good human and other resources, but must be seen to be effective as an institution. RETRIDAL’s reputation has been set back by recent inactivity. COL’s recent – overdue - re-engagement with the institution has begun the rehabilitation process.

226. COL will review its commitment to RETRIDAL in 2013. This should be based on a

further – and more thorough - evaluation of its work in the coming months. If COL decides to continue to support RETRIDAL, it should help the organisation to plan its work more strategically, to monitor outputs more effectively and extend monitoring to outcomes. Outcome monitoring is research and if undertaken effectively will provide valuable feedback to RETRIDAL and its partners’ knowledge base.

227. Issues for COL to consider include:

The balance between work in Nigeria and other countries in the region. Whether capacity workshops should be based on trans-regional or in-country

participation. The former are much more difficult to organise and should probably only be held when there is a clear rationale for trans-regionality.

Whether short workshops are in fact the best mode of capacity building (outcome monitoring will help to answer this question).

The relative value of its research activities and therefore how much emphasis to place on them. Journals quickly acquire a life of their own which can divert effort and funding without sufficient returns.

The possibilities for extending RETRIDAL’s scope and reach through funding partnerships with other agencies.

Conclusions 228. Higher Education’s TYP outcome is: Higher education institutions create policies, systems and staff competencies that

support the use of ODL to increase learner access and completion rates while enhancing curricular content.

34

229. The Performance Indicators in COL’s TYP are:

PIs Progress (by November 2011)

1 4 major HEIs in at least 2 Commonwealth regions begin education, or significantly increase the number educated, through ODL.

2 HEIs (MICO University in the Caribbean and University of the South Pacific) adopted ODL. Work will commence with 4 other HEIs before June 2012, but is not guaranteed to lead to the indicated change.

2 New or improved curriculum content is in use in HEIs in 4 countries.

Course revisions for CEMBA/CEMPA (offered in 9 countries) scheduled to be completed by the end of 2011.

3 QA mechanisms are adopted by 4 tertiary institutions.

3 HEIs have implemented COL-RIM. Another 3 are expected to complete by early 2012. 2 more are entering the pipeline.

230. HE has not been an easy Initiative to evaluate. The area chosen for a longitudinal

study – RETRIDAL – is not a mainstream programme, and in any case has been inactive for most of the last 2-3 years. Other work has been scantily monitored.

231. What has to be said is that COL’s performance in this Initiative has drawn more

criticism from key informants than any other. COL does not seem to have forged strong new partnerships in HE; and informants have said that it has made insufficient effort to cooperate with other international players in the subsector.

232. Policy work with government – hitherto one of COL’s strengths, despite the

difficulties – has made little progress in the last several years. Although COL reports (against PI 1) that two HEIs have adopted ODL in this period, it is not clear what COL’s contribution to this has been.

233. The CEMBA/PA revision has diverted much of the Initiative’s time and other

resources. This was disproportionate and has had a substantial opportunity cost. It is still a valuable programme, but COL should not have a continuing commitment to educational products like CEMBA/PA and is right to be planning an orderly exit from its commitment to manage and revise it.

234. RETRIDAL has achieved very little in the last 2-3 years. It is not easy to see what

COL could have done to remedy that, but there is little evidence of concerted attempts to try.

235. COL RIM is the area of work that has attracted most positive comment. It appears

to be a well crafted instrument that is addressing a clear and substantial demand. The development of systems for quality assurance may have been given a boost in the institutions that have applied COL RIM; but there has not been any outcome monitoring of COL RIM to provide evidence of this or other results.

236. A recent change of leadership in the HE Initiative is making a difference. Work has

been stepped up with the inherited commitments. There are plans to do more work in HEI leadership capacity building, the development and use of OER and in ODL policy support.

237. The HE sector is much larger and more diverse – and the distinction between

distance and face-to-face education less clear-cut - than in the 1990s when COL

35

was a major player in promoting ODL for the HE subsector. Nowadays it is increasingly difficult for COL to find a niche.

238. An important question for COL in the coming triennium is: should it strive to look

for niches through a continuing commitment to a free-standing HE Initiative, or should it contribute to the subsector in more flexible ways – through its championing of OERs for example - and only where major opportunities present themselves?

239. Decisions on this should be made partly on performance over the next 2-3 years.

This means much better monitoring and evaluation as well as wide stakeholder consultation. Stakeholders should be asked to prioritise support from COL in HE in relations to more support in other areas such as skills development. Stakeholders need to understand there is always an opportunity cost to COL’s work in any given area.

240. RETRIDAL is a somewhat separate issue. COL’s continuing commitment should also

be kept under review. It should consider bringing the Southern African regional centre - SADC-CDE - under the same point of oversight, as there would be advantage in comparative monitoring and possibly cooperation in strategy and other aspects of leadership.

The Virtual University for Small States of the Commonwealth

(VUSSC) 241. VUSSC is a group of institutions from small states, brought together through COL

since 2005, to collaboratively develop free content resources for use in an educational context. The aim is to stimulate an increase in learning opportunities by building capacity of existing institutions in these smaller countries to produce and offer relevant courses, sharing material through the medium of OERs and to create a mechanism to enable the transfer of credits and qualified people across borders through the creation of accreditation mechanisms.

242. There are three Activities:

Capacity development for ODL. Networks of ministries and national institutions. Strengthening of quality systems.

243. The VUSSC course materials are non-proprietary and intended to be readily

adaptable to the specific context of each country. They can be used in the offering of credit-bearing qualifications.

244. VUSSC is not a tertiary institution. Rather, it is a collective mechanism for

developing, adapting, and sharing courses and learning materials. It is also a forum for institutions to build capacity and expertise in online collaboration, eLearning and ICTs generally.

245. There are currently 33 countries participating in VUSSC, through interlocutor

(country representative) planning meetings, training and course development workshops, participation in each others’ courses, and gatherings of senior officials for the development, finalisation and implementation of a Transnational Qualifications Framework (TQF).

246. For five years, VUSSC participants chose to focus on the development of post-

secondary, skills-related courses in areas such as tourism, use of ICT, disaster management, transportation and logistics. However, in 2011 VUSSC started the

36

development of its first degree level programme at the request of Commonwealth Education Ministers.

247. The main vehicle for collaborative course development has been the training and

course development workshop, also called “bootcamp”. At each bootcamp around 20 people from different countries meet for 2 to 3 weeks to work on material development for a specific subject. They also have a capacity development component. The immersion format of the bootcamps is popular and seems to have been consistently successful in producing course material of the expected standard. There have been nine bootcamps, in which 190 individuals (99 women, 91 men) from 30 countries have participated. 93 individuals from 19 different countries have participated in additional VUSSC training events.

248. VUSSC participants are encouraged to conduct training and capacity building

workshops in their home countries after returning from the bootcamp. Anecdotal evidence shows that this happens to some extent, although comprehensive data is not available.

249. The TQF is an important component of the VUSSC. It is a 10-level reference

mechanism intended to link up national qualifications systems and frameworks in different small states. It will make different countries’ qualifications, at different levels, translatable. This should encourage employment and educational mobility. A VUSSC TQF portal is being developed, consisting of a searchable database of qualifications and training providers involved in the VUSSC initiative.

250. The TQF still in piloting phase, but it has already been endorsed by three states,

including Malta which should provide a link to the European Higher Education Area. Analysis 251. VUSSC is COL’s largest ever coherent programme in funding terms. Since 2006 it

has attracted CDN$2,303,300 of external funds to complement COL’s CDN$758,300 of programme funding. COL has, since January, 2009, assigned an ES full-time to VUSSC. It is also one of the longest-running interventions.

252. VUSSC’s coverage, and the apparent commitment of member countries, are

impressive. 33 countries are members and 30 have sent participants to the demanding training and course development workshops (appropriately labelled bootcamps).

253. The VUSSC experience - the workshops, other training events and the planning and

management meetings - have received consistently good feedback from participants.

254. The TQF – a potential game-changing mechanism for isolated small states – is on

the verge of becoming fully operational. 255. These results at the output level are encouraging. There was no guarantee that

VUSSC would survive its inception period. It is seen as relevant and is meeting expectations sufficiently to attract both external funding and high levels of participation in its planning and management mechanisms and workshops.

256. Turning to the expected outcome and the performance indicators, however, the

picture is not unequivocally positive.

37

257. The Initiative outcome for the triennium is: VUSSC partner institutions producing and delivering relevant courses using

eLearning strategies that are scalable and sustainable. 258. The TYP Performance Indicators for VUSSC are:

PIs Progress (by November 2011) 1 8 new courses are developed

and 2 institutions in 3 Commonwealth regions offer a total of six VUSSC courses

Content for 15 potential course units have been collaboratively developed; plus a full bachelor degree programme which consists of 20 courses. 2 institutions – University of Samoa (agriculture) and UWI, Open Campus, Barbados (on-line instruction) - have offered one VUSSC course each. 5 other institutions are expected to offer the agriculture course in 2012.

2 Agreements are in place committing 10 member states to take responsibility for the on-going management of VUSSC

A VUSSC Management Committee with representatives of six member states has been set up to secure the on-going management of VUSSC.

3 NQAs in 6 VUSSC member states will ensure the implementation of the TQF

Three states (Malta, Mauritius and Seychelles) so far have referenced their National Qualifications Frameworks to the TQF. Twenty three member states agreed to implement the TQF at the Malaysia meeting in July 2010.

Courses

259. It is clear that the course outcomes have not yet been fully achieved. COL does not

have comprehensive data in this area, but it is thought that only two institutions are delivering new courses based predominantly on material produced at VUSSC bootcamps. One is a course module in on-line instruction. The other is a full two year diploma course at the University of Samoa which the University has customised and blended into a programme for secondary teachers. Bootcamp material is finding its way into existing courses, but usually without fundamentally changing the nature of those courses.

260. There is a consensus among COL stakeholders that progress in the production of

courses has been slower than anticipated. The main stumbling block has been the translation of bootcamp material into actual course modules. This does not happen automatically. It takes a considerable amount of expertise and experience to traverse this “last mile”. The bootcamp is a capacity-building vehicle, but it does not deliver that “last mile” support. Most VUSSC countries do not have much expertise and experience in building courses around OER material, let alone whole curricula.

261. One informant said that in her country, most of the material produced had not

found its way into courses. Time is a factor – people in small states tend to have multiple roles and often find it difficult to focus on the work needed. Participants need incentives to follow through after the bootcamps. In her country, they are now required to sign an MOU containing post-workshop obligations. But institutions and the key individuals also need more capability support to integrate the materials.

262. It was partly this realisation that was behind the move to produce whole courses at

the last two bootcamps – combining curriculum development and instructional design capacity-building.

38

263. The curriculum framework is prepared in a small group which precedes the

bootcamp. This could turn out to be a watershed development. The Agriculture diploma material developed at the Maldives bootcamp was translated relatively quickly into the University of Samoa course; and five other institutions are said to be on the verge of delivering their versions.

264. If this had not happened, serious questions would probably have to be asked about

the current VUSSC model. Bootcamps are very expensive – both in direct expenditure and in use of COL’s human resources. Clearly they are intended to be capacity-building events as much as a materials production mechanism. They are building competency in collaborative tools, in the concept of OERs and increasingly in instructional design. But without the dividend of new courses, the question “capacity for what?” would have to be asked. A successful outcome of VUSSC has to be seen in terms of institutions delivering quality teaching, and students learning effectively.

265. Informants voiced some concerns about the adapted model for the bootcamps. Will

it lead to a loss of ownership? Will the courses be too complete to be easily adaptable to local contexts and needs? Will they be so generic that they resemble what is already available in developed country open source materials?

VUSSC strategies and mechanisms 266. VUSSC is still heavily dependent on COL. COL provides or attracts most of the

funding. It convenes the interlocutor and TQF meetings, and organises the inter-state workshops. There seems to be very little inter-state networking. COL has set up a VUSSC Management Committee with representatives of six member states. This is intended to lead to the agreements for a transfer of ownership referred to in the second PI. But VUSSC is not there yet.

267. Governments need to take ownership. The answer may be to encourage more

regional interaction – at first between Interlocutors and then a wider forum of partners. This should make VUSSC more tangibly relevant. There is inevitably turnover among Interlocutors. It should not be COL’s responsibility to induct the new members. By 2015, COL should be a front - or even back – seat passenger and not the driver.

268. The TQF has taken time to produce, but that may be a good thing. It is a new

concept for most VUSSC countries and it needs time for appreciation of its role to develop. Countries will need help in applying it.

Conclusions 269. VUSSC has been monitored and evaluated quite regularly, but almost exclusively at

the output level. Systematic outcome monitoring is essential if countries’ differing experience of utilisation of VUSSC outputs is to be tracked and fully understood. This will enable VUSSC strategies to be more regularly adjusted to meet countries’ needs. Monitoring processes should be owned by the countries. They should not be complex or arduous, although some capacity-building may be required. Outcome monitoring should be regularly analysed, and not seen simply as a contribution to a triennial evaluation.

270. It is easy to see VUSSC as a vessel serenely sailing around the small-state world. In

fact, with its current methodology, it requires a considerable amount of organisational resource. Organising VUSSC interactions consumes a large proportion of the ES’s time – time that could be used with more strategic matters.

39

VUSSC clearly has stamina. Many participants have a sense of being part of a grand enterprise. This is a considerable achievement after 6-7 years, and few organisations have the staying power to have managed this process in the way that COL has done. COL needs to stay with VUSSC but accelerate the transfer of ownership while reviewing – on the basis of better evidence – the mechanisms, almost certainly introducing more diversity to the model. It also urgently needs to explore sustainable funding mechanisms.

Skills Development 271. COL’s Skills Development Initiative supports ODL for skills development particularly

through new approaches that emphasise technology-enhanced, flexible and blended (FaB) teaching and learning. These approaches, COL believes, are the best way to improve not just access but also quality, gender equity, and efficiency in skills development. There is a particular emphasis in this Initiative on skills for the informal sector which is the most neglected in many countries and yet is increasingly likely to be the destination of school leavers without higher secondary qualifications.

272. COL promotes and facilitates policy development, capacity building and the creation

and sharing of quality learning materials for skills development. This is reflected in the title of the four Activities in the Initiative, currently: Strategic planning and policy development. Capacity building. Models of flexible skills development. Skills development materials.

273. In practice, most of COL’s work in this Initiative combines two or more of these

elements, and therefore cuts across the Activity boundaries. In the current TYP, COL’s work has been mostly in the Pacific and Africa.

274. In the Pacific COL has supported the development of a vocational literacy course by

educators from seven countries to complement courses in basic trades that COL helped to develop in a previous triennium. Following pilot programmes in four of these countries, the learning modules have been collaboratively revised; and several of the countries have produced action plans for rolling out their vocational literacy course.

275. The greater part of COL’s resources in the Initiative have been devoted to a

programme of flexible skills development (FSD) for members of the Commonwealth Association of Polytechnics in Africa (CAPA). This exemplifies COL’s approach in this Initiative and seems set to be the foundation or model for most of the Initiative’s work in the next triennium.

Flexible Skills Development (FSD) for members of the Commonwealth Association of Polytechnics in Africa (CAPA) 276. FSD for CAPA is a coherent, project-like, intervention designed to strengthen

capacity in a number of African technical education institutions for the adoption of flexible and blended (FaB) delivery approaches in technical and vocational skills development. Unlike conventional technical and vocational education and training (TVET), FaB approaches target informal sector learners – young people and adults without advanced qualifications who generally cannot access oversubscribed TVET courses. FSD also aims to impact on the quality of teaching and learning for campus based programmes.

40

277. OER development and use are included in the project, although planned to be activated at a late stage.

278. In COL’s RBM framework this work is spread over three activity logframes, but a

separate integrated plan was drawn up as a guiding framework. 279. According to this framework, the intended outcomes of the FSD project are:

Increased institutional efficiency through use of ICT to both improve administration and release teaching capacity.

Increased access to quality technical and vocational training programmes as released teaching capacity is used to expand programme delivery through flexible and blended approaches.

280. This is to be achieved in three overlapping phases. 281. In Phase One, COL assists institutions to introduce FaB approaches into programme

delivery. This involves building institutional capacity in key areas: Strategic planning and policy development. Flexible resource/capacity planning. ICT systems. Teaching approaches and ICT for teaching. Repurposing OERs for skills training.

282. In Phase Two, institutions engage with people working in the informal sector to

identify training needs and plan to meet those needs. “Technical action groups”, comprising teaching staff, managers, industry representatives and artisans in the informal economy are established at each institution to inform curriculum development for flexible programme delivery.

283. In Phase Two, it is also expected that participating institutions will be providing

collaborative support to each other both in‐country and through the online community.

284. Phase Three – which has not yet been reached will consist of the later stages of

materials development, more advanced inter-institutional collaboration in the delivery of FaB training, and impact assessment.

285. The FSD activity began in June 2010. 11 institutions are involved: five in Kenya,

two in Zambia and one each in Gambia, Ghana, Nigeria, and Tanzania. 286. The main activities have been as follows:

Introductory workshops in three regions, included attendance by senior staff. On-line training – using COL’s Moodle platform - in the management of FSD

and in flexible teaching and learning methodologies. The latter has been delivered through a demanding 10 week course (FLEXTL).

Visits by an Africa-based consultant to the 11 institutions to assess and advise on the development of ICT infrastructure including learning management systems.

A workshop at PCF6 for a number of the Principals. Support visits by COL’s ES or lead consultant to five institutions. The Flexible Skills Development Community Learning Network (CLN): an on-

line forum which currently has around 260 members. Instructional design workshops at selected institutions. Promoting awareness in the wider CAPA community. Advocacy and networking with wider groups. A workshop was held in Nairobi

to introduce Kenyan industry associations, curriculum developers and officials

41

from the Department of Technical Education to what is happening in the project in their country.

287. It is difficult to generalise about where the project has reached. Some capacity

building is still being delivered, but most institutions are already developing FaB curriculum content. Community outreach has begun but more also needs to be done. More work is needed to embed FSD in institutional strategy.

288. In one institution, activity stalled when the leadership changed, but attempts are

being made to re-activate its participation. 289. An evaluation completed in August 2011 built on a 2010 baseline survey of

readiness on four dimensions: Strategic, Organisational, Technical and Pedagogical. The evaluation reported variable progress from the institutional baselines, both among institutions and between the dimensions. In no dimension was progress seen to be exemplary nor very weak. In general the most challenging dimension has been the technical one “in terms of equipping and developing the teaching staff, student outreach & access and the computerisation of administrative systems”. The FLEXTL course - very demanding in content and time (it is 10 weeks long) had a high attrition rate. However those who completed were reported to be very satisfied with the delivery and the progress they felt they had made.

290. Apart from the one institution where participation had stalled, none was reported to

be struggling. Leadership commitment, unsurprisingly, was reported to be the most important success factor.

291. The evaluation reported that the CLN is a major plank of the project, that it is well

used and valued for interaction both within and between institutions. The specialist groups are particularly popular. This was confirmed by a college Principal informant, although a consultant said that, from what he could see, the CLN had not yet become a forum where genuine collaboration is taking place.

Analysis 292. FSD for CAPA has a clear concept and purpose. Delivery is more complex. It is one

of COL’s most multi-faceted projects. This is a challenge for COL which does not have a presence on the project’s ground and no full-time project manager.

293. COL has dealt with this challenge well. It has delivered its services more or less

according to plan, but with the flexibility that working outside of a formal project framework permits. The institutions have mostly played their parts with commitment and in some cases ingenuity.

294. It is early days. Courses are being developed but are not yet being delivered.

Capacity improvements are probably fragile and in some areas – particularly OERs – further iteration of capacity building and handholding will be needed.

295. FSD for CAPA is an ambitious project. The venture into FaB approaches for the

informal sector that it promotes implies a transformative change that most institutions would find difficult. No matter how good the support from COL and its resource persons, it takes commitment and vision from the top of the institutions, time, and a favourable external political environment.

296. The FSD is primarily a bottom-up project. It focuses on the education institutions

and their communities, not on government. Polytechnics are usually not the strongest of education institutions; they lack political clout. Some institutions are hampered by national curricular constraints. There is also resistance from some

42

policymakers to the notion of ODL which is still associated in many minds with second-best education. There is resistance from a variety of stakeholders to teaching outside the framework of formal qualifications.

297. To navigate this policy terrain is asking a lot of college Principals and other staff.

Principals valued COL’s partnership in this effort because it enhances their status with government. A number of government representatives have attended workshops and participated on-line courses. But there is still more considerably more upstream work to do.

298. Kenya is the best example of progress in this area. Two of the FSD institutions have

been invited onto a government TVET forum as a direct result of their participation in the project. There they have a chance to influence policy not just for their own sake but for the country as whole.

299. There are plans to extend the project to other countries in the next triennium, and

the aspiration to include another region further down the line. More consolidation is needed before enough front-running institutions are ready to be used as exemplars or resources for an extension. One stellar performer is not enough – single outliers often defy replication. It would make sense to extend in-country first, and then not until more countries are brought up to speed.

300. If there is to be an expansion of the project – with additional resources – COL

should seriously consider appointing a full-time project manager. This would, amongst other things facilitate the management of the necessary improvements in M&E. So far monitoring has been thin on the ground with heavy reliance on the external evaluator. A somewhat tighter planning framework would probably be useful. The Initiative and Activity logframes are of little value as they are not configured around the project.

Conclusions 301. The Skills Development Initiative outcome is: Training institutions and civil society organisations create the policy frameworks

and increased capacity needed to develop and deliver ODL materials for strengthening knowledge and skills in both formal and non-formal learning environments.

302. This is a very general outcome which could apply to almost any educational or

training context. If we interpret it as applying to TVET, then the outcome is high on the agendas of countries around the developing world and beyond.

303. The TYP Performance Indicators are:

PIs Progress (by November 2011)

1 6 institutions in at least 2 Commonwealth regions begin technical vocational education, or significantly increase the number educated, through ODL.

41 institutions in 15 countries across 4 regions have begun the - or have enhanced their - use of ODL for TVE. Numbers educated are not available.

2 New or improved curriculum content in technical vocational education is in use in institutions in 4 countries.

Broadly the same numbers here as in PI 1, although most non-formal courses are still in development.

3 3 new skills development courses are available as OERs and used by institutions in 4 countries. (This excludes courses developed under

This target has already been exceeded. COL estimates that by June 2012, 12 or more new courses will be available as OERs.

43

VUSSC.) 304. These PI targets have probably all been exceeded – well exceeded in two cases - if

we accept “new courses delivered” as a proxy for increased numbers being educated. This performance against targets is largely explained by four factors: COL has clustered institutions for economies of scale as well as synergies, the

prime example being the CAPA FSD project. COL has worked through AMREF2 to reach larger numbers of institutions. In the Pacific, around 15 institutions – including primary schools – were

helped at arm’s length to adopt vocational literacy courses. The PI targets were insufficiently ambitious.

305. COL’s work is clearly contributing significantly to the Initiative outcome. The policy

aspect is catered for by the FSD project; and capacity development to some degree wherever COL intervenes. The number of institutions engaged is impressive.

306. The scale of impact on learners is modest at present mainly because the flagship project – FSD for CAPA – is young. FSD for CAPA has the potential to create this impact and to be used as a model for replication, but there is probably another 2-3 years of hard work by all stakeholders before COL can be sure of that.

Learning for Farming 307. The Learning for Farming Initiative encompasses what is more commonly known as

the Lifelong Learning for Farmers (L3F) programme. Its basic aim is, through a variety of ODL and ICT methodologies, to build capacity among farmers, landless labourers, and in some cases extension officials, which leads to more productive and sustainable farming and more secure livelihoods. The rural poor still outnumber their urban equivalents in Africa and South Asia and will continue to for some years.

308. At the heart of the L3F programme are three main sustainability factors:

Participatory approaches through which the farmers decide in particular what the focuses for learning should be.

The growth of empowerment and cognitive social capital3 among the communities leading to greater resilience and more options in response to opportunities and external threats (such as climate change).

Linkages with the banking sector and other commercial agencies, providing affordable credit, insurance, ICT and marketing and other services, based primarily on self-interest.

309. The leitmotif of L3F is that banks are willing to lend to these groups because their

collectivisation means lower transaction costs, and their higher productivity – an outcome of the learning process - translates into fewer non-performing assets. The lending in turn reinforces these two features of the communities in a virtuous cycle.

310. The L3F programme was researched and developed by COL and its knowledge

partners4 from 2002, and launched in Tamil Nadu, India in December 2004.

2 African Medical and Research Foundation. 3 Cognitive social capital includes shared norms, values, attitudes, and beliefs. It is thought to predispose people towards mutually beneficial collective action. Another type of social capital – structural – consists of established roles and social networks supplemented by rules, procedures and precedents. It facilitates collective action. 4 Tamil Nadu Agricultural University, Tamil Nadu Veterinary and Animal Sciences University, University of Madras, Anna University, Tamil Nadu Open University, the International Centre for Research in Semi Arid Tropics, and the M. S. Swaminathan Research Foundation.

44

311. There are five Activities in the current logframes:

Implementing L3F in Asia Implementing L3F in Africa Implementing L3F in the Caribbean Implementing L3F in Pacific Voicemail-based L3F

312. As the Activity titles indicate, COL is currently working with L3F in four regions. In

each region there are one or more coherent interventions which this evaluation will call “projects”.

Asia 313. The pioneer project was in two districts of Tamil Nadu, India, and involved the

development of dairy farming facilitated by tailored learning processes centred on village internet kiosks. COL intentionally no longer supports these communities. In one of the two districts – Theni, where kiosks were set up in two villages – the partners at village level have stepped up and diversified their economic activity with no donor support. The number of farmers involved has grown both within and from outside the immediate community. The L3F model seems here to have worked well, although the scaling up has not been dramatic.

314. In the other district, the kiosk closed down after the operator moved away for

personal reasons. Here the project’s roots in the community seem not to have been strong enough to motivate the farmers to look for an alternative kiosk partner.

315. What the pioneer project suggests is that structural social capital is a pre-requisite

for sustainability in L3F. The next iteration– also in Tamil Nadu - therefore was based around a confederation of community-based organisations. This is the “Vidiyal” project which is reviewed in some detail below. Vidiyal uses mobile phone technology as the main channel for information and learning.

316. Another “project” in a neighbouring district has seen the development of certificate

and diploma courses in lifelong learning for rural communities by Arul Andanar College, a university-affiliated institution. The training is conducted in parallel with a community of farmers who acquire knowledge via an off-line voice-integrated touch-screen kiosk which has come to be known as the ‘‘Light on the Wall.’’ The kiosk has a stock of scientifically validated information on locally relevant livelihood systems. The kiosk became a focal point for dialogue, first among the farmers, and then between farmers and other parties, for example over the affordable procurement of fertiliser. As in other L3F communities in Tamil Nadu, state bank representatives have been drawn into a dialogue about credit, attracted by the learning activity and apparent cognitive social capital among the famers.

317. L3F’s first step outside Tamil Nadu was to Sri Lanka. As in the first Tamil Nadu trial,

a tertiary educational institution, ICT providers, and banks have been motivated to help farmers improve their livelihoods. Members of a farmers’ organisation are learning modern agro-technologies to improve their mushroom production. Future plans include expanding marketing efforts beyond the local community, providing lessons to mushroom farmers by mobile phones.

Elsewhere 318. L3F was tentatively introduced in a Nigerian location in 2006 but without gaining

traction. COL turned its attention to locations in Uganda and Kenya in 2008. In both

45

countries, groups of farmers are learning about new agricultural and livestock practices with material provided by national tertiary and research institutions.

319. In Kenya the learning is face to face at designated locations in the villages – both

vertically (through trainers) and horizontally between the farmers. No ICT is involved. The learning process involves a feedback loop from farmers who try out the new practices.

320. In Uganda, mobile phones and handheld DVD players are the channels currently,

but as systematic learning is not yet fully operational, this may change. 321. There are two distinct differences between L3F in the African locations so far

accessed, and the environment in Asia, particularly in the Vidayal project. One is that the degree of community mobilisation is greater in the Asian locations.

322. The other is the role of the banks. In India, state sector retail banks have a

statutory social responsibility for helping rural communities, although not at the expense of balancing their books. In East Africa there are no state sector retail banks, and the private banks have little or no experience in extending credit to members of subsistence farming communities.

323. Despite this, in Kenya at least, both drawbacks are being eroded. With COL’s help,

farmers are being mobilised into a Saving and Credit Cooperative Society (SACCO) through a facilitating NGO which also provides them with training in finance. And two banks have shown interest in lending to the SACCO after COL took representatives to Tamil Nadu to show them the L3F model in action.

324. In the Uganda and Kenya iterations of L3F, COL intervened first at community level,

although it has subsequently brought national banking and knowledge institutions into the frame. In Mauritius and Jamaica, L3F is taking different routes. In both countries, COL has engaged first with government: the Rural Agricultural Development Authority (RADA) in Jamaica, and the Ministry of Gender Equality in Mauritius.

325. In Jamaica the standard L3F model is taking shape after other channels of learning

delivery were discussed with RADA but discounted. The possibility exists of rapid scaling up through RADA if the small scale pilot is successful.

326. In Mauritius, the intervention looks at the moment more like an informal sector

cascaded capacity-building programme. It is designed to enable nearly 100,000 women to assert property rights and develop new livelihoods after the sharp downturn in the country’s cotton industry. But the intention is to involve banking partners further down the line.

327. In the Pacific, COL intends to explore the potential of L3F first with a government

partner in Papua New Guinea, but so far has made little progress because of political changes.

328. The fifth Activity involves the development and piloting – in partnership with the

University of British Columbia’s Networks and Internet Computing (NIC) Lab – of LIVES (Learning through Interactive Voice Educational Systems): an educational software system that offers audio lessons to multiple users over mobile telephone networks. COL and NIC plan to field test the system with L3 farmers in India. Its full potential would be realised if the numbers of farmers with a particular learning need – for goat rearing for example - were substantial.

46

Vidiyal 329. Vidiyal is an NGO in Tamil Nadu that works with a federation of rural women’s self-

help groups (SHGs) – with a total of about 4000 members. Most of the groups score highly in the Indian Government’s SHG rating system, indicating strong social capital.

330. In 2009, about 300 of the poorest members of the SHGs signed up to an L3F

programme of training mainly for goat rearing. They were also given training in business skills.

331. The women developed a business proposal – accepted by a bank - whereby each

member would obtain credit for buying goats and a mobile phone. The main purpose of the mobile phone is to provide a channel for learning. COL and its partners developed the concept of small “granules” of learning that can be delivered aurally; and Vidiyal entered into an agreement with a mobile service provider for sending audio messages and voicemails to the women’s mobile phones.

332. The mobile audio option was preferred by the women compared to computer kiosks.

With mobiles there is no loss of productive time through travel to the kiosk. Audio also appealed because it is the more customary form of communication among groups with low literacy levels.

333. In consultation with representatives of the women and with help from the Tamil

Nadu Veterinary and Animal Sciences University, daily audio messages are produced on such things as buying goats, the management of feed, disease and marketing. Video based multi-media materials are also shown during monthly SHG meetings and are being telecast through local satellite channels.

334. Vidiyal also encourages the women to discuss enterprise issues with one another

using mobile phones. Once a week the members can attend the SHG meetings and share their experiences. Learning notebooks are filled in, usually by the children, This is producing an unintended outcome: the development of the family as a learning centre.

335. In the last 18 months, the scale of the Vidiyal project has increased substantially.

The Vidiyal-facilitated L3F activity has now expanded to more than 5000 women with a commensurate increase in credit. This has gone hand in hand with much greater profile and recognition for the project. The Indian Overseas Bank has appointed two women from the L3F initiative

as its “business correspondents” for transacting the banking operations at the village level.

The mobile phone network started paying the salary of one staff member of Vidiyal for uploading the audio contents.

A memorandum was signed by 25,000 people in the locality and presented to representatives of financial institutions and government, stressing the need for investments from the banking sector on credit-specific capacity building through ODL (the L3F model).

The National Bank for Agriculture and Rural Development (NABARD) has said it is willing to consider adopting the model as national level policy. It wants to see further trials, outside Tamil Nadu and has agreed to coordinate the initiative in two districts in different States with the L3F partners.

336. Probably the main driver for the banks’ sustained and increased interest is the

performance of their loans. The first batch of 320 women repaid 90% of the five

47

year credit, with interest, within three years, compared with a requirement to pay only 75% by that time.

337. An outcome survey was conducted by COL and its partners in mid 2011, building on

baseline data obtained in 2008. Both the 2008 and 2011 surveys incorporated two control groups from the locality: one whose members participated in the SHGs but not L3F, and the other who participated in neither. The survey showed a marked improvement in livelihoods and in indicators of empowerment among the L3F group – significantly greater than the changes among members of the two control groups. Although these groups were not randomised, the L3F group were the poorest at the start, and yet outpaced the other groups. COL acknowledges that the survey only points to the influence of the L3F process as a whole on these outcomes, and does not isolate the learning component. But it is highly plausible that it a major contributory factor.

338. COL has contributed $80,000 in programme funding to Vidiyal since June 2009. Conclusions 339. The Initiative outcome is: Resource-poor farming communities use ODL to increase their knowledge and

skills to access new information, training and financing that improves their livelihoods through various economic activities.

340. L3F is certainly contributing to this outcome, but it is an “entry level” outcome and

unspecific about scale. The Performance Indicators give a more detailed picture of COL’s ambition.

PIs Progress (by November 2011)

1 Members of farming communities in 6 countries in 4 Commonwealth regions have significantly improved economic circumstances through ODL

This may have been achieved already in India, Sri Lanka, Uganda and Kenya (4 countries in 2 regions) – although the only country where it has been fully evaluated is India (and even there the causal link is with L3F in general and not specifically ODL).

2 4 government or international organisations adopt L3F models in 2 Commonwealth regions.

3 government and 2 international organisations in 3 regions have bought into L3F. 3 further government organisations have formally requested to work with the programme.

3 20 organisations in 3 Commonwealth regions strengthen their capacity in ODL to address farmers’ learning needs.

25 organisations are involved, in 4 regions.

341. COL’s evidence shows that the targets for PIs 2 and 3 are likely to have been exceeded by June 2012. PI 1 is longer-term and the innovativeness of L3F means that it needs time – more than anticipated three years ago - to take root in new terrain. The PI 1 target will therefore not be fully met this triennium.

342. L3F is one of COL’s most innovative programmes. COL believes it is still the only

livelihoods programme anywhere in the developing world that brokers the explicit relationship between learning and credit. The attention that L3F consistently gets in public forums around the world, makes this a reasonable assumption.

48

343. L3F is not innovation for its own sake. It has demonstrated, with thorough monitoring and evaluation processes in 2-3 of its projects, that the L3F model can – in certain circumstances - make a significance difference to people’s livelihoods. Furthermore, by empowering people and reinforcing cognitive social capital, it seems to be able to underpin these advances in farming enterprise, building resilience and sustainability.

344. The L3F virtuous cycle – involving learning, credit and livelihood improvements –

has only been fully mobilised in India and Sri Lanka. Here COL and its partners are confident enough in the model to be actively planning a scaling-up process. This has already begun in India. It involves banks – particularly NABARD - taking responsibility for the model and applying it in a larger number of villages in several Indian States. COL’s aim is for L3F to be incorporated in national policy for the state banking sector in India by 2015.

345. It is too early to tell whether this transfer of responsibility will succeed and whether

the model will survive it. If it does, it will be a remarkable achievement with possible wider lessons for sustainable development. If it does not, the standard L3F model will need to be radically re-engineered or put to bed.

346. India is India. The national banks have a statutory obligation to support poor rural

communities. This is not the case in most other countries. In Kenya and Uganda, COL may be pushing water uphill; but for a while, they are probably right to give themselves the benefit of the doubt. However this should be kept under close review to avoid mission drift.

347. In Jamaica and Mauritius, COL is pursuing credibility and scale by partnering with

government institutions. Whether that will help to attract credit to the emerging enterprises on a sustainable basis remains to be seen. If not L3F has the option to diversify down other paths. If these paths lead to innovation which produces sustainable livelihood improvements at scale, L3F model will lose its distinctiveness but still be valuable, as very little attention is being paid anywhere in the world to non-formal learning for the informal rural sector.

348. The L3F PIs emphasise coverage. The elephant in the room is scale in each location.

This should be more of a priority in the next triennium than it has been in this. 349. As long as L3F is innovating, COL and its partners should monitor and evaluate the

separate interventions intensively – primarily for learning and advocacy. L3F has been – and still is - a learning process for COL.

350. Whatever innovation and advances are made at the micro-level, COL now needs to

work hard at finding platforms for the L3F messages that don’t just broadcast its messages but also engage new types of partners for upscaling and diversifying. L3F is old enough to leave the nest and learn to fly. There is a need to demonstrate with clarity what the advantages are. This involves evidence through evaluation but also simpler communications. L3F communications have a tendency to use technical language and acronyms which are not always easy for outsiders to interpret.

351. In 2-3 years time it may be worth conducting a review of the L3F experience

globally, irrespective of its performance by then. Both success and failures will provide important learning for continuing efforts to improve the well being of poor rural communities.

Healthy Communities

49

352. Healthy Communities is another of COL’s highly focused - predominantly single model - Initiatives. It promotes and facilitates participatory learning, predominantly about health, and through the medium of community radio (CR). In this Initiative, unlike CEMCA’s work in India, COL does not help communities to start up CR, but supports its application in participatory learning.

353. Most radio, even if educational in purpose and targeted on specific communities, is

still based on a one-way transmission model. COL’s approach – its “model” in this Initiative – is primarily about community collaboration in the production of programmes, and where possible participation in learning through them. In some contexts this is blended with other learning media.

354. There are two Activities in the Initiative:

Capacity & community learning programme development Materials development and knowledge sharing

355. Broadly the first activity is about helping communities to apply media-enabled

participatory learning, and the second about creating tools and channels for replication. In practice the two often go hand in hand in COL’s work.

356. COL has constructed a model for the effective delivery of participatory learning

through CR, based on a constellation of local partners. The model consists of the following: Media groups who are responsible for content creation and dissemination. CBO networks or federations who are responsible for engaging with (and to

some extend representing) the target learners. Broadcasters with a social development mandate. Health and other specialists who are responsible for ensuring that content is

accurate and credible. Policy groups who are responsible for aligning the programmes to priority

areas. 357. The range of COL’s roles currently consists of:

Advocacy of the educational potential of CR; and of the value and efficacy of collaborative/participatory processes

Advocacy of the partnership model and a certain degree of brokerage where COL is in touch with potential local partners

Injection of capacity building, particularly in collaborative media content creation and participatory learning processes.

358. The comprehensive partnership model may not always be possible to construct in

the short run, and flexibility in implementation is required. 359. COL’s work in this Initiative is widely spread: about 16 countries across four

Commonwealth regions. It is able to do this because it has a fairly standardised package of support which is conceptually uncomplicated and can be applied quickly by the national and regional agencies that COL is now mainly engaging with. Most of the agencies are already working with community organisations and are aware of their need for the kind of support that COL offers.

360. COL therefore engages with communities mainly through intermediaries such as

media training organisations, networks or federations of community based organisations and CR associations (where they exist). COL acknowledges that this arms length approach cannot guarantee good results. A lot depends on the skills and experience of the intermediaries. COL provides capacity development for some of these organisations, but as it is working with around 10 of these organisations, there are limits to this.

50

361. A programme that COL sees as one of the most successful is its partnership with

the South Africa-based Media and Training Centre for Health (MTC) in an HIV/AIDS community learning programme in the country’s Northern Cape Province. MTC is working with approximately 1,000 targeted learners who have been selected by non-governmental organisations and high schools. The distance learning programme is increasing knowledge and support for people living with HIV and AIDS through a four-step process: Identifying needs. MTC conducted an assessment to determine what

information or support learners would find most useful and the appropriate channels for communication.

Monthly contact. Learners receive information and support through participatory radio programmes, SMS messaging, referrals to websites and printed communication.

Face-to-face mentorship. Partner organisations meet with learners to reinforce learning and monitor their progress.

Measuring the impact. Following up on the baseline study conducted at the outset of the project, a second assessment will measure the impact of the intervention.

362. The partnership with COL has led to more work for MTC in South Africa. The

intervention has also attracted UNESCO which is planning to replicate it with different partners in Namibia.

Phukusi La Moyo: a participatory educational radio programme for maternal and child health 363. Phukusi La Moyo is a radio programme broadcasting in Mchinji District in central

Malawi. The estimated population in range of the broadcasts is 360,000 people. 364. The programme is the result of a collaborative effort between an NGO - MaiMwana5

- women’s groups, the Mudzi Wathu Community Radio Station, and the Mchinji District Health Office (DHO).

365. The women’s groups were established by MaiMwana in 2005 and supported to

identify and prioritise their problems, and then to identify and implement solutions. One of the problems that the women’s groups identified was the need to increase their knowledge about maternal health issues. After obtaining mixed results from contact with the government Health Surveillance services, they turned to mass communications and approached the Mudzi Wathu local radio station.

366. COL was initially approached as a source of funding, but it became, in MaiMwana’s

lexicon, a genuine partner. COL identified and funded a Malawi-based NGO called Story Workshop to provide training in participatory radio production. Although the partners were already thinking of the programme along participatory lines, and Mudzi Wathu had had some experience of participatory processes, Story Workshop helped them with the capacity to put it into practice. This was seen by MaiMwana as a pivotal intervention. The workshop enabled the core team to learn skills in message delivery alongside some radio production skills and also crystallised the relationships between the partners.

367. Key characteristics of the programme are:

A collaborative approach, covering both design and operation, among subject experts - MaiMwana and the DHO - and the radio station.

5An NGO established by the University College London Institute of Child Health and the health authorities in Mchinji as part of a research and development project.

51

A high degree of participation by the women’s groups in the creation of content for the programme.

The combination of on-air programming (delivering learning content) with off-air learner support, through a network of listener clubs formed from the women’s groups.

368. The intended outcomes are:

To increase knowledge about maternal health issues amongst communities in Mchinji by educating women about maternal health.

To change attitudes about maternal health solutions. To improve care and care-seeking behaviour.

369. Although COL’s funding was given only for one year, the partners also appreciate

the fact that it has stayed in touch and has signalled that it may offer more help in the future. COL’s interest and faith in what the partners are doing, they say, has given them confidence to look for solutions to the challenges that have encountered. MaiMwana says that although COL did not micro-manage its support – unlike some donors – it has also brought rigour to the programme.

370. The relationship with COL has also introduced the partners to a wider community

media network and provided opportunities for partners to travel to workshops in order to learn and to develop these networks.

371. The external evaluation came to this conclusion: The project was successfully managed and implemented.... The intended outputs

have been achieved and there is qualitative evidence that the programme is achieving the intended outcomes. The project has resulted in a large number of positive unexpected outputs and outcomes.

372. Participation in the production of programmes is high – members of over 100

women’s groups featured in the first 23. Over 3000 women are active in the listener clubs.

373. Among the unexpected outcomes of Phukusi La Moyo, reported by the external

evaluation, are: Raised profile of MaiMwana and Mudzi Wathu within the community. Improved relationship between the DHO and Mudzi Wathu. Improved networking between women’s groups Improved local capacity in project management Improved relations between healthcare workers and community members Encouragement of more open language about health issues. Men in the community took an interest in, and have begun to interact with,

the women’s groups. 374. COL’s support was predicated on promised financial backing from the DHO once

COL’s funding came to an end. Little of the DHO funding materialised, and it dried up altogether in early 2010. There has consequently been no new programme production since June 2010. However, the demand is still high and women’s group members are still meeting as radio listening clubs to hear and discuss repeat episodes of Phukusi La Moyo broadcast on Mudzi Wathu,

375. The key reasons why the partners have not been able to secure the promised

funding from the DHO seem to be: A high turnover of DH Officers. This has made it difficult to keep the project

high on the DHO’s list of priorities. Conventional health promotion activity managed by the DHO is an important

source of income for DHO staff and others.

52

376. Discussions with the DHO are continuing and the partners have not lost hope of

securing stable funding from this source. They are fully aware that they cannot rely on international funding. One of the principal learning points for the partners is the need to focus more on financial sustainability when initially designing and developing the venture.

377. The external evaluation of Phukusi La Moyo’s progress – and COL’s intervention –

up to June 2011 was thorough and well crafted, and provides a good picture of what has happened and what might need to be done in the future in the local context. What is still missing for COL’s modelling and replication purposes is evaluative research on outcomes for the communities from participation in the programme: chiefly changes in knowledge, attitudes and behaviour. MaiMwana intends to do this research, using participatory methods. COL should consider providing support if needed.

Conclusions 378. The Healthy Communities TYP outcome is: Community organisations, NGOs and local public institutions increase their

capacity to create and use ODL materials to improve the health and well-being of their communities.

379. The outcome is focused at the level of capacity and as such is clearly being

achieved. The ODL materials are chiefly radio programme content. Scale of ambition is provided by the Performance Indicators.

PIs Progress (by November 2011)

1 40+ community organisations, NGOs and local public institutions in 20+ countries, in 4 Commonwealth regions, begin to use or significantly enhance or extend their use of ODL for education.

50 district-level and 8 national/regional-level groups, plus 3 international agencies to date. COL expects 13+ additional groups to be using ODL by Jun 2012. This is taking place in 13 countries in 4 regions. COL expects 3 additional countries to be mobilised by Jun 2012.

2 16 new health-related ODL programmes are used by communities in 4 regions of the Commonwealth

13 programmes have been used by communities to date; 10 countries in 3 regions have been involved. COL expects 5+ additional programmes in 3 additional countries and 1 additional region by Jun 2012.

380. COL’s data on PI 1 indicate that the number of organisations beginning to use – or

extending their use of - media for learning has exceeded the target. This refers to all the organisations involved in the making and use of ODL – including radio stations, NGOs and other facilitators as well as the primary users - usually community organisations. So in some cases, several of these organisations are involved in the development of one programme.

381. In reality, not all of these organisations are yet involved in current ODL delivery

programmes. A more accurate form of the indicator would be: the number of organisations whose capacity to use ODL has been significantly enhanced.

382. In terms of the geographical coverage, the PI 1 is expected to be nearly met.

53

383. PI 2 refers to the number of ODL programmes actually in use. In all cases a “programme” refers to a series of radio broadcasts. As has been pointed out above, the number is smaller than that of the organisations in P1, because more than one organisation is involved in the making and use of a programme. COL expects to meet all aspects of this target by the end of the triennium.

384. The main reason why the number of organisations involved has been exceeded is

that COL and its consultants are increasingly working with national or regional organisations to cascade the model, rather than engaging directly with community organisations.

385. Phukusi La Moyo is, according to COL one of the best representations of the model,

particular in terms of the listening-learner groups. The programmes in South Africa are said by COL to be on a par. The majority of others have not been as successful in terms of progress in participation, but COL maintains that with most, there have still been solid outputs in terms of capacity and radio programming.

386. COL believes that its efforts are bearing fruit mainly because they are focused on

something relevant, tangible, and engaging - i.e. the learning programmes. COL says that it generally gets strong buy-in from the communities and that in most cases there is a fairly strong sense of ownership and motivation among local partners because the programmes are helping them fulfil their mandates more effectively.

387. The arms length approach seems to achieve wide reach, although not on a large

scale in any location. Apart from the benefit to the people who directly participate in the programmes that COL assists – perhaps around 100,000 in number – the current value of COL’s work is in the refinement of the model and its testing in a number of different social, cultural, economic and political contexts.

388. The model draws on work that the ES for the Initiative was engaged in with

UNESCO before joining COL; but it has been developed and refined since he joined COL and worked with new partners.

389. As with other models, COL needs to validate it, through the capture (by M&E) and

use of evidence, and if appropriate further refine and diversify it. The evidence strategy is in place for Phukusi La Moyo. This needs to be complemented with similar data from a number of other programmes in different contexts.

390. COL would then have a valuable tool for wider advocacy for its model with a view to

building international strategic partnerships for multiple replication. COL’s work should move further upstream in 2-3 years, by focusing on strategic partnership and supporting the development of remote – through toolkits, eLearning etc - capacity-building of local partners. There should also be closer liaison with CEMCA’s CR work.

Integrating elearning 391. Integrating eLearning is another of COL’s broad-brief Initiatives, but is not sector-

based. It is concerned with promoting and supporting the use of ICT in teaching and learning in both formal and non-formal contexts and at all levels. Although it is located in COL’s Livelihoods and Health Sector, this seems to have been done simply to balance the Logic Model, and its brief extends to the Education Sector as well. In fact most work in eLearning in the TYP to date has been in the formal education sector.

54

392. There are three Activities: Increase digital literacy in institutions and communities. OER content development and support. Policy support for governments.

In practice, much of COL’s work in this Initiative combines two or more of these elements, and therefore cuts across the Activity boundaries.

393. The emphasis in this triennium has been on models for eLearning capacity development, and policy development for integration of ICT in education. In the Caribbean in particular, these two elements have been combined in what amounts to an emergent regional project to support governments in the implementation of ICT policy.

394. In this Initiative, work has continued on OER content development, particularly on

the Commonwealth Computer Navigators’ Certificate; and COL’s Instructional Design Tool has been re-launched in digital form and widely promoted

395. COL has recently joined a two year old partnership between Nokia and various

South African institutions to further develop a low cost OER mLearning model for Maths (known as MoMaths). COL is helping to create the strategy and prepare the content for transferring it to other African countries.

396. The Caribbean and Africa have been the main regional focuses of the Initiative;

although some work has also been done in the Pacific, with the possibility of new activity beginning there on a significant scale before June 2012.

ICT for Education (ICT4E) in the Caribbean 397. COL’s ICT4E work in the Caribbean is the largest and most coherent set of

interventions in this Initiative. It is aimed at integrating ICT in teaching and learning in schools. This is to be achieved mainly through the adoption or adaption of policies and through the institutionalisation of the building of teacher capacity in the integration of ICT.

398. The work, which began in 2009, is carried out with two strategic partners: the

Commonwealth Secretariat (ComSec) and Microsoft. The partnership was forming embryonically when Guyana’s request to ComSec for support in implementing its new ICT in education strategy triggered the activity in the Caribbean.

399. The purpose of the COL-ComSec-Microsoft partnership is to support countries in

integrating ICT in teaching and learning in schools, in the medium term mainly through the application of the UNESCO ICT Competency Framework for Teachers (CFT)6.

400. The process began with capacity-building activity for representatives of Ministries of

Education and the leaders of education institutions in three countries: Guyana, Trinidad and Tobago and Bahamas. It is now extending to Antigua & Barbuda, St Vincent and the Grenadines and Grenada.

401. The main capacity building vehicle that helps countries’ teachers progress through

this framework is a course model known as the Commonwealth Certificate for Teacher ICT Integration (CCTI). The CCTI is an advanced certificate qualification, taken at distance, based on a programme originally designed for school-level educators in South Africa. It is an open educational resource that COL is making

6 The current CFT is currently what might be called a “beta” version. It is not released for global use.

55

available to Ministries of Education and teacher training institutions throughout the Commonwealth and beyond. The material can be used “as is” or can be adapted for use with existing programmes.

402. The CCTI is focused on developing competencies beyond digital literacy with an

emphasis on the pedagogical integration of ICT in learning and teaching. 403. COL has taken responsibility for helping the countries adopt the CCTI model, and to

inject a trainer training component. This is intended to equip participants to develop others in a cascading process. Countries themselves are expected to provide training for the main components of the course. In line with COL’s strategic commitment, materials are to be made available by countries as OERs.

404. The partnership support began in Guyana where the main focus has been on policy

and strategy for teacher development and ICT integration in schools, through the medium of the CFT – or at least some aspects of it. The exposure to international experience is said by Guyana stakeholders to have been a critical factor. Guyana is not ready to adopt the full CCTI model at present, so the partnership is helping them to produce a tailored version.

405. The partnership has also been active in Trinidad & Tobago and The Bahamas. In

these countries, the emphasis has been on developing a cadre of tutors who can facilitate the CCTI as it evolves and grows over time. Accreditation of the course by local tertiary institutions has been completed and the first iteration of the course is underway. This raises hopes for the sustainability of the programme through local ownership. In this way, broader national policy is given concrete expression at local level. Trinidad & Tobago intends to make its own adjustments after the first edition of the course, based on feedback.

406. COL has conducted monitoring and evaluation workshops for all three countries and

supported the development of an M&E strategy for Guyana – a factor for sustainability.

407. The programme is extending its activity to St Vincent and the Grenadines, Antigua

and Barbuda, and Grenada with the intention of even wider application in the Caribbean in 2012 and to the Pacific in 2013.

Analysis 408. COL’s two strategic partners are positive about the work and believe that it is on

the way to having a transformative effect. An external evaluation found that the partnership had already influenced the educational policy terrain in all three countries, and particularly in Guyana. It pointed out that COL had established strong relationships with senior figures in the Ministries of Education and other relevant agencies such as the Ministry of Science and Technology in Trinidad and Tobago (T&T). This is important both for effective implementation and future sustainability. Other key informants saw emerging positive outcomes.

409. The Caribbean would seem to be a good place to launch the CFT. ICT infrastructure

is relatively good, and demand among teachers for capacity-building in ICT is generally high. COL and its partners believe that their experience with the CFT/CCTI package in the Caribbean hold out the immediate prospect of replication – not as a single-solution model, but as a framework with a suite of variations on a theme within it.

410. There is no good reason to doubt their optimism as things stand. The main issue is

timing. As with all replication strategies, there needs to be sound evidence behind

56

the model or framework, to give both advocates and target groups the confidence to invest in the intervention. The ICT4E work has been evaluated but at an interim stage. The continuing rollout of the CFT in the Caribbean should be carefully monitored wherever possible by the country partner institutions. What relay counts is what goes on in the classroom. A further evaluation in 12-18 months would enable short and medium-term outcomes to be traced in Guyana and possibly T&T and Bahamas. This should enable better decision-making in any replication phase.

411. For COL an interesting feature of this work is the strategic partnership. This way of

working is not untypical of COL, but what is relatively unusual for COL is that all the partners are working simultaneously in a coordinated way; and there appears to be no sense of primacy. L3F is not unlike this, but on a completely different scale. COL more typically works with strategic partners either sequentially or in a process where one partner is clearly the lead operator and the other takes a secondary role.

412. In the ICT4E partnership, there are distinct but complementary roles. This is a good

model for multi-stakeholder partnerships in education: it enables the pooling of intellectual as well as financial resources to support change.

413. All three partners, and other informants, testify to its effectiveness. This is

important for two reasons. First, it may be a partnership model that COL would want to replicate in other circumstances, in which case, the partnership processes should be documented. Second, this partnership appears to have the potential to outlive the Caribbean work and could become an important feature of COL’s eLearning work more generally in the next triennium. The ComSec member is already working with COL’s knowledge management team on the development of the Commonwealth Connects portal and he sees the potential for bringing that aspect of COL’s work into the ICT4E frame.

414. Strategic partnerships are not necessarily a one-way street to success. They involve

compromises which are not always in the best interest of the programme partners. Sometimes the presence of one or more of the strategic partners is not wholly welcome by the programme partners. In this example there is understandably a degree of wariness about Microsoft because of fears – so far unfounded – that it will want to impose its proprietary software on the intervention.

415. There is also a danger in strategic partnerships that maintaining them becomes as

an important an objective as providing assistance to programme partners and populations. This can lead to mission drift.

Conclusions 416. The current ICT4E Initiative TYP outcome is: Governments, institutions, individuals and communities:

Integrate ICT into teaching and learning in line with ICT in Education policies and implementation plans.

Use digital technologies to design and develop learning materials that are made available, where possible, as open education resources.

Apply low-cost ICT in Education teacher training models. Provide effective and appropriate skills training.

417. Elements of this outcome statement were added after the beginning of the TYP

period when the current ES joined COL and found her feet. In the previous TYP, the main focus was directly on individual educators through the development of WikiEducator and the rollout of the Learning For Content programme. Although elearning in this TYP has also targeted the competencies of educators, the route has been through governments and key institutions, with more of an emphasis on

57

policy. The inclusion of “governments” and the first and fourth bullet point in the outcome reflect these changes.

418. The current outcome is more multi-faceted than most, but in practice all elements

are all being pursued through the ICT4E work in the Caribbean - and some elsewhere.

419. The first two TYP Performance Indicators were set by a different ES with different

expectations. The third was added later and is specifically designed for the ICT4E in the Caribbean intervention.

PIs Progress (by November 2011)

1 10 major institutions in at least 2 Commonwealth regions use open educational resources for their curricula.

COL believes that its range of capacity interventions in OER under this Initiative has already led to this PI being exceeded, although usage is not yet being monitored.

2 Additional low-cost ICT training models in use in 8 institutions in 4 Commonwealth regions.

11 institutions, all but one in the Caribbean, are using the CCTI model or are expected to be using by June 2012. An mLearning model has been developed and is expected to be in use in 3 African countries by June 2012.

3 4 Governments in one Commonwealth region have integrated ICT in teaching and learning.

Guyana is actively integrating. Trinidad and Tobago, The Bahamas have begun to integrate. The process is about to begin in Antigua and Barbuda, St Vincent and the Grenadines and Grenada.

420. The first PI is not a good fit with the work in the current TYP and it is not possible to

verify it with the data easily available. If the target has been met, new OER usage as a result of COL’s interventions in this period will not be substantial as COL’s works has been upstream with policies, frameworks and capacity building - and actual deployment and usage of OERs will take time to flow.

421. PIs 2 and 3 lock onto the ICT4E work and the targets have either been exceeded

(PI 2) or will be nearly met (PI 3) by the end of the triennium. 422. The change of direction in this Initiative in the first months of the TYP was not easy.

The transfer of management of WikiEducator to a New Zealand institution left a space that was not clearly defined. The new ES brought a different set of experience and skills compared with her predecessor and it took time for COL to identify a new direction to follow. Once this was done, however, the work gathered momentum very quickly. This was helped by two factors: The strategic partnership with ComSec and Microsoft which came together

and took off very quickly The use of pre-existing models and materials that needed adapting and re-

purposing but not re-inventing. Other institutions had done the “heavy lifting”. 423. Both of these strategies by COL have paid off. The work in the Caribbean has been

effective so far and promises more in the future. Other work in the Initiative has been lower profile but may also be producing good outcomes. It is important that the stories are told in an accessible way. This will help COL build on the achievements in the next triennium.

CEMCA

58

424. CEMCA is COL’s regional agency for Asia. It is the only unit outside Vancouver that

COL manages directly7. CEMCA was established in 1995 as a functional entity, focusing exclusively on the application of educational media, particularly in higher education. In its first decade it worked mostly in India.

425. Today CEMCA retains a focus on technology, but operates on a wider front both thematically and geographically. Its mission is to promote “the meaningful, relevant, and appropriate use of information and communication technologies to serve the educational and training needs of Commonwealth member states in Asia”. It received a programme budget of CDN$350,000 in 2011.

426. In COL’s RBM framework, CEMCA is treated as a separate “sector”, although its

Activities – 14 in the last year of the Plan - cluster under five of COL’s Initiatives: Higher Education, Open Schooling, Healthy Communities, Skills Development and Integrating eLearning.

427. Documentary evidence of CEMCA’s work gives the impression of a very active unit

operating on several fronts. Although it is best known at present for its work in CR, CEMCA does not have all its eggs in this basket. Two examples of CEMCA’s non-CR work illustrate this diversity.

428. COL and CEMCA are particularly confident in Easy Now, an open-source software

platform that facilitates the development of ODL content in multiple formats from a single electronic source. It is essentially a low-cost and efficient alternative to the application of multiple pieces of software for content creation. Easy Now was developed and trialled by an Indian IT expert from his own concept. CEMCA provided the opportunity and the platform for leverage. It is currently in use in institutions in four Asian countries.

429. In line with COL’s focus on QA frameworks and tools that can be applied in ODL,

CEMCA is facilitating the development of the Quality Assurance for Multimedia Learning Materials (QAMLM) programme with Government of India Ministry of Human Resource Development, and with the Malaysian Ministry of Higher Education as partners.

Community Radio 430. The largest single area, by far, of CEMCA’s work is the promotion and development

of CR in Asian countries. The modest amounts of CEMCA funding are complemented by external funding (CDN$ 250,000 in 2011).

431. Most of CEMCA’s CR activity has taken place in India, but it has made a concerted

effort in Bangladesh. It has also tried, without success so far, to stimulate activity in the Maldives and Sri Lanka because of lack of support from the governments. Much of CEMCA’s activity has been in advocacy, awareness-raising, and support for licence application; but operational capacity-building as well as content generation has also featured on a small scale.

India

7 SADC-CDE and RETRIDAL are funded by COL and work within COL’s overall mandate, but are managed by other institutions.

59

432. In India most of CEMCA’s activity has been in the context of strategic partnerships

with Indian Government and international institutions, which is reflected in the large scale of the interventions.

433. The growth of CR has been encouraged by the Indian government for several years.

It is seen as an important component in the country’s development strategy, capable of delivering locally relevant messages for health, education, farming etc, in local languages. According to a senior Government of India spokesperson, it is valued for its potential to be a subtle enabler of empowerment for people without a voice.

434. At first the government’s CR strategy was focused on educational institutions; but

from 2006, NGOs were also encouraged to apply for licences. 435. COL has consistently advocated and supported CR in the Commonwealth for over

10 years, but did not play a major role in India until 2007 when the current CEMCA director was appointed. He brought solid experience of CR with him, having been responsible for India’s first CR station at Anna University from 2004.

436. CEMCA’s first venture in CR was a project on women’s health and nutrition with the

National Council for Science and Technology Communication. This work has been sustained and has led to the production of over 4000 programmes in 12 regions. Based on experience at Anna University, the project involved participatory processes, with many women listeners becoming script writers and presenters.

437. The health and nutrition project gave CEMCA credibility in CR and was the

springboard for its participation in a national CR awareness campaign led and funded by the Ministry of Information and Broadcasting (MIB). In 2010, CEMCA leveraged $200,000 of Government of India funding, mainly for workshops.

438. CEMCA has conducted awareness workshops aimed mainly at NGOs and educational institutions. These were designed to motivate the participants and prepare them to undertake the process of securing licenses. COL has also supported the front runners – such as Radio Mattoli in Kerala – on a one-to-one, needs basis. The founder of Radio Mattoli described CEMCA as a “guiding light”. Many of these licence applications are from organisations that have no experience of media broadcasting.

439. To date, CEMCA has organised 27 CR Workshops with about 2500 participants. Since the workshops began, 600 organisations have entered the application process, and there are now 125 operational CR stations, about half of them having gone on air in the last 12 months. The external evaluation reported high levels of satisfaction with the workshops among the participants sampled, despite differing motivations for attending.

440. As well as the workshops CEMCA supports the CR development process with other

services. CEMCA has designed an on-line service – the Community Radio Stations

Management Information System (CRMIS) – on behalf of the MIB. This facilitates the initial application process, and enables organisations to track progress.

It has opened a Facilitation Centre (FC) with grant support from the Ford Foundation. The FC incorporates a telephone help line and an on-line facility offering toolkits, guidelines and other resources. Interns are being trained to staff the FC. It will soon begin a face-to-face service in central Delhi in space provided free by the Indian Government.

60

CEMCA is advising the MIB on setting up a CR Support Fund and is likely to be invited to play a role on its administration.

CEMCA and the MIB have published a Compendium of Community Radio Stations, providing information on matters such as programming focus, partnerships and funding. This is seen by the MIB is a valuable advocacy tool and has led to enquiries about involvement in CR from several other Ministries.

CEMCA promotes and supports internet CR – with five stations so far in operation as a result. Internet CR has the advantage of making programme archives freely available.

CEMCA has helped the Indira Ghandi National Open University (IGNOU) develop a distance certificate course in CR which CEMCA aims to extend to a diploma.

441. In April 2011, COL organised the first National Community Radio “Sammelan”

(colloquium), bringing together 90 CR stations from different parts of the country. For many this was their first opportunity to benchmark and share experiences with other stations and was said to have been an important confidence booster. Participants in the Sammelan included other actual and potential stakeholders – government, NGOs, international agencies and educational foundations – and resulted in new commitments to support, from UNICEF and the Ford Foundation. A second Sammelan is scheduled for February 2012.

442. CEMCA has played a critical role in the recent development of CR in India. It has

been the MIB’s principal operational partner in the workshops, and is seen as a strategic partner for future expansion. It is valued by the MIB both for its technical expertise and organisational capacity.

443. CEMCA is planning, appropriately, to phase out its direct involvement in awareness

workshops early in the next TYP. If it continued there would be a risk of it becoming entrenched as an operational agency of the MIB. There are still thousands of community development blocks in India which are untouched by CR. CEMCA intends to continue to support CR in India but at a more strategic level. Some elements of this role are already in place – the FC for example. Face to face training will still be needed, but this needs to be cascaded by a cadre of master trainers which is not yet in place. CEMCA has begun to develop a small group of multipliers, and this activity should be stepped up.

444. CEMCA intends to focus more on content creation, marketing and operating

capacity in the next TYP. This should also be at the level of trainer training only and there should be a focus on OERs.

445. Longer term, CEMCA sees the recently established Community Radio Association as

its successor in advocacy and capacity building. It will need substantial support to do this.

446. CEMCA is undoubtedly providing valuable services to the CR movement in India –

with good results. Through this work CEMCA has considerably enhanced its profile and reputation, attracting both demand and new partners to help meet the demand. This has not been without complications. UNESCO – an important player in CR in India - has not always agreed with CEMCA’s approaches.

447. There are two issues – one short term and one long term – that COL needs to

address. The short term issue is how to ensure the sustainability of the CR work when the current director retires in 2012. He is so closely identified with the work that there is a risk of a weakening of the momentum. Another CEMCA staff member has been working with CR for over four years, but the incoming director should also

61

take a strong interest in it and be supported in this by COL. This has implications for CEMCA’s participation in COL’s RBM system which are discussed below.

448. The second issue is a potential longer term opportunity. How can COL leverage

CEMCA’s experience and reputation in CR in other countries? Institutional frameworks for CR may be country-specific, but the rationale for, and challenges faced at the operational level by, CR are similar in different contexts.

Bangladesh 449. The Government of Bangladesh introduced a policy and enabling legislation for CR

in 2008, modelled on India’s earlier lead. However it has done little in the way of awareness-raising and facilitation and made no funding available for this.

450. CEMCA decided to push ahead with an intervention of its own rather than pursue

advocacy with the government. It approached a Bangladesh-based non-government development organisation – Rupantar – to be its implementation partner. Rupantar had no history in CR but had used theatre, audio-visual technology and national radio for development awareness. It has a wide network of NGOs; and COL has worked with Rupantar in an intervention managed from Vancouver.

451. CEMCA’s long-term aim with this intervention is to create widespread awareness of

the opportunities CR can provide and build a cadre of master trainers who can cascade capacity building in CR content development and operation.

452. Since 2009, CEMCA has run workshops on participatory content development and

production, managerial and technical aspects of CR for seven out of the 14 organisations granted CR licences by the Government. The other seven have not yet been granted licences by the government.

453. It has also organised three study tours for small teams to CR stations in various

parts of India and intensive training at Bengali-speaking CR stations in East India, establishing a mentoring relationship for some.

454. Satisfaction with the training – and the tours – was reported to be high. However,

delays in the government approvals for go-live led to a leakage of many of the trained staff. One of the functioning stations lost almost all of the trainees.

455. Two stations are now operational – the first since early October 2011. 456. CEMCA is also promoting internet radio in Bangladesh which will have content

sharing advantages. 457. CEMCA is going it alone in Bangladesh. Not only is there no support from

government but also, CEMCA believes, none at present from international organisations or bi-lateral donors. Perhaps even more critically there is a very immature market for CR among the people. Finally stations are concerned about the initial and on-going costs of equipment and expressed a preference for help to find cheap alternatives. This is a fragile environment.

458. CEMCA’s CR work in Bangladesh is well crafted and builds on CEMCA’s and COL’s

experience in CR elsewhere. COL (Vancouver) and CEMCA are in touch about their respective CR activities. Perhaps the most promising path would be analogous to L3F: nurture the development of two or three stations, attempt to link them to sustainable funding sources, at least in the medium term, and use the examples for advocacy aimed both at government and other agencies, in order to scale up.

62

459. If this worked, CR stations in Bangladesh would become part of a network with a critical mass, breaking down the isolation and reducing the vulnerability of these early adopters. Bangladesh has more potential for networking than India because of a largely common language.

460. A comprehensive piece of evaluative research, covering COL’s experience in CR – in

Asia and elsewhere – in about three years’ time would be valuable. Conclusions 461. The last time CEMCA was evaluated externally was in 2004. The evaluation

endorsed COL’s proposal at the time for its transformation from a functional specialised agency to a regional centre, much more closely integrated with COL’s wider remit. An action plan to set this transformation in motion was agreed, but COL’s TYP evaluation in 2006 reported that the proposed transformation had not progressed very far and that CEMCA appeared to be having very little impact with its client base. Part, but not all, of the explanation for this was continuing poor relations with IGNOU, its former host.

462. The comparison with today’s CEMCA is marked, particularly in terms of its

reputation and contribution to outcomes. With modest funding, it is reaching a wide variety of client organisations and in some cases making a noticeable difference. CR in India is the obvious example, but not the only one.

463. CEMCA has extended its work in other countries in the region, although this is still

work in progress. Through no fault of its own, CEMCA has been unable to work in Pakistan and COL is rightly considering an alternative strategy for that important client country.

464. CEMCA’s participation in COL’s RBM framework and processes is still looser than

COL’s programme work managed directly from Vancouver. COL’s outcome statements are not referenced in all of CEMCA’s logframes; and in some cases the logframes do not explicitly subscribe to COL’s TYP PIs. CEMCA generates its own PIs that are mostly at the output level and change from year to year. CEMCA’s Activities are discussed and approved by COL Vancouver, but the RBM follow-through could be more closely aligned with COL Vancouver.

465. This may not have been a negative factor in CEMCA’s recent work. In fact some of

its Activity logframes are more meaningful than some of COL Vancouver’s. Nevertheless, CEMCA’s use of RBM for and in the new TYP, should be better integrated, particularly because of the change of leadership in 2012.

466. The CR work in India has been very successful and some other activities, such as

Easy Now, are also bearing fruit. However there is an impression among informants that CEMCA is trying to do too many things and is not benefiting from the strategic knowledge that good output and outcome monitoring, followed by evaluative reflection, can lead to.

467. The new director should be supported by COL Vancouver to continue CEMCA’s

successful work, as well as explore new directions in a strategic manner. eLearning for International Organisations

63

468. It was not in the scope of this evaluation to assess COL’s eLearning for International Organisations (eLIO) initiative; but a brief review is appropriate as it is a service that COL provides to outside clients by leveraging its core competence in ODL.

469. eLIO develops and delivers technology-mediated distance learning courses for the

staff of international organizations like WHO, ComSec and UNHCR. Courses range from the general (e.g. writing effectively) to the specialised (public debt management). The learning materials are developed and customised for each organization to promote the learning of relevant occupational skills. Tutors are engaged and trained to facilitate the delivery of the programmes.

470. The programme is run as a discrete business stream and is required to cover its full

costs. This is almost being achieved. eLIO’s average annual revenue in the first two years of the triennium was $567,000. This is somewhat lower than the average in the previous triennium mainly because of the strength of the Dollar.

471. About 2,200 learners (of whom 57% were women) from nine international

organisations were trained in the first two years of the triennium. The majority of these are working in non-Commonwealth countries. The average completion rate –80% - is relatively high.

472. eLIO was last evaluated in 2008. The conclusions were very positive. ..all the stated results were met and in some cases exceeded...[and] a rather

overwhelming degree of satisfaction [was] expressed by all the participants, including the learners, the partners, the tutors, the COL staff and the eLIO section employees.

473. While endorsing this conclusion, the 2009 TYP evaluation pointed out there was less

than universal support among Board members for the eLIO programme. They were mainly concerned about mission drift and the diversion of apex management time. Although the evaluation found no evidence to support these concerns, it recommended that COL “try harder to leverage wider benefits from it to support the case for its relevance”.

474. There does not seem to have been any significant change in the way the unit works

or the way COL interacts with it in the current triennium. However the Board has not raised any serious issue about the programme in the triennium, and no-one consulted in this evaluation felt strongly enough about it to suggest any change.

Cross-cutting themes: gender, quality and appropriate technology Gender 475. COL's Gender Policy states that “gender equality is a cross-cutting corporate goal

that requires that both women's and men's views, interests and needs shape its programmes, policies and processes...[and that] the advancement of gender equality and women's empowerment are central to its agenda of learning for development”.

476. COL further states that it aims to be “a model organisation which mainstreams

gender in all its processes and at all levels”. 477. This is an ambitious aim. In pursing it, COL has invested a considerable amount of

time and money in the last decade in creating frameworks and tools for gender mainstreaming in the organisation and its programmes. COL also provides gender-related resources for its programme partners, and other external stakeholders and

64

members of ODL communities, principally through its Gender Micro-site. The Micro-site provides a one-stop platform to access print and electronic resources on gender and ODL with a special emphasis on ICT.

Mainstreaming 478. COL has a Gender Action Plan for the current triennium which is updated annually

and on which it reports to the Board. The Plan has a considerable number of objectives – possibly too many - and is promoted and monitored by a staff gender committee which meets quarterly. There appears to be a strong sense of purpose among staff on the committee.

479. At the centre of the plan are two commitments:

At least one gender-related activity will be implemented in each Initiative. The “longitudinal study” for each Initiative will provide evidence on whether

COL’s interventions have advanced gender equality or not with appropriate data from the field.

480. The former appears to have been achieved. Much of COL’s work addresses gender

deficits intrinsically – particularly relating to women – especially in the Healthy Communities and L3F Initiatives.

481. In terms of the provision of evidence, the second commitment has not been

fulfilled. Several of the evaluations failed to address the question of gender advancement and merely reported on gender balance in monitored activity such as workshops. This could have been due to flaws in the commissioning process, lack of competence in this area by evaluators, or the non-availability of data that could probe the issue of gender more deeply. Gender advancement is an outcome and the evaluators probably came up against a lack of outcome data.

482. This is not to say that COL has failed to advance gender equality through its work.

It is entirely plausible that it has in many – if not all - of its Initiatives. It simply has not provided enough evidence except in L3F.

483. The Stakeholder Survey in November 2010 found that “COL was generally not seen

to demonstrate its commitment to gender equality in its outcomes”. This is echoed in comments from several informants8 in this evaluation. This is probably in part a failure of communication. It may also point to the need to refresh and simplify COL’s use of its gender framework and tools. COL staff in general seem strongly committed to gender equality. More training is probably not needed. COL needs to give its staff one or two effective tools to help them integrate gender equality perspectives into their work.

484. For example, COL has developed a simple checklist tool for gender mainstreaming

in planning, implementation, monitoring and evaluation, known as the Gender Keys. This alone, if applied consistently by all staff, would make a difference in these areas. It should be more widely used.

485. COL also needs to choose, and possibly refine, a gender outcome statement and 3-

4 indicators from the long list in its Gender Action Plan, include them in the Logic Model for the next TYP, and work purposefully towards them. The current TYP Logic Model had no Gender outcome statement, and the indicators, based on the longitudinal study findings, were not well served with evidence.

8 “The issue of gender equality needs to be better highlighted, as the gender perspective has not come out clearly in COL’s work”. (Board member.)

65

Quality 486. COL states in the current TYP that: Quality will be a crosscutting theme in recognition of its pervasive importance.

With its involvement in developments at the international level (such as the Global Initiative for Quality Assurance Capacity and the global campaign against degree mills), COL has the ability to assist Member States with quality issues in an effective and informed manner.

487. There is no specific outcome statement in the TYP for quality as a cross-cutting

theme. The Logic Model however has an indicator: 80% of independent evaluations of COL activities and 80% of stakeholders in the

triennial survey conclude that COL interventions have led to high quality ODL design and delivery.

488. As with gender, not all of the independent evaluations systematically addressed this

issue, although the general impression is that COL has supported its programme partners well in designing and delivering quality ODL in the evaluated interventions. The main exception in this TYP periods is RETRIDAL which has done very little of anything. In the OERs for Open School project, as we have seen, there has been an issue with the concept of “high” quality. This is on the path to resolution.

489. Participants in the 2010 Stakeholder Survey were asked about the quality

dimension to COL’s work. Most felt that without a clearer definition they could not comment meaningfully on the issue.

490. COL is dropping quality as a cross-cutting theme in its next TYP. It feels, with some

justification, it no longer needs to emphasise that it strives for quality in and through its work.

Appropriate technology 491. COL’s TYP states that it will encourage the use of accessible, affordable and

effective technologies in support of learning. 492. Appropriate technology is treated in the same way as quality in the TYP Logic

Model: there is no specific outcome and the indicator is based on evaluation findings.

493. “Appropriate” is by definition context-specific. The evaluations were able to address

the issue and most concluded that the technology was appropriate. There was some concern among the CR stations in Bangladesh contacted by the external evaluator that they would find the technology unsustainable. CEMCA however challenged that, stressing the low cost of the equipment.

494. Because of its context-specificity, few people in the 2010 Stakeholder Survey were

able to comment outside their own immediate experience of COL. In their direct experience however, COL was seen to promote appropriate technology. A few ventured that COL promotes the use of advanced ICT even in challenging environments. However none felt this was inappropriate. They all acknowledged the need to build awareness and capacity for a near future when ICT infrastructure is expected to improve substantially. In the 2006 evaluation, in contrast, several informants were sceptical about this.

495. COL will also drop appropriate technology as a cross-cutting theme in the next TYP. Programme support

66

496. As was mentioned in the Introduction, it is not within the scope of this TYP

evaluation to address programme support. However there are three areas that need to be referenced in this evaluation: RBM, Knowledge Management and specialist staff succession.

Results-Based Management 497. Results-Based Management (RBM) is a strategic approach to management that

translates principles and components of the programme/project cycle to the level of the organisation.

498. It should contain the following main components:

Clarifying realistic expected results for the organisation as a whole, with a particular emphasis on outcomes.

Basing strategic planning on these results, with appropriate attention to risk. Ensuring that planning at operational levels – programmes, projects, services

etc - align with the expected organisation-level results. Identifying indicators and targets at all levels that relate to the planned

results and align between the different levels. Managing for these results. Monitoring whether these results are being achieved. Integrating evaluation to discover how results in the most important areas

were achieved (or why they were not achieved). Analysing and collaboratively reflecting on the evidence emerging from the

monitoring and evaluation. Reporting on performance to stakeholders in a transparent and meaningful

way. Incorporating information and knowledge generated by these processes into

the next cycle of strategic and operational planning. 499. COL adopted the RBM approach with its 2003-06 TYP. Since then it has been

regularly developing and improving its approach. Among other things it produced a practical guide to results-based M&E – which amounts to a large part of the RBM cycle - in 2009 that provides a useful orientation for new staff and is regularly consulted.

500. COL does not have specialist planning and M&E staff but occasionally uses

consultants to facilitate these processes. COL adopts a mainstreamed approach to RBM. This requires capacity building and support. Capacity building can be – and in COL is being – acquired in ways other than formal training. COL staff however may also benefit from structured training in RBM in the next 12-18 months, especially if the recommendation for better outcome monitoring and M&E planning is accepted.

501. COL has made good progress with its RBM so far. Institutionalising RBM and making

it work is a long haul project. Some organisations that started several years before COL are still struggling with it. The detailed assessment of COL’s RBM that follows should be seen in this context. Apart from 2-3 substantial steps that COL could usefully take to further improve, it is mostly fine-tuning.

Defining results, indicators and targets 502. Since 2006, COL’s TYPs have been built around a Logic Model containing, among

other things, statements of impact and outcome plus performance indicators for outcomes and the three cross-cutting dimensions: gender, quality and appropriate technology. An improvement in the 2009-2012 plan was the alignment of indicators

67

with specific outcome statements, as recommended in the evaluation of the previous TYP.

503. Experience with RBM suggests that defining results, indicators and targets should

incorporate both top-down and bottom-up processes. Key stakeholders should be involved and it should be an iterative process.

504. The results statements and indicators for the 2006-09 and 2009-2012 plans were

constructed largely by the ESs in consultation with senior COL management. In two or three cases, the ESs also consulted strategic partners before finalising the components for the 2009-12 plan.

505. This approximates to the recommended approach in most respects, although more

involvement by strategic partners would have been valuable. The process of defining results, indicators and targets was also somewhat perfunctory for the 2009-12 plan. There was little or no iteration. Organisations rarely hit on the optimum results, indicators and targets first time round; and repeat attempts, between periods of reflection and further consultation invariably pay dividends.

506. The indicators for 2009-12 were generally an improvement on those in the previous

plan. They were more closely related to planned interventions. However many were imprecisely defined and lacked the accompanying technical notes that indicators usually need. These notes define terms in the indicators and explain how the indicators should be monitored. This is particularly important where there is likely to be staff changes in an Initiative.

507. Another improvement would be to break down the targets into annual stages. This

can be difficult to get right, but it gives all stakeholders a better idea of when the results are intended to emerge.

508. The 2009-12 indicators for the cross-cutting dimensions relied solely on evaluation

findings and were too general. This has not produced enough evidence to fully assess these dimensions. If COL intends to monitor cross-cutting dimensions it needs to define indicators that relate more closely to what it is trying to achieve in, and with, them.

509. COL’s outcome statements tend to be very generic. In some cases they are so

unspecific that they could be rolled on from TYP to TYP. This is sometimes difficult to avoid, especially where interventions in an Initiative are diverse as in TE for example. However if COL accepts a recommendation of this evaluation that it focuses on a small number of substantial interventions, some of its outcome statements could be built around the results expected from this work.

510. The process for the 2012-15 plan is almost complete. It began in 2010 and an

opportunity to involve stakeholders was provided by a workshop at the PCF6. For most of the Initiatives the results, indicators and targets have been through several iterations. This has been a much better process and should lead to more meaningful results statements and indicators, and more closely calibrated targets. It should also ensure more ownership.

511. Meaningful and realistic outcome statements and indicators at the level of diverse

organisations are difficult to define. COL is now taking the process very seriously and is improving its methodology.

Planning for results

68

512. Although the evaluator has not had direct access to COL’s planning processes this year or in previous evaluations, it is clear that COL’s TYP plans are taken very seriously, are based on wide consultations and are finalised at the same time as closure on the new Logic Model. They also benefit from dialogue during the evaluation process for the previous TYP.

513. COL’s TYP is not a detailed planning instrument in itself. It is primarily a

communication vehicle. It illustrates what COL intends to do and achieve rather than mapping it out in detail. This is widely appreciated by outside stakeholders; few have the time and inclination to wade through dense accounts of proposed operations.

514. However, to translate the expected strategic results into planned operations, other

instruments are required. For this COL uses annual logical frameworks (logframes). They are constructed at the Activity level and are also summarised in Initiative logframes. The logframes do not follow the standard grid design, but include all the normal elements.

515. The Activity logframes are meant to contain:

A description of the Activity. The Initiative outcome statements and indicators. The annual Activity intended outputs and indicators. Means of verification for the Activity indicators. Some reference to baselines. Risks and assumptions for the Activity. Proposed actions for the Activity. Budget.

516. Logframes were introduced comprehensively in the current TYP and a certain

amount of acclimatisation was to be expected. On the whole the logframes are fully completed and there seems to be a reasonable degree of alignment between actions, outputs and outcomes. Output statements and indicators are generally more transparent than at the outcome level.

517. Means of verification are generally rather imprecise, reflecting gaps in COL’s

monitoring processes. Baselines are mostly defined inappropriately and are of little or no use. This section should either be improved or scrapped.

518. CEMCA’s logframes are not consistently aligned with COL’s outcomes. This may not

have been a negative factor in the current TYP period. However it is recommended that CEMCA receives support to integrate fully with COL’s RBM when the incoming director is in place.

519. Logframes are often a target for resistance in organisations – complaints, for

example, that they are inflexible and imply unrealistic linearity. ESs were asked about COIL’s use of logframes and other components of RBM during the evaluation. The absence of serious complaints about them from COL staff is an encouraging sign. Several staff say that they find the logframes useful particularly for planning – providing focus and direction. The elements of the logframes below the outcome level are permitted to change from year to year – or even in-year. This is seen as helpful.

520. A weakness of the logframes in most Initiatives is that they are built around

Activities that are defined generically rather than around a coherent set of interventions. This reflects the fact that the majority of COL’s Initiatives are diverse and are not defined by a single major project or tightly integrated cluster of work.

69

521. The Activity logframes for Skills Development are an example. They are defined simply in terms of four of COL’s five “strategies”: planning and policy development, capacity building, models and materials. Not only does this communicate very little about what COL is intending, it also means that “projects” like the FSD for CAPA do not benefit from the mainstream logframe process. The project’s actions, outputs and indicators have to be disaggregated and distributed among the different Activity logframes. Most of COL’s interventions consist of two or more of the “strategies”, so defining Activities around them leads to this disconnect.

522. This makes little sense and is an impediment to effective planning and meaningful

M&E and reporting. Several of the external evaluators said they found COL’s Activity logframes of little use in their assignments. Of course it is possible to build alternative logframes or other planning and monitoring instruments around project-type interventions, but this is wasteful duplication and reduces the ownership of the mainstream logframes.

523. Most of COL’s Activities and their logframes suffer from this disconnect. Exceptions

are those for the CFS and OERs for Open Schools projects, RETRIDAL, aspects of VUSSC, some of CEMCA’s work, and the L3F work as a whole.

524. Some staff believe that the rationale for this way of defining Activities is that COL

has to report to the Board on spending and results against its five “strategies”. There appear to be misunderstandings about this which the evaluator was not able to resolve. This should be sorted out urgently before the next TYP Activities are set in stone. If it is necessary to report on the separate “strategies” – and this evaluator does not see an overriding need for this – there should be an alternative, uncomplicated, way of doing this. The tail should not wag the dog.

525. The Initiative logframes appear merely to summarise their Activity level

equivalents. As such they do not appear to add much value. However what would almost certainly enhance the planning of COL’s work would be short strategic plans for each Initiative. These should be constructed for the life of the TYP – or even longer in some cases. They should be built around a logic model or theory of change (ToC) that links key actions and outputs for each “Activity” to the Initiative outcome. There would be a strong case for including intermediate outcomes in the models as well. The construction of the logic models/ToCs would best be done collaboratively both within COL and with partners. For both there should be clear benefits for better understanding and scope for synergies with their own plans.

526. The plans should include a proper narrative section about the rationale for the

Initiative and its Activities and how COL is intending to achieve the outcomes. These plans would be summarised in the logframes. Both the plan and the logframes would be reviewed annually.

Monitoring 527. COL monitors its direct actions and outputs consistently in terms of when and

where they happen. There is not consistent monitoring of COL’s reach – details of whom COL and its consultants engage with in workshops and other types of action, nor of the quality of outputs either through real-time user surveys or self-assessment against pre-determined standards. This is not always strictly necessary, but COL does not seem to have any criteria for when it is necessary, or guidance on good practice. Action and output monitoring need the cooperation of consultants and programme partners.

528. There is very little outcome monitoring by COL or its partners, apart from tracking

large milestones like the launching of a new course. There is rarely follow-up with

70

the end-user or even the mediators – e.g. teaching staff and materials writers. L3F is the main exception: it has built pre-planned outcome monitoring, since its inception, into at least one of its projects in each of the three TYP periods.

529. Outcomes are the most important things to monitor for strategic learning. COL’s

rationale is achieving outcomes - not actions and outputs. The development of outcome monitoring, in collaboration with partners, should be the main item on COL’s agenda for improving its RBM system in the next TYP.

530. Up to now COL has “outsourced” outcome monitoring to evaluators. This is not ideal

for two reasons. It does not encourage ownership by partners of the data and the knowledge

that comes from it. Unless the external evaluation is real-time9 – concurrent with the project from

beginning to end - monitoring will normally be designed and implemented retroactively, missing opportunities for data capture early in the intervention.

531. The recommended improvements in monitoring will not happen automatically –

they will require planning, capacity building and recurrent resourcing. To monitor outcomes, COL will have to reach out with surveys, interviews, focus groups and participatory methods. This takes time, expertise and money. Many organisations shy away from this. COL must not. Part of COL’s rationale is to be a source of knowledge. A most important type of knowledge is whether application of its models, and other interventions, are working, and what effects – including unplanned ones - they are having.

532. One way to institutionalise outcome and other monitoring is to create an M&E plan

for the Initiative and Activity at the same time as the general plans are put together. This will only work if the Activities are coherent and the overall plan meaningful (see comments on logframes above). COL should institutionalise M&E planning with the implementation of the new TYP.

Integrating evaluation 533. COL makes considerable use of external evaluation. In the last two TYPs, every ES

identified, at or before the inception of the Three Year Plan, a piece of work to submit to an evaluation.

534. In this TYP, this has mostly involved a two or even three stage process that COL

refers to as a longitudinal study. As all of these stages had to be completed by the end of the second year of the triennium, the longitudinality was somewhat truncated, although in one or two cases – notably OERs for Open Schools and L3F, there was proper baseline research which added considerable value to the summative evaluation. The mid-term formative evaluation for the OERs for Open Schools project also made a pivotal contribution to the project.

535. Of COL’s 10 external evaluations in 2011, about half were not very useful for the

overall TYP evaluation. In this evaluator’s (inevitably subjective) opinion, several were insufficiently strategic, poorly designed or in one or two cases simply unreliable. One evaluation based its conclusions on a survey response of around 5%. Apart from the studies mentioned in the previous paragraph, the evaluations of Phukusi La Moyo (Healthy Communities) and - with some unavoidable limitations - ICT4E in the Caribbean (Integrating eLearning) were also of high quality. Two or three others were of acceptable quality.

9 Real-time external evaluation is proving to be a very effective form of the discipline, but is expensive.

71

536. The main lesson from this is that COL needs to consistently raise the level of its

capability for commissioning evaluations. This implies Being a pro-active partner in evaluation design through terms of reference

and dialogue at the inception stage. Improving the recruitment of evaluators by widening the pool and having

sound selection criteria and processes. 537. COL staff with experience in commissioning effective evaluations should be able to

support those who have had less experience and success. 538. Although there is a strong rationale for commissioning an evaluation in each

Initiative, this would be less strong if effective outcome monitoring were put in place throughout. Outcome monitoring need not lead to an external evaluation in every case. The data from monitoring can be used internally in quasi-evaluative ways. COL should look more strategically at evaluations, and decisions should be shared. There is often greater value in thematic, cross-cutting evaluations which tend to continue to have relevance after specific interventions have been completed.

539. Candidates for thematic evaluation in the next 2-3 years could include:

COL’s work with networks and communities – what value do they add? What works and in what circumstances?

COL’s use of models, both for replication and up-scaling. COL’s work with OERs. The evaluation of COL’s work in 2-3 countries. COL’s work with CR across the board, L3F across the board.

540. COL should also commission a comprehensive evaluation of its Three Year Plan late

in 2014. If it improves its monitoring and commissions a good number of effective evaluations, there is no reason why the same formula as this year should not be applied. If these things are not in place in 2014, then the evaluator should have direct engagement with a sample of interventions, as in 2005/06. COL should look for a new evaluator, one who is reasonably familiar with ODL but has not had much contact with COL in the past.

Analysing, reflecting and incorporating lessons 541. Both monitoring and evaluation should inform decision-making, at operational and

strategic levels. COLs Initiative level evaluations in the final year are timed to feed into decisions for the next Three Year Plan. In several cases, good use of them is also being made for operational improvement.

542. As has already been said, little of COL’s formal monitoring is useful for strategic

decision-making. This needs to change. However, recent Programme Update Meetings (PUM), which are held three times a year, are said to have been very good for sharing information and the creation of knowledge. This is partly because the bare statistics are enriched with anecdotes and other qualitative data that staff provide from their own experience. This is no substitute for systematic outcome monitoring, but at least there seems to be demand for meaningful data among COL staff. Demand is a key factor in the sustainability of any data system.

72

Reporting to stakeholders 543. Reporting is valuable for accountability, stakeholders’ own learning, and advocacy.

But reporting should never be a one-way process and should always provide a feedback path. Feedback paths are in place with various stakeholder groups: particularly other staff (e.g. at PUM as mentioned above) and senior management, the Board of Governors, country Focal Points and other partners.

544. The big question is what value do stakeholders get from COL’s reporting and how

valuable is the feedback for COL? If COL is reporting about the important things – outcomes, output quality, effective processes – then there is a good chance of generating a virtuous cycle of knowledge creation. If the reporting is just about activities and bare output numbers, there is little or no chance of setting that virtuous cycle in motion. Activity and bare output reporting leaves the “so what?” question in the air.

545. With COL, the virtuous cycle is not consistently in motion. There are sometimes

short bursts of momentum. COL reports richly on its work in its four monthly Connections newsletter. This is a valuable source of information on the real stories of what COL does on the ground and, anecdotally at least, some of the effects. The trouble with corporate communications like these is they are not necessarily accepted as representative, particularly by stakeholders who are required to be constructively critical of the organisation. They screen out the “bad stories” or are suspected of doing so.

546. If COL could tell stories in this way through a transparently balanced process, based

on systematic monitoring at all levels of the value chain (from inputs to outcomes or even impact), it would be in the vanguard of qualitative results reporting – an elusive but attainable objective and one that is highly relevant to COL’s role as a leader and custodian of innovation.

547. Apart from this ambitious objective COL could improve easily on the present

position with a few changes. Include progress reports – with short narratives - on the TYP outcome

indicators in its annual reports to the Board. For some reason this is not done at present. The reporting is on output indicators from the logframes.

Streamline the reporting on logframes. Little use seems to be made of the quarterly reports. ESs report three times a year at PUMs and twice a year in personal appraisals with their line manager. Relevant data for these events need to be available. Aside from these needs, however, it may be sufficient to report on logframes annually. Less frequent reporting is likely to lead to more strategic reflection on, and use of, the data.

Consider finding another term for Activities. It is confusing for outsiders –especially those used to logframe terminology – to find solid areas of work defined in this way. Activities in logframes are subsidiary to outputs.

Knowledge Management 548. COL is a knowledge-based organisation. It acts as a knowledge hub and service

provider for several overlapping communities with an interest in ODL. It also generates knowledge from its own work, partly through research, evaluation and good practice toolkits. It does not aspire however to rival academic institutions in research.

549. COL ‘s Knowledge Management (KM) unit supports both aspects of COL as a

knowledge-based organisation. It also manages ICT infrastructure for COL staff and to some extent its programmes. This includes a Moodle LMS which is used primarily

73

by eLIO. It has also been used by COL’s sector programmes for their interventions, although the protocols for this are still being worked out.

550. The KM unit has recently re-strategised and is planning an ambitious and diverse

array of new services including some that reach out to customers rather than reside on its website. In line with COL’s commitment to OERs, it is collaborating with ComSec to create portal access to the rapidly expanding stock of OERs produced in developing countries.

551. COL’s KM facility has great potential to work closely with the eLearning Initiative

and with COL’s other ICT-mediated interventions. These can be genuine collaborations. To optimise them, there needs to be regular dialogue between KM and the ESs. This may need systematising if opportunities are not to be missed.

Specialist staff succession 552. In every triennium one or more ES leaves and is replaced by another. Sometimes

there is a smooth handover, particularly if the new ES has been a consultant or programme partner with the outgoing one. At other times there is little or no communication between the outgoing and incoming ES.

553. The latter has two possible consequences. The incoming ES has difficulty picking up

the thread of work under the previous ES. She or he takes time to find their feet and momentum is lost. More serious is the possibility that the new ES will feel no sense of ownership of the previous ES’s work and decide to plough an entirely new furrow. This may be acceptable or even desirable, but it should not happen by default.

554. When an ES leaves, COL should have a clear medium-term framework for the area

to which it intends to recruit a successor. This implies having a clear profile of the person it should be looking for; and providing an effective induction into this framework for the new ES. Of course the recruitment market may not permit the perfect implementation of this approach – or it may trigger modifications based on new opportunities. COL recruits people with deep background in certain aspects of ODL, and should not try to force a square peg into a round hole.

555. The risk of problems arising as a result of a succession gap in some circumstances

could justify the funding of an overlap between outgoing and incoming ESs, or setting up a short period of cooperation downstream of the appointment.

74

Conclusions and Lessons 556. The Table below summarises COL’s progress against its TYP performance indicators Significantly

exceeded Met or likely to be met

Not likely to be met

Degree of challenge in PI

targets Open Schooling 3 1 High/medium

challenge Teacher Education 1 1 1 Medium/low

challenge Higher Education 2 1 Medium/low

challenge VUSSC 3 Medium

challenge Skills Development 3 Medium/low

challenge Learning for Farming 2 1 High/medium

challenge Healthy Communities 2 High/medium

challenge Integrating eLearning 2 1 High/medium

challenge 557. Progress against COL’s TYP PI targets tell us more about the organisation - and the

outcomes it contributes to - than three years ago; but still not as much as it should. This is partly because the degree of challenge in the targets varies considerably, and partly because most of the outcomes being measured are somewhat generic and de-contextualised. This points to improvements that need to be made in the next planning phase of COL’s RBM framework.

558. The general picture from these results is one of an organisation meeting the

majority of its medium and high challenge targets, as well as all its low ones. It is difficult to generalise but this is not untypical of an organisation effectively pursuing its mandate.

559. On the face of it, no Initiative is seriously underperforming; and none is stellar. But

corporate PIs are a large scale map of the terrain, and as usual the devil is in the detail. Closer scrutiny is needed to say more about COL’s performance and what lies behind it.

Open Schooling 560. COL is an important and valued partner in the open schooling sector in the

developing Commonwealth. In this triennium it has undertaken an ambitious portfolio of interventions including multi-faceted support to new and aspiring open schools; the first institutionalised networking of open schools through COMOSA and an on-line community; and a challenging project to build capacity for OER production in six countries.

561. The externally funded project is the area that was highlighted by an external

evaluation. It has not fully achieved its objectives in this project. It was ambitious – some informants have said over-ambitious. It is also evident that the more formal project mode that was demanded here is not COL’s normal way of operating.

75

562. COL has learnt from the challenges it faced, has responded flexibly and expects to

have achieved one of the two objectives – completed high quality course material - by June 2012.

563. Apart from the lessons from the OER project that COL has already internalised, this

evaluation has no significant recommendations to make about this Initiative except to continue on the same trajectory. It is putting a lot of emphasis on OERs and this seems entirely appropriate for its programme partners’ needs in this subsector.

Teacher Education 564. This Initiative addresses a wide subsector, very important for development. The

subsector’s experience of ODL is maturing, but large scale solutions are proving elusive. It is not difficult for COL to find footholds in this subsector; but it has not always found it easy to obtain leverage, particularly with strategic partners.

565. The CFT project has been well conceived and well executed. There is still work to do

but it is on course to achieve its objectives in most participating countries, and exceed them in the front-runners. Its use of a full-time project manager is unusual for COL, but has worked well and has freed up the ES to pursue other work.

566. COL’s other work in this Initiative has been valuable, if unspectacular. Better M&E

at the outcome level would give COL more of a basis for deciding what works in this subsector. It needs to take a hard look, before the end of the next triennium, at how to optimise its contribution.

Higher Education 567. This is a very wide and diverse subsector – and one where ODL is the most mature.

Over the last two triennia, COL has increasingly found it hard to find a niche that is clearly recognised and valued by a critical mass of stakeholders. This is partly a performance issue. But it may also be a question of life-cycle. HE – unlike other education subsectors - may have outgrown COL’s focus and ways of working, despite apparent demand from COL’s country interlocutor institutions10.

568. COL’s most valued contribution with wide application is the self-administered quality

assurance tool known as COL RIM. There may be a rationale for further development of this tool to provide an institutionalised framework for quality assurance. Before COL moves ahead with this it should consult widely because there appear to be other international players in this area.

569. With the exception of COL RIM, COL’s work over the last few years in HE has not

innovated and has mostly been slow to bear fruit. Unless it identifies and delivers 1-2 products in the next 2-3 years which are widely recognised as making a major contribution to the subsector, the most sensible course may be to wind down HE as a separate Initiative and contribute more flexibly to the subsector where a clear rationale presents itself. This is tied in with the discussion below about how COL configures its work.

570. This Initiative oversees RETRIDAL which has achieved very little in this triennium.

COL is making a great effort to revive the institution, but if it begins to feel consistently like pushing water uphill, it needs to devise an exit strategy.

10 These are mostly responsible for tertiary formal education, so this continuing demand is not entirely surprising.

76

VUSSC 571. VUSSC is COL’s largest ever coherent programme in funding terms. It has mainly

focused on the production of course material for TVET which is seen as highly relevant for today’s challenges. It is helping small countries, most of which have struggled to get on the TVET ladder at all.

572. VUSSC’s geographical coverage, and the apparent commitment of member

countries, are impressive. This coverage is very important for COL which has an obligation to support a wide range of countries in each triennium.

573. At outcome level the results are less impressive. Progress in the production of

courses in the member countries has been slower than anticipated. The main stumbling block has been the translation of bootcamp material into actual course modules.

574. This does not happen automatically. It takes a considerable amount of expertise

and experience to traverse this “last mile”. The bootcamp is a capacity-building vehicle, but it does not deliver that “last mile” support. Most VUSSC countries do not have much expertise and experience for building courses around OER material, let alone whole curricula.

575. To overcome this problem, VUSSC is now concentrating on producing complete

courses at the bootcamps. This is already leading to take-up, so it seems like the right move for now.

576. VUSSC has also completed the construction of the TQF which is designed to

promote educational and labour mobility. 577. Systematic outcome monitoring within VUSSC is essential if countries’ differing

experience of utilisation of VUSSC outputs is to be tracked and fully understood. This will enable VUSSC strategies to be more regularly adjusted to meet countries’ needs.

578. VUSSC clearly has great stamina. Participants have said they feel part of a grand

enterprise. To have maintained this momentum over 6-7 years is a considerable achievement. Few organisations have the staying power to have managed this process in the way that COL has done.

579. COL needs to stay with VUSSC but accelerate the transfer of ownership while

reviewing – on the basis of better evidence – the mechanisms, almost certainly introducing more diversity to the model. It also urgently needs to explore, with its partners, sustainable funding mechanisms.

Skills Development 580. COL’s work in skills development is highly relevant to 21st century challenges. This

Initiative has focused more on bigger countries, which, despite their size, often lack experience in skills development, particularly for the informal sector. This is the niche COL is developing and it is making good progress.

581. Its flagship project - FSD for CAPA – is making headway, particularly in Kenya

where four institutions are participating. In Kenya there is already interest at the

77

policy level, which is very important to engage at some stage in the process, even though the project concept is more bottom-up.

582. There are signs FSD for CAPA could provide a model for other institutions and

regions. But it is early days and the terrain is difficult. It needs to consolidate, monitor, reflect and consult before attempting to branch out. There are probably another 2-3 years of hard work in the project before that.

583. If there is to be an expansion of FSD for CAPA, COL should consider appointing a

full-time project manager. This would, amongst other things, facilitate the management of the necessary improvements in monitoring. So far this has been thin on the ground with heavy reliance on an external evaluator.

Learning for Farming 584. L3F is one of COL’s most innovative programmes. It is innovation for a purpose: to

identify a robust and transferable model for linking credit flows – so important for struggling farmers – to the acquisition of usable knowledge.

585. COL has successfully tested its core L3F model in South Asia. It has shown that it

can – in certain circumstances - make a significance difference to people’s livelihoods.

586. Furthermore, by empowering people and reinforcing cognitive social capital, it

seems to be able to underpin these advances in farming enterprise, building resilience and sustainability.

587. There are now indications that, with some adaptation, L3F may make the transition

to other regions. 588. COL is fully aware that time is running out for the micro field testing phase of L3F

and that it needs to demonstrate that it can rapidly scale up without the intensive care of COL and its development partners. Banks have been identified as promising vehicles for this in India. In other regions, there may have to be different vehicles.

589. COL now needs to work hard at finding advocacy platforms for L3F that don’t just

broadcast its messages but also engage new types of partners for up-scaling and diversifying. L3F is old enough to leave the nest and learn to fly. There is a need to demonstrate with clarity what the advantages are. This involves evidence through evaluation but also simpler communications.

Healthy Communities 590. In this Initiative, COL has developed, and is supporting in an impressive number of

locations, another approach to participatory learning. The learning is mainly about health and is facilitated through CR, sometimes complemented by other media.

591. The wide reach comes from an arms length approach, although the numbers

reached at present are relatively few in any location – as typical of the modelling phase of a COL Initiative.

592. Apart from the benefit to the people who directly participate in the programmes

that COL assists – perhaps around 100,000 in total – the value of COL’s work is in the refinement of the model and its testing in a number of different social, cultural, economic and political contexts.

78

593. As with other models, COL needs to validate it, refine it, and if appropriate diversify it, through the capture (through M&E) and use of evidence. The evidence strategy is in place in one intervention: Phukusi La Moyo in Malwai. This needs to be complemented with similar data from a small number of other programmes in different contexts.

594. COL would then have a valuable tool for wider advocacy for its model with a view to

building international strategic partnerships for multiple replication. COL’s work should move further upstream in 2-3 years, by focusing more on strategic partnerships and supporting the development of remote – through toolkits, eLearning etc - capacity-building of local partners. There should be closer liaison with CEMCA’s CR work.

Integrating eLearning 595. Integrating eLearning is another of COL’s broad-brief Initiatives, but is not sector-

based. It is concerned with promoting and supporting the use of ICT in teaching and learning in both formal and non-formal contexts and at all levels.

596. This Initiative was dominated in the previous triennium by the promotion of

WikiEducator and the Learning for Content courses. The transfer of management of WikiEducator to an institution in New Zealand left a space that was not clearly defined. It took time for COL to identify a new direction to follow. Once this was done, however, the work gathered momentum very quickly. This was helped by two factors: A strategic partnership with ComSec and Microsoft came together and took off

very quickly with work in the Caribbean. The use of pre-existing models and materials that needed adapting and some

re-purposing but not re-inventing. 597. Both of these strategies by COL have paid off. The ICT4E work in the Caribbean is

the largest and most coherent set of interventions in this Initiative. It has already led to some solid early outcomes in three countries, influencing policy and supporting the countries in the adoption of a competency framework for integrating ICT in school teaching.

598. COL’s partners want to push on in the Caribbean and begin similar work in the

Pacific. The latter may be premature without consolidation and research on outcomes. But the apparent effectiveness of the partnership is an important development for COL. Keeping strategic partnerships together however should not become an objective that overrides the needs of programme partners.

599. Other work in the Initiative has been lower profile but may also be producing good

outcomes. It is important that the stories are told in an accessible way to help COL can build on their achievements in the next triennium.

CEMCA 600. CEMCA, in 3-4 years, has become a valued strategic partner with the Government

of India in the promotion of and support for community radio. This is a remarkable achievement, although understandable when the background of CEMCA’s current director is taken into account.

601. CEMCA’s direct work with CR start-ups has equipped around 100 organisations to

apply for licences and begin to broadcast. CEMCA and the Government have big plans for the future of their cooperation in CR which would see CEMCA moving to an even more strategic role.

79

602. There are two issues – one short term and one long term – that COL needs to

address. The short term issue is how to ensure the sustainability of the CR work when the current director retires in 2012. He is so closely identified with the work that there is a risk of a weakening of the momentum.

603. The second issue is a potential longer term opportunity. How can COL leverage

CEMCA’s experience and reputation in CR in other countries? Institutional frameworks for CR may be country-specific, but the rationale for CR, and challenges faced at the operational level are similar in different contexts. CEMCA has made some headway in Bangladesh, but on a small scale so far.

604. CEMCA’s work in other areas does not appear to be suffering from the focus on CR

and there is evidence of genuine innovation. 605. CEMCA is meant to be an integral part of COL, not a separate agency. Much

progress has been made in this over the last 2-3 years, although in terms of its integration in COL’s RBM system there are still improvements to be made.

In general 606. COL has a wide range of stakeholders – for its size, more than most organisations.

Stakeholders have different interests in and expectations of COL. These range from institutions and some governments that are content with a small amounts of conventional technical assistance; to stakeholders that expect COL to be a leader in innovation in ODL. Stakeholders that look to COL for innovation range from those that are interested in innovation per se, to those that expect COL also to apply the innovation at scale, producing real impact on today’s development challenges.

607. Managing this range of expectations is a tough enough task. Delivering on

expectations to achieve impact at scale – bearing in mind COL’s size – is extremely tough. Yet this is what COL claims to be able to do, and how it intends to be evaluated. This evaluation takes COL at its word and looks for evidence that it is making this type of difference and if not, what it might do to enhance its capacity.

608. To this evaluator, COL looks and feels like a different organisation from the one first

encountered in 2005. In all the teams there is a palpable striving to be relevant and create deep and long-term effects. There was evidence of this in the 2003-06 triennium, but it was certainly not universal. Progress was made in the 2006-09 and more since then. This does not always make COL a comfortable place to work, but it is transforming the way COL identifies, and carries out its work.

609. In particular COL understands the need to build sustainability into its interventions.

Sustainability was referenced in 2003-06 but was not a dominant concern. 610. This does not mean that COL has all the answers to questions about relevance and

sustainability, but at least it is actively looking for them. Most of COL’s interventions are medium or even long-term (L3F and VUSSC). Several are multi-faceted, like FSD and CFS, in recognition of the fact that sustainable progress often needs all stakeholders pulling together.

611. COL acknowledged in the 2003-06 triennium that it had to move away from the

one-off workshop as a typical product, but the transition was still work in progress in 2006. Now it is difficult to find a workshop or similar event that is not linked to a longer-term set of activities with a wider outcome in sight. There is altogether more focus and this is not reflected simply in the reduction of sectors from three to two.

80

612. COL’s RBM system has played a role in this. The sheer discipline of RBM and the constant reinforcement of messages about outcomes and sustainability have probably been the main drivers. COL has not yet found a way fully to reap the potential benefits from RBM of strategic learning, but this is not unusual for the state of maturity of COL’s system. The important thing is that it is on the right path to a higher degree of utilisation. This evaluation has looked in detail at COL’s application of RBM and makes a number of recommendations for improvement. If implemented successfully, these could turn COL into a model RBM operator. It has two assets in its favour: its small size and staff who mostly believe in the value of RBM.

613. Apart from RBM, the change has also been brought about through a complete

renewal of ES staff. These staff were recruited to an organisation with expectations that are radically different from 2003. Most ESs also had previous experience of project planning for outcomes and of monitoring and evaluation, if not of strategic RBM.

614. COL’s external environment is also different from 2003, with more competition for

funding and more vocal demands for evidence of results. COL seems to have convinced outside stakeholders that at least it has the will and capacity to provide adequate evidence. Australia’s new commitment to COL, in part at least, signifies confidence in its results-orientation. The doubling of the number of voluntary contributions since 2003 reflects confidence in the profile that COL presents.

615. It is important to stress that, with this transition, COL has not lost its disposition

and ability to be needs-driven and to listen to particular clients. There is always a risk - with pre-set strategic goals, more projectised working, more strategic partnerships and a drive to innovate – of becoming inflexible and insensitive to local needs and inflexible. Getting the balance right is very difficult but COL seems currently to be achieving this.

Models: a core competence 616. Probably the clearest manifestation of COL’s transition to more outcomes-oriented

approaches is its convergence on the development and application of innovative models for ODL. A model, in COL’s terms, is a coherent set of structures and processes which has been shown to achieve similar results in more than one context. Models can range from broad approaches – such as L3F or the blended application of CR for achieving health-related outcomes – to more specific instruments such as COL RIM or the COL ID template.

617. Typically the models are based on concepts developed and often tried out

elsewhere sometimes by newly recruited specialist staff themselves. COL’s participatory work with CR, and also the FSD and CFT projects are examples of this. With L3F, exceptionally, COL itself conceived and gave birth to the concept through painstaking research and stakeholder involvement.

618. In its most comprehensive form, COL’s use of models would be to:

Apply its own thinking and experience – and occasionally research - in preparing the concept or approach for trialling.

Trial it in more than one institutional and geographical context – often in two or more countries or even regions.

Monitor, and assess or evaluate the trial and recycle the learning from this into a new version of the model.

Develop a strategy for replication or up-scaling. (Replication is where a large-scale model - such as the FSD for CAPA, or ICT4E in the Caribbean - is applied on a similar scale in a different context. Up-scaling is where a small-scale

81

model - such as the early iterations of L3F or participatory CR - is mobilised on a much bigger scale through multiplier or cascading processes.) A strategy would typically include documentation with compelling evidence for advocacy; support materials such as toolkits; and, particularly for up-scaling, new partnerships with wide-scoping institutions.

619. In practice, COL has not been able to deliver all of these elements, at least not in

full. The process is often messier, iterative, and incomplete. This is partly because COL has only crystallised its thinking about models in the last 3-4 years. It has not had time to work out fully what it should, and can feasibly, do at all stages of model development and mobilisation.

620. But COL is clearer in this triennium that the development and mobilisation of

models are a core competence. COL’s advantage is that it has the ability both to identify innovative concepts and approaches from a wide external environment, and to find contexts where they can be applied. It achieves this because it is regularly networking in depth with a very wide range of people and organisations working with teaching and learning – in research, formal and non-formal educational delivery, and also in adjacent areas such as social transformation. It is also sharing knowledge internally more than in earlier years, even if cross-Initiative working is still the exception.

621. Models would seem to be the most tangible contribution COL can make to

supporting the mobilisation of ODL for development. COL’s networking, convening power and advocacy are powerful assets, but their results are usually not so tangible.

622. To fully realise the potential of models COL needs to make improvements in the last

two bullets in paragraph 618. It needs to build intensive outcome monitoring and evaluation as standard

into its planning for interventions with models. This will require resources – there is no way round that.

It needs to put more time into researching, discussing, designing and implementing strategies for replication and up-scaling.

Scale of interventions 623. Does this imply longer and deeper commitments to a particular type of

intervention? In some cases, yes. If it is to promote a model for replication or up-scaling it must be sure that the model is sound. It can only do this by continuing to work with the model – and capturing evidence through monitoring and evaluation at both implementation and outcome levels – until that judgment can be safely made. This would have resourcing implications. With a static budget, there would be less funding for other activity.

624. Taking into account the balance of stakeholder opinion, the logic of the extension of

COL’s use of models, and the learning about what works that will come from improvements in outcome M&E, this evaluation recommends that COL focuses even more on a small number of substantial medium to long-term interventions. It should consider devoting perhaps 75% of its programme budget to 4-6 major interventions at any one time. These interventions should be transnational, ensuring that across its interventions all regions get a fair share. However COL should not pursue wide geographical coverage as an overriding priority if it is not compatible with effectiveness, impact and sustainability.

82

625. Should these major interventions generally benefit from external funds? Not necessarily; and the pursuit of external funding should not be main rationale for the identification and design of interventions. COL is clear about this.

626. COL is not always comfortable working with large scale projects, with pre-

determined outputs and outcomes and tight deadlines. It is used to more flexible ways of operating, less full-on and less linear interventions, and emergent definitions of outputs and outcomes. There is a substantial body of opinion that this is a more effective more of operating in development. COL does not have a large pool of programme funds, but this is often not a problem as COL’s expertise, ideas and brokerage are what gives it its edge. Large funding packages can work against local commitment, ownership, and sustainability.

627. However COL should also not shy away from external funding opportunities,

particularly where it is already confident in a model that needs to be taken to a bigger stage. COL has shown that it can operate effectively in an external funding environment, although it should seriously consider appointing project managers in these circumstances if the scale justifies it.

628. Working with a small number of larger-scale interventions should still leave room

for exploratory work, recognising that much of this may not take root. Exploratory work needs to have a clear deadline to prove itself and not drift on.

629. It should also leave room for some smaller scale interventions, although there

should not be diversity for its own sake. All projects must all relate strongly to the Initiative outcome. COL has a unique ability in ODL. Its inter-governmental status, long-term reputation for expertise and professionalism, and access to wider stages and important national and international actors, enable it to amplify and legitimise the voices of less powerful organisations that are innovating in ODL. COL can support these organisations with relatively effort. It should spare time and resources for this.

Configuring COL’s programme work 630. A focus on 4-6 major interventions suggests that COL should reduce the number of

Initiatives. There is no intrinsic reason why COL should remain wedded to the notion of 7-8 separate Initiatives. Although it has tended to retain specialist staff, each with responsibility for a different subsector, this also is not necessarily a “given” for COL. In particular this evaluation is recommending that it keeps under review its commitment to an Initiative devoted entirely to HE.

631. A strong message from a wide range of informants in this evaluation is to move

more resources to skills development. There is a consensus about the unmet need in this area in the developing Commonwealth – and in most countries around the world for that matter. COL is seen to be at the cutting edge of the mobilisation of ODL for skills development.

632. An expansion of work in skills development would have implications for several of

COL’s existing Initiatives. This may require a rethinking of the way COL configures its Initiatives. It is very important that COL’s work in this area benefits from coordination and collaboration across the organisation. COL no longer suffers from “silos”, but it could miss opportunities through inappropriate segmentation.

633. There is a strong suggestion from some stakeholders that COL should identify and

focus on “root” solutions: work which – with minimal adaptation – can support outcomes in more than one subsector. Quality assurance is a well-established root solution; and COL should consider pooling its resources and experience in QA for

83

potential synergies across educational subsectors and beyond into informal learning.

634. OERs are the obvious emerging root area. COL is working with OERs in several

Initiatives and in KM. It is not immediately clear how much collaboration there is within COL on OERs, but there are likely to be advantages in this. It is not inconceivable that in three years’ time COL has a team that works with OERs across the current Initiative boundaries.

635. Policy is another root solution. Most of COL’s interventions include a policy

promotion component for sustainability. While this needs to be tailor-made at the level of institutions, national policy advocacy for ODL is a slower, more detached process. It may be better pursued outside the boundaries of specific Initiatives.

636. Although not a root solution in the same sense of the concept, knowledge

generation for stakeholders about what works in ODL through M&E could develop into a core competence. Approaches are similar in most subsectors, with an important variation where participatory methods are appropriate. M&E is certainly an area in demand and where competencies are lacking among partners. If COL ups its game in this area, it could share both the learning from, and the expertise in, M&E. At the very least, COL should not reinvent the wheel with M&E in each of its Initiatives.

637. There is more that could be said about COL’s ways of working – for example its

approaches to networking and relationships with programme partners. These were highly appreciated by stakeholders in the 2003-06 evaluation. COL has retained these strengths and in some cases improved on them. For example it clearly has a more energetic stakeholder engagement at country level though its Focal Point system. COL asserts that this has been a real advancement in the last few years. However it is outside the scope of this evaluation to examine this aspect of COL. For this, the findings of the 2010 Stakeholder Survey are still relevant.

84

Recommendations (Principal recommendations are underlined) 638. Open Schooling should continue on its current trajectory. Its emphasis on OERs is

entirely appropriate for its programme partners’ needs in this subsector. 639. COL needs to take a hard look, before the end of the next triennium, at how to

optimise its contribution in TE. 640. COL should keep its work in HE under close review. Unless it identifies and delivers

interventions in the next 2-3 years which are widely recognised as making a major contribution to the subsector, the most sensible course may be to wind down HE as a separate Initiative and contribute in other ways to the subsector where a clear rationale presents itself.

641. If COL’s efforts to revive RETRIDAL begin to feel consistently like pushing water

uphill, it needs to devise an exit strategy. 642. COL needs to stay with VUSSC but accelerate the transfer of ownership while

reviewing – on the basis of better evidence – the mechanisms, almost certainly introducing more diversity to the model. It also urgently needs to explore sustainable funding mechanisms.

643. COL needs to consolidate FSD for CAPA, and to monitor, reflect and consult before

attempting to branch out. There are probably another 2-3 years of hard work in the project before that. If there is to be an expansion, COL should consider appointing a full-time project manager.

644. COL needs to demonstrate that it can rapidly scale up L3F without the intensive

care provided by the development partners. COL needs to work hard at finding advocacy platforms for L3F that don’t just broadcast its messages but also engage new types of partners for up-scaling and diversifying. There is a need to demonstrate with clarity what the advantages are. This involves evidence through evaluation but also simpler communications.

645. In 2-3 years time it may be worth conducting a review of the L3F experience

globally, irrespective of its performance by then. Both success and failures will provide important learning for continuing efforts to improve the well being of poor rural communities.

646. As with other models, COL needs to validate its Healthy Communities work, through

the capture (by M&E) and use of evidence, and if appropriate further refine and diversify it.

647. COL’s work in Healthy Communities should move further upstream in 2-3 years, by

focusing on strategic partnership and supporting the development of remote capacity-building of local partners. There should be closer liaison with CEMCA’s CR work.

648. COL’s partners in ICT4E want to push on in the Caribbean and begin similar work in

the Pacific. The latter may be premature without consolidation and research on outcomes.

649. CEMCA is meant to be an integral part of COL, not a separate agency. Much

progress has been made in this over the last 2-3 years, although in terms of its

85

integration in COL’s RBM system there are still improvements to be made. CEMCA should receive support to integrate with COL’s RBM.

650. The new director appointed in 2012 should be supported by COL Vancouver to

continue CEMCA’s successful work, as well as explore new directions in a strategic manner.

651. Models seem to be the most tangible contribution COL can make to supporting the

mobilisation of ODL for development. To fully realise the potential of models COL needs to make improvements: It needs to build intensive outcome monitoring and evaluation as standard

into its planning for interventions with models. This will require resources. It needs to put more time into researching, discussing, designing and

implementing strategies for replication and up-scaling. 652. COL should focus even more on a small number of substantial medium to long-term

interventions. It should consider devoting perhaps 75% of its programme budget to 4-6 major interventions at any one time. These interventions should be transnational, ensuring that across its interventions all regions get a reasonable share. However COL should not pursue wide geographical coverage as an overriding priority if it is not compatible with effectiveness, impact and sustainability.

653. Working with a small number of larger-scale interventions should still leave room

for exploratory work, recognising that much of this may not take root. Exploratory work needs to have a clear deadline to prove itself and not drift on.

654. It should also leave room for some smaller scale interventions, although there

should not be diversity for its own sake. All projects must relate strongly to the Initiative outcome.

655. The evaluation endorses the strong message from a wide range of informants that

COL should move more resources to skills development. This may require a rethinking of the way COL configures its Initiatives. It is very important that COL’s work in this area benefits from coordination and collaboration across the organisation.

656. COL should identify and focus on “root” solutions: work which – with minimal

adaptation – can support outcomes in more than one subsector.

657. Quality assurance is a well-established root solution; and COL should consider pooling its resources and experience in QA for potential synergies across educational subsectors and beyond into informal learning.

658. OERs are the obvious emerging root area. COL is working with OERs in several

Initiatives and in KM. It is not inconceivable that in three years’ time COL has a team that works with OERs across the current Initiative boundaries.

659. Policy is another root solution. While this needs to be tailor-made at the level of

institutions, national policy advocacy for ODL is a slower, more detached process. It may be better pursued outside the boundaries of specific Initiatives.

660. If COL ups its game in M&E area, it could share both the learning from, and its

expertise in, M&E with partners. At the very least, COL should not reinvent the wheel with M&E in each of its Initiatives.

661. Keeping strategic partnerships together should not become an objective that

overrides the needs of programme partners.

86

662. COL needs to give its staff one or two effective tools to help them integrate gender

equality perspectives into their work. For example, COL has developed a simple checklist tool for gender mainstreaming in planning, implementation, monitoring and evaluation, known as the Gender Keys. This should be more widely used.

663. COL also needs to choose, and possibly refine, a gender outcome statement and 3-

4 indicators from the long list in its Gender Action Plan and include them in the Logic Model for the next TYP.

664. Outcome statements in COL’s TYP Logic Model need to be less generic than in 2009-

12. They need to be pitched at medium-term changes. If COL accepts the recommendation that it focuses on a small number of substantial interventions, some of its outcome statements could be built around the results expected from this work.

665. The TYP Performance Indicators need to be more precisely defined with

accompanying technical notes explaining among other things how the indicators will be monitored. This is particularly important where there are likely to be staff changes in an Initiative.

666. The TYP Performance Indicator targets need to be more consistent in their degree

of challenge. 667. Another improvement would be to break down the targets into annual stages. This

can be difficult to get right, but it gives all stakeholders a better idea of when the results are intended to emerge.

668. COL should develop short strategic plans for each Initiative. These should be

constructed for the life of the TYP – or even longer in some cases. They should be built around a logic model or theory of change (ToC) that links key actions and outputs for each “Activity” to the Initiative outcome. There would be a strong case for including intermediate outcomes in the models as well. The construction of the logic models/ToCs would best be done collaboratively both within COL and with partners.

669. The plans should include a proper narrative section about the rationale for the

Initiative and its Activities and how COL is intending to achieve the outcomes. These plans would be summarised in the logframes. Both the plan and the logframes would be reviewed annually.

670. COL should build Activity logframes around coherent packages of work – or projects

– rather than generic types of work – such as COL’s “strategies”. 671. Baselines in COL’s logframes are mostly defined inappropriately and are of little or

no use. This section of the logframe should either be improved or scrapped. 672. The development of outcome monitoring, in collaboration with partners, should be

the main item on COL’s agenda for improving its RBM system in the next TYP. 673. COL should institutionalise M&E planning with the implementation of the new TYP. 674. COL needs to consistently raise the level of its capability for commissioning

evaluations. This implies: Being a pro-active partner in evaluation design through terms of reference

and dialogue at the inception stage

87

Improving the recruitment of evaluators by widening the pool and having sound selection criteria and processes.

675. COL staff with experience in commissioning effective evaluations should be able to

support those who have had less experience and success. 676. COL should look more strategically at evaluations, and decisions about this should

be shared. There is often greater value in thematic, cross-cutting evaluations which tend to continue to have relevance after specific interventions have been completed.

677. COL’s next TYP evaluation will need to include direct engagement with a sample of

important interventions unless the quality of outcome monitoring and Initiative-level external evaluations are markedly improved.

678. COL should explore methodologies for telling stories through a transparently

balanced process, based on systematic monitoring at all levels of the value chain (from inputs to outcomes or even impact).

679. COL should include progress reports – with short narratives - on the TYP outcome

indicators in its annual reports to the Board. For some reason this is not done at present. The reporting is on output indicators from the logframes.

680. COL should streamline the reporting on logframes. Little use seems to be made of the quarterly reports. ESs report three times a year at PUMs and twice a year in personal appraisals with their line manager. Relevant data for these events need to be available. Aside from these needs, however, it may be sufficient to report on logframes annually. Less frequent reporting is likely to lead to more strategic reflection on, and use of, the data.

681. COL should consider finding another term for Activities. It is confusing for outsiders

–especially those used to logframe terminology – to find solid areas of work defined in this way. Activities in logframes are subsidiary to outputs.

682. COL’s KM facility has great potential to work closely with the eLearning Initiative

and with COL’s other ICT-mediated interventions. To optimise this, there needs to be regular dialogue between KM and the ESs. This may need systematising if opportunities are not to be missed.

683. When an ES leaves, COL should have a clear medium-term framework for the area

to which it intends to recruit a successor. This implies having a clear profile of the person it should be looking for; and providing an effective induction into this framework for the new ES.

684. The risk of problems arising as a result of a succession gap in some circumstances

could justify the funding of an overlap between outgoing and incoming ESs, or setting up a short period of cooperation downstream of the appointment.

88

Annex 1: Terms of Reference

The External Evaluation of the 2009-2012 Commonwealth of Learning Strategic

Plan

The Project and what it has set out to achieve

The Commonwealth of Learning commenced its current Three Year Plan, Learning for Development in July 1, 2009. It will be completed on June 30, 2012.

The plan, set within a Results Based Management (RBM) framework, aims to achieve developmental impact through open and distance learning methodologies and the use of appropriate technologies. Overall organisational objectives, medium term outcomes and long term impacts were identified. These are reflected in COL’s programmes, which have two sectors, 8 initiatives and three cross-cutting themes. Performance Indicators were identified at the outcome level and are expressed in the Plan’s Logic Model.

The reasons for the evaluation and the key questions that the Consultant should answer

It is now necessary, in the final financial year of the plan, to evaluate it against the objectives set at the start of the plan and to provide a report to the Board of Governors. The overall purpose of the evaluation is to i) ascertain what difference COL has made over the past two years, moving into the third and final consolidation year and ii) to provide recommendations that can be incorporated into the next TYP 2012-15.

The evaluation should consider both the strategy and the activities at the initiative level, which have been summarised in the logical frameworks for each initiative and activity. An 'evidenced-based' assessment of outcomes is needed with both quantitative and qualitative data. Longer term impacts may be more difficult to ascertain for specific initiatives of less than five years duration. But a general assessment should be made of COL’s impact since the 2003-06 plan cycle, using relevant data and the evaluator’s experience of COL since then. The evaluation should also put forward a plan for impact evaluations in the coming triennium.

The evaluation should be guided by the OECD’s Development Assistance Committee’s international criteria of Relevance; Effectiveness; Efficiency; Impact; and Sustainability11.

The four key questions that the evaluation should answer are:

1. To what extent has the Commonwealth of Learning made a difference in the life

of its current Three Year Plan? 2. What lessons can COL, its Board of Governors and partners, take from the

operation of the plan in to the next TYP? 3. How appropriate are the two strategic sectors, Education, Livelihoods and Health

in realising the aims of the current Three Year Plan? 4. What has been the impact of COL’s work since 2003-2006 to the end of the

current triennium?

11 ‘Criteria for evaluation of humanitarian assistance’, DAC Guidance for evaluating

humanitarian assistance in complex emergencies [Paris: 1999]

89

Who the main client and interested stakeholders are

The main client for the evaluation is COL and its Board of Governors. Interested stakeholders are the 54 Commonwealth countries and those with whom COL works. It would be necessary to consult with Board members and a small number of other key stakeholders to ascertain their views on the evaluation. A rough guide to the tasks to be completed and the schedule

The evaluation should consider both the strategy and the activities at the initiative level. In preparation for this latter examination, COL has conducted 10 ‘longitudinal’ studies or evaluations of key activities in its 8 initiatives and the work of CEMCA. It is envisaged that the evaluation will consider COL’s work over the life of the plan largely through these studies presently being conducted.

There has been constant monitoring over the life of the plan. There is considerable data on how the plan is progressing.

It is envisaged that the methodology would employ a desk study of the data produced by the initiatives; and interviews with the key stakeholders and COL staff. The main sources of data for the evaluation will be:

External evaluations at the Initiative level; Phone interviews with external evaluators; Phone interviews with key partners; Relevant internal monitoring data where it exists; Internal reports, such as the President’s quarterly reports and the annual reports

to the Board; Interviews with COL staff; and Interviews with selected Board members.

What the outputs from the evaluation may be and when they are due with specific dates if the report is required for a particular meeting

A final report will be needed by 15 December 2011. A draft report should be available by 15 November 2011 in order that consultations are completed by 6 December. A progress report should be submitted at the end of September. Information about the size and format of the report

The report should be no more than 50 pages plus appendices in length. Its structure should be:

Title Page Preface and acknowledgements Contents page Acronyms and abbreviations Executive summary The main report Introduction Methodology Findings Conclusions Recommendations Appendices/Annexes

90

Name and contact details of the Evaluation manager and reporting requirements during the evaluation and whom the report should be sent to

The Evaluation manager is Professor Asha Kanwar, Vice President, Commonwealth of Learning, Suite 1200, 1055 West Hastings Street, Vancouver BC V6E 2E9 Canada: akanwar@col.org. The contract manager is Lydia Meister, Programme Coordinator, Commonwealth of Learning. The final report should be sent to Sir John Daniel, President and CEO, Commonwealth of Learning.

91

Annex 2: Evaluation framework COL 2009-2012 TYP Evaluation Framework Headline questions

Detailed questions Indicators/targets Main sources and instruments

What difference has COL made since July 2009?

To what extent has COL helped countries plan and implement open schooling as a means of increasing access to learning opportunities at the secondary level? How have COL’s activities and outputs contributed to these changes, in terms of their Relevance and targeting Design Execution?

2 countries establish new open schools, and 10 existing open schools significantly extended. Open schools in 6 countries offer new high quality courses in 20 subjects. 10 secondary schools adopt open education resources as a central part of their learning strategies. 5 countries have developed and integrated digital content in 10 courses. A plausible trail from COL’s activities and outputs to these changes.

COL quarterly activity and initiative reports and selective reading of supporting data. Telephone interviews with partners and clients. F2F interviews with COL staff. Longitudinal evaluation and telephone interview with evaluator.

To what extent has COL contributed to an increase in teacher education and training institutions using ODL methodologies to train and upgrade teachers? How have COL’s activities and outputs contributed to these changes, in terms of their Relevance and targeting Design Execution?

4 major institutions in at least 3 Commonwealth regions begin to train/upgrade teachers, or significantly increase the number trained or upgraded, through ODL. Teacher education institutions in 4 countries offer significantly improved curriculum content. Institutions in 6 countries adopt the Child Friendly Schools approach. A plausible trail from COL’s activities and outputs to these changes.

COL quarterly activity and initiative reports and selective reading of supporting data. Telephone interviews with partners and clients. F2F interviews with COL staff. Longitudinal evaluation and telephone interview with evaluator.

To what extent has COL helped higher education institutions to create policies, systems and staff competencies that support the use of ODL to increase learner access and completion rates while enhancing curricular content? How have COL’s activities and outputs

4 major higher education institutions in at least 2 Commonwealth regions begin education, or significantly increase the number educated, through ODL. New or improved curriculum content is in use in higher education institutions in 4 countries. Quality Assurance mechanisms are

COL quarterly activity and initiative reports and selective reading of supporting data. Telephone interviews with partners and clients. F2F interviews with COL staff. Longitudinal evaluation and telephone interview with evaluator.

92

contributed to these changes, in terms of their Relevance and targeting Design Execution?

adopted by 4 tertiary institutions. A plausible trail from COL’s activities and outputs to these changes.

To what extent has COL helped VUSSC partner institutions produce and deliver relevant courses using eLearning strategies that are scalable and sustainable? How have COL’s activities and outputs contributed to these changes, in terms of their Relevance and targeting Design Execution?

8 new courses are developed and 2 institutions in 3 Commonwealth regions offer a total of 6 VUSSC courses. Agreements are in place committing 10 member states to take responsibility for the ongoing management of VUSSC. NQAs in 6 VUSSC member states will ensure the implementation of the TQF. A plausible trail from COL’s activities and outputs to these changes.

COL quarterly activity and initiative reports and selective reading of supporting data. Telephone interviews with partners and clients. F2F interviews with COL staff. Longitudinal evaluation and telephone interview with evaluator.

To what extent has COL helped training institutions and civil society organisations to create the policy frameworks and increased capacity needed to develop and deliver ODL materials for strengthening knowledge and skills in both formal and non-formal learning environments? How have COL’s activities and outputs contributed to these changes, in terms of their Relevance and targeting Design Execution?

6 institutions in at least 2 Commonwealth regions begin technical vocational education, or significantly increase the number educated, through ODL. New or improved curriculum content in technical vocational education is in use in institutions in 4 countries. 3 new skills development courses (in addition to VUSSC courses) are available as OERs and used by institutions in 4 countries. A plausible trail from COL’s activities and outputs to these changes.

COL quarterly activity and initiative reports and selective reading of supporting data. Telephone interviews with partners and clients. F2F interviews with COL staff. Longitudinal evaluation and telephone interview with evaluator.

To what extent has COL helped resource-poor farming communities use ODL to increase their knowledge and skills to access new information, training and financing that improves their livelihoods through various economic activities? How have COL’s activities and outputs contributed to these changes, in terms of

Members of farming communities in 6 countries in 4 Commonwealth regions have significantly improved economic circumstances through ODL. 4 government or international organisations adopt L3F models in 2 Commonwealth regions. 20 organisations in 3 Commonwealth regions strengthen their capacity in ODL

COL quarterly activity and initiative reports and selective reading of supporting data. Telephone interviews with partners and clients. F2F interviews with COL staff. Longitudinal evaluation and telephone interview with evaluator.

93

their Relevance and targeting Design Execution?

to address farmers’ learning needs. A plausible trail from COL’s activities and outputs to these changes.

To what extent has COL helped community organisations, NGOs and local public institutions to increase their capacity to create and use ODL materials to improve the health and well-being of their communities? How have COL’s activities and outputs contributed to these changes, in terms of their Relevance and targeting Design Execution?

40 community organisations, NGOs and local public institutions in at least 20 countries in 4 Commonwealth regions begin to use, or significantly enhance or extend their use of, ODL for education. 16 new health-related ODL programmes are used by communities in 4 regions of the Commonwealth. A plausible trail from COL’s activities and outputs to these changes.

COL quarterly activity and initiative reports and selective reading of supporting data. Telephone interviews with partners and clients. F2F interviews with COL staff. Longitudinal evaluation and telephone interview with evaluator.

To what extent has COL helped institutions and communities use digital technologies to design and develop learning materials and models that are made available, where possible, as open educational resources and to provide effective and appropriate skills training? How have COL’s activities and outputs contributed to these changes, in terms of their Relevance and targeting Design Execution?

10 major institutions in at least 2 Commonwealth regions use open educational resources for their curricula. Additional low-cost ICT training models are in use in 8 institutions in 4 Commonwealth regions. A plausible trail from COL’s activities and outputs to these changes.

COL quarterly activity and initiative reports and selective reading of supporting data. Telephone interviews with partners and clients. F2F interviews with COL staff. Longitudinal evaluation and telephone interview with evaluator.

To what other outcomes – both positive and negative – has COL made a significant contribution in this period? How have COL’s activities and outputs contributed to these changes, in terms of their Relevance and targeting Design Execution?

Other significant change in COL’s field of play. A plausible trail from COL’s activities and outputs to these changes.

COL staff focus groups. Telephone interviews with partners, clients and Board members. Reports to the Board.

94

What impact has COL had since July 2003?

To what extent has COL contributed over this longer time span to target countries improving the accessibility and quality of their formal education systems at all levels through the use of ODL tools and strategies?

Sustained improvements in accessibility to and quality of formal education in target countries over this period, where there is plausible evidence that COL has made a significant contribution.

COL staff focus groups and F2F interviews. Telephone interviews with partners, clients and Board members. Reports to the Board. Previous evaluations.

To what extent has COL contributed over this longer time span to improvements in the income, livelihoods and quality of life of communities and their members through new knowledge, skills and economic opportunities gained by means of ODL tools and strategies?

Sustained improvements in the income, livelihoods and quality of life of communities and their members through new knowledge, skills and economic opportunities over this period, where there is plausible evidence that COL has made a significant contribution.

COL staff focus groups and F2F interviews. Telephone interviews with partners, clients and Board members. Reports to the Board. Previous evaluations.

What other impact – both positive and negative – has COL had over this longer time span?

Sustained change in other areas over this period, where there is plausible evidence that COL has made a significant contribution.

COL staff focus groups and F2F interviews. Telephone interviews with partners, clients and Board members. Reports to the Board. Previous evaluations.

How appropriate are the two strategic sectors in realising the aims of the current TYP?

How appropriate are the two strategic sectors in realising COL’s aim of helping governments and institutions advance development by expanding the scope, scale and quality of learning by using new approaches and technologies, especially open and distance learning – bearing in mind COL’s locus, capability and capacity profile at the beginning of the triennium?

COL’s work in its two strategic sectors has made significant contributions to positive change. There is a sustained demand for COL’s interventions in these sectors from partners and clients. No other sectors present themselves as pressing candidates for COL’s interventions bearing in mind COL’s locus, capability and capacity.

COL staff focus groups and F2F interviews. Telephone interviews with partners, clients and Board members.

What are the lessons to take forward into the next TYP?

What areas of COL’s programme are working well and what are the implications of this for the next TYP?

Successful activities and initiatives strongly suggest building on the success; consolidating it; or disengaging through reduced need.

COL staff focus groups and F2F interviews. Telephone interviews with partners, clients and Board members.

What areas of COL’s programme are not working well and what are the implications of this for the next TYP?

Deficits in activities and initiatives suggest strengthening, changing track or disengaging.

COL staff focus groups and F2F interviews. Telephone interviews with partners, clients and Board members.

Are there new programme areas that COL should be including in the next TYP?

New programme areas present themselves strongly for inclusion bearing

COL staff focus groups and F2F interviews.

95

in mind COL’s locus, capability and capacity.

Telephone interviews with partners, clients and Board members.

Does COL need to change or re-balance its core strategies: partnerships, models, policies, capacity and materials?

Evidence of greater demand for particular strategies. Evidence that one or more of the core strategies make a greater difference than others.

COL quarterly activity and initiative reports and selective reading of supporting data. COL staff focus groups and F2F interviews. Telephone interviews with partners, clients and Board members.

Does COL need to change its approach to, or emphasis on, its three cross-cutting themes of quality, gender and appropriate technologies?

Evidence that COL interventions have not sufficiently led to high quality ODL

design and delivery; and sufficiently advanced gender

equality. Evidence that the use of technology advocated by COL has not been relevant to the outcomes sought.

COL quarterly activity and initiative reports and selective reading of supporting data. COL staff focus groups and F2F interviews. Telephone interviews with partners, clients and Board members.

Does COL need to make any changes to its monitoring and evaluation systems and processes?

Evidence that COL’s M&E are not making sufficiently positive contributions to accountability, programme improvement and wider learning?

COL logframes, quarterly activity and initiative reports. COL staff focus groups and F2F interviews. Reports to the Board Telephone interviews with partners and Board members.

What impact evaluation should be conducted in the next TYP period?

What are the priorities and scope for impact evaluation in the next TYP period, bearing in mind the findings and conclusions about COL’s programme work, and the coverage of recent evaluation?

Sustained initiatives or activities where there is insufficient evidence of effectiveness and impact and where there is a compelling need for such evidence either because the intervention is likely to continue beyond the next TYP, or because there are other uses for knowledge about their impact.

COL staff focus groups and F2F interviews. Reports to the Board Telephone interviews with partners, evaluators and Board members.

96

Annex 3: COL Stakeholder Survey 2010 – Conclusions

1. The Survey suggests that COL is helping its partners achieve good results in a wide range of ODL environments. In some cases, particularly in education, COL is contributing to programmes that are achieving large scale impact. In other sectors, it is working innovatively, but in many cases not yet at scale.

2. Partners perceive COL primarily as a capacity builder – directly or indirectly through consultants and network effects. Many see COL’s interventions as important contributions to institutional development.

3. Most see COL’s policy work as important, but some see it as often not achieving its full potential. COL is urged to be more assertive and persistent in its advocacy work with governments.

4. COL is a sought-after partner for a number of reasons. It has the ability to deploy relevant technical inputs from consultants, to bring together people and organisations in communities of purpose, and to give advice directly from its own expert resources. It is also said to be an accessible and reliable partner, one that can be approached with a minimum of bureaucracy and relied upon to communicate in a timely and effective way.

5. One of COL’s hallmark characteristics – one that sets it apart, particularly from donors and larger international organisations - is its disposition to listen and build interventions around partner needs.

6. COL’s ability to look at ODL solutions across regions and sectors is seen as a unique asset, one that COL should leverage as much as possible.

7. COL was not generally seen to demonstrate its commitment to gender equality in its outcomes.

8. The Pan Commonwealth Forum, while valued without exception, is thought to need a thorough conceptual overhaul. In particular it was seen to be formulaic with little opportunity for deep engagement with new ideas and research findings.

9. COL is seen as a more effective organisation than three years ago. Its staff are praised for their experience and energy. COL is also seen to be more focused and results-oriented. Nevertheless, a minority of respondents thought that COL spreads itself too thin.

10. For COL’s next TYP, apart from the call for more confident advocacy of ODL with governments, the area that should receive more resources than now was technical and vocational education.

97

Annex 4: Board member and advisor topic guide The evaluation of COL’s 2009-2012 Three Year Plan (TYP) will focus primarily on its programmes and projects – what COL calls its “Initiatives” and “Activities”. It will use COL’s Logic Model as the primary point of reference. I attach a copy of the Model for reference. It will explore

How relevant the Initiatives and Activities are to COL’s principal aim12. How well targeted they are. How well they are carried out. The changes to which they have contributed.

I would first value your view on these four questions. I appreciate that you may not have detailed perspectives on all of them. If you have first-hand experience of any of COL’s Activities, it would be very useful if you could illustrate your general view with examples from the work you know. Related to the question of relevance, I would be interested to hear you views on

Whether the two sectors currently covered by COL’s work – Education and Livelihoods and Health – are appropriate and sufficient, or whether there is a case for change.

Whether the balance of COL’s resourcing is right between the two sectors13. Whether the 8 Initiatives14 are appropriate and sufficient or whether there is a

case for change. The evaluation will report on the actual scale, and the scalability (through multiplier effects) and replicability (to other contexts) of the changes that COL has helped bring about. You views on these issues would be particularly valuable. COL has three cross-cutting themes or dimensions:

Quality in ODL Gender equality Use of appropriate technology.

COL aims to provide evidence of these three dimensions in the outcomes from its work. Any perspective you have on COL’s performance in relation to these cross-cutting dimensions would be useful. The evaluation will look at COL’s work emanating from its regional centres - particularly CEMCA – as well as from Vancouver. Your views on CEMCA would be appreciated. COL’s 2010 Stakeholder Survey looked at broader issues of

How COL works; and What contribution it makes compared with other development actors.

Although the TYP evaluation has a more focused programme and project perspective, it would be important to capture your current view of any broad issue on which you feel strongly.

12 COL’s aim is paraphrased as: Helping governments and institutions to take development forward by expanding the scope, scale and quality of learning through open and distance learning. 13 Currently about 61% for Education, 31% for Livelihoods and Health (with a further 8% for CEMCA). 14 Open Schooling, Teacher Education, Higher Education, Virtual University for Small States of the Commonwealth (VUSSC), Skills Development, Learning for Farming, Healthy Communities, Integrating eLearning.

98

Although this evaluation will focus primarily on the current TYP period, it will also explore longer-term outcomes and impact to which COL has contributed since 2003, particularly with Initiatives that began in earlier TYP periods. I would appreciate your views on the longer-term outcomes and impact of COL’s work. The evaluation will critically examine COL’s results-based monitoring and evaluation (RBME) framework and system, and the way it reports performance. Any views you may have on this would be welcome. It will specifically make recommendations on areas where evaluations (evaluative research) should focus in the next TYP period. If you think COL should evaluate a particular area of work – or cross-cutting theme - in this period, I would be pleased to know. It would be good if you gave your reason for this choice. Finally if there is anything else you feel should be included in this TYP evaluation, please discuss this with me.

99

Annex 5: Principal COL documents consulted Annual Progress Reports 2009-10 and 2010-11 COL Connections newsletters COL Stakeholder Survey report 2010 COL Website content COL’s Three-Year Plans 2003-06, 2006-09 and 2009-12 Country Action Plans and reports Customised reports on progress against TYP performance indicators Educational Specialists’ selective project plans and reports External evaluation reports on COL projects and Activities Logical frameworks for Activities and Initiatives Minutes of Board meetings and retreats President’s quarterly reports to the Board Quarterly Activity and Initiative Reports Results-Based Monitoring and Evaluation at the Commonwealth of Learning. A Handbook. Glen M Farrell. COL Vancouver 2009 Transnational Qualifications Framework for VUSSC, concept document 2008 TYP evaluation reports 2006 and 2009

100

Annex 6: People consulted COL Staff

Ameresekere, Ruvani Programme Assistant

Askounis, Helen Knowledge Management Coordinator

Balasubramanian, Kodhandaraman

Education Specialist, Agriculture & Livelihoods & Team Leader Livelihoods

Balaji, Venkataraman Director, Technology & Knowledge Management

Daniel, John President and CEO

Ferreira, Frances Education Specialist, Open Schooling

Gracieuse, Sharmila Programme Assistant

Iyer, Hema Project Assistant

Kanwar, Asha Vice President

Kaushik, Madhulika Education Specialist, Higher Education

Kinakin, Rebecca Project Manager, Child Friendly Schools

Kwan, Angela Learning Manager, International Organisations

Lee, Jasmine Executive Assistant

Lesperance, John Education Specialist, Virtual University for Small States of the Commonwealth (VUSSC)

Li, May Programme Assistant

McEachern, Doris Director, Finance, Administration and Human Resources

Mead Richardson, Alison

Education Specialist, Skills Development

Meister, Lydia Programme Coordinator

Monaghan, Amy Information Resource Centre Manager

Pringle, Ian Education Specialist, Media

Schlicht, Patricia Programme Assistant

Sreedher, Ramamurthy Director CEMCA

Tetreault, Christie Course Coordinator, International Organisations

Umar, Abdurrahman Education Specialist, Teacher Education & Team Leader Education

van Wyk, Trudi Education Specialist, eLearning

Vemraju, Rukmini Programme Officer, CEMCA

101

Walker, Carol Programme Assistant

Wilson, Dave Communications Manager

COL Board of Governors and Advisers

Bean, Martin VC, Open University, Milton Keynes, UK

Bourne, Jo Head of Profession, Education, DFID, London, UK

Ecker, Janet President, Toronto Financial Services Alliance, Canada

Glennie, Jennifer Director, South African Institute for Distance Education, Johannesburg, South Africa

Greaves, Pauline Head, Education, Social Transformation Division, Commonwealth Secretariat, London, UK

Katagum, Mariam Yalwaji

Ambassador/Permanent Delegate, Federal Republic of Nigeria, to UNESCO, Paris, France

Procuta, Elena Development Officer, Ministry of Foreign Affairs and Trade, Wellington, New Zealand

Sawyerr, Akilagpa Director of Research, African Association of Universities, Accra, Ghana

Scott, Gordon Governance Advisor, CIDA, Ottawa, Canada

Smith, Ransford Deputy Secretary-General, Commonwealth Secretariat, London, UK

Whiteman, Burchell COL Chair, Kingston, Jamaica

Wood, John Secretary General, Association of Commonwealth Universities, London, UK

External Evaluators

Bialobrzeska, Maryla Programme Specialist, SAIDE, Johannesburg, South Africa

Butcher, Neil Director, Neil Butcher Associates, Johannesburg, South Africa

Chanda, Ipshita Jadavpur University, Kolkata, India

Cunningham-Dunlop, Cathie

Director, Research and Evaluation Unit, Continuing Studies, Simon Fraser University, Vancouver, Canada

Gangopadhyay, Tota Principal, Mascan Consultants, Toronto, Canada

Hellmann, Liora Senior Education Researcher, SAIDE, Johannesburg, South Africa

Isaacs-Bardien, Shafika Independent consultant, Johannesburg, South Africa

Junaid, Muhammad Ibn Executive Secretary, National Commission for Colleges of Education, Abuja, Nigeria

102

Tyrer, Rod Principal Education Advisor, Consult4Change

Venkaiah, Vunnam Director, (Academic) BR Ambedkar Open University, Hyderabad, India

Wood, Cathryn Independent consultant, London, England

Strategic and Programme Partners/Beneficiaries

Amirtham, Thomas Arul Anandar College, India

Banda, Florida MaiMwana, Mchinji, Malawi

Butler, Greg Senior Director, Worldwide Education Leaders Strategy, Microsoft Public Sector, Microsoft, Redmond, USA

Dlamini, Peterson Chief Inspector - Teacher Education, Ministry of Education and Training, Swaziland

Herd, George FSD Lead Consultant, Aberdeen, UK

Idogho, Philippa Rector, Auchi Polytechnic, Nigeria

Ipaye, Babatunde Director, RETRIDAL, Lagos, Nigeria

Jain, Shivani Senior Programme Coordinator, Centre for Environment Education, Ahmedabad, India

Khokan, Rafiqul Islam Executive Director, Rupantar, Khulna, Bangladesh

Ming, Tony Adviser (Informatics), Governance & Institutional Development Division, ComSec, London, UK

Mourtoza, G M Director, CCD Community Radio, Rajshahi, Bangladesh

Murangi, Heroldt Director, Namibian College of Open Learning, Windhoek

Ogada, Charles Ugunja Community Resource Centre, Kenya

Rosato, Mikey Consultant, MaiMwana, Mchinji, Malawi

Rosemin, Karen Director, The Distance Learning Secretariat, Ministry of Science, Technology & Tertiary Education, Trinidad & Tobago

Sahu, Supriya Director, Broadcasting, Ministry of Information and Broadcasting, India

Sampson-Ovid, Lystra Programme Director, Distance Education Unit, Ministry of Education, Trinidad & Tobago

Tamanikaiyaro, Lineta Lecturer, National University of Samoa

Therakam, Thomas Mattoli Community Radio, Kerala, India

Vidanapathirana, Upali Vice Chancellor, Open University of Sri Lanka, Colombo