EvaluationIndependent

Performance Evaluation

Report

Raising development impact through evaluation

Sri Lanka: Distance Education Modernization Project

Reference Number: PPE:SRI 2016-20 Project Number: 33251 Loan Number: 1999 Independent Evaluation: PE-793

Performance Evaluation Report November 2016

Sri Lanka: Distance Education Modernization Project

This document is being disclosed to the public in accordance with the Asian Development Bank’s Public Communications Policy 2011.

NOTE

In this report, “$” refers to US dollars.

In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department (IED) does not intend to make any judgments as to the legal or other status of any territory or area. The guidelines formally adopted by IED on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of the management of IED, there were no conflicts of interest of the persons preparing, reviewing, or approving this report.

Director General M. Taylor-Dormond, Independent Evaluation Department (IED) Director W. Kolkma, Independent Evaluation Division 1, IED

Team leader F. Ahmed, Lead Evaluation Specialist, IED Team members M.A. Chaneco, Associate Knowledge Management Administrator, IED M.J. Dimayuga, Senior Evaluation Officer, IED

Abbreviations ADB – Asian Development Bank DEMP – Distance Education Modernization Project DEPP – Distance Education Partnership Program GCE – General Certificate of Education ICT – information and communications technology IED – Independent Evaluation Department LMS – learning management system MOHE – Ministry of Higher Education NAC - network access center NDEN – National Distance Education Network NODES – National Online Distance Education Service OUSL – Open University of Sri Lanka PCR – project completion report PMO – project management office PPER – project performance evaluation report PSC – project steering committee PVR – project validation report SDR – special drawing rights

Currency Equivalents Currency unit – Sri Lanka rupee (SLRe) At Appraisal

(8 May 2003) At Completion (24 June 2011)

At Independent Evaluation (25 August 2016)

SLRe1.00 = $0.0103 $0.0091 $0.0069 $1.00 = SLRs109.5300 SLRs109.5300 SLRs145.5800

Contents

Acknowledgments vii

Basic Data ix

Executive Summary xi

Introduction 1

A. Evaluation Purpose and Process 1 B. Expected Results of the Project 2 C. Evaluation Method 3

Design and Implementation 4

A. Formulation 4 B. Rationale 5 C. Cost, Financing, and Executing Arrangements 6 D. Procurement, Construction, and Scheduling 7 E. Outputs 7 F. Consultants 10 G. Loan Covenants 10 H. Policy Framework 11

Performance Assessment 12 A. Overall Assessment 12 B. Relevance 12 C. Effectiveness 16 D. Efficiency 19 E. Sustainability 19

Other Assessments 24 A. Impact 24 B. ADB and Executing Agency Performance 27

Issues, Lessons, and Follow-up Actions 28

Appendixes 1. Distance Education Modernization Project – Project Framework 33 2. People Met During the Performance Evaluation Mission 38 3. Appraisal and Actual Project Costs 41 4. Design and Monitoring Framework Showing Achievements, 2011 and 2016 425. Partner Institutions and Online Degree and Diploma Programs, 2011 47 6. Rating Matrix for Core Evaluation Criteria 49 7. Open University of Sri Lanka Enrollment, Financing, and Cost Recovery 50 8. Open University of Sri Lanka Enrollments and Contribution to National

Graduate Output 51 9. Enrollment in Open University of Sri Lanka Study Centers 1999−2014 52

Supplementary Appendix (available on request) Student Survey Questionnaire

Acknowledgments

This report is a product of the Independent Evaluation Department (IED) of the Asian Development Bank (ADB). Farzana Ahmed, Lead Evaluation Specialist, led the evaluation. The team comprised Mary Anne Chaneco (Associate Knowledge Management Administrator), Palita Dammalage (national consultant), Ma. Juana Dimayuga (Senior Evaluation Officer), and Wendy Duncan (international consultant).

The evaluation team acknowledges the support provided by colleagues in the

South Asia Department and the Sri Lanka Resident Mission. The team benefited greatly from discussions with officials of the Government of Sri Lanka, and particularly with staff at the Ministry of Education, Ministry of Higher Education, Ministry of Skill Development and Vocational Training, the Open University of Sri Lanka, the University of Vocational Technology and the National Education Commission, whose staff participated in the interviews and assisted in coordinating the field visits in various districts. We also appreciate discussions with other development partners active in the education sector in Sri Lanka. The feedback from participants in our questionnaire and focus group discussions strengthened the study and is much valued.

The report was peer reviewed by Brahm Prakash, Evaluation Consultant, IED

and was prepared under the overall guidance of Vinod Thomas and Walter Kolkma of IED. IED retains full responsibility for the report.

Basic Data Sri Lanka: Distance Education Modernization Project (Project Number 33251, Loan 1999)

Key Project Data

At Appraisal ($ million)

Actual ($ million)

Total project cost 60.00 46.99 Foreign exchange cost 24.00 24.81 Local currency cost 36.00 22.18 ADB loan amount/utilization 43.30 35.47 ADB = Asian Development Bank. Key Dates Expected Actual Fact-finding 28 July–11 August 2002 Loan negotiations 28–29 April 2003 Board approval 11 June 2003 Loan signing 28 August 2003 Loan effectiveness 16 November 2003 4 November 2003 First disbursement of loan 11 November 2003 Loan closing 31 December 2009 28 July 2010 Borrower: Sri Lanka Executing Agency: Ministry of Education

Type of Mission Number of Missions

Number of Person-Days

Fact-finding 1 No data Appraisal Waived Inception 1 24 Review 6 62 Midterm review 1 22 Special loan administration 2 3 Project completion review 1 36 Independent evaluation 1 28

Project Performance Report Ratings Development

Objectives Implementation June 2003–July 2010 Satisfactory Satisfactory

Executive Summary

Following successful provision of near-universal basic and secondary education, most countries in Asia and the Pacific now face the challenge of providing young people with the knowledge and skills needed to boost employment and economic growth, as economies transition from agriculture to manufacturing and services, and the labor market moves from largely public sector to greater private sector activity, but government budgets remain constrained.

In the early 2000s, Sri Lanka exemplified these dilemmas. While successive Sri

Lankan governments had been very successful in offering basic and secondary education to almost all children, even as a civil war raged, the postsecondary education system was so small that it covered only 3% of young people of the relevant age. At the same time, the rate of unemployment among the small group of post-secondary graduates was higher than the national rate of unemployment, and employers complained they could not find workers with the skills they needed for a modernizing economy. The government of the time recognized that significant reform of postsecondary education was necessary but it lacked the resources to do so. In line with its strong social policies, the government was also committed to ensuring university education remained free, and that all young people, especially those in conflict-affected and plantation areas, could access postsecondary education and training that would be consistent with private sector employment opportunities. An innovative approach was needed to the expansion of postsecondary education and training, outside the conventional tertiary education system.

To assist the government in achieving these goals, the Asian Development Bank

(ADB) supported a project that aimed to expand education and training opportunities for school leavers in rural and conflict-affected areas through the use of online distance education. Although Sri Lanka had a tradition of distance education through printed materials, online education was new. This approach was in line with the government’s strong focus on using information and communications technology (ICT) to modernize the Sri Lankan economy. Distance education learning opportunities were to be expanded through the establishment of a national fiber optic network offering online distance education. Courses were to be provided by public universities, including the Open University of Sri Lanka (OUSL), and private training providers offering more vocationally oriented training. This was another first for Sri Lankan education. While the government recognized the value of greater private sector involvement in education, it had little experience of public–private partnerships. Finally, the project also aimed to upgrade and expand the capacity of the OUSL through the development of online courses and more conventional institutional strengthening and development.

The Distance Education Modernization Project (DEMP) was financed through a

loan of SDR33,005,000 (equivalent to $45 million) from ADB Special Funds resources. The 6-year loan was approved on 11 June 2003, and closed on 28 July 2010 after a 6-month extension. The project was selected for performance evaluation because of its strong elements of innovation. The project performance evaluation report (PPER) provides an opportunity to assess the longer-term impact of the project while also

xii Distance Education Modernization Project identifying lessons generally applicable to the design and implementation of innovative projects.

A. Project Performance

This report presents performance evaluation findings for four evaluation criteria (relevance, effectiveness, efficiency, and sustainability) as well as an additional criterion of impact. Overall, the project is rated less than successful.

Relevance. The project is rated relevant. It responded to government and ADB

national and sector objectives, and it addressed the key issue of tightly restricted access to tertiary education in an imaginative and innovative manner. As a new and innovative project, it faced many risks to success, not all of which were mitigated adequately. The project design was too complex, too broad, and most significantly did not adequately anticipate the resistance to online education that emerged among both students and education providers. More groundwork and investigation was needed before launching such an ambitious investment project.

Effectiveness. The project is rated less than effective. It had many noteworthy

achievements, but on a smaller scale than originally intended. By the end of the project, it had provided opportunities for over 33,000 tertiary students, through online courses that provided job-relevant qualifications and skills. This was about one-third of the original target. The complexity of the project, resistance and caution among the intended partner institutions, the late uploading of online courses, and hesitation about their value on the labor market were the primary factors behind the failure to attract large numbers of students to online education.

Efficiency. The project is rated less than efficient. Implementation was handled

reasonably efficiently given the challenging nature of the project, but the delays in establishing the network and developing courses resulted in too little time for enrolments to grow. The National Online Distance Education Service (NODES) courses were not cost efficient because of the high costs of internet provision in Sri Lanka. The OUSL proved more cost efficient, increasing its level of cost recovery from 25% to 42%, in line with the target.

Sustainability. The project is rated less than likely sustainable. The NODES

system established under the DEMP is no longer in operation. The infrastructure and some staff have been handed over to the OUSL for incorporation into its systems. How the OUSL will utilize the network, and whether it will one day be open to other institutions, is not yet known. However, the introduction of ICT-enabled education is likely to be sustainable in various forms. It is also behind emerging changes in how universities provide their external degrees, and moves by the OUSL to slowly increase online content.

Impact. The project’s impact is rated satisfactory. Of the two major project

components, the capacity development of the OUSL was most successful. The OUSL doubled its enrolments with project support, and in 2014 provided significant proportions of the country’s graduates in high-demand fields such as law, science, and engineering. Other significant impacts occurred through (i) the accreditation and performance standards developed for distance education and now accepted as the basis of a university accreditation system, (ii) the successful strengthening of the management and staff capacity of the OUSL, and (iii) the building of significant capacity across the tertiary education sector for the development of online courses.

Executive Summary xiii Notwithstanding that the online architecture as designed did not take off, the project raised awareness of the potential of online learning and has influenced the way universities have started to explore this option.

B. Innovation and Risk in Project Design and Implementation

The defining characteristic of the DEMP is that it was a visionary project, albeit one that ultimately failed to achieve its vision as initially designed. However, the PPER finds that ADB should have continued to support the development of this concept with the possibility of exploring other avenues of financing and implementation, given the world trend toward online education. Several key issues arise in relation to this nexus between innovation and failure to succeed.

Innovation and risk. Innovative projects are inherently risky in that they almost

always attempt something that has not been tried before. Such projects therefore have a higher degree of risk than others. It is difficult to identify all the possible risks, and to ensure that risk mitigation measures will be effective. All these factors contribute to a higher than usual likelihood that an innovative project will not succeed. At the same time, innovation is being actively encouraged. How much risk is acceptable? If ADB is to consider innovative projects for lending, both parties should explicitly acknowledge the risk of failure.

Time and resource allocation. Projects that are new and innovative require

more time and resources during preparation and implementation. In the case of the DEMP, insufficient time was allocated up front to determine whether the project was feasible. This could have been done by assessing (i) whether the demand for online education was sufficient and (ii) whether the expected public and private partners would commit to the project. Although the DEMP was a 6-year project, this was not long enough to bring about the dramatic changes envisaged. The resources devoted to project implementation by ADB were also insufficient. Before innovative projects proceed, both ADB and the borrower should confirm that they are prepared to commit more time and resources than is usual to optimize the chances of success.

Evaluation of innovative projects. Should projects with a high degree of

innovation be evaluated and rated according to the same strict requirements as other projects? In the case of the DEMP, for example, while it failed to meet most of its specific outcome targets, it did open people’s minds as to the value of ICT in teaching and learning, and laid the groundwork for later developments.

Sustainability and innovation. When new approaches are being tried, it is often

not possible to determine in advance the best way to ensure sustainability. The DEMP had no agreed sustainability strategy initially because it was not known how the distance education network would develop. Neither was a sustainability strategy developed over the course of project implementation, despite intensive effort and lobbying by the project implementers. It is ironic that this flexibility in support of an innovative approach, while positive in itself, allowed the issue to drift unresolved for several years. This led to the ultimate failure of NODES. The competing pressures of flexibility and the need for commitment to a clear path require careful balancing and management.

xiv Distance Education Modernization Project

C. Lessons Arising from the Performance Evaluation

Ensure projects have a clear and achievable focus. The design of the DEMP was very broad, with the project inputs spread too thinly. The project included the construction of a national broadband network; mobilization of nontraditional partners; capacity building and institution strengthening within the OUSL; development of online courses, which is notoriously difficult and slow; and implementation of these courses with the expectation of rapid, high enrolments. The DEMP may have had more chance of success if it had initially focused only on the OUSL, moving to the establishment of NODES as a second phase.

Allocate significant resources for preparation of innovative projects. For innovative projects, significant resources are needed up front. This can include resources for assessing the feasibility of, and demand for, the project objectives and approaches for achieving them, as well as to lay the groundwork for ensuring their acceptability among different interest groups. Resistance to the innovations proposed cannot be overcome at the same time as project implementation is proceeding. The same applies to demand for the project outputs and outcomes. It is critical that a reasonable amount of consensus and commitment toward the project objectives is established before implementation begins.

Require expanded risk matrixes for all innovative projects. Considerable time, effort, and emphasis must be placed on the development and monitoring of the risk matrix for innovative projects. This applies particularly to risk identification, and potential risk mitigation strategies. A range of alternative strategies should be developed in detail for every identified risk, with a common understanding and commitment from ADB and the government. This should be a major focus of ADB-government discussions in project review missions, and should be updated at least annually. A sharper focus on emerging risks, development of appropriate strategies to address these risks as they emerged, and a strong joint commitment to implementation of these strategies would have been very helpful for implementation of the DEMP.

Ensure adequate ADB support for project implementation. By definition, the knowledge, skills, and experience needed to guide and embed innovative change are likely to be scarce in both the borrowing countries and in ADB. This was certainly the case for the DEMP. Additional time and resources are required to support implementation, which should include longer, more intensive, and more frequent review missions, and supplementation of missing technical skill areas on the part of ADB. Additional technical assistance to strengthen the expertise available under the loan and ensure ongoing support to implementation may also be necessary. If the sector department is not committed to providing additional resources and support to a given innovative project, then that project should not proceed.

Ensure sustained investment. Fundamental change to education systems is complex and does not happen quickly. When an entirely new system is being created, as it was under the DEMP, it is critical that a commitment to sustained investment is made. The DEMP was planned, it seems, as a one-off investment. Innovation and reform require long-term commitment to be successful.

Agree on a sound strategy for sustainability. The experience of the DEMP illustrates the importance of developing and operationalizing sound sustainability strategies. The project paved the way for an innovative online learning system, but the lack of a clear long-term plan was one reason why the concept eventually failed. If the

Executive Summary xv best strategy for sustainability is difficult to specify at the outset, specific support and additional time should be allowed in the project design to prepare and operationalize an agreed sustainability plan.

Ensure that the project performance indicators are appropriate for the time frame. In most education projects, the focus of project implementation is on development of systems or programs that are only completed near the end of the project. The final achievement measures, however, are generally framed in terms of the number of student beneficiaries. The DEMP was no exception. Apart from the time required for program development, it generally takes at least one program cycle to convince students that it is worthwhile. The DEMP may have been better as a multiphase project with appropriate targets for each phase, to allow time for incremental development of the national network.

Innovation needs champions. Effective champions are critical for the success of any new initiative. This is even more the case for innovative projects, where strong government and ADB ownership and commitment are critical for project success. The failure to ensure that the DEMP remained physically located within government agencies with a strong commitment to the project, especially following changes of government, was one of the factors that undermined its success. Similarly, the failure of ADB to adequately support project implementation, and to assist the government in identifying and implementing a practical solution for long-term sustainability after the project ended, also contributed to the project’s limited success.

Future potential of online education. The DEMP failed to expand tertiary education opportunities and increase graduate employability to the extent intended. One of the lessons of the DEMP is that online learning is not a magic bullet. Sri Lankan teachers and students were not ready for fully online courses, at least for longer programs. A combination of online learning with a limited amount of face-to-face tutoring appears most appropriate. Some universities have established their own open and distance learning centers, and these could be considered for future ADB assistance, along with support for further capacity expansion at the OUSL.

xvi Distance Education Modernization Project

CHAPTER 1

Introduction 1. This project performance evaluation report (PPER) assesses the Distance Education Modernization Project (DEMP), which aimed to help the Government of Sri Lanka modernize the postsecondary education system, in particular through the introduction of online distance education.1 This was intended to expand postsecondary education opportunities for the large numbers of secondary school graduates unable to gain admission to conventional universities, particularly those in conflict-affected and plantation areas, and was in line with the government’s strategy to strengthen its human resource base. It was intended that this expansion of opportunities would, in turn, promote economic growth and social cohesion across the country.

2. The capacity, quality, and relevance of learning through online distance education was to be boosted through the establishment of a national distance education network for online learning, with participation of both public and private providers, and upgrading and capacity expansion of the Open University of Sri Lanka (OUSL). The promotion of new public-private partnerships to expand education and training opportunities relevant to employment was central to the introduction of online distance education.

A. Evaluation Purpose and Process

3. The project was selected for a performance evaluation because of the innovative approach taken to the expansion of tertiary education opportunities. Both online distance education and public-private partnerships in tertiary education were new to Sri Lanka. The project was evaluated against the four criteria of relevance, effectiveness, efficiency, and sustainability. Although it does not affect the final rating, the impact of the project was also assessed. In particular, the evaluation assesses the project’s success in expanding tertiary education opportunities. The evaluation’s findings will feed into the Sri Lanka Country Partnership Strategy 2016-2018.

4. The project completion report (PCR) was prepared in December 2011 and the project validation report (PVR) in December 2013. 2 The PCR assessed the project outcome and outputs against the revised targets. Both the PCR and the PVR rated the project less than successful. The PCR rated the project highly relevant to the government’s education policies and poverty reduction strategies, as well as to the Asian Development Bank (ADB) country strategy. However, this rating was downgraded to relevant by the PVR. The project received less than satisfactory ratings on all other dimensions. Both the PCR and the PVR rated the project less effective. The project did successfully establish a national platform for online distance education, comprising the

1 ADB. 2003. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to

the Democratic Socialist Republic of Sri Lanka for the Distance Education Modernization Project. Manila (Loan 1999-SRI).

2 ADB. 2011. Completion Report: Distance Education Modernization Project in Sri Lanka. Manila (Loan 1999-SRI); ADB. 2013. Validation Report: Distance Education Modernization Project in Sri Lanka. Manila.

2 Distance Education Modernization Project Distance Education Partnership Program (DEPP) and the National Distance Education Network (NDEN), which embraced the country’s poorer regions. It also expanded student access to a range of postsecondary education opportunities. However, the range of online courses and the number of students accessing them was lower than expected. The project was rated less efficient by both the PCR and PVR because implementation was slow, with the launch of online courses delayed. The resulting lower numbers of students (the project beneficiaries) meant that the cost efficiency was low and the rate of return envisaged at appraisal could not be realized. The project was also rated less likely sustainable because the revenue generated from student fees was insufficient to sustain the organization. In addition, a corporate entity to succeed the DEPP and put it on a business footing had not been established by the time of the PCR.

5. The project was assessed by the PVR as having had a moderate impact, on the basis of what was reported in the PCR. Although the network had not by 2011 managed to solidify its reputation and build a solid customer base, there was potential for this to happen. The project had demonstrated the power of online distance education to reach poorer students, utilizing public-private partnerships, and the potential for achieving this cost-effectively. The downgrading by the PVR of one of the ratings in the PCR, and also its less than satisfactory rating of the PCR itself, was strongly rejected by the division concerned.

6. Because of the potential significance for a follow-on project, the PVR recommended the carrying out of a PPER once the government had implemented a sustainability plan for the DEPP. By that time, the benefits of the project were expected to be more evident.

B. Expected Results of the Project

7. The expected outcome of the project was the development of online distance education to expand postsecondary school enrolments and develop a modern, high-quality human resource base. The outcome was to be achieved through three major outputs: (i) increased capacity and raised quality in online education through the establishment of the DEPP; (ii) better quality programs in distance education through a consortium of public-private partnerships; and (iii) expanded capacity and upgrading of the OUSL. The design and monitoring framework is in Appendix 1. The project was implemented over 6 years, from 4 November 2003 to 28 July 2010, and was national in scope. It should be noted that the DEPP was renamed the National Online Distance Education Service (NODES) by the government in 2011, after the project ended, and is referred to as NODES throughout this report.

8. The outcome and output targets were ambitious and somewhat unrealistic in a country with no previous exposure to online learning and a low level of internet penetration, which continues until today.3 OUSL enrolments were expected to double to 40,000 by 2012, and 150 institutions offering degree and diploma programs were expected to participate in the network. Private institutions were expected to enroll 35,000 students in online diploma programs and public universities 25,000 students in online degree programs. In addition, 20,000 unemployed school leavers were to be given short-term, semiprofessional job training. These targets were revised after the midterm review in 2008, 18 months before the project closed. The target of establishing partnerships with 150 institutions was redefined as the provision of 40 online degree and diploma programs, and the target number of external university

3 World Economic Forum. 2015. The Global Competitiveness Report 2015-2016. Geneva.

Introduction 3 undergraduates 4 and private diploma students was reduced to 10,000 from a combined 60,000. The number of unemployed school leavers to receive skills training was similarly reduced to 8,000 from 20,000. The enrollment target for the OUSL was not changed as this was on track to be achieved.

C. Evaluation Method

9. The PPER reviews the project achievements as of 2016 in terms of the original expected outcome and revised outcome indicators. Interviews were conducted with key stakeholders in government, conventional universities, and the OUSL in Colombo. OUSL staff, students, and other informants were also interviewed in four study centers in the conflict-affected Northern, Eastern and North Central provinces, as well as a tea plantation area. The list of people met is in Appendix 2. Information was gathered to assess the impact of NODES in these areas, its potential for future sustainability, and the success and impact of the upgrading of the OUSL. The evaluation also assessed the future potential for online distance education as a contributor to human resource development in Sri Lanka.

10. The project design and implementation placed a strong emphasis on supply factors but less attention was paid to the demand for online education. One of the consequences, noted in the PCR, was that the project failed to build widespread acceptance of distance education, which contributed to the low enrolments achieved by project end. To shed some light on the experience with, and attitudes toward, online distance education among current students, a short questionnaire was administered to 104 students who were present in three of the OUSL study centers visited during the PPER mission. The interviews with key stakeholders and informants form the basis for the evaluation of the project, illuminated where appropriate by the views of the students.

4 External students are registered in courses with public universities but study the syllabus independently.

Most universities subcontract a private company to do the tuition. External degrees are 3-year general degrees, while degrees for internal students are 4 years with more specialized content. External and internal degrees are recognized by the government as equivalent, as both types of degree are conferred by the University Grants Commission.

CHAPTER 2

Design and Implementation A. Formulation

11. In 2000, the government requested technical assistance from ADB to support postsecondary education outside the conventional university system. 5 The initial proposal from ADB to focus support on the OUSL was subsequently broadened in response to this request. At the same time, the government requested the World Bank to help rationalize conventional universities. The ADB project was prepared during 2002. A strongly participatory strategy was followed during project preparation, including extensive workshops and consultations with public and private employers, the Ceylon Chamber of Commerce, nongovernment organizations, funding agencies, students, university faculty, government policy-makers, and educators. The project interventions were well conceived and were in line with programs implemented by the OUSL since its establishment in 1980.

12. The ministries of finance and education were closely involved in the formulation of the project, and were very supportive of the innovative technical solutions proposed. The mood in the country at the time was optimistic as it was thought the civil war was coming to an end, and the government was keen to unite the country and to boost the economy through development of the private sector. This would require a good supply of well-educated, qualified workers. The vice chancellor of the OUSL was an effective champion of the project as well, recognizing that the OUSL needed to modernize quickly in order to strengthen its reputation as a high-quality institution.

13. However, the political landscape was made up of many different interest groups. The government had already finalized new policies outlined in The Future – Regaining Sri Lanka and e-Sri Lanka: An ICT Development Roadmap, which emphasized the mainstreaming of information and communications technology (ICT) for broad-based tertiary education delivery.6 Work had also begun on a specific ICT policy for education. However, the education community was divided on the issue. Public universities were more interested in upgrading and protecting their established teaching and research programs than in investing resources in distance education. Some, but not all, of the private training institutions were keen to develop online courses but had to be sure of generating a quick profit. Students were opposed to any ‘privatization’ or involvement of the private sector in tertiary education. Even the OUSL itself was divided, with the leadership and regional staff aware of the urgent need for revitalization, while others were reluctant to change their ways. In combination with the ongoing civil war, this made for a difficult and risky environment in which to pursue innovation. While the technical assistance incorporated an understanding of the

5 ADB. 2000. Technical Assistance to the Democratic Socialist Republic of Sri Lanka for the Postsecondary

Education Modernization Project. Manila. 6 Government of Sri Lanka. 2002. The Future – Regaining Sri Lanka. Colombo; Government of Sri Lanka.

2002. E-Sri Lanka: An ICT Development Roadmap. Colombo.

Design and Implementation 5 politics and associated project risks, more attention should have been paid to developing effective risk mitigation strategies, particularly in relation to the likely acceptability of the key project strategy, online distance education. For example, much more could have been done to convince the universities and private institutions of the merits of ICT-enhanced learning.

14. ADB and the World Bank worked alongside each other to develop the two projects and ensure complementarity, which resulted in a consistent approach across the higher education sector. The work being carried out by the governments of Sweden and Norway, the two major bilateral donors, on the reform of university management, information, financial, and accounting systems was also incorporated into the project.

15. The project investment modality was chosen because the new government’s policy reform agenda was still being elaborated in several contentious areas, such as the expansion of private investment in education, the planned liberalization of telecommunications, and the modernization of higher education. In addition, high levels of investment were required for upgrading infrastructure and developing software. ADB provided funding of SDR33,005,000 (equivalent to $45 million) from its Special Funds resources. The 6-year loan was approved on 11 June 2003 and closed on 28 July 2010. A 6-month extension was granted at the end of the project to facilitate financial closing.

B. Rationale

16. The project was formulated to address the need to upgrade the skills of the Sri Lankan workforce in line with the changing economy, the low labor-market relevance of much of the existing higher education, and the limited access to postsecondary education, particularly in rural areas. Low levels of government funding combined with a reluctance to charge fees had constrained the growth of public tertiary institutions. The government had spent only 3% of gross domestic product on education for the previous decade, with only 20% of this going to higher education, while private postsecondary institutions were generally viewed with mistrust and were not yet competitive. At the time the project was designed, only 3% of the relevant age group was enrolled in postsecondary education compared with 8% across South Asia and 20% in Thailand. This was in stark contrast to the situation in primary and secondary education, where sustained investments over several decades meant that 99% of the relevant age group was enrolled in primary education and 90% in lower-secondary education. Participation fell after the General Certificate of Education (GCE) O-Level examinations, when about half of the approximately 400,000 grade 11 students left school and entered the labor market, as only limited training opportunities were available. A further half of the students who continued to grade 12 passed the GCE A-Level examinations after grade 13 and qualified for university.

17. The imbalance between school and tertiary education enrolments meant that far more students qualified for university than could gain admission. In 2001, about 25% of the relevant age cohort, or around 100,000 students, qualified for university. Of this group, only 15% or 15,000 entered conventional universities, 5% went to the OUSL or technical colleges, and 10% subsequently entered external degree programs. This meant that around 65% or 65,000 students who had passed their A-Levels failed to enter a university and had to find other alternatives. External degree programs had been introduced by the government as a strategy to expand access to university education at low cost and were essentially distance education. Students studied for university examinations on their own and paid for private tutoring if possible. By 2001,

6 Distance Education Modernization Project enrolments in these programs had ballooned to a high 166,000 because few students ever succeeded in completing their courses and graduating. The majority of external students were women studying arts and the humanities.

18. Despite this high degree of selectivity, unemployment among educated youth was higher than average. While the national unemployment rate was 8.0%, it was 14.6% for those with A-Levels or higher. Conversely, it was only 1.9% among those with only lower primary schooling. This pattern was due in part to the nature of conventional higher education, much of which did not cater to the emerging needs of the labor market. The economy was in transition, with the share of agriculture declining and manufacturing and services increasing. The public sector was contracting while private sector employment was expanding. The skills required by private sector employers centered on computer science, financial management, marketing, and English, skills which many graduates lacked, especially graduates from external degree programs.

19. The project proposed to address these issues through the use of online distance education. Online education could be used to expand postsecondary education cost-effectively through a new national virtual network, and would be particularly appropriate for expanding access to training opportunities for students in rural, conflict-affected, and plantation areas. The project built on the national outreach system and the print-based distance learning programs of the OUSL, but aimed to broaden the reach of the system by involving conventional public universities and private training institutions as well. It was expected that the private institutions would provide focused, diploma-level skills training better suited to labor market needs, especially in high-demand fields such as business, computer science, English, and engineering. The public institutions were expected to put their external degree programs online, which would be of great benefit to external degree students. The solution was elegant and innovative in the way that it brought together many different elements to solve several problems at once.

C. Cost, Financing, and Executing Arrangements

20. At appraisal, the project cost was estimated at $60 million, toward which ADB provided a loan from its Special Funds resources equivalent to $45 million, the government provided $10 million, and beneficiaries provided $5 million. The actual project cost for ADB, the government, and beneficiaries was $46.99 million, while the amount disbursed from the ADB loan was $36.2 million (see Appendix 3). Three partial loan cancellations were made totaling $14.5 million equivalent. The lower than expected disbursement was due largely to a reduction of output targets for the number of online programs to be developed, but also partly to the appreciation of the special drawing right (SDR) against the US dollar that increased the US dollar loan amount by 13%.

21. Three changes were made in the executing arrangements. Initially, the Ministry of Policy Development and Implementation was the executing agency, with the Ministry of Tertiary Education and Training as the implementing agency. This changed in 2004 when the Ministry of Education was appointed as both the executing and implementing agency. In 2005, these functions passed to the Ministry of Higher Education, where they remained until the project closed.

22. The implementation arrangements were innovative but complex. The project management office (PMO) was established by the Ministry of Tertiary Education and

Design and Implementation 7 Training at the start of the project. The PMO was never physically located within a government department, which did not encourage close day-to-day working relationships. The project steering committee (PSC) was the top decision-making body, chaired by the secretary of the executing agency and with broad membership of all key stakeholders. The PSC met every quarter, and was assessed by the PCR as effective. Several teams operating under these structures were established to implement the different components, all coordinated by one project manager. The PCR noted that, while the implementation arrangements were elaborate and comprehensive, they were marred by a lack of coordination among stakeholders and weak links to policy-makers. These weak links resulted in several policies developed under the project and key to its sustainability failing to gain government approval.

D. Procurement, Construction, and Scheduling

23. The project was implemented from 4 November 2003 to 31 November 2009 in accordance with the original time frame. Financial closing was on 28 July 2010. Project implementation was nevertheless marred by delays stemming partly from an initial 9-month delay in recruiting the consultants. The construction of the national ICT network was particularly affected by the delayed consultant recruitment, as the new courses could not be uploaded until the network was operational. The development of a public-private partnership policy also started 2 years behind schedule, which delayed initiation of the partnerships themselves. The social marketing campaign also occurred too late in the project to attract sizeable numbers of students. All of these delays, combined with the smaller than planned ICT network, meant it was not possible to achieve most of the original outcome and output targets.

24. Civil works and equipment comprised 58% of expenditure, with both requiring additional allocations to cover cost increases and local currency depreciation. Some delays in the procurement of equipment occurred due to limited availability and the lack of local suppliers, but the suppliers generally performed satisfactorily. The performance of the contractors was also generally satisfactory, even though 2 of the 14 civil works contractors did not perform satisfactorily and had their contracts terminated.

E. Outputs

25. The project had three major outputs intended to develop distance education: (i) establishment of the distance education partnership program or NODES, (ii) establishment of a consortium of public-private partnerships, and (iii) expanded capacity at the OUSL. The updated design and monitoring framework summarizing the physical accomplishments is in Appendix 4.

26. Establishment of the National Online Distance Education Service. The project was groundbreaking in its attempt to establish a national online learning network that any education or training provider could use to offer a wide array of courses to students all over Sri Lanka. A national distance education network comprising a network operation center, a content development unit, a disaster recovery center, and 26 multimedia network access centers (NACs) were successfully established to deliver distance education across the country, using up-to-date network technology. However, the building of the network took a year longer than planned as a result of delays in the hiring of technical consultants, and was not completed until the third quarter of 2007. The network was also much smaller than planned: the original target of establishing NACs in 150 partner institutions was found to be unachievable due to the lack of

8 Distance Education Modernization Project willing partners, and the target was reduced at midterm to 40 NACs. Despite this reduction, the network still covered almost every province. All of the NACs had high-speed internet, connected multimedia computers, projectors, videoconferencing, tablets, scanners and printers, an in-house server, and generators. NAC staff members were trained to operate the network. The selection and development of the online platform judged most suitable for course delivery proceeded concurrently with the establishment of the network.

27. To identify the occupational skill areas most in demand, a labor market analysis that included a study of job listings and employer interviews was undertaken early in the project. Subsequently, the specific courses to be developed were identified. The skill areas identified included ICT, engineering technology, agro-technology, biological and physical sciences, business management and administration, marketing, business English, accountancy, teacher education, and hospitality. An online career guidance program with a focus on youth and women did not proceed because an appropriate specialist could not be found. The impact of this was cushioned somewhat by the fact that 75% of all NODES students were already in employment, with most aiming to upgrade through job-relevant qualifications. The career guidance system was to be linked to a new labor market information system (LMIS) that was to be developed under the project. This did not proceed either, as it emerged that the Ministry of Tertiary Education and Training was already developing an LMIS under the World Bank higher education project.

28. A critical part of the network was the content development unit that monitored and supported the development of online courses by partner institutions. This unit undertook an impressive program of capacity building, and succeeded in building a cadre of over 900 partner institution staff skilled in online course design, development, and delivery. Nevertheless, progress in course development was slow. Staff in the partner institutions had first to be convinced of the merits of online education before training could begin, and had only limited time to contribute to the task on top of their regular jobs. The limitations of consultant support also meant that the work of course development had to be staggered among the institutions, which further slowed the pace. As a result, when the network was launched in late 2007, only around 12 courses were available. Better progress was made in the last year of the project, when a further 28 courses were put online. By the time the project ended, there were 42 courses offered through the new network, which just exceeded the revised target of 40 courses (see para 8). The limited number of courses, combined with their late launch, was one of the main factors explaining the low levels of student enrolment achieved.

29. Project marketing activities in the form of general awareness-raising about online education proceeded in parallel with course development, but a more extensive social marketing campaign aimed at attracting students to specific NODES courses did not take place until the last year of the project. The campaign was not successful in delivering its messages to prospective students, which meant that levels of awareness about online education remained low. In any case, the campaign took place too late to have any chance of building up significant student enrolments.

30. Establishment of a consortium of public-private partnerships. A major achievement of the project was the bringing together of public and private training institutions under a common framework for the first time, although not on the scale originally intended. The original target for the consortium was 150 partner institutions; however, by the end of the project, only 24 institutions had joined, comprising 6 public universities (including the OUSL), 3 public institutes, 2 professional organizations and

Design and Implementation 9 13 private postsecondary institutes. A list of the partner institutions, online courses, and 2011 student numbers is in Appendix 5. The private institutions were selected on the basis of labor market demand, and included technical colleges (offering skills development training), private non-profit professional organizations, and private for-profit schools (mostly offering language training, ICT, accounting, business, and commerce). The project provided matching grants to partner institutions to help with the start-up costs for online course development. The requirement to commit funds for these matching grants may have discouraged the private institutions from participating in this new venture. Despite having no financial requirement, the public universities were reluctant to commit to the new venture.

31. Interest in online education among partner institutions did increase during the final year, evidenced by the faster pace of course development at that time. It is possible that a project extension or a follow-on project may have allowed these benefits to be captured and further developed. The opportunity to link up with foreign institutions and offer online programs was also available for 150 institutions, but only two took up the offer. Stipends were available for 20,000 needy students undertaking online skills development courses, but the target for this type of training was too high, and the number of stipends was reduced to 8,000 at the midterm review.

32. Various policy initiatives were planned to facilitate and support the participation of private partner institutions, but were not fully achieved. The most important was a national policy for the assessment and accreditation of distance higher education and an ICT policy for online distance learning. The accreditation policy was developed under the project but was taken over for further development by the National Quality Assurance Agency for Higher Education. The policy developed under the project included accreditation standards, evaluation criteria, an accreditation framework and performance indicators that were considered so impressive by the Commonwealth of Learning that they were adopted for international use. Despite this, the government has approved neither the policy nor the establishment of the accreditation board for distance education, so it is still not possible for private institutions or programs to be accredited. A proposed policy on online education with provision for subsidized internet fees was planned, but it was agreed that this would be undertaken later as part of a broader ICT education policy.

33. Capacity expansion of the Open University of Sri Lanka. The lynchpin of the project was the OUSL, the only partner institution with experience in distance education. To expand the capacity of the OUSL, modern facilities and advanced laboratories for science students were established at the OUSL central campus. The library, multimedia center, and the printing press were upgraded to support the development of the curricula for multimedia online courses by 2006. Academic, technical, and administrative staff members were taught new skills and concepts through an intensive program of staff development, including post-graduate overseas training, long-term domestic training, and short-term overseas training. A management information system was put in place to strengthen and streamline administration, human resources management, budgeting, financing, student admissions, and student services. All of these systems were operating at the time of the PCR. In addition, as part of the physical network, 16 OUSL regional centers and 7 new study centers were built, and 3 existing study centers were renovated. This met the original target of 26 OUSL centers. New centers were established in rural and conflict areas and in poor plantation districts.

10 Distance Education Modernization Project 34. Existing OUSL courses were upgraded through the development of new instructional materials. Staff from all faculties attended training on online course development. As a result, 69 online courses were developed and added to the OUSL program as supplementary courses, of which two were fully online courses. All of these achievements reflect the determination of the OUSL leadership to modernize and move beyond traditional print-based distance education into a new era of e-learning.

F. Consultants

35. The consultants were recruited in three packages, one each for the OUSL, DEPP and NDEN. Recruitment of the consultants for the DEPP and the NDEN was delayed by 9 months. This problem was compounded by the need for several subsequent consultant replacements in the DEPP and OUSL packages. The delay in the recruitment of the NDEN consultants was a major cause of the holdup in establishing the network. Consultants were engaged for a total of 751.8 person months, a significant increase over the appraisal estimate of 552 person months. International consultants were increased by 14 person months, and national consultants by 176 person months. The increased need for consultants reflected the many challenges encountered in developing the network, especially the difficult task of modifying course content for online delivery. In addition to the three consulting packages, individual consultants were engaged for 61.5 person months (of which 59.5 person months were national), primarily to support the modification of course content for online learning systems.

36. Several consultants did not have the required technical skills and had to be replaced. The replacements generally took a long time, which delayed implementation. Finding an appropriate consultant to develop the online course for career guidance proved especially difficult. The consultants generally performed satisfactorily. They had cordial and professional relations with the staff of the executing agencies and their counterparts, technology transfer was effective, and a positive exchange of ideas took place. The international consultant inputs were staggered because the consultants were in high demand, which sometimes caused delays. Identifying local consultants with expertise in instructional design, management information systems, and network technology proved challenging, but those finally engaged proved competent and committed.

G. Loan Covenants

37. The PCR reports that, of the 31 covenants, 23 were assessed as fully or partially complied with, while 18 were assessed as complied with late. One covenant regarding the executing agency was modified following a change in executing agency, and another, concerning a tracer study to examine the employment of graduates from the online programs, was dropped during the midterm review because it was judged to be premature.

38. Covenants relating to the engagement of consultants, procurement of equipment, and civil works were complied with. Project accounts and audit reports were submitted to ADB in a timely manner. Completion of a baseline study of the internal and external efficiency of the OUSL, to determine future growth and funding requirements, was delayed by almost 3 years, mainly because the executing agency delayed the appointment of OUSL staff to the study. Only one covenant was assessed as partially compliant, covering implementation of the government’s national ICT policy in relation to distance education. While a policy paper on ICT in distance education was prepared, the government decided to subsume it into a policy covering ICT in

Design and Implementation 11 education more broadly. A covenant covering the conduct of the public awareness campaign for promotion of the online courses, and establishment of a career guidance program with a focus on women was assessed as compliant even though the public awareness campaign was ineffective and took place late in the project, and the labor market information system that was to be the basis for the career guidance system was dropped. A more rigorous assessment of compliance may have resulted in more covenants being assessed as only partially compliant.

H. Policy Framework

39. The government in place at the time of project design and approval was committed to modernizing the economy and the labor market through a flexible education system with special emphasis on minimizing differences across regions, especially through ICT and new learning technologies. The government’s policy framework set out in The Future – Regaining Sri Lanka signaled its intention to pursue financial investment, public sector reform, and infrastructure development and productivity in key sectors (footnote 6). A central part of government policy was the modernization of the labor force though education and training for lifelong employability, support for self-employment, and encouragement of job creation for small and medium enterprises. At the same time, the government was also investigating education finance reform and public-private participation in tertiary education. Another key vision of the government was of using ICT to provide knowledge connectivity throughout the country. E-Sri Lanka: An ICT Development Roadmap set out a program to (i) develop national hardware and software infrastructure through a specialized agency for ICT strategy and policy, (ii) develop ICT human resource capacity, (iii) use ICT for broad-based tertiary education across the country, and (iv) move toward e-government service delivery (footnote 6). Despite the failure of the 2002 ceasefire agreement, the economy was continuing to expand and the pressure to develop a workforce more suited to the needs of the growing private sector remained strong.

40. The project was closely aligned with all of these policies and enjoyed strong support from key ministries, such as the ministries of planning and finance. However, a change of government in April 2004, just a few months after project approval, reduced this strong support. The new government did not have the same interest in ICT or in advancing online learning. A further blow came with the retirement of the project’s champion, the vice-chancellor of the OUSL, about halfway through project implementation. While OUSL leadership remained committed to the project, the drive to achieve reform across the distance education landscape was diminished. Both of these factors adversely affected the success of the project, as well as the fate of the strong policy agenda designed to support ICT and promote public-private participation in education. Finally, ADB did not demonstrate a consistent and strong commitment to the project, and failed to ensure that adequate technical support was provided.

CHAPTER 3

Performance Assessment 41. The overall assessment is based on the assessment criteria for relevance, effectiveness, efficiency, and sustainability, in accordance with ADB guidelines.7

A. Overall Assessment

42. Overall, the project is rated less than successful. The ratings for the individual criteria are shown in Table 1. The likely impact of the project, as well as the performance of ADB and the borrower, is assessed but not included in the final rating. The rating of less than successful is consistent with the assessment of the PCR and PVR (see Appendix 6 for a comparison of the PCR, PVR, and PPER assessments on the individual criteria).

Table 1: Assessment of Project Performance

Criterion Weight Assessment Rating Value Weighted

Rating Relevance 25 Relevant 2 0.50 Effectiveness 25 Less than effective 1 0.25 Efficiency 25 Less than efficient 1 0.25 Sustainability 25 Less likely sustainable 1 0.25 Impact Less than satisfactory ADB performance Less than satisfactory Executing agency performance

Less than satisfactory

Overall assessment Less than successful 1.25 ADB = Asian Development Bank. Note: Highly successful (>2.5), successful (1.75), less than successful (0.75), unsuccessful (

Performance Assessment 13 tertiary education. The project interventions were well conceived and generally appropriate, and were responsive to the needs of the sector. However, the project did not assess the demand for online education in sufficient depth, and did not include adequate measures to convince universities and external students in particular of the value of online education. The project was complementary to the activities of other development partners, particularly the World Bank. It also complemented other ongoing ADB projects aiming to improve the employability of school-leavers and university graduates through greater exposure to ICT, science, and English in secondary schooling, as well as the successful expansion and upgrading of the postsecondary skills development system.

44. Project justification. At the time of project design, only 3% of the relevant age group was enrolled in the conventional university system, and enrolments had stagnated. The participation rate has increased since then but remains very low, with less than 7% of the age group enrolled in 2014.9 The government was both unwilling and unable to dramatically expand capacity. Other avenues for tertiary education were also limited. The RRP argued that the high level of enrolments in external degree courses indicated a demand for better quality distance education. In the search for alternatives, the project offered a practical and cost-efficient option for rural students in particular to pursue certificate, diploma, and degree programs. The OUSL already offered tertiary education opportunities to a large number of students, but the programs were all print-based. The project took advantage of this infrastructure and the recent innovations in ICT to revitalize the OUSL and establish an expanded national delivery system for online distance education. This involved both public and private partners, offering more employment-relevant training. A set of policies to support the inclusion of private institutions was also proposed. The project was an appropriate and relevant response to the limited access to tertiary education at both the beginning and end of the project. The involvement of skills training institutions offered a further range of job-relevant alternatives for school leavers at the certificate level.

45. Project design. The project outputs corresponded to the three major thrusts of the project and were linked clearly and logically to the project outcome. The link between the project outcome and the project goal or impact was less direct, as many factors intervene between the expansion of postsecondary education and increased socioeconomic growth. One important aspect of the design can be questioned: whether Sri Lankan students were ready for a purely online modality. Sri Lanka’s levels of internet penetration were low, at 3.7% of the population in 2008. In a survey of 70 Asian countries, Sri Lanka ranked 60th on e-learning readiness in 2003 and 59th on e-readiness in 2008.10 In the short survey administered to OUSL students as part of the PPER, half of the responses identified face-to-face teaching as the preferred instructional method. A further 20% preferred online learning supported by an online tutor, while only 10% preferred solely online instruction. These responses are not surprising. Many of the officials and academics interviewed for the PPER also commented that Sri Lankan students have a strong preference for face-to-face teaching and learning. This is likely to be true of the teachers as well, who generally do not have the skills needed to design and deliver online courses. In the context of distance education, where full face-to-face teaching is not an option, an approach blending online learning with face-to-face teaching seems most appropriate. These critical issues should have been explored in much greater depth before the project was designed.

9 University Grants Commission. 2015. Sri Lanka University Statistics 2014. Colombo. 10 ADB. 2012. Access without equity? Finding a better balance in higher education in Asia. Manila.

14 Distance Education Modernization Project 46. Finally, the design of the project was too broad, and many of the targets and time frames were too ambitious for a project introducing complex new infrastructure, new technologies, and new ways of designing and delivering courses—particularly in a context where there was some resistance. The project period should have been longer to allow time to achieve the targets. Alternatively, it may have been better to start with just the OUSL as a first step, moving to the establishment of the national network if online education through the OUSL proved a success. As it was, at the midterm review, the project outcome and output targets for the public and private partner institutions had to be reduced significantly. The targets for the OUSL were not changed. The changes made to the project targets were appropriate given these delays, and the new targets were met. However, the large reductions inevitably limited the impact of the project. No major changes were made to the project scope.

47. Risk analysis and risk mitigation. Any project pursuing significant innovation or reform is inherently risky. The DEMP was bold in its solution to the problem of restricted tertiary opportunities, and incorporated many new elements including new technologies and new partnerships. It also faced resistance from various vested interest groups. Little in the way of past lessons were available to guide the design, as ICT-enabled education had not been attempted in Sri Lanka before. Nevertheless, the project design and monitoring framework managed to identify most of the major risks (described as assumptions) to the project’s success. These assumptions included: (i) that partner institutions would have the motivation and capacity to effectively employ distance education, (ii) that universities would be willing to participate in a joint venture rather than proceeding alone, (iii) that external degree students would recognize the quality and convenience of NODES, (iv) that NODES would provide high-quality programs that would attract students, (v) that the government would implement a public-private partnership policy in postsecondary education, (vi) that public and private institutions would cooperate to improve capacity, and (vii) that the OUSL would be willing to modernize. All of these risks were adequately identified; however, the key risk of the acceptability of online distance education was not recognized.

48. The strategies for risk mitigation were less adequate. A central assumption was that the demand for online distance education would grow rapidly as an automatic response to the huge unmet demand for tertiary education. However, in the 5 years prior to the project, the OUSL had suffered from declining enrolments in its distance education courses, a fall in its reputation, and marginalization from the other universities. While the project aimed to rejuvenate the OUSL, it was likely that the years of decline had given distance education in general a bad name that would take time to counter. Students and their parents were likely to be especially cautious given that fees are paid for all distance education courses. Additionally, no interventions were made to convince universities to put their courses online, nor to persuade external degree students to take up online studies under NODES. The design did include a limited information and awareness-raising campaign to help stimulate student enrolments, but it was ineffective and did not succeed in generating the level of demand expected. Building demand for the new courses was always going to be difficult, given that no prior programs existed by which students and parents could judge the employability of graduates. This should have been a major focus of early project, or even pre-project, activities.

49. Government ownership. Ownership of, and commitment to, the project by the government and the OUSL was high at the time of project design and initial implementation. However, this was weakened over time by political changes and

Performance Assessment 15 ministerial restructuring that led to frequent changes of executing agency and leadership of the project. Over the 6 years of implementation, the project was moved among three different ministries and ministers, and was led by six ministerial secretaries. This made it difficult for the project to establish a clear institutional home and diminished the level of ownership and commitment to the project by key government officials.

50. Continuing relevance of the project. The project focus on identifying alternative avenues and modes of study for higher education, with emphasis on ICT-enabled distance education, remains relevant. The restrictions in access to higher education in Sri Lanka have barely eased, as shown in Table 2. Of the almost 150,000 students who passed the final school examinations (GCE A-Level) in 2014, only 25,200 or 16.8% were admitted to conventional university studies (as on-campus or ‘internal’ students), and another 6,400 or 4% enrolled at the OUSL. This leaves around 80% of the students who passed their A-Levels without any satisfactory options for higher education. Many registered as external university students (43,000 or 30% of students who passed their A-Levels), but the quality and employment outcomes of these programs are extremely poor. This left more than 75,000 students who could not access university studies, in addition to a further 97,000 who did not pass the A-Level examinations and required other forms of tertiary training. There remains good potential for effective distance education to expand educational opportunities to these groups, in fields with good employment prospects, whether fully online or through blended learning with some face-to-face teaching. Online education has exploded in popularity around the world, and Sri Lankan students are now also studying online with private institutions. Different technologies for content delivery, such as mobile phones and tablets, are promising options for countries with low internet usage and coverage.

Table 2. Intake and Output for Public Universities, 2014

Intake Graduate Output

Item Total students

Total Females

% Female

Total Students

% Female

Graduation rate (% of

intake)

A-level students 2014 247,376 143,604 68 149,489 65 60

Students not passing 97,887 46,653 48

Admitted to Degree Programs 2014

On-campus (conventional university studies)

25,200 15,694 62 15,518 61 55

Off-campus (external students) 42,755 32,843 77 11,673 77 41

OUSL 6,367 3573 56 1,041 62 4

Total 74,322 52,110 70 28,231 68 100

A-Level graduates not admitted to degree programs

75,167 27,729 57

% admitted as on-campus students

16.8% 16.2%

% admitted all students (on-campus and off-campus)

50% 54%

OUSL = Open University of Sri Lanka. Notes: Internal students study on-campus and attend lectures, tutorials, seminars, etc. as conventional university students. External students register with a university, receive a syllabus, and then study off-campus independently or through a private tutoring company. Sources: University Grants Commission. 2015. Sri Lanka University Statistics 2014. Colombo; Open University of Sri Lanka. 2014. Statistical Handbook 2014. Colombo.

16 Distance Education Modernization Project 51. In sum, the project is rated relevant because it responded to government and ADB national and sector objectives, and it addressed the key issue of tightly restricted access to tertiary education in an imaginative and innovative manner. The lack of access to tertiary education continues to be a key sector issue today. As a new and innovative project, it faced many risks to success, not all of which were mitigated adequately. The project design was too complex, too broad and, most significantly, did not anticipate the resistance to online education among both students and providers. This only became apparent once project implementation began. While the demand for more traditional distance education was steady, as evidenced by the growth in enrolments at the OUSL, demand for the new online education was weak. More groundwork and investigation was needed before launching such an ambitious investment project.

C. Effectiveness

52. This evaluation rates the project less than effective, as it was by the PCR and PVR. The project outcome was to develop distance education to expand postsecondary school enrolment, in order to develop a modern, high-quality human resource base. Although the project successfully created a national platform and online delivery mechanism for distance learning, utilization of the network by providers and students remained low. The project did not succeed in developing the online distance education system to the level where it could significantly expand postsecondary school enrolments or ease the bottleneck in postsecondary education.

53. The outstanding achievement of the project was the establishment of the NDEN, linking rural and urban regions across the country to Colombo. Online courses could be offered through NODES by both public and private institutions, operating within innovative postsecondary education partnerships established by the project, through a sharing of common resources and activities. The introduction of these partnerships was a noteworthy achievement, in a sociopolitical environment where cooperation with private institutions was not common. Together, these accomplishments meant that, by the end of the project, Sri Lanka had a robust national platform through which all postsecondary institutions could potentially offer online courses.

54. As it was clear that the original outcome targets relating to expansion and utilization of the network could not be met, the following targets were revised at the midterm review: (i) the number of students undertaking external degrees through the NODES system was reduced from 25,000 by 2012 to 10,000 by 2009; (ii) the target of 150 public-private partnerships operating among postsecondary institutions was replaced with a target of 40 programs to be offered by partner institutions by 2009; (iii) the expected increase in private school enrolments was reduced from 35,000 by 2012, to 10,000 by 2009; (iv) the number of unemployed school leavers receiving semiprofessional job training was reduced from 20,000 to 8,000, and the age limit increased from 21 to 30. The targeted increase in OUSL enrolments from 20,000 to 40,000 by 2012 remained unchanged.

55. The project succeeded in providing online degree, diploma, and certificate courses for 33,347 students by project end in 2009, which was one-third of the original target of 100,000 students by 2012, and 85% of the revised target of 40,000 by 2009 (see Table 3). To put this in context, the final enrollments were around 5% of all higher education enrollments. This was far from the original vision of NODES as a major

Performance Assessment 17 facilitator of tertiary education that would provide opportunities to 1.4 million students by 2020.11 Why was the original vision for NODES not achieved?

56. First, the unrealistically high targets would always have been impossible to achieve within the time frame, given the complexity and breadth of the design. This challenge was compounded by the technical difficulties involved in constructing a national high-speed virtual network and developing large numbers of online courses. Second, while some demand existed for online education, which did pick up toward the end of the project, the central issue was that student enrolments were much lower than anticipated. Although mass distance education was beginning to show success in other Asian countries (footnote 12), in Sri Lanka it faced resistance among students and wariness among education and training providers. Many factors explain the failure to attract high numbers of students, of which the most important are discussed below.

Table 2. Students Studying through the National Online Distance Education Service, 2009

2011 Enrolments Original Target

Revised Target Degree Diploma Certificate Total

OUSL 20,000 20,000 20,000 Public universities, external degrees

499 499 25,000 10,000

Private institutions 970 970 35,000 Private institutions, skills training

11,878 11,878 20,000 10,000

Total 33,347 100,000 40,000 Note: OUSL was targeted to increase from 20,000 students pre-project to 40,000 by 2012, an increase of 20,000. Source: Government of Sri Lanka. 2010. Distance Education Modernization Project. Project Completion Report. Colombo.

57. Difficulty of establishing partnerships. A major factor was the lack of success in forming partnerships between NODES and the institutions. This meant that only a small number of online courses were available by the end of the project. The government completion report notes that it was extremely difficult to draw the leadership and staff of universities and public and private institutions into online learning. Even the generous grants available for course development and purchase of materials failed to attract them. The RRP targeted external students as a large source of potential course participants, but did not foresee that the universities would resist putting their courses online to the extent that they did. Officials knew little about online education and were wary of the costs of providing ongoing learner support, which they felt would be too high for students. 12 University investment priorities generally lay elsewhere. The University of Moratuwa, which developed a well-respected and popular fully online degree program in informatics, was one outstanding exception. Some private institutions were initially enthusiastic but found the development process protracted and the financial returns too slow. Given the costs of developing online courses, the matching grant funds required, and the small size of many private institutions, it is not surprising that these institutions were reluctant. Online courses must attract large numbers of students to be profitable.

58. While the decision to include public, private, and professional institutions and organizations in NODES was one of the highlights of this innovative design, it also

11 Footnote 1, p 16. 12 Interviews during the PPER mission; and C. Pereira. Online Learning and Examination System for State

University Entrants. Pre-Orientation Programme. In ADB. 2013. ICT for Development Forum 2013. Manila.

18 Distance Education Modernization Project proved the most difficult to implement. Different approaches to implementing the partnerships could have been trialed before or during the project. More effort was also needed to help overcome their initial resistance.13

59. Slow development of the online courses. A second factor was that the slow development of online courses meant that most became available only in the last year of the project. The first step was to identify courses relevant to labor market needs, which was done through a lengthy labor market survey. The second step, the development of the online courses, is always time-consuming and requires special expertise, a fact that was initially not appreciated by partner institutions. This meant that the project staff first had to do a lot of awareness-raising. Through this process, the project built significant expertise in content development across institutions through the training of 900 staff; nevertheless, the slow development of courses had a significant negative impact on project success.

60. Accessibility and affordability. While it is unlikely that physical accessibility was a major deterrent given the comprehensive network of centers across the island, having all the courses in English would have limited accessibility for rural students. The fees charged for distance education programs are also likely to have been a deterrent for some students, especially non-working students (60% of the students surveyed paid the fees from their salaries).

61. Preference for degrees from conventional universities. Finally, students in Sri Lanka have a strong preference for a conventional university education, which partly reflects the strong preference for public sector jobs. In line with its aim to increase the market relevance of tertiary education, the project favored vocationally oriented certificates and diplomas in addition to the degree programs. It should have been expected that demand for these courses would be low, at least initially. While a degree has a recognized value, this was not the case for the new online diplomas and certificates. In the absence of an accreditation system for distance education, students would wait until the value of such courses was demonstrated through employment outcomes. The lack of external validation would have harmed the potential for private institutions especially to attract students to their courses.

62. The PCR observes that a comprehensive social marketing campaign would have helped generate demand. Some outreach was conducted to officials but the social marketing to students was limited and less effective. It is possible a campaign would have stimulated greater demand to some extent, but caution on the part of students and parents would probably have persisted anyway, especially as the courses were fee-paying and their pay-off in the labor market was not yet evident. Strong demand for courses is generated once they are seen to be successful.

63. In summary, the project had many noteworthy achievements, albeit on a smaller scale than originally intended. By the end of the project, it had provided opportunities for over 33,000 tertiary students in online courses that had been identified as providing skills relevant in the labor market. This was around one-third of the original target and just below the revised target of 40,000 students. The complexity of the project, resistance and caution among the intended partner institutions, the late uploading of online courses, and hesitation about their value on the labor market

13 ADB. 2013. Is ICT the Missing Link in Development Operations? Session on ICT for Education. ICT for

Development Forum 2013. Manila. 28 February–1 March.

Performance Assessment 19 (which was as yet unproven) were the primary factors behind the failure to attract large numbers of students to online education.

D. Efficiency

64. The project is rated less than efficient, which concurs with the assessments of the PCR and PVR. The PCR did not recalculate the economic internal rate of return (EIRR) because the low enrolments rendered it highly unlikely that the EIRR envisaged at appraisal would have been achieved. For the same reason, the decision was taken not to recalculate the EIRR for the PPER.

65. Implementation was handled reasonably efficiently given the challenging nature of the project, in particular, the development of the high-speed optical fiber network. But the delays in establishing the network and developing courses resulted in too little time for courses and enrolments to grow.

66. The National Online Distance Education Service was not cost efficient. One reason distance education is attractive to policy-makers is its perceived cost efficiency. Large numbers of fee-paying students can offset the significant costs of course development and operational costs. In 2011, when NODES was still functioning, the PCR noted that the courses were not cost efficient because of the high costs of internet provision in Sri Lanka. While the DEMP was in operation, all of these costs were covered, but they had to be paid by the partner institutions after project closure. This was unaffordable at the time, given the low enrollments, but this did not mean that NODES would not be cost efficient in the future. The business plan prepared by the DEMP estimated that it could become self-financing in about 6 years, with 16,000 course registrations.

67. The Open University of Sri Lanka proved more cost efficient than the National Online Distance Education Service. From 2003-2014, the OUSL increased its level of cost recovery from 25% to 42%, which exceeded the project target of 40%. Between 2003 and 2014, fee revenue increased more than nine times, while unit costs increased around 2.5 times (see Appendix 7). This was possible partly because of the increase in the number of students that occurred at the same time, which indicates that the OUSL could become even more cost efficient if enrollments were expanded further. However, a balance must be found between cost efficiency and affordability. The current level of OUSL cost recovery appears to be on the cusp of affordability for rural students. Half of those surveyed for the PPER considered the fees charged for their OUSL courses to be manageable, while the other half considered them to be too high. A greater proportion of students in plantation areas would probably find them too high. Therefore, in light of the high costs of internet access through the NACs and the capacity of rural students to