HRD sector evaluation - UNICEF · 2019-06-07 · The Programme was implemented by UNICEF CO in partnership with the MoE and local implementing partners. The Programme had an initial

MONTENEGRO

Evaluation of the Programme “Montenegro –

Investment case on Early Childhood Development”

Final Evaluation Report (vol.1)

18 April 2017

Prepared for UNICEF Country Office in Montenegro by:

Camelia Gheorghe, Team Leader

Ajsa Hadzibegovic, National Consultant

Evaluation of the Programme ”Montenegro – Investment Case on Early Childhood Development” Final Evaluation Report, 18 April 2017

2

ACKNOWLEDGEMENTS

The evaluation team acknowledges the valuable guidance and support provided by the

management and staff of UNICEF Country Office in Montenegro throughout the entire

evaluation process.

The evaluation team also acknowledges the information and data generously provided by the

Ministry of Education, Bureau for Education Services, Ministry of Labour and Social Welfare,

Ministry of Health, Ministry of Finance, Parliament, Council for Regulatory Reform and

Business Environment Improvement, National Council for Children, Pedagogical Centre of

Montenegro, academia, public and private kindergartens, civil society organizations,

international development partners, professionals and parents during interviews, focus groups

and site visits, which were used as part of the analysis.

The evaluation acknowledges the process-oriented approach and continuous partnership

between the Ministry of Education and UNICEF in advancing the reforms aimed to increase

the coverage of children in quality and inclusive early childhood education (ECE). In this

respect, the Programme which was the object of this evaluation is only one of the many

initiatives of the Ministry of Education in the area of ECE carried out in partnership with

UNICEF. The evaluation does not assess the overall work of the Ministry of Education in

inclusive education or of UNICEF Country Office, but a very particular investment case in ECE.

The contents of this evaluation report is the sole responsibility of the contractors and can in no way be taken to reflect the views of UNICEF.


3

Title: Evaluation of the Programme “Montenegro – Investment Case

on Early Childhood development”

Timeframe of the object evaluated: January 2015 – December

2016 (extended until June 2017)

Timeframe of the evaluation: August 2016 – March 2017 Date of the report: 18 April 2017 (final) Country: Montenegro Evaluators: Camelia Gheorghe, International Consultant (Team Leader) and Ajsa Hadzibegovic, National Consultant

Name of the organisation commissioning the evaluation: UNICEF Country Office in Montenegro Name of UNICEF staff contact point for the evaluation: Marija Novkovic, Social Policy Officer Maja Kovacevic, Education (Youth) Officer


4

TABLE OF CONTENTS

Acronyms ............................................................................................................................................... 6 Executive Summary .............................................................................................................................. 7

1. Context of the Evaluation .............................................................................................................. 12 2. Object of the Evaluation ................................................................................................................ 20 3. Objectives, Purpose and Scope of the Evaluation ...................................................................... 26 4. Methodology .................................................................................................................................... 28

4.1 Evaluation Phases ...................................................................................................................... 28

4.2 Evaluation Design ....................................................................................................................... 31 4.3 Stakeholders’ participation in evaluation ..................................................................................... 34 4.4 Evaluation Limitations ................................................................................................................. 35

5. Findings and Analysis of Results .................................................................................................. 37

5.1 Relevance ................................................................................................................................... 37 5.2 Effectiveness ............................................................................................................................... 41 5.3 Efficiency ..................................................................................................................................... 48

5.4 Impact .......................................................................................................................................... 53 5.5 Sustainability ............................................................................................................................... 56

6. Conclusions and Recommendations ............................................................................................ 60

6.1 Conclusions ................................................................................................................................. 60

6.2 Recommendations ...................................................................................................................... 62 6.3 Lessons Learnt ............................................................................................................................ 68

LIST OF TABLES

Table 1. Limitations to the Evaluation and Mitigation Approaches........................................................ 35 Table 2. Beneficiary children of outreach interactive services (status at 30 November 2016) ............. 40

Table 3. Achievement of Outcome 1-related indicators ........................................................................ 42 Table 4. Achievement of Outcome 2-related indicators ........................................................................ 43

Table 5. Beneficiary children of outreach interactive services, by municipality (November 2016) ....... 44 Table 6. Achievement of Outcome 3-related indicators ........................................................................ 45 Table 7. Status of Outreach Interactive Services (November 2016) ..................................................... 57 Table 8. List of Recommendations ........................................................................................................ 63

LIST OF FIGURES

Figure 1. Poverty rate in Montenegro (%), 2006-2014 .......................................................................... 12

Figure 2. Enrolment rate of children aged 3-6 in ECE by municipality (%), 2012 ................................. 14 Figure 3. Proportion of children 3-6 years of age attending ECE (%) by wealth quintiles .................... 14 Figure 4. Theory of Change................................................................................................................... 21

Figure 5. Municipality development index and the coverage rate of children in ECE, 2012 ................. 40 Figure 6. Increase in enrolment in preschool following 2015 campaign (no. of children and %) .......... 46 Figure 7. Increase in enrolment in preschool following 2016 campaign (no. of children and %) .......... 46 Figure 8. Children enrolled in ECE in target municipalities in school year 2015/2016.......................... 54 Figure 9. Increase rate of enrolled children in target municipalities in school year 2015/2016 (%) ...... 54

Figure 10. Effects of the Programme on beneficiary children (parents’ perspective) ........................... 55 Figure 11. Effects of the Programme on beneficiary children (professionals’ perspective) .................. 56 Figure 12. Effects of the Programme on beneficiary children (evaluators’ perspective) ....................... 56


5

ANNEXES (IN VOLUME 2)

Acronyms ............................................................................................................................................... 6 Executive Summary .............................................................................................................................. 7

1. Context of the Evaluation .............................................................................................................. 12 2. Object of the Evaluation ................................................................................................................ 20 3. Objectives, Purpose and Scope of the Evaluation ...................................................................... 26 4. Methodology .................................................................................................................................... 28

4.1 Evaluation Phases ...................................................................................................................... 28

4.2 Evaluation Design ....................................................................................................................... 31 4.3 Stakeholders’ participation in evaluation ..................................................................................... 34 4.4 Evaluation Limitations ................................................................................................................. 35

5. Findings and Analysis of Results .................................................................................................. 37

5.1 Relevance ................................................................................................................................... 37 5.2 Effectiveness ............................................................................................................................... 41 5.3 Efficiency ..................................................................................................................................... 48

5.4 Impact .......................................................................................................................................... 53 5.5 Sustainability ............................................................................................................................... 56

6. Conclusions and Recommendations ............................................................................................ 60

6.1 Conclusions ................................................................................................................................. 60

6.2 Recommendations ...................................................................................................................... 62 6.3 Lessons Learnt ............................................................................................................................ 68


6

ACRONYMS

Acronym Description

BfES Bureau for Education Services

CEDAW Convention on the Elimination of all Forms of Discrimination against Women

CEE/CIS Central and East Europe / Community of Independent States

CO Country Office

CRC Convention on the Rights of the Child

CSO Civil Society Organisation

DCT Direct Cash Transfer

ECD Early Childhood Development

ECE Early Childhood Education

ECEC Early Childhood Education and Care

EQ Evaluation Question

EU European Union

FACE Funding Authorization and Certificate of Expenditures

GDP Gross Domestic Product

GLE General Law on Education

HACT Harmonized Approach to Cash Transfers

HMCF H&M Conscious Foundation

HQ Headquarters

HRBA Human Rights-Based Approach

KAP Knowledge, Attitudes, Practices

LIP Local Implementation Plan

LPSE Law on Preschool Education

M&E Monitoring and Evaluation

MDG Millennium Development Goals

MEIS Montenegrin Education Information System

MICS Multiple Indicator Cluster Survey

MoE Ministry of Education

NGO Non-governmental Organisation

NYHQ New York Headquarters

OECD Organization for Economic Cooperation and Development

PISA Programme for International Student Assessment

PSI Preschool Institution

RE Roma and Egyptian

SDG Sustainable Development Goals

SEN Special Educational Needs

RBM Results-Based Management

ToC Theory of Change

ToR Terms of Reference

UN United Nations

UNDP United Nations Development Programme

UNEG United Nations Evaluation Group

UNESCO United Nations Educational, Scientific and Cultural Organization

UNICEF United Nations Children’s Fund


7

EXECUTIVE SUMMARY This document represents the Evaluation Report of “Montenegro - Investment Case on Early Childhood Development” (hereinafter referred to as “The Programme”), part of a broader collaboration between UNICEF Headquarters (HQ) and the H&M Conscious Foundation (HMCF) entitled “Unleashing Children’s Potential”. The evaluation was conducted between August 2016 and March 2017. Purpose and Objectives of the Evaluation

The purpose of the evaluation was multi-fold: a) to improve accountability for the achievement of results and use of resources; b) to provide recommendations for the scaling-up of the tested services for children aged 3-6 at national level, based on documentation of good practices and lessons learnt; c) to help ensure that the focus is kept on issues of equity, inclusion and quality in the process of implementing the Strategy on Early Childhood and Preschool Education 2016–2020; d) to inform the potential replication of Montenegro investment case in other countries, including by way of capturing the changes at policy and programmatic levels. The evaluation had both summative and formative elements. The main objective of the evaluation was to assess the final (end) results of the Programme in relation to its objectives, the ToC and indicators in the Results Framework. The specific objectives were the following:

1. Assess the relevance, effectiveness, efficiency, sustainability and preliminary impact, i.e. longer term contribution, of the Programme;

2. Extract general lessons learned and provide recommendations aimed to inform the future enhancement of the initiative;

3. Assess the potential for replication and scaling-up of the Programme.

The intended main beneficiaries of the evaluation were UNICEF Montenegro Country Office (CO), the Ministry of Education (MoE) and Bureau for Education Services (BfES), preschool institutions (PSIs), local communities, civil society organisations (CSOs), academia, UNICEF HQ and HMCF. Evaluation Object

The object of the evaluation was the programme “Montenegro - Investment Case on Early Childhood Development”. The Programme aimed to contribute to the improvement of coverage of children aged 3-6 in inclusive and quality pre-school education and their learning outcomes, in line with international human rights standards and the principle of progressive realisation of children’s rights, by achieving three outcomes: enabling policy framework for ECE expansion country-wide; improved access to new ECE services for children aged 3-6 in the targeted municipalities, in particular of vulnerable children; and increased demand of parents/ caregivers for ECE programmes.

The target groups of the Programme included: high-level decision-makers from education, health, social and child protection sectors; professionals working in these sectors whose capacity was strengthened to deliver quality ECE outreach interactive services and alternative 3-hour programmes; parents/caregivers who were informed about the development and learning benefits of enroling their children in preschool education; 12 municipalities where new ECE services have been implemented; and the general public, whose awareness was raised concerning the importance of ECE. The final beneficiaries were preschool age children (3-6 years of age), in particular those living in the poorest municipalities located in the Northern region of Montenegro where the ECE enrolment rate was the lowest in the country.

The Programme was implemented by UNICEF CO in partnership with the MoE and local implementing partners. The Programme had an initial duration of two years (January 2015-December 2016) which was extended with six additional months until June 2017. It had a budget of US$ 406,546, of which US$ 225,000 (55.3%) secured from the grant provided by HMCF and US$ 181,546 (44.7%) representing matching funds from UNICEF. Evaluation Methodology

The evaluation applied a mixed-method approach, including: stakeholder mapping; mapping of situation and contextual analyses, barriers and bottlenecks; in-depth documentary review and structured desk analysis of Programme design, implementation approaches, documenting of results and processes; structured desk analysis of policy documents and legislative frameworks; analysis and testing of the revised Theory of Change (ToC); analysis of results from M&E systems and data at national and municipality level; aggregation and analysis of


8

data collected via the Data Sheet and various mapping templates; contribution analysis to determine factors which promoted or impeded the progress against intended results and attribution analysis to the extent possible; systems analysis of management, monitoring, quality control and assurance strategies; mapping of risks analyses and mitigation measures; financial analysis; analysis of sustainability strategies and systemic barriers to sustainability; in-depth interviews and focus groups; direct observation and guided discussion with children during site visits to municipalities. The evaluation was carried out in three phases. In the Inception Phase, review of key documentation and development of data collection and evaluation tools were done. The Data Review and Collection Phase was devoted to the in-depth documentary review and collection of data from key stakeholders at central and community levels during an in-country visit. In the Analysis and Reporting Phase, the evaluation applied the standard evaluation criteria analysis (relevance, efficiency, effectiveness, impact and sustainability) in combination with Human Rights-Based Approach and Results-Based Management in order to assess the achievement of results, draw informed conclusions and provide recommendations which were validated by the key stakeholders during a workshop. Main Findings and Conclusions

The Programme has won an argument in the public space on the need to invest in ECE by steering informed policy debate and building support in the society for ECE for which UNICEF CO and the MoE, the key driver of the Programme, deserve high praise. Rather than focusing on a specific bottleneck, the two partners opted to implement a comprehensive and concerted action upon all determinants of ECE expansion to produce system change for improved outcomes for children.

The Programme implemented new ECE services where they were unavailable (outreach interactive services in rural North); it campaigned where existing capacities were underutilised in order to create demand (in urban North); and it diversified the ECE service menu in areas with over-demand and over-crowded capacities (alternative 3-hour). Until the date of the evaluation, the immediate net impact was 613 children enrolled in ECE due to the Programme, a number which was planned to increase to 963 children once the interactive services are expanded across the whole North region.

The investment in improving the service provision has been supported by a strong and tireless policy advocacy of UNICEF CO, grounded in solid evidence and based on robust and easy-to-understand arguments. The advocacy efforts have driven a fundamentally different level of debate and understanding on ECE compared to the past, even among stakeholders not necessarily associated or familiar with this area. The result was the creation of prerequisites for an enabling policy framework for universal coverage of children in ECE programmes by 2020, in particular the adoption by the Government of the Strategy for Early Childhood and Preschool Education 2016–2020, which foresees an increased allocation to ECE from 0.38% of GDP to 0.53%, and Action Plan.

More specific findings and conclusions for each evaluation criteria are presented below.

Relevance

The Programme was fully in line with the national priorities in the area of preschool education as well as with the European and international human rights commitments of the country. The ECE services designed, tested and implemented by the Programme proved to be highly relevant for the needs of children and parents. The Programme remained relevant in time, as demonstrated by the strategies and policy documents developed during its lifetime and which the Programme has influenced, and the persistently high demand of parents for ECE services.

The new ECE services and awareness campaigns have increased the number of children receiving early education, including poor children living in rural areas who were the focus of the Programme. At the same time, it had a rather limited relevance for other vulnerable children, in particular children with disabilities and RE children, especially due to the limited duration and resources of the Programme which impeded a much broader scope.

Effectiveness

Implementation of the Programme has delivered valuable results in terms of: a) developed strategic building blocks for ECE expansion country-wide; b) increased access of 613 children aged 3-6 to early learning opportunities in target municipalities, especially of children living in rural areas; c) improved awareness of parents on the importance of ECE, translated into increased enrolment of children aged 3-6 in preschool (5 in 10 parents started the procedure of enrolling their children in kindergarten following the campaign in 2016); and d) strengthened capacity of professionals to run new ECE services and apply child-centred methodologies in their work with children. With few exceptions, the overall


9

effectiveness level of the Programme was found to be adequate and close to what was planned by its designers.

Impact

The Programme increased the overall number of children enrolled in public ECE programmes in target municipalities by 7% to 55% in the school year 2015/2016 (one year after the start of the Programme), the highest impact being recorded in the areas which needed it the most.

At a higher-level results in the ToC, an objective, quantitative assessment of the Programme contribution to learning and development outcomes for children could not be done due to the lack of key prerequisites and resources explained in the section on limitations of the evaluation. Nevertheless, parents, professionals and the evaluation team observed improvements in the language, socio-affective, motor and, to a limited extent, cognitive development of children enrolled in outreach interactive services, which are likely to have been contributed to enhanced learning and development outcomes of the beneficiary children. The qualitative assessment of impact indicates a positive contribution of the new ECE services to the progressive realisation of children’s rights.

Efficiency

The Programme has been implemented in a professional and culturally-sensitive manner. With few exceptions, the funds were allocated and spent in an accountable and cost-effective way. Among the efficiency features figures the use of available premises in rural schools for the delivery of ECE services to children, building the training curricula and endowment specifications on the ones developed in a previous project of UNESCO, and centralised procurement of toys and didactical materials for all target municipalities in order to minimise the costs, ensure unitary endowment of all outreach interactive services and permit as far as possible coordinated implementation schedules at local level.

The average investment was around US$ 1,651/outreach interactive service, meaning an initial investment/child of around US$ 110. The unit cost/child for the 3-hour alternative programme was estimated at € 266.5 annually. The Programme invested US$ 730 for each child who was enrolled in ECE as a result of the awareness raising campaigns “Preschool for All”. The results of the campaigns were fully evaluable, which is rarely the case of campaigns carried out by UNICEF in other countries.

Efficiency of monitoring has been impeded to some extent by the lack of a clear hierarchy of results and targets in the Programme design documents and insufficient attention given to quality issues, in particular related to preschool infrastructure, on the background of system weaknesses related to quality assurance.

Sustainability

As far as sustainability is concerned, one year after their start up in the first six municipalities from North region, the outreach interactive services have been still functioning in 11 out of 19 locations (58%), while four were planned to restart soon and new ones were at the point to be established in other six municipalities in the North. The implementation of the outreach interactive services was instrumental in highlighting what could be done and how, by investing an average amount of US$ 110/child for endowment and capacity building of professionals. Expansion of the services to other municipalities would require a smaller investment since there will be no need to redevelop guides, training curricula, working methodologies and tools or technical specifications for endowment, as these are already available. The current policy and legal framework governing the preschool education system in Montenegro is supportive for the development and expansion of ECE services for children aged 3-6 which were piloted and put in place by the Programme, but their financial sustainability is under risk in some locations.

The testing of new ECE services in the first six municipalities drew the attention on some challenges which might be encountered during implementation and which should be addressed with priority before rolling them out to other municipalities and later on nationally, related to equity and quality.

Equity

The new ECE services and awareness campaigns have increased the number of children receiving early education, including poor children living in rural areas who were the focus of the Programme. At the same time, it had a rather limited relevance for other vulnerable children, in particular children with disabilities and RE children, especially due to the limited duration and resources of the Programme which impeded a much broader scope.

Quality


10

The learning infrastructure in some locations where interactive services are functioning is not adequate, impacting on the quality of the educational process. Due to the education system weaknesses, especially at the level of BfES, implementation of services was not followed by a proper assessment of the learning and development outcomes reached by the beneficiary children in order to be able to analyse the results and take corrective measures, as needed. The absence of children’s voice and views on their learning and the lack of parents’ involvement in monitoring the quality of interactive services on the background of insufficient attention given to the development of positive parenting practices were missed opportunities for increasing the quality of service provision and learning outcomes for children. Other elements which affected quality was staff turnover and no system in place for induction training of new comers or any other systematic form of in-service training, apart from some sporadic mentoring by the principal or pedagogue. These issues might have been addressed more efficiently in case the design of the Programme included a coherent results framework from start, with clear quality safeguards to be monitored during the implementation of services. A more efficient process monitoring would have been also very useful to address some of the quality issues arisen during implementation.

The Programme represents a significant step forward in the process of establishing the underlying policy framework and practices for broadening the access of children aged 3-6 to ECE. The evidence provided by the Programme is a powerful tool in the process of advocating for universal coverage of children in ECE and corresponding public finance. Recommendations

The evaluation provides a number of strategic recommendations (SP) and operational (O) recommendations in line with the ToR and the need to engage all major stakeholders in a concerted effort for the continuation of reforms on the advancement of ECE in Montenegro.

Strategic Recommendations (SR)

SP1: Ensure sustainable equity prerequisites for expanding the ECE services for children aged 3-6 at national level and breaking intergenerational cycles of exclusion (Addressed to: MoE with the support of UNICEF and in partnership with PSIs, Centres for Social Work and Health care providers)

SP2: Ensure that quality safeguards are embedded in the expansion of ECE service provision (Addressed to: BfES, MoE, UNICEF, PSIs, pre-service and in-service training providers and local communities)

SP3: Encourage participation of children and empower parents to become active and efficient partners of the education system in the process of early childhood education and development (Addressed to: UNICEF in partnership with the MoE, PSIs and CSOs)

Operational Recommendations (O)

O1: Carry out a quick quality and financial sustainability assessment of outreach interactive services in the target municipalities supported by UNICEF (first wave) (Addressed to: UNICEF, MoE and BfES)

O2: Develop the Monitoring and Evaluation Framework of the new ECE strategy 2016-2022 (Addressed to: UNICEF and MoE)

O3: Improve the results architecture and evaluability of future UNICEF initiatives (Addressed to: UNICEF CO)

Lessons Leant The Programme provided useful policy and programmatic lessons to the countries which would like to improve the enrolment of children aged 3-6 in inclusive and quality pre-school education. In this respect, the replication of Montenegro experience in other countries would need to consider the following lessons learnt:

1) Development of an enabling framework for universal coverage of children in ECE requires a systematic, comprehensive and concerted approach of all determinants of system change and of all key stakeholders at various levels of intervention. Such approach would imply the following mandatory ‘ingredients’: a) development of solid, credible evidence on needs; b) policy advocacy at national level; c) development and adoption of the strategic framework for ECE expansion and funding scenarios; d) testing, costing and documenting various service provision options at community level; e) capacity building of service providers; f) awareness raising campaign for behaviour change of parents at regional and local levels.


11

Such comprehensive and systematic effort is key to ensuring an informed policy debate and building support in the society for investment in the ECE.

2) Expansion of ECE for children aged 3-6 requires both hard and soft investment. Investing in infrastructure rehabilitation and in the procurement of furniture and didactical materials in parallel with the delivery of early education programmes and capacity building of both professionals and parents is a good mix and a feasible solution for countries where the existing school infrastructure is decrepit and the school network is sub-optimally located across the territory.

3) Any action targeting ECE must engage the parents of beneficiary children from start throughout the whole process of change. The participation of parents in the life of kindergarten and monitoring of activities increases the relevance and contributes to quality assurance of ECE programmes, while the empowerment of parents with positive parental skills yields sustainable effects on children’s educational attainment and development in the coming years.


12

1. CONTEXT OF THE EVALUATION Key features of the country context Montenegro is an upper-middle-income country situated in South-Eastern Europe, with a population of 620,029 inhabitants (20111), of whom 63% live in urban settlements. The most densely populated is the capital of Montenegro, Podgorica, where 30% of the entire population is living. There are 145,126 children, of whom 22,211 children aged 3-6 (10,562 girls and 11,649 boys). Over the last five years, the number of children of pre-school age had a decreasing trend2. Montenegro belongs to the group of countries in the efficiency-driven stage of development3 with a Gross Domestic Product (GDP) per capita of US$ 7,276.10 (2016)4. The country experienced rapid economic growth, ranging between 8.6% and 10.7% of GDP in 2006 and 2007 respectively. As a result of global economic crisis that hit the country in 2008, the real GDP growth rate declined steeply. After slow and modest economic recovery in 2010-2011, the economy went back into recession in 2012 with a real growth rate of GDP of −2.5%5, but recorded growth rates of 1.8% in 2014 and 3.4% in 20156. GDP growth is projected to remain relatively robust at slightly below 4% in 2016 and 20177. The Montenegrin economy is burdened by grey economy and high unemployment (around 20% since 20108). The unemployment rate in Northern Montenegro is as high as 34.9% in 2016. Poverty in Montenegro fell rapidly after independence in 2006, but these gains have been completely reversed since the onset of the crisis in 2008, having a prolonged negative impact on the well-being of its citizen and increasing inequalities in the country. The poverty rate fell from 11.3% in 2006 to 4.9% in 2008, before steadily increasing back to 11.3% in 20129. The poverty rate further decreased in 2013 to 8.6%, but the Gini coefficient has not reversed its negative trend (24.4% in 2006 compared to 26.2% in 2013). In 2014, the poverty rate was 7.5%10 (Figure 1). Figure 1. Poverty rate in Montenegro (%), 2006-2014

Source: MONSTAT, Department for labour market, living conditions, social services and household consumption Children are more exposed to poverty than adults: 10% of children and 6.1% of adults are living in poverty in Montenegro11, with a monthly spending less than €169.13 (the national absolute poverty line

1 Population Census 2011 2 Based on MONSTAT data for age groups 0-4 and 5-9, the evaluation team estimated the total number of children aged 0-5 to have been 43,515 children compared to 46,066 children in 2011, meaning a 5% reduction. MONSTAT methodology does not use age groups corresponding to preschool and school age, but rather 0-4 and 5-9. 3 World Economic Forum (2014), “The Global Competitiveness Report 2014-2015”, Geneva 4 http://www.tradingeconomics.com/montenegro , accessed 15 December 2016 5 MONSTAT, https://www.monstat.org , accessed 14 January 2017 6 MONSTAT, http://www.monstat.org/userfiles/file/GDP/bdp2015/Release_GDP_2015.pdf, accessed in 12 April 2017 7 http://ec.europa.eu/economy_finance/eu/forecasts/2016_spring/cc_montenegro_en.pdf , accessed 20 November 2016 8 Labour Force Surveys, https://www.monstat.org/eng/page.php?id=527&pageid=22, accessed 20 December 2016 9 MONSTAT (2014), “Poverty Analysis in Montenegro in 2013”, Podgorica 10 MONSTAT, Labour Force Survey 2015 11 UNICEF (2012), “Child Poverty in Montenegro”, Podgorica

11.3

8

4.9

6.86.6

9.3

11.3

8.6

7.5

2

4

6

8

10

12

2006 2007 2008 2009 2010 2011 2012 2013 2014

http://www.tradingeconomics.com/montenegro

https://www.monstat.org/

http://ec.europa.eu/economy_finance/eu/forecasts/2016_spring/cc_montenegro_en.pdf

https://www.monstat.org/eng/page.php?id=527&pageid=22


13

in 2009). This means that every 10th child lives in poverty (or 14,500 children). The youngest (children under the age of five) are most vulnerable, with a poverty rate of 13%. Poverty indicators show significant regional12 differences in child poverty. In 2012, the poverty rate in the Northern region was 18.3%, much higher than in the Central region (7.9%) and the Southern region (9%)13. More than three-quarters of all poor children are living in rural areas, and more than half of them are living in the North of the country. Montenegro’s Human Development Index for 2014 was 0.802, positioning the country at 49 out of 188 countries and territories and placing it among the countries with a very high human development14. Montenegro has made significant progress in meeting most Millennium Development Goals (MDG) targets, but still lags behind in relation to MDGs 1, 3 and 7 (poverty reduction, gender equality and environmental sustainability).15 Montenegro was accepted as the 192nd member state of the United Nations (UN) and the 47th member state of the Council of Europe in 2006. A Stabilization and Association Agreement was signed with the European Union (EU) in 2007. Montenegro presented its official application for membership to the EU in 2008 and was awarded the status of candidate country in December 2010. The prospect of EU membership had important implications for the situation of women and children as it motivated important child rights related reforms and initiated an intense process of translating the acquis communautaire into national legislation. Negotiations have been opened on 26 of 33 chapters, including 23 – judiciary and fundamental rights, 24 – justice, freedom and security and chapter 26 – education and culture which was provisionally closed on 15 April 2013. Early Childhood Education for children aged 3-6

Enrolment rates Data on education level in the macroeconomic context are a strong argument for advocating for better and stronger support to Early Childhood Development (ECD) and Early Childhood Education (ECE) as several recent studies confirm that investment in early development and education is the best investment for society and individual. The literature makes reference to research and findings of Nobel Prize for Economics James Hackman: “For each dollar spent on quality programmes for children in early age there is return of 17.07% out of which 12.90 to public budget and 4.17% to individual.”16 ECE is making a tremendous contribution particularly in the case of children from poor and vulnerable families, improving their preparedness for school, and, in the longer term, increasing social mobility and poverty reduction. Montenegro has a relatively low ECE enrolment rate of children aged 3–6, at 52.4% (11,641 children)17, well below the EU target of 95% by 202018 and in contrast with the high enrolment rates in primary and secondary education (98% and 85%, respectively). There are important disparities in the enrolment rate across the municipalities of Montenegro, from 10% in Rozaje (located in the North) to 94% in Budva (South of Montenegro). Figure 2 illustrates the clustering of municipalities in three groups i.e. high-enrolment (88% on average); medium-enrolment (52% on average); and low enrolment (27% on average). The municipalities with the lowest enrolment rate are situated in the Northern region, while the best off in the South of the country.

12 Montenegro is divided into three main regions: Southern region (or coastal region) comprising six municipalities; Central region comprising four municipalities; and Northern region comprising 11 municipalities. 13 MONSTAT (2014), “Poverty Analysis in Montenegro in 2013”, Podgorica 14 UNDP (2015), “Human Development Report 2015. Work for Human Development”, New York 15 Government of Montenegro (2013), “Report on Millennium Development Goals. Montenegro 2010-2013” 16 Pašalić-Kreso, A. (2011), “Nauka za predškolsko obrazovanje, predškolsko obrazovanje za nauku“ u zborniku “Kvalitet predškolskog odgoja i obrazovanja u Bosni i Hercegovini“. 17 Prica, I., Čolić, L., Baronijan, H. (2014), “A Study on Investing in Early Childhood Education in Montenegro”, UNICEF Montenegro, Podgorica 18 According to the Strategic framework for European cooperation in education and training (‘ET 2020’), at least 95% of children (from 4 to compulsory school age) should participate in ECE


14

Figure 2. Enrolment rate of children aged 3-6 in ECE by municipality (%), 2012

Source: Based on Prica, I., Čolić, L., Baronijan, H. (2014), “A Study on Investing in Early Childhood Education in Montenegro”, UNICEF Montenegro, Podgorica

The attendance rate stands at 39.9%19. Slightly more girls attend than boys, i.e. 38.7% compared to 41.5%20. Urban–rural, regional and wealth status differentials are significant: the figure is as high as 51.1% in urban areas compared to 19.5% in rural areas; attendance of ECE programmes is most prevalent in the Central region (53.9%), and the lowest in the North (16.9%); 66% of children living in the richest households attend such programmes, while the figure drops to 7% in the poorest households, almost 10 times less (Figure 3). Figure 3. Proportion of children 3-6 years of age attending ECE (%) by wealth quintiles

Source: MONSTAT and UNICEF Montenegro (2014), ”Montenegro and Montenegro Roma Settlements. Multiple Indicator Cluster Survey 2013”

19 UNICEF Montenegro (2014), ”Montenegro and Montenegro Roma Settlements. Multiple Indicator Cluster Survey 2013” 20 ibid


15

Given the correlation between ECE and learning outcomes of children in the upper education levels21, the low level of attendance of ECE is associated with the low scores achieved in PISA tests, which place Montenegrin 15-year-olds well below OECD average22. Apart from rural children and those living in poor households, there are other two groups of vulnerable children whose participation in ECE is either low or unknown. Coverage of Roma and Egyptian (RE) children in kindergarten is around 21.5%23, or more than two times less than non-RE children. While the attendance rate at national level is 39.9% for the age group 3-6, in case of children living in Roma settlements it is less than half (18.5%24). Failure to attend preschool can make the barriers to linguistic and cultural participation in primary school even greater. The school readiness25 is only 23.8% for children living in Roma settlements compared to the national rate of 45.8%26. A very high percentage of RE children (84%) who are currently out of formal education have never attended a kindergarten compared to 47% who did27. Children with disabilities face specific challenges. There is no accurate data on these children in the country. Despite some positive shifts in behaviours, influenced by nation-wide social change campaigns, most notably UNICEF’s “It’s about ability” campaign, children with disabilities still face stigma and discrimination, and as a result remain hidden. Indeed, a significant number of “invisible children”, particularly children with disabilities, reside in Montenegro out of the grid of health, social and education systems28, including preschool education. The Montenegrin Education Information System (MEIS) registers 78 children with disabilities aged 3-6 in preschool education (2015/16), representing 0.6% of enrolled children in the respective age group; however this figure is inaccurate since there is no uniform practice of registering these children by the Preschool Institutions (PSIs) across the country29. At the same time, the lack of a reliable number concerning the total number of children with disabilities in the country, disaggregated by age, makes impossible the calculation of the enrolment rate and undermines evidence-based policy-making in favour of promoting the inclusion of these children in ECE programmes. The main reasons for low coverage of children in ECE programmes are the following:

a) Insufficient preschool physical capacities in Southern and Central regions to cover the total number of children left out from the ECE system on the background of under-used capacities in the Northern region. The average group size of 3-6-year-olds in the public PSIs varies greatly by municipality: from 41 children per group on average in Herceg Novi and the public preschool institution “Ljubica Popović” in Podgorica to only 12 children per group in Andrijevica. Overcrowding of groups might pose a risk to the quality of the educational process30 with negative impact on learning and development. The highest demand for ECE services is registered in the urban areas, where the PSIs are overcrowded. In Podgorica, the capital city of the country, the demand exceeds by far the available supply. If legally-prescribed norms regarding the number of children in groups are observed31, about 3,377 children would not have a place in the existing PSIs (almost 30% of the total number of children aged 3-6 attending ECE), of whom 1,849 live in Podgorica32. Actually, many kindergartens in the Southern and Central regions work above their capacities and the main reason for not having the full coverage of children in ECE is the lack of space. On the other hand, the Northern municipalities have low coverage rates, and at the same time, their capacities are

21 http://www.oecd.org/edu/school/earlychildhoodeducationandcare.htm , accessed 26 February 2017 22 https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf , accessed 13 January 2017 23 Strategy for Social Inclusion of Roma and Egyptians in Montenegro 2016 - 2020 24 MONSTAT and UNICEF Montenegro (2014), ”Montenegro and Montenegro Roma Settlements. Multiple Indicator Cluster Survey 2013 25 Percentage of children in first grade of primary school who attended pre-school during the previous school year. 26 MONSTAT and UNICEF Montenegro (2014), ”Montenegro and Montenegro Roma Settlements. Multiple Indicator Cluster Survey 2013” 27 Ipsos Strategic Marketing (2013), “Study on the obstacles to education in Montenegro. Focus on Roma and Egyptian children”, prepared by Delić., A. et al. 28 Government of Montenegro (2013), “Strategy for Social and Child Protection in Montenegro 2013-2017” 29 The evaluation was informed that some PSIs register in the MEIS only those children with disabilities referred by the Commissions for Orientation, while others are doing that based on direct observation of the respective child by the teacher. In other cases, children with disabilities are not registered at all, even if they have a referral or the child has obvious special education needs. 30 Wood, J. et al. (2014), “Independent Multi-country Evaluation of Results Area 3: increasing access and equity in early childhood education in CEE/CIS”, RKLA in the CEE/CIS Region, Final evaluation report, November 2014, page 16 31 20 to 25 children/group, depending on age, 20 children/group for mixed-age group in kindergarten (Law on Preschool Education of Montenegro, Article 24) 32Prica, I., Čolić, L., Baronijan, H. (2014), “A Study on Investing in Early Childhood Education in Montenegro”, UNICEF Montenegro, Podgorica

http://www.oecd.org/edu/school/earlychildhoodeducationandcare.htm

https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf


16

underused. This is mainly accounted for by the reasons of low population density and large distance to the nearest preschool facilities, especially for children in rural areas. Almost half of families whose children do not attend preschool live in rural areas with the nearest preschool facility being 9.5 km away from their homes and over ¾ do not have proper public transportation options available33. A study from 2013 identifies the lack of proximity of pre-schools to Roma and Egyptian households as an important barrier in the access of ECE by RE children34. b) Limited financial capacity of parents. According to the law, parents have to pay a monthly contribution of 40 Eur for a full-day programme. In case of children without parental care and those whose families are recipients of social benefits, the contribution is paid by the Centres for Social Work. Nevertheless, there are other costs associated with sending the child to preschool, such as for transportation, clothing or school supplies, etc.35. According to the Survey on Knowledge, Attitudes and Practices (KAP) conducted in 201436 in municipalities with the lowest coverage rate, the main obstacle to enrolling in preschool was the lack of financial means, apart from distance. As seen in Figure 3, the household wealth has an important influence on the attendance of ECE.

c) Insufficient parents’ awareness as to the benefits of preschool education and social norms. The customary multi-generational patterns of family life, in which the care for the preschool children is provided by grandparents, and the insufficient awareness of the importance of preschool education are contributing factor to low enrolment rates. Indeed, one in four parents believes that grandparents provide the best quality of care37. At the same time, some parents believe that there is no reason to pay for a kindergarten if there is someone who can take care of the child at home, that family members can provide better care than kindergarten teachers and that it is hard for mothers to be separated from children. Some unemployed parents may also consider shameful to bring their child to kindergarten once they could stay with the child at home. It shows that parents see ECE through its custodial function rather than the benefits for the development of their children. Around 26% of parents believe that children should start learning only once they start going to school38.

Legal and strategic framework The preschool education in Montenegro is regulated by the General Law on Education (GLE) and the Law on Preschool Education (LPSE), the Law on Primary Education and the Law on Social and Child Welfare. While the GLE sets out the general rules in the education sector (regarding institutional and management infrastructure, provisions for setting up the educational institutions, bodies, budget financing), the LPSE determines in more detail the rules set out in the GLE and develops further rules and standards regarding preschool education (the number of children per group and the number of hours for each programme; rules and standards for teaching and other professional staff). Of particular interest for this evaluation is Article 28b about interactive outreach services which could be established to address some of the needs of children and families in remote rural areas. The main strategic document in the area of ECE is the Strategy for Early Childhood and Preschool Education 2016–2020 which was developed through the Programme which is the object of this evaluation, based on the lessons learnt from the implementation of the former strategy (2010-2015) and on the recommendations of the Study from 2014. The former strategy aimed at providing all children in Montenegro, from birth until the time they start primary school, with good quality services for early development and learning. The efforts of the Ministry of Education (MoE) and the Government have mainly been directed at securing funds for building new kindergartens, since the lack of spatial capacities in kindergartens was perceived at the time to be the main cause of low enrolment rate in preschool education. Progress in this regard has been very slow – only seven new kindergartens will be built in the next three years which will result in a marginal 3% increase in the preschool coverage. The new

33 Proposal of the Ministry of Education “Interactive services programme” submitted to UNICEF Montenegro (Ref.no: 618-22/2015-2, dated 11 November 2015) 34 Ipsos Strategic Marketing (2013), “Study on the obstacles to education in Montenegro. Focus on Roma and Egyptian children”, prepared by Delić., A. et al. 35 Parents may nevertheless get back part of the paid contribution for the days the child has not attended the kindergarten due to illness. 36 IPSOS (2014), “Knowledge, attitudes and practices of parents from the northern municipalities regarding pre-school education” 37 KAP survey (2014) cited in the Proposal of the Ministry of Education “Interactive services programme” submitted to UNICEF Montenegro (Ref.no: 618-22/2015-2, dated 11 November 2015) 38 IPSOS (2014), “Knowledge, attitudes and practices of parents from the northern municipalities regarding pre-school education”


17

Strategy was adopted in November 2015 by the Government along with a fully-fledged Action Plan and required budget. The Strategy sets out three main goals, as follows:

1) Universal preschool coverage of all children aged 3 to school age in line with international standards;

2) Improve the quality of preschool education; and 3) Introduce innovative, optimal and sustainable funding models.

The new Strategy addresses causes of low participation in preschool education and focuses on increasing enrolment in preschool education by improving the infrastructure and expanding interactive services as well as book and toy libraries; introduction of innovative and flexible programmes, including 3-hour programmes; intensive integration of the RE population in ECE; better detection and responses to the needs of children with special educational needs; ensuring active participation of parents; recruitment of sufficient professionals and provision of training opportunities to the existing staff; amendment of regulations to facilitate employment of support staff; new financing options of preschool education and a minimum of 0.53% of GDP as public spending for preschool education; and collection of data with focus on the vulnerable children. The Strategy and Action Plan as five-year implementation framework have not set clear targets in either of three strategic goals. However, the National Plan of Action for Children 2013-2017 defines more concrete measures and indicators under its Strategic Objective 6: Ensure equal access for all children to quality education, constructive spending of free time and cultural contents which are further discussed in the Relevance chapter of this evaluation report. The strategic framework is completed by the Strategy for the Integration of Persons with Disabilities 2016-2020, Strategy for Inclusive Education 2014-2018 and the Strategy of Social and Child Protection System Development 2013-2017 which provide for a range of measures and activities to improve the position of children with development disabilities. Another important strategic document is the Strategy for Social Inclusion of Roma and Egyptians in Montenegro 2016 – 2020 which envisages to increase the percentage of enrolled RE children in kindergartens from the current 21.5% to 40% by 2020. Finally, the National Strategy of Sustainable Development until 2030, which was adopted in July 2016, contains a chapter on ECE with well-defined targets, including the envisaged increase level of GDP.

International human rights commitments Universal principles of human rights are built in the overall legal framework covering education, health and social protection in order to comply with the requirements and recommendations of international conventions and charters. Observance of human rights and in particular the rights of the child is part of the political criterion for accession and has a particularly important imprint in the enlargement policy for Montenegro, in accordance with the EU Charter of Fundamental Rights and the UN Convention on the Rights of the Child (CRC). From an education perspective, the international commitments of the country primarily derive from the ratification of the CRC in 1993. In its Concluding Observations (October 2010)39, the Committee on the Rights of the Child expressed concern on the reduced access to education of children with disabilities and Roma children; and low availability of holistic early childhood development and education, facilities and institutions. The Committee recommended that measures are taken to provide equal education opportunities to these children; further increase the budget allocations for education, health care and family support; further strengthen adequate and systematic training of all professional groups working for and with children, including teachers; raise awareness with respect to preschools and early learning opportunities; and improve data collection and analysis, disaggregated inter alia by age, sex, ethnic or social origin and urban/rural areas, in order to monitor and evaluate progress achieved and assess the impact of policies adopted with respect to children. The country has also ratified the Convention for the Elimination of All Forms of Discrimination Against Women (CEDAW) in 2006. In its latest Concluding Observations40, the CEDAW Committee calls for action towards raising awareness among RE families about the importance of education for the life and career prospects of girls and provision of further incentives to RE parents to send their daughters

39 Committee on the Rights of the Child (2010), “Concluding observations: Montenegro”, Fifty-fifth session, CRC/C/MNE/CO/1 40 Committee on the Elimination of Discrimination against Women (2011), “Concluding observations of the Committee on the Elimination of Discrimination against Women. Montenegro”, Fiftieth session, Geneva, 3-21 October 2011


18

to school. It also calls for the adoption of temporary special measures to eliminate multiple forms of discrimination against RE girls and women, including in education.

Institutional Framework The development, implementation and monitoring of national policies in the area of early childhood education and care are done by the Ministry of Education, Ministry of Health and Ministry of Labour and Social Welfare. Applicable quality standards are also developed by the central public administration. The Bureau for Education Services (BfES) is the operational body in charge of translating education policy to practice. The work of the Bureau is focussed on quality assurance, professional development of teachers and research and development. It is also in charge of accrediting the training of teacher programmes. The institutional framework in education also comprises the Examination Centre, the Textbook Publishing Agency, the Vocational Education Centre and the National Council for Education. The National Council for the Rights of the Child, set up in 2007, has the mission to monitor the implementation of the National Plan of Action for Children; protect and promote the rights of children with regard to social and child protection, health care, education; monitor Montenegro’s fulfilment of commitments arising from the CRC and other international instruments; initiate the adoption of regulations for the promotion and protection of children's rights; enhance cooperation with local authorities and civil society organisations (CSOs) in the implementation and protection of children's rights; inform the public about the rights of the child and report on the state of children's rights. The Council needs capacity building to play a leading role in policy planning and in setting priorities for the implementation of the CRC and Optional Protocols, promote the participation of children and civil society in its proceedings and prepare analytical reports, reviews and case studies as a help in making strategic decisions for children in Montenegro41. Preschool education in Montenegro is mainly delivered through PSIs that can be state-run (21 institutions and 103 educational units with 515 educational children groups) or private (17 institutions in 7 towns including 9 in Podgorica, but none in the Northern region of the country). The monthly cost for a child in private PSIs is in the range of €100-200 which is 2,5 - 5 times higher than in the state-run institutions. The costs may also explain why private PSIs are rather limited in scope, providing services to only 2.5%-3.5% of all children enrolled in pre-school education. Preschool activity also takes place in the children's home "Mladost" in Bijela as well as in resource centres in Kotor and Podgorica which provide continuous education and rehabilitation of children with hearing difficulties, and physical, sensor and combined disabilities respectively. All PSIs follow the publicly approved educational programme. The LPSE differentiates between primary programme, short programme (also for all children aged 5 years, until they go to school, organized as a preschool preparatory programme), specialized programme and individual development and education plans for children with SEN. The teaching staff is supported by other professionals (psychologists, pedagogical experts, speech therapists, rehabilitation and education experts, nutritionists, etc.) and administrative and support staff. Two preschool teachers work in tandem for each group of children aged 3-6. During the 2014/2015 academic year, PSIs in Montenegro employed 1,870 staff.42 Role of international organisations In the area of ECE and child protection, UNICEF is a strategic partner supporting the Government of Montenegro in the design and implementation of key reforms, on the basis of a five-year Country Programme Document. UNICEF also works towards capacity development of key actors, modelling and testing innovations and achieving social change through effective use of communication. Together with partners at policy level and local community level, including civil society, UNICEF has implemented a number of programmes focused on inclusion of marginalized groups, aimed to reduce the inequalities and provide equal opportunities for all. The contribution of UNICEF at system level is acknowledged in few strategic documents adopted by Montenegrin authorities, most notably in the National Plan of Action for Children 2013-2017 and the Strategy for Early Childhood and Preschool Education 2016–2020. Since 2014, UNICEF in partnership with the Government and CSOs has intensified its work aimed at

41 National Plan of Action for Children 2013-2017 42 Strategy for Early and Preschool Education in Montenegro 2016-2020, https://www.unicef.org/montenegro/UNICEF_-

_Strategija_PVO_u_CG_-_ENG.pdf

https://www.unicef.org/montenegro/UNICEF_-_Strategija_PVO_u_CG_-_ENG.pdf

https://www.unicef.org/montenegro/UNICEF_-_Strategija_PVO_u_CG_-_ENG.pdf


19

increasing the access and quality of preschool education. Prior to that, UNICEF was supporting Inclusive Education concept development and implementation in Montenegro with the objective of improving the access to education (including preschool) for children with special developmental and special learning needs. UNICEF continues to have a strong presence in this area, with a focus on increasing the quality of inclusive education practices and enhancing the overall quality of the education system. There are other international agencies and donors that contributed to increasing the access to quality education over the last two decades, most notably UNESCO, British Council, Canadian Development Agency, Government of Finland, OECD, Coordinamento delle Organizzazioni per il Servizio Volontario (known as COSV), Save the Children, Open Society Institute and World Bank.


20

2. OBJECT OF THE EVALUATION The object of this evaluation is the programme “Montenegro - Investment Case on Early Childhood Development” (hereinafter referred to as “The Programme”). The Programme is part of a broader collaboration between UNICEF Headquarters (HQ) and the H&M Conscious Foundation (HMCF) entitled “Unleashing Children’s Potential”, which aims to give young children the best possible start in life and learning by supporting three interrelated results, as follows:

1. Result 1: Global Platform established for advocating the importance of ECD

2. Result 2: ECD placed on the National Development Agenda in three countries (Montenegro, Chile and South Africa)

3. Result 3: Effectiveness of ECD policies and programmes demonstrated for child development and learning in six countries (Benin, Lao PDR, Nepal, Timor-Leste, Rwanda and Mali)

Montenegro programme falls under Result 2, within which the HMCF and UNICEF partnership supports the development of three national investment cases, based on global neuroscientific and programmatic evidence for supporting ECD and country-specific economic evidence for investing in ECD. According to the 2014 Annual Report to the donor, the three countries (including Montenegro) have been selected by UNICEF HQ (ECD section) in close consultation with its regional offices and based on the situation of children, the burden of low child outcomes, the governments’ potential for ECD investment, geographic region, potential to serve as models for scaling-up the investment cases in other countries, etc. The Terms of Reference (ToR) (Annex 1), the concept note, the Theory of Change (ToC) and the Results Framework (including key indicators) provide a good overview of the evaluated Programme. The ToC and the Results Framework have been developed retroactively by the UNICEF Country Office (CO) only for the purpose of this evaluation43 and they attempt to conceptualise the intended change processes

starting with inputs towards impact-level contribution. While activities and outputs are specified in the Programme concept note44, the outcomes, impact statement and key indicators appear only in the retroactive ToC and Results Framework. Since these two documents have been developed only to serve as a framework for the evaluation, the evaluation team has further revised and streamlined them in line with the RBM and the regional UNICEF ToC for Early Learning and School Readiness45. The revised ToC better explains the logical links between needs and bottlenecks on one side, and UNICEF community, regional and national interventions on the other side, and link this work with the higher level results in the hierarchy of change (Figure 4). The main features, logic and changes foreseen to be brought about by the Programme are presented below. It is to be noted that the Programme has exclusively focused on ECE, although its name indicates a broader scope i.e. ECD. The evaluation was informed that it was the decision of each participating country in the HMCF initiative which sector to concentrate on; in the case of Montenegro, the decision to focus on ECE was agreed between the CO and UNICEF HQ, as a logical follow up of the recommendations of the Study from 2014 about the benefits of investing in ECE46.

43 According to UNICEF CO, the TOC and the Results Framework have not been initially requested by the programme intervention and donor. 44 They are consistent with the outputs included in UNICEF HQ’s proposal to the Donor (HMCF), less the ones on “Consensus reached on outcomes for children, families and societies that need to be included in the investment case” and “Financial governance systems set up at national and decentralised levels to allocate funds for ECD programmes in disadvantaged areas and communities”. According to the CO, the concept note for Montenegro was accepted as such by the HQ, along with proposed outputs and activities; any differences between the HQ’s proposal to the Donor and the concept note for Montenegro are due to country specificities. 45 Wood, J. et al. (2014), “Independent Multi-country Evaluation of Results Area 3: increasing access and equity in early childhood education in CEE/CIS”, RKLA in the CEE/CIS Region, Final evaluation report, November 2014 46 Prica, I., Čolić, L., Baronijan, H. (2014), “A Study on Investing in Early Childhood Education in Montenegro”, UNICEF Montenegro, Podgorica


21

Figure 4. Theory of Change

BOTTLENECKS INPUTS MAIN ACTIVITIES OUTPUTS OUTCOMES

(system change)

IMPACT

NATIONAL LEVEL

Quality and efficiency

issues (overcrowded

kindergartens in the

Southern and Central

regions due to lack of

space and underused

capacities in the

Northern region;

average group size of

3-6-year-olds varies

from 12 to 41);

Equity issues (large

gaps in enrolment rates

for poor children,

children living in rural

areas, RE children,

children with

disabilities/SEN)

Large regional

disparities in enrolment

rates (88% in the South

and 27% in the North)

Financial burden for

poor parents (payment

of the monthly

contribution to the cost

of meals) (KAP 2014)

Customary multi-

generational patterns of

family life, with care for

preschool children

done by grandparents

Insufficient awareness

of the importance of

preschool education

and available service

supply among parents/

caregivers (KAP 2014)

Finance

(HMCF and

UNICEF),

staff,

situation

analysis (e.g.

baseline KAP

surveys),

governance,

management

systems,

commitment

and

engagement

of the Ministry

of Education,

respectively

pre-school

institutions

Develop an investment case in ECE based on the study on investing in ECE in Montenegro

Organise a high-level conference on Investing in ECE

Intensive high-level policy advocacy activities to sustain the momentum

1.1. The dialogue on the need and strategic development directions of the ECE in the country initiated through knowledge generation and sustained

1. Enabling policy

framework for

ECE expansion

country-wide

Present to stakeholders and professionals the economic and individual social wellbeing benefits of investing in ECE

Present to stakeholders and professionals the financing scenarios for expansion of preschool education coverage

1.2.Key government stakeholders and professionals have an overview of the long-term benefits of investing in ECE and of possible scenarios for expansion of ECE

Improved

coverage of

children aged

3-6 in inclusive

and quality pre-

school

education and

their learning

outcomes, in

line with

international

human rights

standards and

the principle of

progressive

realisation of

children’s

rights

Draft an ECE expansion strategy for the adoption by the Government

Organise a high-level conference on quality education, including ECE, to present the new strategy

1.3 New ECE expansion strategy developed in line with generated knowledge

Draft the M&E Framework of the ECE expansion strategy

1.4 Framework for M&E of the ECE strategy developed

COMMUNITY LEVEL

Implement outreach interactive pre-school services for children living in rural areas in 5 municipalities

Carry out training of teachers

Present the interactive services to key actors and prepare for expanding it to other 6 municipalities

2.1. Outreach interactive pre-school services made available in targeted municipalities and up-scaled

2. Improved

access to new

ECE services for

children aged 3-6

in the targeted

municipalities, in

particular of

vulnerable children

Draft Local Implementation Plans of ECE expansion in 3 municipalities

2.2 Models for ECE expansion developed

Develop an alternative 3-hour preschool

programme

Present the 3-hour programme to teachers

2.3 Alternative 3-hour preschool programme piloted

REGIONAL LEVEL

Carry out awareness raising campaigns “Preschool for all” in the North of the country to raise the awareness of parents/caregivers on the importance of ECE

3.1 Parents/caregivers in targeted regions understand the importance of the ECE for their children’s development and education

3.Increased

demand of

parents/ caregivers

for ECE

programmes


22

Goal (Impact statement) Based on the revised ToC, the Goal of the Programme was to contribute to the improvement of coverage of children aged 3-6 in inclusive and quality pre-school education and their learning outcomes, in line with international human rights standards and the principle of progressive realisation of children’s rights. Outcomes The Programme aimed to reach its goal by achieving three outcomes:

1. Enabling policy framework for ECE expansion country-wide.

2. Improved access to new ECE services for children aged 3-6 in the targeted municipalities, in particular of vulnerable children47

3. Increased demand of parents/ caregivers for ECE programmes Outputs In line with the intervention logic, the ToC further develops the overarching outcomes into several expected outputs, as follows:

Related to Outcome 1 1.1 The dialogue on the need and strategic development directions of the ECE in the country initiated

through knowledge generation and sustained 1.2 Key government stakeholders and professionals have an overview of the long-term benefits of

investing in ECE and of possible scenarios for expansion of ECE 1.3 New ECE expansion strategy developed in line with generated knowledge 1.4 Framework for M&E of the ECE strategy developed

Related to Outcome 2 2.1 Outreach interactive pre-school services made available in targeted municipalities and up-scaled 2.2 Models for ECE expansion developed 2.3 Alternative 3-hour preschool programme piloted

Related to Outcome 3 3.1 Parents/caregivers in targeted regions understand the importance of the ECE for their children’s

development and education Programme Activities For the achievement of Programme expected outputs and outcomes, a rich set of activities have been planned, as follows:

a) National level:

Development of an investment case in ECE based on the study on investing in ECE in Montenegro

Organisation of a high-level conference on investing in ECE

Intensive high-level policy advocacy activities to sustain the momentum

Presentation to stakeholders and professionals of the economic and individual social wellbeing benefits of investing in ECE

47 The evaluation was informed by the CO that, in the understanding of the Programme, ‘vulnerable children’ included ‘poor children’ and ‘children living in rural areas’. The concept note does not contain any reference to ‘vulnerable children’; outcome 2 in the Results Framework (developed retrospectively by the CO for the purpose of evaluation) embeds ‘vulnerable groups’ in its formulation, still without providing any other details.


23

Presentation to stakeholders and professionals of the financing scenarios for expansion of preschool education coverage

Drafting an ECE expansion strategy for the adoption by the Government

Organisation of a high-level conference on quality education, including ECE, to present the new strategy

Drafting of the M&E Framework of the ECE expansion strategy

b) Community level:

Development and implementation of outreach interactive pre-school services for children living in rural areas in 5 municipalities, including home parenting counselling

Training of teachers

Presentation of outreach interactive services to key actors and preparation for expansion of these services to other 6 municipalities

Development of an alternative 3-hour preschool programme

Drafting of Local Implementation Plans of ECE expansion in 3 municipalities (through 3-hour preschool programme)

Presentation of the 3-hour preschool programme to teachers

c) Regional level:

Awareness raising campaigns “Preschool for all” in the North of the country to raise the awareness of parents/caregivers on the importance of ECE

The grouping of activities per level of intervention and around each output is provided in Figure 4. Target Groups and Final Beneficiaries The Target Groups of the Programme, as mentioned in the annual progress reports of UNICEF CO, included: a) high-level decision-makers from education, health, social and child protection sectors; b) professionals working in these sectors whose capacity was strengthened to deliver quality ECE outreach interactive services and alternative 3-hour programmes; c) parents/caregivers who were informed about the development and learning benefits of enroling their children in preschool education and were also aimed to benefit of home parenting counselling, as the case; d) communities where new ECE services have been/will be implemented (Bijelo Polje, Berane, Plav, Rozaje, Andrijevica, Mojkovac, Kolasin, Savnik, Zabljak, Pluzine, Pljevlja, Bar and Danilovgrad); and e) the general public, whose awareness was raised concerning the importance of ECE. The Final Beneficiaries of the Programme were preschool age children (3-6 years of age), in particular those living in the poorest municipalities located in the North region of Montenegro48. The concept note of the Programme does not specify any intended reach in terms of estimated size of the target groups or number of final beneficiaries. The only estimates appear in the proposals of the MoE on the development and expansion of outreach interactive services, submitted to UNICEF for approval on 11 November 2015 and 29 March 2016, where the number of intended beneficiary children is 300 children for the first wave of target municipalities and 350 children for the second wave. Strategic Approach The review of the Programme concept note and ToC as well as of the ToR of this evaluation indicates that the Programme strategy was focused on the creation of an enabling policy environment for ECE expansion, diversification and outreach of the supply of ECE services and creation of demand of parent/caregivers for

48 The concept note of the Programme makes no reference to final beneficiaries. The UNICEF CO’s progress reports for 2014 and 2015 indicate the final beneficiaries of the Programme to be the preschool age children (3-6 years of age). During the evaluation process, the CO confirmed to the evaluation team that the focus was put on the children living in poorest municipalities of the country i.e. those located in the North region of Montenegro.


24

such services. Such focus was based on the findings and recommendations of the Study from 201449, which was the first analysis of its kind to provide an in depth insight into the organizational and fiscal efficiency of the preschool education system and as such the basis of a strong investment case for ECE. In order to do that and reach the intended results, the following core roles of UNICEF CO have been employed:

Knowledge generation and dissemination: most notably by presenting to decision-makers the results of the Study mentioned above;

Advocacy: through high-level conferences and meetings across Governmental and Parliamentary bodies and institutions aimed to mobilize the political will and initiate the dialogue on the importance of ECE and the need to undertake systemic reforms to increase preschool education coverage; and through information campaigns to raise public awareness and change social norms in order to boost enrolment of children in preschool education;

Policy dialogue and advice: development of cost effective scenarios for the expansion of ECE and of new strategy;

Partnership convening and resource leverage: partnering with the MoE and renown national organisations representing civil society and academia milieu for the implementation of Programme activities; evidence generated by the Study from 2014 used to leverage the influence of major decision makers in the country;

Piloting of new services: development and implementation of outreach interactive services and alternative 3-hour programmes at community level, aimed to inform policy making for scaling up at national level, demonstrate how the system could meaningfully evolve to reduce equity gaps and children’s rights violations and enhance early learning system management, coordination, planning and funding;

Capacity development: through training courses provided to preschool education professionals to increase the quality of service provision for children aged 3-6.

Period of implementation and Financing The Programme had an initial duration of two years (January 2015-December 2016). The timeframe of the overall initiative financed by HMCF has been extended until June 2017, which has automatically been applicable to Montenegro as well. It means that the Programme will last for an additional period of six months. The Programme has a budget of US$ 406,546, of which US$ 225,000 (55.3%) secured from the grant provided by HMCF and US$ 181,546 (44.7%) representing matching funds from UNICEF50. The CO provided these additional funds for the implementation of the activities of the Programme which were considered most useful and needed in terms of enhancing its impact, notably for the “Preschool for All” campaigns (see Annex 8). Implementation mechanism and partners The Programme has been implemented by UNICEF CO in partnership with the MoE and local implementing partners. The MoE has selected the target municipalities and preschools institutions to deliver the ECE services, provided training to teachers (accredited by the BfES), got actively involved in the organisation of the campaigns “Preschool for All” and implemented and monitored the Programme progress jointly with UNICEF CO. At community level, the main Programme partners were the following:

Pre-school institutions - through participation in the “Preschool for All” campaigns, the delivery of outreach interactive services and alternative 3-hour programmes to beneficiary children aged 3-6 in targeted municipalities;

49 Prica, I., Čolić, L., Baronijan, H. (2014), “A Study on Investing in Early Childhood Education in Montenegro”, UNICEF Montenegro, Podgorica 50 Funds provided by UNICEF until 20 September 2016 according to information submitted by the CO “UNICEF Contribution to ECE related initiatives”, 10/02/2014-20/09/2016


25

Pre-school professionals - through participation in training activities, participation in the “Preschool for All” campaigns, delivery of the new ECE services to children, mobilization and counselling of parents;

CSOs, including associations of parents – through active participation in the campaigns “Preschool for All” and advocacy for the expansion of ECE.

The role of UNICEF CO in the Programme was two-fold: to ensure the technical and financial management of the Programme, while also bringing in the technical expertise and policy advocacy leverage for the attainment of envisaged outcomes. UNICEF HQ and Regional Office were in charge of monitoring the progress of the Programme in Montenegro in line with the estimated results at global level for Result 2 and based on progress reports provided by UNICEF CO, as well as of providing peer support and expert advice to the CO (as needed) for smooth implementation of the Programme. The role of HMCF (Donor) was to provide the funds for the implementation of interventions in Montenegro, as part of the overall grant for broader collaboration with UNICEF HQ “Unleashing Children’s Potential”.


26

3. OBJECTIVES, PURPOSE AND SCOPE OF THE EVALUATION The main objective of the evaluation is to assess the final (end) results of the Programme in relation to its objectives, the ToC and indicators in the Results Framework. Beyond the main objective of evaluation, the ToR include several specific objectives which can be summarised as follows:

4. Assess the relevance, effectiveness, efficiency, sustainability and preliminary impact, i.e. longer term contribution, of the Programme;

5. Extract general lessons learned and provide recommendations aimed to inform the future enhancement of the initiative;

6. Assess the potential for replication and scaling-up of the Programme.

The purpose of the evaluation is multi-fold: a) to improve accountability for the achievement of results and use of resources; b) to provide recommendations for the scaling-up of the tested services for children aged 3-6 at national level, based on documentation of good practices and lessons learnt; c) to help ensure that the focus is kept on issues of equity, inclusion and quality in the process of implementing the Strategy on Early Childhood and Preschool Education 2016–2020; d) to inform the potential replication of Montenegro investment case in other countries, including by way of capturing the changes at the policy level. The evaluation meets the UN statutory requirement for evaluation of UN development assistance at least once in its cycle. The Programme is, at the moment of releasing this final report (April 2017), in its last three months of implementation. The evaluation was scheduled for the last quarter of 2016 in order to provide usable recommendations for the process of preparing the enhancement of this initiative which will begin in the first half of 2017, including within the framework of the new 2017-2020 UNICEF Country Programme and recently-adopted strategy on expansion of ECE services country-wide. This means that the evaluation has a summative element focused on accountability for results achieved to date and a formative element focused on learning and provision of recommendations relevant for the upcoming programming process and new strategy implementation. Beneficiaries of the evaluation. The evidence and knowledge generated by the evaluation, the lessons learnt and its recommendations are aimed to be used by the Government of Montenegro, in particular the MoE and BfES, for further scaling-up the services tested by the Programme in order to expand the coverage of children aged 3-6 in preschool education in line with the principles of equity and quality. The results of the evaluation are also envisaged to be used by the PSIs and local communities for increasing the access to preschool education for all children, especially for the most vulnerable. Based on the evidence provided by the evaluation, it is assumed that the associations of parents and other interested CSOs will further strengthen their monitoring and advocacy efforts for an equitable access of children aged 3-6 to quality preschool education. UNICEF CO intends to use the results of the evaluation for future programming and support to implementation of the Strategy for Early and Preschool Education in Montenegro 2016-2020 as well as for informing the implementation of the 2017-2021 Country Programme. The evaluation is also aimed to provide information to UNICEF HQ (Education Section) and HMCF on the effectiveness and (to the extent possible) impact of the Programme in Montenegro and potential for replication of similar initiatives in other countries. The Scope of the evaluation is defined by the following elements:

- Programme implementation period to be covered: January 2015-December 2016; - focus on vulnerable children and their parents/caregivers living in the poorest municipalities with

the lowest enrolment rate in ECE; - geographic coverage: focus on 13 targeted municipalities where particular Programme

interventions have been conducted (interactive outreach services, “Preschool for All” campaign) or planned to be implemented (alternative 3-hour preschool programmes);

- level of change/intervention: community level (e.g. new service provision, teacher training, development of parenting skills); regional level (campaign for social change); national level system change (e.g. evidence generation, advocacy and policy dialogue, new policies advancing ECE


27

reforms in the country); - evaluation criteria used to guide the evaluation thematically: relevance, effectiveness, efficiency,

impact and sustainability (discussed in detail in the methodology chapter of this report). The main Units of Analysis for this evaluation are the following:

- MoE, as main partner and beneficiary of support for the development of the new ECE strategy 2016-2020, improvement of the ECE service provision in municipalities with the lowest enrolment rate and funding scenarios for ensuring universal coverage of children aged 3-6 in preschool;

- preschool education professionals (principals, teachers, support staff), as recipients of trainings and providers of ECE services to children aged 3-6 in targeted municipalities;

- parents/caregivers (especially the vulnerable ones), as beneficiaries of awareness raising actions on the importance of ECE and home parenting counselling;

- children aged 3-6, notably vulnerable ones (meaning ‘poor children’ and ‘children living in rural areas’ in the understanding of the Programme), as final beneficiaries of the new ECE services implemented in targeted municipalities.

Other relevant line ministries, parliamentary officials, CSOs and international development partners were also consulted during the Data Review and Collection Phase in relation to their role in the implementation of the Programme and/or of the overall Strategy for Early and Preschool Education 2016-2020 in Montenegro, but not as main units of analysis.


28

4. METHODOLOGY 4.1 Evaluation Phases

The evaluation was carried out in three consecutive phases, as shown in the figure below.

I. Inception Phase Kick-off discussion. In anticipation of the kick-off discussion, a list of needed data, documents and information was sent to UNICEF CO. The kick-off discussion (skype conference call), attended by the International Consultant (Team Leader) and UNICEF CO (Education Officer and Social Policy Officer – M&E focal point), took place on 9 September 2016. It was used to understand the client’s expectations and ensure that the evaluator is fully aware of the nature of the desired end-product; to get key information about the background of the object of evaluation, in particular the Montenegrin ECE system and bottlenecks addressed by the Programme; to clarify some Programme-related aspects (e.g. difference between outreach interactive services and alternative 3-hour programme; timing for the second wave of six municipalities to implement outreach interactive services; final beneficiaries of the Programme; process of developing the LIPs in three municipalities; etc.); to discuss the sampling of municipalities for site visits during the in-country mission; to learn about the local level key actors and interlocutors for the evaluation; to discuss the role and responsibilities of the local consultant to be recruited before the in-country mission in order to support the overall evaluation process; to agree procedures in order to have access to all needed documents, information and key informants throughout the evaluation (including organization of focus groups); to agree the timing of the in-country mission for primary data collection. During the skype conference call, the need for additional information to inform the evaluation was discussed and agreed to be provided by the CO within the deadlines indicated by the International Consultant. Preliminary Desk Review of Secondary Sources of Information. UNICEF CO has provided a comprehensive set of documents and links to relevant studies, surveys and web stories relevant for this evaluation. The Team Leader has also downloaded from internet other relevant strategies, reports, studies, statistical data, etc. She undertook a preliminary review of key programme documents (concept notes, ToC, Results Framework, MoE proposals, progress reports, resource utilization reports, field monitoring reports, RWPs, UNICEF’s situation analyses, Study from 2014, Country Programme Document and its Results Matrix, UNICEF annual reports and Mid-Term Review of the current Country Programme, several Programme deliverables) and available databases (e.g. MICS, TransMonee). She reviewed the human rights international commitments of Montenegro, in particular those deriving from the Concluding Observations of CRC and CEDAW. The review continued during the next phase of the evaluation when more systematic and structured review of key documentation related to the work of UNICEF and its partners in the target municipalities was done. Evaluability Assessment. An important element of the Inception Phase was to undertake an evaluability assessment of the scope of work planned, based on resources available. Within the scope of the current assignment, the evaluability assessment considered the Evaluation Questions (EQs), intervention logic and the associated indicators (descriptors) to determine the type of information that could be sourced from secondary sources and the areas which require primary data collection. The evaluability assessment indicated that there was a satisfactory level of information to answer most of the EQs, able to be processed in a concise and useful form, and that additional required information could be obtained within the existing resource constraints. A full evaluability assessment was provided in the Inception Report.

Inception Phase - 9 August to 5 October 2016

Data Review and Collection Phase - 10 October to 30 November 2016

Analysis and Reporting Phase - 1 December 2016 to 31 March 2017


29

Development of Methods and Tools for Primary Data Collection and Evaluation. Primary data collection methods and tools were designed to cover the identified information needs and ensure the required level of information needed to answer the EQs in relation to the various units of analysis. The data collection methods are presented in Annex 2, while the reasoning for their selection is explained below in the section 4.2 related to evaluation design. The following primary data collection tools have been developed during the Inception Phase:

Guides for in-depth interviews and focus groups as well as a Data Sheet for collecting information on the overall enrolment rates (Annex 3);

Mapping templates to be used for collecting information on Programme achievements (Annex 3);

Template with revised Programme indicators (based on the revised ToC and Results Framework), assigned along the hierarchy of results, to be used for mapping and analysing progress against set targets (completed version presented in the Effectiveness chapter).

The main evaluation tool which was designed during the Inception Phase was the Evaluation Matrix, grouping the evaluation questions under the evaluation criteria (Annex 4). A Sample of municipalities and a Control Group have been also constructed (Annex 5). The Inception Phase ended with the drafting of an Inception Report, which included the methodology, a detailed work plan, the full set of the proposed data collection and evaluation tools, sample and control group, proposed roles and responsibilities of parties in the evaluation process as well as an annotated outline of the final report. This Phase lasted from 9 August to 5 October 2016. II. Data Review and Collection Phase This phase consisted of an in-depth documentary review to gather secondary quantitative and qualitative data as well as of field work to collect primary data from key stakeholders at national and local levels, based on the data collection tools developed in the Inception Phase. The phase also included a 2-week in-country mission of the evaluation team (International and National Consultants) which took place in the period 31 October – 11 November 2016. During the in-depth documentary review, the evaluation team reviewed the remaining documents provided by UNICEF and implementing partners, key laws, policy papers and strategies in the area of education, health and child protection, an example of a local development plan prepared with the support of the Programme, relevant national and local level statistics/databases, independent reports and studies of CSOs, think tanks and international organisations and donors. The documents consulted for this evaluation are presented in Annex 6. Field data collection was carried out in order to respond to several overarching evaluation questions which required consultation of key stakeholders, deepening the analysis in the target municipalities and understanding the operations at the local level. In-depth interviews and focus groups with a representative number of key stakeholders were carried out in Podgorica and in the sampled municipalities. The municipalities forming the control group were contacted in writing and asked to provide information on the basis of the Data Sheet template mentioned above. The Data Review and Collection Phase lasted from 10 October to 30 November 2016, including a debriefing session with UNICEF on the preliminary findings of the evaluation.

III. Analysis and Reporting Phase Information and facts collected during the first two phases were analysed and integrated in the Draft Evaluation Report, in line with the ”UNICEF Evaluation Report Standards” (Evaluation Office, UNICEF NYHQ, September 2004). The analysis was based on the revised ToC and the Evaluation Matrix developed during the Inception Phase of the evaluation process. All standard OECD/DAC evaluation criteria, set in the ToR, were endorsed by the evaluation team for the following reasons:

- they are sufficient to provide a sound assessment of the quality, value and significance of the aid intervention, are all necessary and equally important;

- they are fully appropriate for the evaluation purpose, after careful examination of the Programme’s strategies and ToC;


30

- they are in line with internationally recognised best practice for a summative evaluation of a Programme and also consistent with recommended methodologies for a final evaluation of external assistance.

The standard evaluation criteria were approached as follows:

a) Relevance: alignment of the Programme to Montenegro’s priorities, international commitments and needs of stakeholders and beneficiaries. The assessment of the Programme relevance was based on the analysis of the national and local context and the challenges of the ECE and preschool education reforms to advance the realisation of children’s rights. The relevance was assessed in relation to country policies, strategies and priorities, equity agenda, European and international human rights commitments of Montenegro and, most notably, in relation to the needs and priorities of vulnerable children aged 3-6.

b) Effectiveness: the extent to which the Programme attained planned objectives and results. Using the ToC, the evaluation analysed to what extent the results obtained to date following the implementation of the Programme have contributed to the attainment of the planned objectives from the perspective of policies pertaining to ECE expansion, children’s access to quality and equitable ECE services as well as knowledge, attitudes and behaviour of parents/caregivers towards ECE. Using evidence collected, the report explains the factors that contributed or hampered the achievement of results and any unplanned significant contribution towards improvement of ECE service provision for children aged 3-6 in targeted municipalities. Evaluation of effectiveness paid special attention to the demonstration of/collecting strong evidence on the causal link between UNICEF intervention and the observed changes in the existing operating space of Montenegro.

c) Efficiency: the extent to which the management of the Programme ensured timeliness and an efficient utilization of resources to achieve its objectives. The report analyses how well UNICEF CO organized itself in delivering its work with regard to managerial, monitoring and budget efficiency of interventions at national, regional and community level. Analysis of efficiency was based on the assessment of outputs in relation to Programme inputs, costs and planned timelines. The report also incorporates a discussion of issues related to the assessment of cost-effectiveness of services developed by the Programme, using as far as possible benchmarking with similar services in the region. Complementarity and synergy with other relevant ECE interventions for children aged 3-6 and coordination with relevant stakeholders in the design and implementation of the Programme were reviewed as well.

d) Impact: long-term effects produced by the Programme interventions upon the social inclusion of vulnerable children and progressive realisation of children’s rights. The impact evaluation practice defines two types of impact evaluation: theory-based and counterfactual. The ToR request a theory-based approach reasoned by the fact that a non-intervention group of municipalities not exposed to the national and regional interventions of the Programme cannot be identified (page 12). While a counterfactual for the entire Programme was not feasible, a control group has nevertheless been constructed for the community level interventions of the Programme in an attempt to assess the preliminary quantitative impact of the Programme upon rights-holders, i.e. by comparing the effects of the Programme in the targeted municipalities (‘treatment’ group) to those not supported (‘control’ group). The results of the counterfactual were however not solid enough (Annex 9) and thus not finally used in the analysis of the quantitative net effect of the Programme upon the enrolment of children in ECE services. The assessment of the impact in terms of learning outcomes for children was primarily done from a qualitative perspective due to the limitations detailed in section 4.4 below.

e) Sustainability: extent to which the benefits (results) achieved by the Programme are

sustainable (are likely to continue beyond the intervention cycle). The evaluation assessed to what extent the outcomes and benefits generated by the Programme continue to exist without or with a lower level of external support. It also assessed the extent to which prerequisites for sustainability have been put in place and the concrete measures undertaken to date to ensure sustainability of results. Sustainability was analysed from various perspectives: legal/policy, institutional, capacity building, financial. The report highlights the factors that facilitated or hindered the sustainability of the results for children and their families.


31

The additional evaluation areas i.e. Human Rights Based Approach (HRBA) and cross-cutting issues specified in the ToR (page 10) have been integrated into the evaluation criteria above, as presented in the Inception Report. The Analysis and Reporting Phase took place from 1 December 2016 until 31 March 2017. The findings, conclusions and recommendations of the evaluation have been disseminated by the CO to national stakeholders and discussed with them during a validation workshop51 which took place on 24 March 2017. The evaluation team has also received written comments. The feedback from national stakeholders and the external UNICEF reviewer was used to draft the Final Evaluation Report released on 18 April 2017. 4.2 Evaluation Design

The design of the evaluation was based on a careful selection of data sources, data collection methods and data analysis methods, as detailed below. Data sources For each EQ in the Evaluation Matrix, specific data sources were identified and included in the Evaluation Matrix (see Annex 4, last column). Their selection was based on the following considerations:

They were relevant for informing the analysis of relevance, effectiveness, efficiency, impact and sustainability;

Used in a combined manner, they were able to increase the accuracy of information and mitigate the information gaps and limitations (see also Table 1 on evaluation limitations and mitigation approaches);

They were appropriate for embedding a diversity of perspectives in the analysis: planners, implementers and final beneficiaries; policy makers at central, regional and municipality level; external evaluators and reviewers; international development partners; civil society.

Data collection methods The methodology model designed for this evaluation aimed to utilize the best mix of data gathering tools to yield the most reliable and valid answers to the EQs and generate maximum learning within the limits of resources and availability of data. In order to serve its purpose, the evaluation applied a mixed-method approach52, including: stakeholder mapping; mapping of situation and contextual analyses, barriers and bottlenecks; in-depth documentary review and structured desk analysis of Programme design, implementation approaches, documenting of results and processes; structured desk analysis of policy documents and legislative frameworks; analysis and testing of the revised ToC; analysis of results from M&E systems and data at national and municipality level; aggregation and analysis of data collected via the Data Sheet and various mapping templates; contribution analysis to determine factors which promoted or impeded the progress against intended results and attribution analysis to the extent possible; systems analysis of management, monitoring, quality control and assurance strategies; mapping of risks analyses and mitigation measures; financial analysis; analysis of sustainability strategies and systemic barriers to sustainability; in-depth interviews and focus groups; direct observation and guided discussion with children during site visits to municipalities. Specific data collection methods have been assigned to each EQ in the Evaluation Matrix. These methods were selected because of the following reasons:

They were appropriate for the Programme strategy and intervention-level inquiry in municipalities.

They were feasible and sensible, following the preliminary data review in the Inception Phase.

They formed a relatively effective means of triangulation if used in a combined manner.

51 The workshop was not initially envisaged in the ToR. In order to ensure that the evaluation is highly participative in all its phases, a validation workshop was suggested by the International Consultant and accepted by the CO. 52 It combines various methods as a way to overcome limitations and highlight strengths - see Stern et al (2012), “Broadening the range of designs and methods for impact evaluations”, DFID, Working Paper 38.


32

Use of interviews, focus groups and direct observation in the field increased the breath of perspectives and data that could be secured; relying only on secondary data would have been insufficient; thus the field study was essential to ensure a degree of validity, as follows:

- In-depth interviews with key informants at national and local level were used to collect qualitative data and capture various perspectives on the way the Programme addressed the complexity of issues in ECE of children aged 3-6, how the Programme worked and how could have worked better for informing policy-making and reforms in ECE;

- The evaluation had to be informed by feedback from as many key stakeholders as possible, but it was not feasible to conduct individual interviews with all key informants given the resource constraints. Therefore, the evaluation used focus groups with preschool education professionals and parents/caregivers from all targeted municipalities;

- Based on the research experience of the International Consultant, the focus groups with professionals and parents/caregivers were organised in Bjelo Polje rather than on site, in the community, for the following reasons: the object of evaluation at local level is the testing of alternative ECE services rather than the community (municipality); there are risks associated with the presence of local leaders in the focus groups organised in the community who could affect the accuracy of the facts disclosed by participants and introduce a subjectivity element; some participants might not be able to interrupt the working hours in order to attend the focus group;

- Contact with final beneficiaries, as rights holders, i.e. children aged 3-6, was crucial for the evaluation. Given their small age, no interviews or focus groups were possible. Instead, the evaluation used focus groups with parents/caregivers and professionals to collect vital impressions on the relevance, accessibility, quality and impact of provided services on beneficiary children. The focus group with parents/caregivers were completed by several in-depth interviews with parents/caregivers of vulnerable children to add more insight evidence on issues of access. Those interviews also served as a back-up data collection method in case vulnerable parents/caregivers were unable to participate in the focus group (impossibility to leave the children alone at home while coming to Bjelo Polje, sensitivity of issues which might render a discussion in public/focus group not appropriate for ethical reasons, etc.). Direct observation of beneficiary children was also used during the site visits to sampled municipalities in combination with guided discussion to the extent possible;

Disaggregated data on results obtained by the preschools in target municipalities were needed to inform the assessment of relevance, effectiveness and impact; given significant data gaps, disaggregated data were collected via the Data Sheet and mapping templates referred above.

A representative sample of four municipalities (Bijelo Polje, Andrijevica, Danilovgrad and Pljevlja) for site visits has been constructed for the purpose of conducting in-depth interviews with key informants and getting factual evidence through direct observation. The sampling criteria are detailed in Annex 5;

A ‘control group’ of three municipalities (Bar, Niksic and Ulcinj) was also constructed, based on the quasi-experimental method of ‘simple difference’53, to allow a comparison with the targeted

municipalities (forming the ‘treatment group’) in terms of effects of the Programme on preschool education enrolment. The criteria used for the selection of municipalities in the control group are presented in Annex 5.

Data analysis The evaluation was designed to assess the relevance, efficiency, effectiveness, impact and sustainability of the Programme based on the concept paper, revised ToC, the Results Framework and indicators used for monitoring performance and attainment of estimated results. The ToR specify 18 EQs linked to the evaluation criteria mentioned. Based on this, a core analytical framework was drawn up in the Inception Phase in the form of an Evaluation Matrix (Annex 4), against which data were gathered and analysed. All EQs – formulated as in the ToR or reformulated, regrouped

53 It measures the difference between program participants and non-participants after the program is completed. The control group is made of individuals who did not participate in the program, but for whom data were collected after the program. Non-participants are identical to participants except for Project participation and were equally likely to enter the program before it started. Required data: after data for program participants and non-participants. Source: https://www.povertyactionlab.org/sites/default/files/documents/Experimental%20Methodology%20Table.pdf

https://www.povertyactionlab.org/sites/default/files/documents/Experimental%20Methodology%20Table.pdf


33

and streamlined on the basis of the logic model reflected by the revised ToC - have been assigned to a specific evaluation criterion (relevance, effectiveness, efficiency, impact and sustainability). New EQs have been introduced to capture important analysis dimensions and fuel the recommendations and lessons learnt54, while a number of EQs have been either removed if redundant or absorbed/bundled in broader questions. Two of the three EQs suggested in the ToR for evaluating the impact have been moved to the effectiveness section as they address the institutional/system changes (awareness raising of government and parliamentary institutions on the importance of ECE expansion) and behavioural changes (knowledge, attitudes and behaviour of parents), being linked to the Programme outcomes (1 and 3) in the revised ToC rather than the impact. The additional EQs in the ToR related to the application of the HRBA and cross-cutting issues have been incorporated in the relevance, effectiveness and impact EQs and corresponding indicators/descriptors (EQ3, EQ5 and EQ11). As a result, 14 EQs have been introduced in the Evaluation Matrix. For each EQ, related indicators/descriptors, data collection methods and sources of information have been specified in the Matrix. The indicators and methods were based on the preliminary documentary review conducted during the Inception Phase, experience from similar evaluations and a review of available data. Methods were set out per question, and the forms of triangulation between them made clear. An overview of the grouping of questions around the main evaluation criteria was presented in the Inception Report. The questions in the Evaluation Matrix have been to the extent possible framed based on the units of analysis, indicating their relative importance and the relative emphasis on the different target groups and beneficiaries. Analysis took place against the analytical framework in the following ways: across the analytical fields, which are drawn from the Evaluation Matrix; common trends, contradictions and differences were sought out and explored; b) the different pathways of contribution at different levels of results were tracked, identified and triangulated; explanatory factors related to the operating space, internal environment, design of the Programme approach and implementation of related interventions at municipality level were assessed; gaps in information available were identified and reported. Validity and reliability of analysis was ensured through triangulation, complementarity and interrogation55. Impartiality and lack of bias were safeguarded by the evaluation methodology which relied on a cross-section of information sources and used a blended methodological approach (quantitative, qualitative and participatory) to ensure triangulation of information through a variety of means. The design of the evaluation methodology combined a Results-Based Management with a HRBA to programming and evaluation i.e. achievement of planned results through morally-acceptable processes to realise human rights, in particular children’s rights. The HRBA applied by the evaluation was guided by five core principles: normativity, participation, non-discrimination, accountability and transparency, and by the Common Understanding on HRBA to Development Cooperation and Programming (United Nations Development Group, 2003):

All programmes of development cooperation, policies and technical assistance should further the realization of human rights as laid down in the Universal Declaration of Human Rights and other international human rights instruments.

Human rights standards contained in, and principles derived from, the Universal Declaration of Human Rights and other international human rights instruments guide all development cooperation and programming in all sectors and in all phases of the programming process.

Development cooperation contributes to the development of the capacities of ‘duty bearers’ to meet their obligations and/or of ‘rights-holders’ to claim their rights.

This evaluation identifies the “rights holders” of the Programme the children of pre-school age (3-6 years old), particularly those living in the poorest municipalities in the North of Montenegro, and their parents. Attention was given to the evaluation of the gender equality and mainstreaming issues reflected by the Programme, in line with Montenegro’s international commitments, most notably CEDAW and

54 i.e. European commitments (relevance); bottlenecks and barriers in access of vulnerable children to ECE (effectiveness); management, monitoring and reporting (efficiency); learning outcomes (impact) 55 Triangulation to confirm and corroborate results reached by different methods, complementarity to explain and understand findings obtained by one method by applying second and interrogation where diverging results emerge from the application of different methods.


34

UNICEF Gender Policy (2010)56 which states that UNICEF aims to work with partners to pursue gender equality and the equal rights of girls and boys “to contribute to poverty reduction and the achievement of the Millennium Development Goals (MDGs) through results-oriented, effective and coordinated action that achieves the protection, survival and development of girls and boys on an equal basis.” The evaluation assessed the extent to which the achieved Programme results contributed to the progressive realisation of children’s rights, equity and gender equality. In this respect, the Evaluation Matrix included specific EQs and indicators/descriptors i.e. EQs 3, 5 and 11. The evaluation used to the extent possible disaggregated data by gender and vulnerability profile. Ethical aspects During data collection, attention was paid to ensuring that the evaluation process was ethical and that participants in the process could openly express their opinions, protecting the confidentiality of their answers. Overall, the UNEG Code of Conduct for Evaluation in the UN System (March, 2008) was strictly respected, notably independence of judgement, impartiality, honesty and integrity, accountability, respect and protection of the rights and welfare of human subjects and communities, confidentiality, avoidance of risks, harm to and burdens on those participating in the evaluation, accuracy, completeness and reliability of report, transparency. The evaluators were sensitive to beliefs, manners and customs and acted with integrity and honesty in their relationships with all stakeholders, ensured that their contacts with individuals were characterized by respect, protect the anonymity and confidentiality of individual information. The process of recruiting stakeholders from different institutional levels followed a standard procedure in order to ensure an informed consent to participate in the evaluation (letter of introduction from UNICEF CO presenting the evaluation process, protection of privacy and information confidentiality, followed up by written/verbal communication regarding the interview/focus group/site visit details). Participation in the evaluation was voluntary and opinions are presented in the report in an anonymous manner. Selection of parents/caregivers (part of the target group, unit of analysis) for the focus groups and interviews was done in cooperation with the PSIs from the targeted municipalities, based on specific recruitment criteria57. Parents/caregivers were informed about the scope of the focus group/interview and its main discussion topics. Parents’ consent to participate was secured. Parents of vulnerable children were approached in a culturally-sensitive manner, based on UNEG Ethical Guidelines58. They were offered the opportunity to participate in the focus groups or to be interviewed individually on site. Given the small age of the children (final beneficiaries and rights-holders), the evaluation team used direct observation of several children groups in sampled municipalities. Their recruitment59 was done with the support of the teachers from targeted PSIs and based on prior consent of parents. UNEG Norms and Standards, including Guidance on Human Rights and Gender Equality in Evaluation60, were fully applied throughout the evaluation. 4.3 Stakeholders’ participation in evaluation

Involvement of stakeholders in the evaluation was of utmost importance for the collection of vital data and critical insights, but also for validating findings and conclusions as well as checking the feasibility of recommendations while ensuring buy-in. In gathering data and views from stakeholders, the evaluation considered a cross-section of stakeholders with potentially diverse views to ensure that the evaluation findings are as impartial and representative as possible (see Annexes 2 and 7). The evaluation was highly participative. In total, a

56 UNICEF (2010), “Working for an Equal Future. UNICEF Policy on Gender Equality and the Empowerment of Girls and Women”, New York 57 Gender balance (both mothers and fathers); employment status of mothers; vulnerability profile of parents and their children (children with disabilities/SEN, RE, poor, single-parent families, many children in the family, etc.); parents who were visited at home for tailored parenting counselling 58 UNEG (2008), “Ethical Guidelines for Evaluation”, http://www.unevaluation.org/document/detail/102, accessed 14 November 2016 59 Recruitment criteria: homogeneous groups in terms of gender and age (preferably 5-6 years old); children whose parents’ consent was obtained prior to the discussion 60 UNEG (2014), ”Integrating Human Rights and Gender Equality in Evaluations”, http://www.uneval.org/document/detail/980 , accessed 14 November 2016

http://www.unevaluation.org/document/detail/102

http://www.uneval.org/document/detail/980


35

number of 121 people have been involved during the evaluation61, representing key stakeholders of the Project: parents/caregivers, children, professionals, central public authorities, service providers (kindergartens), UNICEF and international development partners, academia, civil society organisations. The evaluation was so designed to ensure the involvement of stakeholders at three levels: information, participation and consultation, depending on the nature of each stakeholder and engagement in the implementation of the Programme. Interviews and focus groups were used for:

1) Informing the evaluation: primary data collection to cover gaps and add multiple informant perspectives;

2) Enabling participation of stakeholders in the evaluation: collection of qualitative, critical insights from stakeholders; direct participation of stakeholders in the testing of the revised ToC and results indicators as well as in the analysis and evaluation of the results and impact, also via self-evaluation;

3) Engaging stakeholders in consultation on future priorities and reforms: checking the perceived priorities for the continuation of the reforms in improving the access to quality ECE as well as the role each of them could play in the future. In this respect, specific questions have been included in the guides for interviews and focus groups (Annex 3) to capture the various perspectives of a large range of stakeholders.

This was very useful for informing the recommendations of the evaluation based on an open and participatory process started during the field data collection and which will continue on the occasion of the round table when draft findings, conclusions and recommendations will be presented, discussed and validated by the national stakeholders, and embedded in the Final Evaluation Report. 4.4 Evaluation Limitations

The ToR specify several limitations for this evaluation. The International Consultant has identified additional ones, which are presented in Table 1 together with corresponding mitigation approaches. Table 1. Limitations to the Evaluation and Mitigation Approaches

Potential Limitations/Constraints Mitigation approaches

Insufficient reliable and disaggregated data (by

municipality, by sex, vulnerability profile, etc.). No

basic demographic data on population aged 3-6. No

baselines and targets for several indicators in the

Programme concept note, Results Framework and

reporting documents, in particular on beneficiary

children (identified in the ToR and by the International

Consultant) – these data gaps might hinder the

robustness of assessment, in particular the Programme

relevance for vulnerable children and the measurement

of achievements against planned outcomes and outputs

from an equity and quality perspective. They might also

limit the full application of contribution and attribution

analysis at the higher levels of results (e.g. trends in

enrolment rates in the ‘treatment’ and ‘control’ groups of

municipalities).

Together with UNICEF CO, its national

counterparts and local implementing partners as

well as based on primary data collection methods

(interviews, focus groups), the evaluation team

strived to reconstruct the baselines and targets

retroactively and gather missing disaggregated

data to the extent possible. In case of persistent

information gaps, a more qualitative appraisal was

used, applying contribution analysis at different

levels, rather than a full inventory and assessment

of inputs, respectively results. The use of proxy

indicators and alternative datasets in cases of

simply unavailable data was also done in parallel

with the adoption of a mixed-method approach

(described in section 4.2) to ensure triangulation

through the use of multiple qualitative and

quantitative sources and cross-checking these on

an on-going basis.

Resourcing restrictions which limit the ability

to collect primary data and assess the Programme

strategy areas in a satisfactory manner (identified by the

International Consultant)

Agreed with UNICEF CO to contract a local

consultant to support the compilation of data in a

usable format and in a timely manner as well as to

contextualise the evaluation

61 33 interviews, 3 focus groups, 4 site visits, guided discussion and direct observation of 6 children groups (35 children) and validation workshop.


36

Potential Limitations/Constraints Mitigation approaches

Challenges in assessing the learning outcomes of

beneficiary children (impact level) as a result of

attending ECE programmes (identified by the

International Consultant) – The limited financial means of

the Programme and the rather short duration of piloting

alternative preschool services on the background of poor

system capacities, in particular of the Bureau for

Education Services, to measure the learning outcomes in

preschool education made the CO to avoid inclusion in

the Results Framework of learning outcomes.

Given the fact that the impact evaluation of an

ECE programme should invariably look at the

learning outcomes for children and having in view

the identified limitations, the impact of the

Programme from this perspective was still

analysed, but from a rather qualitative point of

view, based on the assessment of professionals

and perceptions of parents/caregivers as well as

expert opinion of selected key informants and

direct observation of children during site visits. In

this respect, corresponding qualitative indicators

have been developed and included in the

Evaluation Matrix (based on revised ToC) and the

template for the mapping of progress against

Programme indicators.

Unavailability of key informants (staff turnover in the

light of recent political developments in the country,

sensitivity issues, etc.) (identified by the International

Consultant)

The evaluation team mapped the key informants

and organised, in close cooperation with UNICEF

CO, interviews with people that left the respective

institutional stakeholder, if information obtained

from the existing technical staff was insufficient;

interviews were face-to-face or by Skype. In case

vulnerable parents/caregivers were not able to

attend the focus group in Bjelo Polje due to

sensitivity issues or impossibility to leave the

children unattended at home, the methodology has

included, as back up (but also for gathering more

insights), in-depth interviews with such

parents/caregivers during the site visits to

municipalities. Questions were formulated in a

manner that avoided any discomfort and

sensitivity. The evaluators created an open,

trustful atmosphere to facilitate open sharing of

views and opinions in full confidentiality.

Validity and reliability of interview data, particularly in

an environment where national sensitivities exist on

multiple issues, and where local political conditions

shape interaction with international consultants (identified

by the International Consultant)

Strict application of UNEG Ethical Guidelines and

Code of Conduct, ensuring confidentiality and

anonymity of interviewees at the start of interview.

Recognising the likely limitations of interview data

from some sources, and placing major efforts on

triangulation.


37

5. FINDINGS AND ANALYSIS OF RESULTS 5.1 Relevance

Evaluation of relevance was based on the following evaluation questions:

Alignment with country’s policies, strategies and priorities The evaluation has found evidence through desk research and consultation with key informants that there is alignment of the Programme design and its objectives with the needs and priorities of Montenegro in the area of early childhood education. It is acknowledged by decision-makers as essential for the holistic development of children aged 3-6 and as such integrated in the General Law on Education and the Law on Preschool Education. The Programme is in line with the country strategies and policy papers in the area of preschool education and child protection. The Strategy for Early and Preschool Education 2010-2015, which was in place when the Programme was designed, included among its objectives: to increase the coverage of children in early and preschool education through ensuring the development of diverse and expanded models of services of early and preschool education (objective 1); to create possibilities for the continuous professional development of staff working in the early and preschool education system (objective 3); to ensure the advocacy and promotion of the Strategy for Early and Preschool Education through the active involvement of the media and other forms of social communication (objective 6). The Programme has addressed these strategic objectives by designing and implementing outreach interactive services and alternative 3-hour services in order to boost the enrolment rates in ECE in the most needy municipalities. The setting up of these services were complemented by an intensive public campaigning effort to promote the importance of ECE among parents and change their attitudes towards enrolment of their children in ECE programmes. By end October 2016, with the support of the Programme, four training courses were delivered to PSI principals, teachers and support staff (psychologists, pedagogues) aimed at updating their knowledge and improving their practices in dealing with children aged 3-6 enrolled in outreach interactive services. One of the strategic objectives of the National Plan of Action for Children 2013-2017 is to ensure equal access for all children to quality education (Strategic Objective 6). The Plan defines more concrete measures and indicators under its Specific Objective 6.1 which aims ”to increase the coverage of all children with the early development and preschool education systems”, several of them being specifically addressed by the Programme, such as the introduction of new services in preschool education (interactive services and alike) by 2015 and attainment of the predicted coverage rates of children in ECE on an annual basis. By targeting children from remote rural areas in the poorest region of the country, the Programme has also addressed another indicator of the National Plan i.e. ”By 2017, increased number of children from vulnerable groups covered by preschool education”. Relevance for the European and internationally agreed goals and commitments The Programme is consistent with the European goals and commitments of Montenegro. It is aligned with current European strategies, most notably the Strategic framework for European cooperation in education and training (ET 2020) which calls for ”increasing participation in early

To what extent was the Programme aligned with the country policies, strategies and priorities?

Has the Programme been relevant in terms of European and internationally agreed goals and commitments of the country, norms and standards guiding the work of UNICEF in Montenegro and the Government?

Has the Programme addressed the underlying causes of low coverage of children in preschool education in Montenegro and responded to the needs of children aged 3-6, especially of the most vulnerable children?


38

childhood education as a foundation for later educational success”62. This is all the more relevant since participation in early childhood education and care in Montenegro is well below the EU target of 95% by 2020, as highlighted in the recent EU Progress Report.63 The Programme is fully in line with the international human rights standards ratified by Montenegro, in particular the CRC. The Programme has addressed the needs identified in the Concluding Observations of the CRC Committee64, for:

increased budget allocations for education (point 14), through: evidence-based policy advocacy and provision of technical assistance to the Ministry of Education for the development of the Strategy for Early Childhood and Preschool Education 2016–2020, which includes a recommendation to increase the preschool education budget from 0.38% to 0.53% of GDP (Goal 3, page 27);

strengthened skills of teachers (point 20), through: the development of manuals and guidelines for the teachers as well as the delivery of training courses to PSI principals, teachers and other professionals dealing with children aged 3-6;

improved awareness with respect to pre-schools and early-learning opportunities (point 60h), through: two awareness raising campaigns carried out in the Northern region.

The CRC Concluding Observations also calls for increasing access to early education of RE children and children with disabilities (points 14, 48g, 48h, 59c, 60d). As detailed in Chapter 2, the Programme has planned to increase the enrolment in ECE of children of preschool age (3-6 years of age) in the targeted municipalities, i.e. of those living in the poorest municipalities located in the North region of Montenegro. This might have had a spillover effect on other vulnerable groups (RE children and children with disabilities/SEN) - an important equity dimension which should be pursued and secured in the future, all the more since these two groups figure high among the priorities of the new Strategy for Early Childhood and Preschool Education 2016–2020 (Goal 1: intensive integration of the RE population in ECE; better detection and responses to the needs of children with SEN).

As far as CRC recommendation on establishing a consolidated system for comprehensive collection and analysis of data, disaggregated by age, sex, ethnic or social origin, urban/rural areas (point 16) is concerned, the Programme supported the improvement of MEIS which can now provide data on RE children and children with SEN, including children with disabilities. However, as explained in the previous chapter on country context, the lack of uniform registration in MEIS of children with disabilities on the background of lack of accurate data on the overall number of children with disabilities and methodological issues at the level of MONSTAT in recording the preschool population (based on age groups which do not correspond to preschool age groups) impede the possibility to “effectively analyze, monitor and assess the impact of laws, policies and programmes for all the areas covered by the Convention”65, as recommended by the CRC Committee. The Programme is relevant for the priorities of UNICEF in Montenegro, as addressed in the Country Programme Action Plan 2012-2016 (outcome 1, output 1.1). The Programme remained relevant in time, as demonstrated by the strategies and policy documents developed during its lifetime, in particular the Strategy for Early Childhood and Preschool Education 2016–2020 and 2030 National Strategy for Sustainable Development of Montenegro. The latter is based on the international Agenda 2030 and incorporates all its 17 Sustainable Development Goals (SDGs), including SDG 4 whose target 4.2 “By 2030, ensure that all girls and boys have access to quality early childhood development, care and pre-primary education so that they are ready for primary education” places an even greater emphasis on the significance of preschool education than has been the case with the MDGs. At the same time, there is a persistently high parents’ demand for ECE programmes such as the ones introduced by the Programme, and which explains the expansion of outreach services across the whole North region (from an initial number of 5 municipalities to 6 additional ones) as well as the piloting of alternative solutions (3-hour preschool programme). Relevance to needs of children and underlying causes of low coverage in preschool education

62 Council of the European Union (2009), “Council Conclusions on a strategic framework for European cooperation in education and training (ET2020)”, Brussels 63 European Commission (2016), “Montenegro 2016 Report”, Brussels, SWD(2016) 360 final 64 Committee on the Rights of the Child (2010), “Concluding observations: Montenegro”, Fifty-fifth session, CRC/C/MNE/CO/1 65 ibid, page 4


39

The stakeholders consulted for the purpose of evaluation – ministries, Parliament, professionals, parents/caregivers, academia, CSOs - have all confirmed the great need for ECE services for children aged 3-6 and were thus praising the MoE and UNICEF for its commendable initiative to pilot outreach interactive services in the Northern part of the country and promote enrolment in preschool education for all. The evaluation was informed that this Programme is the only one which addressed in a systematic and innovative way the needs of children of preschool age living in remote rural areas, highlighting the strategic value added which was contributed by UNICEF CO in Montenegro to the national efforts of enhancing the participation of children aged 3-6 in ECE. The importance of targeting this age group stems from evidence provided by developmental and neuroscience research that early childhood is the most critical developmental phase in life. Formation of cognitive and non-cognitive skills takes place mainly in the first 4 years. A child that has attended kindergarten only one year (or less) is performing better at the age of 15 in terms of learning outcomes. A child that attended kindergarten for more than one year is performing significantly better at the age of 15 and is ahead of his/her peers who haven’t attended preschool by one academic year66. Adverse childhood experiences including deprivation, abuse, neglect and malnutrition can have a lifelong impact on development, mental and physical health, and productivity.67 MoE and UNICEF CO’s decision to invest in developing services for children aged 3-6, as rights-holders, was informed by the results of the Study from 201468, according to which the overall enrolment rate in ECE is very low and there are high disparities in the access to and enrolment in preschool services across the country, with coverage of children varying from quite high (88%) in the Southern and Central regions to very low in the North (27%). Out of 14 municipalities listed in the Study (Table 3, page 36 in the Study) as having an attendance rate below the national average of 52.4%, the Programme worked in 10 of them of which six (Bijelo Polje, Berane, Andrijevica, Rozaje, Savnik and Pluzine) belong to the group of municipalities with the lowest enrolment rate in the country (27%) (Figure 2). There is no data available concerning the socio-economic status of the beneficiary children69. Nevertheless, given the fact that the Northern municipalities are also the most under-developed and poorest in the country and that three-thirds of children affected by poverty live in rural areas in the North, the Programme has most likely addressed the poor children, who are known to gain the most from preschool education and overcome poverty status in their lifetime. There is a strong correlation between the level of development of a municipality and the number of children who attend ECE (Figure 5) and it could be seen that the Programme has operated in the ones with the lowest development level.

66 Presentation of B. Perks, UNICEF Representative, called “Expanding Early Childhood Education in Montenegro” to high level conference in July 2014, Podgorica 67 Engle, P. (2011), “Early child development: Why should we invest in the health and development of young children?” in Early Child Development: What Parliamentarians Need to Know and Do, UNICEF; Shonkoff, J., Boyce, W. T., and McEwen, B. S. (2009), “Neuroscience, Molecular Biology, and the Childhood Roots of Disparities: Building a New Framework for Health Promotion and Disease Prevention. Summary of Essential Findings”, in JAMA, 3 June 2009, Vol. 301, No. 21, pp. 2252–2259. 68 Prica, I., Čolić, L., Baronijan, H. (2014), “A Study on Investing in Early Childhood Education in Montenegro”, UNICEF Montenegro, Podgorica 69 In the absence of data on poverty status of children enrolled in the outreach interactive services, the evaluation team requested the PSIs to provide information concerning the employment status of parents or the recipients of Material Assistance (the main social assistance means-tested benefit for poor families), in an attempt to use it as a proxy indicator. However, the PSIs were unable to provide such information, except for two cases.


40

Figure 5. Municipality development index and the coverage rate of children in ECE, 2012

Prica, I., Čolić, L., Baronijan, H. (2014), “A Study on Investing in Early Childhood Education in Montenegro”, UNICEF Montenegro

Data indicate that the Programme has also successfully reached children aged 3-6 from rural areas (Table 2) where the attendance rate of ECE is almost three times less compared to the one in urban areas, as already mentioned in the previous section of this report. Table 2. Beneficiary children of outreach interactive services (status at 30 November 2016)

Total* Gender

Rural children Children with

disabilities/SEN RE children

girls boys

391 174

(45%)

217

(55%)

391

(100%)

5

(1.3%)

0

(0%)

* School years 2015/2016 and 2016/2017, first wave of municipalities (Danilovgrad, Rozaje, Berane, Andrijevica, Bijelo Polje) Source: Programme data collected from beneficiary PSI and MEIS

There are very few children with disabilities/SEN among the final beneficiaries and no RE children (who live in uran areas), demonstrating the fact that they were not targeted by the outreach interactive services. As pointed out by UNICEF CO, the limited scope and resources of the Programme have not allowed the inclusion of all children in need among the planned final beneficiaries, but they will be considered in the upcoming programmatic efforts of UNICEF and Government counterparts. The Programme has been guided by the HRBA to programming. The overarching human rights principles of non-discrimination, equality, rule of law and use of maximum available resources were implicit in the implementation and monitoring of the Programme. As mentioned above, there have been some equity issues raised by the design of the Programme. The participation principle was well represented at the level of education authorities and professionals, in their capacity of duty-bearers, by engaging them in various Programme activities (design of the new strategy, design and delivery of new ECE services, conferences, trainings, awareness campaigns, monitoring). Even though the associations of parents were involved in the conferences and in the activities of campaigns and have participated in the working group for development of new Strategy for ECE, the engagement of parents whose children attended new ECE services was seemingly limited. Namely, apart from awareness campaigns and initial meetings to present the interactive services, there is no evidence that participation of parents was specifically encouraged in, for instance, the daily activities of the interactive services or in the monitoring of the quality of services. Similarly, there is no evidence of promoting the voice and views of children aged 4-6 in the design, delivery or quality monitoring of the new ECE services implemented by the Programme. The empowerment principle worked well at the level of duty-bearers, notably the MoE, PSI


41

principals and professionals working in interactive services, but it was rather negligible as far as parents were concerned. According to the feedback from the focus group with professionals, home visits to vulnerable parents for parenting counselling took place in only two of the six municipalities where interactive services have been implemented; no space was arranged in the location of the interactive services for individual and confidential discussion with parents70; and it is unclear which was the benefit of the workshops and meetings with parents reported by the professionals during the third training course. Given its profound CRC-oriented nature, the budget of the Programme was fully rights-based. Although gender equality was not an explicit strategy pursued by the Programme, there was a rather good gender balance among the beneficiary children and professionals. The concept note does not include any reference to gender issues. The formulation of target groups, final beneficiaries and indicators is gender-blind. Progress reports do not report gender disaggregated data nor do they contain specific discussion on any possible gender issues raised during the implementation of the Programme. Following the request of the evaluation team, UNICEF and preschools have kindly compiled gender disaggregated information. According to data made available, boys were better represented among the final beneficiaries of the Programme (55% boys compared to 45% girls), due to their overrepresentation in the overall population of preschool age in targeted municipalities. The Programme data regarding the professionals who benefitted of training courses show that 87% were women, reflecting the female-dominated feature of the education sector71 rather than a result of a specific gender-sensitive strategy employed by the Programme. 5.2 Effectiveness

Evaluation of effectiveness considered the following evaluation questions:

Main Programme achievements at outcome and output levels Analysis of Programme effectiveness has been a challenging exercise for the evaluation team given the lack of targets for many indicators and unclarity concerning outcomes and outputs in the Programme design documents on the background of a ToC which required a significant revision. As explained in section 2 above, the evaluation has revised the ToC and assigned indicators to various Programme outputs and outcomes, based on the indicators in the Results Framework. In the case of results where indicators were not specified in the Programme documents or in the Results Framework, the evaluation team reconstructed the respective indicators and targets, to the extent possible, as they were needed to measure the progress achieved along the results chain in the revised ToC and respond to the EQs in the ToR. They were presented in the Inception Report and approved by UNICEF. Reconstructed targets and ToC were validated with key stakeholders during the in-country mission. Data reflecting the achievements of various results (outputs and outcomes) which were not reported in the progress reports were then compiled by UNICEF, following the request of the evaluators, and triangulated with feedback from interviews and primary data collected from principals of PSIs and MEIS.

70 Report of the training course, 24-25 March 2016, page 3 (English version) 71 In EU the gendering of the early childhood education and care (ECEC) is a matter of concern. In its Communication from 2011 “Early Childhood Education and Care: Providing all our children with the best start for the world of tomorrow”, the European Commission states that there is a very important issue of gender balance among ECEC, as almost all of them are women. A few countries have set targets for the recruitment of men into ECEC or sought to redesign the profession to reduce gendering. The Communication calls for measures to make a career in the ECEC sector more attractive to men in all EU countries.

What is the achievement level of planned results (quantitative and qualitative) compared to stated objectives?

To what extent did the Programme interventions contribute to expanding ECE in Montenegro and reducing bottlenecks and barriers in the access of children to quality and equitable ECE, especially of the most vulnerable ones?

What were the main factors which contributed or hindered the achievement of the intended Programme outcomes to date?

Has the implementation of the Programme produced any additional, unplanned effect (positive or negative)?


42

According to the revised ToC, the Programme envisaged to achieve three outcomes and eight outputs. Their achievement level until 30 November 2016 (the cut-off date of primary data collection for the evaluation) is assessed below. It is to be noted that the Programme will be implemented until end June 2017 and there are big chances that all envisaged results will be fully achieved.

► Outcome 1 and its related outputs have been partially achieved (Table 3) Table 3. Achievement of Outcome 1-related indicators

Results Indicators Baseline/Target Status at

30.11.2016 Assessment

Outcome 1: Enabling policy framework for ECE expansion country-wide

Budgeted, accountable and evaluable ECE expansion strategy and action plan approved by the Government

No/Yes

Strategy and action plan approved, but not evaluable

Partially achieved

Output 1.1: The dialogue on the need and strategic development directions of the ECE in the country initiated through knowledge generation and sustained

Evidence-based ECE Investment Case available for public presentation

Number of high-level conferences held

Not available/ Available* 0 (2014)/1

Available 1

Achieved Achieved

Output 1.2: Key government stakeholders and professionals have an overview of the long-term benefits of investing in ECE and of possible scenarios for expansion of ECE

Number of high-level meetings held 2 (2015)/14 14 Overly-achieved

Output 1.3: New ECE expansion strategy developed in line with the generated knowledge products

Existence of a new, knowledge informed ECE expansion strategy

Number of high-level conferences on quality education, including ECE, to present the expansion strategy

No (2015)/Yes 0/1

Yes 1

Achieved Achieved

Output 1.4: Framework for Monitoring and Evaluation of the ECE strategy developed

Availability of a Framework for Monitoring and Evaluation

No (2016)/Yes No Not achieved

* baseline and target reconstructed by the evaluation team and validated with UNICEF and national stakeholders72

The Programme created the overall prerequisites of an enabling policy framework for ECE expansion at the country level. It did so by supporting the development of an evidence-based Strategy for Early Childhood and Preschool Education 2016–2020 and Action Plan which were further adopted by the Government (November 2015). The Strategy sets the goals and directions for preschool education in terms of improving its accessibility, quality and sustainability, it is budgeted but not evaluable at the level of its three goals and thus not sufficiently accountable. The Strategy lacks concrete targets73 and a M&E Framework; the indicators included in the action plan are process or output indicators at the most. Nevertheless, the feedback from interviews and focus groups with a wide range of stakeholders from both central and local level acknowledged that the Strategy is a valuable guiding document for decision-makers, professionals, academia, CSOs and community members in designing, implementing and monitoring progress in the area of ECE reforms. It is also appreciated as being “short and concrete” (Government informant). With few exceptions, the strategy and its action plan are well known by the stakeholders due to the highly participative nature of its design and the intensive policy advocacy of UNICEF in partnership with the MoE (high-level conferences and bilateral meetings). Grounded in a solid evidence-based Investment Case in ECE, provided in the Study from 2014, the robust and easy-to-understand advocacy arguments have driven a fundamentally different level of

72 As retroactive reconstruction of targets might induce bias in assessment, the evaluation team strived to reduce it to the extent possible by asking the national stakeholders to recollect past decisions and desiderata, and by systematically applying triangulation to validate the reconstructed targets for all relevant outcome and output indicators. 73 The first draft of the Strategy contained a target of 95% enrolment rate to match the EU target, which was however removed by the Government from the final version, being replaced by the syntagma “in line with international standards”.


43

debate and understanding on ECE compared to the past, even among stakeholders not necessarily associated or familiar with this area. The Programme has thus won an argument in the public space on the need to invest in ECE by steering informed policy debate and building support in the society for ECE for which UNICEF and the MoE need to be highly commended.

► Outcome 2 and its related outputs have been achieved (Table 4) Table 4. Achievement of Outcome 2-related indicators



Outcome 2: Improved access to new ECE services for children aged 3-6, in particular of vulnerable children in the targeted municipalities

Number of children enrolled in outreach interactive services, disaggregated by residence (urban/rural), sex

% of children aged 3-6 enrolled in PSIs supported by the Programme of the total number of eligible children in the targeted municipalities (villages)

0/300, of whom: 300 in rural areas (target not able to be reconstructed for age disaggregation) n.a./target not able to be reconstructed due to gaps in data on the overall population 3-6

391, of whom: 391 rural; 174 girls and 217 boys Unknown

Overly-achieved for overall number and rural children (not possible for other disaggregation, lack of target) Not possible, lack of target

Output 2.1: Outreach interactive pre-school services made available in targeted municipalities and up-scaled

Number of municipalities offering interactive services (first wave)

Inclusiveness of enrolment criteria in outreach interactive services

Availability of a plan for upscale Number of municipalities where

interactive services have been up-scaled (second wave)

No. of professionals who completed the training courses in interactive services, disaggregated by sex

% of trained professionals who apply acquired knowledge and skills in delivering interactive services

Parents/caregivers at risk are able to indicate at least 2 positive parenting practices that they acquired following the family visits of professionals

0 (2015)/5 n.a./High* No (2016)/Yes 0 (2015)/6 (2016) 0/36* 85%* Not applicable/Yes

6 Moderate (high for rural/remote children, but not prioritising the most vulnerable (e.g. the poorest, children with disabilities/ SEN) Yes 6 46, of whom 40 women 85% Unknown

Overly-achieved Partially achieved Achieved Achieved Overly-achieved Achieved Not possible, lack of data on progress

Output 2.2: Models for ECE expansion developed

Number of developed LIPs for ECE expansion

0 (2015)/3 3 Achieved

Output 2.3: Alternative 3-hour preschool programme piloted

Availability of a 3-hour programme

Number of municipalities where the 3-hour programme is piloted

No. of professionals who attended the presentation of the 3-hour programme and were supported to design implementation activities

% of professionals who consider that they are able to deliver the 3-hour programme at the required standard due to the attendance of presentation and support

No (2016)/Yes 0 (2016)/3 0/9* n.a./100%*

Yes 3 14, of whom 13 women 100%

Achieved Achieved Overly-achieved Achieved

* targets reconstructed by the evaluation team and validated with UNICEF and national stakeholders


44

The Programme has managed to improve the access of 391 rural children aged 3-6 (key rights-holders) to ECE by implementing free of charge outreach interactive services in the rural areas of six municipalities (first wave: Bijelo Polje, Berane, Andrijevica, Rozaje, Plav, Danilovgrad) where no ECE opportunity existed (Table 5). The planned number of beneficiary children was exceeded by 30.3%, indicating the big demand for ECE in the target municipalities, a fact which was confirmed by parents, principals of PSIs and professionals who were interviewed during the site visits. Nevertheless, the lack of data concerning the overall number of children aged 3-6 (for reasons explained in Chapter 1), who would have been eligible for ECE services, made impossible a precise assessment of the extent to which the Programme met the needs in the villages where interactive services were implemented. Table 5. Beneficiary children of outreach interactive services, by municipality (November 2016)

Municipality Total Year of enrolment

2015/2016 2016/2017*

Andrijevica 40 35 5

Berane 109 75 34

Bijelo Polje 80 51 29

Plav 22 22 n.a.

Rozaje 73 53 20

Danilovgrad 67 51 16

TOTAL 391 287 104

*Enrolment was just starting when data was collected for the evaluation. It is expected that many more children will enrol. There is also evidence that some former beneficiaries ‘migrated’ towards the alternative 3-hour programme.

As a result of vigorous awareness raising campaigns, demand has been created in other municipalities, hence the development of a plan for upscaling of interactive services to an additional number of six municipalities (second wave: Kolasin, Pljevlja, Zabljak, Mojkovac, Pluzine, Savnik) in the Northern region and which will increase the number of beneficiary children by around 350 until the end of the Programme74. The evaluation identified some quality issues which may put at risk the excellent advocacy results and recently-created demand and expectations of parents. Quality loss is mainly caused by the following reasons:

Inadequate learning infrastructure in some locations: obsolete heating system, lack of running water, electricity cuts, outdoor/age inappropriate toilets;

Mixed-age large groups: difficulties for the teachers to cope with large heterogeneous groups and stimulate an attractive and interactive learning environment for all;

Insufficient induction training for ‘new comers’: no systematic training in place for new teachers, which is essential for quality assurance given significant turnover among preschool teachers;

Ununiform duration of the education programme, varying from 2 hours per week in some locations to 3 hours per week;

No parents’ engagement in quality monitoring of interactive services. Capacity building of professionals included several initiatives carried out under Outcome 2. A “Manual for kindergarten teachers – learn through play” and “Guidelines and teaching tools on developmental and educational phases” were developed by the Pedagogical Center of Montenegro and the MoE, respectively, and made available to all relevant professionals; the interviews with key informants and focus group with professionals did not confirm however the extent to which these materials are used in the everyday work routine. In addition to manual and guidelines, professionals benefited of four accredited training courses aimed to increase their capacity to deliver the interactive services and stimulate the development of children. A number of 46 professionals (principals of PSIs, teachers, psychologists, pedagogues) from 12 target municipalities graduated the training courses provided by the Programme. In total, around 60 hours of training have been provided until October 2016. Review of training reports and satisfaction questionnaires indicate that the trainings for outreach interactive services were appreciated due to their informative, relevant, practical and interactive nature.

74 According to planned number in the proposal of the MoE Ref.no: 600-48/2016.


45

“The guidelines on how to approach parents were very useful for us to reach-out to parents in distant rural areas.” “The toy library model that was introduced to us was very useful and applicable”

(focus group with professionals, November 2016)

According to the feedback received from key informants during interviews and focus group, the majority of trained professionals (85%75) has managed to internalise the methodologies and make full use of what they learnt, thus being able to stimulate child development and learning. The evaluation team wanted to learn about the extent to which parents/caregivers at risk have acquired positive parenting practices following the family visits of professionals involved in interactive services, and the overall partnership with parents for the purpose of stimulating healthy and well balanced development of a child. The review of Programme documentation and the feedback from the focus group with professionals and parents confirmed that the participation of parents in this process was minimal, while family visits were very few. The limited number of interviews with parents at risk afforded by the evaluation and the lack of any monitoring/progress data related to the work with parents have impeded a more thorough assessment. In addition to interactive services, capacity building has also addressed 14 professionals planned to be involved in the piloting of the alternative 3-hour programme in three municipalities (Andrijevica, Bar and Danilovgrad) located in each of the regions of the country, for whom specific curricula (approved by the National Council of Education in July 2016) and Local Implementation Plans were designed with the support of the Programme. Compared to outreach interactive services, the alternative 3-hour programme is also free of charge, but it has a longer duration (3 hours/day compared to 2-3 hours/week), is provided in city kindergartens (urban areas compared to rural areas) and is aimed to address the demand in excess primarily in the Southern and Central regions of the country which creates overcrowding of children groups in the kindergarten. It is to be mentioned that the decision to design and implement both outreach interactive services and alternative 3-hour programme was based on the Study from 2014 and the new strategy for 2016-2020 mentioned several times above.

► Outcome 3 and its related outputs have been overly-achieved (Table 6). Table 6. Achievement of Outcome 3-related indicators



Outcome 3: Increased demand of parents/ caregivers for ECE programmes

Rate of increase in enrolment of children (3 – 6 years) in municipalities targeted by the campaign, in comparison to the year prior to the campaign

n.a./5-10% increase*

As a result of 2015 campaign: increase of 4%76 to 24% (mostly above 5%) As a result of 2016 campaign: increase of 11% to 31%

Overly-achieved

Output 3.1: Parents/caregivers in targeted regions understand the importance of the ECE for their children’s development and education

Awareness raising campaign on ECE importance

Percentage of parents/caregivers who declared that the campaign encouraged them to enrol their children in preschool

Percentage of parents/caregivers who started the procedure of enrolling their children in kindergarten following the campaign

0/1 0/20%* 0/10%*

2 As a result of 2015 campaign: 43% As a result of 2016 campaign: 23% As a result of 2015 campaign: 44% As a result of 2016 campaign: 53%

Overly-achieved Overly-achieved Overly-achieved

* targets reconstructed by the evaluation team in cooperation with UNICEF and validated by national stakeholders

The Programme has increased the demand of parents/caregivers for ECE programmes which translated into increased enrolment of children aged 3-6 in preschool far beyond planned levels.

75 The remaining ones are not able to fully use what they learnt at the training courses due to constraints specific to each location (e.g. lack of space in the kindergarten to host the toy library), teachers’ turnover, etc. 76 A modest result was obtained in Rozaje – only 4% increase – due to massive labour migration, including families with children of preschool age which decreased the demand for ECE services.


46

The KAP surveys77 show that as a result of the awareness raising campaigns “Preschool for All” organised by the Programme in 2015 and 2016 in the municipalities of the North region, the rate of increase in enrolment was around 18.6%. It meant that 222 children would have not been enrolled in ECE without the campaigns implemented by the Programme (Figures 6 and 7), in addition to the 391 children enrolled in interactive services. Indeed, the campaigns mainly targeted the urban areas in the North region, with under-used capacities and low demand, while enrolment in interactive services was primarily due to the information meetings of PSIs with parents in the villages, as confirmed by the key informants during the interviews and focus groups. Figure 6. Increase in enrolment in preschool following 2015 campaign (no. of children and %)

Figure 7. Increase in enrolment in preschool following 2016 campaign (no. of children and %)

This was the net effect of the campaigns as confirmed by parents and given the fact that there was no other intervention of the Government, development partner, CSO or alike to compute the results. In 2015, 4 in 10 parents started the procedure of enrolling their children in kindergarten following the campaign, while 5 in 10 parents did so in 2016, compared to planned 1 in 10. Nevertheless, the effect of the campaigns on the enrolment of the most vulnerable children is unknown, as vulnerability was not explicitly targeted, monitored and assessed. According to a local implementation partner, only 15 parents of children with disabilities/SEN participated in the events of the campaign, of whom 8 parents with children already enrolled in kindergarten. Factors promoting or hindering effectiveness The factors which increased the effectiveness of the Programme were the following:

- evidence-based, systemic and multi-angled approach of issues and stakeholders to create consensus in society on the need to invest in ECE: this approach made possible an

77 KAP July 2015 and KAP July 2016


47

increased awareness of the government and other key decision-makers on the importance of expanding the ECE and a decisive change of knowledge, attitudes and behaviour of parents in favour of ECE. It also helped to address in a concerted way many of the bottlenecks and barriers in the access of children to quality and equitable ECE, notably financial barriers, lack of preschools in the proximity and unfavourable social norms. It provided solid arguments, based on scientific evidence and costed scenarios, of the benefits of investing in ECE.

- high level of leadership, commitment and in-house technical expertise of UNICEF CO materialised in tenacious and ceaseless efforts towards ensuring efficient strategic guidance and technical assistance for the development of an enabling environment and expansion of service provision in parallel with stimulation of demand from parents to enrol their children in preschool.

- blended implementation strategies which were used in concert to influence the determinants of low access to ECE towards positive outcomes for children, i.e. knowledge generation, policy advocacy and dialogue, communication for social change, piloting and institutionalisation of new services;

- efficient team work between the MoE, UNICEF CO, local kindergartens, NGOs and academia (Faculty of Philosophy), as acknowledged by participants in the validation workshop.

At the same time, there were several factors which hindered the effectiveness of the Programme, as noted in the achievement level of certain planned results, as follows:

- quality safeguards: more effective quality safeguards and monitoring would have been needed to be built by the MoE in partnership with UNICEF in order to ensure observance of minimum quality requirements in service provision, in particular: a) the learning infrastructure, since the Programme has not envisaged any investment in the rehabilitation of the space provided by rural schools for outreach interactive services, compared to a similar project in neighbouring Serbia (this is a quality shortcoming fervently highlighted by parents during interviews and focus group, but also by professionals as early as February 201678); b) the size of mixed-age groups and its impact on the quality of the learning process; c) induction training for new teachers; d) sufficiency of teachers79 (see unplanned results below and analysis in the Sustainability chapter); and e) participation of parents in quality monitoring;

- empowerment of parents: apart from initial meetings to inform them about the benefits of ECE and home visits (which were very few and in only two municipalities), no other specific activities have been envisaged to empower the parents with effective positive parenting practices in order to set the building blocks of a partnership with teachers towards the common goal of stimulating child development and improving learning outcomes.

Unplanned results and their implications There is evidence of some additional, unplanned results of the Programme identified during the interviews with key informants and focus group with professionals. Examples of such results and their implications (effects) include the following:

setting up of ECE services in other localities of the target municipalities, triggered by the Programme; effect: apart from the Programme beneficiary children, an additional number of around 50 children aged 3-6 benefit of outreach interactive services (Luzac/Berane - 20) and 3-hour programme (Odzak/Pljevlja – 20, and Kralje/Andrijevica - 10) and have early learning opportunities; their mothers have the possibility to return on the labour market and reconcile the professional and private life; teachers could make use of various deliverables of the Programme, most notably curricula for 3-hour programme, manual and guidelines;

implementation of 3-hour programme in an additional number of 4 municipalities compared to the initially planned 3 municipalities; effect: as in the case above, an additional number of children aged 3-6 benefit of ECE opportunities; return on the labour market of mothers; use of curricula for 3-hour programme, manual and guidelines;

78 See Report of the training course, 4-5 February 2016, page 3, English version 79 During the validation workshop which took place in March 2017, the MoE and a PSI principal confirmed the employment of new teachers in an attempt to overcome staff shortage.


48

job creation; effect: seven professionals (teachers and support staff) have been employed as a result of setting up new ECE services in their municipality

provision of volunteering opportunities to students; effect: by engaging 100 students from Pedagogical faculty in the organisation of the awareness raising campaigns, their hands-on knowledge about the challenges of the ECE reform has been improved; it has also steered reflection on their possible role in pioneering change in their future profession at the very practical level of work with children, parents and community as a whole;

shortening of the preschool hours for children enrolled in regular kindergarten in one of the target municipalities in order to allow their teachers to teach in interactive services (due to lack of resources to employ other teachers); effect: substitution effect, as children attending the kindergarten are affected in their learning process so as other children have access to ECE via interactive services.

5.3 Efficiency

Evaluation of efficiency was based on the following evaluation questions:

Programme management and monitoring The Programme has been managed in a professional, efficient and culturally-sensitive manner. The Programme management was ensured by UNICEF CO (social policy and education sections) in partnership with the MoE and local implementing partners. The framework for cooperation with the MoE was represented by the Rolling Work Plans (RWPs) signed by the Government and UNICEF, while a more detailed description of the work to be done and required financing were provided in two proposals submitted by the Ministry to UNICEF (as a basis for Direct Cash Transfers – DCTs) as follows: “Interactive Services Programme” (Ref.no: 618-22/2015-2, dated 11 November 2015); and “Support to Preschool Coverage and Quality of Delivery” (Ref.no: 600-48/2016, dated 29 March 2016). The local implementing partners (two non-governmental organisations - NGOs, one school and one faculty) were selected based on a direct agreement procedure, given the comparative advantage of each of them in the implementation of specific activities of the “Preschool for All” campaign, as follows:

- the Pedagogical Centre of Montenegro, which was in charge of community mobilisation in partnership with preschools, is the only Montenegrin member organisation of the International Step-by-Step Association80 promoting early childhood development across Europe and Central Asia; the Centre is a highly regarded NGO for its expertise and has excellent connections within the preschool network which facilitated its work during the campaign;

- the Association of Music Schools, the Music and Ballet School and the Faculty of Dramatic Arts were the best positioned organisations to perform the artistic activities on the agenda of the campaign.

80 http://www.issa.nl/content/who-we-are, accessed 20 December 2016

How well has the implementation of the Programme been managed? What monitoring and reporting systems and tools have been used and how did they contribute to the Programme management process?

Were financial resources used appropriately? Were funds managed in a cost-effective manner to achieve expected results? Could the same results have been achieved with fewer resources?

To what extent and how has the implementation strategies of the Programme contributed to achieving better synergies, avoiding overlaps with similar interventions?

http://www.issa.nl/content/who-we-are


49

The local partners were contracted by UNICEF through a Small Scale Funding Agreement, following a micro-assessment of their technical and financial management capacity, which is the standard procedure. There was no Steering Committee or alike as an organised form of multi-stakeholder engagement in strategic decision-making and strategic orientation of the Programme, to integrate in a more straightforward manner the ECE within the broader early childhood development reform in the country and ensure a more efficient approach of equity and quality issues highlighted in the previous chapters. Nevertheless, the Council for Structural Reforms and Improvement of the Business Environment - the highest governmental body in charge of initiating reforms – was informed about the Programme and was a key target of policy advocacy by UNICEF as were the two most relevant line ministries i.e. the Ministry of Labour and Social Welfare and the Ministry of Health, and the Ministry of Finance. In additions, presentations were organised for Parliamentary Committees for education, health, gender, human and minority rights. Internally, the evaluation was informed that all issues, bottlenecks, progress made and successes achieved during the implementation of the Programme were discussed at weekly staff meetings as well as at regular Program Coordination Meetings and meetings of the Country Management Team. UNICEF CO was in charge of the technical and financial management, monitoring and quality assurance of the Programme as well as reporting to UNICEF HQ and indirectly to the donor. The feedback from stakeholders confirms that the management of the Programme was conducted professionally and in an efficient and culturally-sensitive manner. The Programme stakeholders depict UNICEF as “a professional, respected and reliable partner”, with a “high level of leadership and commitment”, “understanding well the context of our country, which is not the case with other international partners”, being “open to new solutions”, bringing “value added” and being “best placed to support the Government in the implementation of ECE reforms in our country”. UNICEF is also popular among the parents who are grateful for the support received so far, but have some higher expectations concerning the quality of service provision as openly spoken out during interviews and focus group (see the Chapter on Effectiveness). The system used by UNICEF and its implementing partners to monitor the Programme is presented in the Box below.

Programme monitoring

The overall Programme has been monitored by UNICEF Montenegro CO. Day-to-day monitoring of community-level activities i.e. outreach interactive services and alternative 3-hour programme, has been ensured by the MoE, according to their proposals which based the DCTs.

Monitoring has been done on the basis of the following documents:

a) Programme concept note; b) Proposals of the MoE mentioned above and corresponding budgets; c) Quarterly progress reports of the MoE; d) FACE (Funding Authorization and Certificate of Expenditures) forms filled in by the MoE e) Small Scale Funding Agreements with the local implementing partners, and their progress reports; f) Fund monitoring by grant template; g) Mid-term and annual progress reports of the CO to UNICEF HQ; h) UNICEF reports of programmatic visits to the MoE for the DCTs.

Results monitoring has been impeded by the lack of a TOC and Results Framework81, with complete and clear results chain and targets for all indicators, as already explained in the Effectiveness chapter of

the report.

Process monitoring strived to assist the adaptation of Programme strategies to meet the arising needs in the field and cope with risks, but has not always managed to do it successfully. The Programme

concept note does not contain any monitoring provisions (e.g. process, instruments, responsibilities, etc.) and risk mitigation strategies. Notwithstanding, interventions at national and regional levels were adequately monitored, and fast and firm action was taken when changes in the political environment put at risk the timely implementation of the Programme. However, this was not the case at community level. UNICEF CO paid site visits to several target municipalities and rural schools hosting the interactive services82, but no trip reports

81 These documents have been developed retroactively, only for the purpose of the evaluation, as mentioned in Chapter 2. 82 Andrijevica, Bijelo Polje, Lazine, Pavino Polje


50

are available, apart from reports on site programmatic visits to the MoE and one report following a visit to two outreach sites. The CO informed the evaluation that such reports were not requested, but it is good practice to document the monitoring scope, process and follow-up. There is no evidence that quality issues (see section 5.2 above) related to enrolment policy, respectively the learning environment, staff turnover, teachers’ shortage etc. were identified during monitoring by the MoE, the BfES or the preschool teams and reported back to UNICEF. The proposal of the MoE on interactive services (Ref.no: 618-22/2015-2), which based the DCT and thus raised contractual obligations, included site monitoring activities and a specially dedicated budget. The site monitoring visits were planned to be carried out by MoE, BfES and PSI teams. The interviews carried out for the purpose of this evaluation and review of Programme documentation revealed the following:

- the MoE paid visits in the field primarily for the promotion of the importance of ECE and specific services (outreach interactive services and alternative 3-hour programme) rather than for actual monitoring; the evaluation was informed that this was partly due to the severe understaffing of the Preschool and Inclusive Education Section which was in charge of the Programme within the MoE; nevertheless, the principals who participated in the validation workshop confirmed the valuable support received from the MoE which helped them in the implementation of the new ECE services;

- there were no site visits aimed to monitor the interactive services or the operation of toy libraries carried out by the BfES (also confronted by insufficient human resources);

- the PSIs teams paid rare site visits to the rural schools hosting the interactive services; they partly or fully used the monitoring funds for other needs (e.g. payment of fees and transportation for teachers, purchase of furniture and didactical materials, etc.) since they were not aware of the specific monitoring destination of these funds.

The progress reports of UNICEF CO follow the template provided by UNICEF HQ for reporting to the donor. They are well structured and informative, providing useful information about the implemented activities, obtained results and utilised budget; however, disaggregated data (by gender, deprivation, ethnicity,

disability, etc.) were not collected and reported.

The progress reports of the MoE are also well structured and informative, but less analytical, with incomplete reporting on constraints and ways how they were overcome. Apart from progress reports and FACE, day-to-day communication and regular meetings between the MoE and UNICEF were used to monitor the progress of the Programme. Nevertheless, these meetings have not been minuted and there is no evidence of how issues were discussed and followed up with remedial action.

Human rights principles of non-discrimination, equality, rule of law, accountability and progressive

realisation of children’s rights were embedded in the monitoring work. The participation and inclusion principle was given less attention, particularly in relation to parents’ engagement in quality monitoring and inclusiveness of the enrolment policy of the target preschools respectively.

There have been some delays in the implementation of the Programme due to changes in the leadership of the MoE, which affected the timely adoption of the new ECE strategy and shortened the duration of the interactive services with consequences on the learning outcomes of beneficiary children. In such circumstances, UNICEF CO has mobilised itself in an exemplary way and pushed the strategic decision-making process through a strong and tireless advocacy effort, demonstrating that progress could be achieved although functioning in a very challenging context. A six-month no-cost extension was granted to UNICEF HQ for the overall initiative “Unleashing Children’s Potential”, including the Programme in Montenegro, that will allow the finalisation of all activities in good condition and settling any implementation issues. Most outputs are of good quality and accessible to relevant stakeholders, and the immediate impact and results of the Programme are significant as confirmed by a large number of people consulted for this evaluation. Financial management and cost-effectiveness Financial monitoring was based on regular financial reports (of UNICEF CO to HQ/Donor and of implementing partners to UNICEF CO), HACT (Harmonized Approach to Cash Transfers) rules and regulations, including FACE83, spot checks on utilization of funds and site monitoring visits. The reported financial data indicate that there was straightforward spending as per approved budget by UNICEF, with funds spent in an accountable and cost-effective manner. Most grant funds were allocated to the setting up and running of outreach interactive services, design and capacity building for piloting the alternative 3-hour programme, development of LIPs and consultancy (over 55%84), while most of UNICEF own contribution was used for the implementation of the “Preschool for All’ campaign and associated KAP surveys (around 87%). The Programme management and overhead costs (coordinator,

83 All UNICEF partners have been trained in HACT. 84 According to ‘Fund Monitoring by grant’ at 19.01.2017 provided by UNICEF CO.


51

printing of materials, etc.) of the MoE, representing 4.4% of the total budget of its planned interventions85, were in line with the usual thresholds for such costs, the market prices and salary levels for the respective level of qualification, experience and responsibility in an international Programme. All implementing partners have strictly followed UNICEF requirements for budget reporting and breakdown. According to financial reports, the amount that was spent until mid-January 2017 was US$ 403,961 (Annex 8), representing 99.4% of the total budget. The remaining amount was planned to be used for this evaluation and for the organisation of the validation workshop. As far as cost-effectiveness of human resources is concerned, internal staff of UNICEF was engaged to supervise the overall implementation of the Programme, review and comment on various Programme outputs, without any additional costs for the Programme. Local experts and trainers were hired to the extent possible, leading to important cost savings for fees and translation costs. Additional efficiency features of the Programme consisted of the use of available premises in rural schools for the delivery of ECE services to children, building the training curricula and endowment specifications on the ones developed in a previous project of UNESCO86, and centralised procurement of toys and didactical materials for all target municipalities in order to minimise the costs, ensure unitary endowment of all outreach interactive services and permit as far as possible coordinated implementation schedules at local level. Centralised procurement by UNICEF was also efficient time wise – if the MoE were to do it using public tender procedures, it could have taken much more time and created delays in Programme implementation. The procurement process was transparent and guided by the value-for-money principle. The average investment was around € 1,566 (US$ 1,65187)/outreach interactive service, meaning an initial investment/child of around € 104 (US$ 110)88. There are no available national benchmarks or in neighbour countries, since these outreach interactive services are rather new to the country, while the other countries opted for different types of services. As far as 3-hour alternative programme is concerned, the Study in 201489, which informed its piloting by the Programme in three municipalities, has estimated the unit cost/child at € 266.5 annually. A similar 4-hour programme implemented in neighbour Serbia costed € 489 annually90, meaning around € 366.8 if adjusted to 3-hour format to compare with the Montenegrin model. From this perspective, the services that have been piloted by the Programme in Montenegro and which are going to be expanded country-wide seem to be more cost-efficient. However, it should be noted that the unit cost in Serbia included preschool rehabilitation works as well. As far as the awareness raising campaigns “Preschool for All” are concerned, the Programme invested US$ 73091 for each child who was enrolled in ECE as a result of the campaigns in 2015 and 2016. It is highly commendable that the campaign has been fully evaluable from the point of view of its results, a feature which is many times absent in campaigns carried out by UNICEF in other countries. The CO in Montenegro has rigorously set the baseline through KAP surveys which were repeated after the campaign to measure the results i.e. change in attitudes and demand for ECE as well as increase in enrolment rates. The Programme has a significant efficiency characteristic given by its very nature, i.e. investment in early childhood education, which brings about tremendous returns in the future for the individuals, communities and society generally. Numerous studies demonstrated a large range of benefits, from improved academic outcomes for children who benefitted of ECE and reduced

85 Based on budgets attached to the MoE proposals Ref.no: 618-22/2015-2 and Ref.no: 600-48/2016, which grounded the DCTs from UNICEF. It does not include UNICEF overhead costs. 86 Pilot project implemented in Danilovgrad, Berane and Rozaje in 2013. 87 Exchange rate 30.12.2016, http://www.cb-cg.org/eng/ 88 It includes the initial endowment, capacity building of staff and monitoring divided by the number of beneficiary children in school year 2015-2016 in the first six municipalities (first wave). Note: generally, monitoring is not a direct investment, but in this particular case the budget for monitoring was actually used for other purpose (as explained in Box 1), i.e. costs which fall under the category of direct investment, and therefore has been considered in the calculation of the unit cost. 89 Prica, I., Čolić, L., Baronijan, H. (2014), “A Study on Investing in Early Childhood Education in Montenegro”, UNICEF Montenegro, Podgorica 90 UNICEF (2012), “Investing in Early Childhood Education in Serbia”, Belgrade, cited in the Final Evaluation Report of the Project “Expanding Early Learning Opportunities for Vulnerable Children’’, financed by IKEA and implemented by UNICEF Serbia. 91 Grant and UNICEF own contribution less KAP costs.

http://www.cb-cg.org/eng/


52

dropout rates to economic savings for states92. Another important reference document is the 2011 Communication of the European Commission93 which notes that early childhood education and care is an essential foundation for successful lifelong learning, social integration, personal development and later employability and that it is particularly beneficial for the disadvantaged and can help to lift children out of poverty and family dysfunction. Initial Programme investment in the endowment and development of guides and training modules for professionals could be quickly recovered in the coming years as more and more children will enrol in the ECE services and benefit of child-centred early learning programmes. In other words, education and development returns in the future will pay for initial investment. From this perspective, efficiency of the Programme might have been higher on long-term in case the final beneficiaries included vulnerable children to a bigger extent (children with disabilities/SEN, RE children, etc.). As put it by James J. Heckman, Nobel Prize laureate in economics, "The highest rate of return in early childhood development comes from investing as early as possible, from birth through age five, in disadvantaged families."94 The evaluation has nevertheless identified some efficiency issues. Firstly, apart from providing space in existing rural schools to host the outreach interactive services, the Programme design has not envisaged any cost sharing whereby target municipalities were required to cover a certain level of costs or additional needs (e.g. basic rehabilitation works of locations where parents and teachers recalled poor learning conditions). There are very few examples of voluntary local contribution (e.g. carpet, wood for heating, some furniture, school bus, breakfast for children) and it seems that this was a missed opportunity for UNICEF to have used its core role of leveraging resources from local communities for the benefit of children aged 3-6. Secondly, outreach interactive services have been implemented in Danilovgrad which was part of the first wave of target municipalities. Its selection is questionable for various reasons: 1) it belongs to the group of municipalities with high development level (group 5-6 of development) rather than of the poorest; 2) it had an enrolment rate in preschool of 47% in 2013, making it part of the municipalities with a medium level of enrolment rather than low; 3) it benefitted in the past of a similar programme financed by UNESCO, so there was no need for additional endowment of one of its locations (Slap) and for a repetition of training of the principal and teacher all the more since the training curriculum was the same. Thirdly, the discontinuation of interactive services in four locations and staff turnover produced some additional investment loss. Complementarity, synergy and cooperation with relevant stakeholders The evaluation found out that the Programme built on the existing policy and legal environment, formal partnerships and cooperation mechanisms with the Government and experience of working at community level. The Programme has also built on the results of a former pilot project of UNESCO (2013) by using the training curriculum in its training operations as well as the technical specifications for the endowment of the interactive services, as already mentioned above. Nevertheless, it is unclear to what extent the Programme was complementary to the UNICEF-supported work on ECD (parenting, nutrition, health care) and inclusive education95 and what were the benefits of synergy if it did so. British Council in Montenegro has worked on quality assurance and trainings for English teachers. According to the feedback received from its representative, British Council is increasingly interested in ECE, particularly from the perspective of introducing English in preschools. To date, there has been exchange of information between UNICEF CO and British Council, and there are opportunities for future cooperation in the area of monitoring and quality assurance of teaching in preschool. Cooperation of UNICEF with central authorities, in particular with the MoE, but also with other relevant line ministries (labour and social welfare, health, finance) and the Council for Structural Reforms and Improvement of the Business Environment went very well. Interviews with key informants confirmed that

92 Reynolds, A., Rolnick, A., Englund, M. & Temple, J. (2010), “Early childhood development and human capital”; Heckman, J., Moon, S.H., Pinto, R., Savelyev, P.A., Yavitz, A. (2009), “The Rate of Return to the High/Scope Perry Preschool Program”; Hill, C. (1999), “Early Learning, Later Success: The Abecedarian Study”, Highlights of the Age 21 Follow-up Study: University of North Carolina, FPG Child Development Center 93 European Commission (2011), “Early Childhood Education and Care: Providing all our children with the best start for the world of tomorrow”, Communication from the Commission, COM(2011) 66 final, Brussels, 17.2.2011 94 http://www.heckmanequation.org/content/resource/invest-early-childhood-development-reduce-deficits-strengthen-economy, accessed 11 December 2016 95 For instance, with the training of preschool teachers and member of local commissions for orientation on early detection and intervention - see Briefing Points for the meeting at MoE, 11 February 2015, under heading “Inclusive education - Current initiatives (2014-2015)”

http://www.heckmanequation.org/content/resource/invest-early-childhood-development-reduce-deficits-strengthen-economy


53

the MoE in partnership with UNICEF have consulted them on the needs and priorities of ECE and of children aged 3-6. They were also extensively informed about the results of the Study from 2014 and cost-effective scenarios for the expansion of ECE in an attempt to achieving universal coverage by 2020. In this respect, they appreciated the fact that suggested scenarios have not been focused on investing in new facilities and infrastructure, but rather on increased organizational/budgetary effectiveness and efficiency, and also flexibility in modalities of delivery of preschool education. At local level, cooperation of UNICEF and the MoE with preschools and rural schools went also very well and there have been no particular issues reported to the evaluation team. There were extensive consultations with the local communities to present the benefits of early education for children aged 3-6 and get the local commitment for a successful implementation of the Programme in the respective communities. The staff was cooperative and the deadlines were generally respected. The only area where better coordination and more active engagement would have been required refer to the health and social protection services as far as identification of vulnerable children for priority enrolment in interactive services was concerned. 5.4 Impact

Evaluation of impact considered the following evaluation questions:

According to revised ToC, the Programme aimed to contribute to increased coverage of children aged 3-6 in preschool education and improved learning outcomes for beneficiary children in line with international human rights standards and the principle of progressive realisation of children’s rights. As explained in the evaluation limitations in section 4.4 above, the assessment of the impact of the Programme was a very challenging undertaking, given basic demographic data gaps (population aged 3-6) and the lack of monitoring data on school readiness and other relevant development outcomes for beneficiary children. Therefore, the evaluation used proxy indicators to highlight to the extent possible the impact of the Programme in the enrolment and learning outcome areas. Impact on the coverage of children aged 3-6 in pre-school education

Indicator: Increase rate in the number of enrolled children in ECE in the target municipalities

The data collected from the PSIs (first wave) triangulated with data retrieved from MEIS and monitoring data indicate that the Programme increased the overall number of children enrolled in public ECE programmes in target municipalities from the North region by 7% to 55% in the school year 2015/2016 (one year after the start of the Programme) (Figures 8 and 9 below).The highest impact was recorded in Andrijevica, a predominantly rural municipality, with villages located often high in the mountains, badly connected to towns by public transport and where children and families were not exposed to ECE through indirect experience of relatives or neighbours. The Programme has thus more impact exactly in areas which needed the most. Increase in enrolment would have been higher if not affected by migration of families with children of preschool age to more affluent South or Central regions or for work abroad.

Has the Programme contributed to an increase in the enrolment of children in preschool education, especially of the vulnerable children, in targeted municipalities?

To what extent has the Programme impacted on the learning outcomes of children aged 3-6 in the targeted municipalities?


54

Figure 8. Children enrolled in ECE in target municipalities in school year 2015/2016

Figure 9. Increase rate of enrolled children in target municipalities in school year 2015/2016 (%)

Impact on the learning outcomes of children aged 3-6 A quantitative assessment of the Programme impact upon the learning outcomes of children who benefitted of outreach interactive services was not possible for the reasons explained in section 4.4. Nonetheless, the evaluation has sought to assign plausible qualitative contributions of the Programme to higher level results using the available Programme data, triangulated with feedback from focus groups with parents and professionals, guided discussion and direct observation of children during site visits and expert opinion of selected key informants.

Indicator: Satisfaction level of parents/caregivers concerning the effects of new services on the development of their children (low/moderate/high)

The evaluation assesses the satisfaction level of parents to be moderate-to-high. Parents, in their capacity of both duty-bearers and rights-holders, are grateful for the learning opportunities provided

64

522

743

355

198

150

3575

51 5122

53

-50

50

150

250

350

450

550

650

750

Andrijevica Berane Bijelo Polje Danilovgrad Plav Rozaje

Total no. of children enrolled in ECE No. of children enrolled in interactive services

55%

14%

7%

14%

11%

35%

0%

10%

20%

30%

40%

50%

60%

Andrijevica Berane Bijelo Polje Danilovgrad Plav Rozaje


55

to their children in the outreach interactive services set up with the support of the Programme. Most appreciated are the daily activities, the interaction with teachers and the overall staff professionalism. Due to teachers, children go with pleasure to kindergarten and are very attached to them. Concerns were raised mainly in relation to large mixed-age groups and improper learning infrastructure (see Effectiveness chapter) as well as the short duration of the education programme in the interactive services (2-3 hours/week) which made some parents move the children to regular urban kindergartens to the extent possible or just keep them at home when conditions were improper. As far as observed learning and development outcomes are concerned, parents were able to identify several positive effects on their children as a result of attending the interactive service:

“My child now reminds her older siblings to wash their hands and take care of their things.” “He has learned about responsibility and work habits.” “My daughter learnt to behave properly when eating and she is teaching her younger siblings now.” “My child didn’t want to speak before coming to the preschool, and he was 3 years old!”

(focus group with parents, November 2016)

The figure below presents in a more structured way the feedback received from parents96. Figure 10. Effects of the Programme on beneficiary children (parents’ perspective)

Indicator: Preschool professionals are able to provide at least 2 examples of progress made by beneficiary children in each of the following areas of child development and learning: language, social-affective, cognitive, motor, as a result of the Programme (yes/no/partially)

The evaluation was informed by the professionals who were consulted during interviews and focus group that the impact of attending the interactive services on beneficiary children is obvious in terms of language and socio-affective development, being able to identify 3-5 effects in each of these child development areas (Figure 11)97. At the same time, they consider that the effect is modest on the motor development and insignificant as far as cognitive development is concerned due to the very short duration of the education programme. It is nevertheless to be noted that their assessment is based on children’s portfolios and observation checklists discussed during the trainings and not a rigorous assessment of the learning outcomes.

96 The evaluation considered the following elements in analysing the four domains of development: Motor development - in

particular fine motor skills needed to engage in smaller, precise movements, normally using the hands and fingers; Socio-affective

development - compliance to rules (ability of the child to understand and respect rules), social interactions (ability of the child to

initiate and maintain positive social interactions with both children and adults), prosocial behaviour, comprehension of emotions,

emotional expression and emotional self-regulation; Cognitive development - child’s ability to process information (attention,

memory, categorization, planning and problem solving); Language development - production of speech or verbal fluency, ability

to generate words according to semantic categories. 97 Based on children’s portfolios and observation check-lists.


56

Figure 11. Effects of the Programme on beneficiary children (professionals’ perspective)

As mentioned in the methodology, the evaluation team used guided discussion and direct observation of children (10% of beneficiary children) during the site visits and shares the views of the professionals. As seen in Figure 12, the children enrolled in 3-hour programmes (meaning 3 hours/day, 5 days/week) are more developmentally on track compared to those enrolled in the interactive services, especially as far as cognitive development is concerned. Figure 12. Effects of the Programme on beneficiary children (evaluators’ perspective)

Indicator: Contribution of the Programme to the progressive realisation of children’s rights in target communities (none/modest/good/significant)

The Programme had a significant contribution to the implementation of the principle of progressive realisation of children’s rights in Montenegro which was part of the monitoring of human rights principles in general. It did so through: analytical and assessment work to provide evidence on the supply, demand and financing options for universal coverage of children in ECE; opening up of access to organised ECE programmes for children aged 3-6; introduction of child-centred practices in ECE of children; development of the understanding of professionals, as duty-bearers, on their role in providing quality education services and of their capacity to allow them perform at highest professional standards. 5.5 Sustainability

Evaluation of sustainability considered the following evaluation questions:


57

Current functioning of ECE services in target municipalities and scaling-up One year after their start up in the first six municipalities from North region, the outreach interactive services are still functioning in 11 out of 19 locations (58%). According to the feedback of principals of PSIs (November 2016), other four are planned to re-start soon, while other four were discontinued especially because of improper conditions (Table 7). Table 7. Status of Outreach Interactive Services (November 2016)

Municipality Location Status at 9.11.2016

functioning discontinued planned to re-start soon

Rožaje Balotiće

Kalače

Skarepača

Plav Bogajiće

Brezojevice Berane Buče

Budimlje

Donja Ržanica *

Dolac Andrijevica Trešnjevo *

Gračanica

Seoce **

Slatina ** Bijelo Polje Poda

Pavino Polje

Kanje Danilovgrad Slap

Lazine

Novo Selo * TOTAL 19 11 4 4

* transformed into daily 3-hour programme; **children transported to regular kindergarten in Andrijevica town

The positive aspect is that three of the still functioning interactive services have been transformed into longer programmes and children enrolled in two of the discontinued services are currently attending the regular city kindergarten. The worrying aspect is that the outreach interactive services (both those which are still functioning and planned to re-start soon) are under financial risk especially due to difficulties in paying the teachers, a matter which needs to be urgently sorted out by the MoE. In some municipalities, teachers who were full-time employed in kindergartens were also used to teach in the interactive services, as an additional task98. The need for new recruitments was brought to the attention of the MoE

98 According to focus group with professionals, interviews with principals of PSI and Report of the training course 24-25 March 2016 (page 3, English version)

To what extent are the results (benefits) from the Programme sustainable or likely to be maintained over time?

What measures has UNICEF undertaken to ensure that results to which it contributed are not lost?


58

during the first training course in December 201599 and apparently it was an issue which could have been settled rather smoothly given the provisions of the legal framework regulating the interactive services. The issue has been reiterated during the second training course in February 2016100. While in some municipalities additional staff was recruited, in other municipalities this was not the case, although badly needed. Since enrolment in the new school year 2016/2017 was just starting at the time of the evaluation, it was not possible to check if the number of beneficiary children was kept more or less identical compared to the previous year. The working hours are still the same i.e. 2-3 hours/week. According to the principals of PSIs, not all teachers trained by the Programme are in place. In several cases, the respective teachers were transferred to regular kindergartens, while in other cases this was due to staff turnover; trained teachers were replaced by new comers who were however not provided systematic training in running the service, working with children and parents, etc., apart from sporadic mentoring by the principal or pedagogue of the respective PSI. The didactical materials and toy libraries procured by the Programme are fully used in the still functioning services, but it is unclear what happened with them in the four locations where services were discontinued. According to the upscaling plan101, an additional number of 350 children from rural communities of Kolasin, Pljevlja, Zabljak, Mojkovac, Pluzine and Savnik is planned to be enrolled in outreach interactive services. In addition, expansion of ECE will be also done through the implementation of alternative 3-hour programmes across the country. At the time of the evaluation, they were in a piloting stage in three municipalities for whom LIPs were developed by a team of consultants; according to principals of PSIs, the engagement of the community in the process was minimal (only provision of some data) at the cost of low ownership and certain reservations about the applicability of the recommended lines of action. The evaluation has also some reservations concerning the working hypotheses used by the authors to estimate the need for ECE102 and has already brought them to the attention of UNICEF CO during the debriefing at the end of the in-country mission. Measures put in place to ensure sustainability of results

► Changes in policy and strategic framework sustaining Programme results The current policy and legal framework governing the preschool education system in Montenegro is supportive for the development and expansion of early childhood education services for children aged 3-6 which were piloted and put in place by the Programme. As already explained in the Relevance chapter, the Programme goes in line with the existing policies and legislation which it directly enriched and influenced, particularly the new Strategy for Early Childhood and Preschool Education 2016–2020 and the amended Law on Preschool Education. In addition, ECE figures explicitly in the new 2030 National Strategy for Sustainable Development of Montenegro through specific targets and indicators mirroring the SDG 4 “Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all”. This overall policy and strategic framework provides a good basis for expanding the access and coverage of preschool education over the next years. The evidence provided by the Programme concerning the benefits of ECE upon child development as well as UNICEF work on costing and piloting new services (outreach interactive services and alternative 3-hour programmes) will form an important body of arguments for the continuation of reforms and expansion of preschool education provision for children aged 3-6 across the country. According to various studies consulted for the purpose of this evaluation and interviews with key informants at central level as well as feedback from principals of PSIs (during interviews and validation workshop) and parents, there is a high demand for ECE services in practically all regions of the country, which is likely to act as a lever for resource mobilisation in the future. The results of this evaluation are also aimed to provide information, findings and recommendations for improved policies in the area of ECE and changes in the regulatory framework.

99 Report of the Training course, 24-25 December 2015, page 2 (English version) 100 Report of the Training course, 4-5 February 2016, page 3 (English version) 101 The MoE proposal “Support to Preschool Coverage and Quality of Delivery” (Ref.no: 600-48/2016, dated 29 March 2016) includes the scaling up plan and was the basis for the second DCT to the Ministry. 102 For instance, in the LIP for Andrijevica, the needs analysis is based on the hypothesis that there will be no changes in the number of newborns or the depopulation rates, so that the number of children aged 3-6 will be the same over the period between 2015 and 2020 (footnote 1, page 5, English version of LIP), which is highly doubtful.


59

► Capacity of preschool education professionals

Investment of the Programme in human capital was aimed to familiarise the ECE professionals with setting up and running outreach interactive services, adaptation of teaching curricula to these services context and use of tailor-made tools and improvement of work with children and parents to stimulate a healthy, harmonious and comprehensive child development. According to the findings of the evaluation, all professionals who benefitted of these learning opportunities reported positive experience from the trainings in terms of knowledge acquisition and skills development. During the focus group discussion, professionals were able to give practical examples to illustrate their statements and highlight specific areas concerning children’s rights that they would like to improve in their work. As mentioned in the Effectiveness chapter, most professionals (85%) have embedded the acquired knowledge and skills in their usual work routine with children and parents. Parents, in their capacity of both duty-bearers and rights-holders, have expressed particular appreciation for the teachers whom they consider to be well qualified and mastering efficient methods of working with their children, with good learning results. However, as mentioned above, not all trained teachers are still working with children in the interactive services and there was no solid mitigation strategy put in place for coping with staff turnover (e.g. induction training, work shadowing, etc.). The accredited training curricula, the “Manual for kindergarten teachers – learn through play” and “Guidelines and teaching tools on developmental and educational phases” developed by the Programme provide nevertheless an excellent starting point for developing a standard programme of induction training to be attended by all professionals who work with children aged 3-6 in outreach interactive services or alike.

► Financial resources At national level, the Programme managed to influence the prerequisites for financial sustainability through the recommendation to increase the allocation to ECE from 0.38% of GDP to 0.53% included in the new Strategy for Early Childhood and Preschool Education 2016-2020. At the same time, UNICEF has presented various forms of ECE service provision and corresponding funding scenarios to decision-makers aimed to ensure a full coverage of children aged 3-6 in ECE by 2020 (based on the Study from 2014). According to interviews with key informants from Government, these proposals were welcome and will be considered in the planning of national budgets in the years to come. Based on the respective proposals, the MoE has already developed its own document looking into most feasible scenarios, entitled ”The State-of-play in Preschool Education: Infrastructure, Coverage, Teaching Staff and Equipment”. At municipality level, the financial sustainability of outreach interactive services implemented by the Programme is more challenging, as already mentioned above, and it might have been overcome in case two prerequisites were embedded in the design and implementation work, as follows: 1) the setting up of interactive services would have been conditioned upon an official, contractual commitment of the MoE and respective PSIs to ensure the running of the services after the end of Programme funding at an appropriate level of quality; and 2) systematic monitoring of the sustainability prerequisites would have been carried out during implementation by the MoE and UNICEF to address some of the quality and financial constraints issues identified by the evaluation. Regretfully, this was not the case. Interviews with the MoE and principals of PSIs tend to confirm that the outreach interactive services set up with the support of the Programme are currently part of the regular service provision for children aged 3-6 in the respective municipalities. Nevertheless, it is the opinion of the evaluation that support is badly needed to overcome the current financial constraints and increase access of most vulnerable children to ECE until solid rights-based foundations of practices and procedures are built and sustainable capacities are in place to ensure that services run effectively and at the required quality standards. This is all the more important since the outreach interactive services have been extended starting with school year 2016-2017 to other six municipalities in the North region and they may benefit of lessons learnt with shortcomings overcome at a quicker speed.


60

6. CONCLUSIONS AND RECOMMENDATIONS 6.1 Conclusions

The Programme has won an argument in the public space on the need to invest in ECE by steering informed policy debate and building support in the society for ECE for which UNICEF CO and the MoE, the key driver of the Programme (as acknowledged during the validation workshop) deserve high praise. Rather than focusing on a specific bottleneck, the two partners opted to implement a comprehensive and concerted action upon all determinants of ECE expansion (enabling environment, supply and demand) to produce system change for improved outcomes for children.

The Programme implemented new ECE services where they were unavailable (outreach interactive services in rural North); it campaigned where existing capacities were underutilised in order to create demand (in urban North); and it diversified the ECE service menu in areas with over-demand and over-crowded capacities (alternative 3-hour programme). The immediate net impact was 613 children enrolled in ECE due to the Programme, a number which is planned to increase to 963 children once the interactive services are expanded across the whole North region.

The investment in improving the service provision has been supported by a strong and tireless policy advocacy of UNICEF CO, grounded in solid evidence and based on robust and easy-to-understand arguments. The advocacy efforts have driven a fundamentally different level of debate and understanding on ECE compared to the past, even among stakeholders which were not necessarily associated or familiar with this area. The result was the creation of prerequisites for an enabling policy framework for universal coverage of children in ECE programmes by 2020, in particular the adoption by the Government of the Strategy for Early Childhood and Preschool Education 2016–2020, which foresees an increased allocation to ECE from 0.38% of GDP to 0.53%, and Action Plan.

The Programme was fully in line with the national priorities in the area of preschool education as well as with the European and international human rights commitments of the country. The ECE services designed, tested and implemented by the Programme proved to be highly relevant for the needs of children and parents, although it paid insufficient attention to the empowerment of parents with positive parenting practices. The Programme remained relevant in time, as demonstrated by the strategies and policy documents developed during its lifetime and which the Programme has influenced, and the persistently high demand of parents for ECE services.

Implementation of the Programme has delivered valuable results in terms of: a) developed strategic building blocks for ECE expansion country-wide; b) increased access of 613 children aged 3-6 to early learning opportunities in target municipalities, especially of children living in rural areas; c) improved awareness of parents on the importance of ECE, translated into increased enrolment of children aged 3-6 in preschool (5 in 10 parents started the procedure of enrolling their children in kindergarten following the campaign in 2016); and d) strengthened capacity of professionals to run new ECE services and apply child-centred methodologies in their work with children. With few exceptions, the overall effectiveness level of the Programme was found to be adequate and close to what was planned by its designers.

At a higher-level results in the ToC, an objective, quantitative assessment of the Programme contribution to learning and development outcomes for children could not be done due to the lack of key prerequisites and resources explained in the section on limitations of the evaluation. Nevertheless, parents, professionals and the evaluation team observed improvements in the language, socio-affective, motor and, to a limited extent, cognitive development of children enrolled in outreach interactive services, which are likely to have been contributed to enhanced learning and development outcomes of the beneficiary children. The qualitative assessment of impact indicates a positive contribution of the new ECE services to the progressive realisation of children’s rights.

The Programme has been implemented in a professional and culturally-sensitive manner. With few exceptions, the funds were allocated and spent in an accountable and cost-effective way.


61

Efficiency of monitoring has been impeded by the lack of a clear hierarchy of results and targets in the Programme design documents and insufficient attention given to quality and risks monitoring during the implementation of outreach interactive services.

As far as sustainability is concerned, one year after their start up in the first six municipalities from North region, the outreach interactive services have been still functioning in 11 out of 19 locations (58%), while four were planned to restart soon. The implementation of the outreach interactive services was instrumental in highlighting what could be done and how, by investing an average amount of US$ 110/child for endowment and capacity building of professionals. Expansion of the services to other municipalities would require a smaller investment since there will be no need to redevelop guides, training curricula, working methodologies and tools or technical specifications for endowment, as these are already available. The current policy and legal framework governing the preschool education system in Montenegro is supportive for the development and expansion of ECE services for children aged 3-6 which were piloted and put in place by the Programme, but their financial sustainability is under risk in some locations.

The testing of new ECE services in the first six municipalities drew the attention on some challenges which might be encountered during implementation and which should be addressed with priority before rolling them out to other municipalities and later on nationally, related to equity and quality.

Equity: The new ECE services and awareness campaigns have increased the number of children receiving early education, including poor children living in rural areas who were the focus of the Programme. At the same time, it had a rather limited relevance for other vulnerable children, in particular children with disabilities and RE children, especially due to the limited duration and resources of the Programme which impeded a much broader scope. This does not mean that these children were forgotten or neglected; it rather indicates the limited scope of the Programme caused by its short duration and limited resources. Quality: The learning infrastructure in some locations were interactive services are functioning is not adequate, impacting on the quality of the educational process. Due to the education system weaknesses, the design of the services has not envisaged a proper assessment of the learning and development outcomes reached by the beneficiary children in order to be able to analyse the results and take corrective measures, as needed. The absence of children’s voice and views on their learning and the lack of parents’ involvement in monitoring the quality of interactive services on the background of insufficient attention given to the development of positive parenting practices were missed opportunities for increasing the quality of service provision and learning outcomes for children. Another element which affected quality was staff turnover. There was no system in place for induction training of new comers or any other systematic form of in-service training, apart from some sporadic mentoring by the principal or pedagogue. These issues might have been addressed more efficiently in case the design of the Programme included a coherent results framework from start, with clear quality safeguards to be monitored during the implementation of services. A more efficient process monitoring would have been also very useful to address some of the quality issues arisen during implementation. The need for a quality-focussed ECE expansion process has been strongly advocated by the participants in the validation workshop, who also stressed the key role the BfES should play in quality assurance.

The Programme represents a significant step forward in the process of establishing the underlying policy framework and practices for broadening the access of children aged 3-6 to ECE. The evidence provided by the Programme is a powerful tool in the process of advocating for universal coverage of children in ECE and corresponding public finance. Equity and quality should remain at the heart of the rolling out process at national level and measures need to be taken to ensure that the underlying causes of exclusion are addressed and that the service provision strictly comply with quality standards.


62

6.2 Recommendations

The recommendations presented in Table 8 below are based on the findings and conclusions of the evaluation as well as on active consultation with all key stakeholders that were interviewed during the in-country mission. Each interview and focus group has checked the perceptions of various stakeholders (UNICEF, ministries and governmental agencies, Parliament, PSI principals, preschool professionals, parents, development partners, CSOs, etc.) concerning the top priorities of the ECE reforms in Montenegro that need to be addressed in the coming years and consequently the role each of these stakeholders should play (see Guides in Annex 4). Validation of recommendations was done in two phases: 1) by UNICEF CO, following submission of the draft report; and 2) by national stakeholders, through written feedback and during the validation workshop where evaluation findings, conclusions and preliminary recommendations were presented and discussed. The feedback from key stakeholders was incorporated in the Final Evaluation Report. The recommendations are addressed to the Government of Montenegro, UNICEF and stakeholders at national and local levels, in line with the ToR and the need to engage all major stakeholders in a concerted effort for the continuation of reforms on the advancement of child rights in Montenegro. Recommendations are divided in two categories, as follows:

Recommendations are prioritised, have a timeframe and indicate an addressee. Each recommendation includes references to findings of the evaluation to substantiate the proposed course of action.

O - Operational recommendations

SP - Strategic and Programmatic recommendations


63

Table 8. List of Recommendations

No Recommendations Priority Timeframe Addressee

Strategic and Programmatic Recommendations (SP)

SP1 Ensure sustainable equity prerequisites for expanding the ECE services for children aged 3-6 at national level and breaking intergenerational cycles of exclusion

Description: Equity should be at the core of scaling up strategy of alternative ECE programmes, acknowledging that priority must be given to the most vulnerable children even in circumstances where supply limits access to relatively low percentages of the population. The parents’ demand for ECE services (also acknowledged during the validation workshop) and the overall society support for ECE which were successfully created by the Programme could lead to a rapid expansion of services; nevertheless, if ECE services expand rapidly without a sufficiently pronounced equity focus (as happened in the case of outreach services, see Relevance sub-chapter), equity gaps might grow and become increasingly institutionalised.

The MoE should continue to focus its interventions on poor children (as did with the support of the Programme), but also prioritise children with disabilities/SEN and RE children given their very poor participation in ECE as highlighted in a large number of studies and analytical work, the recommendations of the treaty bodies (CRC103 and CEDAW104 committees) and EU105 as well as the objectives of several national strategies106, on the top of scientific evidence that the most vulnerable benefit the most from early learning.

At the same time, the MoE and its partners should put in place coordination and outreach policies and systems involving various professional practices to identify, assess, refer, support and monitor children with various needs; hence an increased role for a cross-sectoral approach to early learning within the broader ECD framework (validated by the national stakeholders who participated in the workshop and who sent written comments), whereby education, health and social protection systems articulate their sphere of influence and responsibilities in order to produce equity gains and ensure that no child is left behind.

UNICEF can play a significant role in this process, by bringing together these various systems to identify and address the needs and maximise the benefits of ECE services in the realisation of children’s rights

High Short to Medium-Term

(2017-2019)

MoE with the support of UNICEF CO and in partnership with PSIs, Centres for Social Work and Health care providers

103 Recommendations 14, 48g, 48h and 60d, Committee on the Rights of the Child (2010), “Concluding observations: Montenegro”, Fifty-fifth session, CRC/C/MNE/CO/1 104 Recommendations 27a and 27d, Committee on the Elimination of Discrimination against Women (2011), “Concluding observations of the Committee on the Elimination of Discrimination against Women. Montenegro”, Fiftieth session, Geneva 105 “Increasing preschool participation, especially from Roma and other vulnerable groups, would significantly improve their success and continued participation in later schooling”, in European Commission (2016), “Montenegro 2016 Report”, Brussels, SWD(2016) 360 final, page 75 106Strategy for Inclusive Education 2014-2018, Strategy for Improving the Position of Roma and Egyptians in Montenegro 2012-2016 and the recent Strategy for Early Childhood and Preschool Education 2016–2020 which calls for intensive integration of the RE population in ECE as well as better detection and responses to the needs of children with SEN.


64


and breaking intergenerational cycles of exclusion. It can also play a crucial role in embedding diversity and inclusion within preschool environment and practice, starting with influencing an inclusive enrolment policy of PSIs and providing support for teaching and learning materials that reflect ethnic and linguistic diversity. Development of early detection and intervention services for identification of small children with disabilities/SEN and provision of targeted support to attend ECE are also key equity prerequisites.

SP2 Ensure that quality safeguards are embedded in the expansion of ECE service provision

This recommendation has been fully validated by the stakeholders during the validation workshop, indicating that focus on quality is undisputable.

Description: Specific quality assurance procedures should be put in place by the BfES from the very beginning to ensure adequate preschool infrastructure, mitigate staff turnover and, in general, enable the achievement of optimal learning and development outcomes for children (thus avoiding several quality shortcomings described in the Effectiveness sub-chapter). In case of funding shortage, resources could be leveraged from local communities (local authorities, parents, private sector), as done in other countries of the region.

Where the existing quality standards do not match the modalities of provision of ECE services tested by the Programme (2-3 hours/week, 3 hours/day), the BfES are recommended to prepare an additional guideline to operationalise the standards within provision that has a shorter timing. Such quality standards and guidelines would enable a more efficient monitoring of the quality of services which could initially be done jointly by UNICEF, MoE and BfES, while at the local level by the PSIs, teachers themselves and, very importantly, by parents (see Recommendation S3 below)

An essential quality dimension which should be given utmost consideration in preparing the scaling up of the new ECE services refers to the assessment of learning and development outcomes for enrolled children. UNICEF can play a key role in developing jointly with the MoE and BfES a unitary assessment framework and standardised instruments to measure progress and allow timely remedies.

Capacity building is key to ensuring a successful replication of the new ECE services nationally. In this respect, the guides, training curricula and methodologies tested during the implementation of outreach interactive services and currently piloted for the alternative 3-hour programme could be put of good use to ensure the development of a ‘critical mass’ of trained teachers throughout the country. The MoE and PSIs should ensure a mandatory induction training for any new teacher involved in delivering the new ECE services, followed by efficient supervision/mentoring (as indicated by the analysis in the Sustainability sub-chapter). Trends in the integration of childcare, health and education are increasing the professionalisation of staff working in early childhood services. UNICEF can play a key role in promoting comprehensive professional development for staff working with children of preschool age that in addition to ECE topics

High Short to Medium-Term

(2017-2019)

BfES, PSIs and local communities

BfES in partnership with the MoE and UNICEF CO

MoE, BfES in partnership with UNICEF CO

MoE and PSIs in partnership with UNICEF CO, pre-service and in-service training providers


65


also includes training in child development assessment and basic knowledge on health, nutrition, child protection, vulnerability, diversity and parenting.

SP3 Encourage participation of children and empower parents to become active and efficient partners of the education system in the process of early childhood education and development

Description: As mentioned in Relevance sub-chapter, there is no evidence of promoting the voice and views of children in the design, delivery and monitoring of the ECE services implemented by the Programme. It is recommended that UNICEF CO develops guidance and tools for child participation from age 4 to 6 to ensure children’s voice is heard and integrated into the general monitoring and assessment of ECE quality.

Systematic and culturally-sensitive awareness raising of parents on the benefits of early learning (especially of vulnerable parents), but equally on their role in the education of children, followed by empowering actions is of utmost importance. The Programme has insufficiently encouraged the participation of parents in the educational process of their children, while its contribution towards building positive parenting skills was insignificant, as evidentiated by this evaluation (sub-chapters on Relevance and Effectiveness). Raising awareness of parents and development of parenting skills should mandatorily complement the expansion and diversification of ECE. At the same time, empowerment activities are needed to develop parents’ capacity to play a meaningful role in the design and monitoring of the quality of early education programmes for their children.

UNICEF is best placed to advocate for policies of parental education programmes focused on the most at-risk children (poor children, RE children, children with disabilities) and contribute, in partnership with the MoE, to the development of more systematic involvement of parents as early educators and supporters of young children.

Medium Short to Medium-Term

(2017-2019)

UNICEF in partnership with the MoE, PSIs and CSOs

Operational Recommendations (O)

O1 Carry out a quick quality and financial sustainability assessment of outreach interactive services in the target municipalities supported by UNICEF (first wave)

Description: As mentioned in the Findings chapter of the report, the evaluation identified several quality and financial sustainability issues related to the outreach interactive services i.e. learning environment; staffing; financial risks.

High Short-Term

(2017)

UNICEF CO, MoE and BfES


66


UNICEF CO is recommended to carry out a quick assessment of the situation in cooperation with the MoE and BfES, and take remedial action in line with the revised ToC which provides for inclusive and sustainable coverage of preschool children in quality ECE.

For the assessment of quality, existing quality standards applicable to kindergartens should be adapted to a shorter duration programme and be used in checking the quality issues signalled by this evaluation.

Assessment of financial risks would need to analyse existing staffing levels and gaps, based on enrolments in the new school year 2016/2017.

Lack of adequate premises should not be a reason for closing down the interactive services or an obstacle for reaching out to the most vulnerable children. Mobile interactive services, for instance, have been suggested by some key informants as a quick and cost-effective solution until rehabilitation works are carried out or new kindergartens are built (which are time-consuming and rather costly).

O2 Develop the Monitoring and Evaluation Framework of the new ECE strategy 2016-2022

Description: As mentioned in the Effectiveness sub-chapter, the Strategy lacks a M&E Framework, which makes it hardly evaluable and accountable. It also impedes its monitoring to ensure that it is implemented in line with principles of quality and equity. UNICEF CO is recommended to provide assistance to the MoE to develop this framework which would comprise the following main components:

Narrative component, describing how M&E will be undertaken, how data quality control will be done, the feedback process and how participation will be ensured;

Results-Based Monitoring Plan (Matrix), where each result (output, outcome, impact) is assigned specific indicators with baselines and targets, monitoring events and data collection methods, schedule/frequency of data collection, responsible staff/institution for monitoring and reporting, means of verification, monitoring budget and, finally, planned use of monitoring results;

Risks Monitoring Plan (Matrix), identifying the risks, their probability of occurrence and potential impact (low, moderate, high), prevention/mitigation strategy, responsibilities and monitoring actions;

Evaluation Plan for summative and formative evaluations, or any other type of evaluation considered necessary, including the body which commission the evaluation, estimated start and completion dates, estimated cost and sources of funding;

Capacity building provisions for M&E, as the case, aimed to enhance the knowledge and skills of staff in charge of monitoring and evaluation.

Medium Short-Term

(2017)

UNICEF CO and MoE


67


The M&E Framework may be complemented by an Indicators Tracker, aimed at facilitating the monitoring of progress in the achievement of targeted levels in time. Developing a M&E framework is not a one-time event. Each time a significant change to the planned results or to the political, legal and institutional context occurs and might affect the results, the M&E framework should be revised. Annual work planning is the most pertinent point for updating the framework.

O3 Improve the results architecture and evaluability of future UNICEF initiatives

Description: The ToC and the Results Framework were developed retrospectively for the purpose of this evaluation. It is much more useful to construct them at the beginning of an intervention, with the participation of key stakeholders, as outcomes and processes are viewed differently with hindsight. Future interventions should be designed from start in a way which allows their evaluability and performance accountability of both UNICEF and Montenegrin partners, with clear baseline indicators and targets, disaggregated by gender, residence, age, disability, ethnicity, etc. It is recommended that a coherent risk mitigation strategy is designed during the programming phase and regularly reviewed and updated, to inform UNICEF and its partners about risks and constraints of the operating space which could affect the estimated results and guide the prevention and remedial actions, as needed.

A very important dimension of the programming process is the integration of equity, quality and inter-sectorality into the monitoring frameworks of UNICEF CO future initiatives. Monitoring should be both results and process-based. Any monitoring meeting of the CO with Government counterparts and each field visit to communities targeted by a specific intervention should be documented in a minutes (highlighting the decisions taken), respectively in a trip report (including the monitoring scope, key findings and follow-up).

Medium Whenever programming new initiatives

UNICEF CO


68

6.3 Lessons Learnt

Several key lessons emerge from the issues highlighted in this evaluation:

1) Development of an enabling framework for universal coverage of children in ECE requires a systematic, comprehensive and concerted approach of all determinants of system change and of all key stakeholders at various levels of intervention. Such approach would imply the following mandatory ‘ingredients’: a) development of solid, credible evidence on needs; b) policy advocacy at national level (Government, Parliament, CSOs, development partners); c) development and adoption of the strategic framework for ECE expansion and funding scenarios; d) testing, costing and documenting various service provision options at community level; e) capacity building of service providers; f) awareness raising campaign for behaviour change of parents at regional and local levels. Such comprehensive and systematic effort is key to ensuring an informed policy debate and building support in the society for investment in the ECE.

2) Expansion of ECE for children aged 3-6 requires both hard and soft investment. Investing in infrastructure rehabilitation and in the procurement of furniture and didactical materials in parallel with the delivery of early education programmes and capacity building of both professionals and parents is a good mix and a feasible solution for countries where the existing school infrastructure is decrepit and the school network is sub-optimally located across the territory. Any attempt to expand the preschool education for this age group should consider the investment in physical infrastructure, too.

3) Any action targeting ECE must engage the parents of beneficiary children from start

throughout the whole process. Parents are both duty-bearers and rights-holders. They have a fundamental role in the education and upbringing of their children, being key partners of the education and social welfare systems. The participation of parents in the life of kindergarten and monitoring of activities increases their relevance and contributes to quality assurance, while the empowerment of parents with positive parental skills yields sustainable effects on children’s educational attainment and development in the coming years. Working with parents needs to be culturally-sensitive, especially in the case of parents belonging to ethnic minority groups, but also in the case of parents living in rural areas, given the traditional family roles and more influential community norms and values than for urban parents. The role of grandparents should not be underestimated, as they can substantially facilitate or impede the enrolment of their grandchildren in preschool education.

Documents

HRD sector evaluation - UNICEF · 2019-06-07 · The Programme was implemented by UNICEF CO in partnership with the MoE and local implementing partners. The Programme had an initial