Evaluation Report

Evaluation of the project “Using textbooks in DAISY format - Primary school teacher training in using audio textbooks”

Montenegro

Project “Using textbooks in DAISY format” – Draft Evaluation Report 2.

Name of the project: “Using textbooks in DAISY format – Primary school teacher training in using audio textbooks” Country of the project: Montenegro Timeframe of the evaluation and date of the report: April–August 2015 Locations (country, region, etc.) of the evaluation object: Montenegro, 24 primary schools and the resource centre (south, central and north region) Evaluator: Salvador Bustamante Aragonés Name of the organization commissioning the evaluation: UNICEF Names of UNICEF staff contact point for the evaluation: Marija Manojlovic / Maja Kovacevic The evaluator acknowledges the support provided by UNICEF Montenegro staff and all relevant stakeholders involved in the evaluation, for their participation, logistics arrangements and insights.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 3.

TABLE OF CONTENTS

COUNTRY OF THE PROJECT: MONTENEGRO ...................................................................... 2

TIMEFRAME OF THE EVALUATION AND DATE OF THE REPORT: APRIL–AUGUST 2015 . 2

LOCATIONS (COUNTRY, REGION, ETC.) OF THE EVALUATION OBJECT: MONTENEGRO, 24 PRIMARY SCHOOLS AND THE RESOURCE CENTRE (SOUTH, CENTRAL AND NORTH REGION) .................................................................................................................................... 2

EVALUATOR: SALVADOR BUSTAMANTE ARAGONÉS ........................................................ 2

NAME OF THE ORGANIZATION COMMISSIONING THE EVALUATION: UNICEF ................. 2

NAMES OF UNICEF STAFF CONTACT POINT FOR THE EVALUATION: MARIJA MANOJLOVIC / MAJA KOVACEVIC ........................................................................................ 2

TABLE OF CONTENTS ............................................................................................................. 3

LIST OF FIGURES ................................................................................................................... 26

LIST OF TABLES .................................................................................................................... 26

LIST OF ACRONYMS .............................................................................................................. 27

1. EXECUTIVE SUMMARY................................................................................................. 28

OVERVIEW OF THE OBJECT OF EVALUATION ................................................................... 28

EVALUATION OBJECTIVES AND INTENDED AUDIENCES ................................................. 28

EVALUATION METHODOLOGY ............................................................................................. 29

MOST IMPORTANT FINDINGS AND CONCLUSIONS ........................................................... 30

KEY RESULTS ACHIEVED ..................................................................................................... 30

PROJECT DESIGN .................................................................................................................. 31

DAISY PRODUCTION AND PROVISION OF CAPACITY-BUILDING ACTIVITIES ................. 31

STAKEHOLDERS AND BENEFICIARIES’ ENGAGEMENT .................................................... 31

CONTRIBUTION TO FUTURE IMPACT EVALUATION .......................................................... 32

LONG-TERM SUSTAINABILITY ............................................................................................. 32

MAIN RECOMMENDATIONS .................................................................................................. 32

RECOMMENDATION 1 ........................................................................................................... 32

RECOMMENDATION 2 ........................................................................................................... 33

RECOMMENDATION 3 ........................................................................................................... 33

RECOMMENDATION 4 ........................................................................................................... 34

RECOMMENDATION 5 ........................................................................................................... 34

LESSONS LEARNED .............................................................................................................. 34

2. OBJECT OF THE EVALUATION .................................................................................... 36

INTRODUCTION ............................................................................................................. 36

OBJECT OF THE EVALUATION ..................................................................................... 36

INTERNATIONAL CONTEXT .......................................................................................... 41

NATIONAL CONTEXT .................................................................................................... 42

UNICEF CONTEXT ......................................................................................................... 43

Project “Using textbooks in DAISY format” – Draft Evaluation Report 4.

SITUATION ANALYSIS ................................................................................................... 44

3. EVALUATION PURPOSE, OBJECTIVES AND SCOPE ................................................ 46

THE EVALUATION PURPOSE AND OBJECTIVES ........................................................ 47

SCOPE OF THE EVALUATION ...................................................................................... 47

4. EVALUATION METHODOLOGY .................................................................................... 48

5. FINDINGS AND ASSESSMENT ..................................................................................... 53

RELEVANCE .................................................................................................................. 53

(i) Is the project in line with national strategies and policies on education? ..................... 53 (ii) Does the project address the underlying causes of exclusion of CwSEN in primary schools and respond to their needs? ................................................................................. 54

FIGURE 2: SURVEY RESULTS ON DAISY RESPONDING TO CWSEN NEEDS ................... 55

(iii) Is the design of the project adequate for the achievement of its objectives in the given period of time? .................................................................................................................. 55 (iv) Was the project designed according to international norms on human rights and gender equality and in line with national strategies in those areas? ............................................... 56 (v) Are training sessions, materials and the DAISY audio-visual textbooks produced suitable for the intended audiences and did they contribute to goal achievement? ............ 57

FIGURE 4: OVERALL LEVEL OF SATISFACTION (TOTALS) BY STAKEHOLDERS ........... 59

EFFICIENCY ................................................................................................................... 60

(i) How well was the implementation of activities managed? ........................................... 60 (ii) How well were the financial resources used? .............................................................. 60 (iii) Was the intervention adequately coordinated? ........................................................... 61

FIGURE 5: OVERALL LEVEL OF SATISFACTION WITH IMPLEMENTATION QUALITY ..... 61

EFFECTIVENESS ........................................................................................................... 62

(i) Are teachers and parents aware of the benefits of using DAISY textbooks and motivated to work with them? ............................................................................................ 62 (ii) Project contribution to developing the skills of teachers and parents in the use of DAISY ............................................................................................................................... 63 (iii) Accessibility to the produced learning materials to students ....................................... 64 (iv) Project contribution to developing student’s skills on the use of DAISY ...................... 65 (v) Project contribution towards better organizing the teaching process for joint learning . 66 (vi) Unintended effects (positive and negative) ................................................................. 66

IMPACT .......................................................................................................................... 67

(i) Project contribution towards improving learning group dynamics ................................ 67

FIGURE 6: LEVEL OF PARTICIPATION IN CLASS WHEN USING DAISY TEXTBOOKS ..... 67

(ii) The project’s contribution to changing students’ attitudes toward school and the learning process. ............................................................................................................... 67 (iii) Project contribution to improving the confidence of CwSEN ........................................ 68 (iv) Project contribution to better integration of CwSEN in the teaching and learning process ............................................................................................................................. 68 (v) The project’s contribution to developing an improved teaching process ...................... 68

SUSTAINABILITY........................................................................................................... 70

(i) Establishment of a clear and appropriate sustainability and exit strategy .................... 70 (ii) Sustainability of project’s main activities ..................................................................... 70 (iii) The extent to which knowledge and skills will be integrated in schools ....................... 70

Project “Using textbooks in DAISY format” – Draft Evaluation Report 5.

CROSS-CUTTING ISSUES ............................................................................................ 72

(i) Equality focus and gender mainstreaming in the project ............................................. 72

6. CONCLUSIONS .............................................................................................................. 73

7. LESSONS LEARNED ..................................................................................................... 75

8. RECOMMENDATIONS ................................................................................................... 76

9. ANNEXES ....................................................................................................................... 78

ANNEX 1 – TERMS OF REFERENCE ..................................................................................... 79

ANNEX 3 – LIST OF DOCUMENTS REVIEWED ..................................................................... 91

ANNEX 5 – LIST OF PARTICIPATING SCHOOLS AND DISAGGREGATED NUMBER OF STUDENTS ............................................................................................................................ 102

ANNEX 6 – QUESTIONNAIRES ............................................................................................ 104

1. UNICEF’s support through the DAISY project is relevant for the achievement of Montenegro’s overall inclusive education goals. ....................................................... 104 (ii) In case strongly agree / agree: Could you please name the national education goals to which DAISY project contributed? ................................................................................... 104 2. UNICEF’s support through the DAISY project responds to the needs of CwD in primary schools............................................................................................................. 104 3. The design of the DAISY project… ........................................................................ 104 (iii) In case strongly agree / agree: Could you please name the national strategies on human rights and gender equality to which the DAISY project contributed?........... 104 4. Could you name and categorize the DAISY project activities as follows? Please provide an answer for each item and briefly include the main reason for this categorization ................................................................................................................ 105 5. Could you please rate the DAISY project capacity-building activities (training, workshops, sessions) according to: ............................................................................ 105 6. Could you please rate the DAISY project training materials and guidelines according to: ................................................................................................................. 105 7. Could you please rate the produced DAISY textbooks according to: ................ 105 8. Other relevant activities, please specify ___________________________________: .................................................................. 106 9. How would you rate the level of coordination with other implementing partners? 106 10. Did the intervention include an appropriate sustainability and exit strategy to ensure the benefits generated by the project continue after donor funding has been withdrawn? .................................................................................................................... 106 YES ................................................................................................................................. 106 106 NO 106 106 N/A 106 106 11. Could you please rate the overall level of satisfaction with the quality of Project DAISY ............................................................................................................................. 106 Could you please elaborate on your answer/ indicate reasons? ............................... 106

UNICEF “USING TEXTBOOKS IN DAISY FORMAT” – SURVEY FOR DIRECT BENEFICIARIES (FAMILIES WITH CWD / WITH CHILDREN WITHOUT DISABILITIES) .... 107

DATE: .................................................................................................................................... 107

TYPE OF RESPONDENT: ..................................................................................................... 107

FAMILY WITH CWD .............................................................................................................. 107

Project “Using textbooks in DAISY format” – Draft Evaluation Report 6.

............................................................................................................................................ 107

FAMILY WITH CHILDREN WITHOUT DISABILITIES ........................................................... 107

............................................................................................................................................ 107

NUMBER OF CWD AT SCHOOL: ......................................................................................... 107

ASSOCIATION/SCHOOL: ..................................................................................................... 107

1. DID YOU ATTEND THE TRAINING PROVIDED BY THE DAISY PROJECT ON THE USE OF DAISY AUDIO-VISUAL TEXTBOOKS? .................................................................. 107

NO ......................................................................................................................................... 107

PARTIALLY ........................................................................................................................... 107

COMPLETELY ....................................................................................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

2. COULD YOU PLEASE RATE THE DAISY PROJECT CAPACITY-BUILDING ACTIVITIES (TRAINING, WORKSHOPS, SESSIONS) ACCORDING TO: ............................ 107

VERY HIGH ........................................................................................................................... 107

HIGH ...................................................................................................................................... 107

MODERATE ........................................................................................................................... 107

LOW ....................................................................................................................................... 107

VERY LOW ............................................................................................................................ 107

N/A ......................................................................................................................................... 107

QUALITY OF CONTENT CONCERNING DAISY BOOKS ..................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

QUALITY OF CONTENT CONCERNING GENERAL INCLUSION STRATEGIES ................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

USEFULNESS ....................................................................................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

Project “Using textbooks in DAISY format” – Draft Evaluation Report 7.

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

TIMELINESS .......................................................................................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

YOUR OVERALL LEVEL OF SATISFACTION...................................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

3. COULD YOU PLEASE RATE THE DAISY PROJECT TRAINING MATERIALS AND GUIDELINES ACCORDING TO: ........................................................................................... 107

VERY HIGH ........................................................................................................................... 107

HIGH ...................................................................................................................................... 107

MODERATE ........................................................................................................................... 107

LOW ....................................................................................................................................... 107

VERY LOW ............................................................................................................................ 107

N/A ......................................................................................................................................... 107

QUALITY OF CONTENT ....................................................................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

USEFULNESS ....................................................................................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

Project “Using textbooks in DAISY format” – Draft Evaluation Report 8.

............................................................................................................................................ 107

............................................................................................................................................ 107

YOUR OVERALL LEVEL OF SATISFACTION...................................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

4. COULD YOU PLEASE RATE THE PRODUCED DAISY AUDIO-VISUAL TEXTBOOKS ACCORDING TO: .................................................................................................................. 107

VERY HIGH ........................................................................................................................... 107

HIGH ...................................................................................................................................... 107

MODERATE ........................................................................................................................... 107

LOW ....................................................................................................................................... 107

VERY LOW ............................................................................................................................ 107

N/A ......................................................................................................................................... 107

QUALITY OF NARRATION ................................................................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

USEFULNESS (LEARNING) ................................................................................................. 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

TIMELY INTRODUCTION IN SCHOOLS. .............................................................................. 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

Project “Using textbooks in DAISY format” – Draft Evaluation Report 9.

AVAILABILITY OF TECHNICAL SUPPORT ......................................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

YOUR OVERALL LEVEL OF SATISFACTION...................................................................... 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

............................................................................................................................................ 107

5. HOW WOULD YOU RATE YOUR SKILLS IN THE USE OF DAISY AUDIO-VISUAL TEXTBOOKS? ....................................................................................................................... 108

VERY HIGH ........................................................................................................................... 108

HIGH ...................................................................................................................................... 108

MODERATE ........................................................................................................................... 108

LOW ....................................................................................................................................... 108

VERY LOW ............................................................................................................................ 108

N/A ......................................................................................................................................... 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

6. HOW WOULD YOU RATE YOUR MOTIVATION TO USE DAISY AUDIO-VISUAL TEXTBOOKS? ....................................................................................................................... 108

VERY HIGH ........................................................................................................................... 108

HIGH ...................................................................................................................................... 108

MODERATE ........................................................................................................................... 108

LOW ....................................................................................................................................... 108

VERY LOW ............................................................................................................................ 108

N/A ......................................................................................................................................... 108

............................................................................................................................................ 108

............................................................................................................................................ 108

Project “Using textbooks in DAISY format” – Draft Evaluation Report 10.

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

7. DO YOU HAVE ACCESS TO DAISY AUDIO-VISUAL TEXTBOOKS AT HOME? (IF ANSWER IS “NO” OR “N/A”, PLEASE GO TO Q14) ........................................................... 108

YES ........................................................................................................................................ 108

............................................................................................................................................ 108

NO ......................................................................................................................................... 108

............................................................................................................................................ 108

N/A ......................................................................................................................................... 108

............................................................................................................................................ 108

8. HOW OFTEN DOES YOUR CHILD USE DAISY AUDIO-VISUAL TEXTBOOKS FOR HIS/HER INDIVIDUAL LEARNING? ...................................................................................... 108

DAILY .................................................................................................................................... 108

WEEKLY ................................................................................................................................ 108

MONTHLY ............................................................................................................................. 108

NEVER ................................................................................................................................... 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

9. HOW OFTEN DO YOU PROVIDE ASSISTANCE TO YOUR CHILDREN IN THE USE OF DAISY AUDIO-VISUAL TEXTBOOKS?................................................................................. 108

DAILY .................................................................................................................................... 108

WEEKLY ................................................................................................................................ 108

MONTHLY ............................................................................................................................. 108

NEVER ................................................................................................................................... 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

10. HOW OFTEN ARE YOU ABLE TO SOLVE YOUR CHILD’S DOUBTS ABOUT THE USE OF DAISY AUDIO-VISUAL TEXTBOOKS? ........................................................................... 108

ALWAYS / ALMOST ALWAYS ............................................................................................. 108

USUALLY / MOST OF THE TIME .......................................................................................... 108

OFTEN ................................................................................................................................... 108

SELDOM / RARELY .............................................................................................................. 108

NEVER / ALMOST NEVER .................................................................................................... 108

Project “Using textbooks in DAISY format” – Draft Evaluation Report 11.

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

11. COULD YOU PLEASE RATE THE OVERALL LEVEL OF SATISFACTION WITH THE QUALITY OF PROJECT DAISY ............................................................................................ 108

VERY HIGH ........................................................................................................................... 108

HIGH ...................................................................................................................................... 108

MODERATE ........................................................................................................................... 108

LOW ....................................................................................................................................... 108

VERY LOW ............................................................................................................................ 108

N/A ......................................................................................................................................... 108

QUALITY OF IMPLEMENTATION ......................................................................................... 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

USEFULNESS ....................................................................................................................... 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

TIMELINESS .......................................................................................................................... 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

EQUAL ACCESS TO BOYS AND GIRLS (EQUITY FOCUS) ................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

Project “Using textbooks in DAISY format” – Draft Evaluation Report 12.

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

YOUR OVERALL LEVEL OF SATISFACTION...................................................................... 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

............................................................................................................................................ 108

COULD YOU PLEASE ELABORATE ON YOUR ANSWER/ INDICATE REASONS? .......... 108

UNICEF “USING TEXTBOOKS IN DAISY FORMAT” – SURVEY FOR DIRECT BENEFICIARIES (TEACHERS AND SCHOOL MANAGERS) ............................................... 109

DATE: .................................................................................................................................... 109

TYPE OF RESPONDENT: ..................................................................................................... 109

TEACHER .............................................................................................................................. 109

............................................................................................................................................ 109

SUPERVISOR ........................................................................................................................ 109

............................................................................................................................................ 109

SCHOOL MANAGEMENT ..................................................................................................... 109

............................................................................................................................................ 109

POSITION: ............................................................................................................................. 109

ASSOCIATION/SCHOOL: ..................................................................................................... 109

2. UNICEF’S SUPPORT THROUGH INTRODUCTION OF DAISY AUDIO-VISUAL TEXTBOOKS PROJECT IS RELEVANT FOR THE ACHIEVEMENT OF MONTENEGRO’S OVERALL INCLUSIVE EDUCATION GOALS. ...................................................................... 109

STRONGLY AGREE .............................................................................................................. 109

AGREE .................................................................................................................................. 109

NEITHER AGREE NOR DISAGREE ...................................................................................... 109

DISAGREE ............................................................................................................................ 109

STRONGLY DISAGREE ........................................................................................................ 109

N/A ......................................................................................................................................... 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

Project “Using textbooks in DAISY format” – Draft Evaluation Report 13.

3. IN CASE STRONGLY AGREE / AGREE: COULD YOU PLEASE NAME THE NATIONAL EDUCATION GOALS TO WHICH THE DAISY PROJECT CONTRIBUTED? .... 109

4. UNICEF’S SUPPORT THROUGH THE INTRODUCTION OF DAISY AUDIO-VISUAL TEXTBOOKS PROJECT RESPONDS TO THE NEEDS OF CWD IN INCLUSIVE PRIMARY SCHOOLS ............................................................................................................................. 109

STRONGLY AGREE .............................................................................................................. 109

AGREE .................................................................................................................................. 109

NEITHER AGREE NOR DISAGREE ...................................................................................... 109

DISAGREE ............................................................................................................................ 109

STRONGLY DISAGREE ........................................................................................................ 109

N/A ......................................................................................................................................... 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

5. THE DESIGN OF THE DAISY PROJECT… ................................................................. 109

STRONGLY AGREE .............................................................................................................. 109

AGREE .................................................................................................................................. 109

NEITHER AGREE NOR DISAGREE ...................................................................................... 109

DISAGREE ............................................................................................................................ 109

STRONGLY DISAGREE ........................................................................................................ 109

N/A ......................................................................................................................................... 109

..IS IN LINE WITH NATIONAL STRATEGIES IN INCLUSIVE EDUCATION (2014–18) ........ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

…IS IN LINE WITH NATIONAL STRATEGIES ON GENDER EQUALITY ............................. 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

Project “Using textbooks in DAISY format” – Draft Evaluation Report 14.

…IS ADEQUATE TO ACHIEVE THE INTENDED GOALS AND OBJECTIVES SET FOR THE PROJECT. ............................................................................................................................. 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

............................................................................................................................................ 109

6. IN CASE STRONGLY AGREE / AGREE: COULD YOU PLEASE NAME THE NATIONAL STRATEGIES ON HUMAN RIGHTS AND GENDER EQUALITY TO WHICH THE DAISY PROJECT CONTRIBUTED? ...................................................................................... 110

7. COULD YOU NAME AND CATEGORIZE THE DAISY PROJECT ACTIVITIES AS FOLLOWS? PLEASE PROVIDE AN ANSWER FOR EACH ITEM AND BRIEFLY INCLUDE THE MAIN REASON FOR THIS CATEGORIZATION ........................................................... 110

CATEGORY ........................................................................................................................... 110

SERVICES ............................................................................................................................. 110

WHY? .................................................................................................................................... 110

ESSENTIAL AND EXCELLENT (ONLY THE PROJECT OFFER THESE ACTIVITIES) ........ 110

ESSENTIAL ACTIVITIES BUT SHOULD BE IMPROVED ..................................................... 110

USEFUL BUT THEY CAN BE OFFERED BY OTHER PARTNERS ...................................... 110

ACTIVITIES NO LONGER NEEDED ..................................................................................... 110

8. COULD YOU PLEASE RATE THE TRAINING ON THE USE OF DAISY AUDIO-VISUAL TEXTBOOKS (BOTH TECHNICAL AND PEDAGOGICAL ASPECTS – OR SEPARATELY): 110

VERY HIGH ........................................................................................................................... 110

HIGH ...................................................................................................................................... 110

MODERATE ........................................................................................................................... 110

LOW ....................................................................................................................................... 110

VERY LOW ............................................................................................................................ 110

N/A ......................................................................................................................................... 110

QUALITY OF CONTENT CONCERNING DAISY BOOKS ..................................................... 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

QUALITY OF CONTENT CONCERNING GENERAL INCLUSION STRATEGIES ................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

Project “Using textbooks in DAISY format” – Draft Evaluation Report 15.

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

USEFULNESS ....................................................................................................................... 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

TIMELINESS .......................................................................................................................... 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

YOUR OVERALL LEVEL OF SATISFACTION...................................................................... 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

9. COULD YOU PLEASE RATE THE DAISY PROJECT TRAINING MATERIALS AND GUIDELINES ACCORDING TO: ........................................................................................... 110

VERY HIGH ........................................................................................................................... 110

HIGH ...................................................................................................................................... 110

MODERATE ........................................................................................................................... 110

LOW ....................................................................................................................................... 110

VERY LOW ............................................................................................................................ 110

N/A ......................................................................................................................................... 110

QUALITY OF CONTENT ....................................................................................................... 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

Project “Using textbooks in DAISY format” – Draft Evaluation Report 16.

............................................................................................................................................ 110

............................................................................................................................................ 110

USEFULNESS ....................................................................................................................... 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

YOUR OVERALL LEVEL OF SATISFACTION...................................................................... 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

............................................................................................................................................ 110

10. COULD YOU PLEASE RATE THE PRODUCED DAISY AUDIO-VISUAL TEXTBOOKS ACCORDING TO: .................................................................................................................. 111

VERY HIGH ........................................................................................................................... 111

HIGH ...................................................................................................................................... 111

MODERATE ........................................................................................................................... 111

LOW ....................................................................................................................................... 111

VERY LOW ............................................................................................................................ 111

N/A ......................................................................................................................................... 111

QUALITY OF NARRATION ................................................................................................... 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

USEFULNESS (TEACHING) ................................................................................................. 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

Project “Using textbooks in DAISY format” – Draft Evaluation Report 17.

TIMELY INTRODUCTION IN SCHOOLS. .............................................................................. 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

AVAILABILITY OF TECHNICAL SUPPORT ......................................................................... 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

YOUR OVERALL LEVEL OF SATISFACTION...................................................................... 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

12. OTHER RELEVANT ACTIVITIES, PLEASE SPECIFY ___________________________________: ......................................................................... 111

VERY HIGH ........................................................................................................................... 111

HIGH ...................................................................................................................................... 111

MODERATE ........................................................................................................................... 111

LOW ....................................................................................................................................... 111

VERY LOW ............................................................................................................................ 111

N/A ......................................................................................................................................... 111

QUALITY OF CONTENT ....................................................................................................... 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

USEFULNESS ....................................................................................................................... 111

............................................................................................................................................ 111

Project “Using textbooks in DAISY format” – Draft Evaluation Report 18.

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

TIMELINESS .......................................................................................................................... 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

YOUR OVERALL LEVEL OF SATISFACTION...................................................................... 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

13. HOW WOULD YOU RATE YOUR SKILLS IN THE USE OF DAISY AUDIO-VISUAL TEXTBOOKS? ....................................................................................................................... 111

VERY HIGH ........................................................................................................................... 111

HIGH ...................................................................................................................................... 111

MODERATE ........................................................................................................................... 111

LOW ....................................................................................................................................... 111

VERY LOW ............................................................................................................................ 111

N/A ......................................................................................................................................... 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

14. HOW WOULD YOU RATE YOUR MOTIVATION TO USE DAISY AUDIO-VISUAL TEXTBOOKS? ....................................................................................................................... 111

VERY HIGH ........................................................................................................................... 111

HIGH ...................................................................................................................................... 111

MODERATE ........................................................................................................................... 111

Project “Using textbooks in DAISY format” – Draft Evaluation Report 19.

LOW ....................................................................................................................................... 111

VERY LOW ............................................................................................................................ 111

N/A ......................................................................................................................................... 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

15. HOW OFTEN HAVE YOU CONDUCTED INDIVIDUAL/GROUP LEARNING SESSIONS IN THE USE OF DAISY AUDIO-VISUAL TEXTBOOKS? ...................................................... 111

INDIVIDUAL SESSIONS ........................................................................................................ 111

GROUP SESSIONS ............................................................................................................... 111

DAILY .................................................................................................................................... 111

WEEKLY ................................................................................................................................ 111

MONTHLY ............................................................................................................................. 111

DAILY .................................................................................................................................... 111

WEEKLY ................................................................................................................................ 111

MONTHLY ............................................................................................................................. 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

TOTAL NUMBER OF INDIVIDUAL SESSIONS CONDUCTED______ ................................. 111

TOTAL NUMBER OF GROUP SESSIONS CONDUCTED__________ ................................. 111

16. HOW OFTEN HAVE YOU CONDUCTED INDIVIDUAL/GROUP LEARNING SESSIONS WITH DAISY AUDIO-VISUAL TEXTBOOKS? ...................................................................... 111

INDIVIDUAL SESSIONS ........................................................................................................ 111

GROUP SESSIONS ............................................................................................................... 111

DAILY .................................................................................................................................... 111

WEEKLY ................................................................................................................................ 111

MONTHLY ............................................................................................................................. 111

DAILY .................................................................................................................................... 111

WEEKLY ................................................................................................................................ 111

MONTHLY ............................................................................................................................. 111

............................................................................................................................................ 111

Project “Using textbooks in DAISY format” – Draft Evaluation Report 20.

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

............................................................................................................................................ 111

TOTAL NUMBER OF INDIVIDUAL SESSIONS CONDUCTED______ ................................. 111

TOTAL NUMBER OF GROUP SESSIONS CONDUCTED__________ ................................. 111

17. DO YOU THINK YOU WILL BE ABLE AND WILLING TO CONTINUE USING DAISY AUDIO-VISUAL TEXTBOOKS ONCE THE PROJECT FINISHES? ...................................... 112

YES ........................................................................................................................................ 112

............................................................................................................................................ 112

NO ......................................................................................................................................... 112

............................................................................................................................................ 112

N/A ......................................................................................................................................... 112

............................................................................................................................................ 112

IN CASE YOUR ANSWER TO Q10 IS “YES”, COULD YOU PLEASE PROVIDE SOME EXAMPLES? IN CASE YOUR ANSWER TO Q10 IS “NO”, COULD YOU PLEASE PROVIDE REASONS? ........................................................................................................................... 112

18. COULD YOU PLEASE RATE THE OVERALL LEVEL OF SATISFACTION WITH THE QUALITY OF PROJECT DAISY AUDIO-VISUAL TEXTBOOKS?......................................... 112

VERY HIGH ........................................................................................................................... 112

HIGH ...................................................................................................................................... 112

MODERATE ........................................................................................................................... 112

LOW ....................................................................................................................................... 112

VERY LOW ............................................................................................................................ 112

N/A ......................................................................................................................................... 112

QUALITY OF IMPLEMENTATION ......................................................................................... 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

USEFULNESS ....................................................................................................................... 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

Project “Using textbooks in DAISY format” – Draft Evaluation Report 21.

............................................................................................................................................ 112

TIMELINESS .......................................................................................................................... 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

EQUAL ACCESS TO BOYS AND GIRLS (EQUITY FOCUS) ................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

YOUR OVERALL LEVEL OF SATISFACTION...................................................................... 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

............................................................................................................................................ 112

COULD YOU PLEASE ELABORATE ON YOUR ANSWER/ INDICATE REASONS? .......... 112

UNICEF “USING TEXTBOOKS IN DAISY FORMAT” – SURVEY FOR STUDENTS ............ 113

DATE: .................................................................................................................................... 113

GRADE: ................................................................................................................................. 113

SCHOOL:............................................................................................................................... 113

TYPE OF RESPONDENT: ..................................................................................................... 113

CWD ...................................................................................................................................... 113

............................................................................................................................................ 113

CHILDREN WITHOUT DISABILITIES ................................................................................... 113

............................................................................................................................................ 113

GENDER:............................................................................................................................... 113

MALE ..................................................................................................................................... 113

............................................................................................................................................ 113

FEMALE ................................................................................................................................ 113

............................................................................................................................................ 113

Project “Using textbooks in DAISY format” – Draft Evaluation Report 22.

USE THE FOLLOWING FACES TO RESPOND AND DESCRIBE HOW YOU FEEL ABOUT EACH SENTENCE. PLEASE CIRCLE THE FACE TO ANSWER. ........................................... 113

STRONGLY AGREE // AGREE // NEITHER AGREE NOR DISAGREE //

DISAGREE // STRONGLY DISAGREE // N/A – DO NOT KNOW – NO OPINION ..... 113

STRONGLY AGREE .............................................................................................................. 113

AGREE .................................................................................................................................. 113

NEITHER AGREE .................................................................................................................. 113

NOR DISAGREE .................................................................................................................... 113

DISAGREE ............................................................................................................................ 113

STRONGLY ........................................................................................................................... 113

DISAGREE ............................................................................................................................ 113

N/A – DO NOT KNOW / ......................................................................................................... 113

NO OPINION .......................................................................................................................... 113

1. I LIKE BEING AT THE SCHOOL .................................................................................. 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

2. MY TEACHER KNOWS HOW TO USE DAISY AUDIO-VISUAL TEXTBOOKS ........... 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

3. MY TEACHERS HELP ME IN THE USE OF DAISY AUDIO-VISUAL TEXTBOOKS .... 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

Project “Using textbooks in DAISY format” – Draft Evaluation Report 23.

4. I LIKE TO PARTICIPATE IN CLASS ............................................................................ 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

5. I PARTICIPATE MORE IN CLASS WHEN NOT USING DAISY AUDIO-VISUAL TEXTBOOKS ......................................................................................................................... 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

6. OTHER CHILDREN HELP ME AT SCHOOL ................................................................ 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

7. THE OTHER STUDENTS ACCEPT ME AS I AM ......................................................... 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

8. I KNOW HOW TO USE AUDIO-VISUAL DAISY TEXTBOOKS .................................... 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

Project “Using textbooks in DAISY format” – Draft Evaluation Report 24.

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

9. I LIKE STUDYING WITH AUDIO-VISUAL DAISY TEXTBOOKS ................................. 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

10. I PREFER GROUP ACTIVITIES IN CLASS WHEN WE DO NOT USE DAISY AUDIO-VISUAL TEXTBOOKS ........................................................................................................... 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

11. I THINK DAISY AUDIO-VISUAL TEXTBOOKS HELP ME TO STUDY ........................ 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

12. MY PARENTS KNOW HOW TO USE DAISY AUDIO-VISUAL TEXTBOOKS .............. 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

Project “Using textbooks in DAISY format” – Draft Evaluation Report 25.

............................................................................................................................................ 113

13. MY PARENTS HELP ME TO USE DAISY AUDIO-VISUAL TEXTBOOKS AT HOME . 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

14. I STUDY MORE BY MYSELF SINCE I STARTED USING DAISY AUDIO-VISUAL TEXTBOOKS ......................................................................................................................... 113

...................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

.......................................................................................................................................... 113

...................................................................................................................................... 113

............................................................................................................................................ 113

Project “Using textbooks in DAISY format” – Draft Evaluation Report 26.

LIST OF FIGURES

Figure 1. Programme logic model ............................................................................................. 39

Figure 2. Survey results on DAISY responding to CwSEN needs ............................................. 55

Figure 3. Survey results for DAISY audio-visual textbook usefulness. ...................................... 59

Figure 4. Overall level of satisfaction (totals) by stakeholders .................................................. 59

Figure 5. Overall level of satisfaction with implementation quality ............................................. 61

Figure 6. Level of participation in class when using DAISY textbooks ...................................... 67

LIST OF TABLES

Table 1: Methods of data collection .......................................................................................... 49

Table 2: Sample stakeholder coverage .................................................................................... 51

Project “Using textbooks in DAISY format” – Draft Evaluation Report 27.

LIST OF ACRONYMS

BoE Bureau of Education

CwD Children with Disabilities

CwLD Children with Learning Difficulties

CwSEN Children with Special Education Needs

DAC Development Assistance Committee

DAISY Digital Accessible Information System

EFA Education For All

ESC Evaluation Steering Committee

EU European Union

FG Focus Group

GE Gender Equality

HR Human Rights

IT Information Technology

ITTA Institute for Textbooks and Teaching Aids

MDG Millennium Development Goals

MoE Ministry of Education

MoU Memorandum of Understanding

NGO Non Governmental Organisation

OECD The Organisation for Economic Co-operation and Development

PISA Programme for International Student Assessment

RC The “Podgorica” Resource Centre for Children and Youth

SEN Special Education Needs

TIGAR Trusted Intermediary Global Accessible Resources

ToR Terms of Reference

UNESCO United Nations Educational, Scientific and Cultural Organization

UNICEF United Nations International Children’s Emergency Fund

WIPO World Intellectual Property Organization

Project “Using textbooks in DAISY format” – Draft Evaluation Report 28.

Overview of the object of evaluation This evaluation presents the results of the project “Using textbooks in DAISY format – Primary school teacher training in using audio textbooks” implemented from September 2014 to June 2015 in Montenegro. The main objective of the DAISY project is to contribute towards the development of an inclusive education system, allowing CwSEN (Children with Special Education Needs) to have the same learning opportunities as their peers. A DAISY (Digital Accessible Information System) audio-visual textbook refers to the content of a printed textbook in digital audio format1. In order to achieve the intended goals, the project has produced DAISY audio-visual textbooks and introduced them in participating schools (24 schools and the Resource Centre of Podgorica), so that all students can benefit from these didactic tools. After the first year of implementation, the involved stakeholders decided to conduct an evaluation to build knowledge for decision making at different levels, establish goal achievement and define evidence-based findings, conclusions and recommendations for future planning and eventual replication and scaling up. The evaluation was conducted from April to August 2015. Evaluation objectives and intended audiences The main objectives of the evaluation are to:

a) Build knowledge for decision making at different levels (strategic, design and management).

b) Define the level of accomplishment of the expected results and objectives, taking into consideration both positive and negative unintended effects.

c) Provide evidence-based findings, recommendations and lessons learned, with the aim of improving the quality of the intervention and of defining future programme frameworks and scaling up.

The knowledge generated by the evaluation should be used by:

a) Implementing partners – the Resource Centre, Ministry of Education and the Institute for Textbooks and Teaching Aids, as important sources of information for the further scaling up of this initiative – more specifically to identify lessons learned during the implementation, understand the impact of the use of DAISY textbooks in the teaching and learning processes in classrooms, identify areas that require additional attention, potential barriers and bottlenecks in implementation, etc.;

b) Participating schools and other primary schools in order to introduce DAISY textbooks into the teaching and learning processes;

c) Associations of Parents of Children with Disabilities and Children without Disabilities, and parents in general, representing the interest of children to further strengthen their

1 Kuismanen & Holopainen, 2014, “Use of DAISY talking textbooks as study aids”.

1. EXECUTIVE SUMMARY

Project “Using textbooks in DAISY format” – Draft Evaluation Report 29.

monitoring and advocacy efforts;

d) UNICEF Montenegro Office for future programming and support for strengthening the inclusiveness of education for children with disabilities, support for scaling up this same initiative and similar initiatives; and

e) UNICEF Headquarters Disability Section – for future programming, support and replication of similar initiatives in other countries.

Evaluation methodology The evaluation is guided by the criteria proposed in the terms of reference as follows:

a) Relevance: the extent to which the project objectives are consistent with country’s needs, global priorities and policies, beneficiaries’ requirements and UNICEF’s goals. Aspects related to alignment and coherence have been also included;

b) Efficiency: a measure of the achievement of results in relation to the resources used in the search for an optimal allocation thereof, including internal and external coordination;

c) Effectiveness: the degree of achievement of short-term objectives and results through activities;

d) Impact: the positive and negative medium and long-term changes produced by the project, directly or indirectly, intended or unintended. This criterion covers from the fourth level up to the sixth level of the logic framework;

e) Sustainability: future viability of the intervention once support is withdrawn. Assessment of the likelihood of continuation of project benefits after the assistance has been completed; and

f) Cross-cutting issues concerning gender equality and human rights will be taken into account, as well as other aspects and assumptions that could positively and/or negatively influence the programme.

Additionally, the evaluator included a programme theory approach. This perspective states that every project is built upon explicit or implicit models on how the intervention will lead to the desired results. To identify these models, a theoretical model of the programme was designed to establish the underlying intervention rationales, the causal linkages between all elements in the programme that produce results. This theoretical model includes all the relevant aspects of the programme: the results and objectives, processes and activities undertaken to achieve them and the means and resources that are in place to carry out these activities. This approach of studying the intervention based on all its elements is known as the systemic perspective. The relevance of this approach lies in the usefulness it provides in designing an evaluation, allowing a better understanding of the programme, including the cause–effect links generated at all levels. The methodological approach includes quantitative and qualitative strategies. Both quantitative and qualitative techniques have been used to obtain the information needed to answer the evaluation questions. Particular emphasis was given to cross-validation of data and an assessment of plausibility of the results obtained. The methodological mix included a desk review of existing literature, semi-structured individual interviews, focus groups, a survey and direct observation.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 30.

Most Important Findings and Conclusions Key results achieved 2. The intervention is highly relevant, it is in line with national strategies and policies on inclusive education, addresses the main underlying causes of exclusion for CwSEN and it has the potential to respond to their needs (Finding 1). The project design and rationales also reflect international and national norms and strategies on Human Rights and Gender Equality. (Finding 3)

Conclusion 1: The project rationale and objectives are relevant to national priorities and policies on inclusive education, and are suited to address the main causes of exclusion for CwSEN. Human Rights and Gender Equality issues are adequately included in the project design based on international and national norms and standards. (Conclusion 1 is linked to Findings 1 and 3, Relevance). Project activities have contributed to partial achievement of 5 out of the 8 short-term results defined in the evaluation matrix. The key elements that contributed to the achievement of results were:

a) Implementing partners’ high levels of coordination and commitment towards the project;

b) An extremely efficient use of resources based on the volunteer contribution of stakeholders at all levels. No activities were cancelled or not carried out due to lack of funds (Finding 6);

c) Timely production of adequate teaching tools. DAISY textbooks were produced in a

timely manner and are currently available on the RC and ITTA websites (Finding 8); and

d) Provision of adapted capacity building sessions, training materials and production

of DAISY textbooks. Trainers, trainees and students have expressed high levels of satisfaction with the training sessions, materials and produced DAISY textbooks (Finding 4). Teachers’ skills in the use of DAISY are adequate for the effective introduction of these teaching tools in schools. Conversely, parents lack basic skills in the use of DAISY textbooks (Finding 7).

Conclusion 3: The intervention has been implemented in a highly efficient manner. Implementing partners managed to carry out all the planned activities with a very limited budget. (Conclusion 3 is linked to Finding 6, Efficiency). Early indications of the project’s contribution to the medium and long-term objectives could be defined, but impact analysis in this evaluation has been limited by a lack of baseline data and the project’s early implementation stage. The results of the project are likely sustainable, given Implementing Partners’ levels of ownership and responsibility towards the achievement of results. This framework will potentially ensure the continuation of activities, even in case of an eventual withdrawal of external funding. The education staff who are familiarized with DAISY audio-visual textbooks are willing to keep using them in their schools (Finding 12). Regardless of these positive results, the evaluation has identified areas in need of improvement, as expressed in the following findings and conclusions:

Project “Using textbooks in DAISY format” – Draft Evaluation Report 31.

Project design

The project documentation only provides a general overview of the intervention. The project design is better articulated and defined for short-term achievements, so it has been suitable for the given period of time. The current monitoring system lacks specific indicators and benchmarks (Finding 2). Conclusion 2: Despite the fact that the project design leads to the achievement of results during the implementation period, the current result-based framework does not effectively reflect the expected medium and long-term objectives and lacks specific indicators and standards to measure their level of completion. The absence of specific indicators and benchmarks in the monitoring framework prevents the obtaining of reliable performance information. (Conclusion 2 is linked to Finding 2, Efficiency).

DAISY production and provision of capacity-building activities Training sessions have been focused on developing teachers’ skills in order to introduce DAISY audio-visual textbooks in class. The benefits and adequacy of using DAISY audio-visual textbooks with children with no SEN have not been stressed enough during training. Production issues have been detected (inadequate intonation and modulation, voices by role, page information and other interruptions, etc.), lessening the potential impact of the intervention (Finding 5). Education staff are aware of general strategies on inclusive education, but lack the specific skills and practice in how to effectively introduce those concepts during sessions with DAISY (Finding 11). The use of DAISY should be combined with traditional activities during each session and long listening periods avoided to produce a more dynamic session, otherwise it is difficult to keep the students’ attention. Participation of teachers and parents in capacity-building sessions has not reached the minimum standards set by the project (Finding 7). Conclusion 4: Training sessions have been very relevant for teachers, since the sessions particularly focused on the satisfactory introduction of DAISY audio-visual textbooks in class. The project relied on schools to actively inform and motivate parents to join the training sessions, but this was not effectively conducted. A wider approach, putting more effort into directly addressing the need to involve all potential users and beneficiaries of the intervention (especially parents and students) not only positively affects the involvement of indirect beneficiaries, it also provides the grounds for developing a more inclusive educational environment (Conclusion 4, linked to Finding 4, Effectiveness)

Stakeholders and beneficiaries’ engagement A lack of information and awareness on the potential of DAISY textbooks prevents teachers and particularly parents from being more involved in the initiative. Parents requested greater involvement in the project through adapted training, so that they could support their children according to their specific needs. Motivation to use DAISY is higher among parents who know about the project (particularly parents of CwD), but they are not sufficiently involved (Finding 7). Students and parents are not informed about the availability of DAISY textbooks. Students do not know how to use DAISY textbooks, but are motivated to learn and use them at home. DAISY textbooks are not used at home yet. A lack of technical equipment at schools has been identified as a widespread threat to the effective introduction of DAISY textbooks. The reading software needs a relatively new and updated computer to run all its features, and that could hinder its effective introduction in schools (Finding 9). Conclusion 5: Promoting the project is essential in order to ensure the engagement of the direct beneficiaries and students. This is the very first step towards achievement of the goals, affecting

Project “Using textbooks in DAISY format” – Draft Evaluation Report 32.

the performance of the results at higher levels. Information-sharing and awareness-raising activities are crucial, since the majority of direct beneficiaries, target groups and indirect beneficiaries have expressed their motivation to use DAISY audio-visual textbooks, but are not adequately informed. Information converts scepticism towards the project into a willingness to be engaged, and that is something the project has to develop further (Conclusion 5 is linked to Findings 7 and 8, Effectiveness).

Conclusion 6: A lack of technical equipment in schools and DAISY software requirements are a threat to the effective introduction of DAISY in schools and at home. Production issues hinder the potential impact of the project (Conclusion 6 is linked to Findings 5 and 9, Effectiveness).

Contribution to future impact evaluation

Early evidence of DAISY sessions improving group dynamics and CwSEN participation in class has been identified. Students have a positive attitude towards the use of DAISY textbooks. The project’s contribution to improving CwSEN confidence could not be clearly established, but the findings indicate a potential positive effect that can be attributed to the intervention. No evidence is available on the project’s contribution to better integration of CwSEN in the teaching and learning processes (Finding 10). As mentioned in the report, the project is currently in its pilot stage, and DAISY textbooks were only used a few times (on average, three to five sessions per school) over a short period of time (less than six months).

Conclusion 7: The evaluation could not establish clear and solid evidence of the project’s contribution to its medium- and long-term objectives. Findings only provide grounds to determine early evidence of a contribution or positive tendencies. DAISY has the potential to support teachers in the process of developing and improving the inclusive teaching process, so that all students can benefit from it. Specific training on inclusive strategies is required, as well as the inclusion of a practical approach on how to effectively introduce DAISY into the classroom (Conclusion 7, linked to Findings 10 and 11, Impact).

Long-term sustainability

No formal sustainability or exit strategies have been agreed among partners, but they all share their commitment and willingness to continue with project activities. Education staff that are informed and have conducted sessions with DAISY will continue with the initiative even without the support of the project. The sustainability of the production of DAISY audio-textbooks at the same rate is not ensured (Finding 12).

Conclusion 8: Implementing partners have developed a level of ownership and responsibility towards the achievement of results that will potentially ensure the continuation of activities even considering an eventual withdrawal of external funding. Nevertheless, a clear sustainability and exit strategy is required to ensure project autonomy in case external financial support is withdrawn. Education staff that are familiarized with DAISY audio-visual textbooks are willing to keep using them in their schools. The current production rate is not sustainable in its present form (Conclusion 8, linked to Finding 12, Sustainability). Main Recommendations Recommendation 1 Implementing partners should develop a fully fledged, results-based framework and monitoring system, clearly stating medium- and long-terms goals, SMART indicators for every activity and result, and the monitoring strategy to track goal achievement and potential deviations. (Linked to Conclusion 2 and Finding 2.) Priority level: High

Project “Using textbooks in DAISY format” – Draft Evaluation Report 33.

Recommendation 2 Implementing partners should take into consideration the following proposals for future training and DAISY textbook production:

a) Develop specific training for parents, focusing on their particular role in supporting the process of learning.

b) Training sessions and materials should highlight the adequacy and relevance of using DAISY textbooks with all students, not only CwSEN, and include a practical session on how to conduct a class with DAISY textbooks, with a focus on inclusiveness. IT teachers should participate in training, so that the rest of the teachers can rely on them if help with technical issues is required in schools.

c) Produce a manual on the use of DAISY based on the training experiences and lessons

learned, including best practices. It should describe how to work with all types of children, including specific strategies for different types of disabilities.

d) To the extent possible, the project should address identified production issues (inadequate

intonation and modulation, voices by role, page information and other interruptions, etc.). Implementing partners should include these items in the monitoring system (supervisory reports and lesson observation forms) to ensure the timely identification of adaptation issues and their correction, and develop a clear quality assurance plan and mechanisms.

e) Concerning copyright issues, a standard procedure to engage authors should be

developed. This could take the form of legal agreements signed by both parties and in which the project ensures a faithful adaptation of the original material. The project could also consider joining WIPO’s (World Intellectual Property Organization) TIGAR (Trusted Intermediary Global Accessible Resources) service, which allows participating institutions to search internationally for books in an accessible format and to exchange them. Participation in the service is free and is supported by the DAISY consortium, among others. By joining the service, the project would have access to over 286,000 titles in 55 different languages, and that would for example be very relevant for foreign language classes.

f) To the extent possible, the intervention should ensure adequate equipment is available in

participating schools for the effective introduction of DAISY audio-visual textbooks. This could be done through monitoring activities and IT support, especially when it comes to software requirements. The IT teachers in every school should provide regular computer maintenance to ensure that DAISY software smoothly runs its basic features in those computers.

(Linked to Conclusions 4 and 6 and Findings 4, 5, 9 and 11.) Priority level: Very high Recommendation 3 Implementing partners should extensively conduct promotional and awareness-raising activities in the areas of intervention, addressing education staff, associations of parents, other families and students (especially children with no special education needs). These activities should focus on promoting the project, its objectives (short-, medium- and long-term), conduct practical sessions with DAISY audio-visual textbooks and inform about their availability. Promotional activities could be strengthened using specific tools such as flyers, leaflets, videos and

Project “Using textbooks in DAISY format” – Draft Evaluation Report 34.

promotional CDs comprising all DAISY textbooks and reading software. Stronger partnerships with parents’ associations would enhance the scope of the project and its potential. The MoE could play a major role in the promotion of the DAISY project, given its credibility as a governmental body. In addition, the school management is also a key stakeholder in this matter, in promoting the project within schools and among parents. Finally, the RC could also play a major role in promoting the use of DAISY textbooks among children and parents.

(Linked to Conclusion 5 and Findings 7 and 8.) Priority level: Very high

Recommendation 4

The definition of a baseline would provide extremely relevant information for eventual future impact evaluation. Implementing partners could also contemplate the idea of defining the control group schools to conduct this type of evaluation.

(Linked to Conclusions 2 and 7 and Findings 2 and 10.) Priority level: High

Recommendation 5

Implementing partners should define a clear and realistic sustainability and exit strategy, including shared and individual responsibilities, continuity plans, benchmarks, funding opportunities and timelines. These documents should be periodically reviewed and adapted to the political and economic context that might affect the project. The sustainability and exit strategy should be integrated into all aspects of programming, aimed at ensuring sustainable outcomes. In line with this, implementing partners should define specific action strategies for the production of DAISY audio-visual textbooks, either by reducing the production rate or by providing additional support (professional or voluntary, considering some monetary contribution for the production process.

(Linked to Conclusion 8 and Findings 5 and 12.) Priority level: High

Lessons Learned 1. Access to educational needs for CwSEN and gender mainstreaming: Students with

special education needs face difficulties and constraints in accessing their education needs in mainstream schools in Montenegro. Moreover, the evaluation process has identified that differences in equal access to the project and its benefits are issues that also affect children based on their level of disability. When conducting the analysis of gender mainstreaming issues related to the project, cross-referencing of information obtained from different stakeholders showed that the imbalanced access to educational needs in mainstream schools is based on the student’s grade of disability, and not due to his/her gender. No limitations in accessing those rights could be attributed to gender inequalities. This lesson learned should be taken into consideration when developing scaling-up strategies and studying other contexts and situations. Other potential constraints should be also studied, especially those that could affect marginalized groups such as ethnic minorities or children in vulnerable families.

2. Stakeholder engagement and efficiency levels: Several difficulties are encountered when

introducing a pilot project in any kind of context. The lack of previous experience and knowledge about the intervention are factors that can seriously hinder the involvement of stakeholders at all levels, an essential condition to ensuring an effective and sustainable intervention. This affects not only the target group and indirect beneficiaries, but also the implementing partners. The level of commitment among implementing partners may vary, especially if an intervention is based on voluntary support, as is the case in the DAISY project. This situation has enabled a highly efficient management of resources to achieve results. The downside of this approach is the difficulty in securing stakeholders’ commitment in the medium and long terms. Implementers should be open to sacrificing efficiency levels to some extent

Project “Using textbooks in DAISY format” – Draft Evaluation Report 35.

in order to ensure the full commitment and availability of all relevant stakeholders, especially those involved in the production process.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 36.

2. OBJECT OF THE EVALUATION

INTRODUCTION

1. This report documents the results of the evaluation of the project “Using textbooks in DAISY format – Primary school teacher training in using audio textbooks”, funded by UNICEF Montenegro in collaboration with the “Podgorica” Resource Centre for Children and Youth and implemented by the Resource Centre in collaboration with the Ministry of Education, the Bureau of Educational Services, the Institute for Textbooks and Teaching Aids and the Drama Academy. The report provides information on the context within which UNICEF operates in Montenegro and specifies the evaluation methodology, which thus determines the exact focus and scope of the exercise, and the results of the evaluation. The evaluation was carried out between May 2014 and July 2015 by an external evaluator. The UNICEF Montenegro Country Office – Evaluation Steering Committee (ESC) will review the draft evaluation report, provide comments and sign-off the final evaluation report.

OBJECT OF THE EVALUATION

3. The project “Using textbooks in DAISY format – Primary school teacher training in using audio textbooks” represents the efforts of UNICEF, the “Podgorica” Resource Centre for Children and Youth and all relevant stakeholders involved, to develop a more inclusive education system in Montenegro. The intervention in its pilot phase provides accessible and adapted teaching and learning materials, DAISY audio-visual textbooks, for all students in 24 schools and the Resource Centre of Podgorica.

4. Montenegro is the first country that has adapted a set of regular textbooks into the DAISY format with the intention to introduce them as regular teaching material in mainstream schools, piloting the intervention in 24 primary schools. The particular quality of this initiative lies in the close partnership between educational institutions, from the Ministry of Education (MoE) to the Bureau of Education (BoE), the Institute for Textbooks and Teaching Aids (ITTA) and the Resource Centre (RC), as the main implementing agency, with the Faculty of Dramatic Arts whose teachers and students have volunteered voices for the production of DAISY audio-visual textbooks. The results of this intervention might define future directions in the education reform process when it comes to the issues of access as well as quality of teaching and learning for children with disabilities.

5. The “Podgorica” Resource Centre for Children and Youth launched the project “Using textbooks in DAISY format – Primary school teacher training in using audio textbooks” in June 2013. An editorial team comprising all the implementing partners (the Resource Centre, the Ministry of Education, the Bureau of Educational Services, the Institute for Textbooks and Teaching Aids and the Drama Academy) was created in order to monitor the adaptation of DAISY audio-visual textbooks. Additionally, a production team (manager and technical editor from the RC) was designated and trained by international consultants in the process of recording and adapting standard textbooks to DAISY format. By July 2014, the production team, in collaboration with volunteers from the Drama Academy, had produced the first set of DAISY audio-visual textbooks (three textbooks) that are published on the RC and ITTA websites. The production continued and currently there are 6 DAISY audio-visual textbooks on language and literature, available for grades 4 to 7.

6. Selection of schools was carried out based on a needs assessment and the number of CwSEN per school. Participating schools were grouped into “primary-focus schools” and “other

Project “Using textbooks in DAISY format” – Draft Evaluation Report 37.

schools”. There are 11 schools in the “primary focus” group, and 14 schools in the group of “other schools”.

7. Effective project implementation took place from September 2014 until June 2015. The geographical areas where the intervention took place in Montenegro are the following: Podgorica, Bijelo Polje, Berane, Pljevlja, Nikšic, Danilovgrad, Tivat, Bar, Kotor and Herceg Novi. During this period of time, five training sessions for education staff and parents in the use of DAISY were completed as follows:

(a) First training held on 11 October 2014 at the Resource Centre Podgorica. A total of 16 teachers, one parent and one representative from the BoE attended the session;

(b) Second training, held on 26 October 2014 at the Resource Centre Podgorica. A total of nine teachers, three parents and one representative from the BoE attended;

(c) Third training held on 8 November 2014 at “Marko Miljanov” Primary School. A total of 18 teachers, two representatives from day care centres (“Tisa” and “Pljevija”) and two representatives from the BoE attended;

(d) Fourth training held on 14 November 2014 at “Braca Ribar” Primary School. A total of 24 teachers, one representative from the association of persons with disabilities Niksic, one representative from the parents’ association (“Let’s grow together”) and two representatives from the day care centre for CwD in Niksic attended; and

(e) Fifth training held on 20 December 2014 in Kotor. A total of 19 teachers attended the session.

8. Monitoring visits to schools were conducted from February 2015 until May 2015. UNICEF provided a total of 29 completed monitoring forms from 11 different schools (Braca Ribar, Dragiša Ivanovic, Luka Simonovic, Maksim Gorki, Milan Vukovic, Njegoš Kotor, Savo Pejanovic, Mileva Lajovic Lalatovic, Ratko Žaric, Njegoš Spuz and Vladimir Nazor). Students with special education needs participated in the sessions with DAISY audio-visual textbooks in eight of these schools.

Project Objectives and goals – The programme’s logic model

9. The intervention was articulated through a theory of change designed by UNICEF that was developed after the establishment of the project design, with the aim of contributing to its evaluability. A logic model of the project was designed after reviewing the theory of change provided by UNICEF. The logic model includes all relevant aspects of the programme: resources and structure, activities, results, outputs and outcomes. Additionally, several assumptions have been detected at all levels to ensure that the causal linkages between all elements occur.

10. Overall objective: The final and ultimate objective of the project is to contribute to the development of an effective inclusive education system in Montenegro. This is a long-term objective that requires the support and collaboration of different stakeholders (institutions, national bodies, organizations, civil society, etc.) and frameworks (projects, laws, strategies, etc.). This overall objective has been defined as “All students have the same opportunities in a developed inclusive education system” in the theory of change.

11. Specific Objective 1: The first specific objective identified is “Teaching process improved in 24 inclusive schools and the Resource Centre”. It is one of the two main objectives defined, specifically referred to the contribution of the project to develop education staff skills and attitudes to ensure an inclusive and effective teaching environment. The achievement of this objective requires both the use of new technologies (DAISY) and classical educational techniques for CwSEN. Educational staff includes teachers, education specialists and directors, but parents also play an important role by supporting their children in their learning process. “The influence of

Project “Using textbooks in DAISY format” – Draft Evaluation Report 38.

teacher attitudes emanates from the fact that the individual classroom teacher remains the most effective variable to influence the performance of students.”2

12. Specific objective 2: The second specific objective of the project is to improve the learning process for all students in 24 inclusive schools and the Resource Centre. This objective refers to the contribution of the project to developing an improved learning environment, focusing on the changes produced on students. Students (especially CwSEN) should experience an improved learning environment and develop positive attitudes toward school and individual learning at home through reliable modern learning technologies. Students should be accepted among them as social members of the teaching groups. Learning process improvement could be measured based on changes in attitudes towards school, emotional satisfaction and usefulness concerning the use of DAISY, engagement in students’ own learning, and the development of learning opportunities for all students. The two specific objectives are intimately intertwined, since the teaching and learning processes can be understood as one. For evaluative reasons, these two aspects have been separated and studied individually, based on the stakeholders involved in each one.

13. Result 1: “Teachers able to better organize the teaching process for joint learning of CwD and children without disability”. This result, in contrast to Objective 1, specifically refers to the ability of teachers to adapt the teaching process to the use of DAISY audio-visual textbooks, and conduct joint learning strategies to ensure that all students participate in the class dynamics. Teachers have skills to deal with individual differences and instructional processes developed for CwD, such as cooperative learning. Teaching staff are motivated and have a positive attitude towards the inclusion of CwD in their class and the use of DAISY audio-visual textbooks. As mentioned in a previous study, “teachers should encourage the students to commit to the talking textbooks because not all children have the initiative to start listening, students should not be left alone with their talking textbooks – they need someone to listen with them, ask questions and point out important parts.”3

14. Result 2: “Parents of CwD supportive of providing assistance to their children in the learning process through DAISY”. Parents support the learning process with DAISY audio-visual textbooks books by providing assistance to their children in the use of these devices. Families have access to the required hardware at home. As mentioned in a previous study “the use of a talking textbook requires motivation and family support, emphasizing the importance of support from the family, both in terms of starting to use the textbooks and in monitoring their use and providing assistance.” 4

15. Result 3: “CwD feel less isolated from the teaching process and more integrated in the work of the class”. This objective is based on the assumptions that, once DAISY audio-visual textbooks are available and all students learn together with them, it will provide the grounds for CwD to experience the benefits of using DAISY audio-visual textbooks, ultimately stimulating group work and feeling less isolated from the class dynamics.

16. Result 4: “CwD gain more confidence through facilitated learning in a group and independently”. Based on the same causal chain as Result 3, this result includes the benefits of facilitated learning both in groups and independently, thus increasing CwD’s confidence. The feedback loop in the logic model between Results 3 and 4 represents the hypothetical positive correlation between the level of integration and the level of confidence, meaning the values of these variables increase or decrease together.

2 Gudyanga E. Challenges Faced by Students with Hearing Impairment in Bulawayo Urban Regular Schools 3,12 Kuismanen & Holopainen, 2014, “Use of DAISY talking Textbooks as study aids”.

Figure 1: PROGRAMME LOGIC MODEL

All students have the same

opportunities in a developed

inclusive education system

Teaching process improved

in 24 inclusive schools and

the Resource center

Learning process improved for

all students in 24 inclusive

schools and the Resource

center

IMP

AC

T O

BJE

CTI

VES

Physical environment

(classroom design, noise, etc.)

adapted to CwD needs

2N

D G

RO

UP

OF

RES

ULT

S

All teachers trained in

general inclusive education

Teachers able to better

organize the teaching

process for joint learning

of CwD and children

without disability

Parents of CwD

supportive of providing

assistance to their

children in the learning

process through DAISY

CwD feel less isolated

from the teaching

process and more

integrated in the work

of the class

CwD gain more

confidence through

facilitated learning in

the group and

independently

ASSUMPTIONS

Accessible technical support

and equipment Students EXPERIENCE the benefits of

using DAISY books

School management supportive

on the use of DAISY in classroom

Parents of CwD have

electronic devices at

home

1ST

GR

OU

P O

F R

ESU

LTS

Programme staff

understands the

intervention and its

objectives

Training materials

and strategy reflect

the intended impact

Teacher and parents are

motivated and aware of the

benefits of using DAISY books

Teachers and parents

gain the skills to use

DAISY textbooks

Daisy books are

accessible and available

CwD and children without

disabilities learn together

through the use of DAISY books

Teachers and parents attend training

AC

TIV

ITIE

S

Identify needs and

beneficiaries

Develop training

materials, manuals

and guidelines

Raise awareness of the

benefits of using DAISY

books among teachers

and parents

Train teachers, supervisors and

parents on the use of DAISY

books (technical and

pedagogical)

Daisy textbooks in mother

tongue produced and

introduced in 24 schools and

the resource centre for all

students

Individual and group learning

sessions in the use of DAISY

books facilitated to all students

Management structures (Ministry of Education, Bureau for Education, Resource Centre, Textbook Publishing Agency, UNICEF, schools), human and financial resources, technical expertise, Drama

Academy, project design STR

UC

TUR

E

17. First group of results:

(a) Programme staff understand the intervention and its objectives

(b) Teachers and parents are motivated and aware of the benefits of using DAISY audio-visual textbooks.

(c) Teachers and parents gain skills in the use of DAISY audio-visual textbooks.

(d) DAISY audio-visual textbooks are produced and accessible

(e) All students learn together how to use DAISY audio-visual textbooks.

18. Activities (in logic sequence):

(a) Identify needs and beneficiaries

(b) Develop training materials, manuals and presentations

(c) Inform and raise awareness on the benefits of using DAISY audio-visual textbooks

(d) Train direct beneficiaries on the use of DAISY audio-visual textbooks (technical and pedagogical)

(e) Produce and introduce DAISY audio-visual textbooks

(f) Facilitate individual and group learning sessions in the use of DAISY to all students

(g) Facilitate session with DAISY in class

19. Assumptions: external presuppositions that need to occur at all levels for the achievement of results and objectives.

20. The logic model comprises six different levels, each of them referring to a particular sequence of the intervention, assuming that the availability and successful completion of a lower level leads directly to the next one.

(a) First level – Resources and structure: Includes all the available resources needed for the implementation of the project, all elements that allow processes and activities to be carried out, e.g. management structures, human and financial resources, technical expertise, legal framework, and strategies on inclusive education.

(b) Second level – Processes and activities: Includes the sequence of activities (process) that initiates the causal chain of changes, leading to the first level of results.

(c) Third level – First group of results: Represents the short-term consequence of the implemented activities. In this sense, if an effective awareness-raising activity is conducted with parents and teachers, it is expected that they will be motivated and aware of the benefits of using DAISY.

(d) Fourth level – Second group of results: Results arising as a consequence of partial or complete achievement of the first group of results. This level comprises medium-term results, e.g. if parents are motivated in the use of DAISY audio-visual textbooks and have gained enough skills to use them, they will be supportive on providing assistance to their children at home. Please notice that a few assumptions included in the logic model should be considered under this logic chain, such as the availability of electronic devices to parents at home.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 41.

(e) Fifth level – Objectives: This level includes the two main long-term objectives of the programme.

(f) Sixth level – Impact: The ultimate goal of the intervention – what the project aims to contribute to in the long term – that all students have the same opportunities in a developed inclusive education system in Montenegro.

21. The level of completion of results and objectives has been rated in the evaluation matrix (Annex 4) as follows:

(a) Fully achieved: very few or no shortcomings;

(b) Largely achieved: only a few shortcomings;

(c) Partially achieved: benefits and shortcomings finely balanced;

(d) Very limited achievement: extensive shortcomings; and

(e) Not achieved

INTERNATIONAL CONTEXT

22. At the international level, the main conventions on inclusive education are:

(a) The United Nations Convention on the Rights of the Child (1989), that protects and promotes the rights of all children, including disabled children. Articles 2, 23, 28 and 29 directly refer to the rights of Children with Disabilities (CwD);

(b) The “Salamanca Statement, (1994)”, a document that represents the recognition of the need to work towards institutions that include everybody, celebrate differences, support learning and respond to individual needs5;

(c) The UN Convention on Rights of People with Disabilities (2006), which Montenegro ratified in 2009, adopted a broader categorization of persons with disabilities and reaffirmed that all persons with all types of disabilities must enjoy all human rights and fundamental freedoms; and

(d) The Dakar Framework and the Education for All initiative are also key documents aimed at promoting the rights of students with special education needs.

23. The Education for All (EFA) initiative is a global commitment first launched in 1990 aimed at providing quality basic education for all children. Among its six main goals, two are directly linked to the scope of the DAISY project:

a) Ensuring that the learning needs of all young people and adults are met through equitable access to appropriate learning and life-skills programmes; and

b) Eliminating gender disparities in primary and secondary education by 2005, and achieving gender equality in education by 2015, with a focus on ensuring girls' full and equal access to and achievement in basic education of good quality.

5 The Salamanca Statement and Framework for Action on Special Needs Education (June 1994)

Project “Using textbooks in DAISY format” – Draft Evaluation Report 42.

NATIONAL CONTEXT

24. Montenegro gained its independence and became a member of the United Nations in 2006. The Constitution of Montenegro (2007) guarantees basic human rights and freedoms and protects all citizens from all forms of discrimination. The evolution of Montenegro’s education system is based not only on its constitution, but also on all relevant international agreements and declarations related to the preservation of human dignity. Despite its successful economic growth since 2000, the country faces important challenges in complying with the Education for All (EFA) and Millennium Development Goals (MDG) standards. The delayed transition of Montenegro has had an adverse impact on the education sector which remains disconnected from the latest thinking and innovations, as well as from labour market needs – PISA results show that Montenegrin students lag behind their OECD peers by two years in terms of the skills and competencies necessary for their future education and success in life and employment (50% of Montenegrin students have scored below basic functional literacy levels in mathematics, science and reading).6

25. The main education reforms, laws and strategies concerning the rights and features of Montenegrin children with special educational needs include:

a) Elaboration of the “Book of Changes of the Education System of Montenegro” (2001), a comprehensive and progressive reform aimed at improving the quality, access and equality of the Montenegrin education system. This reform states the need to prepare the suitable conditions in order to include students with special needs in mainstream schools. The General Law on Education (2002, 2013) and other laws that regulate education are based on the principles included in the “Book of Changes”;

b) Establishment of a national forum for EFA, aimed at monitoring the progress towards achieving the EFA goals;

c) The Plan for Action for Children 2004–2010 that defines key strategies in health, education and child protection, to alleviate poverty and meet the MDGs;

d) The Law on Education of Children with Disability (2004);

e) The Inclusive Education Strategy (2008–2013); and

f) The strategy of inclusive education in Montenegro (2014–2018).

26. Children with special education needs (CwSEN)7 are among the most vulnerable groups in Montenegro in terms of access to fundamental rights. Moreover, CwSEN do not represent a homogeneous group, and other aspects of their identity such as economic status, gender or ethnicity can play a major role in their vulnerability and social inclusion. Therefore, disability has to be understood as one aspect to be taken into consideration when studying discrimination, but not as an isolated form of exclusion. Nevertheless, late reforms in the education system have put special effort into developing a more inclusive framework for CwSEN. The adoption of two inclusive education strategies for the periods 2008–13 and 2014–18 provides the basis for a systematic development of the sector, in line with best international practices. The leading principles guiding these documents describe how to achieve accessible education for children with special education needs in accordance with their interests, abilities and needs. Overall, the main goal is to provide equal educational opportunities at all levels for all children.

6 “Using textbooks in DAISY format” Terms of Reference 7 The term refers to both children with disabilities (physical or intellectual) and children with learning difficulties.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 43.

27. The Ministry of Education of Montenegro has implemented a large number of initiatives and has undertaken comprehensive reforms in order to develop an inclusive education system. Here we will mention just a few:

a) “Integration of children with special needs into regular kindergartens” pilot project (1998);

b) The Active Learning Project, implemented by the MoE and UNICEF (1994–2004);

c) Establishment of the Commissions for the Orientation of Children With Disabilities at the municipal level, consisting of paediatricians, social workers and school psychologists/pedagogues, who are to assess and propose, with the help of parents, the abilities of the child and the level and scope of the educational programme (2008);

d) “It’s About Ability” – a comprehensive public awareness campaign conducted with the support of UNICEF and in partnership with schools, central educational institutions, businesses, etc., aimed at changing the attitudes of parents and the general public, as well as teachers and schools, regarding the status, position and rights of children with special education needs (2010–2013). The campaign was accompanied by a variety of programme activities, including the training of teachers, capacity building of local commissions, etc., which led to an almost five-fold increase in the number of CwSEN in primary schools in the period from 2010 (654 children) to 2015 (1,450 children). The campaign itself had a great impact on the attitudes of the public. For example, the number of citizens who found it unacceptable that their child should go to the same class as a child with disability fell from 64% in 2010 to 39% at the end of 2012;

e) The Network for Support for Inclusive Education, with the support of UNICEF (2013);

f) EU IPA project: Inclusive Education Services (2011–2013), including three main components: a) transformation of special institutions into resource centres, b) increasing the number of Roma children in preschools and primary schools, and c) improving the systems and mechanisms for the monitoring and assessment of progress of CwD;

g) Comprehensive Support for Quality Continuation of Inclusive Education, with the support of UNICEF (2014–2015), including enhancement of IDEPs (individual development and educational plans for CwSEN) and establishment of ITPs (individual transition plans for CwD who are to enrol in secondary schools);

h) From Inclusion Index to an Inclusive Culture, with the support of Save the Children (currently ongoing); and

i) The “Inclusive education” project implemented by the MoE in partnership with UNICEF, Save the Children and the NGO sector.

UNICEF CONTEXT

28. As part of its mandate, UNICEF Montenegro has a commitment to ensure the inclusion of all children in the education system, particularly those with special education needs. To achieve this goal, it is necessary to build inclusive education systems that promote schools based upon “a child-centred pedagogy capable of successfully educating all children, including those who have serious disadvantages and disabilities. The merit of such schools is not only that they are capable of providing quality education to all children; their establishment is a crucial step in helping to change discriminatory attitudes, in creating welcoming communities and in developing an inclusive society. A change in social perspective is imperative.”8

8 UNICEF, 2005, Innocenti Insight, Children and Disability in Transition in CEE/CIS and Baltic States

Project “Using textbooks in DAISY format” – Draft Evaluation Report 44.

29. UNICEF’s core framework for quality education comprises five key components: inclusiveness, effectiveness, a healthy and protective, gender-sensitive and participatory approach that promotes the participation of children, parents and communities in the education system. Focusing on the inclusiveness aspect, UNESCO defines inclusive education as “a process of addressing and responding to the diversity of needs of all learners through increasing participation in learning, cultures and communities, and reducing exclusion within and from education. It involves changes and modifications in content, approaches, structures and strategies, with a common vision which covers all children of the appropriate age range and a conviction that it is the responsibility of the state to educate all children.”9 Therefore, inclusion is an aspect that involves all children, not only CwSEN. The main idea is to provide the grounds for all children to socialize in an environment where diversity is the norm rather than the exception, generating inclusion and acceptance.

30. Several initiatives have been undertaken in the last few years in order to assist the country to develop an inclusive environment for CwSEN, particularly through capacity building, awareness-raising campaigns and legislative interventions. The C4D campaign “It’s About Ability” was launched in September 2010, having a recorded positive impact on people’s attitudes towards CwD. Since 2014, the implementation of the “Using textbooks in DAISY format” project complements this line of action.

SITUATION ANALYSIS

31. The nature and extent of the main problem that the programme addresses:

a) What is the problem, issue or opportunity? CwSEN do not have the same learning opportunities as their non-disabled peers. They are disadvantaged in relation to their classmates due to the lack of access to adapted teaching and learning processes that ensure their inclusion in the educational system and achievement of their full potential. There is an expressed need to further develop and improve the teaching and learning environment in schools that could benefit both CwSEN and children without special education needs. The “Using textbooks in DAISY format” project aims to tackle this problem by adapting mother tongue textbooks into DAISY audio-visual format and introducing them in participant schools, so that all students can benefit from these didactic tools. The project ensures training to teaching staff, parents and students on the didactic and pedagogical use of DAISY audio-visual textbooks, not only for educational purposes, but also to initiate the causal chain leading to the final project objectives of developing an improved teaching and learning environment. Many educational materials are only available in regular print, which is frequently an important part of the classroom activities. This is an opportunity that the project identified and took into account by enabling the use of these teaching materials by all the students. However, equal access to DAISY audio-visual textbooks does not necessarily bring equal opportunities to all students: other aspects concerning the learning environment play a major role such as social interaction, context, support from parents and teachers, etc. The adaptation of these particular teaching materials to the context where they are introduced is also essential, as well as their availability. In this regard, an improvement of accessibility does not necessarily leads to an improvement on usability, and this is another aspect the evaluation has considered.

b) For whom does this problem exist? Children with disabilities, defined as those who have long-term physical, mental, intellectual or sensory impairments, which in interaction with various barriers may hinder their full and effective participation in society on an equal basis with others.10 “Disability” describes the condition whereby physical and/or social

9 UNESCO 2005, Guidelines for Inclusion: ensuring access to education for all, Paris 10 United Nations, Convention on the Rights of Persons with Disabilities, United Nations, New York, 2006. See <www.un.org/esa/socdev/enable>.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 45.

barriers prevent a person with impairment from taking part in the normal life of the community. The concept of inclusion implies a sense of belonging and acceptance (Chakuchichi et al, 2002), a positive response to individual differences (Pijl, Meijer and Hegarty (1997) and developing inclusive environments that promote educational success for all students (Hodkinson and Vickerman, 2009). The impact of the project is directly linked to these concepts, and could be partially measured in terms of contribution to the development of an inclusive educational system in Montenegro.

c) What is the history of the problem? Montenegro has undertaken major reforms in the education system since 2001, including strategies in inclusive education. In addition, an extensive education curriculum has been developed, addressing ethnic divisions and national cohesion. The Montenegrin Law on Education allows CwSEN to enrol in any school, but effectively the availability of schools for them is limited due to the lack of preparedness of educational staff to deal with CwSEN, lack of adequate didactic materials, some schools lacking an adequate physical environment, issues with engagement of teaching assistants, etc.

32. Cause and contributing factors:

a) Why does this problem occur? What are its causes? Are some causes more important or influential than others? There is a consensus between decision-makers, education professionals and parents that significant challenges remain when it comes to consistent implementation of adopted policies, particularly those concerning the development of teachers’ skills in inclusive strategies, the appropriate allocation of resources, inter-sectoral collaboration and adapted teaching materials adjusted to the needs of CwD.11 The work with CwD in inclusive education is quite new for teachers, who lack key competences in this area. Professional development programmes organized by the Department of Education only reach a reduced group of teachers who go through a number of training sessions, while the vast majority are left behind12. Teachers are also very critical concerning the support received from central institutions, due to the high administrative requirements that they have to comply with and the lack of real support from those institutions. Inter-sectoral collaboration appears as an essential aspect to be developed. Finally, adapted teaching materials have been limited to the use of Braille books in class.

33. What, if anything, is known about what has and has not been effective in addressing the problem through the use of DAISY audio-visual textbooks? Available research found on the use of DAISY audio-visual textbooks and/or other type of electronic textbooks is quite limited. Scandinavian countries such as Norway (Nes, 2007), Sweden (Lundh, 2013), Finland (Kuismanen & Holopainen, 2014) and Denmark (Nota, 2011) have put effort into studying the impact of introducing DAISY audio-visual textbooks in both primary and secondary schools. Other pilot projects have been also implemented in New Zealand (Clunie, 2009), UK (Russell & Draffan, 2011), Canada (Lockerby, 2006) and the Netherlands (DEDICON 2007). Information obtained from those studies was considered during the evaluation design process, and reflected in some of the main evaluation questions included in the evaluation matrix.

Groups of identified stakeholders

34. The following groups of stakeholders were identified:

a) Implementing partners: Editorial team (The Resource Centre, Ministry of Education, Bureau of Education, Institute for Textbooks and Teaching Aids and the Drama Academy).

11 “Using textbooks in DAISY format” Terms of Reference 12 Micanovic, V & Vuckovic, D. Some aspects of the primary education reform process in Montenegro from the perspective of teachers, 2014.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 46.

b) Donor: UNICEF (in collaboration with the Resource Centre)

c) Target group: Children with special education needs (grades 4 to 9) in 25 participating schools. The target group comprises CwD and CwLD. CwD are represented by children with visual impairment, children with physical disabilities, children with intellectual disabilities, children with autism, children with speech disorder and children with combined disabilities. The CwLD group includes children with behavioural disorders, chronic illness, long-lasting illness, emotional difficulties, learning difficulties and children facing language, social and cultural barriers. Considering all participating schools, a total of 777 students with special education needs (grades 4 to 9) are potential beneficiaries (target group) of the DAISY project. This group comprises 296 CwD and 481 CwLD.

d) Direct beneficiaries: Primary school teachers in 25 participating school, education specialist representatives, representatives of the associations of parents of CwD, supervisors/education advisors from the Bureau of Educational Services and families of children with and without disabilities. The project goal for the first year of implementation was to train 110 teachers from participating schools and 15 parents and/or representatives from parents’ associations.

e) Indirect beneficiaries: Students with no special education needs (grades 4 to 9) in 25 participant schools. A total of 14,112 students with no special education needs could potentially benefit from the project.

35. A total budget of €7,553 (€6,563 contributed by UNICEF and €990 contributed by the Resource Centre) was available to finance proposed activities during the implementation period (September 2014 to June 2015).

36. Other implementing partners contributed to the project as follows:

(a) The Drama Academy: A total of 11 people were directly involved in the production of DAISY audio-visual textbooks on a voluntary basis. A speech teacher and six students recorded all the material, and four other teachers at the academy revised the final products.

(b) The Resource Centre Information Technology (IT) team (three people) adapted recordings to the specific DAISY format on a voluntary basis.

(c) Representatives from the BoE (two people) participated in the school selection process, supervised the production of DAISY audio-visual textbooks, facilitated training sessions and conducted monitoring visits to participating schools.

(d) The MoE (one person) supported the DAISY initiative and provided advice through their specialist in inclusive and special education.

(e) The ITTA (one person) provided support for editing the DAISY audio-visual textbook contents and facilitating training sessions.

(f) UNICEF supported the RC’s original idea of introducing DAISY audio-visual textbooks, providing not only financial support but also guidance and support in programme and management matters, as well as the credibility associated with the agency, which allowed the engagement of other implementing partners.

3. EVALUATION PURPOSE, OBJECTIVES AND SCOPE

Project “Using textbooks in DAISY format” – Draft Evaluation Report 47.

THE EVALUATION PURPOSE AND OBJECTIVES

37. The aim of this summative evaluation is to provide feedback to UNICEF and all relevant stakeholders on the relevance, effectiveness, efficiency, impact and sustainability of the “Using textbooks in DAISY format” project implemented in 24 primary schools and the Resource Centre from December 2014 until June 2015 in the Republic of Montenegro.

38. The specific evaluation objectives are the following:

a) Build knowledge for decision making at different levels (strategic, design and management).

b) Define level of accomplishment of expected results and objectives, taking into consideration both positive and negative unintended effects.

c) Provide evidence-based findings, recommendations and lessons learned, aimed at improving the quality of the intervention and defining future programme frameworks and scale-up.

39. The knowledge generated by the evaluation should be used by:

a) Implementing partners – the Resource Centre, Ministry of Education and the Institute for Textbooks and Teaching Aids, as an important source of information for the further scaling up of this initiative – more specifically to identify lessons learned during the implementation, understand the impact of the use of DAISY textbooks in the teaching and learning processes in classrooms, areas that require additional attention, potential barriers and bottlenecks in implementation, etc.;

b) Participating schools and other primary schools to introduce DAISY textbooks in the teaching and learning processes;

f) Associations of parents of children with disabilities and children without disabilities, and parents in general, representing the interest of children to further strengthen their monitoring and advocacy efforts;

c) UNICEF Montenegro Office for future programming and support for strengthening the inclusiveness of education for children with disabilities, support for scaling up of the same initiative and similar initiatives; and

d) UNICEF Headquarter Disability Section for future programming, support and replication of similar initiatives in other countries.

SCOPE OF THE EVALUATION

40. The evaluation covers the entire project implementation period (September 2014 to June 2015) and all the geographical areas where the intervention is taking place in Montenegro: Podgorica, Bijelo Polje, Berane, Pljevlja, Nikšic, Danilovgrad, Tivat, Bar, Kotor and Herceg Novi.

41. The methodology for the evaluation has sought the provision of evaluative insights to provide the basis for evidence-based decision making. The conclusions and recommendations generated have been built upon the information gathered through the evaluation questions and criteria.

42. The evaluation has been conducted based on the following work plan:

Project “Using textbooks in DAISY format” – Draft Evaluation Report 48.

a) Inception phase: The main goal of the inception phase was to produce an evaluation

design adapted to the information needs of the commissioner, allowing a systematic study of all the relevant intervention components (structure, processes and outcomes). Consensus between the evaluation consultant and the organization on the evaluation design was key to ensuring a relevant and feasible study. All documentation generated during the programme cycle was reviewed, and the evaluation strategies, methodologies, techniques and sampling strategies were defined. The main outputs of this stage were a comprehensive evaluation design and the Inception Report.

b) Field phase: Collection of primary data based on the sampling strategy and tools defined during the inception phase.

c) Synthesis phase: All relevant information gathered has been analysed and included in this draft evaluation report, along with the main conclusions and recommendations.

4. EVALUATION METHODOLOGY

43. As indicated in the Inception Report, the following methodological approach has been developed:

(a) An external goal-based evaluation has been performed. Although this summative evaluation model does not necessarily include the study of other aspects of the programme, such as processes and/or resources, they have been included in the scope of the evaluation to allow greater understanding of the intervention, helping to establish the focus of the analysis and, ultimately, defining not only the results generated but also the underlying reasons (how and why) these goals were achieved or not, including lessons learned.

(b) The evaluation is based on a comprehensive and sensitive model, which basically depends on complete understanding of the project to be able to evaluate it. Firstly, the intervention is “photographed” for its comprehension, and then evaluated for its improvement. The evaluation has considered a gender-sensitive approach, keeping this aspect present throughout the evaluation process. Cross-cutting issues concerning human rights have been also considered.

(c) The evaluation has been guided by the criteria proposed in the terms of reference. Nevertheless, a theoretical model of the programme was designed based on the theory of change provided. This approach states that every project is built upon explicit or implicit models on how the intervention will lead to the desired results. To identify these models, a theoretical model of the programme was designed to establish the underlying intervention rationales, the causal linkages between all elements in the programme that produce results. This theoretical model includes all the relevant aspects of the programme: results and objectives, processes and activities undertaken to achieve them and the means and resources that are in place to carry out these activities. The relevance of this approach lies in the usefulness it provides in designing an evaluation, allowing a better understanding of the programme, including the cause–effect links generated at all levels. Additionally, this approach provides opportunities to identify those aspects of the programme that may be key or relevant for inclusion in the evaluation and that are outside of the area of influence of the criteria proposed in the terms of reference (standard DAC criteria).

(d) Once the programme’s theoretical model was designed, UNICEF validated it. During this process, some evaluation questions were modified or new ones included.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 49.

44. At least one indicator has been designed for each evaluation question. The indicator allows measurement of the concept to which the evaluation question refers., Some indicators appear in different evaluation questions, according to their relevance in measuring key concepts. All the proposed indicators in the ToR were studied, and linked to their relevant evaluation question or questions. Additional indicators have been identified and included.

45. This process was summarized in the evaluation matrix, a tool comprising all the key elements in conducting the evaluation (criteria, dimensions, evaluation questions, indicators, techniques to gather data, recipients, etc.). The evaluation matrix can be found in Annex 4.

46. The evaluation applied the following criteria:

a) Relevance: The extent to which the project objectives are consistent with country’s needs, global priorities and policies and beneficiaries’ requirements and UNICEF’s goals. Aspects related to alignment and coherence have been also included;

b) Efficiency: A measure of the achievement of results in relation to the resources used in the search for an optimal allocation thereof, including internal and external coordination;

c) Effectiveness: The degree of achievement of short-term objectives and results through activities. This criterion covers up to the third level of results (see paragraph 20 of this report) defined in the logic model;

d) Impact: The positive and negative medium- and long-term changes produced by the project, directly or indirectly, intended or unintended. This criterion covers from the fourth level up to the sixth level of the logic framework;

e) Sustainability: future viability of the intervention once support is withdrawn. Assessment of the likelihood of continuation of project benefits after the assistance has been completed; and

f) Additionally, cross-cutting issues concerning gender equity and human rights will be taken into account, as well as other aspects and assumptions that could positively and/or negatively influence the programme.

47. The methodological approach included quantitative and qualitative strategies. Taking into consideration the existing time and resource constraints associated with this evaluation, the evaluator conducted in-depth document reviews, focus group consultations and surveys as key methods of data collection. These methods have been complemented with interviews with a limited number of selected strategic stakeholders and direct observation. All these data-gathering techniques combined ensured a reliable source of primary and secondary data from which the evaluative analysis and evidence-based assessment are derived.

48. Interviews and focus groups were conducted based on a set of protocols designed. Four different protocols were produced, one for each type of respondent (implementing partners, teachers and school management, parents and students).

49. A set of four different questionnaires were also produced, one per group of respondents (implementing partners, teachers and school management, parents and students). Some questions were included in all questionnaires (such as the overall level of satisfaction with the project), whereas others only applied to each particular group of respondents.

Table 1: Methods of data collection

Proposed methods

Description Minimum–Maximum Number

Addressee

Project “Using textbooks in DAISY format” – Draft Evaluation Report 50.

In-depth document

review

In-depth review of all documentation related and generated by the project, including monitoring records, reports, etc.

N/A

UNICEF Montenegro

Participant schools

Implementing partners

Thematic focus groups

The evaluator conducted focus group sessions with each thematic group. Focus group protocols were designed to guide the sessions.

A total of 12 focus groups (three per school, in the four

schools selected) were conducted

Education staff

Parents

Students

Individual semi-

structured interviews

The evaluator conducted semi-structured individual interviews with selected stakeholders. The designed interview protocols guided these sessions

A total of 16 interviews were conducted with

selected stakeholders

Implementing partners

Teachers

Survey

The evaluator will complete individual interviews with written inquiries and/or (a) ad-hoc short survey(s) to crosscheck and/or confirm initial findings.

80 questionnaires completed

Education staff

Parents

Implementing Partners

Students

UNICEF

Observation Used as a complementary method of data collection.

N/A N/A

50. Selection of the participating stakeholders was based on purposive sampling, meaning that stakeholders were selected for the following reasons:

a) Schools were selected based on the reported number of CwD participating in activities with DAISY audio-visual textbooks, the number of teachers per school that attended training sessions, the total number of CwD registered at each school and their earlier initiation of activities (introduction of DAISY audio-visual textbooks in the class). According to these criteria, two schools in Podgorica (“Vladimir Nazor” and The Resource Centre), one in Danilovgrad (“Njegos”) and one in Berane (“Vuk Karadzic”) were visited. Three of these schools are included in the “primary focus” group, and the other (Vladimir Nazor) is part of the group of “other schools”.

b) Representatives from all implementing partners were interviewed and invited to complete anonymous surveys.

c) The consultant met the target group, and the direct and indirect beneficiaries during the visits to proposed schools.

Teachers that attended trainings (along with non-participating colleagues) were invited to participate in focus group sessions during the school visits.

Representatives from associations of parents and individual parents that attended training sessions were invited to participate in focus groups, as well as other parents of students with or without disabilities who had/had not attended DAISY classes.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 51.

Students (both children with SEN and children without disabilities) were invited to jointly participate in focus groups during the visits to selected schools.

Only interviewed stakeholders were asked to complete anonymous surveys. Gender balance participation was taken into consideration for both the target group and indirect beneficiaries.

Table 2: Sample stakeholder coverage

Stakeholder group Interview sample Focus group sample Survey sample (completed

questionnaires)

Implementing partners

10 representatives - 6 questionnaires

Donors 1 representative - -

Target group / indirect beneficiaries

- 29 students 23 questionnaires

Direct beneficiaries 4 school directors and 1 assistant

27 parents /27 teachers 51 questionnaires

51. Overall, 16 personal interviews and 12 focus groups were conducted, reaching a total of 99 key stakeholders. Additionally, 80 questionnaires were completed. 13 participants (six students and seven parents) could not complete the questionnaire due to a lack of information and knowledge about the project. The complete list of interviewed people (excluding focus group participants) can be found in Annex 2.

52. Ethical considerations were taken into consideration at all stages in order to ensure the integrity of the evaluation. Independence of judgment, impartiality, honesty, competence, accountability, confidentiality, transparency, accuracy, integrity and the obligations to participants are key concepts that every evaluator is required to commit to when conducting any kind of evaluation, and particularly within the UN system.13 All informants were invited to participate in the evaluation and were requested to provide consent. Participation was voluntary and this report presents the information extracted from those sessions in an anonymous fashion. At the beginning of every session, the evaluator provided information about the evaluation and its objectives, so that every participant could understand the purpose of the exercise. The process ensured that all informants could openly and freely express their views during the field sessions. In this regard, providing anonymous questionnaires to all participants in FG ensured confidentiality. Additionally, FG sessions were not recorded unless all participants provided explicit consent.

Development of recommendations

53. Recommendations were established based on the main findings and conclusions identified. Once the draft evaluation report was completed, it was shared with UNICEF for comments, which resulted in some amendments. Once the final report is ready, it will be translated into Montenegrin

13 UNEG Code of conduct for Evaluation in the UN system, 2008.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 52.

and circulated among key stakeholders (implementing partners, participants, etc.) for revision. This process aims to increase the utility and ownership of the evaluation process.

Limitations of the evaluation

54. The main limitation found for the evaluation is the definition of impact or contribution to the higher levels of outcomes established in the programme’s logic model (Overall objective and specific objectives). As indicated in the inception report, this is due to the fact that the initiative was operating for a limited period of time (less than six months of use of DAISY audio-visual textbooks in classes). In addition, no baseline data was available. This type of data is essential in order to estimate impact even for methodologically weak impact studies such as a before–after approach with no control group. To overcome this difficulty, the evaluation approach to estimate early contribution towards medium- and long-term goals considered only those students that were involved in sessions with DAISY textbooks, and studied the differences between perceptions from CwSEN and children with no special education needs.

55. Limited detailed and exhaustive project documents (logical framework, full list of indicators per activity, monitoring sheets, etc.) can reduce the adequacy of the evaluation design, and as a result may adversely affect the evaluation process. To overcome this disadvantage, the evaluator designed a theoretical model of the programme and a set of indicators to ensure that evaluation questions could be fully addressed. This design was included in the inception report and approved by the ESC, ensuring suitable adaptation to the agency’s requirements and the reality of the intervention.

56. The survey and the evaluator’s field visit were conducted simultaneously. Therefore, some results and findings arising from this technique could not be explored further through other techniques such as interviews and focus groups. To address this issue, information collected during interviews complemented data already gained through questionnaires, since interview protocols were designed taking this aspect into consideration. In any case, after analysis of the completed questionnaires and particularly the open-ended questions, it was clear that a point of saturation in the discourse had been reached.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 53.

5. FINDINGS AND ASSESSMENT

RELEVANCE

57. This section illustrates the extent to which:

(a) The project design and its objectives are in line with national strategies and policies on education;

(b) The project addresses the underlying causes of exclusion of CwSEN in primary schools and responds to their needs;

(c) The design of the project is adequate for the achievement of its objectives in the given period of time;

(d) The project was designed according to international norms on human rights and gender equality and in line with national strategies in those areas; and

(e) Training sessions, materials, guidelines and the DAISY audio-visual textbooks produced are suitable for the intended audience and contribute to the goal achievement.

(i) Is the project in line with national strategies and policies on education?

58. As mentioned in paragraph 5 of this report, Montenegro has invested effort in the past few years to developing a legal and operational framework to provide the grounds for an equitable and accessible education system. The latest Inclusive Education Strategy (2014–2018) comprises the objectives and directions for the education of CwSEN, based on the experiences, analysis and lessons learned from the previous Inclusive Education Strategy (2008–2013). These strategies are in line with best international practices and national and regional experiences.

59. The defined guiding principle of the Inclusive Education Strategy (IES, 2014–2018) is the respect for the rights and features of children with special educational needs whose development and educational capacities need to be fulfilled and met through good-quality education with a view to making them ready for independent living. This guiding principle is built upon the objectives of the previous IES (2008–2013). In this document, the first objective is “to provide equal education possibilities for all children”. This objective directly relates to the final long-term goal established for the DAISY project.

60. In terms of the main tasks included in the IES (2014–2018), three out of five are directly in line with the scope of the DAISY project as follows:

(a) Task 2: Enable students with SEN access to and continuity of good quality education to prepare them for independent life and work;

(b) Task 3: Ensure support for teaching and learning processes at all levels; and

(c) Task 4: Improve pre-service education, specialized training and professional development for teachers and experts.

61. Project activities are in line with the main tasks included in IES (2014–2018), and particularly with Task 4, referring to the improvement of specialized training and professional development

Project “Using textbooks in DAISY format” – Draft Evaluation Report 54.

for teachers and experts. In this regard, the provision of five training sessions in the use of DAISY (technical and pedagogical) is considered highly relevant.

62. Before the project was presented to all implementing partners, the MoE had already ascertained the necessity of adapting teaching materials to the needs of CwSEN, and the initiative was already considered highly relevant. It was UNICEF, based on RC’s proposals, that engaged all the other stakeholders in the project.

63. The DAISY project is the result of continuous work on developing an inclusive education system in Montenegro. Systemic efforts of the Government accompanied by UNICEF’s massive C4D campaign ”It’s About Ability”, has resulted in a five-fold incremental increase in the number of CwD attending mainstream schools in the past five years.14 Given this context, the introduction of DAISY audio-visual textbooks can be understood as the next natural step in this transitional period.

64. The survey reveals that 100% of respondents strongly agree (61%) or agree (39%) that UNICEF’s support through DAISY project is relevant for the achievement of Montenegro’s overall inclusive education goals. This question was included only in the survey for implementing partners and education staff.

(ii) Does the project address the underlying causes of exclusion of CwSEN in primary schools and respond to their needs?

65. The main underlying cause of exclusion for CwSEN identified is the lack of access to the same learning opportunities as their non-disabled peers. DAISY audio-visual textbooks are designed to contribute towards alleviating this issue, and all stakeholders agree that they are a sound complementary educational tool, easy to use and intuitive, that allows all students to learn at the same pace, reduce learning differences among students and improve inclusion by facilitating individual and group learning. DAISY audio-visual textbooks increase attention and concentration, are easy to follow and improve reading skills. Both teachers and students agree that they make the classes more interesting.

66. CwSEN do not only face difficulties in class, but also at home, since they regularly require the support of their parents during the learning process. This is particularly accentuated for sight-impaired students, who usually need their parents’ support when faced with standard textbooks. Allowing self-study for these students is one of the key potentials of DAISY audio-visual textbooks, and both students and parents could benefit from the introduction of DAISY at home.

67. Nevertheless, stakeholders also agree that DAISY audio-visual textbooks should be used as a complementary tool, and sessions should follow a clear teaching strategy, combining different teaching strategies during the sessions (role playing, repetition by heart, etc.).

68. DAISY books have some advantages compared to traditional textbooks and particularly to Braille books, since many textbooks can be stored on a single CD, are easy to transport, do not wear out at the same rate of use and can be used on many different devices, enabling higher accessibility. Books in DAISY format are easy to correct and edit: once an erratum is detected, it is relatively easy to amend the mistake. Misprints in traditional textbooks are amended in new editions, and this process usually requires time. This also applies when a curriculum is revised, usually every three years.

69. Features such as voice repetition, navigation and visual support have been consistently brought up during the evaluation as excellent support when using DAISY audio-visual textbooks.

14 ToR DAISY Project

Project “Using textbooks in DAISY format” – Draft Evaluation Report 55.

70. In financial terms, DAISY audio-visual textbooks are more economical and potentially represent a financial alleviation for parents: they are available for free and can be used on a wide range of devices.

71. Stakeholders have determined the following essential activities of the project:

(a) Training of education staff and parents. They have also raised their concerns and proposals to improve the process, as included in the effectiveness section of this report;

(b) Production of DAISY audio-visual textbooks, comprising both recording and adaptation to DAISY format. More information about the specific issues tha came to light is included in the effectiveness section of this report; and

(c) Monitoring visits, to follow up and support the project implementation.

72. Survey analysis reveals that 97% of the surveyed stakeholders (in this case implementing partners and education staff) strongly agree (50%) or agree (47%) that the project responds to the needs of CwD.

Figure 2: Survey results on DAISY responding to CwSEN needs

Finding 1: The project objectives are in line with national strategies and policies on inclusive education, both in terms of guiding principles and operational tasks. Intervention addresses the main underlying causes of exclusion for CwSEN and has the potential to respond to their needs.

(iii) Is the design of the project adequate for the achievement of its objectives in the given period of time?

Project “Using textbooks in DAISY format” – Draft Evaluation Report 56.

73. The intervention was articulated through a theory of change designed by UNICEF that was developed after the establishment of the project design, with the aim of contributing to its evaluability. The project design and rationale is contained in two documents, a project description and the terms of reference for the evaluation. No logical framework has been designed for the project. Monitoring tools have been used during the implementation period, but a fully fledged monitoring system is unavailable. The underlying logic of the theoretical model of the programme was correct, although it was revised and restructured during the inception phase of the evaluation to express additional aspects and assumptions that were found essential. The project’s main objective was narrowly defined, focusing on improving the quality of teaching but not mentioning the key aspect around which the project should revolve – inclusiveness. The available documentation provides a general overview of the intervention, but specific key information, such as SMART15 indicators per activity and result, is not clearly defined. Although long-term objectives are not completely defined and long-term indicators do not capture the essence of what needs to be measured, this did not theoretically affect the achievement of short- and medium-term goals in the given period of time.

74. The survey reveals that 81% of respondents (implementing partners, education staff and parents) strongly agree (28%) or agree (53%) that the design of the project is adequate to achieve the intended goals. (36 respondents)

(iv) Was the project designed according to international norms on human rights and gender equality and in line with national strategies in those areas?

75. The project is in line with international norms on human rights, specifically the UN Convention of the Rights of Persons with Disabilities (2006, ratified by Montenegro in 2009). In this treaty, State parties commit to ensuring an inclusive education system at all levels, where CwD are not excluded from the general education system and can access inclusive, quality and free primary and secondary education. The treaty also aims to ensure that persons with disabilities receive the support required, within the general education system, to facilitate their effective education.16

76. The Action Plan for Achieving Gender Equality in Montenegro (2013–2017) is the main national document for the implementation of gender equality policy. It is based on international and national legal documents (UN standards for achieving gender equality, EU standards for achieving gender equality, legal acts of both the European Union and the Council of Europe referring to gender equality and equal treatment of women and men, etc.). One of the main strategic goals defined in the plan directly refers to introducing gender-sensitive education at all levels. Among the set of activities defined for this strategic goal, the MoE and the BoE are in charge of conducting analysis of textbooks in primary and secondary schools and take into consideration gender mainstreaming issues. This activity is carried out in collaboration with the ITTA, and includes revision of adapted DAISY audio-visual textbooks. Given the nature of their work, and their alignment and involvement with national policies such as the aforementioned Action Plan, all implementing partners have taken into consideration gender mainstreaming issues when designing the intervention.

77. Implementing partners, education staff and parents responded the following way when asked about the project design:

(a) 89% consider the design of the project in line with IES (2014–2018), 11% N/A (33 respondents).

15 Specific, Measurable, Attainable, Relevant, Time-bound. 16 Convention on the Rights of Persons with Disabilities, (2006)

Project “Using textbooks in DAISY format” – Draft Evaluation Report 57.

(b) 64% strongly agree (22%) or agree (42%) that the design of the project is in line with national strategies on gender equality. 33% answered not applicable or no answer and 3% neither agree nor disagree (36 respondents).

Finding 2: The project documentation only provides a general overview of the intervention. The project design has been better articulated and defined for short-term achievements, so it was suitable for the given period of time. Although long-term objectives are not completely defined and long-term indicators do not capture the essence of what needs to be measured, this did not theoretically affect the achievement of short- and medium-term goals in the given period of time. The current monitoring system lacks specific indicators and benchmarks.

Finding 3: The project design and rationale reflect international and national norms and strategies on human rights and gender equality.

(v) Are training sessions, materials and the DAISY audio-visual textbooks produced suitable for the intended audiences and did they contribute to goal achievement?

Training and materials

78. The training sessions were implemented in an adequate and timely manner. They were focused on the use of DAISY with CwSEN, but the possibilities and benefits of using DAISY audio-visual textbooks with all students were pointed out.

79. Training covered not only technical aspects on how to operate DAISY audio-visual textbooks but also general strategies and pedagogical aspects for their effective use in class. Each trainer covered different themes and developed a specific area on the use of DAISY textbooks, allowing adaptation to the different needs and goals of each session. Among these pedagogical strategies, respondents considered educational activities such as role-playing as very relevant. General strategies for the use of DAISY in class did not include enough aspects on how to use these textbooks in classes where there are children with disabilities in terms of their better inclusion and integration in the class environment and dynamics. General strategies were focused on the role of teachers in the project.

80. Although the relevance of using DAISY audio-visual textbooks with all students was pointed out, it was not sufficiently stressed. Some teachers without CwSEN in their class did not understand the potential benefits of using DAISY audio-visual textbooks, and therefore did not conduct any sessions.

81. The overall level of satisfaction with capacity building activities is very high or high for 70% of respondents (including implementing partners, educational staff and parents). Moderate satisfaction was indicated by 11% of the total sample, and 19% responded N/A (57 respondents).

82. The training materials are suitable for the intended users (especially teachers) and their content is clear and concise. The quality of training content was rated as very high or high by 67% of respondents (includes educational staff, parents and implementing partners). 7% considered the quality of contents moderately satisfactory, 3% low and 23% answered N/A (57 respondents in total).

Produced DAISY audio-visual textbooks

83. Adaptation of standard textbooks to the DAISY format was a challenge for the production team, which managed to produce high-quality DAISY textbooks in spite of technical and financial

Project “Using textbooks in DAISY format” – Draft Evaluation Report 58.

constraints, and lacking previous experience in the matter. The process is time-consuming and requires very specific skills for both actors/actresses and the technical team. Particularly challenging are literature textbooks, where different texts require different intonation and modulation, and the recorders need to act accordingly. In these cases adequate acting ability is also essential to engage the audience. Not only is acting important, the pronunciation also has to be correct so that the final user learns to pronounce words properly.

84. The RC, in collaboration with the ITTA, was in charge of supervising the production and edition of DAISY audio-visual textbooks.

85. Stakeholders identified potential new subjects for future DAISY textbooks:

(a) Foreign languages: the adaptation of these types of textbooks has the great potential to develop good pronunciation and listening skills and to support the parents’ role in the learning process, especially those that do not speak other languages.

(b) History and geography: these subjects should be relatively easy to adapt and have great potential to engage students.

(c) Compulsory literature reading.

86. The production team faced complications when adapting standard textbooks to the DAISY format, and the process was particularly challenging in terms of image and picture adaptation. Moreover, the use of copyrighted materials entailed additional difficulties when adapting original textbooks, as in the particular case of an author who did not find DAISY format suitable for adequately reflecting the original material. In any case, the project successfully overcame all these difficulties, but similar issues could hinder the potential of future adaptations or slow the production rate.

87. Speech and intonation issues have been pointed out by users and implementing partners. Some texts are not adapted correctly, recording is not lively and engaging, and pronunciation mistakes were also detected. A particular exercise (radio role play) was produced in a professional recording studio with professional actors/actresses, and the results stand out when compared to the rest of materials. Sometimes the voices do not correspond to the roles they represent (a woman’s role played by a man’s voice). Additionally, interruptions while reading texts (e.g. turning pages) negatively affect the students’ attention and negatively affect the atmosphere generated.

88. With regards to the usefulness of produced DAISY textbooks, 82% of survey respondents consider the adapted materials very useful or useful (57 respondents from implementing partners, education staff and parents).

89. All surveyed students (23) liked very much (61%) or liked (39%) studying with DAISY audio-visual textbooks.

90. The overall level of satisfaction with the produced DAISY audio-visual textbooks was very high or high for 72% of respondents. 14% of the surveyed stakeholders were moderately satisfied with produced textbooks in the DAISY format, 5% expressed a low level of satisfaction and 9% responded N/A.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 59.

Figure 3: Usefulness of DAISY

audio-visual textbooks

Figure 4: Overall level of satisfaction (totals) by stakeholders

Finding 4: In general terms, training sessions, materials and produced DAISY audio-visual textbooks are suitable for the intended audiences and users. Trainers, trainees and students expressed high levels of satisfaction with the training sessions, materials and produced DAISY textbooks. Training sessions were focused on developing teachers’ skills in introducing DAISY audio-visual textbooks in class. The benefits and adequacy of using DAISY audio-visual textbooks with children with no SEN was not stressed enough during training.

Finding 5: Production issues were detected (inadequate intonation and modulation, voices by role, page information and other interruptions, etc.), lessening the potential impact of the intervention.

0

2

4

6

8

10

12

14

16

2

4

0 0 0 0

4

16

7

10

2

78

12

0

3

IMPLEMENTINGPARTNERS

TEACHERS

PARENTS

Project “Using textbooks in DAISY format” – Draft Evaluation Report 60.

EFFICIENCY

91. This section presents findings based on the timely and efficient utilization of resources for the following concepts:

(a) Implementation management;

(b) Efficient use of financial resources; and

(c) Coordination.

(i) How well was the implementation of activities managed?

92. The project develops three main processes: production of DAISY audio-visual textbooks, training sessions and introduction of material in schools. All activities involved were completed in a timely manner as per the agreed chronogram.

93. Implementing partners developed a monitoring and evaluation system, focusing on training sessions and the effective introduction of DAISY textbooks in schools. Representatives from the BoE prepared reports for each of the training sessions conducted, and trainees had the opportunity to assess the quality of the session via a survey. With regards to the monitoring of sessions with DAISY in schools, teachers received “lesson observation” forms for them to complete after each session. Finally, representatives from the RC and ITTA visited schools, attended sessions with DAISY and produced supervisory reports. All in all, four monitoring tools were available to track performance and to ensure progress was made towards the achievement of results.

94. The evaluation had access to the following documents:

(a) Training reports (5)

(b) Trainee surveys

(c) List of training participants

(d) Supervisory reports (5)

(e) Lesson observation forms (31)

95. Efficiency analysis for monitoring visits is only partially available since no specific plan, indicator or benchmark is included in the monitoring framework. “Lesson observation” forms were provided by 10 out of 24 schools (four of these schools were part of the “primary focus” group and six of them from the “other schools” group). The completion rate of monitoring forms among participant schools is 41.6%. All participants completed training quality surveys.

(ii) How well were the financial resources used?

96. All implementing partners were strongly committed to the project, and contributed to its implementation as described in paragraph 37 of this report. The availability of volunteers in both the production stage and the implementation of training allowed implementing partners to run a highly efficient project. The project excelled in mobilizing all available resources (human and financial) towards the completion of planned activities, while incurring the lowest possible costs.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 61.

97. The total budget (€7,553) was allocated to: personnel (20.1%), materials (5.3%), support costs (13.5%) and implementation of activities (61.1%). The financial reports are clear and soundly structured. Professional rates were in line with UNICEF’s standards and the context. All stakeholders expressed their satisfaction with the cost–activity ratio and agreed that no other alternatives could have led to the same results with lower costs. All activities were carried out according to the defined work plan and budget.

(iii) Was the intervention adequately coordinated?

98. All implementing partners pointed out the high-quality and good coordination between them, as well as their pride at being a part of the project. All of them considered the level of coordination achieved as very high (83.3%) or high (16.6%). The RC, UNICEF or any other partner could not have completed any of the project activities without each other’s support. The Internal Coordination System developed allowed the completion of all activities in a timely and efficient manner.

99. The RC was very proactive since the early stages of the project, not only by bringing the idea to the rest of the stakeholders, but also leading the implementation process. The complexity of the initiative, however, requires strong management and coordination capacities. The rest of partners did not always reciprocate this proactivity. The MoE embraced the initiative, and given its sphere of influence and role at a legislative level, relied on the BoE to support the implementation of the project. The BoE has better capacity and resources, including trained personnel, to provide direct assistance to the intervention.

100. Coordination with other similar interventions was not possible due to the project’s uniqueness. All the internal goals and operating principles of implementing partners are in line with project objectives.

101. The surveyed stakeholders (implementing partners, education staff and parents, 57 in total) agree in 72% of cases that the quality of the implementation is very high or high. A representative of an implementing partner, one parent and five teachers provided “not applicable” responses (12%).

Figure 5: Overall level of satisfaction with implementation quality

Story 1: Coordination Success “Coordination with all partners has been excellent: we have maintained continuous and fluid communication and UNICEF has play a major role in this matter, being the force that brought partners together.” “I am excited at being part of this initiative, what we are doing in Montenegro has not yet been done in any other country. This is a pioneering project and it is very inspiring to see all the partners working together towards the achievement of the intended goals”.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 62.

Finding 6: Implementing partners carried out activities in a highly efficient manner, and no alternatives to deliver the same services incurring in lower costs could be identified. No activities were cancelled or not carried out due to a lack of funds.

EFFECTIVENESS

102. This section details the main findings with regards to the achievement of short-term results included in the logic model (third level).

103. The DAISY project is an innovative and pioneering intervention in Montenegro. The people involved in the initiative do not necessarily have extensive experience in this type of enterprise. However, the high levels of commitment and determination from the involved stakeholders allowed them to overcome the initial difficulties and drawbacks and provide the basis for goal achievement.

(i) Are teachers and parents aware of the benefits of using DAISY textbooks and motivated to work with them?

104. Increasing awareness among teachers and parents on the benefits of using DAISY audio-visual textbooks is the first result in the logic model that aims to initiate the causal chain of effect, leading to the achievement of the rest of goals. Awareness raising and informative activities were conducted and introduced during the capacity-building sessions.

105. Teachers are to some extent informed and aware of the benefits of using DAISY textbooks, but this mainly applies to those who attended training sessions and used them in class, and in particular for language and literature teachers (as the books that have been produced so far are only for those subjects). Although some constraints were identified (see paragraph 123), the teachers that conducted sessions with DAISY are satisfied with the results. Teachers considered CwD as the main beneficiaries of these tools, and that better adaptation during the production process could lead to better engagement among all students.

106. Parents lack information about the project, its objectives and DAISY textbooks. Patterns in respondents’ verbalization show their concern about not being involved and informed about the initiative. Most of the parents got to know about DAISY through their children, from whom they received very positive feedback, but they were lacking specific information about the project. Most of the parents’ negative feedback about the project is explained by a lack of information. In line with this, parents are not aware that textbooks in DAISY format are available on the RC and ITTA websites.

107. Based on the pattern in parents’ verbalization of their knowledge about the project and DAISY textbook, it can be confirmed that they did not experience the benefits of using DAISY textbooks due to a lack of information and engagement. Although some efforts were made at schools to improve parent participation, they were not enough to change parents’ scepticism

Story 2: Information pays off As mentioned by parents in one of the sessions “DAISY should not be used by all students, they get lazy and discouraged from reading standard books, and this is very negative considering the current poor reading skills among students in all grades”. During this particular session, information about the project and DAISY textbooks was provided, and given the previous comment, parents were informed of all features available on the use of these teaching aids, and in particular the possibility of projecting the text and read while listening. After explanations were given, all of them had already changed their views and actually commented about the potential benefits of using DAISY with all students.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 63.

towards the project. Only parents with CwSEN attended sessions with DAISY at schools, clearly showing the existing awareness imbalance between parents with and without CwSEN.

108. Parents (particularly those with CwSEN) identified great potential in DAISY textbooks to enhance students’ individual learning, since they improve the learning process and students require less attention. Parents requested a higher involvement in the project through adapted training, so that they can support their children according to their specific needs.

109. 81% of surveyed parents (21 respondents in total) consider their level of motivation to use DAISY as very high (48%) or high (33%). Moderate motivation is shown by 19% of cases. It is important to point out that seven parents did not fill out the questionnaire since they did not know about the project.

110. A lack of communication or miscommunication has prevented greater promotion of the project through teachers and parents that attended the training (see paragraph 112). Even though some successful cases were reported, the expected multiplicative effect of promoting the project through its direct beneficiaries did not occur.

111. Beneficiaries consider DAISY a great teaching aid tool, improving reading and pronunciation, etc., but not a tool to promote inclusion, the final goal of the project. Direct beneficiaries should be aware that the project does not only intend to improve the learning and teaching processes, but also, and more importantly, to develop a more inclusive education system.

(ii) Project contribution to developing the skills of teachers and parents in the use of DAISY

112. One of the key short-term objectives of the project was to ensure the effective development of direct beneficiaries’ skills in the use of DAISY audio-visual textbooks. Once participants completed the training, they were expected to conduct informative sessions either at schools or at parents’ associations (where applicable). This strategy seeks a multiplicative effect that was partially achieved, as previously mentioned.

113. The results-based framework indicators established a minimum of 110 teachers, five supervisors and 15 parents trained in the use of DAISY audio-visual textbooks. Based on the attendance reports provided, 88 teachers, four supervisors and eight parents completed the sessions.

114. Teachers’ self-assessment shows that 40% consider their skills in the use of DAISY textbooks as very high (10%) or high (30%). 30% consider their skills moderate, 7% low and 23% did not answer the question or indicated that the question did not apply to them (teachers of other subjects for which no DAISY textbook was available). Teachers that attended training gained enough skills to effectively introduce DAISY textbooks in class.

115. Students agree that their teachers have sufficient skills to operate DAISY; it was a process that positively evolved over time. All of the surveyed students strongly agree (48%) or agree (52%) that their teacher knows how to use DAISY.

116. A total of 23 students completed questionnaires, and 79% strongly disagree (35%) or disagree (44%) when asked whether their parents know how to use DAISY textbooks. 4% neither agree nor disagree with the question, and 8% strongly agree (4%) or agree (4%) that their parents know how to operate these teaching tools.

117. Teachers who attended training allowed other teachers to be present during DAISY sessions at school. Some teachers from the RC visited other schools to share their skills, although no reports or evidence were provided during the evaluation.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 64.

Parents’ participation in the training was very limited, and the sessions where not focused on their role in the project, but rather on the teachers’. There is only one case identified of a parent that conducted an informative session on DAISY with other parents after the training, who expressed concerns about the lack of enthusiasm and engagement perceived. Parents, although not highly involved in the project, are grateful for the initiative and keen to participate.

(iii) Accessibility to the produced learning materials to students

118. The DAISY audio-textbooks were available in a timely manner for introduction into schools once the capacity-building sessions were completed. The production team managed to maintain a high production rate, completing a total of six textbooks (literature and language) for grades 4 to 9.

119. The Resource Centre and ITTA provide free access to DAISY textbooks on their website, so anyone can download the materials and reading software for their own use.

120. Students with visual impairment pointed out the need for specific voice control software to control the computer, enabling a better individual experience when using DAISY textbooks at home. The Government launched a programme to refund those families who acquire specific tools such as the aforementioned, but still many families cannot afford the investment. In any case, these students will only need parents’ support to launch the software/application, and once it is loaded, they can easily navigate through the DAISY contents.

121. Teachers from lower grades have conducted sessions with DAISY textbooks even though the materials were not specifically designed for their grades and curricula (3rd grade), broadening the accessibility of DAISY textbooks to students.

122. Students showed interest in using DAISY textbooks at home, but they were not properly informed, i.e. they did not know about the availability of free DAISY textbooks on the RC and ITTA websites. DAISY textbooks are not available in schools’ IT rooms, which would enhance their accessibility.

123. A number of teachers are reluctant to introduce new teaching materials and strategies in class, especially if new skills need to be acquired. Teachers gained enough skills to effectively introduce DAISY in class; the main constrain is their lack of willingness to do so. Older teachers are not willing to change their teaching strategies, and it has been difficult to involve them in the use of DAISY. Additionally, there is a minor concern that DAISY textbooks could replace traditional textbooks in the future. However, younger generations are open to new technologies and that is an opportunity that implementing partners are aware of and are trying to further develop.

124. The lack of adequate technical equipment at schools has been identified as a widespread threat to the effective introduction of DAISY textbooks. This situation discourages teachers from using DAISY. As an example, two teachers had to buy one laptop and speakers for the use of DAISY textbooks at school. Other teachers bring their own laptops since they would not be able to conduct the sessions otherwise, and in some cases also in order to avoid responsibilities with school equipment. The use of visuals during DAISY sessions has been limited, a relevant issue considering the poor reading skills among student identified by stakeholders.

125. The reading software needs a relatively new and updated computer to run all its features. Therefore, many schools experience limitations in being able to simultaneously play audio and video, enlarge fonts, use the “karaoke” feature, etc. Issues concerning the software’s limited ability to run under certain operating systems were also detected. The use of DAISY textbooks in other devices is also limited, and powerful/advanced mobiles or tablets are required to display all

Project “Using textbooks in DAISY format” – Draft Evaluation Report 65.

the major features. DAISY audio devices (specifically designed to run DAISY textbooks) provide a user-friendly experience but are expensive.

126. Adaptation of DAISY audio-visual textbooks relied highly on the volunteers’ (Drama Academy students’) availability to record materials, potentially leading to delays in production. Once the recording was completed, the technical team could finalize the process relatively quickly.

(iv) Project contribution to developing student’s skills on the use of DAISY

127. Although no activity concerning the development of students’ skills on the use of DAISY was originally included in the project design, it was finally identified as relevant for the evaluation process. The reason to include this question lies in the need to identify to what extent students could use DAISY textbooks without the assistance of teachers and parents, to enhance their individual learning.

128. Neither individual nor group sessions on the use of DAISY were conducted at schools, and students expressed their lack of knowledge of operating DAISY; 48% strongly disagree (9%) and disagree (39%) when faced with the sentence “I know how to use DAISY audio-visual textbooks”, whereas 13% of the surveyed students neither agreed nor disagreed and 26% did not know or did not answer the question. Only 13% strongly agreed (9%) or agreed (4%) with the sentence. Nevertheless, students expressed their motivation to learn and study with DAISY textbooks. In general terms, respondents indicated the need for students to further develop their IT skills.

129. The development of students’ skills on the use of DAISY directly relates to the next evaluation question, which tries to identify to what extent students were empowered in individual learning. Teachers verbalized their positive thoughts concerning the capacity of DAISY textbooks to improve individual learning, but also stated that they were introduced only recently and that it takes time for the children to adapt to these materials, so they focused on group learning. However, during group sessions with DAISY in class, individual learning was stimulated in terms of listening to different texts for their analysis. Some children with no special education needs considered that DAISY is not for them, only for CwSEN.

130. DAISY textbooks have not been introduced at home yet. As previously mentioned, this is mainly because parents lack information about the project, but also due to a lack of required technical equipment or internet access. Internet users in Montenegro represent 55.8% of the total population. Computer use in households (49.8% of total households) differs by the type of settlement (59.5% in urban areas and 40.2% in rural areas).17 Based on the aforementioned finding, neither the evolution of students’ need for parental support in the use of DAISY textbooks nor the evolution of individual learning during the project life could be determined. A case was found where a family confirmed the use of DAISY textbooks at home, considering it a very good experience that improves children’s attention and concentration.

Finding 7: A lack of information and awareness on the potential of DAISY textbooks prevent teachers and particularly parents from being more involved in the initiative. Parents requested a higher level of involvement in the project through adapted training sessions, so that they can support their children according to their specific needs. Motivation to use DAISY is higher among parents who know about the project (particularly parents of CwD), but they are not sufficiently involved. Participation of teachers and parents in capacity building sessions has not reached the minimum standards set by the project. Teachers’ skills in the use of DAISY are adequate for the effective introduction of these teaching tools at schools. Conversely, parents lack basic skills in the use of DAISY textbooks.

17 Montenegro Statistical Office N 281

Project “Using textbooks in DAISY format” – Draft Evaluation Report 66.

(v) Project contribution towards better organizing the teaching process for joint learning

131. This objective specifically refers to the ability of teachers to adapt the teaching process to the use of DAISY audio-visual textbooks, and conduct joint learning strategies to ensure that all students participate in the class dynamics.

132. DAISY sessions were conducted in language and literature classes. Teachers selected different poems and classic texts to listen to, and students had to listen carefully as they were asked to analyse and comment on them afterwards. On average, teachers conducted from three to five sessions with DAISY, reporting that students look forward to these sessions, are more concentrated and relaxed, and consequently the learning process is improved.

133. During training, different strategies on how to approach a session with DAISY textbooks were provided. In addition to playing a text or a poem for analysis and comment, strategies such as role playing (where students are involved in the text by playing different roles), uncommon word analysis and group analysis proved to be very effective in stimulating students’ engagement.

134. Teachers reported that DAISY is particularly effective when used with sight-impaired students, especially for those teachers without sufficient knowledge of the Braille alphabet. The audio-visual textbooks allow them to better organize classes with these students. However, all teachers agree that Braille books are irreplaceable and that DAISY textbooks should be only used as a complementary tool. Braille is the standard tactile writing and reading system that every sight-impaired student learns during the first years at school, and teachers consider Braille books as key tools not only for learning different subjects but also for the students to become familiar with the system, which they will constantly use in the future.

135. Information from supervisory activities does not provide a clear pattern on the project’s effective contribution to better organize of the class. On one hand, it has been reported that teachers are effectively adapting classes for the use of DAISY, involving all students in the process. On the other hand, informants mentioned that teachers need more time to adapt to these new devices, since CwD have not been sufficiently involved in the process.

(vi) Unintended effects (positive and negative)

136. No major unintended effects were detected or experienced as a result of the intervention.

Finding 8: DAISY textbooks were produced in a timely manner and are currently available on the RC and ITTA websites. Nonetheless, students and parents are not informed about the availability of DAISY textbooks. Students do not know how to use DAISY textbooks, but are motivated to learn and use them at home. DAISY textbooks are not used at home yet.

Finding 9: A lack of technical equipment at schools has been identified as a widespread threat to the effective introduction of DAISY textbooks. The reading software needs a relatively new and updated computer to run all its features, and that could hinder its effective introduction in schools.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 67.

IMPACT

137. This section presents the findings with regards to the achievement of the medium- and long-term objectives included in the logic model (fourth to sixth level). As predicted and included in the inception report, the project is currently at too early a stage to produce the changes in attitudes and/or behaviour required to comply with medium and long-term goals. In addition, the lack of baseline data and a control group prevents a methodologically robust impact study. In any case, the following findings represent early contributions and tendencies identified towards the achievement of the aforementioned objectives.

(i) Project contribution towards improving learning group dynamics

138. The education staff that conducted sessions with DAISY agree that these activities stimulate group dynamics, creativity, attention and participation. The sessions generate a pleasant and relaxing learning atmosphere. The evaluator could not establish an evolution on the improvement of group activities based on the monitoring records, but only identify the aforementioned positive effects.

139. Students verbalized their higher participation in class during sessions with DAISY, as indicated in Figure 6. It has to be noticed that 23 students answered this question, both students with and without SEN. An interesting pattern emerged from the results of this question – the fact that 87.5% of CwSEN strongly disagree with the sentence “I participate more in class when not using DAISY”, and the remaining 12.5% disagree. This fact could be considered as an early indication that CwSEN have been actually participating more in class since the introduction of DAISY compared to their peers.

Figure 6: Level of participation in class when using DAISY textbooks

(ii) The project’s contribution to changing students’ attitudes toward school and the learning process.

140. When faced with this question, education staff agree that the project is still at too early a stage to be able to produce such changes in students. However, both teachers and parents confirmed students’ positive attitude towards the use of DAISY audio-visual textbooks. In this regard, the survey addressed students’ emotional satisfaction with the use of DAISY textbooks.

141. All of the students strongly agree (61%) or agree (39%) they like studying with DAISY textbooks. Disaggregated information shows that in the case of CwSEN, the percentage of students that strongly agree with the sentence “I like studying with DAISY textbooks” rises to 88%. Additionally, students verbalized how useful DAISY textbooks are for the learning process, and

44% 26% 13% 13% 4%

"I participate more in class when NOT using DAISY"

Highly disagree Disagree Neither agree nor disagree Agree Strongly agree

Project “Using textbooks in DAISY format” – Draft Evaluation Report 68.

explained that this is one of the aspects of why they like these learning tools. Finally, attitudes towards school were found to be very positive, but there are no grounds to confirm that the intervention contributed to this situation.

142. When referring to the project’s contribution in actively engaging students in their own learning, information extracted from the monitoring sheets reflects how DAISY sessions produce a positive effect on students’ individual learning. Activities such as learning a role by heart improve the capacity of the students to memorize and understand texts; they follow the play and the responses of all other characters, which makes it easier and faster to learn the part.

(iii) Project contribution to improving the confidence of CwSEN

143. The evaluation could not establish solid grounds to provide evidence on the project’s contribution to improving CwSEN’s confidence. Only early evidence of positive effects can be provided in this evaluation. Based on the findings expressed in paragraph 136, it could be established that higher participation in class when using DAISY textbooks could be also interpreted as an increase in students’ confidence. Additionally, students prefer group activities with DAISY in 53% of cases, while 26% either did not respond to the question or gave no clear indication (neither agreed nor disagreed).

(iv) Project contribution to better integration of CwSEN in the teaching and learning process

144. The evaluation could not establish clear evidence of the project’s contribution towards better integration of CwSEN in the teaching and learning process. This objective implies behavioural and attitude changes that cannot emerge in the short term, especially in the education sector and in particular given Montenegro’s current context. The country has experienced a significant increase in the number of CwSEN enrolling in mainstream schools, and teachers face difficulties in adapting to this new situation. The DAISY project has the potential to support this transitional period, but it is still at too early an implementation stage to produce this type of result. The evaluation can only provide some indications about the stated objective, as follows:

(a) All surveyed CwSEN strongly agree that the other students in school accept them as they are;

(b) 89% of CwSEN strongly agree (55.5%) or agree (33.3%) that the other children at school help them when required; and

(c) All surveyed CwSEN strongly agree they like to participate in class, and 55% of students strongly agree (44%) or agree (11%) that their teachers help them during DAISY sessions (33% did not respond to this question).

Finding 10: Early evidence of DAISY sessions improving group dynamics and CwSEN participation in class identified. Students have a positive attitude towards the use of DAISY textbooks. The project’s contribution to improving CwSEN’s confidence could not be clearly established, but findings indicate a potential positive effect that can be attributed to the intervention. No evidence is available of the project’s contribution to better integration of CwSEN in the teaching and learning processes.

(v) The project’s contribution to developing an improved teaching process

Project “Using textbooks in DAISY format” – Draft Evaluation Report 69.

145. This objective addresses the extent to which teachers have been empowered not only in developing their skills on how to operate DAISY textbooks (effectiveness, section II) but also in developing general inclusive strategies that contribute to improve the teaching process.

146. As already mentioned in paragraph 76, training sessions included different strategies on the use of DAISY, but the perception is that the sessions did not focus enough on developing teachers’ skills in specific issues about inclusiveness and providing practical guidelines on how to conduct inclusive DAISY sessions. Generally speaking, teachers are aware and familiar with general inclusive teaching strategies. There are many existing accredited training courses for teachers in this subject and the BoE provides certified courses. The ITTA also produces manuals in inclusive education that are accessible to education staff. However, the recent increase in the number of CwSEN in mainstream schools has evidenced that education staff lack practical experience conducting classes with an inclusive focus. This situation is aggravated in the specific cases of sight-impaired and autistic students.

Finding 11: Education staff are aware of general strategies on inclusive teaching, but lack the specific skills and practice in how to effectively introduce these concepts during sessions with DAISY. The use of DAISY should be combined with traditional activities during each session and long listening periods avoided to produce a more dynamic session, otherwise it is difficult to keep students’ attention.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 70.

SUSTAINABILITY

147. This section illustrates the main findings concerning partners’ viability and capacity to maintain the introduced activities and achieved results over time.

(i) Establishment of a clear and appropriate sustainability and exit strategy

148. Implementing stakeholders shared a strong level of ownership of the initiative, and all have agreed to continue with the project even if the source of funding changes. If current funds were to be withdrawn, the MoE and the rest of partners would look for other partners and financial sources to continue developing this intervention. In any case, implementing partners do not recognize only UNICEF’s financial support, but also its involvement and the credibility the agency provides. The perception of being part of a unique and relevant project ensures that implementing partners’ efforts will be focused on the continuation of activities.

149. No official meetings to discuss sustainability or exit strategies were conducted, and no agreements were reached on this matter, therefore there is no document establishing sustainability strategies for the coming years. Nevertheless, they all share the idea of continuation, scaling up to an eventual nationwide expansion.

(ii) Sustainability of project’s main activities

150. Revision of training sessions and material is to be done every year, based on quality surveys, monitoring sheets and supervisory reports. All partners agree that this is necessary to ensure that activities and produced material are in line with project objectives. The Government, and particularly the BoE, plays a key role in this matter, since it has the capacity to conduct continuous monitoring activities.

151. The sustainability of DAISY production at the same rate is not guaranteed, particularly when it comes to recording. The Drama Academy made a major effort to ensure the production of the first six DAISY textbooks, but this production rate is not sustainable over time. They estimate that in the future they will be only able to produce one DAISY textbook per year. There are different reasons that explain this situation. First of all, due to a lack of students experienced in the recording of DAISY. The speech subject represents only 15% of the total first school year in the academy, and volunteers want to further develop other skills after their time volunteering for the project. By the time the students are sufficiently experienced in the recording process, most of them decide to explore other ways of improving their acting skills. They lose motivation, especially considering the commitment and time the recording process requires. New students from the first academic year might join, but the experience and know-how of their predecessors is lost. The students involved in the project receive credit from UNICEF and the Drama Academy and they are proud to be helping in the production of materials that will eventually help other students, particularly CwSEN. The emotional reward that community work provides and the feeling of being part of the development of a better education system is a great incentive. This has to some extent ensured motivation was not an issue, but it is not a sustainable approach.

152. The Bureau of Education has requested official accreditation for training in DAISY, so it would be considered as another accredited training from the Government (MoE), reaching a higher number of direct beneficiaries and eventually ensuring the availability of these capacity-building sessions in the future.

(iii) The extent to which knowledge and skills will be integrated in schools

Project “Using textbooks in DAISY format” – Draft Evaluation Report 71.

153. Education staff fully agree they will keep conducting DAISY sessions even if no more textbooks are produced and the project does not ensure its sustainability. Teachers that are informed about the project and have experienced the benefits of these textbooks want to continue using them.

Finding 12: No formal sustainability or exit strategies have been agreed among partners, but they all share their commitment and willingness to continue with project activities. Education staff that are informed and have conducted sessions with DAISY will continue with the initiative even without the project support. The sustainability of DAISY audio-visual textbook production at the same rate is not ensured.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 72.

CROSS-CUTTING ISSUES

154. This section illustrates the main findings concerning cross-cutting issues taken into consideration during the evaluation.

(i) Equality focus and gender mainstreaming in the project

155. None of the informants identified any constraints or difficulties in terms of equal access to the project and produced materials for boys and girls. No gender inequalities were identified for students in accessing DAISY audio-visual textbooks.

156. The emotional satisfaction at the use of DAISY textbooks expressed by students shows no major deviations when disaggregated by gender. Similar results were obtained when studying the expressed level of participation in class, the level of acceptance and positive attitudes towards schools.

157. A gender-imbalanced participation of parents during focus groups was identified, with a larger number of mothers joining the sessions.

RELEVANCE

Conclusion 1: The project rationale and objectives are relevant to national priorities and policies on inclusive education, and suited to addressing the main causes of exclusion for CwSEN. Human rights and gender equality issues are adequately included in the project design based on international and national norms and standards. (Linked to Findings 1 and 3)

Conclusion 2: Despite the fact that the project design leads to the achievement of results during the implementation period, the current results-based framework does not effectively reflect the expected medium- and long-tem objectives, and lacks specific indicators and standards whereby to measure their level of completion, and that will negatively affect the project in the future. (Linked to Finding 2)

EFFICIENCY

Conclusion 3: The intervention was implemented in a highly efficient manner. Implementing partners managed to carry out all the planned activities with a very limited budget. (Linked to Finding 6)

EFFECTIVENESS

Conclusion 4: Training was very relevant for teachers, since the sessions particularly focused on the satisfactory introduction of DAISY audio-visual textbooks in class. The project relied on schools to actively inform and motivate parents to join training, but this was not effectively conducted. A wider approach, putting more effort into directly addressing the need to involve all potential users and beneficiaries of the intervention (especially parents and students) do not only positively affect involvement of indirect beneficiaries, it also provides the basis to develop a more inclusive educational environment. (Linked to Finding 4)

Conclusion 5: Promoting the project is essential in ensuring engagement of the direct beneficiaries and students. This is the very first step towards goal achievement, affecting the performance of results at higher levels. Information-sharing and awareness-raising activities are essential, since the majority of direct beneficiaries, target groups and indirect beneficiaries expressed their motivation to use DAISY audio-visual textbooks, but were not adequately informed. Information turns scepticism towards the project into willingness to be engaged, and that is something the project has to develop further. (Linked to Findings 7 and 8)

Conclusion 6: The lack of technical equipment at schools and DAISY’s software requirements are a threat to the effective introduction of DAISY in schools and at home. Production issues hinder the potential impact of the project. (Linked to Findings 5 and 9)

6. CONCLUSIONS

Project “Using textbooks in DAISY format” – Draft Evaluation Report 74.

IMPACT

Conclusion 7: The evaluation could not establish clear and solid evidence of the project’s contribution to its medium- and long-term objectives. The findings only provide grounds for determining early evidence of contribution or positive tendencies. DAISY has the potential to support teachers in the process of developing and improving the inclusive teaching process, so that all students can benefit from it. Specific training on inclusive strategies is required, as well as the inclusion of a practical approach on how to effectively introduce DAISY in class. (Linked to Findings 10 and 11)

SUSTAINABILITY

Conclusion 8: Implementing partners developed a level of ownership and responsibility towards the achievement of results that will potentially ensure the continuation of activities even in case of an eventual withdrawal of external funding. Nevertheless, a clear sustainability and exit strategy is required to ensure project autonomy in case external financial support is withdrawn. The current production rate is not sustainable in its present form. Education staff familiar with DAISY audio-visual textbooks are willing to keep using them in their schools. (Linked to Finding 12)

Project “Using textbooks in DAISY format” – Draft Evaluation Report 75.

3. Access to educational needs for CwSEN and gender mainstreaming

Students with special education needs face difficulties and constrains to accessing their education needs in mainstream schools in Montenegro. Moreover, the evaluation process identified that differences in equal access to the project and its benefits are issues that also affect children based on their level of disability. When conducting the analysis of gender mainstreaming issues related to the project, triangulation of information obtained from different stakeholders showed that the unbalanced access to educational needs in mainstream schools is based on the students’ degree of disability, but is not due to their gender. No limitations in accessing those rights could be attributed to gender inequalities. This lesson should be taken into consideration when developing scaling-up strategies and studying other contexts and situations. Other potential constraints should be also studied, especially those that could affect marginalized groups such as: ethnic minorities or children in vulnerable families.

4. Stakeholder engagement and efficiency levels Several difficulties are encountered when introducing pilot projects in any kind of context. The lack of previous experience and knowledge about the intervention are factors that can seriously hinder the involvement of stakeholders at all levels – an essential condition to ensuring an effective and sustainable intervention. This affects not only the target group and indirect beneficiaries, but also the implementing partners. The level of commitment among implementing partners may vary, especially if an intervention is based on voluntary support, as is the case in the DAISY project. This situation has enabled a highly efficient management of resources in achieving results. The downside of this approach is the difficulty of securing stakeholders’ commitment in the medium and long term. Implementers should be open to sacrificing efficiency levels to some extent in order to ensure the full commitment and availability of all relevant stakeholders, especially those involved in the production process.

7. LESSONS LEARNED

8. RECOMMENDATIONS

RECOMMENDATIONS Responsible stakeholder

Deadline Priority

Recommendation 1 Implementing partners should develop a fully fledged results-based framework and monitoring system, clearly stating medium- and long-terms goals, SMART indicators for each activity and result and the monitoring strategy to track goal achievement and potential deviations. (Linked to Conclusion 2 and Finding 2)

Implementing partners (led by

UNICEF)

December 2015

High

Recommendation 2 Implementing partners should take into consideration the following proposals for future training and DAISY textbook production: a) Develop specific training for parents, focusing on their particular role in supporting the process of learning.

b) Training sessions and materials should highlight the adequacy and relevance of using DAISY textbooks with all

students, not only CwSEN, and include a practical session on how to conduct a class with DAISY textbooks with a focus on inclusiveness. IT teachers should participate in the training sessions, so that the rest of the teachers can rely on them if help is required in schools.

c) Produce a manual on the use of DAISY based on the training experiences and lessons learned, including best

practices. It should describe how to work with all types of children, including specific strategies for different types of disabilities.

d) To the extent possible, the project should address the identified production issues (inadequate intonation and

modulation, voices by role, page information and other interruptions, etc.). Implementing partners should include these items in the monitoring system (supervisory reports and lesson observation forms) to ensure the timely identification of adaptation issues and their correction, and develop a clear quality assurance plan and mechanisms.

e) Concerning copyright issues, a standard procedure to engage authors should be developed. This could take

the form of legal agreements signed by both parties and in which the project ensures faithful adaptation of the original material. The project could also consider joining WIPO’s (World Intellectual Property Organization) TIGAR (Trusted Intermediary Global Accessible Resources) service, which allows participating institutions to search internationally for books in an accessible format, and to exchange them. Participation in the service is free, and supported by, among others, the DAISY consortium. By joining the service, the project would have access to 286,000 titles in 55 different languages, which would for example be very relevant for foreign language classes.

Implementing partners

(led by RC)

December 2015

Very High

Project “Using textbooks in DAISY format” – Draft Evaluation Report 77.

f) The intervention should ensure adequate equipment is available at participating schools for the effective

introduction of DAISY audio-visual textbooks. This could be done through monitoring activities and IT support, especially when it comes to software requirements. IT teachers at every school should provide regular computer maintenance to ensure that DAISY software smoothly runs its basic features on those computers. (Linked to Conclusions 4 and 6, and Findings 4, 5, 9 and 11)

Recommendation 3 Implementing partners should extensively conduct promotion and awareness-raising activities in the areas of intervention, addressing education staff, associations of parents, other families and students (especially children with no special education needs). These activities should focus on promoting the project, its objectives (short-, medium- and long-term), conduct practical sessions with DAISY audio-visual textbooks and inform about their availability. Promotional activities could be strengthened using specific tools such as flyers, leaflets, videos and promotional CDs comprising all DAISY textbooks and reading software. Stronger partnership with associations of parents would enhance the scope of the project and its potential. The MoE could play a major role in the promotion of the DAISY project given its credibility as a governmental body. In addition, the school management is also a key stakeholder in this matter, promoting the project within schools and among parents. Finally, the RC could also play a major role on promoting the use of DAISY textbooks among children and parents. (Linked to Conclusion 5 and Findings 7 and 8)

Implementing partners (led by

UNICEF, MoE and RC)

Before the next

school year (Sept

2015)

Very high

Recommendation 4 The definition of a baseline would provide extremely relevant information for potential future impact evaluations. Implementing partners could also consider the idea of defining control-group schools to conduct this type of evaluation. (Linked to Conclusions 2 and 7, and Findings 2 and 10)

Implementing partners (led by

UNICEF)

December 2015

High

Recommendation 5 Implementing partners should define a clear and realistic sustainability and exit strategy, including shared and individual responsibilities, continuity plans, benchmarks, funding opportunities and timelines. These documents should be periodically reviewed and adapted to the political and economic context that might affect the project. The sustainability and exit strategy should be integrated into all aspects of programming, aimed at ensuring sustainable outcomes. In line with this, implementing partners should define specific action strategies for the production of DAISY audio-visual textbooks, either by reducing the production rate or by providing additional support (professional or voluntary, considering some monetary compensation) for the production process. (Linked to Conclusion 8 and Findings 5 and 12)

Implementing partners (led by UNICEF)

2016 High

9. ANNEXES

Annex 1. Terms of Reference

Annex 2. List of interviewed stakeholders

Annex 3. List of documents reviewed

Annex 4. Evaluation Matrix

Annex 5. List of participating schools and disaggregated number of students

Annex 6. Questionnaires

Project “Using textbooks in DAISY format” – Draft Evaluation Report 79.

ANNEX 1 – Terms of Reference

Consultant to conduct an evaluation of the project “USING TEXTBOOKS IN DAISY FORMAT – Primary school teacher training in using audio textbooks

1. Context Since becoming an independent state in 2006, Montenegro has officially joined the international community and in doing so a new blueprint has emerged for nation building. On 28 June 2006, it became the 192nd Member State of the United Nations, and on 11 May 2007, the 47th Member State of the Council of Europe. Montenegro presented its official application for membership of the European Union in December 2008 and two years later it became a candidate country for EU membership. Like its neighbouring countries, it has a rich cultural heritage, a diverse history and is well poised to enjoy integration with the European community. Montenegro belongs to a group of upper middle-income countries with a per-capita GDP of €5,063 in 2012. Montenegro experienced rapid economic growth ranging between 8.6 and 10.7 percent of GDP in 2006 and 2007 respectively. However, as a result of the global economic crisis that hit the country in the second half of 2008, the real GDP growth rate saw a steep decline. After slow and modest economic recovery in the period 2010–2011, the Montenegrin economy went back into recession in 2012 with a –2.5% real growth rate of GDP (MONSTAT, 2013). According to the latest estimates, the real growth rate of GDP in 2013 will be positive but modest (1.8%) (World Bank, December 2013). The global economic crisis hit the economy of Montenegro hard, having a prolonged negative impact on well-being of citizens. The economic recession resulted in a further increase of the poverty rate in the country from 9.3% in 2011 to 11.3% in 2012. There are also significant spatial disparities in terms of poverty that represent a serious impediment to the fulfilment of human and children’s rights. In 2012, the poverty rate in the northern region was 18.3%, which was much higher than in the central region (7.9%) and southern region (9%) (MONSTAT, 2013). According to the Child Poverty Study in Montenegro (UNICEF, 2012), child poverty is concentrated in rural areas and in the north of the country with more than three-quarters of all poor children living in rural areas, and more than half of them living in the north of the country. The delayed transition of Montenegro has had an adverse impact on the education sector which remains disconnected from the latest thinking and innovations, as well as from the labour market’s needs – PISA results show that Montenegrin students lag behind their OECD peers by two years in terms of the skills and competencies necessary for their future education and success in life and employment (50% of Montenegrin students score below the basic functional literacy levels in mathematics, science and reading). The compromised quality of education also questions the quality of inclusion of children with disabilities (CwD). It must be emphasized though that the most visible results of the education reform are those related to inclusion of CwD. So far, the Government has adopted two Inclusive Education Strategies for the period 2008–2012 and 2014–2018 that are in line with the best international practices (EU countries, above all Finland, then Great Britain, Israel, etc.) and rich national and regional experiences. The leading principles guiding these documents are good quality, accessible education for children with special education needs, in accordance with their interests, abilities and needs. Systemic efforts of the Government accompanied by UNICEF’s massive C4D campaign “It’s About Ability”, resulted in a five-fold incremental increase in the number of CwD attending mainstream schools in the past five years. However, CwD still do not have the same learning opportunities. It must be noted that there is no verified evidence of the noted inequalities affecting girls more severely than boys or vice versa. The Evaluation of Education Reform in Montenegro conducted in 2012 emphasizes that, in terms

Project “Using textbooks in DAISY format” – Draft Evaluation Report 80.

of the legal framework and education policies concerning the establishment of an inclusive education system, Montenegro has made significant improvements, and a systemic environment for the effective inclusion of children with special needs has been created. However, there is a consensus between decision makers, education professionals, teachers and parents that significant challenges remain when it comes to consistent implementation of adopted policies, specifically in relation to: enhancement of teacher training in this area (pre- and in-service), appropriate allocation of resources for the implementation of inclusion, inter-sectoral collaboration and activities, teaching material adapted to the needs of CwDs, etc. In addition, there are no reliable estimates on the prevalence of child disability due to data gaps in all sectors. The child disability rate in the 2011 Census was 1.1%, while MICS 2005 pointed to a suspected prevalence of as much as 12.5% for the 2–9 year age group. According to MONSTAT data from 2013, 1,399 children were recipients of child disability benefits, while the number of children with disabilities enrolled in primary schools in April 2014 was 1,350. With the aim of providing accessible and adapted teaching and learning materials for CwD, in June 2013 UNICEF Montenegro Office, the Ministry of Education, the Textbook Publishing Agency and the Resource Centre for CwD initiated the first-ever production of audio textbooks in DAISY format for children with visual impairments and children with learning difficulties. The initiative was implemented with the generous contribution of the Drama Academy, whose students recorded the books on a voluntary basis. Six audio textbooks/primary school readers in the mother tongue have been produced so far and approved by the National Education Council. This initiative is ongoing and the production team at the Resources Centre is continuing to work on making more textbooks accessible in DAISY audio format. However, further challenges are linked to scaling up the efforts, regular updates of the material and ensuring their effective use in classrooms. 2. Project to be evaluated: “USING TEXTBOOKS IN DAISY FORMAT – Primary school teacher training in using audio textbooks” As a continuation of the initiative started in June 2013, the project “USING TEXTBOOKS IN DAISY FORMAT – Primary school teacher training in using audio textbooks” aims to ensure that DAISY audio textbooks are effectively used in classrooms and that both children with and without disabilities are benefiting from an improved teaching and learning environment. The project is being implemented by The Resource Centre for Children and Youth “Podgorica″ (Resource Centre) in close cooperation with the Ministry of Education, Textbook Publishing Agency and Bureau for Education, with financial support from UNICEF Montenegro. The total budget for this project is €7,553.00 (UNICEF’s contribution: €6,563.00, Resource Centre’s contribution: €990.00). The project timeframe covers the period from September 2014 to June 2015. Overall project objective: Train teachers and parents to ensure quality teaching and application of reliable and modern learning methodologies for children with disabilities, as well as for all other children in 24 primary schools and the Resource Centre in Montenegro. Specific objectives:

- informing teachers and parents about how innovative methods and modern technologies can be used to support the teaching process;

- training teachers and parents in how to use DAISY audio books and provide learner support for students/children;

- developing the capacity of teachers to use audio books in such a way as to support the quality and objectives of the teaching process;

- using audio books in the classroom, monitoring/testing students’ ability to listen attentively and accurately present the content to others;

Project “Using textbooks in DAISY format” – Draft Evaluation Report 81.

- helping students to acquire knowledge from DAISY books as a modern teaching aid; - making an inventory of good practices, identifying lessons learned in using this book

format for future improvements in the teaching process; - developing the training programme “Using Audio Textbooks in DAISY Format” and having

it accredited by the Bureau for Education The rights holders of the project “USING TEXTBOOKS IN DAISY FORMAT – Primary school teacher training in using audio textbooks” are children with and without disabilities attending inclusive primary schools and their parents. The project was managed by the Editorial Team composed of representatives of the Resource Centre, Ministry of education, Textbook Publishing Agency and Bureau for Education. The Theory of Change (ToC) for the Project was developed after the inception of the project with the aim of contributing to its evaluability and better defining expected outputs, outcomes and impact. The ToC, as well as the Project Document, can be accessed here:

Theory of

Change.v3.docx

TT Audio Textbooks

English.doc

3. Rationale for the Evaluation Montenegro is the first country in the world to produce a set of DAISY audio textbooks for inclusive education. Similar efforts have been made in other countries, but not led by the main educational institutions seeking to make the use of the audio textbooks part of the regular teaching material and introduce them in all primary schools in the country. Also, this is the first time that DAISY audio textbooks will be made accessible and downloadable to everyone. As such, this project has an enormous potential for knowledge generation and compilation of lessons learned to be replicated elsewhere. UNICEF Headquarters’ Disability Section has recognized this potential and made available additional funds for evaluating this Project. The knowledge generated by the evaluation should be used by:

Project Implementing Institutions – Resource Centre, Ministry of Education, Textbook Publishing Agency, as an important source of information for the further scaling-up of this initiative – more specifically to identify lessons learned during the implementation, understand the impact of the use of DAISY textbooks in the teaching and learning processes in classrooms, areas that require additional attention, potential barriers and bottlenecks in implementation, etc.;

Other primary schools in the introduction of DAISY textbooks in the teaching and learning process;

Associations of Parents of Children with Disabilities and Children without Disabilities representing the interests of children to further strengthen their monitoring and advocacy efforts;

UNICEF Montenegro Office for future programming and support for strengthening the inclusiveness of education for children with disabilities, support for scaling-up of the same initiative and similar initiatives;

UNICEF Headquarters’ Disability Section – for future programming, support and replication of similar initiatives in other countries.

Given that this is the first time that the DAISY textbooks are being used in teaching and learning in primary schools, the recommendations will help UNICEF to ensure provision of the most

Project “Using textbooks in DAISY format” – Draft Evaluation Report 82.

efficient and effective assistance to education institutions in Montenegro in defining the future direction of the reform process with a focus on inclusiveness in education. 4. Objectives of the Evaluation Given that this is the first time that the DAISY textbooks are being used in teaching and learning in primary schools, the main immediate purpose of this summative evaluation is to evaluate the final (end) results and achievements of the project in relation to the project theory of change and indicators from the project document. More specifically the evaluation objectives are to:

1. Provide feedback to the UNICEF Montenegro office and its national counterparts on the soundness (defined as relevance, effectiveness, efficiency and sustainability) and the impact of the project approach in introducing the DAISY textbooks in 24 primary schools and the Resource Centre for the benefit of improved teaching and learning of CwD and children without disabilities, with the aim of:

a. Revealing good practices and gaps in approaches, b. Evaluating the Project Impact following the Project Document and Theory of

Change. 2. Based on the experience from the Project implementation, extract general lessons learned

and recommendations aimed at further enhancement of the initiative; 3. Provide UNICEF Headquarters’ Disability Section with information on the impact of the

project and the potential for replication and scaling up.

5. Scope of the Final Evaluation The project evaluation should cover the entire project implementation period (September 2014–June 2015). The geographical coverage is the 10 municipalities where the initiative is being implemented: Podgorica, Bijelo Polje, Berane, Pljevlja, Nikšic, Danilovgrad, Tivat, Bar, Kotor and Herceg Novi (that is, 24 primary schools from these municipalities and the Resource Centre). Project monitoring data as well as other data sources that should inform the evaluation are listed within section 7 on Methodology and will enable assessment of the project’s achievements. The Project Summary is portrayed through the Theory of Change table (attached above). This Theory of Change table should be used as the main reference point – together with the project document, because it captures the activities undertaken, the expected changes they were to produce and the intended impact. There are no major limitations anticipated in relation to this evaluation. However, given that at the time the Project Document was being developed, there were no plans to conduct a fully fledged evaluation of this initiative, the document does not contain an exhaustive list of indicators or the log frame. To overcome this, a ToC has been developed to guide the evaluation process and allow for development of appropriate indicators to measure and establish the evidence of impact. This is especially pertinent, given that the project implementation period is fairly short and that the evaluation will be conducted towards the end of the project. 6. Final Evaluation Questions

Project “Using textbooks in DAISY format” – Draft Evaluation Report 83.

The focus of the evaluation is articulated under five main evaluation criteria (relevance, effectiveness, efficiency, impact and sustainability)18, each accompanied by guiding questions for the evaluation. These are not exhaustive and should be clarified at the start of the work when developing the evaluation framework: Assessing relevance / To what extent is the project responding to the needs of stakeholders and beneficiaries?

To what extent does the project address the underlying causes of exclusion of CwD from the education system and respond to the needs of CwD in inclusive primary schools?

To what extent is the project design relevant vis-à-vis the overall project goal and the achievement of its objectives in the given period of time?

To what extent are the project design and its objectives relevant vis-à-vis national policies and strategies?

Was the project designed according to international norms and agreements on human rights (HR) and gender equality (GE) and in line with national strategies to advance HR & GE?

Assessing effectiveness / To what extent does the project meet the outcomes as defined by the project log-frame and the Theory of Change?

To what extent have the planned results (quantitative and qualitative) been achieved to date?

To what extent and how has the project contributed to enabling teachers to use the DAISY textbooks in the teaching and learning processes?

To what extent has the project contributed to strengthening the skills of parents to better support their children in the learning process?

To what extent has the project enabled better accessibility of the learning material to children with disabilities and learning difficulties?

To what extent has the project contributed to better individual and group learning of children with and without disabilities in selected schools?

What were the major factors influencing the achievement or non-achievement of the project objectives to date?

Has the project provided any additional (i.e. not directly planned by the project) significant contributions/outcomes?

Assessing efficiency / To what extent did the management of the project ensure timelines and efficient utilization of resources?

How well was the implementation of activities managed? To what extent were activities implemented as scheduled? What management and monitoring tools were used?

How well were the financial resources used? Were funds managed in a cost-effective manner? What is the correlation between funds utilized and outputs or results achieved? Could the same results be achieved with fewer resources?

Did the project ensure co-ordination with other similar interventions to encourage synergy and avoid overlaps?

Assessing impact / To what extent has the project increased the system’s capacities to ensure that more vulnerable and excluded children benefit from community services in a way which contributes to their social inclusion?

Is there early evidence of the project having contributed to better integration of children with disabilities in the teaching and learning processes in selected schools as compared to the period before the introduction of DAISY textbooks?

Is there early evidence of the project having contributed to improved dynamics of group learning between children with and without disabilities in selected schools as compared to the period before the introduction of DAISY textbooks?

18 The humanitarian criteria for evaluation (coverage, coordination and coherence) are not applicable for the Montenegrin context or the context of this project.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 84.

Is there early evidence of the project having contributed to improving the confidence and

empowering children with disabilities in individual learning as compared to the period before the introduction of DAISY textbooks?

Is there early evidence of the project having contributed to improving the teaching and learning environment in selected schools as compared to the period before the introduction of DAISY textbooks?

Is there early evidence of the project having contributed to changing the attitudes and knowledge of teachers, schools management and parents on inclusive teaching and learning practices?

Is there an indication that the project results/contributions have affected girls and boys differently, and, if yes, why?

Assessing sustainability / To what extent are the achieved project outcomes sustainable? Did the project design include an appropriate sustainability strategy to support positive

changes in the selected schools after the end of the intervention? To what extent do the developed training programme, related manuals and guidelines for

teachers, supervisors and parents provide a basis for sustainable and coordinated scaling up of the project?

To what extent are national-level mechanisms strengthened for the monitoring and quality assurance of introduction of DAISY textbooks in teaching and learning in Montenegro?

To what extent will the new knowledge and skills be integrated into the regular activities of education professionals working with children with disabilities?

What is the level of ownership of the initiative within the Ministry of Education, Bureau for Education, Textbook Publishing Agency and schools, and what are the prospects for further development of related interventions after the end of external support?

To what extent has the project promoted the strengthening of already existing partnerships and the establishment of new ones?

In addition to the five main evaluation criteria, the evaluation will also focus on assessing the human rights-based approach and relevant cross-cutting issues. More specifically, it will look into the extent of the project outcomes’ contribution to achievement of children’s rights and how the project contributed to addressing key cross-cutting issues.

Does the project actively contribute to the promotion of children’s rights? To what extent and how does the project ensure an equity focus? Does the project reflect gender mainstreaming issues? To what extent are HR & GE a priority in the overall project budget and implementation? Was the design of the project ethical? How was the balance of cost and benefits to

participants (including possible negative impact) considered during the project implementation?

7. Methodology The evaluation methodology will be guided by the Norms and Standards of the United Nation Evaluation Group (UNEG): (http://www.uneval.org/normsandstandards/indexs.isp?doc_cat_source_id=4)19. Evaluability assessment The Theory of Change and data available allow for assessment of the progress achieved and evaluation of the project impact and results. These existing sources of information are assessed as the most reliable:

a. Ministry of Education Information System (MEIS) – specifically disaggregated data on the number of children with disabilities in primary schools;

19 UNEG Norms: http://www.uneval.org/indexAction.cfm?module=Library&action=GetFile&DocumentAttachmentID=1491 UNEG Standards: http://www.uneval.org/indexAction.cfm?module=Library&action=GetFile&DocumentAttachmentID=1496

Project “Using textbooks in DAISY format” – Draft Evaluation Report 85.

b. Data from the Resource Centre – number of direct and indirect project beneficiaries

involved in project activities and implementation, evaluation form from the participants of the trainings on the usefulness of the same, etc.;

c. Bureau for Education – data on supervisory visits to primary schools;

It is expected that the evaluation methodology and instruments used will enable the gathering of information on the situation before and after the intervention, specifically through the use of qualitative data collected from in-depth interviews, focus groups and a survey conducted on an appropriate sample. The contact list of all the project relevant stakeholders, project implementing partners and consultants will be provided to the evaluator once a contractual agreement has been made. Approach The overall evaluation approach to be applied will rely on the project’s Theory of Change and will have an equity focus. It will combine qualitative and quantitative data and apply a data-collection strategy that relies on primary and secondary data collection and non-experimental design.

The consultant will use the desk review to become familiar with the policy basis, relevant project documents and other means of verification / the sources of information listed above.

The consultant should develop more precise evaluation work. In addition to interviews with the Editorial Team and representatives of the Ministry of

Education, the Resource Centre, the Textbook Publishing Agency and UNICEF, the opinions of the following actors will also be acquired and analysed:

Key stakeholders and project beneficiaries: - Associations of Parents of Children with Disabilities; - NGOs representing parents; - School Parents’ Councils; - Teachers; - School management; - Children with disabilities and children without disabilities in selected schools; - Families of children with disabilities and of children without disabilities.

The guiding questions for the evaluation against the defined evaluation criteria will be further elaborated and used as a basis for development of the main data collection instruments (interviews and focus groups). All the data gathered will be analysed by the evaluator/s. Triangulation of data will be used to increase reliability of findings and conclusions. Special measures will be put in place to ensure that the evaluation process is ethical and that participants in the evaluation process can openly express their opinions. Also, if proved necessary, it will be ensured that relevant experienced professionals be present and provide support during the interviews/focus groups with children with disabilities. The sources of information will be protected, and known only to the evaluator/s. The Evaluation Team will ensure that the evaluation process is in line with UNEG Ethical Guidelines. The consultant must ensure that it is clear to all subjects that their participation in the evaluation is voluntary. All participants should be informed or advised of the context and purpose of the evaluation, as well as the privacy and confidentiality of discussions. 8. Work Plan

Description Responsible Timeline

Project “Using textbooks in DAISY format” – Draft Evaluation Report 86.

Preparatory phase:

Development of ToR UNICEF By 12 December 2014

Selection/contracting of consultant UNICEF By 26 December 2014

Data collection:

Desk review of the existing documents Consultant

May 2015

Logistics (arranging meetings / interviews) Consultant End May 2015

Field visit to Montenegro (meeting / interviews with UNICEF, MoLSW, EU Delegation, key partners and stakeholders)

Consultant with the support of UNICEF

Beginning June 2015

Debriefing meeting with Implementing Partners and UNICEF

Consultant

Beginning June 2015

Reporting:

Preparing inception report (methodology and work plan)

Consultant Mid-May 2015

Submission of the draft report Consultant

Mid/End June 2015

Feedback on the draft report from UNICEF and Implementing Partners

UNICEF End June 2015

Submission of the final report Consultant

Beginning July 2015

Use of evaluation findings:

Dissemination of the final report to all partners and stakeholders

UNICEF Beginning July 2015

Agreement reached with implementing partners on how to translate key findings into activities and integrate them into further activities / programming

UNICEF Mid-July 2015

The consultant is expected to produce the following key deliverables with the following tentative deadlines:

1. Inception Report, by 15 May 2015. 2. Draft project evaluation report, by 20 June 2015. 3. Final project evaluation report including executive summary and annexes, by 10 July 2015.

All deliverables should be submitted in English. UNICEF will ensure translation in Montenegrin. The timeframe for this work assignment is from 26 December 2014 to 10 July 2015. During that period the total number of consultancy days available is up to a maximum of 19, with the estimated share of days as following:

Desk review and submission of Inception Report – 3 days, Field visits and debriefing – 7 days, Draft report development – 6 days, Final report submission and presentation at the final conference – 3 days

The consultant is expected to observe the UNEG ethical guidance to evaluation as a guiding principle to ensure the quality of evaluation process. (http://www.uneval.org/search/index.jsp?q=ETHICAL+GUIDELINES)

Project “Using textbooks in DAISY format” – Draft Evaluation Report 87.

Management and Organization Management: The evaluation will be managed by the UNICEF Country Office – Evaluation Steering Committee consisting of Programme Specialist, Social Policy and Equity Officer (M&E focal point) and Education Specialist. The consultant will be supervised by the Programme Specialist. The management of the evaluation will include development of the terms of reference, assignment of the evaluation team, liaison between the evaluation team and partners/stakeholders involved (supporting organization of meetings, interviews and field visits), as well as quality assurance of the report. Representatives of the Implementing Partners will be involved in designing the evaluation and will participate in the definition of recommendations through active contribution during debriefing meetings and providing feedback on the draft report. Organization: Individual consultancy is required for this task. Schedule: This assignment will commence on 26 December 2014. Key intermediate tasks of the consultant:

Desk review of relevant documents and reports; Develop a more detailed evaluation methodology and work plan – draft to be submitted as a

part of the Inception Report to UNICEF for approval, including key instruments and interview questions;

Conduct data collection through field visits (carry out interviews/focus groups with selected stakeholders, partners and beneficiaries);

Present initial findings through a debrief meeting with UNICEF and implementing partners; Prepare the draft report with key findings, recommendations and lessons learned, based on

all sources of information used; Based on feedback provided by UNICEF and implementing partners, prepare and submit the

Final Report with all key findings, recommendations and lessons learned, following the UNICEF Evaluation Report Standards.

Qualifications/specialized knowledge/experience required to complete the task The consultant will be selected based on the following criteria:

Experience of conducting project and programme evaluations; Technical expertise in education and especially inclusive education; Knowledge of the education system in Montenegro; Knowledge of the UN Convention on the Rights of the Child and relevant international

standards in the area of education; Previous experience in working with UNICEF; Communication skills.

The qualifications and skill areas required include:

Consultant with expertise in the area of education with a focus on inclusive education and children with special educational needs;

Documented extensive evaluation expertise and experience; Proven knowledge of the education system in Montenegro, knowledge of the region is an

asset; Excellent report-writing skills; Good communication and presentation skills; Excellent written and spoken English; Ability to keep to strict deadlines;

Project “Using textbooks in DAISY format” – Draft Evaluation Report 88.

Knowledgeable of UN Evaluation policy; Experience in working with the UN or UNICEF (obligatory).

The consultant should be sensitive to beliefs and act with integrity and respect for all stakeholders. In the report the consultant may not refer to individual children. The consultant may not share findings with the media in Montenegro or abroad concerning individual children or individual institutions. The UNICEF premises will be available during the time spent in Montenegro, if needed. Printers, photocopying services and other similar services will be provided by UNICEF. It is expected that the consultant will bring his/her own laptop. Budget and remuneration The request for services under this contract will require the prospective consultant to indicate his/her fees for the services to be provided. As part of the selection process, the office will select the consultant that has quoted the lowest fee from the list of prospective consultants who are deemed suitable for achieving all the tasks on time, and as per the criteria and deliverables stipulated in the Terms of Reference. Resource requirements Total funds not to exceed US$7,000.00 Funding source for Evaluation: FR 9000007626 Project: “USING TEXTBOOKS IN DAISY FORMAT – Primary school teacher training in using audio textbooks” 9. Structure of the Evaluation Report The final evaluation report should follow UNEG Norms and Standards, UNICEF Evaluation Report Standards and should follow the GEROS Quality Assessment System.

UNEG_UNICEF Eval Report Standards.pdf

The report template to be used includes:

Title page and opening pages

Executive summary

Project description (including the logic of the project design and/or expected results chain)

The role of UNICEF and Implementing Partners and other stakeholders involved

Purpose of the evaluation

Evaluation criteria

Evaluation scope and objectives

Evaluation design

Description of methodology

Stakeholders’ participation

Ethical issues

Findings

Analysis of results

Constraints

Project “Using textbooks in DAISY format” – Draft Evaluation Report 89.

Conclusions

Recommendations

Lessons learned

Annexes The evaluation report will be assessed and rated against the following assessment tool available here:

2013 Evaluation TOR review template.xlsx

Project “Using textbooks in DAISY format” – Draft Evaluation Report 90.

ANNEX 2 – LIST OF INTERVIEWED STAKEHOLDERS

NAME POSITION ORGANIZATION

Benjamin Perks Country Representative UNICEF

Ana Zec Programme Specialist UNICEF

Maja Kovačevic Education Specialist UNICEF

Z.Boškovic Director Resource Centre

Mladen Perazic IT Team Production team

Aleksandra Vešovic Supervisor and teacher trainer

BoE

Nađa Durkovic Textbook editor/teacher

trainer/supervisor

ITTA

Anita Maric Special education teacher BoE

Mirjana Raškovic Teacher and supervisor Resource Centre

Dubravka Drakic Teacher Drama Academy

Ms. Krivokapic Director Vladimir Nazor School

Mr. Sekularac Director Karadžic School

Non available Director assistant Karadžic School

Ms. Pavicevic Director Njegoš School

Tamara Milic Advisor for Students with

Special Education Needs

MoE

Toni Koletic IT Team Production team

Project “Using textbooks in DAISY format” – Draft Evaluation Report 91.

ANNEX 3 – LIST OF DOCUMENTS REVIEWED

DESCRIPTION SOURCE STATUS

Main project document

design and work plan

UNICEF Montenegro REVIEWED

Annual agency report (2013) UNICEF Montenegro REVIEWED

Monitoring sheets completed

by participating schools

UNICEF Montenegro PARTIALLY REVIEWED

Evaluation terms of

reference and rationales

UNICEF Montenegro REVIEWED

Other studies and evaluation

concerning the introduction

of DAISY audio-visual

textbooks in schools

VARIOUS REVIEWED

MoU and agreements signed

with partners

UNICEF Montenegro and relevant

partners

N/A

Detailed budget and financial

records

UNICEF Montenegro REVIEWED

Montenegrin laws on primary

education and inclusion

UNICEF Montenegro, Ministry of

Education

REVIEWED

Activity reports UNICEF Montenegro PARTIALLY REVIEWED

List of participants (schools

and students)

UNICEF Montenegro REVIEWED

Documents from meetings

organized by UNICEF

UNICEF Montenegro N/A

List of participants – training UNICEF Montenegro REVIEWED

Reports from

training/seminar

Resource Centre REVIEWED

Manual for the Use of DAISY

Textbooks developed by the

Technical Team

UNICEF Montenegro REVIEWED

ANNEX 4 – EVALUATION MATRIX

CRITERIA EVALUATION QUESTIONS

CODE SUB-QUESTIONS INDICATORS CONCEPTS ADDRESSEES TECHNIQUES LEVEL OF

COMPLETION

RELEVANCE

To what extent is the

project responding to the needs of stakeholders and users?

1.1

To what extent are the project design and its objectives relevant to national policies and strategies on education?

The number and description of articles in national policies in line with project design and objectives. The number and type of activities implemented addressing Montenegro development needs in the area of education. At least 70% of respondents agree / strongly agree that the project is relevant for the achievement of national education strategies.

Alignment of project design with national policies and strategies on education

UNICEF, implementing

partners, direct beneficiaries

Document review,

interviews, survey

Fully Achieved

1.2

To what extent does the project address the underlying causes of exclusion for CwD in the educational system and respond to the needs of CwD in Montenegrin inclusive primary schools?

Pattern in stakeholders’ identification of project’s essential activities.

Extent to which project reached intended beneficiaries and their high-priority need in inclusive education.

UNICEF, implementing

partners, direct beneficiaries

Survey

Fully achieved

Pattern in stakeholders’ verbalization of DAISY’s usefulness.

Interviews, FG, survey

At least 70% of respondents agree / strongly agree that the project responds to the needs of CwD in primary schools.

Survey

Identification of underlying causes of exclusion and their link to project activities and goals.

Document review,

interviews, FG

1.3

To what extent is the project design relevant to the overall project goal and the achievement of its objectives in the given period of time?

Analysis of project design rationale, logic and theory of action. At least 70% of respondents agree / strongly agree that the project design reflects the goals and objectives.

Adequacy of project design to tackle defined goals and impact

UNICEF, implementing

partners, direct beneficiaries

Document review, survey

Largely achieved

1.4

Was the project designed according to international norms on human rights and gender equality and in line with national strategies in those areas?

Analysis of project design linked to international and national norms and agreements on HR and GE. At least 70% of respondents agree / strongly agree that the project reflects national strategies on GE and HR.

Alignment of project design with international and national standard norms on human rights and gender equality.

UNICEF, implementing

partners, direct beneficiaries

Document review, survey

Fully achieved

Project “Using textbooks in DAISY format” – Draft Evaluation Report 93.

ANNEX 4 – EVALUATION MATRIX

CRITERIA EVALUATION QUESTIONS

CODE SUB-QUESTIONS INDICATORS CONCEPTS ADDRESSEES TECHNIQUES LEVEL OF

COMPLETION

RELEVANCE

To what extent is the

project responding to the needs of stakeholders and users?

1.5

Were the training sessions, materials and produced digital textbooks adequate for the intended audiences and did they contribute to goal achievement?

At least 60% of respondents rate their overall level of satisfaction with materials and guidelines/digital textbooks as very high / high. Addressees’ verbalization on training materials and digital textbooks’ adequateness for goal achievement.

Adequacy of developed materials for intended audiences (manuals, guidelines, DAISY contents in Montenegrin)

UNICEF, implementing partners and

direct beneficiaries

Survey, interviews, FG

Largely achieved

At least 60% of final DAISY book users consider produced digital books in mother tongue useful / very useful for the teaching and learning processes

Adequacy of developed materials for intended audiences (manuals, guidelines, DAISY contents in Montenegrin)

Target group, direct and

indirect beneficiaries

Survey, interviews, FG

Achieved

EFFECTIVENESS

To what extent does the project meet the

results and outcomes as

defined in the theory of change?

2.1

To what extent and how did the project contribute to enabling teachers to effectively introduce DAISY books and better organize the teaching process for joint learning?

The number and type of strategies used by teachers for joint learning with DAISY books as a result of the training provided by the project. A pattern in teachers’ verbalization about better organizing of the teaching process for joint learning thanks to the intervention. The number of strategies that respond to student diversity in class.

Teachers’ ability to adapt the teaching process to the use of DAISY books (e.g. cooperative learning, peer tutoring) to ensure all student participate in the class dynamics. Measure attitudes and knowledge. Learning activities respond to student diversity.

UNICEF, implementing

partners, teachers

Interviews, FG, document

review

Partially achieved

2.2

Do parents effectively and regularly provide assistance to their children in the use of DAISY books?

Frequent and correct assistance provided by parents expressed by both parents and students.

Parents provide frequent and correct support on the use of DAISY to their children

Target group, indirect

beneficiaries and parents

Survey, FG Not achieved

2.3 Have CwD been empowered

in individual learning?

Pattern in teachers, parents and students’ verbalization about evolution in individual learning during the project life

CwD gain in confidence as a result of the intervention

and individual learning activities. Evolution of learning independence

based on the decrease in the need for parental

support. Linked to 4.4

Parents, teachers,

directors and CwD students

FG

Limited achievement

Frequency and number of individual learning activities conducted in class/at home

Survey

Evolution of students’ need of parental support on the use of DAISY

FG

Project “Using textbooks in DAISY format” – Draft Evaluation Report 94.

ANNEX 4 – EVALUATION MATRIX

CRITERIA EVALUATION QUESTIONS

CODE SUB-QUESTIONS INDICATORS CONCEPTS ADDRESSEES TECHNIQUES LEVEL OF

COMPLETION

EFECTIVENESS

To what extent does the project meet the

results and outcomes as

defined in the theory of change?

2.4

Are teachers and parents aware of the benefits of using DAISY textbooks and motivated to work with them?

Pattern in verbalization of DAISY usefulness expressed by addressees Positive attitudes towards

DAISY books and the project

Parents and teachers

Interview, FG, survey

Limited achievement Pattern in addressees’ verbalization

about motivation and positive attitudes towards the use of DAISY textbooks.

Interview, FG, survey

2.5

To what extent has the project contributed to strengthening of the skills of parents to better support their children in the learning process with DAISY?

At least 15 parent representatives of the association of CwD acquiring the technical skills to use DAISY books. The number and percentage of parents that successfully finished the training.

Project contribution to developing parents’ skills in

the use of DAISY Parents, UNICEF

Document review

Not achieved Number and percentage of parents that attended the training

Document review

Parents’ verbalization of the usefulness and adequacy of training provided at the end of the implementation period

Survey, FG

2.6

To what extent has the project develop teachers’ skills in the use of DAISY books?

At least 110 teachers from 24 primary schools and 5 supervisors/education advisors trained in using DAISY books.

Project contribution to developing teachers’ skills

in the use of DAISY

Teachers, directors,

students and UNICEF

Document review

Partially achieved

Number and percentage of teachers that attended the training

Document review

Teachers’ self-assessment on their capacities to use DAISY at the end of the implementation period.

Survey

Students’ assessment of teachers’ capacity to use DAISY.

Survey, FG

2.7

To what extent has the project enabled better accessibility to learning materials (DAISY) to children with disabilities and learning difficulties?

Pattern in students’ verbalization about accessibility of DAISY textbooks

Accessibility and availability of learning materials to students

UNICEF, teachers, directors, parents,

implementing partners

FG

Partially achieved

Timely introduction of devices in participating schools.

Document review,

interviews, FG, survey

Project “Using textbooks in DAISY format” – Draft Evaluation Report 95.

ANNEX 4 – EVALUATION MATRIX

CRITERIA EVALUATION QUESTIONS

CODE SUB-QUESTIONS INDICATORS CONCEPTS ADDRESSEES TECHNIQUES LEVEL OF

COMPLETION

EFFECTIVENESS

To what extent does the project meet the

results and outcomes as

defined in the theory of change?

2.8

Has the project developed students’ skills in the use of DAISY through individual and group sessions?

The number and frequency of individual and group sessions conducted in the use of DAISY. At least 1 group/individual session per class conducted.

Effectiveness of individual and group sessions on

developing student’s skills on the use of DAISY.

Direct beneficiaries, target group and indirect beneficiaries

Document review, survey

Not achieved

N/A

Teachers, students and parents’ perception concerning the coverage and usefulness of these sessions.

FG, interviews

2.9

Has the project provided any additional (not directly planned by the project) significant contributions / outcomes?

Identification of unintended effects, both positive and/or negative based on facts identified by stakeholders. Identification of unintended effects from the review of similar interventions

Identification of both positive and negative unanticipated consequences

All relevant stakeholder

Observation, document

review, interviews and

FG

EFFICIENCY

To what extent did the management of the project

ensure timelines and

efficient utilization of resources?

3.1 How well has the implementation of activities been managed?

The number of monitoring visits conducted in each school vs. planned visits. The number of monitoring surveys completed and completion ratio among all participating schools. Reporting frequency. Percentage of activities monitored and evaluated.

Efficiency on implementation management

UNICEF, implementing

partners, direct beneficiaries

Document review,

interviews, FG

Largely achieved

3.2 Were activities implemented as scheduled?

Study of the expected chronogram and monitoring data to establish timely implementation of activities.

Timely implementation of activities

UNICEF, implementing

partners, direct beneficiaries

Document review,

interviews, FG, survey

Fully achieved

3.3 Are monitoring systems adequate?

List of tools used to manage and monitor the project. Study of the monitoring plan designed by implementing partners

Management and monitoring tools

UNICEF, direct beneficiaries

implementing partners

Document review

Partially achieved

Project “Using textbooks in DAISY format” – Draft Evaluation Report 96.

ANNEX 4 – EVALUATION MATRIX

CRITERIA EVALUATION QUESTIONS

CODE SUB-QUESTIONS INDICATORS CONCEPTS ADDRESSEES TECHNIQUES LEVEL OF

COMPLETION

EFFICIENCY

To what

extent did the management of the project

ensure timelines and

efficient utilization of resources?

3.4

How well were the financial resources used / were funds managed in a cost-effective manner? What is the correlation between funds utilized and outputs/results achieved? Could the same results have been achieved with fewer resources?

The number and percentage of activities that were undertaken within the planned budget and work plan. Percentage of staff acknowledging the adequacy of the time and resources spent in X activity.

Cost-effectiveness analysis. Implementation alternatives and costs (cost-minimization analysis). Cost of similar interventions in others areas/contexts could be analysed and compared. Costs per client served. Analysis would be conducted based on UNICEF’s perspective.

UNICEF, implementing

partners

Document review,

interviews, FG

Fully achieved

Was the intervention coordinated

internally and externally

with relevant partners and

similar interventions?

3.5

Did the project ensure coordination with other similar interventions to encourage synergy and avoid overlaps?

Agreements/MoU signed with other organizations implementing similar projects in other areas/countries

Coordination with other organizations implementing similar interventions. External coordination

UNICEF, implementing

partners Interviews, FG N/A

3.6 Was the intervention adequately coordinated internally?

Pattern in verbalization of effective internal coordination during the implementation. The number of joint activities delivered in a coordinated and timely manner. The number and percentage of activities that would not have been achievable without partners’ support. Percentage of partners that share same goals and operating principles as UNICEF.

Internal coordination UNICEF,

implementing partners

Interviews, FG, survey

Fully achieved

What is the overall level of satisfaction with the intervention?

3.7 How would you rate the quality of the assistance provided?

At least 70% of stakeholders are very satisfied / satisfied with the quality of the activities implemented.

Client/user satisfaction survey

All relevant stakeholders

Survey Fully

achieved

Project “Using textbooks in DAISY format” – Draft Evaluation Report 97.

ANNEX 4 – EVALUATION MATRIX

CRITERIA EVALUATION QUESTIONS

CODE SUB-QUESTIONS INDICATORS CONCEPTS ADDRESSEES TECHNIQUES LEVEL OF

COMPLETION

IMPACT

To what extent has the project increased the system’s capacities to ensure CwSEN benefit from an improved teaching and learning environment that contributes to their social inclusion?

4.1

Has the teaching process evolved and improved as a result of the project? Early evidence of the project having contributed to changing the attitudes and knowledge of teachers, schools management and parents on the inclusive teaching and learning practices

Identification of type and number of educational techniques learned and implemented as a result of the project that ensure greater involvement for all students. .

Linked to questions included in effectiveness. Educational staff are capable of implementing a teaching process where all students are involved, with the support of new technologies (DAISY) as well as classical education techniques that ensure an inclusive and effective teaching environment.

UNICEF, teachers and

directors

Document review,

interviews, FG

Limited achievement Pattern in teachers, parents and

directors’ verbalization on whether the project has contributed to change their attitudes and increase knowledge on inclusive teaching and learning practices.

Teachers, directors and

parents, UNICEF Interviews, FG

To what extent has the project increased the system’s capacities to ensure that more vulnerable and excluded children benefit from an improved teaching and learning

4.2

Has the learning process of CwD improved as a result of the project? Early evidence of the project having contributed to changing students’ attitudes toward school and the learning process.

Pattern in CwD’s verbalization of positive attitudes towards schools and the new educational techniques introduced since the project started.

Learning process improved based on changes on attitudes towards school, emotional satisfaction and usefulness concerning the use of DAISY, engagement in own learning, and the development of learning opportunities

CwD, teachers, parents

Interview, FG

Limited achievement

Students’ emotional satisfaction in the use of DAISY books

CwD FG, survey

Verbalization of DAISY’s usefulness in the learning process recorded by CwD

CwD FG, survey

Students are actively engaged in their own learning.

Teachers, parents and CwD

Document reviews,

interviews, FG, survey

Project “Using textbooks in DAISY format” – Draft Evaluation Report 98.

ANNEX 4 – EVALUATION MATRIX

CRITERIA EVALUATION QUESTIONS

CODE SUB-QUESTIONS INDICATORS CONCEPTS ADDRESSEES TECHNIQUES LEVEL OF

COMPLETION

IMPACT

environment in a way that contributes to their social inclusion? 4.3

Is there early evidence of the project having contributed to improve learning group dynamics between children with and without disabilities in selected schools as compared to the period before the introduction of DAISY textbooks?

Pattern in stakeholders’ verbalization of project’s contribution to improving learning group dynamics in participating schools. Evolution of monitoring records about the level of improvement concerning learning group dynamics. Percentage of students who strongly disagree / disagree with Q.15 in questionnaire.

Improvement of learning group dynamic. Linked to 4.4 and 4.5 (Higher integration builds higher confidence)

Students, teachers, directors

Document review,

interviews, FG, survey

Limited achievement

4.4

Is there early evidence of the project having contributed to improving the confidence and empowering the CwD through group and individual learning as compared to the period before the introduction of DAISY books?

Pattern in stakeholders’ verbalization of project’s contribution to improve CwD confidence in schools. Evolution of CwD’s voluntary participation in the class dynamics since the introduction of DAISY textbooks. The number and frequency of group and individual learning activities conducted per school. Percentage of students who strongly disagree / disagree with Q.9 in questionnaire.

CwD gain confidence as a result of the intervention and through individual and group learning activities

Teachers, directors, parents, students

Document review,

interviews, FG, survey

N/A

To what extent has the project increased the system’s capacities to ensure that more vulnerable and excluded children

4.5

Is there early evidence of the project having contributed to better integration of CwD in the teaching and learning processes in selected schools as compared to period before the introduction of DAISY books?

Pattern in stakeholders’ verbalization of project’s contribution to better integration of CwD in participating schools. Diversity is valued as an enriching aspect of the school environment. Support for positive behaviour is embedded in classroom and school-wide routines to support learning and social participation.

Identification of project’s contribution to better integration of CwD in participating schools

Students, parents, teachers, directors,

implementing partners

Document review,

interviews, FG N/A

Evolution of monitoring records concerning the level of integration achieved at every stage.

Document review

Project “Using textbooks in DAISY format” – Draft Evaluation Report 99.

ANNEX 4 – EVALUATION MATRIX

CRITERIA EVALUATION QUESTIONS

CODE SUB-QUESTIONS INDICATORS CONCEPTS ADDRESSEES TECHNIQUES LEVEL OF

COMPLETION

IMPACT

benefit from an improved teaching and learning environment in a way that contributes to their social inclusion?

4.5

Is there early evidence of the project having contributed to better integration of CwD in the teaching and learning processes in selected schools as compared to the period before the introduction of DAISY books?

CwD’s expression of acceptance from other students as active social members (Q.10, 11 & 12 in students’ questionnaire). Students have opportunities to demonstrate leadership. Compare CwD’s positive attitude towards group activities in class using/not using DAISY (Q.15). Evolution of social dynamics during recess at school and extracurricular activities since the introduction of DAISY. Relationships between CwD and other students evolve outside the school (birthdays, house visits) since the introduction of DAISY. Children without disabilities learn tolerance, acceptance of difference and respect for diversity

Students FG, survey

SUSTAINABILITY

To what extent are the

achieved project

outcomes sustainable?

5.1

Did the project design include an appropriate sustainability and exit strategy to support positive changes in selected schools after the end of the project intervention?

Existence of an effective and realistic exit strategy shared and agreed among partners. Level of ownership of the initiative by partners. Pattern in stakeholders’ verbalization on their level of appropriation and prospects to further develop the project after the end of the external support. Number of meetings with key partners to discuss project’s future. Number of new partnerships/ partnership signed or under negotiation

Existence of a clear and realistic exit strategy shared by all parties. Prospects for further development of related interventions after the end of the external support. Promotion of already existing partnerships and establishment of new ones

All relevant stakeholder

Document review,

interviews, FG, survey

Partially achieved

5.2

To what extent do the developed training programme, related manuals and guidelines for teachers, supervisors and parents provide grounds for sustainable and coordinated scaling up of the project?

Existence of an effective and realistic exit strategy shared and agreed with main partners. Frequency of the revision of materials and guidelines for teachers planned by partners. Number and percentage of trained participants that express very high / high levels of satisfaction with training materials six months after the training.

Adequateness of materials for eventual project scaling up

UNICEF, teacher,

directors and parents

Document review, survey

Largely achieved

Project “Using textbooks in DAISY format” – Draft Evaluation Report 100.

ANNEX 4 – EVALUATION MATRIX

CRITERIA EVALUATION QUESTIONS

CODE SUB-QUESTIONS INDICATORS CONCEPTS ADDRESSEES TECHNIQUES LEVEL OF

COMPLETION

SUSTAINABILITY

To what extent are the

achieved project

outcomes sustainable?

5.3

To what extent are national level mechanisms strengthened for monitoring and quality assurance of introduction of DAISY books in teaching and learning in Montenegro?

Existence of mechanisms at the national level to ensure eventual introduction of DAISY books in Montenegrin primary schools

Adequateness of national mechanisms to ensure quality assurance on the eventual introduction of DAISY books at the national level

Governmental bodies (Ministry

of Education, national

education council), UNICEF,

Resource Centre

Interviews Partially achieved

5.4

To what extent will the new knowledge and skills be integrated into the regular activities of educational professionals working with CwD?

Pattern in addressees’ verbalization on the extent to which the new knowledge acquired will be integrated into regular activities

Teachers and

directors Interviews

Largely achieved

CROSS-CUTTING

ISSUES

Gender streaming

6.1

To what extent and how does the project ensure an equity focus and reflect gender mainstreaming issues?

Strategies identified aimed at ensuring gender equity at all levels. Study on gender-equal access to DAISY books. To what extent do the project results reach boys and girls equally? Measure of functional usefulness and emotional satisfaction expressed by students on the use of DAISY disaggregated by gender. Study of results on level of inclusion and acceptance, voluntary participation and positive attitude towards school generated by the intervention disaggregated by gender.

Equal contribution to inclusion for boys and girls, and reasons.

All relevant stakeholders

Document review,

interviews, FG, survey

Fully achieved

ASSUMPTIONS

Factors that

might positively or negatively

influence the project

7.1 Was enough funding available to cover the intervention’s needs?

Identification of proposed activities not carried out due to lack of funds. Constraints in designed implementation due to lack of funding.

Availability of funds to cover needs of intervention

Implementing partners, UNICEF

Document review,

interviews

Fully achieved

7.2 Is the physical environment of the school welcoming to all students?

Existence of physical barriers hindering CwD performance in schools

Adequateness of school environment to CwD needs

Students, parents, teachers, directors, UNICEF

Observation, interviews, FG

N/A

Project “Using textbooks in DAISY format” – Draft Evaluation Report 101.

ANNEX 4 – EVALUATION MATRIX

CRITERIA EVALUATION QUESTIONS

CODE SUB-QUESTIONS INDICATORS CONCEPTS ADDRESSEES TECHNIQUES LEVEL OF

COMPLETION

ASSUMPTIONS

Factors that might

positively or negatively

influence the project

7.3 Have students experienced the benefits of using DAISY textbooks?

Emotional satisfaction expressed by students on the use of DAISY books. Pattern in addressees’ verbalization on benefits experienced by students.

Linked to 2.4, 4.4, 4.6

Students, parents, teachers, directors

Interviews, FG, survey

Partially achieved

7.4

Do parents have electronic devices at home to support their children’s learning process?

The number and percentage of families with access to electronic devices and DAISY books at home

Access to DAISY books at home

Parents FG, survey N/A

7.5

Do teacher have enough skills concerning general inclusion teaching strategies?

The number and type of strategies used by teachers for joint and inclusive learning (not related to DAISY books) and not related to training provided by the project.

Knowledge about general inclusion is required besides DAISY-specific knowledge. Teachers should have a good understanding of key inclusion strategies not related to DAISY. Project might not provide this type of training, but is key for its success. Teachers work on students’ self-esteem, have positive attitude towards CwD, know about cooperative learning, and focus on the process and not on results.

Teachers, directors, parents, UNICEF

Interviews, FG N/A

Project “Using textbooks in DAISY format” – Draft Evaluation Report 102.

ANNEX 5 – List of participating schools and disaggregated number of students

Municipality and Primary Schools

Total Number of

Children Grades 4–9

Total Number of

Children with SEN

Grades 4–9

Girls with SEN

Grades 4–9

Boys with SEN

Grades 4-9

Number of Children

with Disability

Children with

Physical Disability

Children with

Intellectual Disability

Children with Visual

Impairments

Children with

Hearing Impairments

Children with

Speech Disorders

Children with

Autism

Children with

Combined Disabilities

Number of

Children with

Learning Difficulties

Primary-focus Schools:

Podgorica, JU OŠ “Pavle Rovinski”

1000 22 5 17 11 2 0 2 0 2 1 4 11

Podgorica, JU OŠ “Savo Pejanović”

478 27 16 11 5 3 0 1 0 0 0 1 22

Podgorica, JU OŠ “Dr Dragiša Ivanović”

1090 6 4 2 6 2 2 1 0 0 0 1 0

Danilvograd, JU OŠ “Njegoš”

428 17 4 13 9 4 2 1 0 1 0 1 8

Niksic, JU OŠ “Mileva Lajović Lalatović”

694 17 7 10 7 1 0 3 0 0 2 1 10

Niksic, JU OŠ “Luka Simonović”

515 17 5 12 9 3 1 0 2 1 0 2 8

Bijelo Polje, JU OŠ “Dušan Korać”

543 13 5 8 11 1 4 0 0 0 0 6 2

Pljevlja, JU OŠ “Boško Buha”

402 11 5 6 6 1 2 1 0 0 0 2 5

Berane, JU OŠ “Vuk Karadžić”

821 102 34 68 29 2 8 0 1 13 0 5 73

Tivat, JU OŠ “Drago Milović”

822 7 1 6 5 1 1 0 0 1 1 1 2

Podgorica, Resource Centre for Children with Disabilities

19 5 14 19 4 1 4 0 0 0 10 0

Other schools:

Podgorica, JU OŠ “Branko Božović”

698 7 5 2 5 1 2 1 0 0 0 1 2

Project “Using textbooks in DAISY format” – Draft Evaluation Report 103.

Municipality and Primary Schools

Total Number of

Children Grades 4–9

Total Number of

Children with SEN

Grades 4–9

Girls with SEN

Grades 4–9

Boys with SEN

Grades 4-9

Number of Children

with Disability

Children with

Physical Disability

Children with

Intellectual Disability

Children with Visual

Impairments

Children with

Hearing Impairments

Children with

Speech Disorders

Children with

Autism

Children with

Combined Disabilities

Number of

Children with

Learning Difficulties

Podgorica, JU OŠ “Milorad Musa Burzan”

879 57 19 38 15 1 1 6 1 5 0 1 42

Podgorica, JU OŠ “Maksim Gorki”

1017 113 36 77 23 2 3 2 3 13 0 0 90

Podgorica, JU OŠ “Vladimir Nazor”

609 14 9 5 7 1 2 0 2 0 1 1 7

Niksic, JU OŠ “Olga Golović”

546 18 9 9 12 1 5 2 0 1 0 3 6

Niksic, JU OŠ “Ratko Žarić”

595 23 8 15 9 0 1 0 2 5 0 1 14

Niksic, JU OŠ “Braća Ribar”

493 37 16 21 8 1 0 2 1 3 0 1 29

Bijelo Polje, JU OŠ “Risto Ratković”

384 4 3 1 1 0 0 1 0 0 0 0 3

Bijelo Polje, JU OŠ “Marko Miljanov”

655 5 1 4 1 0 0 1 0 0 0 0 4

Bijelo Polje, JU OŠ “Aleksa Đilas Bećo”

112 6 0 6 1 0 1 0 0 0 0 0 5

Herceg Novi, JU OŠ “Ilija Kišić”

283 51 28 23 27 0 10 0 1 7 0 9 24

Herceg Novi, JU OŠ “Milan Vuković”

505 80 32 48 26 0 7 2 2 14 0 1 54

Bar, JU OŠ “Jugoslavija” 813 25 14 11 18 2 8 2 1 1 2 2 7

Kotor, JU OŠ “Njegoš” 507 79 20 59 26 1 5 0 0 14 1 5 53

ANNEX 6 – QUESTIONNAIRES

UNICEF “Using textbooks in DAISY format” – SURVEY FOR IMPLEMENTING PARTNERS

Organization:

Date:

1. UNICEF’s support through the DAISY project is relevant for the achievement of Montenegro’s overall inclusive education goals.

Strongly Agree Agree Neither agree nor

disagree Disagree Strongly Disagree N/A

(ii) In case strongly agree / agree: Could you please name the national education goals to which DAISY project contributed?

2. UNICEF’s support through the DAISY project responds to the needs of CwD in primary schools

Strongly Agree Agree Neither agree nor

disagree Disagree Strongly Disagree N/A

3. The design of the DAISY project…

Strongly Agree

Agree Neither agree nor disagree

Disagree Strongly Disagree

N/A

..is in line with national strategies in inclusive education (2014–18)

…is in line with national strategies on gender equality

…is adequate to achieve intended goals and objectives set for the project

(iii) In case strongly agree / agree: Could you please name the national strategies on human rights and gender equality to which the DAISY project contributed?

Project “Using textbooks in DAISY format” – Draft Evaluation Report 105.

4. Could you name and categorize the DAISY project activities as follows? Please provide an answer for each item and briefly include the main reason for this categorization

Category Activities Why?

Essential and excellent (only the

project offers these activities)

Essential activities but should be improved

Useful but they can be offered by other partners

Activities no longer needed

5. Could you please rate the DAISY project capacity-building activities (training, workshops, sessions) according to:

Very high High Moderate Low Very low N/A

Quality of content concerning DAISY books

Quality of content concerning general inclusion strategies

Usefulness

Timeliness

Your overall level of satisfaction

6. Could you please rate the DAISY project training materials and guidelines according to:

Very high High Moderate Low Very low N/A

Quality of content

Usefulness

Your overall level of satisfaction

7. Could you please rate the produced DAISY textbooks according to:

Very high High Moderate Low Very low N/A

Quality of narration

Usefulness

Timely introduction in schools

Availability of technical support

Your overall level of satisfaction

Project “Using textbooks in DAISY format” – Draft Evaluation Report 106.

8. Other relevant activities, please specify ___________________________________:

Very high High Moderate Low Very low N/A

Quality of content

Usefulness

Timeliness

Your overall level of satisfaction

9. How would you rate the level of coordination with other implementing partners?

Very High High Moderate Low Very low N/A

10. Did the intervention include an appropriate sustainability and exit strategy to ensure the benefits generated by the project continue after donor funding has been withdrawn?

YES NO N/A

In case your answer to Q10 is “YES”, could you please provide some examples? In case your answer to Q10 is “NO”, could you please provide reasons?

11. Could you please rate the overall level of satisfaction with the quality of Project DAISY

Very high High Moderate Low Very low N/A

Quality of implementation

Usefulness

Timeliness

Equal access for boys and girls (equity focus)

Your overall level of satisfaction

Could you please elaborate on your answer/ indicate reasons?

Project “Using textbooks in DAISY format” – Draft Evaluation Report 107.

UNICEF “Using textbooks in DAISY format” – SURVEY FOR DIRECT BENEFICIARIES (Families with CwD / with children without disabilities)

Date:

Type of respondent:

Family with CwD

Family with children without disabilities

Number of CwD at school:

Association/school:

1. Did you attend the training provided by the DAISY project on the use of DAISY audio-visual textbooks?

No Partially Completely

2. Could you please rate the DAISY project capacity-building activities (training, workshops, sessions) according to:

Very high High Moderate Low Very low N/A

Quality of content concerning DAISY books

Quality of content concerning general inclusion strategies

Usefulness

Timeliness

Your overall level of satisfaction

3. Could you please rate the DAISY project training materials and guidelines according to:

Very high High Moderate Low Very low N/A

Quality of content

Usefulness

Your overall level of satisfaction

4. Could you please rate the produced DAISY audio-visual textbooks according to:

Very high High Moderate Low Very low N/A

Quality of narration

Usefulness (learning)

Timely introduction in schools.

Availability of technical support

Your overall level of satisfaction

Project “Using textbooks in DAISY format” – Draft Evaluation Report 108.

5. How would you rate your skills in the use of DAISY audio-visual textbooks?

Very High High Moderate Low Very low N/A

6. How would you rate your motivation to use DAISY audio-visual textbooks?

Very High High Moderate Low Very low N/A

7. Do you have access to DAISY audio-visual textbooks at home? (If answer is “NO” or “N/A”, please go to Q14)

YES NO N/A

8. How often does your child use DAISY audio-visual textbooks for his/her individual learning?

Daily Weekly Monthly Never

9. How often do you provide assistance to your children in the use of DAISY audio-visual textbooks?

Daily Weekly Monthly Never

10. How often are you able to solve your child’s doubts about the use of DAISY audio-visual textbooks?

Always / Almost always

Usually / Most of the time

Often Seldom / Rarely Never / Almost

never

11. Could you please rate the overall level of satisfaction with the quality of Project DAISY

Very high High Moderate Low Very low N/A

Quality of implementation

Usefulness

Timeliness

Equal access to boys and girls (equity focus)

Your overall level of satisfaction

Could you please elaborate on your answer/ indicate reasons?

Project “Using textbooks in DAISY format” – Draft Evaluation Report 109.

UNICEF “Using textbooks in DAISY format” – SURVEY FOR DIRECT BENEFICIARIES (Teachers and school managers)

Date:

Type of respondent:

Teacher

Supervisor

School Management

Position:

Association/School: 2. UNICEF’s support through introduction of DAISY audio-visual textbooks project is relevant for the achievement of Montenegro’s overall inclusive education goals.

Strongly Agree Agree Neither agree nor disagree

Disagree Strongly Disagree N/A

3. In case strongly agree / agree: Could you please name the national education goals to which the DAISY project contributed?

4. UNICEF’s support through the introduction of DAISY audio-visual textbooks project responds to the needs of CwD in inclusive primary schools

Strongly Agree Agree Neither agree nor disagree

Disagree Strongly Disagree N/A

5. The design of the DAISY project…

Strongly Agree

Agree Neither

agree nor disagree

Disagree Strongly Disagree

N/A

..is in line with national strategies in inclusive education (2014–18)

…is in line with national strategies on gender equality

…is adequate to achieve the intended goals and objectives set for the project.

Project “Using textbooks in DAISY format” – Draft Evaluation Report 110.

6. In case strongly agree / agree: Could you please name the national strategies on Human Rights and Gender equality to which the DAISY project contributed?

7. Could you name and categorize the DAISY project activities as follows? Please provide an answer for each item and briefly include the main reason for this categorization

Category Services Why?

Essential and excellent (only the project offer these activities)

Essential activities but should be improved

Useful but they can be offered by other partners

Activities no longer needed

8. Could you please rate the training on the use of DAISY audio-visual textbooks (both technical and pedagogical aspects – or separately):

Very high High Moderate Low Very low N/A

Quality of content concerning DAISY books

Quality of content concerning general inclusion strategies

Usefulness

Timeliness

Your overall level of satisfaction

9. Could you please rate the DAISY project training materials and guidelines according to:

Very high High Moderate Low Very low N/A

Quality of content

Usefulness

Your overall level of satisfaction

Project “Using textbooks in DAISY format” – Draft Evaluation Report 111.

10. Could you please rate the produced DAISY audio-visual textbooks according to:

Very high High Moderate Low Very low N/A

Quality of narration

Usefulness (teaching)

Timely introduction in schools.

Availability of technical support

Your overall level of satisfaction

12. Other relevant activities, please specify ___________________________________:

Very high High Moderate Low Very low N/A

Quality of content

Usefulness

Timeliness

Your overall level of satisfaction

13. How would you rate your skills in the use of DAISY audio-visual textbooks?

Very High High Moderate Low Very low N/A

14. How would you rate your motivation to use DAISY audio-visual textbooks?

Very High High Moderate Low Very low N/A

15. How often have you conducted individual/group learning sessions in the use of DAISY audio-visual textbooks?

Individual Sessions Group sessions

Daily Weekly Monthly Daily Weekly Monthly

Total number of individual sessions conducted______

Total number of group sessions conducted__________

16. How often have you conducted individual/group learning sessions with DAISY audio-visual textbooks?

Individual Sessions Group sessions

Daily Weekly Monthly Daily Weekly Monthly

Total number of individual sessions conducted______

Total number of group sessions conducted__________

Project “Using textbooks in DAISY format” – Draft Evaluation Report 112.

17. Do you think you will be able and willing to continue using DAISY audio-visual textbooks once the project finishes?

YES NO N/A

In case your answer to Q10 is “YES”, could you please provide some examples? In case your answer to Q10 is “NO”, could you please provide reasons?

18. Could you please rate the overall level of satisfaction with the quality of Project DAISY audio-visual textbooks?

Very high High Moderate Low Very low N/A

Quality of implementation

Usefulness

Timeliness

Equal access to boys and girls (equity focus)

Your overall level of satisfaction

Could you please elaborate on your answer/ indicate reasons?

Project “Using textbooks in DAISY format” – Draft Evaluation Report 113.

UNICEF “Using textbooks in DAISY format” – SURVEY FOR STUDENTS

Date:

Grade:

School:

Type of respondent:

CwD

Children without disabilities

Gender:

MALE FEMALE Use the following faces to respond and describe how you feel about each sentence. Please circle the face to answer.

Strongly agree // Agree // Neither agree nor disagree // Disagree // Strongly disagree // N/A – Do not know – No opinion

Strongly agree

Agree

Neither agree nor disagree

Disagree

Strongly disagree

N/A – Do not know /

No opinion

1. I like being at the school 2. My teacher knows how to

use DAISY audio-visual textbooks

3. My teachers help me in the use of DAISY audio-visual textbooks

4. I like to participate in class

5. I participate more in class when not using DAISY audio-visual textbooks

6. Other children help me at school

7. The other students accept me as I am

8. I know how to use audio-visual DAISY textbooks

9. I like studying with audio-visual DAISY textbooks

10. I prefer group activities in class when we do NOT use DAISY audio-visual textbooks

11. I think DAISY audio-visual textbooks help me to study

12. My parents know how to use DAISY audio-visual textbooks

13. My parents help me to use DAISY audio-visual textbooks at home

14. I study more by myself since I started using DAISY audio-visual textbooks

Project “Using textbooks in DAISY format” – Draft Evaluation Report 114.