SABER EMIS Romania Reportwbgfiles.worldbank.org › documents › hdn › ed › saber › ... · Romania EDUCATION MANAGEMENT INFORMATION SYSTEMS SABER Country Report 2016 Key Policy

Romania

EDUCATION MANAGEMENT INFORMATION SYSTEMS SABER Country Report

2016

Key Policy Areas Status

1. Enabling Environment Ministerial Order 5760/2006 established the National Education Database (BDNE), which became the mandatory system for education data collection in 2006. BDNE later was transformed into the Integrated IT System of Romanian Education (SIIIR), which is currently used for all data collection, validation, and analysis. A recently signed agreement between the Ministry of National Education and Scientific Research (MNESR) and the National Institute for Statistics (NIS) has fully institutionalized SIIIR as the sole provider of education data in data collection, validation, processing, and dissemination, also to be directly integrated with the tertiary education sector by 2017. In accordance with the Confidentiality and Data Privacy Law 677/2001, the Romanian Data Protection Agency conducted an extensive data protection audit in 2008. It has since established EMIS compliance with data protection laws, defining responsibilities, roles, and guidelines for the EMIS. EMIS staffing, however, is not yet institutionalized. The hiring and retention of highly qualified and skilled human resources remain a substantial challenge and threat to the sustainability of the EMIS. EMIS positions tend to lack competitiveness and appeal in comparison with the private sector. Moreover, no institutionalized professional development and training plans have been set forth by the MNESR.

2. System Soundness EMIS architecture and design establish the functionality and operability of the Romanian EMIS. The system is able to produce quality and timely data but is not yet fully integrated. The EMIS would benefit from smart and dynamic features such as feedback mechanisms to bridge the information gap between data providers and data users. A fully integrated early warning system would enable teachers, parents, and school administrators to identify students at risk and ensure timely intervention to improve learning outcomes. Additional analytics features of the EMIS may include projections (e.g., student population), what-if analysis, and data associations. To ensure full data coverage, particular importance should be paid to socially disadvantaged groups, such as Roma, because the system currently fails to collect data on them. The EMIS is web-based and has been customized employing open-source Oracle BI Discover software. It is maintained, audited, and updated on a regular basis by the independent vendor Siveco Romania SA. These audits are essential components in ensuring that the system is adaptable to change but would benefit from expanding it to include additional smart and dynamic analytics features.

3. Quality Data Data collected by the Romanian EMIS are comprehensive but would benefit from improved accuracy and reliability measures. The data collected cover a wide range of important indicators in administrative, human resource, financial, and learning outcome data. To ensure timely annual statistics reports, data compliance policies and penalties for schools have been put into place. However, the education indicators are often not clearly defined, causing conflicting results in, for example, reporting of dropout rates. The initial data collection and entry are carried out by administrative staff, who would benefit from additional instruction and systematic training to ensure accurate and reliable data (e.g., on school infrastructure). School inspectorates are tasked to track data production processes to certify that best practice standards are employed.

4. Utilization for Decision Making Despite comprehensive and quality data, the EMIS remains closed and underutilized. The system does not foster data access, sharing, and utilization. The establishment of SIIIR has improved data sharing, transparency, and access to a certain degree, but more efforts should be made, and utilization of the available information remains challenging. Policy makers do not yet utilize data for decision making and could benefit from sharing data with research agencies to design education policies. Currently an integrated data utilization system is not in place for professional development and continuous training for teachers and EMIS staff. Moreover, even though schools, teachers, and parents are given access to education data in the form of national reports, they are often unable to use the data effectively. Once the Electronic National Grade-book System is established and operational, it will enable schools to make use of the data collected by the EMIS in the context of informed decision making to improve learning outcomes of students.

ROMANIA ǀ EMIS SABER COUNTRY REPORT |2016

SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 2

Introduction

The Systems Approach for Better Education Results (SABER) is an established tool designed to support countries in systematically examining and strengthening the performance of their education systems. Part of the World Bank’s Education Sector Strategy, SABER uses diagnostic tools for examining education systems and their component policy domains against global standards and best practices and in comparison with the policies and practices of countries around the world. By leveraging this global knowledge, the SABER tools fill a gap in the availability of data and evidence on what matters most to improve the quality of education and achievement of better results. This report discusses the results of applying the SABER–Education Management Information Systems (EMIS) tool in Romania. The objectives of this report are to examine the system according to key policy areas, identify successes and challenges in the system, and provide recommendations to support the continued advancement of the EMIS in Romania.

Approach of SABER‐EMIS

Information is a key ingredient in an effective education system. SABER–EMIS aims to help countries improve data collection, data and system management, and data use in decision making. SABER‐EMIS assesses the effectiveness of a country’s EMIS, with the goal of informing policy dialogue and helping countries better manage education inputs and processes to achieve overall efficiency and strong learning outcomes. A successful EMIS is credible and operational in planning and policy dialogue, as well as teaching and learning. It produces and monitors education statistics within an education system and has a multifaceted structure, comprising the technological and institutional arrangements for collecting, processing, and disseminating data (Abdul‐Hamid 2014). It is crucial for tracking changes, ensuring data quality and timely reporting of information, and facilitating the utilization of information in decision making. The SABER‐EMIS assessment methodology is built on four key policy areas that are essential to an EMIS and must be assessed to understand and ultimately strengthen the system. Each policy goal is defined by a set of policy levers (actions that help governments reach the policy goal) and indicators (measuring the extent to which the policy levers are achieved) (figure 1). A strong enabling environment lays the foundation for an effective EMIS. Enabling environment refers to the laws, policies, structure, resources, and culture surrounding an EMIS that make data collection, management, and access possible. In essence, this policy area is the context in which an EMIS exists. This defined scope of an enabling environment builds on lessons learned from studies of education management systems.

Figure 1: SABER‐EMIS Policy Areas and Levers

Enabling Environment

System Soundness

Quality Data

Utilization for Decision Making

Policy Levers: legal framework, organizational structure and institutionalized processes, human resources, infrastructural capacity, budget, data‐driven culture

Policy Levers: data architecture, data coverage, data analytics, dynamic system, serviceability

Policy Levers: methodological soundness, accuracy and reliability, integrity, periodicity, and timeliness

Policy Levers: openness to EMIS users, operational use, accessibility, effectiveness in disseminating findings

Policy Areas

Source: Abdul‐Hamid 2014.



System soundness ensures key processes, structures and integration capabilities in an effective EMIS. Education data are sourced from different institutions, but all data feed into and make up EMIS. Databases within an EMIS are not viewed as separate databases but as part of the whole EMIS. Key aspects of system soundness include what data are covered in the EMIS and how they come together in the overarching system. Quality data establish the mechanisms required to collect, save, produce, and utilize information in an accurate, secure, and timely manner. Data quality is a multidimensional concept that encompasses more than just the underlying accuracy of the statistics produced. It means that not only are the data accurate, but that the data address specific needs in a timely fashion. Quality data lay the groundwork for utilization. An effective EMIS is utilized in decision making by all users (parents, students, teachers, principals, and policy makers) across the education system. An EMIS needs to be used so that measures can be taken to improve educational quality. Accurate information on education sector performance enables the design of more informed policies and programs. It is imperative to understand where decision making occurs, if the capacity to analyze and interpret education data exists, and if specific data are available to inform decisions. Using the EMIS data collection instrument, policy levers are scored on a four‐level scale (latent, emerging, established, and advanced) to assess the extent to which both policy intent and implementation are achieved (Figure 2).

AssessingPolicyIntentandImplementation

The EMIS assessment examines policy intent and the degree to which intended policies are effectively implemented on the ground (figure 3). Intent refers to the way in which the EMIS and its overarching purpose are articulated by decision makers and documented in policies and legislation, as well as standards and strategy documents. Assessing intent alone reveals only part of the picture. As such, this EMIS assessment also evaluates policy execution. Implementation refers to the degree to which intentions take place during the day‐to‐day activities of stakeholders (e.g.,

1 Latent

2 Emerging

3 Established

4 Advanced

Figure 2: SABER Scoring and EMIS Development

Limited enabling environment, processes, structure, data management, utilization

Basic enabling environment, processes, structure, data management, utilization

Enabling environment, processes, structure, data management, utilization in place with some integration

Comprehensive enabling environment, processes, structure, data management, utilization, and integration in place, with intelligent analytics

Source: Abdul‐Hamid 2014.

Implementation

Outcomes

Intent

Utilization Processes Institutionalization Budget Human Resources Communication

Policies Vision & Buy‐in Standards Strategy

Documents

Teaching & Learning Management Planning & Monitoring Transparency Governance

Figure 3: Policy Intent, Implementation, and Outcomes Cycle,with Examples



policy makers, county administrators, principals, teachers, and students). Implementation can be observed through utilization of the EMIS by stakeholders, budget allocation, distribution of human resources, availability of professional development activities, communication, and dissemination of information, as well as the extent of institutionalization across the system. Once policy intent and implementation are analyzed, the EMIS assessment explores the results of these two key components, with a focus on system effectiveness and efficiency, in addition to teaching and learning, and management and planning. Strong education systems will ultimately use these outcomes to inform the effectiveness of policies and education strategies and make adjustments as necessary, creating the cyclical process illustrated in figure 3. In Romania, EMIS intent and implementation were assessed through desk research and analysis of system applications and utilization, as well as interviews with a variety of stakeholders (table 1).

Table 1: Measuring Policy Intent and Implementation in Romania

Policy Intent Processes Policy Implementation

Multiple meetings with MNESR

Extensive review of relevant policies, national strategies, standards, and planning documents

Analysis of data quality and comprehensiveness

Examination of professional development activities

Interviews at national, district, and school levels including policy makers and education stakeholders

Methodology

The EMIS assessment methodology consists of a review of written policies and technical documents as well as interviews with key stakeholders across the education system to ensure proper implementation. Research and investigation for the Romania EMIS assessment took place from August to October 2016. The authors conducted a comprehensive review of policies, as well as technical documents and other background materials. To further examine policy intent and implementation, a series of interviews and meetings took place with the following entities:

1. Ministry of National Education and Scientific Research 2. School Inspectorate 3. Teachers and principals in rural and urban areas (in public schools)



Country Overview

Before the 2008 financial crisis, Romania was among the fastest growing economies in Europe, experiencing rapid poverty reduction. Growth has now returned and is projected to be around 3 percent in the coming years. Before the financial crisis, growth was marked by a shift in labor allocation from less productive sectors including agriculture into higher productive sectors including services and construction. Based on the Europe and Central Asia regional poverty line of 5 USD per day (in constant 2005 USD PPP), the poverty rate declined from 2006 to 33 percent. The financial crisis severely disrupted poverty reduction and growth, and the poverty rate did not change from 2008 to 2010. Growth has now resumed and is projected by the IMF to be approximately 3 percent for the coming year. Despite the progress made before the financial crisis and the resumption of growth since then, the poverty rate in Romania still remains among the highest in the European Union (EU).

Romania like other Eastern European countries is experiencing a rapid population decline emphasizing the importance of strengthening the workforce’s education and skills. The United Nations predicts Romania’s population will decline by approximately 19 percent between 2010 and 2050, because of lower birth rates but also emigration; approximately 2.7 million young workers have migrated to other parts of Europe in the past 10 years. Emigrants tend to be higher skilled (World Bank 2014).

Strengthening public administration is central to the government’s national reform strategies. Romania’s public administration is overwhelmed by poor quality public investment, an inconsistent legal framework, poor interministerial relations, poor use and programming of EU funds, and ineffective oversight of state‐owned enterprises. Improving public financial management, tax regulation, and performance of state‐owned enterprises and reforming the judiciary are key areas of engagement with the World Bank.

As part of Romania’s efforts improve public administration, a series of four strategies related to education have recently been approved with heavy emphasis on monitoring and evaluation. The sector strategies are part of the ex‐ante conditionality of an integrated approach to the 2014–20 EU fund programming period. These strategies cover (1) early school leaving, (2) tertiary education, (3) vocational

Table 2: Public Expenditure on Education (2012)

Public Expenditure on Education

Percent of Government Expenditure

Percent of GDP

Pre‐primary 0.9% —

Primary 1.5 0.5%

Secondary 3.3% 1.1%

Tertiary 2.1% 0.7%

Source: EdStats 2016.

Table 3: Pupil‐Teacher Ratio (2012)

Pupil‐Teacher Ratio

Primary 17.56

Secondary 12.27 Source: EdStats 2016.

‐10

‐5

0

5

10

2002 2004 2006 2008 2010 2012 2014

Percent chan

ge

Source: World Bank 2016.

Figure 4: Percentage Change of Real GDP Growth inRomania



and technical training, and (4) lifelong learning. They coordinate the government’s efforts at achieving the Europe 2020 goals.

Increasing enrollment, especially for minorities and improving quality of education, are key challenges that Romania’s education system currently faces. Enrollment rates for secondary, pre‐primary, and especially tertiary‐level education have grown since 2000, although tertiary education has declined since the 2008 financial crisis. Despite this progress, significant disparities in access to education exist. For example, only a small proportion of Roma men and women complete secondary school, 12 and 6 percent, respectively (World Bank 2015). Ninety‐five percent of Roma leave school early.

Romania has made significant progress improving learning outcomes for 15‐year‐old students, as measured in the 2015 Organisation for Economic Co‐operation and Development (OECD) Programme for International Student Assessment (PISA) study, but a large percentage still do not achieve minimum proficiency. Between 2006 and 2015, the proportion of students who achieved PISA level 2 or higher in all three subjects increased substantially. For example, in 2006 only 46 percent of students attained this basic level of proficiency in reading, but by 2015, 61 percent attained this level. Similar improvements occurred in mathematics and science as well. However, more than one‐third of Romania’s 15‐year‐olds still do not achieve minimum proficiency.

A strong EMIS is needed to monitor and evaluate the education sector strategies, to contribute to improving public sector management more generally, and to help achieve the Europe 2020 goals. Monitoring indicators for all four strategies rely on data about education institutions. The role of the EMIS, especially for the early school leaving strategy, is central to providing the data needed to measure whether the strategy is achieving its goals or not.

Table 4: Education Data in Romania (2012–14)

Absolute enrollment number

Gross enrollment ratio (both sexes)

Percentage of female students (private and public)

Percentage private institutions

Pre‐primary 581,144 — — 2.7%

Primary 807,047 95.5% 48.2% 0.4%

Secondary 1,609,259 94.8% 48.4% 1.2%

Tertiary 578,706 52.3% 53.8%

—

Source: EdStats 2016.

Figure 5: Percentage of 15‐Year‐Old Students Achieving Minimum Proficiency in PISA by Domain (Level 2 or Above)

47 465353

60 5959 63 6360 61 61

0

25

50

75

Math Reading Science

Percentage

Source: Based on OECD PISA 2015 data.

2006 2009 2012 2015



Romania EMIS Results

This section presents the main results of EMIS diagnostics described in the previous sections. Results and scores for each policy goal are presented, along with supporting evidence.

PolicyArea1:EnablingEnvironment

Established Romania’s enabling environment was assessed in the following areas: (1) Legal Framework; (2) Organizational Structure and Institutionalized Processes; (3) Human Resources; (4) Infrastructural Capacity; (5) Budget; and (6) Data‐Driven Culture.

Ministerial Order Number 5760/2006 established the National Education Database (BDNE) in 2006, which was later upgraded to the Integrated Romania Education Information Technology (IT) System (SIIIR), which has recently become the single data provider in the education system. The BDNE was introduced in 2005 and became operational following Ministerial Order 5760/2006, which established the BDNE as the official system for data collection, validation, and processing. However, in 2013 the BDNE was replaced with the Integrated Romania Education IT System (SIIIR), which has been in use since the 2014/15 school year. Recently, an agreement between the Ministry of National Education and Scientific Research (MNESR) and the National Institute for Statistics (NIS) has been signed stating, among other things, that the SIIIR is the single data provider for the education system. The organizational structure of the Romanian EMIS is well established and supported by the operational manual. There is a clear workflow and designated staff roles and responsibilities. The EMIS architecture facilitates information flow from the central to the regional and local levels as well as vice versa. The operational manual sent to all school inspectorates and schools requires the nomination of regional and local SIIIR administrators and users, who will carry out the tasks outlined in the operational manual. The system is supported by functional hardware and software. The hardware and software needs of SIIIR have been established in the project’s tender documents, such as the Task Book. This process included the replacement of almost all paper‐based data collection instruments with digital means, such as SIIIR. Currently the Romanian EMIS uses Oracle BI Discoverer software and has a valid support, maintenance, and development contract with Oracle until the end of 2016. Internet connectivity is well established in Romania, which facilitates the use of digital technology. Even though Romania lacks an official written policy for data backup, procedures and processes are in place to manage and archive data. Backup procedures have been implemented, but they are not yet fully institutionalized. In general, education data are fully archived without a time limit. However, a data backup process is not specified at this point, and a written policy is not available. A standardized backup process would ensure the availability of data and may be useful to evaluate long‐term education developments. A draft government decision has been prepared to establish EMIS staff as a distinct unit. The Romanian EMIS is not staffed with quality assurance specialists, communications specialists, or data analysts. These are crucial roles in the EMIS structure and should be taken into consideration while making staffing decisions. In August 2016, a Government Decision Draft was written to address this and introduce a separate unit for IT within the MNESR. Such a unit would need to be included in the annual education budget.



The EMIS is included as a separate item in the in the MNESR annual budget, but a general budgetary EMIS framework is still needed. The infrastructure needed to collect and validate data as well as staffing are financed by the MNESR annual budget. The budget currently records a separate budget for the physical EMIS structure (hardware, software, and technical support) but does not specify EMIS staffing expenses or reporting costs (publications, costs of website maintenance, etc.). Maintenance of the EMIS system is currently included in the MNESR annual budget. This lack of specificity is not a problem related to funds, but rather is more about lack of a general framework for a number of years and more importance that should be given to the needs of using the system. No provision is in place for professional development of EMIS staff despite annual evaluations. According to the standards of the Romanian Labor Code, annual staff evaluations are carried out for all EMIS personnel. However, these are not used for targeted professional development and trainings. Only some inspectorates organize ad‐hoc trainings in their counties for those responsible for data collection and data entry. The National SIIIR Working Group, established in October 2016, is tasked to assess what EMIS activities are currently carried out and how to move forward. This also includes the evaluation of the need for professional development and learning materials. Often highly skilled staff are financed by donors. In general, the financing of highly skilled staff has so far been provided by donor agencies, namely, the EU. Often IT specialists are paid on a temporary basis, drawing on the funds of ongoing EU operations. Overreliance on donor funds may decrease EMIS financial sustainability and can potentially overlook important staff positions that should be filled by such quality assurance specialists, communications specialists, or data analysts. Given the absence of clear professional development and pay level and staffing policies, it is challenging to attract and retain highly specialized staff. Currently no policy is in place regarding EMIS staffing including number, training, or competencies, and it is currently difficult to attract and retain specialized staff because of pay. The payment of IT specialists to staff the system is too low to be competitive with the private sector. In addition, the positions are not recognized as professional IT that is cutting edge or can bring value‐added to the ministry but rather as part of the general administration of the ministry. Financing has not been allocated for staff training or professional development, which often encourages staff to look for outside professional opportunities. In fact, in September 2016 EMIS lost a considerable amount of human resources. The government aims to change both the attractiveness of employment within the ministry and, thus, attract and retain more specialized staff. EMIS data are disseminated to the public only in the form of official reports and national statistics; parents are also able to access data about their schools online. Generally, education stakeholders need to actively look for education data, which are not automatically supplied by the government. Often the available data are in aggregate terms. Sharing EMIS data with other ministries or organizations takes place only with bilateral agreements. Schools and parents do have access to data regarding school infrastructure, staffing, and facilities. Parents and teachers can access data about national examinations at the individual level. Many countries such as Chile and Cameroon have active education data strategies to ensure that all education stakeholders are reached (box 1).



PolicyArea2:SystemSoundness

Established Romania’s EMIS soundness was assessed in five critical areas: (1) Data Architecture; (2) Data Coverage; (3) Data Analytics; (4) Dynamic System; and (5) Serviceability.

The architecture for Romania’s EMIS is designed to ensure system functionality. The system is based on a clearly defined framework with specifications guiding the structure of all underlying data sources. The database is structured according to relational standards and is well documented and secure. The system is based on off‐the‐shelf Oracle software that has been customized and primarily managed by Siveco Romania SA, an independent vendor. The vendor was selected competitively following Romanian Public Acquisition Law requirements. The contract with Siveco Romania SA requires monthly maintenance tests to ensure data security and confidentiality. These factors establish a sound EMIS and ensure compliance with privacy requirements. The EMIS is dynamic and able to respond to changes in data requirements and education sector structure. Stakeholders periodically meet to review the portfolio of education indicators, review existing education reports, and identify new data requirements. The system is able to integrate new student demographic data and changes in school organization as well as other structural changes to the education system. The system can also add new types of functionalities and integrate with older versions of the database. However, the system does not yet incorporate feedback mechanisms from the local to the central level and vice versa. Such feedback mechanisms are essential in any well‐functioning EMIS (figure 6). At the same time, a need is seen for a balance of tracking data at the national and subnational level with support for end users at the local level for better school management, effective teaching, and assessment of student learning.

The Ministry of Education in Chile has introduced scorecards for schools to monitor learning improvements. These scorecards are made available to the public and allow for annual school classifications based on student learning outcomes. The schools are grouped according to socioeconomic standards (low or high), rural or urban. Points are awarded based on different indicators such as social, personal, and academic. This allows for a fair comparison between schools, and the best performing schools are rewarded with financial benefits. In addition, these scorecards are made available to the public on the Ministry of Education’s website. This allows stakeholders to search for school results by name or unique school ID number and fosters openness as well as transparency (Bruns, Filmer, and Patrinos 2011). Even in countries with low resources, data dissemination strategies are in place. In Cameroon, the Ministry of Education publishes education data on the radio, in the local press, and on television. These are highly effective channels of publication with a deep reach within the population. Many other countries do not make sufficient use of such channels of communication and therefore may forego the chance of reaching some potential data users. Making data available is a crucial aspect in data usage, but just as important is communicating the findings to the general population even if they may not be actively looking for it (UNESCO 2012).

Box 1: Examples of Active Data Dissemination Strategies



The MNESR collects enrollment data, and the Ministry of Public Finance (MoPF) collects financial data; both provide data to the NIS. Both ministries collect data relevant to schools, teachers, and students, but the two databases remain unlinked. Expenditure data are collected by the MoPF through its own mechanisms at a very detailed (budgetary) level. Enrollment and other education data are collected by the MNESR. A high benefit exists in combining the two independent education databases. Relating learning outcomes to education data provide valuable insights for the allocation of resources. If collected properly, validated, and verified, this ensures a high level of accuracy informing on improvements to student learning outcomes. Both ministries provide their datasets to NIS, which then forwards statistical data to the EU. The EMIS already covers data on students and school facilities, public financing, teachers, and national assessments but should be expanded to achieve full education data coverage. The National Evaluation and Examination Centre is responsible for the processing and distribution of sensitive documents for the national assessments, in cooperation with the Special Telecommunications Agency. Student‐level data include basic demographics such as age and gender as well as data on socioeconomic status and ethnicity. Officially, regulation does not yet mandate data collection on socioeconomic backgrounds or ethnicity, but the information is habitually collected. Data on student transition rates into higher education are also recorded. However, no data on health issues, such as immunization of students, are collected except for learning and physical disabilities. A wide range of data on school facilities and infrastructure are collected, but data on implementing and achieving milestones in school plans are not. Financial data are collected only on government spending; private expenditure information including tuition, grants, donations, and other school revenue is not. The data do not yet fully cover teacher data. Data are available on teacher salaries, experience, and qualifications but are not collected on professional development and trainings. Schools and students benefit directly from effective teacher training and professional development plans. Teachers are instrumental in the classroom and, thus, for learning outcomes. A good example of a functioning system is the teacher information system in Bihar, India, which focuses on teacher qualifications and professional development (box 2).

Feedback

Research/analysis Reporting

Data

collection Data

utilization Schools

Annual

School

Census

Data

validation

& analysis

Decision making

and policy making

School Census &

other outcomes

and policy actions Central

Government, Local

Government, &

School Level

Source: Adapted from Abdul‐Hamid 2014.

Figure 6: Information Feedback Loop

Planning

& design

of Census



The EMIS does not yet collect data on disadvantaged social groups such as Roma. Although the SIIIR collects data on students and school facilities, public financing, teachers, and national assessments, data on disadvantaged groups, including Roma, are not collected. Given that 95 percent of Roma leave school early, these are important data and need to be accounted for in the system architecture. Special attention should be paid to disadvantaged communities. Although capacity to compute and provide descriptive analysis of the EMIS data exists, processes are not established to analyze associations or correlations between indicators. The Romania EMIS has processes to conduct descriptive data analysis (e.g., means and totals) as well as cross‐tabulations to summarize indicators across different values of other indicators. Data interpretation and manipulation are generally carried out by the IT in Education unit or manipulated by the maintenance provider, Siveco Romania SA. Capacity is lacking to do more in‐depth analysis including understanding the relationship between multiple indicators and forecasting or projections. Professional development and software are needed to conduct these analyses. In Romania, no mechanism exists to flag potential issues or early warnings for the performance of students, teachers or schools. The aim of any successful EMIS is to provide education stakeholders with the data and information necessary to better learning outcomes for students. An important factor is an early warning system that prevents students from falling behind and tracks student progress. Some advanced systems such as Maryland’s (box 3) have successfully implemented an academic indexing procedure, which identifies and flags students at risk. This is an immensely valuable tool for teachers.

The East Indian state of Bihar has integrated what was initially a teachers’ attendance monitoring system into a professional and human resource development one. The Teacher Education Management Information System (TEMIS) uses modern technology to track teachers’ training and has successfully established a link to the technology‐based pre‐service training. In the future all teacher training will be tracked through that system. This provides administrators with more transparency in the teacher selection process for in‐service trainings as well as professional development programs. TEMIS includes data on the teachers’ educational level, professional qualification, and personal history in addition to current school posting, commencement date of teaching service, training status, and foreseeable retirement date. In addition, TEMIS is a longitudinal tracking system of teachers’ qualification and performance. Source: World Bank 2015.

Box 2: Teacher Education Management Information System in Bihar, India



The Romanian EMIS has not yet established a real‐time procedure to monitor and record student and teacher performance. To ensure student learning and effective human resource allocation, the EMIS should focus on student and teacher needs. This starts with implementing efficient deployment mechanisms for teachers and tracking student learning. Policy makers and school administrators should be able to rely on the EMIS to inform them at any point on the academic and professional progress of students and teachers. This data in combination with school inspection data can lead to direct improvements in student learning and course completion rates. Tracking schools, teachers, and students simultaneously identifies potential correlations between student learning and school or teacher performance. With the availability of tracking data, policy makers will be able to answer the question if there is a link between low‐performing schools and teachers. Such an EMIS also enables decision makers to identify areas for improvements and map schools accordingly. These indicators may be linked to the achievement of milestones in individual school development plans. Therefore, it is necessary to collect data on school, teacher, and student behavior even in established systems. Howard County in Maryland, U.S.A., believes that monitoring was a key factor in improving learning outcomes (box 4).

Howard County in Maryland states that one of the key success factors to improve student learning was monitoring. Daily and weekly monitoring (of students, teachers, and schools) contributed greatly to the achievement of a solid foundation for data‐driven target setting. The presence of teachers and effective classroom monitoring was crucial in the process. Countries with similar capacities may benefit from the experiences in Howard County. Source: Howard County 2015.

Box 4: An Example of How Regular MonitoringImproves Education Outcomes

High Risk 12 students (3.5%) Medium Risk 35 students (10.3%) Low Risk 286 students (84.1%) No Data Available 7 students (2.1%)

Academic indexing supports all education stakeholders in identifying students at risk. The value of data is even apparent at the microlevel. For instance, in Maryland a risk index was developed, which identifies students at risk. The figure to the right illustrates the color coding for students at no risk, medium risk, and high risk. When parents and students have real‐time access to student learning data, as they do in Maryland, then they are able to track their own progress and identify deficiencies in their performance early. Just as importantly, teachers may use data to track progress toward Student Learning Objectives. By tracking each of their students’ progress in a computerized system, it is easier to identify and carry out necessary intervention for students at risk. By comparing student records against their classmates, but also longitudinally, support can be given in a more targeted, efficient, and effective manner. Principals and school administrators actively use data to evaluate teachers, monitor school progress, and manage school plans. Policy makers use data to monitor education quality and equity, improve accountability, and gauge effectiveness of policies and programs. Source: Provided to authors by Cecil County, 2014.

Box 3: Academic Indexing in Maryland



The system may benefit from improvements in formal external quality assurance procedures. To maintain quality and system soundness, internal and external reviews should be performed. The International Organization for Standardization Series ISO 9000 framework acts as the backbone of an EMIS (figure 7). Assessing the management of an EMIS system ensures the continuity of operations. More specifically, the government is the managing authority responsible for the system, which manages its architecture and infrastructure and the realization of such products as statistical reports. Processes are in place that focus on data quality, collection monitoring, processing and dissemination of education statistics, and inclusion of data quality in statistical planning. Currently, Romania does not follow internationally recognized quality assurance measures such as ISO 9000 nor is it subject to external reviews by auditing or quality management agencies.

PolicyArea3:QualityData

Established The quality of data captured by Romania’s EMIS was assessed in four areas: (1) Methodological Soundness; (2) Accuracy and Reliability; (3) Integrity; and (4) Periodicity and Timeliness.

Romania’s EMIS is equipped with operation manuals, but some definitions of indicators could benefit from a higher level of standardization and clarity. Official operations manuals exist that define the system’s specific terminology related to data collection, data management, data fields, indicators, and archival procedures. However, not all definitions are clear, resulting in confusion on how to define certain

Figure 7: Use of ISO 9000 Quality Processes to Assess the Soundness of an EMIS

Source: Adapted from ISO 9000 (2016).



indicators. Moreover, some indicator definitions are missing. For example, SIIIR has introduced new modules (e.g., on school infrastructure), which are not accompanied by explanations on how to calculate and/or differentiate data (e.g., school areas). In theory, data collectors and data management staff should be aware of how data are supposed to be collected and defined. This should include any differences between concepts and definitions used in source data versus the EMIS database. However, in practice this is often not the case, which is why some important indicators are not reliable and accurate. This is particularly true with regard to school dropout rates. Dropout rates are important and are basic indicators in any set of education statistics, but the Romanian EMIS fails to accurately calculate these because of a lack of a clear definition. Without appropriate systemic data training and clear definitions, data may be conflicted. Any error made at the data collection and data entry level reduces data quality. Systematic training and instruction for data collectors would directly benefit data quality. However, often data collection and entry are done by administrative staff with minimal training, which may cause misunderstandings of the definitions. This holds particularly true in rural areas, where data entry is often conducted by secretary‐level staff with minimal statistical and data entry training. In addition, in large schools—especially in urban areas, where the challenge is to input large volumes of data—professional development mechanisms are limited for data collectors or statistical staff to enhance the accuracy of the collected data. Limited communication of the existing data definitions to the data‐collecting units potentially lowers data quality. In general, the Romanian education system classifications are based on international guidelines, mainly those of the EU and UNESCO’s Institute of Statistics International Standard Classification of Education. The MNESR and NIS follow their procedures and methodologies set forth by these institutions and their partners. The MNESR and NIS coordinate on data definition because the NIS supplies data to European and international statistical institutes. In theory, Law Number 677/2001 prescribes that users are made aware in advance of major changes to methodology, source data, and statistical techniques. It is questionable to what extent users are notified of any changes and the depth of the reach within the pool of data collectors. The government has established data procedures aimed at ensuring accuracy and reliability, but challenges remain. The design of the data collection instruments is intended to facilitate easy completion and enable direct data entry into the SIIIR web‐based front end. Even through a registry of school units exists, a request of a list of high schools classified by selected school indicators raised reliability questions because of different ways of calculating those indicators at the school level (e.g., average completion rate, average passing rate at the Baccalaureate exams) and/or lack of updated information at the school or inspectorate level. A common understanding of definitions at the level of all schools is an important step toward comprehensive and quality data. Without it, data are likely to be missing and to lower the accuracy and reliability of the education statistics. The Romanian EMIS aims to promote accountability and transparency. Accountability and transparency are essential in managing the interplay of different education stakeholders in assigning and identifying data responsibilities. The public is aware of and able to access education statistics, which include publicly available documentation on how education statistics have been collected and compiled and their utilization. In theory, this encourages transparency and accountability, but it is questionable how often these data sources are actually used by parents, teachers, policy makers, and education researchers. The government is in the process of designing extensive data validation for school inspectors. The need exists to establish very clear and accurate data validation and verification procedures. Ensuring quality



source data directly improves data quality overall. School inspectors are responsible for data collection, validation, and verification, but a lack of clear validation procedures persists. In fact, school inspectorates are currently in the process of designing extensive inspection procedures and SIIIR inspection checklists for use in schools. The school inspectorates are tasked with tracking which schools have and have not yet responded to the data collection requests. Schools have access to SIIIR, and with the appropriate built‐in data validation mechanisms, a high level of data quality can be achieved. Granting data providers (e.g., teachers, school administrators) access to education data makes them accountable for the data. Analytical data are entered directly into the SIIIR’s web‐based application, and every school is granted access to the system via the internet. It is essential to include built‐in data validation mechanisms in the software at the point of data entry. For instance, mandatory reviews of entered data before final submission can directly improve data quality. Penalties are in place in case of disclosure of confidential data, but clear guidelines on ethical behavior have not been established. A special focus is placed on privacy and confidentiality measures because they are regularly audited given the maintenance contract with Siveco Romania SA. In addition, Law 677/2001 and the EMIS job contract prescribe penalties if EMIS staff disclose confidential information. However, clear guidelines are not in place for staff regarding unethical behavior (e.g., misuse of statistics or alteration of statistics) or clear guidelines about conflicts of interest for EMIS staff. EMIS staff are not protected by an officially endorsed code of professional ethics. There is also no internal or external review process for research on methodology or use of data; staff do not attend conferences or publish methodological papers. The EMIS is able to produce timely education statistics. Data collection from schools is conducted generally twice a year. The data are updated periodically to reflect if students transfer to a new school or drop out or if new students enter the system. Final education statistics are disseminated within 6 to 12 months of the start of the year. The periodicity and timeliness of data appear to be sufficient for data users’ needs.

PolicyArea4:UtilizationforDecisionMaking

Emerging The utilization of Romania’s EMIS was assessed by examining four areas: (1) Openness; (2) Operational Use; (3) Accessibility; and (4) Effectiveness in Disseminating Findings.

The EMIS in Romania is established, but its data often remain unused. Any EMIS is only as good as its utilization. A well‐functioning EMIS utilizes data at every level of the decision‐making process, starting at the school and regional levels and going all the way to the central level. In Romania, it is exactly in education data utilization that the EMIS is still challenged the most. Evidence shows that the data are not used by policy makers to introduce data‐based education policy with the ultimate objective to improve learning outcomes of students. Many Latin American countries have established the utilization of information obtained by an EMIS at the core of any education policy decision (box 5).



The MNESR provides access to the EMIS, but the data are not used widely. Access to education data is not sufficient. Education stakeholders need to be trained to digest and effectively utilize the data available to them from the EMIS. The Romania EMIS focuses on the accessibility of data but not its usage. Indeed, data are accessible not only to the MNESR administration but also the school inspectorate, regional education offices, schools, parents, researchers, nongovernmental organizations, and international organizations. Some EMIS reports are publicized via the internet and pamphlets on the internet. However, no training is provided to EMIS users on how to access and use the system. Effectively, users do not benefit from the potential that the EMIS holds. The establishment of SIIIR has improved data sharing and transparency but is not automatically

translated into data utilization. SIIIR has improved the education governance system by evaluating the

system and schools. The EMIS provides reports on school performance as well on student performance

including graduation and transition rates. The Education Status Report is published annually and provides

an overall evaluation of the education system based on the EMIS, as well as on other sources. Education

agencies including regional offices and the inspectorate use EMIS data for evaluating schools and for the

allocation of financial resources and teaching staff. A need is seen to build additional capacities and

processes at all levels to enable stakeholders to make use of the data. Ideally, capacity building should be

increased at the school level to help those at the “frontline” of education (teachers, school staff, et al.)

and nourish a data culture.

The Romanian government has a data dissemination strategy in place, but it has not yet triggered the desired data utilization effect. The strategy involves bulletins and newspapers as well as annual reports. Data are disseminated according to a fixed schedule that is also communicated to the public in advance. Typically, the release of a comprehensive report is preceded by an informational release, to make it easy for the audience to digest the information. The government aims to report data in a manner that is easy for the public to understand. Metadata are also collected and disseminated, although not yet updated for 2016. Data that are not published but nonconfidential are available upon request. Despite the availability of data, the utilization remains low. A general framework of education data sharing at the central level is needed. Data sharing can reap benefits far beyond the education system. Education statistics represent valuable information for other ministries such as the Ministry of Health and Ministry of Labor. Many Ministries of Education have started data‐sharing initiatives, which benefit a broad range of stakeholder groups (box 6). In Romania, data sharing takes place only under individual agreements settled with other institutions and ministries. For

Many Latin American countries have successfully established education systems that base their policy decisions on the information and data obtained through an EMIS. Chile is at the forefront of data‐driven education systems and has institutionalized an education system, where decision making at every level (central, regional, and local) is deeply founded on data. In an educational environment, where most of the schools are privately run, data are crucial to ensure that all schools, private and public, comply with learning standards set for by the central government. Funding for schools is also distributed according to education data. Chile has established a well‐structured private school system with an advanced voucher system. The vouchers are allocated to individuals, and so the decision on where parents send their children to school has a direct effect on the funding for schools and gives importance to their decisions. For parents to make informed decisions, the Ministry of Education in Chile publishes a school ranking every year, which makes schools comparable for parents.

Box 5: Data Utilization in Chile



national reports, the government generally publishes data related to school network, educational areas, levels, specializations, and results for national examinations on their website, data.gov.ro. Although schools use the EMIS to some extent, they are not using it to its full potential. Schools tend to use the EMIS for school improvement decisions and for financial management using the EduSal financial module in the EMIS. However, they do not use it for allocating resources, to evaluate faculty or student progress. They currently do not use it for teacher evaluation or to make teaching decisions or to compare their schools’ academic performances to other schools, but these will be possible when Electronic National Gradebook System (ENGS) is implemented.

Data sharing should be extended to parents and teachers. It is crucial that data sharing be extended from the central level to regional and local education stakeholders. Currently, data‐sharing strategies are not established with parents and teachers except for the dissemination of general reports and final statistics. School administrators, however, are granted access to their own data, actual and historical, in the SIIIR. The system includes data on school networks and individual school such as location, infrastructure, facilities, number of teachers, students, and classrooms, but no school performance data. Data on national examinations, Grade 8 or Baccalaureate, are published on an aggregate level and can be accessed at the individual level by any parent or teacher. Australia is a good example of how individual school data can be easily communicated to the public and shared (box 7).

In Australia, the mySchool platform (myschool.edu.au) provides education stakeholders, including parents and the community, with the opportunity to compare and assess individual schools. The Australian Curriculum, Assessment and Reporting Authority calculates an Index of Community Socio‐Educational Advantage score for each school, which is accessible online. The online tool enables fair comparisons between schools across Australia and incentivizes teachers, parents, and students to make use of the tool. The website mySchool thus supports data collection, analysis, assessment, and utilization of education data. Access to education data for the broader community raises data awareness and inspires a data‐driven culture at all levels: national, regional, and local. MySchool also provides a detailed analysis of enrollment numbers, student background, and staff. In addition, the mySchool website also provides information on financial resources and expenditures of every school. The platform is designed to enable education stakeholders to make effective use of the data. Source: ACARA 2016.

Box 7: Data Access and Usage for All Education Stakeholders

Many countries have started initiatives of data sharing with different ministries. The aim is to utilize data that can be gained through the annual census and other education data sources. For instance, data on vaccination coverage, enrollment in school sports programs, and students’ height and weight can help paint a picture of the overall health of the youth in any given country. For instance, the Fijian Ministry of Education has established a Body Mass Index page to collect data on the height and weight of students. The intent of the initiative was to use this as a vehicle to engage the Ministry of Health and enhance the prospects of data collaboration. Collaboration with the Judicial Ministry, which is responsible for birth certificate IDs, may reflect data on children‐out‐of‐school and possibly introduce a longitudinal tracking system.

Box 6: Example of Linkages of Education Data with Other Ministries



Summary

Romania has established a functioning EMIS system. Ministerial Order Number 5760/2006 has successfully laid the foundation for a legal framework of a National Education Database (BDNE), which later was developed into the Integrated Romanian IT System (SIIIR). Recently SIIIR has been institutionalized through a new agreement between the Ministry of National Education and Scientific Research and National Institute for Statistics (NIS). The system is well established, supported by a sound EMIS structure, and produces comprehensive data. However, effective data utilization remains challenging. The education data system has been designed to ensure system functionality, but additional features such as early warning mechanisms and additional quality assurance procedures should also be included. The organizational EMIS structure ensures the production of data that cover a wide range of indicators: administrative, financial, human resource, and learning data. However, the EMIS does not yet have the ability to track teacher and student attendance or flag potential early warning signs in student, teacher, or school performance. The system is dynamic and thus able to respond to changes in demands on the education system, but currently lacks effective feedback mechanisms reaching from the central to the local level. The Romanian EMIS produces comprehensive data but may benefit from more accuracy and reliability measures. A review and standardization of definitions may be beneficial to ensure that international best practices are in place and data can be shared and publicized accordingly. School census data are produced at least twice a year and made publicly available within six months. Confidentiality measures are in place, but more need exists for clear guidelines on ethical behavior with regard to education data. This assessment highlighted some areas for improvement in the current EMIS, in particular with regard to data utilization. Timely, accurate, reliable, and useful data should be the base for informed decision making, but in Romania data often remain unused. The Romanian EMIS still requires some improvements in the area of data utilization. Extensive data sharing is a key component in the data utilization process.



Recommendations and Proposed Activities

This section presents a set of recommendations and proposed activities based on the assessment of the EMIS in Romania (table 5).

Recommendations and activities aim to improve the overall EMIS functionality in a sustainable and effective manner, to ensure better access and use of information for decision making, planning, and student learning. Future activities to improve the EMIS should be strategically designed such that they incrementally boost dimensions of the EMIS to a more advanced level, ultimately improving overall EMIS functionality in a sustainable and effective manner. The Strengths Weaknesses Opportunities Threats (SWOT) identification (table 6) summarizes key points from the needs assessment and informs recommendations.

Table 5: Romania EMIS Rankings

1. Enabling Environment Established

2. System Soundness Established

3. Quality Data Established

4. Utilization for Decision Making Emerging

Table 6: Romania EMIS SWOT Profile

Strengths

Signed an agreement to fully institutionalize SIIIR as the main education data system

Well‐established enabling environment and system soundness

Established quality data

A sound web‐based data system with the potential for real‐time data

Support and buy‐in from international organizations, in particular the EU

Weaknesses

Lack of clear definitions resulting in lower quality data

Slow progress integrating human resources and financial data with the EMIS

Lack of collaboration between ministries and institutions

Lack of professional development plans for EMIS staff

Opportunities

A sound system with comprehensive data, which is ready for utilization

Expansion of assessment and instructional data programs to closely monitor learning outcomes

Exclusively web‐based applications function well with the highly developed internet infrastructure in Romania

Expanding ministry capacity by creating new units; e.g., the potential for a new IT unit for education has been recently discussed

Electronic National Grade‐book System (ENGS) will enable schools to take full advantage of the EMIS

Threats

Need for data utilization at every level of the decision‐making process

No systematic strategy for country‐wide data utilization training for schools, teachers, and parents

Lack of appeal and competitiveness of EMIS positions in the IT department of the MNESR

Lack of information sharing within the government may prevent important development opportunities

Separate data collection on financial data of schools by the Ministry of Finance requires strict data‐sharing policies



Recommendation1:ImprovingDataUtilization

Romania has established an operational EMIS; building demand for its data presents a key next step. The Romanian EMIS produces a wide range of data that are often underused. This assessment emphasizes the need for data utilization across the entire education system. Considerable opportunity exists to improve system‐wide efficiency, teaching practices, and student learning. The aim of any EMIS is to provide reliable, accurate, and useful data for decision‐making process; and an EMIS is only as good as its usage. The EMIS would benefit from feedback loops. Feedback loops ensure the connection between data providers (e.g., teachers, students, parents, and the community) and data users (e.g., government, education policy makers, teachers, and principals). They facilitate data system openness by granting access to a broad spectrum of data users. Information needs to flow from the central to the regional and local level and vice versa. Schools, teachers, and parents need to be presented with education data in such a manner that they can utilize the information and improve learning outcomes. The EMIS in Maryland, U.S.A., provides effective feedback to schools in the form of the School Progress Index. It is also used by policy makers to identify schools in need of intervention (box 8).

Data allow for smart decisions and efficient resource allocation. Data need to be fully integrated into the EMIS to allow for strategic planning and efficient utilization. Data offer detailed information on the needs of resources and with a central planning tool, which facilitates the use of scalable options. These will improve efficiency and offer cost reduction opportunities, for instance, in the area of procurement of school equipment such as computers. The possibility of effective and targeted strategic planning and resource allocation underscores the benefits of an EMIS. High‐quality data are necessary for effective data utilization. Effective data utilization is possible if the data are reliable, accurate, and useful. In Romania, experience has shown that reporting on different indicators at the school level (e.g., average completion rate, average passing rates) can be challenging. This clearly indicates a stronger need for clearer definitions to ensure high‐quality data. A review of current definitions could ensure that they comply with international best practice standards and are utilized accordingly. High‐quality data can be obtained by institutionalizing a code or standardized definitions and guidelines that ensure the compliance with internationally accepted best practice norms. In addition, best practice standards also plan for data utilization and data sharing. The definitions should

In Maryland, U.S.A., the School Progress Index is a successful feedback mechanism for schools and is utilized on the central, regional, and local levels to identify potential need for intervention. It evaluates schools on a continuous scale based on the variables of Achievement, Growth, Gap Reduction, and College‐ and Career‐Readiness. It also makes results of each school available publicly via the annual Maryland Report Card, which also contains demographic data, enrollment and attendance rates, absentee rates, student mobility, teacher qualifications, and data about students receiving special services. The index identifies schools for intervention, support, and recognition, depending on their progress. The state affords top‐performing schools with greater flexibility, while lower‐performing schools receive progressively more prescriptive technical assistance, targets for performance, and monitoring. To effectively position schools for success, national, state, and local governments provide schools with data that they can use to benchmark their students’ performance against student performance metrics at local and national levels, as well as tools to organize and analyze those data.

Source: MSDE 2012.

Box 8: Feedback to Schools in the Form of the School Progress Index in Maryland, U.S.A.



be aligned with international standards to ensure their effective utilization and comparability across time and potentially also borders. Capacity building is needed for schools on how to take full advantage of the SIIIR and ENGS, when implemented. Although schools use the EMIS to some extent, they do not use it evaluate their performance with other schools or for informing teaching. The establishment of the ENGS will improve the potential for schools to use the data, but capacity building at the school level would help ensure the EMIS is fully used; the school level is important because this is the point of contact with student learning. The United Kingdom was set to implement a similar database, enabling all education stakeholders to compare and analyze school performance (box 9).

Opportunities exist for new mechanisms to leverage the quality of Romania’s EMIS that may help improve student educational outcomes: (1) establishment of an “early warning system” to identify children at risk, (2) improve tracking of disadvantaged groups including Roma, (3) longitudinal tracking, and (4) teacher training to use EMIS data to improve teaching. These four mechanisms build on the existing EMIS and present opportunity to improve learning outcomes of students using an EMIS. The establishment of an “early warning system” to identify children at risk of dropping out or poor achievement should be explored. The SIIIR tracks students and their performance across time. Given the rich data collected by the SIIIR, an early warning system to help identify and predict future problems in the performance of schools, students, and teachers would help policy makers intervene earlier and more effectively. An important factor is academic indexing for students at risk. The academic indexing system in Maryland, U.S.A., is an excellent example of how early warning signs are identified and communicated to teachers, who then have the ability to intervene early to prevent students from dropping out or failing courses. EMIS has the potential to track outcomes of disadvantaged groups including Roma to help improve their education outcomes. Currently, SIIIR does not collect specific data on Roma, but given their specific challenges in the Romanian education system, such data are crucial. Data should be collected and utilized in the design of effective education policies targeting disadvantaged social groups. These will address Roma‐specific issues to ensure improved learning outcomes and effective education. Other countries such as Australia face similar challenges and address these by putting a particular focus on marginalized groups (box 10).

The United Kingdom’s Ministry of Education has set forth ambitious plans to improve data access, sharing, and utilization. First, it aims to unite all eight existing databases to establish a single depository for all education data. It aims to bring “together for the first time school spending data, school performance data, pupil cohort data, and Ofsted [Office for Standards in Education] judgments in a parent‐friendly portal, searchable by postcode” (U.K. Department for Education 2012, 22). It means to facilitate comparison between schools and analysis of school performance. Even though the project has been postponed and is not yet operational, it demonstrates the importance of data access, transparency, and utilization in advanced education systems. Source: du Preez 2012.

Box 9: An Example of Plans to Fully Integrate Education Databases and Effective Community Outreach



The system would benefit from longitudinal tracking of students. Longitudinal tracking of students allows for important policy conclusions, which can be utilized directly to improve student learning outcomes. Longitudinal tracking in combination with predictive analytical tools reaps high benefits for long‐term planning and education policy decisions. The tracking systems need to be directly incorporated into the EMIS structure and cover students throughout their entire academic career. The system is already linked to the tertiary education sector by sharing student data with universities. As SIIIR is already used as the main source of education data, it should be complemented with a longitudinal student tracking system. Florida, U.S.A., has implemented a successful longitudinal student tracking system from kindergarten until graduation from college, which may serve as a model system (figure 8 and box 11).

Florida, U.S.A., serves as an example on how to build a successful unique student ID (UID) system with a focus on longitudinal student tracking. Every student is issued a unique tracking number, which is applied through their entire academic career, including tertiary education, within the state. Florida is one of the country’s pioneers in collecting and tracking student‐level data with the oldest longitudinal data system in the country, dating back to 1995. In 2014, the Florida Department of Education served nearly 2.7 million students, 4,200 public schools, 28 colleges, 192,000 teachers, 47,000 college professors and administrators, and 321,000 full‐time staff throughout the state. The UID system in Florida provides an example of a fully operational and utilized longitudinal data system. In fact, Florida’s statewide longitudinal data system helped to institutionalize EMIS in the education system, while cultivating a data‐driven culture. State educational accountability policies date back nearly 50 years, positioning Florida at the forefront of efforts to track and learn from education data. Source: McQuiggan and Sapp 2014.

Box 11: Successful Longitudinal Tracking in Florida, U.S.A.

Because of various factors Australian indigenous groups face considerable education challenges compared to other ethnic groups. This is why the Australian government has made education for indigenous Australians one of its priorities. In the “Melbourne Declaration on Educational Goals for Young Australians,” one of the key policy documents defining the legal framework for Australian education, one of the seven commitments targets improving educational outcomes for indigenous youth and disadvantaged young Australians, especially those from low socioeconomic backgrounds. The Australian government aims to narrow the gap for young indigenous Australians by providing targeted support and focusing on school improvements in low socioeconomic communities. In the process the Australian government aims to collect and utilize data to ensure the bettering of education outcomes for disadvantaged social groups. Source: MCEETYA 2008.

Box 10: Australia’s Commitment to Better Education Outcomes for Disadvantaged Groups

Source: Adapted from Florida Department of Education 2012.

Figure 8: Example of Longitudinal Tracking



Considerable potential exists to expand data utilization to the classroom with teacher training in data literacy. The Data Quality Campaign (DQC) in the United States has increasingly advocated including data literacy in teacher training. Teachers not only provide data, which enables policy makers to carry out informed decisions, but they should also make use of data in the classrooms. The DQC views data literacy as an essential tool for teachers, who are entitled to data and should enjoy the needed infrastructure to use data on a daily basis in classroom instruction (DQC 2014). Following this logic, the Oregon DATA Project set out to train teachers on the value and use of data to enhance classroom practice and improve student learning, by giving them the resources to collect, analyze, and use longitudinal data (DQC 2015). Data literacy training should be included in continuous training and professional development programs. Teachers are often seen as data providers, but with appropriate training and skills they can also become data users. In combination with early warning systems and effective data access as well as utilization at the school level, excellent potential exists to improve learning outcomes. The Oregon, U.S.A., DATA Project has already demonstrated that data literacy training can change the way teachers value and use data. The Urban Teacher Residency United program has presented an example of how a teacher’s data literacy not only facilitates the teacher’s curriculum planning, but also directly benefits students (box 12).

Recommendation2:BuildingInstitutionalandStaffCapacity

Institutional and staff capacity could be strengthened by (1) formalizing the regulatory structure of the SIIIR, (2) professional development and regularization of EMIS staff, and (3) adopting external quality assurance methods. These three areas have emerged as important needs based on this review. Recently SIIIR has been institutionalized and has officially become the main education data system in Romania. A well‐defined EMIS policy should have clearly stated procedures and regulations for (a) central, local, and school reporting requirements, (b) defined responsibility and ownership of data, (c) allocation of the EMIS budget, (d) data collection processes, (e) data submission requirements, (f) procurement guidelines if purchasing the software and hardware from external vendors, (g) technical specifications of the EMIS, (h) type of data collected, (i) data validation mechanisms (internal and external), (j) extent to

Urban Teacher Residency United’s video A Data Literate Teacher profiles Micah O’Hare, a member of the Memphis Teacher Residency program, as he uses data in the classroom. O’Hare explains that real‐time data at the daily level, weekly level, and unit level are all critical to his teaching strategy. With this insight into his students’ progress he is able to customize his lessons. He explains that data allow him to be purposeful with his time: “I know when I can release the majority of students to do something, and know that they can do it, and during that time, who are the three students in the room that need me to stop by and give them a quick reminder while I watch and make sure that they can do that skill.” O’Hare explains that he uses data through the lens of growth toward something. In this way data are not a final compliance metric, but instead an active, real‐time tool that puts information at his fingertips so he can more effectively meet the unique needs of each student. In this way, he also uses data to empower individual students to take ownership of their progress: “I like to use the data as a sales pitch. I use it to make success attainable for students.” Whether using data to track growth, top scores, or personal best, data open up new access points to be successful and engage students on their learning journey.

Source: Urban Teacher Residency United 2014.

Box 12: A Data‐Literate Teacher in Action



which data should be utilized by stakeholders, (m) code of conduct for staff, and (n) professional development activities. Updating the regulatory framework is needed to reflect the SIIIR as the replacement for the BDNE. The SIIIR has a clear workflow and architecture, covers a wide range of indicators, and provides public access to its data, but is not yet institutionalized through legal policies. Although SIIIR has effectively replaced the BDNE, which was the official education database, this has not been established in regulation. The BDNE was legally established in 2006 by Ministerial Order 5760/2006, but although the SIIIR has replaced the BDNE in practice, it does not yet have a legal framework. Professional development plans and trainings will enable EMIS staff and education stakeholders to strengthen their analytical skills. No provision is made for EMIS staff professional development in the EMIS annual budget. Staff do not publish methodological papers or attend conferences, which limits their exposure to obtaining new analytical skills to keep the SIIIR up to date. Annual evaluations of EMIS staff as stipulated in the Labor Code are important and should be continued. Here the crucial factor is the effective utilization of the evaluations to improve human resources within EMIS staff through targeted professional development plans. Strengthening data quality by instructing, training, and clearly defining indicators for data collectors is essential. The government needs to clearly define each indicator in accordance with international standards, so that staff can collect reliable and accurate data. If any indicators, such as dropout rates, are not clearly defined, this dilutes data quality. Consequently, staff need to be made fully aware of each indicator and their definition. As the quality of source data builds the foundation for quality education statistics, training and instructing data collectors and data entry staff is essential. This is why instruction, training, and professional development mechanisms can be beneficial in improving data collection and, thus, data quality. Training should aim to implement international best practices and internalize standardized definitions of indicators. Special attention should be paid to rural areas, where data entry is often carried out by underqualified staff. Increasing appeal and competitiveness of EMIS positions can help the government ensure that specialized staff can be hired and retained. Qualified human personnel are essential in any EMIS. On September 1, 2016, most EMIS staff left, which created a severe loss of human resources. In Romania, several human resource issues hinder the growth and effectiveness of the system, including lack of professional development of staff and certain specialized staff including statisticians, policy analysts, and communication experts. These positions are a crucial component in an advanced education data system. Currently regulation is needed to formally define these positions including their skill requirements and enable the MNESR to pay them competitively with the private sector. In general, the lack of appeal and (financial) competitiveness with the private sector is a factor deterring highly qualified staff from accepting employment within the MNESR. Reliance on EU project funds for financing specialized staff decreases sustainability of the EMIS. Currently, the EU funds specialized EMIS staff. However, overreliance on EU funding is not sustainable in the long term. It is necessary to develop stable and long‐term positions for EMIS, data specialists, statisticians, and IT personnel within the Romanian MNESR to cement a well‐functioning EMIS. The EMIS should include clear guidelines of ethical standards for staff to ensure data integrity. NIS adheres to the principles of the Code of Practice of European Statistics, adopted by the European Statistical System Committee (Eurostat 2017). However, ethical standards and conflict‐of‐interest



guidelines are lacking, which may limit data integrity, as well as a lack of internal or external quality reviews. Furthermore, the lack of publication of methodological papers hinders the integrity of the data. An established set of ethical guidelines is also lacking, which is needed for the professionalization of staff. The current EMIS may be expanded by providing deep‐reaching quality assurance measures. Quality assurance could be strengthened by following and implementing an internationally accepted quality assurance management approach such as the International Organization for Standardization’s series ISO 9000, the Education Data Quality Assessment Framework, and Utilization‐Focused Evaluation. To maintain quality, internal and external reviews are performed. Processes are in place that focus on data quality, collection monitoring, processing, and dissemination of education statistics as well as inclusion of data quality in statistical planning.



Acknowledgments

This report was prepared by SABER‐EMIS team members Husein Abdul‐Hamid (Task Team Leader) and Diana Mayrhofer in collaboration with the Country Project Team Leader, Mariana Doina Moarcas, and consultants, Kevin Alan David Macdonald, Valentin Bucur, and Alina Sava. We also thank Dingyong Hou and Janssen Edelweiss Nunes Teixeira, both Senior Education Specialists, for their review and insights. The report benefited immensely from the guidance and support of Daniel Bojte and Valentin Popescu from the Ministry of National Education and Scientific Research, as well as from School Inspectorates. Finally, the SABER EMIS team is especially grateful for the many parents, teachers and principals who shared their time, feedback and insights. Special thanks to colleagues providing support: Namrata Saraogi, Manar El‐Iriqsousi, and Cassia Miranda.

Acronyms

BDNE National Education Database DQC Data Quality Campaign EMIS Education Management Information Systems ENGS Electronic National Grade‐book System EU European Union IT Information Technology MNESR Ministry of National Education and Scientific Research MoPF Ministry of Public Finance NIS National Institute for Statistics OECD Organisation for Economic Co‐operation and Development PISA Programme for International Student Assessment SABER Systems Approach for Better Education Results SIIIR Integrated IT System of Romanian Education



References

Abdul‐Hamid, Husein. 2014. “SABER EMIS Framework Paper.” World Bank, Washington, DC.

Australian Curriculum, Assessment and Reporting Authority (ACARA). 2016. “MySchool.” ACARA, Sydney. http://www.myschool.edu.au/.

Bruns, Barbara, Deon Filmer, and Harry D. Patrinos. 2011. “Making Schools Work: New Evidence on Accountability Reforms.” World Bank Group, Washington, DC.

Data Quality Campaign (DQC). 2014. “Teacher Data Literacy: It’s about Time.” DQC, Washington, DC.

———. 2015. “EMIS in Different Country Contexts.” Paper presented by Paige Kowalski at the World Bank Education Staff Development Program, February. Washington, DC.

du Preez, Derek. 2012. “Department for Education Plans £31M Data Aggregation Project.” Computerworld UK. http://www.computerworlduk.com/news/public‐sector/3370673/department‐for‐education‐plans‐31m‐data‐aggregation‐project/.

Education Statistics (EdStats). 2016. “World Bank Education Statistics—All Indicators.” World Bank, Washington DC. http://databank.worldbank.org/data/reports.aspx?source=Education%20Statistics&preview=off.

Eurostat. “European Statistics Code of Practice.” http://ec.europa.eu/eurostat/web/quality/european‐statistics‐code‐of‐practice

Howard County. 2015. “Integrated Systems for HCPSS.” Paper presented by Grace Chesney and Justin Benedict to the World Bank, April. Washington, DC.

International Organization for Standardization (ISO) 9000, 2016. “ISO 9000 Quality Management.” http://www.iso.org/iso/home/standards/management‐standards/iso_9000.htm.

Maryland State Department of Education (MSDE). 2012. “School Progress Index.” http://reportcard.msde.maryland.gov/Entity.aspx?WDATA=State.

McQuiggan, James and Armistead Sapp. 2014. Implement, Improve and Expand Your Statewide Longitudinal Data System: Creating a Culture of Data in Education. New York: Wiley.

Ministerial Council on Education, Employment, Training and Youth Affairs (MCEETYA). 2008. “Melbourne Declaration on Educational Goals for Young Australians.” MCEETYA, Melbourne, Australia.

United Kingdom. Department for Education. 2012. “Department for Education Departmental Digital Strategy.” DFE‐30014‐2012. Department for Education, London. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/192204/dfe_departmental_digital_strategy.pdf.

Urban Teacher Residency United. 2014. “A Data Literate Teacher.” Video. https://vimeo.com/94399186.

World Bank. 2014. “Project Information Document: Concept Stage”. World Bank, Washington, DC. http://documents.worldbank.org/curated/en/436801468092382153/pdf/PID‐Print‐P148585‐11‐18‐2014‐1416332567376.pdf

___. 2015. “Program Appraisal Document: India—Enhancing Teacher Effectiveness in Bihar Operation Project.” World Bank, Washington, DC. http://documents.worldbank.org/curated/en/184631468000251240/India‐Enhancing‐Teacher‐Effectiveness‐in‐Bihar‐Operation‐Project;jsessionid=cdCk62fl5npMvn7DapJx8N2a.

———. 2016. “World Bank Data Romania.” World Bank, Washington DC. http://data.worldbank.org/country/romania.



Appendix A: Summary of Policy Lever Benchmarking

Policy goal Policy lever Scorea Weight Benchmark

Enabling environment

Legal framework 2.81 17% Established

Organizational structure and institutionalized processes

2.67 17% Established

Human resources 1.71 17% Emerging

Infrastructural capacity 2.81 17% Established

Budget 2.00 17% Emerging

Data‐driven culture 2.29 15% Established

System soundness

Data architecture 3.76 20% Advanced

Data coverage 1.74 30% Emerging

Data analytics 1.00 15% Emerging

Dynamic system 2.67 15% Established

Serviceability 2.97 20% Established

Quality data

Methodological soundness 2.70 25% Established

Accuracy and reliability 2.56 25% Established

Integrity 2.96 25% Established

Periodicity and timeliness 3.70 25% Advanced

Utilization in decision making

Openness 0.74 15% Latent

Operational use 2.04 50% Established

Accessibility 2.47 20% Established

Effectiveness in disseminating findings 1.20 15% Emerging a. 0–0.99 = Latent; 1–1.9 = Emerging; 2–2.9 = Established; 3–4 = Advanced.



Appendix B: Extended Rubric, Romania Scores Highlighted in Red

Policy levers Indicators Description of best practices

Scoring

Latent Emerging Established Advanced

POLICY AREA 1: ENABLING ENVIRONMENT

The system contains crucial components of a comprehensive enabling environment, which addresses related policy elements and enables the functioning of an effective and dynamic system

The system lacks major components of a comprehensive enabling environment

The system contains basic components of a comprehensive enabling environment

The system contains most components of a comprehensive enabling environment

The system contains crucial components of a comprehensive enabling environment

1.1 Legal framework

Institutionalization of system: the EMIS is institutionalized as an integral part of the education system and the government

An existing legal framework supports a fully functioning EMIS

A legal framework is not in place

Basic components of a legal framework or informal mechanisms are in place

Most elements of a legal framework are in place

There is an existing legal framework to support a fully functioning EMIS

Responsibility: responsibility for collecting, processing, and disseminating education statistics is given to a clearly designated institution or agency

Dynamic framework: the legal framework is dynamic and elastic so that it can adapt to advancements in technology

Data supply: the legal framework mandates that schools participate in the EMIS by providing education data Comprehensive, quality data: the requirement for comprehensive, quality data is clearly specified in the EMIS legal framework




Scoring


Data sharing and coordination: the legal framework allows for adequate data sharing and coordination between the Ministry of Education and agencies and/or institutions that require education data

Utilization: the legal framework emphasizes data‐driven education policy

Budget: the education system budget includes a line item for the EMIS

Confidentiality: the legal framework guarantees that respondents’ data are confidential and used for the sole purpose of statistics

1.2



The system is institutionalized within the government, has well‐defined organizational processes, and has several functionalities beyond statistical reporting

The system is not specified in policies, and what exists does not have well‐defined organizational processes; the EMIS has limited functionalities

The institutional structure of the system is not clearly specified in policies, it has some organizational processes, and its functionalities are limited

The institutional structure of the system is defined within the government and it has defined organizational processes, but its functionalities are limited

The system is institutionalized within the government, has well‐defined organizational processes, and has several functionalities beyond statistical reporting

1.3 Human resources Personnel: the core tasks of the EMIS are identified, and the EMIS is staffed with qualified people

Qualified staff operate the system, and opportunities are available to improve their performance and retention

Minimum standards of qualification are not met for the majority of staff that operate the system, and opportunities are not available to

Some staff are qualified to operate the system, and limited opportunities are available to improve staff

The majority of staff are qualified to operate the system, and frequent opportunities are available to improve

All staff are qualified to operate the system, and well‐established opportunities are constantly available to improve staff




Scoring


Professional development: professional training is available for EMIS staff

improve their performance and retention

performance and retention

staff performance and retention

performance and retention

1.4 Infrastructural capacity

Data collection: tools for data collection are available

The system has a well‐defined infrastructure to perform data collection and management, and dissemination functions in an integral manner

The system lacks a well‐defined infrastructure

The system has a basic or incomplete infrastructure

The system has an infrastructure that allows it to perform some of its functions in an integral manner

The system has a well‐defined infrastructure to fully perform its data collection and management, and dissemination functions in an integral manner

Database(s): databases exist under the umbrella of the data warehouse and have both hardware and software means

Data management system: a system is in place that manages data collection, processing, and reporting

Data dissemination: data dissemination tools are available and maintained by the agency producing education statistics

1.5 Budget

Personnel and professional development: the EMIS budget contains a specific budget for EMIS personnel and their professional development

The system budget is comprehensive, ensuring that the system is sustainable and efficient

The system suffers from serious budgetary issues

The system has a basic or incomplete budget

The system budget contains the majority of required categories to ensure that most parts of the system are sustainable and efficient

The system budget is comprehensive, ensuring that the system is sustainable and efficient

Maintenance: the EMIS budget contains a specific budget for system maintenance and recurrent costs

Reporting: the EMIS budget contains a specific budget for reporting costs

Physical infrastructure: the EMIS budget contains a specific budget for physical infrastructure costs




Scoring


Efficient use of resources: processes and procedures are in place to ensure that resources are used efficiently

Data‐driven culture Data‐driven culture

A data‐driven culture prioritizes data as a fundamental element of operations and decision making, both inside and outside of the education system

The system suffers because there is not a data‐driven culture that prioritizes data management and data utilization in decision making

The system has a data‐driven culture that demonstrates a basic appreciation of data and interest in developing better data utilization practices

A data‐driven culture exists that prioritizes data management and utilization within and beyond the education system

A data‐driven culture exists that prioritizes data management and utilization within and beyond the education system, and evidence of that culture is present in daily interaction and decision making at all levels

POLICY AREA 2: SYSTEM SOUNDNESS

The processes and structure of the EMIS are sound and support the components of an integrated system

The system lacks processes and structure

The system has basic processes and a structure that do not support the components of an integrated system

The system has some processes and a structure, but they do not fully support the components of an integrated system

The processes and structure of the system are sound and support the components of an integrated system

2.1 Data architecture Data architecture

The data architecture is well defined to ensure full system functionality

The system’s data structure does not have a well‐defined data architecture

The system’s data architecture includes some components; however, it is incomplete

The system’s data structure has most elements of the data architecture; however, it has some deficiencies that affect the system’s functionality

The data architecture is well defined to ensure full system functionality

2.2 Data coverage

Administrative data: the EMIS contains administrative data

The data in the system are comprehensive and cover administrative, financial, human resources, and learning outcomes data

The data in the system are far from being comprehensive, and coverage is limited

The data in the system include some of the data areas

The data in the system include most but not all of the data areas

The data in the system are comprehensive and cover all data areas

Financial data: the EMIS contains financial data

Human resources data: the EMIS contains human resources data




Scoring


Learning outcomes data: the EMIS contains learning outcomes data

2.3 Data analytics Data analytics

Tools and processes are available to perform data analytics at different levels on a regular basis

Tools and processes are used to perform limited tabulations

Basic tools and processes are available, but the system is not capable of conducting advanced analytical steps (e.g., predictive models, projections)

Tools and processes are available; however, data analytics are not performed regularly

Tools and processes are available to perform data analytics at different levels on a regular basis

2.4 Dynamic system

Quality assurance measures: the system is dynamic and maintains quality assurance measures The system in place

is elastic and easily adaptable to allow for changes /advancements in data needs

The system in place is not easily adaptable to changes /advancements in data needs, as no quality assurance standards are used

The system in place is not easily adaptable and requires significant time and resources to accommodate changes and/or advancements

The system in place is easily adaptable, but it remains reasonably complex

The system in place is elastic and easily adaptable to allow for changes/ advancements in data needs

Data requirements and considerations: mechanisms exist for addressing new and emerging data requirements

System adaptability: the EMIS is elastic and easily adaptable to allow for changes and/or advancements in data needs

2.5 Serviceability

Validity across data sources: information brought together from different data and/or statistical frameworks in the EMIS is placed within the data warehouse using structural and consistency measures

Services provided by the system are valid across data sources, integrate non‐education databases into the EMIS, and archive data at the service of EMIS clients by ensuring the relevance, consistency, usefulness, and timeliness of its statistics

Serious issues exist related to data validity and consistency

Inconsistencies exist related to data validity and consistency

The data are consistent and valid; however, some concerns still exist

Services provided by the system are valid across data sources, integrate non‐education databases into the EMIS, and archive data at the service of EMIS clients by ensuring the relevance, consistency, usefulness, and timeliness of its statistics

Integration of noneducation databases into the EMIS: data from sources collected by agencies outside the EMIS are integrated into the EMIS data warehouse

Archiving data: multiple years of data are archived, including source data, metadata, and statistical results




Scoring


Services to EMIS clients: services provided by the system to EMIS clients include ensuring the relevance, consistency, usefulness, and timeliness of its statistics

POLICY AREA 3: QUALITY DATA

The system has the mechanisms required to collect, save, produce, and utilize information, which ensures accuracy, security, and timely, high‐quality information for use in decision making

The system lacks mechanisms to collect, save, or produce timely, high‐quality information for decision making

The system has basic mechanisms to collect, save, and produce timely, quality information; however, its accuracy might be questionable

The system has most mechanisms in place needed to collect, save, and produce timely, high‐quality information for use in decision making; however, some additional measures are needed to ensure accuracy, security, and/ or timely information that can be used for decision making

The system has the required mechanisms in place to collect, save, produce, and utilize information, which ensures accuracy, security, and timely, high‐quality information for use in decision making

3.1 Methodological soundness

Concepts and definitions: data fields, records, concepts, indicators, and metadata are defined and documented in official operations manuals along with other national datasets and endorsed by the government

The methodological basis for producing educational statistics from raw data follows internationally accepted standards, guidelines, and good practices

The methodological basis for producing educational statistics does not follow internationally accepted standards, guidelines, or good practices

The methodological basis for producing educational statistics follows the basics of internationally accepted standards, guidelines, and good practices

The methodological basis for producing educational statistics follows most required internationally accepted standards, guidelines, and good practices

The methodological basis for producing educational statistics from raw data follows internationally accepted standards, guidelines, and good practices

Classification: defined education system classifications are based on technical guidelines and manuals

Scope: the scope of education statistics is broader than and not limited to a small number of indicators (e.g., measurements of enrollment, class size, and completion)




Scoring


Basis for recording: data‐recording systems follow internationally accepted standards, guidelines, and good practices

3.2 Accuracy and reliability

Source data: available source data provide an adequate basis for compiling statistics Source data and

statistical techniques are sound and reliable, and statistical outputs sufficiently portray reality

Source data and statistical techniques lack soundness and reliability

Source data and statistical techniques have basic soundness and reliability, but statistical outputs do not portray reality

Source data and statistical techniques follow most required elements to be sound and reliable, but statistical outputs do not portray reality

Source data and statistical techniques are sound and reliable, and statistical outputs sufficiently portray reality

Validation of source data: source data are consistent with the definition, scope, and classification as well as time of recording, reference periods, and valuation of education statistics

Statistical techniques: statistical techniques are used to calculate accurate rates and derived indicators

3.3 Integrity

Professionalism: EMIS staff exercise their profession with technical independence and without outside interference that could result in the violation of the public trust in EMIS statistics and the EMIS itself

Education statistics contained within the system are guided by principles of integrity

Education statistics contained within the system are not guided by principles of integrity

Education statistics contained within the system are guided by limited principles of integrity (one of the three principles of professionalism, transparency, and ethical standards)

Education statistics contained within the system are mostly guided by principles of integrity (two of the three principles of professionalism, transparency, and ethical standards)

Education statistics contained within the system are guided by all three principles of integrity: professionalism, transparency, and ethical standards

Transparency: statistical policies and practices are transparent

Ethical standards: policies and practices in education statistics are guided by ethical standards

3.4 Periodicity and timeliness

Periodicity: the production of reports and other outputs from the data warehouse occur in accordance with cycles in the education system

The system produces data and statistics periodically in a timely manner

The system produces data and statistics neither periodically nor in a timely manner

The system produces some data and statistics periodically and in a timely manner

The system produces most data and statistics periodically and in a timely manner

The system produces all data and statistics periodically and in a timely manner Timeliness: final statistics and financial

statistics are both disseminated in a timely manner




Scoring


POLICY AREA 4: UTILIZATION FOR DECISION MAKING

The system is wholly utilized by different users for decision making at different levels of the education system

There are no signs that the EMIS is utilized in decision making by the majority of education stakeholders

The system is used by some education stakeholders, but not for major policy decision making

The system is used by most education stakeholders but is not fully operational in governmental decision making

The system is wholly utilized by different users for decision making at different levels of the education system

4.1 Openness

EMIS stakeholders: EMIS primary stakeholders are identified and use the system in accordance with the legal framework

The system is open to education stakeholders in terms of their awareness and capacity to utilize the system

The system lacks openness to education stakeholders in terms of their awareness and capacity to utilize the system

The system is open to some education stakeholders in terms of their awareness and capacity to utilize the system

The system is open to the majority of education stakeholders in terms of their awareness and capacity to utilize the system

The system is open to all education stakeholders in terms of their awareness and capacity to utilize the system

User awareness: current and potential EMIS users are aware of the EMIS and its outputs

User capacity: EMIS users have the skills to interpret, manipulate, and utilize the data produced by the system to ultimately disseminate findings

4.2 Operational use

Utilization in evaluation: data produced by the EMIS are used to assess the education system

Data produced by the system are used in practice by the main education stakeholders

Data produced by the system are not used in practice by education stakeholders

Data produced by the system are used in practice by some education stakeholders

Data produced by the system are used in practice by the majority of education stakeholders

Data produced by the system are used in practice by the main education stakeholders

Utilization in governance: data produced by the EMIS are used for governance purposes

Utilization by schools: data produced by the EMIS are used by schools

Utilization by clients: data produced by the EMIS are used by clients (including parents, communities, and other actors) Utilization by government: the system is able to produce summative indicators (derived variables) to monitor education system

4.3 Accessibility Understandable data: data are presented in an easily digestible manner

Education statistics are presented in an understandable

The system suffers from serious accessibility issues

The system has major accessibility issues

The system has minor accessibility issues

Education statistics are presented in an understandable




Scoring


Widely disseminated data: education statistics are disseminated beyond the Ministry of Education and/or the education statistics–producing agency to other EMIS stakeholders

manner and are widely disseminated using clear platforms for utilization, complemented by user support

manner and are widely disseminated using a clear platform for utilization, complemented by user support Platforms for utilization: platforms are

standardized across the EMIS and are customizable to user needs

User support: assistance is provided to EMIS users upon request to help them access the data

4.4 Effectiveness in disseminating findings

Dissemination strategy: national governments have an information dissemination strategy in place Dissemination of

education statistics via the EMIS is strategic and effective

Dissemination is neither strategic nor effective

Dissemination is reasonably strategic, but ineffective

A dissemination plan has been implemented; however, room exists for improvement (for full effectiveness in relation to strategic engagement)

The dissemination of education statistics via the EMIS is strategic and effective

Dissemination effectiveness: dissemination of EMIS statistics is effective

SURINAME ǀ EMIS SABER COUNTRY REPORT |2016


The Systems Approach for Better Education Results (SABER) initiative collects data on the policies and institutions of education systems around the world and benchmarks them against practices associated with student learning. SABER aims to give all parties with a stake in educational results—from students, administrators, teachers, and parents to policy makers and business people—an accessible, detailed, objective snapshot of how well the policies of their country’s education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of Education Management Information Systems.

This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries.

www.worldbank.org/education/saber