The use of performance indicators in a school improvement ...mvalcke/CV/Paper Londen 2003 MVA.pdf · research traditions might be favourable to foster the quality of education. A

The use of performance indicators in a school improvement policy: the theoretical and empirical context. Department of Education, Ghent University H. Dunantlaan 1, 9000 Gent, Belgium Tel: + 32 (0)9 264 86 74; Fax: + 32 (0)9 264 86 88 [email protected] http://simsim.rug.ac.be Total number of words: 5918 List of tables Table 1: Differences between school effectiveness and school improvement research

(based on: Creemers & Reezigt, 1997) List of Figures Figure 1: Model of school effectiveness (based on Scheerens, 1992) Figure 2: Integrated model of school improvement Figure 3: Merging SIR and SER in a model for School Effectiveness and Improvement in

terms of Performance Indicators (SIPI) Figure 4: Conditions to make school performance feedback useful for school

improvement

The use of performance indicators in a school improvement policy: the theoretical and empirical context.

2

The use of performance indicators in a school improvement policy: the theoretical and empirical context. Abstract School effectiveness research and school improvement research are central in this article. Based on a study of the literature, the hypothesis is put forward that merging both research traditions might be favourable to foster the quality of education. A theoretical model is developed for School Improvement in terms of Performance Indicators (SIPI). This model is linked to the Flemish school effectiveness approach that is based on the CIPO-model in which Context, Input, Process and Output indicators are distinguished. A central issue in the SIPI-model is the use of performance indicators and the way schools receive feedback when participating in performance indicator studies, such as PISA or TIMSS. The article concludes with the outlining of number of conditions that have to be met at school level to improve the educational quality. Key words: Performance indicators School effectiveness School improvement School performance feedback


3

1. Introduction and research question Performance indicators help to describe and analyse key aspects of schooling. They help to evaluate and monitor the quality of education. Indicators provide an ‘at a glance’ indication of current conditions and may augur future prospects (Oakes, 1986). The question can be raised whether these indicators can be a basis for school improvement. For years, Flanders has participated in comprehensive international performance indicator studies, such as the Programme for International Student Assessment (PISA) and the Third International Mathematics and Science Study (TIMSS). Consequently, there is a wealth of information available that is useful to direct educational policy making at macro-level. In the present article we put forward the following general research question: How useful are performance indicators for a school improvement policy? To answer this question, two sub-questions are raised: (1) What is school improvement? and (2) What methodology – based on performance indicators - facilitates a school improvement policy? We first investigate how school improvement is defined in the literature and which performance indicators might fit into a school improvement approach. The latter introduces the topic of school performance feedback as a methodology to foster school improvement, based on performance indicator studies. 2. School Effectiveness Research (SER) The aim of school effectiveness research is to explain the differences between schools by means of specific criteria. School effectiveness research explores possible differences in learning output and whether these differences are related to teacher, class or school characteristics. Research-based estimates focusing on the size of explained variance by such school and classroom characteristics vary from 8% to 14% (Reynolds, Creemers, Stringfield, Teddlie, & Schaffer, 2002). In terms of pupil progress the explained variance by class/school related variables is more influential than background variables of the individual students (Sammons & Reynolds, 1997). This grounds the added value of schools in relation to educational development.

2.1 Historical traditions in SER

The school effectiveness research has a long tradition. Its history can be outlined in a variety of ways. In appendix 1 an overview is presented that is based on the model of Scheerens (Scheerens, 1992 and 2000). The next paragraphs present a condensed outline of the SER-history. The origin of school effectiveness research can be traced back to the sixties and is rooted in equal opportunities research. The research of Coleman (Coleman et al., 1966) represents the foundations of school effectiveness studies. This study intended to reveal the extent to which school achievement is related to students’ ethnic and social background. The potential contribution of the school to learning achievement was examined. The results presented in the Coleman Report are extremely pessimistic: school characteristics account for only 10% of the variance in pupil performance and schools have little effect on students’ achievement that is independent of their family background and social context. In other words “Schools don’t matter”. Jencks et al. (1972) come to the same conclusions, namely school achievement is largely determined by the family circumstances of a pupil; all other factors are of secondary importance. As a response to these pessimistic views, other research started in the seventies to study more closely factors that are connected to school effectiveness. A second generation of studies represents the economic approaches to school effectiveness. These studies investigate the influence of input and resources on the


4

school output. In this view, school effectiveness research is the study of ‘the scientific properties of school effects evolving from input-output studies to current research utilizing multilevel models’ (Reynolds, Teddlie, Creemers, Scheerens, & Townsend, 2000). A third type of studies focuses on compensatory programmes. Compensatory programmes manipulate school conditions in order to raise achievement levels of disadvantaged groups of pupils; however, no overwhelming success could be established. A fourth generation of studies attempts to disclose the ‘black box’ of the input-output approach towards the added-value of schooling. These studies focus on the organisation, structure and curriculum-content of schools. They are also labelled as ‘input-process-output’ research. Initially these studies try to identify and investigate unusually effective schools (outlier studies). More recently, the research pays attention to the ‘added value’ to analyse potential school effects: “Raw results tell parents the grades their children have obtained but they can say nothing about how well the school attended has performed. In contrast, value added results tell parents how effective their children’s school is in promoting achievement.” (Mortimore & Sammons, 1994). An example of this type of research is set up by Edmonds (1979) who listed salient features of effective schools: strong leadership, high expectations of pupils’ achievement, orderly atmosphere conducive to learning, emphasis on basic-skill acquisition, and frequent monitoring of students’ progress. Later effectiveness studies have replicated this five factor model, and have added other factors and/or applied another labelling of the factors (e.g. Mortimore, Sammons, Stoll, Lewis, & Ecob, 1988 and Levine & Lezotte, 1990). A fifth tradition focuses on instructional efficacy. Teaching and classroom processes are central to this research of teachers and instruction. The studies can be subdivided into (1) research that examines the effectiveness of certain personal characteristics of teachers, (2) research that takes the teacher behaviour during lessons into account and (3) instruction effectiveness research. Current effectiveness research focuses on the complex interaction of a large set of variables and processes to study the effectiveness of education; context, school, teacher and classroom characteristics are taken into account. The research is guided by integrated research models and reflects an international dimension. Reynolds (2000) states: “The benefits of international studies are potentially very great - in the chance to examine a greater range of educational factors and in the chance to generate middle range theory based upon the analyses generated by seeing which effectiveness factors are important in which country cultural contexts”. Despite the – historical - differences in approaches to school effectiveness research (SER), there is a consensus about the basic set of process variables that correlate with school effectiveness:

− School climate and culture, such as a safe and orderly climate, high expectations and achievement orientation

− Purposeful teaching and class management − Concentration on teaching and learning and opportunity to learn − Co-operation, shared visions and goals − Educational or professional leadership − Frequent monitoring, evaluation and positive reinforcement − Communication between the actors, pupil rights and expectations − Parental involvement

Examination of these qualities of effective schools does not provide direct information to guide the use of these findings. School effectiveness research rather presents a vision of an effective schools, and hardly gives insight into ways to realize effective schools (Lezotte, 1989). Purkey and Smith (1983) come to the same conclusion: “Most current


5

school effectiveness research lists have a variety of potential ingredients, but offers little direction for mixing them together.” In other words, the variables are tendencies, not certainties and should not be regarded as a blueprint for realizing effective schools. This is in large due to, since the factors are interdependent, vary from school to school and only explain a part of the variance. Because the list of effective school factors cannot reproduce the complexity of education, researchers complement the research field with context-specific models of school effectiveness. Examples of these models are the integrated model of school effectiveness (Scheerens, 1992), the hierarchical elementary education effects model (Stringfield & Slavin, 1992) and Creemers’ comprehensive model of educational effectiveness (Creemers, 1994). Considering SER-research, a long list of performance indicators has been studied. In figure 1 we list this large set of indicators, using the CIPO-model of Scheerens (1992) as a frame of reference. In view of the central research questions in this article, figure 1 will be discussed again when we present the analysis from the perspective of school improvement research.

<Insert Figure 1 about here>

2.2 Controversies about school effectiveness research The school effectiveness approach has provoked a number of controversies in the literature. Three types of criticisms have been raised: methodological, ideological and pragmatic criticisms. From a methodological point of view 4 issues of concern are being discussed. (1) The school effectiveness paradigm adopts a very mechanistic methodology and an instrumentalist approach to educational processes that fits into the ‘school accountability’ movement. (2) School effectiveness research – when it comes to student performance - builds mainly on performance tests that reflect a narrow knowledge domain: mathematics and literacy tests. Moreover, mostly only cognitive skills are being measured. This emphasis on examination results and test scores as key performance indicators results in a narrow analysis of what is considered as a good school. Educational success is reduced only to factors that can be measured (Morley & Rassool, 1999). This is a critical issue as it is also stated by Rose (1995): “If we think about education largely in relation to economic competitiveness, then we lose sight of the fact that the school has to be about more than economy. If we determine success primarily in terms of test scores, then we ignore the social, moral and aesthetic dimensions of teaching and learning – and, as well, we’ll miss those considerable intellectual achievements which aren’t easily quantifiable.”. (3) There are concerns about the possibility to generalise findings of SER research. Ouston (1999) wonders whether the features of effective schools really relate to all schools or just to those researched. (4) A fourth methodological concern fits in the debate about quantitative versus qualitative research traditions. In most SER-research there is an over-reliance on large-scale quantitative research at the expense of studies employing more detailed qualitative methodologies (Thrupp, 2001). Secondly, there are critiques that centre on ideological issues. School effectiveness research does hardly pay attention to the overall goals of schools and education. Only the most obvious goal is taken into account, namely cognitive achievement. The diversity of school aims is neglected (Van Petegem, 1997). There is also an ideological concern that centre on assumptions about cause and effect. Are the characteristics of effective schools the result of being effective or the cause of the effectiveness (Ouston, 1999)? School effectiveness research is also criticised for the absence of a sound theoretical base. Researchers often randomly select factors to be studied without a priori postulation that


6

indicate the nature of the relationships. A number of the ideological criticisms can be countered by adopting the more recent integrated models for SER. From a pragmatic point of view, a third set of critiques is put forward. School effectiveness research is often characterised by over-estimating the impact size of its findings. As stated earlier, only 8 to 14 % of the variance in pupils’ achievement is accounted for by school/classroom variables. But school effectiveness researchers, like Stoll and Fink (Stoll & Fink, 1996), stress that this limited proportion of explained variance might be the crucial difference between success and failure of schools. Also Sammons and Reynolds (Sammons & Reynolds, 1997) come to this conclusion. Moreover, they stress that these school and classroom variables are of importance since they can be manipulated more easily than background characteristics and prior attainment of pupils. A second pragmatic point of criticism is the limited value of the promising SER-features to guide school and classroom practices. A third point of criticism is the failure of the school effectiveness movement to control the political abuse of its findings. These findings are often used in a too simplistic way by politicians and government officials (Elliott, 1996). 3. School Improvement Research (SIR) The school improvement movement emerged initially as a reaction against school effectiveness research. The school improvement movement is both practice and policy oriented. Its major focus is on helping education to change or develop in a desired direction (Creemers, 2002). It is an approach to educational change that fosters student outcomes and strengthens the school’s capacity for managing chance. In the context of this article, school improvement is defined as a dynamic, planned and rational change process with structural and cultural aspects. School improvement is a process that is planned along three stages, namely initiation, implementation and institutionalisation. It especially centres on changing the conditions of/for learning and related internal conditions in schools. Its ultimate goal ins the more effective realisation of educational goals (van Velzen, Miles, Ekholm, Hameyer, & Robin, 1985; Lagerweij & Haak, 1994; Appelhof et al., 1986).

3.1 The development of the school improvement research tradition The historical development of the school improvement movement can be described in five stages (Reynolds, Teddlie, Hopkins, & Stringfield, 2000; Fullan, 1998 and 1989; Hopkins & Reynolds, 2001). The first stage is set in the mid 1960s and labelled as the curriculum reform movement. It was characterised by an emphasis on the adoption of fitting curriculum materials. This reform movement failed to have a long-lasting impact on teaching. Teachers were not involved in the production process and the teacher in-service training was often rudimentary. Also, hardly questions were asked about what was expected to happen after the adoption of the new curriculum materials. The second stage reflected a period of documenting the failure of the initial SIR approach. Fullan (1989, p.3) e.g., states: “We learned more about what not to do than anything else (don’t ignore local needs; don’t introduce complex, vague innovations, don’t ignore training needs; don’t ignore local leaders and opinion makers; and so forth.)”. It became apparent that the implementation of changes was a complex and lengthy process that required a sensitive combination of strategic planning, individual learning and teacher commitment. During the third phase a series of several successful school improvement projects were set up and studied. Evidence of success was defined as increases in student


7

achievement, degree of institutionalisation or teacher practice mastery, teacher change and teacher commitment. Much was learned about the “dynamics of change” in the projects. However, the clear descriptions were not very helpful to manage the change processes (Fullan, 1989). The shortcomings of the former stage, are dealt with in the next developmental stage of the SIR. Researchers and practitioners try to relate their knowledge about dynamics of change to the school realities. There is a shift from the “study of change” to the actual participation in school development. Attention is paid to influencing changes in the educational processes instead of focusing solely on school management issues or organisational characteristics. A bottom up approach is applied, in which problem owners at school level are actively involved in implementing improvement attempts. In current school improvement research attention is paid to as well the school level as the classroom level. There is an emphasis on changing both school processes and pupil outcomes. Staff development, capacity-building, medium-term strategic planning, change strategies which utilise pressure and support and the intelligent use of external support agencies are considered to be of importance. Current school improvement research adopts a mixed methodological orientation in which bodies of quantitative data plus qualitative data are used to determine quality. These data are also passed on to the school. As such, feedback-loops can be developed to support improvement processes of schools. This fits clearly into the focus of the current article that looks at ways to use performance indicators in a school improvement oriented policy. The current position of school improvement research can be best described by citing van Velzen et al. (1985). They summarize the state-of-the –art in SIR by referring to a number of assumptions:

− “The school is the centre of change. Reforms need to be sensitive to the context of the individual school and classroom.

− There is a systematic approach to change. School improvement is a carefully planned and managed process.

− The key focuses for change are the internal conditions of schools. These include for example the teaching-learning activities in the school, the schools’ procedures, role allocations and resource use that support the teaching and learning process.

− Schools must accomplish educational goals more effectively. The educational goals reflect the mission of the school. This suggests a broader definition of outcomes than scores on achievement tests. Schools also have to serve the more general developmental needs of students, the professional development needs of teachers and the needs of the community.

− There is a multi-level perspective. If quality has to be achieved, then the school is embedded in an educational system that works collaboratively. The roles of all actors should be defined and committed to the process of school improvement.

− Integrated implementation strategies are used. There is a link between top down and bottom up strategies. The former strategy provides a framework, resources and alternatives; the latter provides the energy and the school-based implementation.

− There is a drive towards institutionalisation. Implementation must be a part of the natural behaviour of the actors.”

In parallel to the model of school effectiveness that was presented in section 2.1, we can also structure the variables and processes that are considered of importance in SIR-research. We suggest doing this again by taking the CIPO-model as a frame of reference (Scheerens, 1992), although this model is rarely referred to in the school improvement literature. However, in the context of the present article, we state that adopting a comparable frame of reference is necessary in order to be able to link SER and SIR. The model as depicted in figure 2, is based on Fullan (1989), Fullan & Stiegelbauer (1991), Lagerweij & Lagerweij-Voogt (2004) and Appelhof et al., (1986).


8


3.2 Controversies about school improvement research Also the school improvement research tradition has been criticized. Again, three types of critiques are” being put forward: methodological, ideological and pragmatic criticisms. An important methodological point of criticism on the school improvement movement is that it focuses mainly on practice and that scientific research is not at the centre of this movement (Stringfield, 1995, Stringfield, 1995; Creemers & Reezigt, 1997). From an ideological point of view, Van Petegem (1998) states that the school improvement movement does hardly consider clear educational goals. A first pragmatic criticique is that school improvement research rather provides schools with promising lines of thinking but never clear lines of action (Fullan, 2000). A second issue is that the change strategies are often treated as goals on theitr own instead of means to result in improved learning (Murphy, Evertson, & Radnofsky, 1991). Another pragmatic criticism concerns the major focus on the school level, and the limited focus on the classroom level. This results in a limited impact on pupils performance (Hopkins & Levin, 2000). Finally, SIR is said to adopt a too rational and technical model of change. This is in contrast with reality. Education in schools is a very complex, highly uncertain, and unpredictable system about which our knowledge is still incomplete (Glass, 1979). Moreover, school changes occur often in an intuitive and ad hoc way. School development is not always based on clear-cut and integral policies. Considering these critiques, there is a need to extend the current school improvement approach, e.g., by merging the school effectiveness and school improvement research approaches. 4. Merging school effectiveness and school improvement research The school effectiveness and school improvement studies build on separate bodies of knowledge and research traditions. This is not surprising, considering the rise of the school improvement movement as a reaction against the school effectiveness movement. In general, the school effectiveness movement investigates what constitutes school effectiveness, whereas the school improvement movement examines how schools improve or change. The main differences in approach are summarised in table 1.

<Insert Table 1 about here> The discrepancies between the school effectiveness and the school improvement research are especially situated at the theoretical level. From a practical point of view both approaches are rather complementary since both aim at improving the quality of education. Many researchers have explored how findings of school effectiveness research can inform the developmental work of school improvers. Reynolds et al. (2000) state in this context: “Effectiveness research can provide the knowledge base concerning what to do, and the improvement community the vehicle within the knowledge base sits.” In other words, school effectiveness research and school improvement research might benefit from one another. However, in reality this relationship is rather troublesome. For example, the information needed to guide improvement is not always available in school effectiveness research. Furthermore, the take up of improvement and effectiveness knowledge by practitioners has been patchy, partial and unsystematic (Creemers et al., 1998). Therefore, the need has grown to adapt findings of effectiveness research in function of their use in the context of school improvement. The present research about the use of school performance feedback in a school improvement policy intends to overcome these discrepancies between the school effectiveness and the school


9

improvement movement and might contribute to the integration of both research paradigms. It is in view of the former that we try to merge – both at a theoretical and practical level - the indicators that fit in both the school effectiveness and school improvement tradition. This results in the SIPI-model: School Improvement in terms of Performance Indicators and is represented in figure 3. It is noteworthy that in the SIPI-model assessment and evaluation are important indicators, including both assessment and evaluation of pupils, and assessment of school performance. It is hypothesized that feedback at school level about performance of pupils, class and school related variables is an integral part of this assessment an evaluation that guides the improvement processes at both micro- and meso-level.

<Insert Figure 3 about here> 5. Performance indicators as the base for a methodology for school improvement In its widest sense, a performance indicator summarizes statistical data about the ‘quality’ of an institute or system (Goldstein & Spiegelhalter, 1996). Educational performance indicators provide information about the current performance of a system, suggest whether good progress is being made, and caution about observed problem issues (Oakes, 1986). Quantitative or qualitative data are used to give an ‘at glance’ indication of the actual “state” of a school and may foretell future projections. Obviously, indicators do not reveal everything about a system. Since it is complicated to determine the quality of a school, an indicator is seen as a proxy, instead of a direct measure of school performance. These proxies may deal with different aspects of the system as presented in the frame of reference that has been applied in figure 1 and 2, i.e. context, input, processes and output indicators. Indicators may serve three basic functions (Bertzeletou & Stavrou, 1997). Their first function is the measurement function, in order to undertake analysis, assessment, monitoring or evaluation of the quality of individual schools and education in general. The second purpose of indicators is communication. Indicators help to inform relevant stakeholders or society about the state or development of a particular educational system. Finally, indicators can be used for normative or standardisation purposes. Following this function, indicators show to what extent the actual situation deviates from an established bench, predefined standards or a set of goals. In other words, performance indicators allow value judgements about key aspects of the functioning of educational systems in a quantified way (Scheerens, 1990). The latter function clearly fits into a call for school accountability. Many researchers believe that indicators are a valuable base to direct quality improvement of schools. Bosker and Scheerens (2000) conclude that paying attention to performance indices as part of the school culture can result in improved performance. Also Fullan and Hargreaves (1992) note that pupil achievement and performance data should be used as a springboard for action and that effective schools are actively interested in how well they are doing, and seek evaluative data to monitor and improve their progress. Smith et al. (2001) come to the conclusion that indicators play a role in improving the performance of schools since they can be used to assess a school’s performance, and indicate whether changes are necessary and/or assess whether changes result in improved performance”. Researchers do not only believe that performance indicators are useful for school improvement, also current practice mirrors promising approaches (e.g. NCREL, 2000; Houston, 2002 and Osler, 2002). In conclusion, indicators can lead to powerful opportunities for individual schools to analyse and improve their educational quality. However, an indicator system is only a


10

starting point. The screening instruments are only helpful to indicate where certain problems may be present, but they are not helpful to give a complete and precise diagnosis (Yang, Goldstein, Rath, & Hill, 1999). Performance indicators should be considered as a first step along the way to trying to understand what works, and how schooling can be improved (Fitz-Gibbon & Tymms, 2002). The present article focuses on an additional step to guarantee the relevance of performance indicators for supporting school improvement. This additional step builds on school feedback that makes use of the information from performance indicator studies. Van Petegem and Vanhoof (2004) indicate that this new emerging practice is valuable from both a governmental and school perspective: 1. By providing feedback to individual schools, the government entices schools to study

and document their own operation. From a government point of view, the emphasis is not immediately on judging the quality of individual schools. The feedback of indicators is rather a tool to help schools to improve their quality.

2. From the point of view of the schools, feedback is a starting point to question and optimize underlying processes and to improve process and output elements. The feedback is a mirror to get a realistic picture of critical variables and to understand a need for change. Visscher and Coe (2003), state that successful innovations are much more likely to occur if schools themselves are convinced that changes are needed. They also believe that school performance feedback has a high potential to create this kind of understanding.

Giving feedback to schools is but one step. The next step is considering the way the feedback is used in the school. Rossi & Freeman (1993) and Smith (1995) distinguish different types of school feedback use. The information can be used instrumentally (i.e. as a basis for decision making), conceptually (i.e. influencing the thinking of decision-makers), or in a symbolic way (i.e. a selective use to substantiate a certain position or opinion). One can also distinguish a strategic use of the information that e.g., leads to selective student admission, teaching to the test, consciously depressing baseline test scores, etc. In this way school look more effective but the objective of school improvement will not be attained. A number of conditions must be taken into account to make school feedback based on performance indicators useful and to assure the school improvement orientation. Building on the available literature we can present a first outline of a methodology that directs useful school feedback practices (Fitz-Gibbon & Kochan, 2000; Fitz-Gibbon & Tymms, 2002; Tymms & Albone, 2002; Van Petegem, Vanhoof, Daems, & Mahieu, 2004; Visscher, 2002 and Visscher & Coe, 2002). First of all, the content of the school feedback should be perceived as relevant, accurate, up-to-date, reliable and valid, and inviting to set goals for improvement. This could imply fair comparisons with other schools. However, ranking of schools should be avoided and anonymity and confidentiality ensured. Feedback will lead to a better understanding of a school’s functioning if relationships between indicators are revealed, for instance indicating a school’s differential effectiveness. With respect to the presentation of the feedback, attention is drawn to the representation modes (e.g. charts, graphs), the use of relative versus absolute data, the use of percentiles, T-scores, etc., as well as to the accessibility of the information (e.g. electronic, allowing tailored feedback, allowing schools to perform own analyses). Furthermore, confidentiality and anonymity of the feedback must be guaranteed. This emphasis on confidentiality is based on a concern to avoid unintended and undesirable side effects. Also the support schools receive is of importance to guide correct interpretation and appropriate use of the indicators. The need for support to schools in interpreting and using the feedback is widely emphasized in the literature and corroborate by findings from school improvement research (e.g. Fullan, 1998; Slavin, 1998; and Smith, 1998. These basic conditions to base a sound methodology for school performance feedback have been summarized in figure 4.


11


With regard to a number of elements discussed above, the Flemish educational system take a very different position in the international context. For example, no central examinations are being organised. Schools are autonomous in granting diplomas. No performance based league tables have ever been published in Flanders; the government explicitly tries to prevent the establishment of such approaches. Nevertheless, there is a growing demand of schools to receive relevant and accurate information about the quality of their school performance (Van Petegem & Van Hoof, 2002). Until now, no national school performance feedback system has been set up. Only in the context of the TIMSS and PISA-studies, first experiences with school feedback to schools who participated in the studies have been organised. These individual feedback reports were rather an incentive for participation in the study and were not developed as a tool for self-evaluation or improvement. Moreover, the PISA and TIMSS studies also present some limitations in view of a use in the context of a school improvement approach. For example, the studies are limited to output measurements from a restricted sample of students in a number of classes at a certain moment. Hardly information about background variables of students, classes and schools have been gathered. A longitudinal perspective paying attention to a larger set of output and process measures would have been more adequate. PISA and TIMSS are rather intended to investigate the performance of general educational system and provide information at a national and international level. Therefore, PISA and TIMSS are not that valuable for grounding a school improvement approach at the meso (school) or micro (classroom) level. 6. Conclusion School improvement was at the centre of this article. Both at theoretical and empirical level, school improvement was studied in relation to earlier conceptions that reflect a school effectiveness approach. From the study of the literature, we concluded that the school effectiveness and the school improvement paradigm can be linked in view of promoting quality improvement of education. In merging both approaches, performance indicators were discussed and their role in the assessment and evaluation of school quality. In the latter context the PISA and TIMSS studies were introduced. The potential of a school feedback system based on the indicator studies was explored. Critical issues and especially a number of condition were discussed that are essential to develop a methodology for school performance feedback.


12

7. References Appelhof, P. N., Deen, N., Dubois-Reijmond, M., Van de Grift, W., Lagerweij, N. A. J.,

Seashore-Louis, K. et al. (1986). Naar beter onderwijs. Tilburg: Uitgeverij Zwijsen.

Bertzeletou, T. & Stavrou, S. (1997). Indicators in Perspective. The Use of Quality Indicators in Vocational Education and Training CEDEFOP.

Bosker, R. J. & Scheerens, J. (2000). Publishing school performance data. European Education, 32.

Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, F. D. et al. (1966). Equality of Educational Opportunity. Washington: Government Printing Office.

Creemers, B. (1994). The Effective Classroom. London: Cassell.

Creemers, B. (2002). From school effectiveness and school improvement to effective school improvement: background, theoretical analysis, and outline of the empirical study. Educational Research and Evaluation, 8, 343-362.

Creemers, B. & Reezigt, G. J. (1997). School effectiveness and school improvement: sustaining links. School Effectiveness and School Improvement, 8, 396-429.

Creemers, B., Reynolds, D., Chrispeels, J., Mortimore, P., Murphy, J., Stringfield, S. et al. (1998). The future of school effectiveness and improvement: a report on the special sessions and plenary at ICSEI 1998 in Manchester, UK (Rep. No. vol.9, n° 2). Swets & Zeitlinger.

Edmonds, R. R. (1979). Effective schools for the urban poor. Educational Leadership, 37, 15-24.

Elliott, J. (1996). School effectiveness research and its critics: alternative visions of schooling. Cambridge Journal of education, 26, 199-223.

Fitz-Gibbon, C. & Kochan, S. (2000). School effectiveness and education indicators. In The International Handbook of School Effectiveness Research (pp. 257-282). London & New York: Falmer Press.

Fitz-Gibbon, C. T. & Tymms, P. (2002). Technical and ethical issues in indicator systems: doing things right and doing wrong things. Education Policy Analysis Archives, 10.

Fullan, M. (1989). Implementing Educational Change: What We Know. Education and Employment Division Population and Human Resource Department.

Fullan, M. (1998). The meaning of educational change: a quarter of a century of learning. In A.Hargreaves, A. Lieberman, M. Fullan, & D. Hopkins (Eds.), International Handbook of Educational Change (pp. 214-228). Boston & Dordrecht & London: Kluwer Academic Publishers.

Fullan, M. (2000). The three stories of education reform. Phi Delta Kappan, 81, 581.

Fullan, M. & Hargreaves, A. (1992). What's Worth Fighting for in your School? Working Together for Improvement. Buckingham: Open University Press.


13

Fullan, M. & Stiegelbauer, S. (1991). The New Meaning of Educational Change. New York: Teachers College Press.

Glass, G. V. (1979). Policy for the unpredictable (uncertainty research and policy). Educational Researcher, 12-14.

Goldstein, H. & Spiegelhalter, D. J. (1996). League Tables and their Limitations: Statistical Issues in Comparisons of Institutional Performance.

Hopkins, D. & Levin, B. (2000). Educational review and school improvement. NIRA review, 21-26.

Hopkins, D. & Reynolds, D. (2001). The past, present and future of school improvement: towards the third age. British Educational Research Journal, 27, 459-467.

Houston, P. (2002). Using Data to Improve Schools. What's Working? American Association of School Administrators.

Lagerweij, N. & Lagerweij-Voogt, J. (2004). Anders Kijken. De Dynamiek van een Eeuw Onderwijsverandering. Antwerpen & Apeldoorn: Garant.

Lagerweij, N. & Haak, E. (1994). Eerst goed kijken ... de dynamiek van scholen-in-ontwikkeling. Leuven / Apeldoorn: Garant.

Levine, D. U. & Lezotte, L. W. (1990). Unusually Effective Schools: A Review and Analysis of Research and Practice. Madison: WI: National Center for Effective Schools Research and Development.

Lezotte, L. W. (1989). School improvement based on the effective schools research. International Journal of Educational Research, 13, 815-825.

Morley, L. & Rassool, N. (1999). School Effectiveness: Fracturing the Discourse. London & New York: Falmer press.

Mortimore, P. & Sammons, P. (1994). School effectiveness and value added measures. Assessment in Education: Principles, Policy and Practice, 1, 315.

Mortimore, P., Sammons, P., Stoll, L., Lewis, D., & Ecob, R. (1988). School Matters. The Junior Years. Somerset: Open books.

Murphy, J., Evertson, C. M., & Radnofsky, M. L. (1991). Restructuring schools: 14 elementary and secondary teachers' perspective on reform. The elementary school journal, 92, 135-148.

NCREL (2000). Using Data to Bring about Positive Results in School Improvement Efforts Oakbrook, Illinois.

Oakes, J. (1986). Educational indicators: A guide for policymakers. New Brunswick: Rutgers University, Center for Policy Research in Education.

Osler, D. A. (2002). How Good is our School? Self-evaluation Using Quality Indicators Norwich.

Ouston, J. (1999). School effectiveness and Improvement: Critique of a movement. In T.Bush, L. Bell, R. Bolman, R. Glatter, & P. Ribbins (Eds.), Education Management: Redefining Theory, Policy and Practice ( London: Paul Chapman Publishing / Sage Publishing.


14

Purkey, S. & Smith, M. (1983). Effective schools. The elementary school journal, 83, 27-52.

Reynolds, D. (2000). School effectiveness: the international dimension. In C.Teddlie & D. Reynolds (Eds.), The International Handbook of School Effectiveness Research (pp. 232-256). London & New York: Falmer press.

Reynolds, D., Creemers, B., Stringfield, S., Teddlie, C., & Schaffer, G. (2002). World Class Schools: International Perspectives on School Effectiveness. London & New York: Routledge Falmer.

Reynolds, D., Teddlie, C., Creemers, B., Scheerens, J., & Townsend, T. (2000). An introduction to School Effectiveness Research. In C.Teddlie & D. Reynolds (Eds.), The International Handbook of School Effectiveness Research (pp. 3-25). London & New York: Falmer Press.

Reynolds, D., Teddlie, C., Hopkins, D., & Stringfield, S. (2000). Linking school effectiveness and school improvement. In C.Teddlie & D. Reynolds (Eds.), The International Handbook of School Effectiveness Research (pp. 206-231). London & New York: Falmer Press.

Rose, M. (1995). Possible lives – The promise of public education in America. New York: Penguin books.

Rossi, P. H. & Freeman, H. E. (1993). Evaluation: A Systematic Approach. Newbury Park & London & New Delhi: Sage.

Sammons, P. & Reynolds, D. (1997). A partisan evaluation - John Elliott on school effectiveness. Cambridge Journal of Education, 27, 123.

Scheerens, J. (1990). School Effectiveness Research and the development of process indicators of school functioning. School Effectiveness and School Improvement, 1, 61-80.

Scheerens, J. (1992). Effective Schooling. Research, Theory and Practice. London & New York: Cassell.

Scheerens, J. (2000). Improving School Effectiveness. Paris: UNESCO.

Slavin, R. (1998). Sands, bricks and seeds: school change strategies and readiness for reform. In A.Hargreaves, A. Lieberman, M. Fullan, & D. Hopkins (Eds.), International Handbook of Educational Change (pp. 1299-1313). Dordrecht & Boston & London: Kluwer Academic Publishers.

Slee, R. & Weiner, G. (1998). Introduction: school effectiveness from whom? In R.Slee, G. Weiner, & S. Tomlinson (Eds.), School Effectiveness from Whom? Challenges to the School Effectiveness and School Improvement Movements. (pp. 1-11). London / Bristol: Falmer Press.

Smith, L. M. (1998). A kind of educational idealism: integrating realism and reform. In A.Hargreaves, A. Lieberman, M. Fullan, & D. Hopkins (Eds.), International Handbook of Educational Change (pp. 100-120). Dordrecht & Boston & London: Kluwer Academic Publisher.

Smith, P. (1995). On the unintended consequences of publishing performance data in the public sector. International Journal of Public Administration, 18, 277-310.


15

Smith, W. J., Bordonaro, T., & Lusthaus, C. (2001). School-based Indicators of Performance Ed-Lex.

Smith, W. J. & Sparkes, C. S. (1999). Schools speaking to stakeholders: putting the school performance puzzle together. -25. Canada, Ed-Lex. Research Network on Education, Law and Policy. Faculty of Law. McGill University. Ref Type: Generic

Stoll, L. & Fink, D. (1996). Changing our Schools. Linking School Effectiveness and School Improvement. Buckingham Philadelphia: Open University Press.

Stringfield, S. (1995). Attempting to enhance students' learning through innovative programs: The case for schools evolving into into High Reliability Organisations. School Effectiveness and School Improvement, 6, 67-96.

Stringfield, S. & Slavin, R. E. (1992). A hierarchical longitudinal model for elementary school effects. In B.Creemers & G. J. Reezigt (Eds.), Evaluation of Educational Effectiveness (pp. 35-69). Groningen: ICO.

Thrupp, M. (2001). Sociological and political concerns about school effectiveness research: Time for a new research agenda. School Effectiveness and School Improvement, 12, 99-113.

Tymms, P. B. & Albone, S. (2002). Performance indicators in primary schools. In A.J.Visscher & R. Coe (Eds.), School Improvement through Performance Feedback (pp. 191-218). Lisse: Swets & Zeitlinger.

Van Petegem, P. (1997). Scholen op zoek naar hun kwaliteit. Effectieve-scholenonderzoek als inspiratiebron voor de zelfevaluatie van scholen. doctor Universiteit Gent.

Van Petegem, P. (1998). Vorm Geven aan Schoolbeleid. Leuven: Acco.

Van Petegem, P. & Van Hoof, J. (2002). Reports on the interviews with school principals - Flanders. In J.Scheerens & M. Hendriks (Eds.), Benchmarking the Quality of Education (pp. 263-274). Enschede: Universiteit Twente.

Van Petegem, P., Vanhoof, J., Daems, F., & Mahieu, P. (2004). Benchmarking the quality of school education: enhancing the impact of indicators. accepted for publication in Assessment in Education.

van Velzen, W., Miles, M., Ekholm, M., Hameyer, U., & Robin, D. (1985). Making School Improvement Work: A Conceptual Guide to Practice. Leuven: Acco.

Visscher, A. J. & Coe, R. (2002). School Improvement through Performance Feedback. Lisse: Swets & Zeitlinger.

Visscher, A. J. (2002). A framework for studying school performance feedback systems. In A.J.Visscher & R. Coe (Eds.), School Improvement through Performance Feedback (pp. 41-71). Lisse: Swets & Zeitlinger.

Yang, M., Goldstein, H., Rath, T., & Hill, N. (1999). The use of assessment data for school improvement purposes. Oxford Review of Education, 25, 469-483.


16

Appendices

Appendix 1: General characteristics of types of school effectiveness research (Scheerens, 1992) Independent

variable type Dependent variable type

Discipline Main study type

(Un)equal opportunities

Socio-economic status and IQ of pupil, material school characteristics

Attainment Sociology Survey

Production functions Material school characteristics

Achievement level

Economics Survey

Evaluation of compensatory programmes

Specific curricula Achievement level

Interdisciplinary pedagogy

Quasi-experiment

Effective schools Process’ characteristics of schools

Achievement level


Case study

Effective instruction Characteristics of teachers, instruction, class organisation

Achievement level


Experiment, observation

Effective education Characteristics of teachers, classes and schools

Achievement level (cognitive and non-cognitive)


Quantitative and qualitative studies


17

School Effectiveness Research School Improvement Research What needs to be changed in schools in order to become more effective? Focus on theory and explanations Programme for research Searching for stable causes and effects Searching for objective knowledge Interested in abstraction and Generalisation Output criterion, mostly cognitive skills Roots in scientific research Top down Quantitative techniques, sophisticated techniques for data analysis Focus on student learning / classroom level School = static

How do schools bring about the processes of change? Focus on change in educational practice Programme for innovation Dealing with changing goals and means Dealing with subjective knowledge Interested in local factors Processes Roots in educational practice Bottom up Qualitative techniques Focus on school level School = dynamic

Table 1: Differences between school effectiveness and school improvement research (based on: (Creemers & Reezigt, 1997)


18

Figure 1: Model of school effectiveness (based on Scheerens, 1992)

Context SER

- National, political, economical, international, … context - Educational policy - School sector, school category - Study orientation - School and class size - School facilities, accommodation - Location (urban/rural), recruitment area

Process SER

− School climate (orderly, safe), a learning environment

− Achievement orientation, high expectations

− Leadership − Consensus, cohesion − Curriculum quality − Assessment (district and

school level) − A learning organisation,

professional development − Focus on teaching and

learning − Effective learning time,

time-on-task − Structured teaching,

classroom management − Purposeful teaching − Opportunity to learn − Task oriented environment − Teacher - student

interactions − Focus on basic skills − Monitoring pupils’ progress

and reinforcement − Parent and community

support

Inputs SER

− Policy support − Budget, per pupil

expenditure − Material resources − Absenteeism,

delinquency − Teacher and

student body composition

− Classroom: calm class group, problematic behaviour, study orientation, intelligence level

− Pupil: gender, age, socio-economic status, ethnicity, prior attainment, intelligence, motivation, education level of the parents

− Teacher: teacher education, experience, salary, gender, assignment size, staff stability

Outputs SER

− Student (cognitive) achievement, adjusted for:

Previous achievement

Intelligence Socio-

economic status

− Also non-

cognitive outputs, such as wellbeing, self-image, …


19

Figure 2: Integrated model of school improvement

Context SIR

− Political, economical, social, cultural, demographical context

− Educational policy; decentralisation − Scaling-up − Degree of effectiveness − School sector, school category − Location (urban/rural), recruitment area − Local characteristics (school district, school board,

the community) − Local market situation − Incentives from the environment

Process SIR

− Leadership − Professional development − Support (internal or external support,

critical friend) − Pressure − School policy (unambiguous goals,

consensus, but also conflicts) − Assessment or evaluation − Team development − School culture − School climate − School structure − Instructional organisation − Networks − High expectations (for students and

staff) − Parents and community support − Characteristics of change projects:

content, improvement experience, need, clarity, complexity, quality and practicality of program, change, active initiation, participation, development plan with specific goals, vision-building, top down and bottom up approaches, problem solving capacity, …)

Inputs SIR

− Budget, per pupil expenditure

− School facilities, accommodation

− Time and energy − Teacher and

student body composition

− School and class size

− Pupil: gender, age, socio-economic status, ethnicity, prior attainment, intelligence, motivation, education level of the parents

− Teacher: teacher education, experience, salary, gender, assignment size, staff stability, engagement, collegiality

Outputs SIR

− Strengthening the change capacity of the school

− Restructuring − Reculturing − Improving pupil’s

results − Improving the

progress of the pupils


20

ContextNational, political, economical, international contextEducational policyLocation, recruitement aera, local characteristics, local market situationSchool sector, school category, school facilities and accomodation

InputBudget, material resources, school facilities, accommodationTeacher and student body composition, school and class size, pupilcharacteristics, teacher characteristics, classroom characteristics, absenteeism,delinquency, ...

OutcomesINTERMEDIATE OUTCOMES: Restructuring & reculturing of the school

PUPIL OUTCOMES: Student achievement (cognitive and non-cognitive outcomes),attitudes, skills; improving pupils’ results, improving the progress of pupils

Process of changeMESO: leadership, assessment and evaluation, parent and community support, professionaldevelopment, team development, school structure, school culture, shared visions and goals,school ethos, school climate, internal and external support and pressure, networks, highexpectations...

MICRO: achievement orientation, curriculum quality, effective learning time, time-on-task,structured teaching, classroom management, purposeful teaching, opportunity to learn, taskoriented environment, teacher-students interaction, monitoring pupils’ progress andreinforcement

Impl

emen

tatio

n

Incorporation

Adoptation

Figure 3: Merging SIR and SER in a model for School Effectiveness and Improvement in terms of Performance Indicators (SIPI)


21

Use of performance feedback for SI

SchoolFeedback

ContentRelevantReliableValideUp-to-dateAnonymousConfidentialDifferential effectivenessPositive reactivity

Representation

Accesible informationNo classificationsStandard and tailoredinformation

Problem solving capacityInnovation capacityInnovation AttitudeAllocation extra resourcesTailored user-training and school support

Educational system

Figure 4: Conditions to make school performance feedback useful for school improvement

Documents

The use of performance indicators in a school improvement ...mvalcke/CV/Paper Londen 2003 MVA.pdf · research traditions might be favourable to foster the quality of education. A