KATHRYN RILEY, WARD HENEVELD and ALMA HARRIS

BIG CHANGE QUESTIONSARE SCHOOL EFFECTIVENESS MEASURES SUSPECT IN

HELPING US IDENTIFY WHAT IS NEEDED TO TRANSFORMTEACHING AND LEARNING?

KATHRYN RILEY

School effectiveness research has contributed much to our current thinkingabout differences and similarities between schools. However, the validityof some of the measures used, and the extent to which the research hasprovided us with the information we need to improve schools, has beenthe source of much debate. The two authors who have responded to the BigChange Question, Ward Heneveld and Alma Harris, come to the questionfrom two very different vantage points: Ward as a leading education expertin the World Bank, and Alma as an academic who has been firmly rootedin the school improvement camp for some years.

Despite these two different vantage points, they share a number ofcommon perspectives, most notably the importance of linking any schooleffectiveness measures to strategies aimed at creating a better environmentfor teaching and learning. I hope you will enjoy their stimulating contri-butions. As I have just taken over as section editor of The Big ChangeQuestion, I would be delighted if you would send me your thoughts aboutother important Big Change Questions we should tackle.

Institute of EducationUniversity of London, UKE-mail: kriley@dial.pipex.com

WARD HENEVELD

My first inclination in response to the “big change question” above is tosay, “it depends.” It depends first on what one means by “school effective-ness” and by “measures”, and second, on what one thinks it means “totransform teaching and learning.” Let’s start with the definitions.

Journal of Educational Change 3: 407–415, 2002.© 2002 Kluwer Academic Publishers. Printed in the Netherlands.

408 KATHRYN RILEY ET AL.

In the school effectiveness literature, and even more in practice,educators talk about both the educational experience and the learning thatstudents take away from that experience as indicators of effectiveness.Sometimes, people confuse the two, using them more or less interchange-ably. Some years ago I came across a definition of school effectivenessthat I have used since because it resolves this question for me. In theirsmall book, Planning the Quality of Education: The Collection and Useof Data for Informed Decision-Making, Ross and Mahlck (1990) definequality in education as “an improvement in the environment in which thestudent work(s) with the aids to learning provided for that purpose by theschool system” so that there are “detectable gains in the knowledge, skills,and values acquired by students.” (p. 6) An effective school does this: it isalways seeking to improve the environment for learning in ways that willhelp children learn more. In both parts of this definition, one should notethat the emphasis is on change, on “an improvement” and on “gains.” Eventhe best school and the worst school in a system can be more effective ifthey are more effective tomorrow than they are today.

“Measures” is harder for me to define. My Webster’s Dictionaryprovides two definitions that could apply here: “a basis or standard ofcomparison: criterion” and “a step planned or taken as a means to anend.” Although the context of the question above clearly favors the firstdefinition, personally, I don’t like applying it to schools because it impliesa normative distinction among schools that tells some they are good andothers that they are bad. Labeling hurts, no matter what form it takes, asI have witnessed in almost forty years of school experience in both poorcountries in Africa, Asia and South America and in rich countries like theUSA and Great Britain. This labeling is particularly pernicious in settingslike Africa where neither the government nor the communities can providethe resources for all schools to reach any standard that may be set for thelearning environment. I’m much more comfortable with the second defini-tion, using “measures” to indicate steps that can be taken to change thelearning environment so that children learn more. So, if “measures” meansstandards, I conclude that they are suspect in helping us identify what isneeded to transform teaching and learning.1 I’ll stick with the meaningthat suggests steps to be taken, despite the risk that many people mayconsider whatever is said to be standards. We have stuck to this meaningof “measures” as conditions to be achieved in the work on improving thequality of primary education that we have engaged in at the World Bankover the last ten years.

When I joined the World Bank in 1990 I was asked to analyze whyWorld Bank projects in Africa that had specific objectives to improve

BIG CHANGE QUESTION 409

primary school effectiveness (or “quality”) were not producing betterstudent learning results. A look at a few project design documents thatGovernments and the World Bank had prepared together made it clear thatthe definition of an effective school was not explicit in any project design orreform program. Furthermore, none of the project designs even mentioned,let alone focused on, teaching/learning processes. This observation on thedesign of interventions made even more striking the finding of a reviewof over 200 World Bank education projects that the closer the locus ofdecision-making was to the schools, the more successful the project hadbeen (Verspoor, 1989). Obviously, if a school’s effectiveness was goingto improve, interventions had to deal more explicitly with what goes oninside schools and classrooms, and the desired in-school factor had tofigure prominently in planning any intervention. In our research, we setout to look more closely at the generic characteristics of effective schoolsas a guide to what might be more explicitly included in efforts to reformAfrican primary education. This search, started with analysis in mind, hasled to a fruitful history of collaboration for change in primary education inAfrica and, more recently, in Asia. The conceptual framework and some ofour early experiences using it are presented in Heneveld and Craig (1996).

Based on a review of all the literature we could find on schooleffectiveness and school improvement, we identified eighteen “factorsthat determine school effectiveness” under five headings: Supportinginputs, children’s characteristics, school climate, enabling conditions (inthe school), teaching/learning process, and student outcomes. Then, usingthe research literature, we prepared a definition and listed indicators foreach of the factors; that is, what one would expect to observe in a schoolthat was “effective” with respect to that factor. For example, one of the“enabling conditions” is “effective leadership.” One of the ways to knowthat a school has effective leadership (part of its definition) is if “the Headcommunicates regularly and effectively with teachers, with parents, andwith others in the community.” One knows this factor exists when onevisits a school if he/she finds that staff meetings, communications withteachers, school public events, and the Head’s involvement in activitiesoutside the school are “frequent and productive.” What this indicator maybe in a given context still needs specification if it is to be useful operation-ally. For example, perhaps half of the primary schools in India have one ortwo teachers where staff meetings are not necessary and there is no formalschool head to engage the community. Still, one can expect that if the linksamong stakeholders that the framework defines are attended to, the schoolwill be more effective.

410 KATHRYN RILEY ET AL.

When we shared the conceptual framework of factors, definitions, andindicators with African educators, they started using it immediately todefine their own conceptual framework to fit their immediate priorities, toset improvement targets using that framework, and to plan interventions forthose school characteristics that they thought were most important at thatmoment in their systems. The response in the Bank was also supportive,but many people expected to have quantities specified for the indicators.If one responded to this request it would mean, for example, using theresearch to decide and advertise how frequently a school head shouldhave public events at a school. We resisted this normative definition of“measure” in order to let educators set their own quantitative targets basedon where their school or system was starting from.

We have found that when people are clear about the characteristics ofschools that they think will help children learn more in their own contextthey are then able to set their own targets, or “measures,” to be achievednext. Many groups in Africa and in other parts of the world have sincefound the approach helpful. But this is only the first half of the issue,identifying the “measures” that can guide reform. The next challenge isto use those measures to affect teaching and learning. This requires influ-encing teachers and students to behave differently. Here, I find the issuesas perplexing as when I resorted to the definitions at the beginning ofthis commentary. For I don’t know how, or even if, creating the condi-tions that respond to my definition of “school effectiveness measures”can be certain to affect what teachers do in the classroom and what theadult school community does to maintain a supportive school climate. Thecharacteristics we describe in the conceptual framework that we developeddefine the action, but they do not clarify how to implement them in a givensetting. Even the most motivated teachers are limited in their teachingrepertoire by education, training, experience, and personality. Also, thestudents bring their own uniqueness to the relationship in the classroom.These realities influence how student-teacher relationships play them-selves out, confounding further any prescriptive certainty we may have thatour measures will improve learning. Instead, we may expect that by payingattention to school effectiveness measures as guides, not prescriptions, theconditions will become more conducive to teachers taking advantage ofthe changed environment to become better teachers.

And even with all the research that’s been done, we cannot be sure whatconditions will “transform teaching and learning.” As a reminder of thisreality, since graduate school I have kept on my shelf a qualifying paperwritten at the end of the sixties that summarizes a broad range of studiesthat suggest that the best teaching process in a given classroom is defined

BIG CHANGE QUESTION 411

by the characteristics of the teacher, the students, and the subject beingtaught. Children with little discipline at home, whether studying mathe-matics or civics, perform better when the teacher is more didactic. Teacherswho think clearly and love their subject but who may not have excellentsocial skills, may also be effective directive teachers, even with studentswhose home environments have provided the motivation and discipline tostrike out more on their own. There is no one best way to teach! I thinkit is the realization of the unique dynamics in the interaction of teacher,students, and subject that has encouraged the experimentation with devel-opment schools, teacher apprenticeships, in-service mentoring programs,and reflective teacher research that mature education systems are nowimplementing. In summary, “school effectiveness measures” grounded inanalyzed experience are not directly helpful in the existential activity oftransforming teaching and learning in a particular school or classroomunless the teacher has been able to adapt the new information to his orher reality. Experience hasn’t given us the measures to generalize fromthe unique experience that happens every day among millions of childrenand teachers in settings all over the world. However, if there aren’t any“measures” towards which to strive to improve teaching and learning,changes in the teaching environment and in learning are also not likelyto occur. As the American baseball philosopher Yogi Berra used to say, “ifyou don’t know where you’re going, you probably won’t get there.”

So, I come back to my first point: “It depends” whether school effective-ness measures can add much to what is needed to transform teaching andlearning. The best one can hope for is that paying attention to changingthese measures, by creating a better environment for learning, teachers willbe encouraged, and then forced by dedication, opportunity and hopefullysome nudges from the supporting system to become better than they arenow. If they seize that opportunity in spaces given to them by the managersof education (shortened teaching hours, accessible resource centers, at-hand mentors, etc.), children’s learning can be transformed. However, it’sup to each teacher, each time they enter the classroom to make effectiveuse of the conditions that their setting provides them. If we use measuresto tell teachers and communities that there are standards that they don’tmeasure up to and which they can never see their school attaining, thetransformation that we all hope for is very unlikely to happen. On the otherhand, if we don’t have “measures” that stimulate them to transform theirteaching, schools will probably not become more effective.

412 KATHRYN RILEY ET AL.

NOTE

1 Though Brazil’s use of “minimum conditions for learning” to describe the desiredcharacteristics of school that government has already estimated it can finance does reducethe pejorative implications of these “standards.”

REFERENCES

Geneveld, W. & Craig, H. (1996). Schools Count: World Bank Project Designs and theQuality of Primary Education in Africa. Washington, DC: World Bank Technical PaperNo. 303.

Ross, K. & Mahlck, L. (1990). Planning the Quality of Education: The Collection and Useof Data for Informed Decision-Making. Paris: UNESCO/Pergamon Press.

Verspoor, A. (1989). Pathways to Change: Improving the Quality of Education inDeveloping Countries. Washington, DC: World Bank Discussion Paper 53.

The World BankNew Delhi, IndiaE-mail: e.heneveld@worldbank.org

ALMA HARRIS

School effectiveness is a research field which has gained academicprominence over the last 30 years (Teddlie and Reynolds, 2000). Ina growing number of countries research has been undertaken that hasinvestigated the impact that schools have upon their students’ educationalachievements. As a research paradigm, school effectiveness is premisedupon the measurement of outcomes and quantifying differences betweenschools. The concept of school effectiveness is closely related to a ‘means–end’ relationship. In broader terms, the effectiveness research tradition isconcerned with the extent to which schools differ from one another interms of student outcomes.

To date, school effectiveness research has produced a significantreorientation of thinking about what educational systems can do andwhat might be expected of them. The field has made a very importantcontribution to understanding why and how schools perform differently.Most significantly, the school effectiveness field has challenged the ideathat schools make little difference to the life chances of pupils and hashighlighted what schools are able to achieve by focusing upon pupil attain-ment as the basis of judging effectiveness. Furthermore, it has encouraged

BIG CHANGE QUESTION 413

schools to think more critically about their performance and exposed andexplained differential effectiveness within schools.

Recently, concerns have been raised over whether the school isthe appropriate unit of analysis. Work by Hopkins and Harris (2000)has emphasised differential teacher effectiveness and the importance ofviewing the classroom as the unit of change. Such arguments suggest thatto analyse pupil achievement school by school is to overlook much closerinfluences on individual pupil performance, and that narrow measures ofachievement are unlikely to explain or to assist improvements in teachingand learning. Indeed from the very outset, school effectiveness writershave focused on a very limited number of pupil achievement scores. Inthe U.S.A. and the Netherlands, for example, they have tended to measureachievement scores in English and Mathematics tests, while in Englandand Wales the focus has been external examination and key stage testresults. Consequently, school effectiveness researchers have been criti-cized by opponents of the movement (e.g. Slee and Weiner, 1998) for usingnarrow and limited measures of achievement and for proposing a simpletraditional relationship between teaching and learning. They point to thefailure to move beyond correlational association to seek causal evidence,thereby implying a simple relationship between teaching and learning.

It could be argued that an emphasis on the measurable becomes a justi-fication for ignoring less measurable but equally important outcomes ofschooling. This has implications for the kinds of teaching and learningprocesses deemed to be appropriate. Defining certain goals leads inevitablyto particular definitions of “good practice”. Hence, a narrow set of goals oroutcomes is equally likely to produce a narrow and uncritical interpretationof how to achieve those goals: Consequently, a rather restricted view ofwhat counts as effective teaching and learning (and the process by whichthis is achieved) ensues.

Externally defined measures of effectiveness do not necessarily helpschools or teachers to become more effective. While lists of character-istics of school or teacher effectiveness may assist in diagnosing areas forfuture change or development, they do not offer any operational means toimprovement. How schools and teachers interpret, understand and respondto such lists is key to linking effectiveness and improvement at the teacher,school and departmental level (Harris, 2001). Recent work has demon-strated that there is potential for linking effectiveness outcomes and theprocess of improvement but only within a much broader interpretation oforganizational change and development (Bennett & Harris, 2001). Thisimplies a much more complex, dynamic and turbulent relationship betweenteaching and learning where diversity rather than consistency is desirable.

414 KATHRYN RILEY ET AL.

Jamieson and Wikely (2000) argue that even the most well thoughtthrough and tried and tested strategies are unlikely to be very effectiveif students are not engaged with school and motivated to learn. The failureof large scale, externally prescribed reform is well documented and wellknown (Fullan, 2000), indicating that students learn with more depthand understanding when they feel some ownership or involvement in theprocess of learning. The limitations of externally prescribed reform initi-atives reside in their inability to be sensitive or responsive to the learningneeds of individual students and to widely differing school contexts. Thedrive for consistency, uniformity and standardization of teaching andlearning, however well intentioned, has meant that students are factoredout of the teaching and learning equation and have less control over boththe process and the outcomes.

This appeal for individuality rather than conformity rests upon anumber of key assumptions about effective learning. Firstly, knowledgeand beliefs are formed within the learner and cannot be imposed. In thissense the process of coming to know is an interactive one where learnersbring experience and understanding to the classroom. Secondly, the valuesand beliefs that learners have already formed help them to interpret andassign meaning, as do their interactions with others. Hence, meaning isconstructed and shaded by the previous individual experience. Thirdly,learners play a critical role in assessing their own learning and teachersplay a critical part in establishing learning goals and criteria for success.Where outcomes are tightly prescribed, there is little scope for negotiationand less opportunity for forms of assessment that provide an insight forthe teacher into how students develop and view their own learning. Thisis unlikely to increase the motivation to learn if students are only able toachieve in terms that the school is allowed to recognize and endorse.

It is unfortunate that the school effectiveness research movement hasprovided a rationale and platform for the standards-based reform agendain a number of countries. The seductive appeal of consistency, uniformityand certainty in educational practice has overshadowed the complexity andpracticality of improving teaching and learning. Change at the classroomlevel requires investing in approaches that promote and mutually supportteacher and student learning. It requires ownership over the change processwith particular attention to context, individual needs and the constructednature of learning. It also necessitates viewing schooling as intrinsicallyconcerned with building professional learning communities where learningis a shared endeavour and where communal relationships can be built(Wenger, 1998).

BIG CHANGE QUESTION 415

Mitchell and Sackney (2000: 78) note that professional learningcommunities are built where ‘individuals feel confident in their owncapacity, in the capacity of their colleagues and in the capacity of theschool to promote professional learning’. Similarly, Hopkins and Levin(2001: 24) argue that ‘changes in teaching practice will only occur whenthere is clarity and coherence in the minds of teachers and that this is bestachieved by building the capacity of the school as a learning organisation’.This will require paying careful attention to how collaborative processesin schools and classrooms are fostered and in deciding upon shared goalsand outcomes. It will require embracing a much broader and creative set ofteaching and learning approaches and allowing those who work in schoolsto decide what counts as effective and most importantly, how to achievethis together.

REFERENCES

Bennett, N & Harris, A. (2001). School effectiveness and school improvement: Futurechallenges and possibilities. In A. Harris & N. Bennett (eds), School Effectiveness andSchool Improvement Alternative Perspectives. London: Cassell.

Fullan, M. (2000). Leading in a Culture of Change. San Francisco: Jossey Bass.Harris, A. (2001). Department improvement and school improvement: A missing link?

British Educational Research Journal 27(4), 477–487.Hopkins, D. & Harris, A. (2000). Creating the Conditions for Teaching and Learning.

London: David Fulton Press.Hopkins, D. & Levin, B. (2000). Government policy and school development. School

Leadership and Management 20(1), 15–30.Jamieson, I. & Wikely, F. (2000). School effectiveness and school improvement: Fitting

school round the need of students. In A. Harris & N. Bennett (eds), School Effectivenessand School Improvement: Alternative Perspectives. London: Cassell.

Mitchell, C. & Sackney, L. ( 2001). Profound Improvement: Building Capacity for aLearning Community. Lisse: Swets and Zeitlinger.

Slee, R. & Weiner, G. with Tomlinson, S. (eds), (1998). School Effectiveness for Whom?Challenges to the School Effectiveness and School Improvement Movements. London:Falmer.

Teddlie, C. & Reynolds, D. (2000). The International Handbook of School EffectivenessResearch. London: Falmer Press.

Wenger, E. (1998). Communities of Practice: Learning, Meaning and Identity. Cambridge:Cambridge University Press.

University of WarwickUnited KingdomE-mail: alma.harris@nottingham.ac.uk