From project to policy evaluation in vocational - Cedefop - Europa

From project to policy evaluation in vocational education and training – possible concepts and tools. Evidence from countries in transition

Evelyn Viertel, Søren P. Nielsen, David L. Parkes, Søren Poulsen

In:

Descy, P.; Tessaring, M. (eds)

The foundations of evaluation and impact researchThird report on vocational training research in Europe: background report.

Luxembourg: Office for Official Publications of the European Communities, 2004(Cedefop Reference series, 58)

Reproduction is authorised provided the source is acknowledged

Additional information on Cedefop’s research reports can be found on:http://www.trainingvillage.gr/etv/Projects_Networks/ResearchLab/

For your information:

• the background report to the third report on vocational training research in Europe contains originalcontributions from researchers. They are regrouped in three volumes published separately in English only.A list of contents is on the next page.

• A synthesis report based on these contributions and with additional research findings is being published inEnglish, French and German.

Bibliographical reference of the English version:Descy, P.; Tessaring, M. Evaluation and impact of education and training: the value of learning. Thirdreport on vocational training research in Europe: synthesis report. Luxembourg: Office for OfficialPublications of the European Communities (Cedefop Reference series)

• In addition, an executive summary in all EU languages will be available.

The background and synthesis reports will be available from national EU sales offices or from Cedefop.

For further information contact:

Cedefop, PO Box 22427, GR-55102 ThessalonikiTel.: (30)2310 490 111Fax: (30)2310 490 102E-mail: [email protected]: www.cedefop.eu.intInteractive website: www.trainingvillage.gr

http://www.trainingvillage.gr/etv/Projects_Networks/ResearchLab/

www.cedefop.eu.int

www.trainingvillage.gr

Contributions to the background report of the third research report

Impact of education and training

Preface

The impact of human capital on economic growth: areviewRob A. Wilson, Geoff Briscoe

Empirical analysis of human capital development andeconomic growth in European regionsHiro Izushi, Robert Huggins

Non-material benefits of education, training and skillsat a macro levelAndy Green, John Preston, Lars-Erik Malmberg

Macroeconometric evaluation of active labour-marketpolicy – a case study for GermanyReinhard Hujer, Marco Caliendo, Christopher Zeiss

Active policies and measures: impact on integrationand reintegration in the labour market and social lifeKenneth Walsh and David J. Parsons

The impact of human capital and human capitalinvestments on company performance Evidence fromliterature and European survey resultsBo Hansson, Ulf Johanson, Karl-Heinz Leitner

The benefits of education, training and skills from anindividual life-course perspective with a particularfocus on life-course and biographical researchMaren Heise, Wolfgang Meyer

The foundations of evaluation andimpact research

Preface

Philosophies and types of evaluation researchElliot Stern

Developing standards to evaluate vocational educationand training programmesWolfgang Beywl; Sandra Speer

Methods and limitations of evaluation and impactresearchReinhard Hujer, Marco Caliendo, Dubravko Radic

From project to policy evaluation in vocationaleducation and training – possible concepts and tools.Evidence from countries in transition.Evelyn Viertel, Søren P. Nielsen, David L. Parkes,Søren Poulsen

Look, listen and learn: an international evaluation ofadult learningBeatriz Pont and Patrick Werquin

Measurement and evaluation of competenceGerald A. Straka

An overarching conceptual framework for assessingkey competences. Lessons from an interdisciplinaryand policy-oriented approachDominique Simone Rychen

Evaluation of systems andprogrammes

Preface

Evaluating the impact of reforms of vocationaleducation and training: examples of practiceMike Coles

Evaluating systems’ reform in vocational educationand training. Learning from Danish and Dutch casesLoek Nieuwenhuis, Hanne Shapiro

Evaluation of EU and international programmes andinitiatives promoting mobility – selected case studiesWolfgang Hellwig, Uwe Lauterbach,Hermann-Günter Hesse, Sabine Fabriz

Consultancy for free? Evaluation practice in theEuropean Union and central and eastern EuropeFindings from selected EU programmesBernd Baumgartl, Olga Strietska-Ilina,Gerhard Schaumberger

Quasi-market reforms in employment and trainingservices: first experiences and evaluation resultsLudo Struyven, Geert Steurs

Evaluation activities in the European CommissionJosep Molsosa

From project to policy evaluation invocational education and training –

possible concepts and toolsEvidence from countries in transition

Evelyn Viertel, Søren P. Nielsen, David L. Parkes, Søren Poulsen (1)

AbstractThis study follows the international trend in evaluation approaches in vocational education and training(VET), moving away from single projects towards policy evaluations. It examines the contribution ofresearch and a number of examples from transition countries in central and south-east European coun-tries with a view to helping clarify underlying concepts, methods and possible tools used when under-taking VET policy evaluations.Using VET policy evaluations as a platform for identifying and designing reform proposals in transitioncountries, as well as monitoring reform progress, the study is as much concerned with aspects of systemanalysis, as it is with elements of implementation research. The study argues that there is no holy grail in terms of conceptualisation or methodology related to VETpolicy evaluations. The engineer’s toolbox is of limited use. Similarly, management approaches based onrefined analytical frameworks have been found to obscure rather than illuminate VET, including relianceon tools derived from system analysis. Thus, the only remedy seems to be the evaluator’s broad under-standing of the essential components of VET, of the relationships between them, of the fundamental logicbetween the system and its environment and, of change levers. This is an understanding that developsonly through many years of apprenticeship and first-hand experience with VET policy evaluations, andnot least through a close dialogue with key local actors. Evaluators need to develop an understanding of both the starting points for reform in the given countrycontext and strategic levers of change. The study emphasises that process evaluation, which acknowl-edges the role of organisational or administrative learning as part of an overall evaluation activity, is asimportant as impact studies. The study also argues that any VET policy evaluation needs to pay specifictribute, amongst others, to the professionalisation and continuous development of teachers and othereducation specialists, as they are key to the success of any systemic VET reform effort.Presented in this study are the first results of a kind of ground-clearing work in VET policy evaluations.As such, we see our study as a basis for discussion, which opens a number of fields for further researchand debate.

(1) In cooperation with Jean-Marc Castejon, Henrik Faudel, Barbara Kuta and Piero Benazzo.

1. Introduction 195

1.1. Methods of work 197

2. The limited transparency of concepts and methods used in VET policy evaluations 198

2.1. OECD 198

2.2. The building-blocks approach 198

3. Humanistic approaches to understanding VET systems 201

3.1. The hermeneutic approach 201

3.2. The structural-historical approach 201

3.3. Understanding and explanation 202

4. VET as a (sub)system and issues of internal and external consistency 204

4.1. System analysis 204

4.2. VET as a subsystem 205

4.3. Internal and external consistency 205

4.3.1. Internal consistency 207

4.3.1.1. Consistency between subsystems and clarification of pathways 207

4.3.1.2. Regional consistency 207

4.3.1.3. Consistency for the individual learner 207

4.3.2. External consistency 207

4.3.3. Regulating mechanisms 208

4.4. Examples of lack of internal and external consistency of VET reforms in Bosnia-Herzegovina,

Estonia and Romania 209

4.4.1. Teacher training in Bosnia-Herzegovina (Durand-Drouhin and Bertrand, 1994) 209

4.4.1.1. Pedagogical Academy in Sarajevo (PAS) in charge of education of primary school

teachers 209

4.4.1.2. Sarajevo University, Faculty of Philosophy, Pedagogical Department, in charge of

pre-service VET teacher training 209

4.4.1.3. Pedagogical Institute, Canton Sarajevo, in charge of in-service VET teacher training 210

4.4.2. Curriculum reform in Estonia (Nielsen, 1999b) 210

4.4.3. National dissemination of pilot project results in Romania (Nielsen, 1999a) 211

4.5. Example of stressing system connections in a VET policy evaluation for Croatia 212

4.5.1. Decentralisation and school headmasters’ and teachers’ training 212

4.5.1.1. Governance 212

4.5.1.2. Finance 213

4.5.1.3. Curriculum (content) 213

4.5.2. Curriculum reform and teacher training 213

4.5.2.1. Design and development of new curricula 213

4.5.2.2. New equipment 213

4.5.2.3. Curriculum delivery 213

4.5.2.4. Teams of teachers 213

4.5.2.5. Curriculum assessment and revision 214

Table of contents

From project to policy evaluation in vocational education and training – possible concepts and toolsEvidence from countries in transition 193

5. System change and the process and actors in VET policy evaluations 216

5.1. The complexity of systemic reforms 216

5.2. The path dependency of institutional change 217

5.3. Accomplishing change 217

5.4. The role of evaluations 218

6. Criteria, values and indicators underpinning VET policy evaluations 220

6.1. Evaluation criteria 220

6.2. Similarities and differences in VET reforms in western European countries 220

6.2.1. Improving progression from initial VET to higher education 221

6.2.2. Improving links between vocational education and employers 221

6.2.3. Improving the status of VET teachers and trainers 221

6.2.4. Improving the vocational curriculum 222

6.2.5. Summary 222

6.3. Potential and limits of the benchmarking approach 223

7. Conclusions and needs for further research 226

7.1. Conclusions 226

7.2. Needs for further research 229

List of abbreviations 231

Bibliography and references 232

List of figures

FiguresFigure 1: Basic concepts in evaluation 195

Figure 2: Denmark’s ranking – education sector 224

‘We shall not cease from explorationAnd the end of all our exploringWill be to arrive where we startedAnd know the place for the first time.’T.S. Elliot, Little Gidding, V

National and international donor agencies have ashared understanding of how and why to under-take evaluations. Evaluation in its classical defini-tion aims at comparing the design and implemen-tation of projects or programmes to actualoutcomes by analysing:(a) relevance to objectives and defined needs;(b) efficiency in providing inputs promptly and at

lowest cost;(c) effectiveness in achieving planned outputs

and immediate results;(d) impact on higher level objectives to which the

result should contribute;(e) sustainability over time, and especially after

the project/programme inputs have beenprovided and/or external assistance stops(Grootings, 2000) (2).

The following scheme summarises the rela-tions between these different criteria and aspectsof projects/programmes:

The scheme illustrates that:(a) efficiency evaluation has to do with the rela-

tion of means and expected results. Itassumes that means and expected results areclearly formulated;

(b) effectiveness evaluation deals with the rela-tionship between achieved results and imme-diate objectives. Again, this assumes theexistence of clearly formulated objectives;

(c) impact evaluation concerns the relationshipbetween immediate objectives and overallobjectives. This assumes that both immediateobjectives and overall objectives are clear(Grootings, 2000) (3).

However, evaluation increasingly becomes acontinuous cycle that takes place at all stages ofa reform or change process. The aim is to reflecton policies and practice with a view to influencingthem. Such evaluation is meant to have animpact on decision-making processes, on howreform actions are designed or redesigned, oncosts, communications, etc.

According to Windham (2000), there are ‘threeprimary purposes of all assessment work in thefield of education policy: to assess the nature andmagnitude of the opportunities and constraintsthat face the systems that provide education andtraining; to assist the government, the privatesector and the individuals in establishing prioritieswithin a resource-constrained environment; andto specify options for exploiting the identifiedopportunities and for dealing with the identifiedconstraints. Alternative goals and strategies willbe proposed from which government and its

1. Introduction

Figure 1: Basic concepts in evaluation

Overall

Project purpose

Results

Activities

Means

Allocation

Action

Utilisation

Change Impact evaluation

Effectiveness evaluation

Efficiency evaluation

(2) Evaluation criteria and their definition are based on documents from the European Commission’s Joint RELEX service for themanagement of Community aid to non-Member States (SCR – now EuropeAid Cooperation Office). However, identical presen-tations can be found with the World Bank, the Organisation for Economic Cooperation and Development (OECD), UnitedNations organisations and others.

(3) Ibidem

partners can make concrete recommendationsfor policy reform.’

Evaluation in this sense is no longer just areflection on the outcomes of some project orprogramme, but a type of consultancy workproviding advice and guidance and the basis forinformed decision-making. This makes criteria,such as relevance, effectiveness, efficiency,impact and sustainability relevant or irrelevant.However, project/programme evaluation andpolicy evaluation are often closely linked and feedeach other.

Also, international literature suggests a trendaway from evaluating the impact of singleprojects. A review of evaluation practice ineducation (OECD), VET (Grubb and Ryan, 1999)and labour market policy (Schütz et al., 1998,among others) signals a significant change inevaluation approaches towards target-orientedapproaches in labour market policy and asystemic approach in VET. These shifts arecaused by a growing awareness of the restrictedcontributions of single projects and the limitedimpact they can have, even if highly sustainable.Reviews of international cooperation (e.g. Kingand Buchert, 1999) similarly observe a shift awayfrom project-based funding to policy assistance.As relevant publications suggest, the Danishinternational development assistance (Danida)and the UK Department for International Develop-ment (DFID) are increasingly following thisapproach, as is the European Commission inforeign aid policy vis-à-vis the countries and terri-tories eligible for the MEDA programme (theEuro-Mediterranean partnership). However, whilea pretty robust methodology for project orprogramme evaluation exists (even up to thepoint that there is now an evaluation of the evalu-ation methods), this is not the case for policyevaluation (Grootings, 2000).

We admit that the distinction between policy,which broadly encompasses political intentionand may be expressed through severalprogrammes, and particular programmes, whichare themselves made up of many projects, maybe somewhat arbitrary. Sometimes a policy is

virtually identical with a programme. However, therationale for this distinction is to emphasise thatwe are more and more involved in broader policyevaluation. The evolving human capital imperativeand its interrelationship with education encour-ages decision-makers to expand or alter theirviews regarding the nature of useful evaluation.Policy-makers increasingly want informationregarding systemic, not only operating unit,performance. This is also the case for the partnercountries with which the European Training Foun-dation (ETF) (4) is working. They ask for advicerelated to future policy directions rather than justsupport with the design of a certain project.

The ETF has both an analytical and a develop-mental role, i.e. directly or indirectly the Founda-tion is involved with the implementation outcomesof its evaluations. The main tasks include:(a) to identify gaps and needs for intervention or

assistance;(b) to assist with the identification of needs or

the formulation of policy objectives;(c) to identify and design reform proposals in

transition countries;(d) to review progress and evaluate the

outcomes of VET reform programmes;(e) to manage the dialogue with policy-makers and

other stakeholders from our partner countries.The analysis of an individual project or a

limited number of themes would be too restrictiveand the strategic perspective too narrow for theabove assignments. Hence, VET policy evalua-tions are of strategic importance for the Founda-tion or, generally, for anyone who assumes, orprepares for, a development role in a domestic ora foreign context.

We explored the contribution of research andexamined a number of examples from transitioncountries with a view to illuminating, wherepossible, VET policy evaluations. In line with theremit of the ETF, we have concentrated on, andused examples from, transition countries incentral, eastern and south-eastern Europe, aswell as from our Mediterranean partner countries.

The aim of our study is to clarify underlyingconcepts, methods and possible tools used in

The foundations of evaluation and impact research196

(4) The European Training Foundation is an agency of the European Union whose mandate is to support VET reform in its40 partner countries, comprising central, east and south-east European countries, the successor States of the former USSRand Mongolia, as well as our Mediterranean partner countries and territories

VET policy evaluations. Our search for moreconceptual and contextual clarity and possibleinstruments is, not least, connected with thehope that a shared understanding among evalua-tors can contribute to an increased coherence ofpolicy recommendations and a clearer rationalefor establishing priorities.

Chapter 2 points to the fact that there is rarelytransparency for the methodologies deployed in(vocational) education (and training) policy evalu-ations, quoting the example of the Organisationfor Economic Cooperation and Development(OECD). However, in a number of evaluationscarried out or commissioned by the ETF, there isevidence of a building-blocks approach whichwas used to structure the evaluation and which isfurther explained in this chapter.

Given the limits of methods derived frommanagement thinking, we argue in favour ofhumanistic approaches to understanding VETsystems. Describing the hermeneutic approach,Chapter 3 discusses how a particular system canbe understood. It also refers to the need to putcertain phenomena in their historical context (thestructural-historical approach) which helps tounderstand why existing practices, functions andstructures are as they are. Finally, we refer to thecontinuous interaction between understandingand explanation, as explanations are neededwhen we fail to understand.

Chapter 4 examines VET as a (sub)system andelaborates on both the concepts of internal andexternal consistency of VET systems and theinherent tensions between flexibility and consis-tency, which an evaluator must grasp whenundertaking VET policy evaluations.

Chapter 5 places VET policy evaluations in thecontext of system change. Viewing policy evalua-tions as a consciously organised, critical reflec-tion and learning process involving the socialactors and, hence, as formative and construc-tivistic, we emphasise their indispensability in anydevelopment process.

Chapter 6 explores the possibility of estab-lishing predefined lists of evaluation criteria,values, policy priorities, indicators or otherassumptions underpinning VET policy evaluationsand argues why these should be subject to nego-tiations with the social actors involved. Further-more, we review first experience with the use ofthe benchmarking approach on an internationalscale and point to its potential and limits.

Chapter 7 includes the executive summary andoutlines needs for further research.

Finally, there is a list of references and a struc-tured bibliography of background literature on thetopic.

1.1. Methods of work

A working group was set up within the ETF toexamine and compare concepts and methodsused for VET policy evaluations. Group membersreached understanding through collaboration andmutual learning. The study method was a struc-tured collaborative writing process where workinggroup members generated the results of theiranalysis in and through writing. Group membersengaged both in an extensive literature reviewthat included the European Commission, OECD,Cedefop, the World Bank, ILO, BiBB, the Leipzigand Vienna groups of transformation researchers,plus selected evaluation reports from our partnercountries and many others.

The project also benefited from workshopsinvolving both working group members and inter-national experts. Two expert meetings wereorganised: one in February 2001 bringingtogether representatives from major internationalorganisations operating in transition countries,and a second in November 2001 with evaluationexperts from European countries. Moreover, allgroup members were involved, over the past twoyears, in a number of VET policy evaluations invarious transition countries.


VET policy evaluations cover a multi-disciplinary,highly complex terrain. Concepts and methodsare derived from sociology, economics,psychology, history, pedagogy, philosophy,management and organisational theory, compara-tive education and VET research, etc., whosescientific theoretical positions and methods haveto be integrated. Furthermore, using VET policyevaluations as a platform for identifying,designing or redesigning and monitoring policiesor programmes/projects, it was clear that somekind of implementation research would be asimportant, since, unavoidably, some form of inter-vention, or a type of applied, forward-lookingresearch, is involved.

However, the methods and tools that supportfirst analysis and then implementation areexplored in international literature only in outlineand often remain implicit rather than explicit. Thatliterature conveys other aspects: the values andassumptions that underpin policy; the informationand analysis that feeds in and out of policyformulation and determination; and the researchbase that exists or is created to support policydetermination and implementation.

2.1. OECD

The OECD as one of the major internationalplayers in the field of (education) policy evaluationsuggests that theoretical underpinnings derivefrom the choice of experts rather than being anexplicit part of an overall approach. At the Foun-dation’s workshop in November 2001, Whitman(2001) set out the approach of the OECD tosystem reviews: client country consensus; aframework of values and criteria; a careful choiceof experts carrying their own cultural andresearch hinterland; and legitimacy from both theclient country and the general secretariat of theOECD. The main purposes of the evaluations assuch are:(a) to link educational policy to national, regional

and economic issues;

(b) to place each country’s system in an interna-tional comparative perspective;

(c) to make policy recommendations to govern-ment (being of greater benefit to partnercountries and other donors by not itself repre-senting a donor organisation).

While the two underlying themes (or values)are: (a) the emphasis on active labour market policies

and their effectiveness;(b) improving labour force skills and compe-

tences through wide-ranging changes ineducation and training systems.

Whitman’s exposition is useful, since else-where in OECD literature (even in country reviewoutcomes) there is rarely transparency for themethodologies deployed.

2.2. The building-blocks approach

Analysing evaluations that had been carried outor commissioned by the ETF, we found that someform of the building-blocks approach had beenapplied in certain cases. The notion of buildingblocks was first mooted by Parkes (1995a). Theapproach was used for evaluations both inBosnia-Herzegovina and in Moldova as a basisfor constructing the VET ‘green’ and ‘white’papers. Furthermore, the approach was used inthe Maghreb countries (Algeria, Morocco, Tunisia)as a common framework for comparison,comprehension and subsequent cooperation;and in Uzbekistan as a means of evaluatingpolicy determination and implementation. Theapproach has also appeared in project terms ofreference drawn up by the ETF, most recently forArmenia, Montenegro and Turkey, etc.

The metaphor ‘building blocks’ referring to aneffective VET (sub)system was introduced first tokey actors from Baltic countries and then to areview of curricular reforms of the 1990s in ninecentral and east European countries. In theirinitial use, building blocks were defined in termsof functions or process. It was suggested that allsuccessful VET systems, independent of their

2. The limited transparency of concepts and methods used in VET policy evaluations


cultural and historic contexts, need to possessthe following ingredients (ETF, 1996a) (Parkeset al., 1999, p. 27): (a) ‘to be able to define occupational sector

priorities (on the best possible evidence avail-able);

(b) to be able to identify the appropriate occupa-tional sector competences and skills required(and to construct the institutions and tools todo this);

(c) to be able to turn these into curricular profilesand programmes and measurable standards;

(d) to deliver these at school level (including thecapacity to transfer from pilot to systemlevel);

(e) to help make the processes attractive tostudents and teachers (transferability, visibilityand portability of qualifications for studentsand working conditions for teachers);

(f) to provide for timely and effective feedbackthrough evaluation, monitoring, quality controland tracer studies of school-leavers.’

It was recommended that these elements beundertaken in the context of establishing thebalance between general and vocational provi-sion and in the context of transparent andaccepted approaches to standards, certificationand qualification. They also have to be related toother factors: financing mechanisms; changes inthe location of decision-making; a credibleresearch base; and the development of manage-ment capacity, the acquisition of appropriatetools in management, as well as curriculumdevelopment per se.

While we can agree with the specified func-tions, which need to be undertaken in anymodern VET system, there are some problemswith the use of metaphors. It is, in essence, afunctionalist analysis and should, hence, becalled this way.

Added later on to the notion of function orprocess was the development of an architecturalelement. This was to offer an operational modelfor transition country working groups set up toanalyse existing structures and practices and tomake proposals for change in a consistent way,in order to have a common agenda amongministries and agencies. Eight topics (buildingblocks) were formulated:(a) educational management and administration; (b) curriculum, assessment and certification;

(c) financing of VET;(d) the labour market and social partnership; (e) educational standards and quality control; (f) in-service teacher training;(g) legislation;(h) labour market and adult education.

Each of the building blocks was then analysedin detail, for example, by answering the followingthree key policy and action questions for thebuilding block of legislation: (a) should there be separate or integrated legis-

lation for general education and VET?(b) should existing legislation be left largely

unchanged with modification to regulationsonly or should there be clear, new legislation,indeed an Education Reform Act?

(c) if new legislation, should it be short, simple andtransparent, leaving the detail to regulation?

Judging from the evaluation report of therespective Phare project under which the exer-cise was undertaken (Nielsen, 2001),Bosnia-Herzegovina can be considered a verysuccessful example, in which the building-blocksapproach was used to structure and systematisethe analyses by the actors involved and then toprovide a framework for a concept paper forlegislation. The building-blocks approach helpedestablish a common conceptual grasp of theissues at stake and a common language in whicha relatively large group of key actors coulddiscuss structures, functions and institutions of aVET system in transformation. According to thereport, it turned out to be a useful tool for speci-fying well-grounded and specified VET reformstrategies, encompassing all elements of VETsystems.

In the cases we reviewed, the building-blocksapproach was an attempt to provide a simple,transparent vehicle for managing the dialogueamong country key actors, donor representativesand consultants, i.e. between the researcher andthe practitioner. It was an attempt to balanceactual complexity with simplicity and trans-parency in the use of tools, which is consistentwith Grubb and Ryan (1999) who recognise theneed to find a way to express a holistic approachin a manner convincing to policy-makers: ‘In theselective use of evidence, policy makers haveavoided the complexities of theory and method-ology. Instead they have often used socialscience results in simplified ways, to tell almost

commonsensical stories consistent with theparticular ways in which they want to frame prob-lems. In this habit they are reinforced by journal-ists, whose methods of personalising issues byconcentrating on the experience of individuals –

story telling – makes the issues more vivid in thepublic mind, but [...]’.

However, as the ending indicates, the approachalso leaves a number of unresolved questionswhich will be discussed in the following chapters.


Instead of methods derived from managementthinking, we agreed on a phenomenologicalapproach, in which the emphasis is on describingphenomena, as they appear to the observer,instead of attempting to guess or theorise aboutwhat might be behind these phenomena. Thebasis is first and foremost to establish a genuineunderstanding of what you observe.

Having said this, we asked ourselves whetherwe can identify a number of methods or tools(Krücken für die Gedanken) actually made use ofby experienced VET policy evaluators. Whichmethods do the latter use to come to an under-standing of VET? And what is the correct rela-tionship in evaluations between understandingand explanation?

3.1. The hermeneutic approach

In more specific terms, we may talk of ahermeneutic approach that covers a number ofstipulations and various methods related to theunderstanding and interpretation of phenomena.Hermeneutics play an important role in humanand social sciences; an important thinker in thisfield is Hans-Georg Gadamer (1972).

A hermeneutic approach seeks to elucidatewhat happens when you try to come to grips witha text or any other phenomenon in society. In thedescription of how to come to an understanding,a specific concept is normally used: the conceptof the hermeneutic circle. This refers to the factthat the particular element is understood in termsof the totality, while the totality is, at the sametime, understood through its constituent parts.Through our interpretation work we will achieve amore and more secure, detailed and variedunderstanding by way of a continued circularalternation between studying parts (systemelements) and totalities (the total system). Thereis an additional circular process in the interpreta-tion work: the interaction between the precon-ception (Vor-Verstehen) of the phenomenon,which the researcher/evaluator brings with

him/her, and the new understandings worked outthrough the process.

An important point for discussion onhermeneutic approaches is the question ofwhether objectivity is possible. Some hermeneu-tics would argue so. Karl Popper, for example,claims that you can avoid wild interpretations andarbitrary postulations, which are a clear risk inhermeneutic understanding, by going through anumber of well-conceived loops for testing yourinterpretation hypotheses. However, Gadamerquestions this position.

Gadamer (1972) insists that it is not possible tocome to an understanding of a phenomenon, forinstance a foreign VET system, just by followingcertain methodical procedures. According toGadamer, the interpreter cannot abstracthim/herself from history and culture in trying tounderstand, as the researcher is her/himself partof history and culture. Understanding alwaystakes place between two entities both placed intheir own historical and cultural contexts. Whatthe researcher/evaluator must do is openly recog-nise and clearly articulate this tension. Anotherconclusion for us is the involvement of evaluatorscoming from the reviewed country itself or, in thecontext of external peer reviews, from countrieswith similar education systems.

3.2. The structural-historicalapproach

Building blocks or ingredients may be a useful,simple tool to structure (or manage) evaluation,but the approach also leaves a number of unre-solved questions. One is the use of metaphorsinstead of clearer concepts; but let us continuemetaphorically. If these (eight) building blocks arethe bricks, what then is the mortar holding thebuilding together? Can we provide a conceptualframework for grasping not only the parts(building blocks) of the VET system, but also itshistoric roots and dynamic relationships? Whicharchitecture structures the VET system under

3. Humanistic approaches to understanding VET systems

observation and gives the constituent parts theirspecific meaning? Are there different systemiclogics behind the way building blocks are puttogether in the different countries? How can wedig deeper into the inner driving forces behind theconfiguration of the building erected over manyyears?

A structural-historical approach (Strukturell-genetischer Zugang) could be applied. Here, theanalysis would start with an overview of existingpractices, functions and structures as theyappear to the (experienced) evaluator. Essentialstructures are then traced back to their origin(genesis), and an understanding/explanation ofthe historical context is reached in which thespecific phenomenon was born or established.Described through the comparative, structural-functionalist mirror, for instance, the overallDanish VET system may seem a wonderfullyharmonic building that consists of highly refinedbuilding blocks (Cort, 2002). But analysed histor-ically (through the crystal ball instead of themirror), this VET system is a patchwork of polit-ical compromises reflecting what was achievableunder given historical-cultural circumstances overthe last 100 years. How do we come to such anunderstanding of other VET systems in evalua-tions, using the building-blocks approach?

Another question related to the understandingof the systemic logic behind the building blocksis the challenge of change. How do VET systemschange and what are the driving forces behindchange processes? Even if we get the descriptionand understanding of the building blocks right inVET policy evaluations, we cannot at all be sureto have fully established the capacity to predictchange or to formulate the right interventionstrategies to achieve change. Is there a strategiclever at force in VET system change? What is(empirically and theoretically) known aboutdynamic forces, catalysts for change, etc., andwhat is the right balance between top-down (poli-tics) and bottom-up (market) approaches? Whatfollows is that the building-blocks approach mustbe accompanied by a deeper insight into organi-sational and institutional processes of change.We will come back to these issues in Chapter 5.

Yet another problem is how to cope with therisk of preconceptions and nationally biasedunderstandings of what makes up a good VETsystem. It is not uncommon to see the foreign

evaluator measure the distance between VETreform initiatives in a transformation societyagainst the advanced state of his/her ownsystem. The first question is: are the buildingblocks per se culturally bound? The next questionis: how can we make use of them in evaluations,being nevertheless reasonably objective andtransparent? One way around this dilemma is tomake statements of values behind evaluationsexplicit, when using the building-blocks method.Chapter 6 will deal specifically with this issue.

3.3. Understanding andexplanation

It has been argued above that the phenomeno-logical/hermeneutic method is primarily used toachieve an understanding rather than for expla-nations. If we accept this point of departure, thenwe would argue that we should not give up usingthe building blocks as an interpretative frame-work between the evaluator and the VET systemunder observation, but we should definitely putmore emphasis on explanations.

Understanding and explanation are differentways of (re)cognition. Understanding is a moreimmediate experience or recognition of aphenomenon. Sometimes, if a phenomenon isdifficult to understand, we need explanations tocome fully to grips with, say, a component of aVET system and its precise role within the system.When you explain something, you establish somedistance to the phenomenon under observation.On the basis of explanations, you may oftenunderstand the phenomenon better or view it froma different perspective. To explain something is tostate causes behind the phenomenon under anal-ysis, external to the phenomenon itself. To under-stand a phenomenon is to give a reason for it,which is internal; meaningful understandingpresupposes a communicative communitybetween the evaluator and what is being evalu-ated. Understanding a phenomenon, in casu theVET system, means that one can supposedly see(recognise, realise) how the elements fit together,and see the meaning of the phenomenon (inten-tion, purpose, function), often in an immediate,even intuitive way.

The relationships between understanding andexplanation are tricky; one form of recognition is



not better or more correct than the other. Theargument here is simply that we should reflect onthese connections and deliberately seek toestablish a continuous interaction betweenunderstanding and explanation.

The following are but a few examples of usefulexplanatory frameworks:(a) economic or labour market or sociological

laws and other forms of determination of theconcrete conditions under which aphenomenon is active, for example, the condi-tioning factors behind VET systems, such asdemography, trends in labour market develop-ments and broader educational policies;

(b) functionalist explanations referring to thetotality in which the phenomenon to beexplained is placed and a description of theprecise function it serves; for example the roleof a national VET system for societal functionssuch as qualifying, socialising, sorting, andbeing a depository of young people;

(c) historical-cultural explanations behind theoccurrence of different institutions and prac-tices in contemporary VET systems, forexample: why is there such an assortment ofindependent VET providers (productionschools, etc.) in Denmark compared toSweden? Because of Grundtvig and the freeschool and folk high school tradition;

(d) structural(ist) explanations of phenomenawhich can only be uncovered through anin-depth analysis, such as the specificconnections between the economic, socialand political systems in social structures,making transition of VET systems in manyformer communist countries very difficult;

(e) system-analytical explanations showing howchanges in one component will havespill-over effects on other system compo-nents or on the total system, cf. the followingdiscussion on internal and external consis-tency, such as the role played by access tohigher non-university education as a determi-nant for the attraction of VET.

Central to this reasoning is that understandingand explanation feed each other, that under-standing is the point of departure for VET policyevaluations, and that explanations are neededwhen we fail to understand.

But we easily and often fail to understand.Awareness of one’s own limited or culturally bound

understanding is as important as ever. Four exam-ples may illustrate the problems evaluators face:(a) the trendy catchwords broadcasted by the

OECD, the European Commission, the WorldBank, etc., in recent years have mainlyfocused on modern competences, such asadaptability, changeability, responsibility forone’s own learning, etc.;

(b) the cultural school of thought in EuropeanVET research (around Anja Heikkinen fromJyväskylä University, Finland) argues that thisis a frontal attack against VET institutionswhich the labour movement has been able toestablish only after more than 100 years ofhard struggle. The change of emphasis fromthe sociological qualification to the psycho-logical competence is bound to remove socialcontrol from VET. And it undermines thesocial category of the skilled worker;

(c) the lifelong learning concept is now recom-mended everywhere. As evaluators we tendto measure VET system developmentsagainst the indicators found in or around theEuropean Commission’s memorandum onlifelong learning (European Commission,2000a). We tend to do so even in countrieswhere the most rudimentary conditions forinitial VET are not in place;

(d) in our thinking on modern and adequate VETprovision, we tend to promote output-based,modularised and credit accumulation-basedmodels with little emphasis on process factors(curricula, teachers, pedagogics, time-servicingrequirements). We promote the Anglo-Saxonmodels without remembering why they had nochoice but to configure the system the way it is;

(e) the complete destruction of the East Germanpolytechnic education system was the result ofa lack of understanding of the intrinsic qualitiesof that system. Entirely lost is the expertisedeveloped in work-based learning principles(Lernen in der Arbeit), didactics and practices,which were further developed in the socialistcountries at the time than anywhere else.

Therefore, besides developing further thebuilding-blocks approach by including explana-tions, and in order to avoid hobbyhorse explana-tions, we have to cultivate, individually and in acommunity of VET evaluation practitioners, a newattitude towards argumentation. We need a moreopen discussion practice: a discourse.

In our effort to understand VET systems (or thesystem context of VET policies), it seems obviousthat we have to apply some kind of system anal-ysis. However, it is quite difficult to define a VETsystem: (a) how is the system universe to be delineated? (b) what are the constituent components, parts,

units? (c) which relationships exist between units?(d) what matrix of dominance patterns the inter-

play of units?(e) where are the boundaries of the system

located?(f) which relationships (metabolism) exist

between the system and its environment? (g) what is the prime mover or the strategic lever

for change? The set of questions is very complex and

forms part of a broader scientific context. It ishelpful to go back to the roots of system analysisand to analyse, from a theoretical standpoint,issues related to internal consistency andexternal consistency of VET systems and theirconsequences for VET reforms, and hence VETpolicy evaluations.

4.1. System analysis

The VET field is short of a comprehensive theoryand in many ways remains a very young researchfield. Perhaps one could classify the researchtradition as follows. First, there is work within thetradition of general educational theory. A secondgroup of topics relates to the study of thoseorgans of the VET system which exist by law andare recognised in official documents about VETstructure, functions and actors. Third are studieswhich take these formal institutions as given andexplore the forces, including the informal institu-

tions, which set them in motion and link theiractivities.

A whole range of topics is opened for discus-sion, as soon as one asks the question: whatmakes the VET system tick? One can thenexplain things in terms of functions; we can lookfor systemic interrelations between the data, witha view to discover what goes with what, whatchanges in parts produce what reactions in thewhole and in other parts. We can treat functionsas heuristic, as a notion that will help us to spotrelationships and so to define structure as asystem of parts with internal and external rela-tions. An interesting example of this approachwas the early 1990s research component of theEuropean Union’s (EU) PETRA programme oninnovation of VET programmes. The researchquestion asked was ‘through which mechanismsare VET programmes renewed?’ instead of asking‘who does what when?’.

General systems theory is a field of thoughtthat has been growing in strength over more than50 years. Its origins go back to biologists whoopposed reductionism, i.e. an analysis of livingthings merely as a sum of parts (which was aview contradicted by observation). Parallel move-ments existed in the psychology of perceptionand learning (Köhler, 1959) (5), and in socialanthropology, in which structure/function analysisalready began to flourish in the 1920s. But thesystemic approach is more than just an exampleof the use of biological analogies to describesociety. Much impetus was given by mathemati-cians and electrical engineers during World War IIand immediately after.

General systems theory would not claim to beempirical; rather it claims to be axiomatic, in thatthe mathematics conceived first for the electronicsystems has been so far generalised that it isapplicable to any systems whatever, provided thatthe rules are followed: systems are isomorphic.

4. VET as a (sub)system and issues of internal and external consistency

(5) Interestingly, the Gestalt concept has been applied very fruitfully by German VET researchers; cf. Felix Rauner, University ofBremen/Institut Technik und Bildung (ITB), and Gerald Heidegger, Universität Flensburg/ Berufsbildungsinstitut Arbeit undTechnik (BIAT), who make use of the Prinzip Gestaltung as a way to conceptualise innovation in VET.


The meaning of the term ‘system’ is oftenconfused. The most general definition is formu-lated by the founding father of the general systemtheory, von Bertalanffy (1950), after he hadnoticed that in physics, biology, psychology andsocial sciences it was no longer acceptable ‘toexplain phenomena by reducing them to an inter-play of elementary units which could be investi-gated independently’. ‘A system is a set of unitswith relationships among them; the word “set”implies that the units have common properties –the state of each unit is constrained by, condi-tioned by, or dependent on the state of otherunits. The units are coupled. Moreover, thesystem as a whole has “got something” which itscomponents separately have not got. Systemsmay be concrete or abstract, systems may be“open” or “closed”, and systems can be analysedon the principle of systems and sub-systems –systems within systems within […] ultimately awholly general system. There are theoretical diffi-culties about this ultimate system: but the prob-lems of supra-systems and sub-systems, levelsand boundaries, etc., are generally manageable ina practical way.’

The German sociologist/philosopher NiklasLuhmann (2001) has built up a general systemtheory of society, where the educational system isa subsystem having its own logic, laws of motion,discourse, etc., and is even further differentiated.

4.2. VET as a subsystem

VET as a subsystem can be seen like other livingorganisms as an open system in a steady state; itdepends on self-regulating mechanisms to main-tain its boundaries and its continued existencewithin these boundaries (meaning primarilyboundaries which distinguish the VET subsystemfrom other (sub)systems of the same society).The steady state depends on a balance of inputsand outputs. The inputs are demands andsupport: support makes the system strongenough to process demands and to produceoutputs in the form of qualifications. The VETsystem sits in an environment – the total social

system – and there is a continuous feedback intothe VET subsystem, signalling whether outputsproduce good or adverse effects in the environ-ment. A systemic approach to VET evaluationsfocuses on the analysis of relationships, commu-nication channels, and responsiveness andadaptability, based on the fundamental under-standing that changes in one component lead tochanges in other components and in the systemas a whole.

It is important to understand the specific logicof the VET system, to explore whether the systemis internally and externally consistent, and inwhich way the system needs to be regulated tobalance available tensions.

4.3. Internal and externalconsistency (6)

Improving VET is possible only through asystemic approach, and we provide argumentsbelow why this is so. Evaluations would have toestablish to what extent VET policies havefollowed a systemic approach, applying in partic-ular the concepts of internal and external consis-tency. At the same time, the evaluator needs tobe aware of important system-inherent tensionsthat are not at all easy to reconcile.

In transformation societies the main driver forreforms is the fact that the whole logic of VETsystems has radically changed. Before thecollapse of the planned economies, there was aclose relationship between the human resourceneeds of companies and vocational schools.Based on a precise manpower forecasting (plan-ning) model and a vocational guidance systemwhose main aim was to steer young people intocertain occupations in line with available job posi-tions, schools could almost exactly cater for theneeds identified by companies. People tended toremain in their jobs for a lifetime. With the changeto market economy principles, this system nolonger worked. The labour market underwentdramatic changes within a short period, callingfor different skills than those catered for byschools. Schools had been slow to adapt, results

(6) Section 4.3 and subsections 4.3.1–4.3.3 draw heavily on Durand-Drouhin and Bertrand, 1994.

include a large number of young unemployed andinadequate skills upgrading or retraining systemfor adults. There are tremendous conceptual,political, financial, structural, organisational andcontent-related problems to be solved at thesame time. This concerns internal factors of VET,but above all the need to establish new mediatingmechanisms between education and work toreplace the former administrative ones.

Most countries have placed new emphasis onthe need for more responsiveness of VET systemsto changing needs for education and training. Twokey concepts, flexibility and consistency, are asso-ciated, to variable degrees and in variable combi-nations, with the search for responsiveness.

Responsiveness is needed with regard to:(a) constantly changing and largely unpredictable

requirements of labour markets for skilledlabour;

(b) growing demands by young people for higherlevels of education and for more diversifiedand individualised learning processes andpathways.

Responsiveness can be interpreted and maybe achieved in a variety of ways. In institutionaland management terms, three major develop-ments can be identified.

First, there is a clear trend towards moredecentralised governance and management ofeducation systems in general and of VET inparticular. Examples include France, which in theearly 1980s, delegated authority and responsi-bility for the location and equipment of vocationalschools largely to the regional level, as well asseveral Scandinavian countries which transferredresponsibility to local authorities and schools.The latter were given a great deal of initiative,including curriculum design and implementation,in order to satisfy diverse needs in localeconomies and of individual students.

Second, the role of central government ineducational planning is declining, while industryis called upon to participate more actively in thedesign and provision of VET and of related certi-fication and qualification systems. While theparticipation of employers and their organisationsis expected to contribute to improved respon-siveness of VET to both quantitative and qualita-tive demands in the labour market, the involve-ment of trade union organisations and workers’councils varies from country to country.

A third trend is the development of training inenterprises and in private institutions and therelated emergence of a training market wherepublic and private institutions may be competing.This trend is related to the growing importance ofadult training, which can itself be seen largely asa response to the demand for a more flexibleworkforce.

In terms of the structure, content and organisa-tion of VET, the search for responsiveness has inparticular led to:(a) the diversification of training programmes and

pathways;(b) the broadening of curricula;(b) the increased duration of initial training;and in some countries:(a) the modularisation of courses and certificates;(b) the introduction of competence-based learning

and assessment.All these developments are assumed to

contribute to the adaptability of individualworkers and to the flexibility of educationsystems. However, they also raise serious ques-tions about the types of qualifications whichyoung people acquire, about the costs and effec-tiveness of education pathways and about theconsistency of the rules of the game prevailing indifferent sectors of national education systemsand in the labour market.

These issues seem to be of interest to centraland east European countries, which are goingthrough a process of transition, from centralisedand rigid education and training systems towardsother types of systems, expected to meet thedemands of market economies. Considering that,in the present context of uncertainty, thesedemands are to a large extent unpredictable, theneed for adaptability and flexibility would seem tobe even more pressing in countries in transitionthan in most west European countries.

The implications of such orientations concerningthe structure, organisation and management of VETsystems should be carefully examined. There arecertain tensions between objectives of flexibility inVET on the one hand and consistency on the other.The underlying question is whether flexibility andconsistency can be conceived of as complemen-tary rather than contradictory features of educationand employment systems and which regulatorymechanisms deserve particular attention in thisrespect.



This question will be considered from threepoints of view: the internal flexibility and consis-tency of education and training systems; theexternal responsiveness of education and trainingsystems and the consistency between theirinternal organisation and objectives and the func-tioning of labour markets; and, finally, differentapproaches to system regulation.

4.3.1. Internal consistency

4.3.1.1. Consistency between subsystems andclarification of pathways

Introducing flexibility by diversifying educationpathways has also meant the creation of evermore sinuous, long and unfocused educationalcareers. Moreover, while countries have movedcloser to flexible and open systems, their strate-gies of reform have tended to concentrate on onetype and/or level of education and training at atime, rather than explicitly aiming at complemen-tary transformations of all stages and in all rele-vant areas. And yet the consistency of learningopportunities and pathways across types andlevels of education and training is essential.

One important aspect of the internal consis-tency issue is the relative attractiveness of VETvis-à-vis academic education. Many efforts toattract young people into vocational streams donot take a sufficiently broad view of the relation-ship between the different streams. For instance,one of the reasons for the preference foracademic streams is the wider opportunities thatthey provide for further studies. But attempts toopen up access to higher education to vocationalstudents often fail, because universities operateon a completely different logic and use differentselection criteria.

The two examples illustrate that improving VETis possible only through a systemic approach,encompassing all the components of the system.Such an approach may conflict with a policywhich would promote a high degree of decentral-isation and of privatisation.

4.3.1.2. Regional consistencyWhen far more autonomy is given to schools, notonly from the point of view of management butalso of teaching content, the problem of possibleimbalances and inequalities between regions,schools and individual students might arise. Forthe central and east European countries, with

their strong egalitarian tradition, this raises thequestion of how to overcome such problems.

Most English-speaking countries, with tradi-tionally highly decentralised systems, shared aconcern about the multiplicity of assessment andcertification systems. This was one of the reasonswhy the UK and other English-speaking countriesengaged in the development of national voca-tional qualifications. They are not, in contrast tomany continental European countries, related toremuneration systems through collectivebargaining at national or sectoral levels. This isconsidered as an additional element of flexibility.However, the value of such qualifications andtheir motivating effect on young people and adultlearners may also be weaker than that of moreholistic and rigid qualification systems, such asexist in Germany.

4.3.1.3. Consistency for the individual learnerIn countries like the UK, the definition of nationalvocational qualifications is associated with amodular approach to training, allowing studentsto choose their own personal range andsequence of course/modules (which can beprovided by different institutions) and to accumu-late corresponding certification units. Youngpeople are thus – to some extent – encouraged todefine their own individual learning pathways.Modularisation is expected to allow for morerapid responses to changing labour demand.Flexibility is pursued both from the point of viewof students’ interests and of labour marketrequirements.

One of the possible drawbacks of modularapproaches is the risk that young people willacquire only loose collections or incompletepuzzles of narrowly defined skills or compe-tences. In such systems, responsibilities for thelevel of skills and the relevance and complete-ness of skill profiles with which young peopleleave the education system is largely left to youngpeople whose information, strategic planningcapacities and financial possibilities vary greatly.

4.3.2. External consistencyExternal consistency refers to the relationshipsbetween educational objectives and theeconomic and social context. In reality, there arepotential problems, such as the fact that theemployment system is not always able to articu-late its needs and that there are tensions between

the pedagogical logic of school systems andcompetence needs of employment. Furthermore,the drive to maintain a holistic approach oftraining for an occupation, and to developnational qualification systems as a means toincrease internal and individual learner consis-tency, may conflict with deregulated labourmarkets and the short-term recruitment andtraining practices of companies. It is also realisedthat education cannot by itself solve the prob-lems of the employment system. See, forexample, Hodgson (2001), Paquet (2001), Miegeland Nölke (1996), as well as Grubb and Ryan(1999): ‘Education without suitable employment,and specific skill training without jobs requiringsuch skills may be valuable in their own right butthey cannot enhance economic conditions. Andso the other conditions necessary for educationand training to be effective – the employmentnecessary, the capital required, the institutionsthat can give these arrangements some perma-nence – also need to be carefully understood,and the most successful programmes carefullyconsider the nature of local employment.’

In evaluations commissioned by the ETF wehave found a number of examples of internal andexternal consistency lacking in the VET policiesimplemented by transition countries of centraland south-eastern Europe. We include some ofthem in Section 4.4. In contrast, in an evaluationin Croatia, we have tried to highlight systemconnections which need to be considered whendesigning a consistent (in the given case,EU-funded school headmasters’ and teachers’training) programme (Section 4.5).

4.3.3. Regulating mechanisms The foregoing observations underline the need forregulation of VET systems themselves and of theinterface with their social and economic environ-ment in order to find the narrow path betweentwo dangers. On the one hand ‘a centralised,uniform and rigid system would not meet theneeds of a modern economy, would limit adapt-ability and threaten the ability to innovate’ (Wolf,1993). On the other hand, leaving individualschools and training institutions without anycommon framework of objectives and organisa-tional conditions would not satisfy the long-termeconomic goals of quality and effectiveness andis likely to increase geographic imbalances andsocial inequalities. Ideally, successful system

regulation requires both clear policy orientationsand effective regulation mechanisms, which needto be reviewed by VET policy evaluations. Clearpolicy orientations are particularly important in acontext of strongly decentralised systems. As faras possible, policies should be based on both asufficient capacity of technical analysis and ahigh degree of social dialogue. In institutionalterms, this implies the need for:(a) ‘a framework for consultation among the

various actors, which, at the national, localand/or sectoral level, guarantees somedegree of continuity, coherence and consis-tency, especially between education systemsand labour markets;

(b) some kind of research and developmentstructure, providing information and technicalsupport to decision-makers; […] this includes[…] the development and monitoring oflabour force information systems, the devel-opment of learning theory and appropriateteaching and training methods, etc.;

(c) clearly defined and agreed financing agree-ments which are a major element of thesystem regulation. This raises policy – orpolitical – questions about the respective roleof governments, enterprises and individuals;[…]

(d) finally, there is the broader issue of evalua-tion. The more decentralised the system, themore necessary it is to set up mechanismsallowing for the evaluation of its output andeffectiveness. Certification is an importantelement of evaluation and it has to be seen,not only from the point of view of recognitionon the labour market. The above commentsconcerning the limitations of a market-drivenapproach do not deny the fact that the ulti-mate value of qualifications depends on themarket and not on administrative decisions.But the need for consistency pleads for somedegree of cooperation between the actorsand for some form of institutionalisation inorder to regulate the market and transcend itsmost immediate and specific demands.’(Durand-Drouhin and Bertrand, 1994).

This brief overview of requirements or condi-tions of successful regulation refers to an idealsituation. In reality, it is clear that such require-ments cannot easily be fulfilled, especially in thepresent context of transition countries. To



mention only a few points, the institutions and theidentity of the various actors are only emergingand the inadequacy of technical instruments isreinforced by the lack of resources. But, in oneway or the other, responses to these problems,adapted to national cultures and traditions willhave to be found.

4.4. Examples of lack of internaland external consistency of VET reforms inBosnia-Herzegovina, Estonia and Romania

To understand the importance of systemic consis-tency (and to which phenomena the conceptsrefer) when evaluating VET policies, examples shallbe provided from three transition countries. Moreexamples are available from the ETF.

4.4.1. Teacher training in Bosnia-Herzegovina(Durand-Drouhin and Bertrand, 1994)

Part of the questions during the evaluation wereaddressed to actors and institutions outside theEU Phare VET reform programme with a view toassessing whether framework conditions wouldbe conducive to the longer-term sustainability ofthe pilot project. Obviously, one central factorhere is vocational teacher training, which had notoriginally been a project component. The pilotschools had been asked only to specifyteacher-training needs in relation to newly intro-duced curricula. This was changed later, and abudget reserve of EUR 200 000 allocated to run aprogramme for teacher and management training.

The following questions were asked in threeinstitutions visited: were existing teacher traininginstitutions involved? And were they familiar withthe curriculum principles applied in the Phare VETreform programme?

4.4.1.1. Pedagogical Academy in Sarajevo (PAS)in charge of education of primary schoolteachers

The PAS department for Chemistry was visited.The reason for choosing chemistry was that inalmost all pilot VET schools visited, it was arguedthat pupils are below standard and that particu-larly the new Phare module in chemistry is veryproblematic and probably the most difficult

subject. The two assistant professors interviewedhad never heard of the Phare VET programmeand its renewed module in chemistry, neitherwere they aware of poor end results of pupils inprimary schools.

The conclusion is that a clear problem existshere in terms of the vertical integration of theeducation system, which has a negative impacton Phare programme results. Only the Ministry ofEducation can solve this problem by promotinginternal consistency measures.

4.4.1.2. Sarajevo University, Faculty of Philosophy,Pedagogical Department, in charge ofpre-service VET teacher training

An interview was carried out with the Dean, Prof.Dr Vlado Sucic. Pre-service vocational teachertraining is subject-based and covers the fullrange of subjects taught in secondary schools.Teacher training is carried out according to aconsecutive model: first, students graduate in asubject (e.g. engineer), then there is a six-monthpedagogical-psychological-didactical-methodicaltraining course, which includes school practice.After two years of teaching practice at a VETschool, the candidate takes the final state exami-nation and becomes a certified teacher. Thismodel is also offered for teachers of practicalsubjects.

Until now, the University’s Pedagogical Depart-ment had always been involved in the design ofnew curricula for all subjects taught. This wasdone to ensure that new subject matter and inno-vative methods were integrated immediately intoteacher training. However, the department wasnot involved in the Phare programme and did noteven know of its existence.

The fact that no inputs from changes in VETequipment, curricula and methodology are fedforward and channelled directly into pre-serviceteacher training is a serious problem. The devel-opment model is not optimal: too little emphasisis placed on the teacher training system and thetraining of teacher educators. In addition, VETteacher training would capitalise on becoming animportant research field in Bosnia-Herzegovina.Any support programme should focus on theteacher educators to ensure a multiplying andlasting effect and to ensure that activities are‘anchored up’ within existing providers ofpre-service (and in-service) teacher training.

4.4.1.3. Pedagogical Institute, Canton Sarajevo, incharge of in-service VET teacher training

Tasks of the pedagogical institutes in Bosnia-Herzegovina in general include:(a) advising ministries of education on curricula

and new legislation;(b) collecting statistical and other information

from schools;(c) evaluating the quality of schools;(d) advising on the appointment of teachers and

assessing and confirming their qualifications;(e) providing professional development opportu-

nities for teachers and setting up in-servicetraining courses;

(f) evaluating school development plans. The institute has traditionally promoted a

subject-based approach to curriculum design anddelivery, which led to some opposition with respectto the newly introduced module-based curricula inPhare pilot schools. An interview was held with MrVreto Sadik, inspector in the field of mechanicalengineering at VET schools. His specific responsi-bility covers subject matter and training practice inthese schools. He is familiar with the Phareprogramme and had participated in two seminarsin the previous year, one on curriculum develop-ment and teacher training, the other on the dissem-ination on the Green Paper. He formed part of theteam who approved the new equipment specifica-tions. But he is only one of very few.

Pedagogical institutes need to be involved indevelopment projects along the lines of the PhareVET programme. The latter focused on 12 pilotschools, while the pedagogical institutes weretotally forgotten. Subject inspectors did notreceive any training; they see their subjects disap-pear in the long run or being parcelled out intonew modules, and they are less and less able toperform their supposed roles when going to theschools. This is a serious barrier to broad accep-tance of the reform. In fact, it may give ground toa large group of key actors who veto reforms.

The above three examples demonstrate asub-optimal reform environment resulting from alack of internal consistency of reform measuresintroduced.

4.4.2. Curriculum reform in Estonia (Nielsen,1999b)

The EU-funded Phare 1995 VET Reformprogramme in Estonia aimed at assisting thecountry in kick-starting VET reform, which would

be key to developing a workforce required for amodern market-driven economy. The specificactivities to be undertaken embraced thefollowing main components:(a) curriculum development;(b) teacher training;(c) provision of equipment;(d) establishment of links with partner schools in

European Union countries;(e) support to the design of a VET reform

strategy and dissemination of results.The curriculum design principle chosen in

Estonia is a flexible, modular-based andemployer-led model. In the development ofcurricula, Estonia follows a model similar to theIrish (FAS) one, but adapted to the Estoniancontext. 13 selected pilot schools developedmodularised curricula to nationally approvedindustry standards. Social partners, in particularthe Chamber of Commerce and Industry, areactively involved, at national level, in the definitionand approval of occupational profiles and, atlocal level, through school-based curriculumdesign groups developing course modules tonational standards.

The system logic of VET has changed. Amethodology has been developed and success-fully piloted to identify skill needs and educationalgoals and to translate occupational profiles intocurricula. This has functioned well. Employersand their representatives have started to influ-ence VET provision by articulating their skillrequirements and requiring schools to produceskilled workers according to these needs. Thedynamics of the VET system have been increasedand self-regulating mechanisms built into the verycore VET.

The results achieved show that the adoptedstrategy served as a catalyst for building thethrust and the momentum needed in the newapproach to curriculum development. The basicinstitutional frameworks are in place.

However, the curriculum support infrastructureis extremely fragile. Human resources in the VETDepartment of the Ministry of Education areseverely limited. Some years ago the Ministry hadalready transferred its responsibilities forcurriculum innovation to the national centre forexamination and qualifications (NCEQ). The centre,currently engaged in other assignments such ashandling national examinations, is methodologi-



cally and technically clearly too weak. It has notbeen in a position to develop sufficient capacitiesto absorb Phare programme results, such asendorsing the newly developed modules or drivingthe reform process forward. In 1999, about130 modular descriptors that had been submittedto the centre were still awaiting their approval andaccreditation. The centre, in its function as thenational vocational qualification institution in theAnglo-Saxon philosophy, constitutes the basiccornerstone in the whole VET system logic. If thesystem is to work well, the capacities of the NCEQto set up a national qualification structure and toact as the national award and accreditation bodywill have to be built up quickly. Staff needs to berecruited and trained.

This is a clear threat to sustaining VET reformresults. The NCEQ needs to be given a clearmandate and to attract new experienced staff. Inaddition, it requires immediate technical supportwith a view to building up its capacity. Simply notendorsing the modular descriptors developedunder the Phare programme, as was the case,puts both the momentum of the whole curriculuminnovation process and the still fragile interactionbetween the employment and the educationsystem seriously at risk.

It follows that there is a lack of external consis-tency, owing to problems of consistency betweeneducational objectives on the one hand and theeconomic and social context on the other.Although attempts to increase the responsivenessof VET to the changing needs of the economywere a stated priority, there was only limitedevidence of the implementation of this objective,and the educational infrastructure has continuedto operate primarily according to its own logic.

4.4.3. National dissemination of pilot projectresults in Romania (Nielsen, 1999a)

The Phare VET Reform project in Romaniaselected 75 VET schools to act as pilot schools,some 10 % of all schools. At the end of thereform programme the country was faced withthe challenge of generalisation of results. Thegovernment then decided that the new curriculawere obligatory in all schools, without any newequipment or any proper training. The nationalinspectorate, whose actors were themselves onlyhalf-familiar with the new curriculum principles,was supposed to support this vast implementa-tion effort.

The decision to generalise the pilot schoolcurricula indicated an intention to take the PhareVET reform methodologies from the experimentalto the systemic level. However, it is difficult to seehow this could be achieved without a consider-able investment in equipment and training of bothteachers and school managers, in short strength-ening material as well as human resources. Pilotschools and teachers are not in a position togenerate their own resources for this huge effort.The necessary critical mass of change agentsmay be in place to consolidate what has beenachieved. The teachers and curriculum develop-ment officers trained under the Phare programmecould have played a crucial role in training thestaff of the majority of vocational schools notinvolved in the Phare programme. However, abroad nation-wide implementation effort requirescareful planning and substantial resources.

In all types of pilot projects there is a risk ofencapsulation. As a rule, the best institutions areselected and, during the piloting phase, they aregranted good development conditions, i.e. they arefreed from the difficulties that exist in the ordinarystructure. When it comes to the transfer and broadsystemic implementation of pilot results, all the realworld problems and barriers suddenly turn upagain. The risk of barriers in the ordinary structure isalways a serious factor to be reckoned with whengoing from the phase of ‘VET reform (pilot) project’to the phase of ‘national VET reform strategy’.

One other impeding factor in the model schoolapproach is the fact that a number of elitistschools are given all the equipment, all thecoaching, all the study tours, all the developmentassignments, etc., leaving the initially backwardschools another couple of years behind. It is notclear how Romania (or any other country) willcope with this problem.

Another threat to the wider dissemination ofcurricular reforms, as initiated under the Phareprogramme, is the relatively low level of teacherqualifications. A major national in-service teachertraining programme is required to make VETreforms happen at school level. Modularisation, asthe basic pedagogical principle, implies a radicalchange of working conditions for Romanian voca-tional school teachers. In many ways, the teachingof modules could be seen as reduced qualificationfrom the point of view of the professional teacher.So what is needed is not only technical up-skilling

but also awareness-promoting initiatives on abroad scale.

The reform of VET systems is more than a singleact establishing a new legal framework. A changein the logic of the system can only be realisedgradually by those who are involved in VET them-selves, especially at the local level and in theschools. Since there are many people involved, atremendous learning process is necessary, whichneeds to be facilitated and intensified throughproper intervention and guidance.

After a long period of central regulation andbureaucratic administration, it is particularlyimportant to depart from the traditional top-downconception of VET reforms and promoteself-responsibility and self-initiative for localactors. But such bottom-up strategies, that givethe initiative for educational change to the indi-vidual schools, may easily lead to fragmentationand competition between schools, as a conse-quence of the pilot-school based reform strategy.

Romania shares this challenge with othercentral and east European countries. It will benecessary to integrate the various changes intro-duced in a more coherent change strategy. Froma strategic point of view, one should reflect onceagain on the interrelationships between thevarious aspects and issues and ask the funda-mental question: where should one start in orderto trigger a development process that will lead tothe desired outcomes? In particular, it is impor-tant to explore whether the applied strategy ofbottom-up curriculum-modernisation-through-pilot-schools, which has been forced upon thecountry by the EU Phare programme, has had theintended impact. And how, given the experienceso far, is it possible to transcend from the level ofpiloting to the level of systemic reform. A funda-mental discussion about the role of the vocationalteacher and trainer may now well be of a morestrategic value than the continuation of curricularreform, which has dominated VET reform so far.

We conclude that the challenge of nationaldissemination has caused tensions between theobjective of flexibility in VET on the one hand andconsistency on the other. This also raises thequestion of different approaches to regulatorymechanisms with a view to ensuring internal andexternal consistency.

4.5. Example of stressing systemconnections in a VET policyevaluation for Croatia (7)

While the examples given in Section 4.4. indicatea lack of consistency in one form or another, afurther example shall show how emphasis hasbeen placed on system connections in a VETpolicy evaluation for Croatia. The original assign-ment given to the ETF was to analyse the trainingof VET teachers and trainers in Croatia and, onthis basis, to come up with proposals for anEU-funded development programme.

Basic problems identified in Croatia include: (a) the non-existence of a genuine and holistic

pre-service VET teacher training system(would-be VET teachers read a technicalsubject plus general pedagogy; there are noin-company training practices);

(b) the in-school training year of would-beteachers is not well supervised;

(c) the central Institute for Educational Develop-ment offers, through a centrally publishedcatalogue, a list of teacher training coursesestablished centrally on the basis ofperceived needs for teacher training andavailable capacities;

(d) in-service teacher and school headmastertraining schemes are highly limited in general;training is neither linked to school develop-ment nor to teacher performance evaluationor career progression.

Based on the fact that changes to one part ofthe system have an impact on other parts of thesystem, the challenge for us was to design areform process that builds upon the internal linksand develops both the framework structures andthe actors at the same time. We highlighted thefollowing system connections.

4.5.1. Decentralisation and schoolheadmasters’ and teachers’ training

4.5.1.1. GovernanceSchool headmasters, administrators and teacherswill have to share the burden for a number of neweducational management issues. This implies aradical change, as schools must be managed in a


(7) Viertel, 2003. The text draws on Nielsen, 2002.


different way. Apart from pedagogical innovation,schools will have to establish a strategic planningand budget execution capacity. Both schoolheadmasters and selected teachers will beinvolved at the strategic level of the new schoolorganisation. This calls for training to develop thenecessary skills.

4.5.1.2. FinanceMoney will (partly) be transferred in the form ofblock grants, and schools will have to strengthentheir capacity for planning and economising inthe context of budgetary restrictions. They mayalso have to look for new income sources and willhave to form local partnerships with parents,education authorities, employment services andemployers. This will have immediate conse-quences for school headmasters, administratorsand teachers alike: all of them will have to learnhow to act as entrepreneurs.

4.5.1.3. Curriculum (content) A modern VET system is based on a frameworkrather than detailed regulations, to give somefreedom of action to the institutions providing theeducation and training services. Curriculumdevelopment and delivery will, therefore, bedecentralised to some extent. At the central level,both clear objectives in terms of learning achieve-ments (competences) and frameworks forcontent will be formulated, while schools willhave to determine, to a higher degree, both themore specific content elements and how learningobjectives are to be achieved (learning methods).As emphasised also in both the White Paper andthe Ministry’s concept paper, modern learningtheory calls for a shift from teaching to learningand from a teacher to a learner-centred focus. Inthis context, headmasters will play a key role inpromoting new pedagogical approaches in theirschools, and VET teachers and trainers have tobe equipped with the necessary skills to masterthese new assignments.

We are aware that the needs for training inrelation to the change processes are enormous,and that training should start as soon as possible.Without such training and school developmentprocesses being initiated on a massive scale,curriculum change and VET modernisation willnot be accomplished. As this is also anever-ending process, resources have to be freedup in the budget for this task.

4.5.2. Curriculum reform and teacher training A decentralised VET system ascribes new rolesand responsibilities to all involved. Newcurriculum principles will involve all teachers at avocational school. Curriculum-led VET reform hasconsiderable implications, including the following.

4.5.2.1. Design and development of new curriculaWithin given objectives and frameworks definedat national level, teachers will be actively involvedin curriculum development: they will have totranslate overall learning goals into specificsubject matter, and determine the actual teachingplans and the methods of teaching/learning. Inaddition, selected teachers will also have greaterresponsibilities in other fields as outlined above.

4.5.2.2. New equipment New curricula imply the use of new or updatedlearning (pedagogical) equipment. The modernisa-tion of school workshops calls, first, for teachersand instructors to master the new machines them-selves and, second, for the ability to make goodpedagogical use of it, i.e. to create meaningfullearning opportunities for students. An intensivetraining in vocational didactics is necessary. The(practice-related, partly in-company) programmefor the training of teachers and trainers within thedual VET system, which was implemented by theChamber of Crafts and Trades in collaboration withwestern partners, may serve as a good example inthis respect. Although we are aware that it isimpossible to send all teachers and trainersabroad for a certain period, learning on the spotfrom the best vocational schools or companies isprobably the most effective way to learn about thenew, often tacit knowledge and skills required.

4.5.2.3. Curriculum deliveryNew student-activating methods must be intro-duced, such as problem-solving organised asgroup work, project work, etc. The focus is ontraining teachers in developing the capacity toorganise new learning processes, to stimulate thelearning motivation of students and to establishbetter learning environments. This would have togo hand-in-hand with the development of newteaching material.

4.5.2.4. Teams of teachersThe work of teachers and instructors will follownew principles of organisation. The need to

develop key occupational competences on thebasis of a much better integration of generalsubjects, subjects of vocational theory andsubjects of vocational practice requires a muchcloser cooperation between all teachers andtrainers in a vocational school. We have witnessedan excellent illustration of this new approach in aprivate economics gymnasium, where varioussubject teachers work together to cluster anumber of formerly separate, but related subjectsinto broader subject fields. The new approachpresents a radical challenge vis-à-vis currentschool practices: the teacher becomes part of acollective planning body composed of severalteacher colleagues who altogether feel responsiblefor the holistic (more broadly occupation-orientedrather than narrowly subject-oriented) competencedevelopment of an individual student. To achievethis goal, a lot of school-based teacher and trainertraining and development will be required in thecoming years, as an inherent feature of truecurriculum reform. Again, the school headmasterhas to understand these processes to be able toprovide direction and leadership, orchestrating theefforts of the teacher teams.

4.5.2.5. Curriculum assessment and revisionIn a decentralised VET system leaving substantialfreeway to the schools, mechanisms will neverthe-less have to be introduced to assure quality to thelevel of agreed national standards and apurposeful and efficient use of public funds. As aresult of pressures from the financing authorities,but also a number of civil and economic actors towhich the school becomes increasingly account-able (students, parents, employers, etc.), voca-tional schools will have to reflect on the range oftheir programmes and the quality of their services.The best schools in Europe have developed agenuine evaluation culture as a shared preoccupa-tion by all school staff, which encourages feed-back from their external clients with a view toimproving learning processes and outcomescontinuously. Such an evaluation culture can onlybe established on the basis of mutual trust andmust reflect the values of the school, which canvary considerably from school to school. Oncemore, school headmasters would have to play apivotal role in establishing such a culture.

As not all the necessary changes in the Croa-tian VET system could be fully implementedimmediately, a careful choice had to be made ofelements that are critical to reform. Our choice of

priorities was based on the following criteria:instead of a series of separate interventions in thefields of pre-service and in-service VET teachertraining, we recommended trying out a coherent,systemic approach based on partnershipsbetween schools, universities, teacher traininginstitutions and companies.

Furthermore, our approach focused on the prin-ciples of learning on the job or action learning,meaning that learning should be as closely relatedto the changing day-to-day work processes aspossible. This way, theoretical knowledge would becombined with experience-based learning as partof the change process and become applied knowl-edge or action competence (Handlungskompetenz).Rather than taking teachers out on external courses(as is the current practice in Croatia), learningoutcomes are optimised when teachers and schoolheadmasters learn through first-hand experience,being supervised by teacher-mentors in their orig-inal work environments. Having gone through sucha process themselves, teachers are more likely topractice the same new style of learning with theirstudents in day-to-day teaching practice after-wards. Moreover, involving all school staff in acollective learning process, rather than trainingschool headmasters and teachers individually,seemed the most promising way forward towardsschool development. Headmasters and teachersneed to share the same vision and see a role forthemselves in a future reformed system. Alsoteacher trade unions needed to be taken aboard.

The activities which were finally recommendedfor inclusion in the EU-funded developmentprogramme were, amongst others: (a) the development of change agent team

strategy and the training of a sufficientnumber of change agents as a way to intro-duce new VET reform concepts and trainteachers at school;

(b) the development of the mentor as one of theprofiles of education specialists, the selectionof excellent teachers specialising in certainfields and their training as mentors;

(c) the design of a training programme for schoolheadmasters which should be owned andmultiplied later on by the School for SchoolHeadmasters in Croatia;

(d) the design of new methods for analysingtraining needs, the pilot-testing of trainingcourses delivered in VET schools and theimplementation of a system to monitor the



relevance and effectiveness of training before,during and after the courses;

(e) the implementation of a number of schooldevelopment projects as the principalapproach to training;

(f) the identification of a central teacher traininginstitution that would accompany the devel-opment process, and own and disseminatefurther all innovations introduced into thesystem.

Apart from the systemic approach, the evaluatorneeds to develop an understanding of howsystems change in the interactions between actors,that innovations need to be taken forward not onlyto implementation but also to institutionalisation,and what are possible strategic levers for change.

5.1. The complexity of systemicreforms

To illustrate the complexity of systemic reforms inVET and the role of evaluations in them, let usfirst have a closer look at how they are defined.While, currently, there is no agreed definition ofsystemic reforms, Grootings (1993) describesthem in the context of the following post-socialiststages of reform in transition countries (the firststage was added later by Bîrzea, 1997):(a) corrective reforms that are initiated with

immediate repairing objectives in countriesemerging from a war or deep economicregression period;

(b) modernising reforms that are interventionsaimed at reducing gaps and catching up withwestern institutions. They are especially activeat the level of curricula, teaching and learningmethods, examinations, and school textbooks;

(c) structural reforms that are targeted at thestructures, legal framework and managementof educational systems;

(d) systemic reforms that are deeper and have aglobal character because they call for agenuine change of paradigm in terms ofeducational policy. They are aimed not only atthe curricula or the legislative framework, butat the internal logic of education and its rela-tionships with the global social system. Asystemic reform examines the key elementsof every educational policy: the role of thestate, relations with the labour market, thefinancing system, efficiency control, thenormative role of national standards, etc.

A study by the US Department of Education(1995) offers the following definitions for systemicreforms in education:

(a) ‘systemic reform addresses all of the mutuallyreinforcing structures, processes, and activi-ties within the educational system, recog-nising that altering any one part of the systemnecessarily impacts on all other parts;

(b) systemic reform requires system coherencethrough the integration of policy and practice;

(c) systemic reform constitutes a mainstreamactivity of all organisations involved, not analternative or special programme;

(d) systemic reform requires strategies that helpdevelop and mobilise the conceptions, skills,and motivation in the minds and hearts ofscores of educators;

(e) systemic reform requires the development ofroutine mechanisms for bringing peopletogether across roles, within and acrossorganisations, for developing and maintainingshared direction and understanding; and tomaintain strong communication among all ofthe constituent parts of the system;

(f) systemic reform in education addresses thepreparation, continuing learning and workingconditions of […] educators in all roles –teachers, principals, counsellors, specialists,paraprofessionals, central office and highereducation personnel.’

What is interesting to note in the above defini-tions is that: (a) the professionalisation and further develop-

ment, in a structured and continuous way, ofteachers and other education specialists areconsidered key to the success of anysystemic reform effort;

(b) higher education plays a central role in imple-menting reforms of the VET system. Much willtherefore depend on new school-universitypartnerships.

In other words, changing the VET system – asany social system – is a complex, multi-layeredprocess, comprising all system levels: (a) the primary learning process; (b) the organisational level;(c) the institutional frame;(d) the policy and legal frame.

5. System change and the process and actors in VETpolicy evaluations


5.2. The path dependency ofinstitutional change

The organisational settings and arrangements,and related social values and behaviouralpatterns, have grown historically as the result oflong, non-linear social development processes.According to Grootings (2002a), ‘institutions resultfrom social relations among people and implytherefore power and trust. They are embedded inhistorical contexts and part of wider institutionalframeworks in which different institutions arerelated to each other’. Taking the example ofcurriculum reforms, Parkes et al. (1999) illustratetheir institutional embeddedness as follows:

‘Curriculum is perceived as a balanced relationbetween the potential and interests of individualsand the requirements of society. From the indi-vidual point of view “it” can be seen as the totalityof measures, interactions and experiences withinan organised learning process. However, individ-uals can make up their curricula only withinpatterns laid down by organised bodies and socialinstitutions within society. These institutionalframeworks are anchored in the specific socialsystem of a society. They offer opportunities tothe individual who in turn is constrained byrequirements and standards to become a“responsible” member of society, in whatever waythis is defined. The educational field has its ownbodies and institutions to develop a framework torespond to the temporary and long-term needs ofsociety, as well as the wishes of the individual topromote his/her personal development. … Themain actions, actors and institutions involved inthe development of a curriculum are part of “insti-tution-building” involving developing “institutionalhomes” for key functions and activities.’

Grootings (2002b) argues that there is an insti-tutional logic that is country-dependent, andreform through institutional change is determinedby the institutional context of the country. Theyrepresent what was possible to achieve at acertain point in time rather than being a perfectmaking. They are unique for each country, whichexplains the uniqueness of reform designs foreach country. Similarly, current OECD economicthinking (the development of social capital)suggests that reform, and economic develop-ment, is best undertaken through the mutation ofhistoric institutional structures.

5.3. Accomplishing change

It is also important for the evaluator to under-stand the ways in which understanding andsupport by the actors necessary for implementa-tion can be gained, as a lack of this has often ledto failures of reform projects. Reforms will have tobe carried out by existing staff, and despitewidespread agreement with the global policyobjectives and the recommended improved poli-cies, there may still be great discomfort felt bythe reform actors at the prospect of changingtraditional ways. Accomplishing change is aboutreversing deeply embedded policies and stronglyheld beliefs. It is about preventing resistance tochange or attenuating its impact.

The prevailing political culture will determinethe accentuation of approaches to reform – withtop-down and bottom-up reform approaches atthe two ends of the spectrum. Again, there is nowrong or right in favour of the one or otherapproach. Our experience tells that contrastingorganisational forms are required for interventionsat the central level and the level of organisations,respectively. A high degree of bureaucracy islikely to make structural reform easier, whilenon-bureaucratic reforms are needed for curric-ular innovation and learning.

While, in transition countries, system changewas traditionally designed by the centre anddecreed from the top, the political culture is nowchanging. Local agents claim a stake in thedesign of reform concepts and, as we have seen,resistance may build up at an early stage whenGovernments, or foreign consultants, presentready-made concepts rather than involving localchange agents in reform design from the veryoutset.

According to Paquet (2001), the defining char-acteristics of democratic change includebottom-up governance schemes and a new formof transversal coordination, while collaborationbecomes the new categorical imperative. Thecentre will focus on norms and the periphery ondelivery; for the two to function together, a clearsense of public purpose is needed, as well asnew partnerships and skills. The agencies needto be granted the necessary powers to organiseactivities and become negotiating arenas,providing space for interaction, the (re-)definitionof activities, as well as monitoring. A shift is

taking place in the policy management of publicservices: the stress is no longer on inputs but onoutputs. The government defines objectives andsets frameworks while allowing a flexible style ofdecentralised management that is capable ofrapidly adapting to new demands. This in turncalls for a decentralised structure, and for newforms of horizontal accountability, for the systemof governance to be effective.

Such a process of cultural transformation andthe translation of policy into practice is almostalways an extremely lengthy process. McLeishargues in her introduction to Process of transitionin education systems (McLeish and Phillips, 1998)that the completion of the transition process atthe structural-legislative level in no way impliesthat educational transition at the micro level hasbeen achieved. To change a label is easy, toeffect a comprehensive change in practice is verydifficult. Peter Drucker (1999), whose speciality,since the 1950s, has been to distil currentthinking and research for the practical manager atthe level of the organisation, makes the followingremarks in his last book: ‘There is an enormousamount of work to be done in organisation theoryand organisation. There are only organisationswhich have distinct strengths, limitations andspecific applications. A given organisation fitscertain tasks in certain conditions and at certaintimes [...] One cannot manage change; one canonly be ahead of it [...] Change is necessarily aseries of small incremental changes.’

We conclude that system change will have tobuild on the given historically grown institutionalstructures. It is likely to be achieved only throughsmall, incremental change in narrow and targetedareas and only where there is equilibriumbetween radical change and traditional forces.Change requires, according to Paquet (2001, seeabove) a clear sense of public purpose, new part-nerships and new skills, as well as careful policycoordination, compensatory mechanisms andcollaboration in adequate forums for consultationand decision making.

5.4. The role of evaluations

Planning system change would be the art (ratherthan the science) of designing a multi-purpose,multi-actor, multi-stage process comprising

coherent actions across many interdependentareas. This process must be based on a soundunderstanding of the institutional logic of thesystem and a number of hypotheses about thereasons for lack of internal and/or externalconsistency and how they could be overcome.Due to the multitude of factors impeding asystem that is in a steady state of development,and the unpredictability of human behaviour,such planning models and their outcomes mustalways remain imprecise. OECD (2000a) arguesthat ‘the outcomes of educational processes aremulti-dimensional, underlying factors remainimperfectly defined.’

Being aware of the imperfection of any plan-ning model, it becomes essential to build into thechange process, strategic phases where socialactors come together and reflect and learn withthe ultimate aim of verifying or rectifyinghypotheses. This explains the need for evalua-tions not as one-off events, but as a continuouscycle.

Evaluations take on different roles in such achange process. They can be commissioned bythe (national or international) funders of reformprojects with a view to providing feedback onprogress and establishing whether they have mettheir targets. In this case, evaluations would besummative (evaluation of results/impacts); theywould serve as accountability tools and be largelylimited to the projects in question. However, asalready argued in the introductory chapter,project evaluations provide too limited a scopefor assessing the complex nature of changes inVET within their social/ institutional contexts. Thatis why also ‘summative’ evaluations need to becomplemented by formative elements. Whatformative literally means is having influence informing or developing.

Evaluations can be organised as a conscious,critical reflection process at various stages of thereform process. They involve the reform actors,arrange for learning and negotiating opportunitiesand help shape or reshape policies. As such, theyare part of the development process. They areconstructivistic in that they deny that there is anobjective knowledge of the world (the VETsystem) and in that they put actors at the centreof the evaluation rather than its outcomes.

Stern (2003) describes the following steps ofsuch an evaluation:



Consensus creates one priority. A second needis capacity building by means of a dialogue, insimple and transparent terms, with the key actorsso that it is possible for them to internalise andown the outcomes of evaluation. This will, subse-quently, allow them to have the understandingand political will to move to strategic implementa-tion both at macro and micro levels.

Inspiration and guidelines can be found in theso-called empowerment evaluation school in USuniversity evaluation research. But this approachis also well known in Europe. For example, theUK Tavistock Institute of Human Relations (TIHR)makes use of an approach described as ‘devel-opment evaluation’ (Stern, 1989). It involves anumber of elements: (a) it offers participants in the setting being eval-

uated a voice in shaping the evaluationagenda;

(b) it involves an active feedback policy;(c) it sets a clear focus on utilisation and imple-

mentation. The approach is based on clusters of shared

professional values, one of which is the commit-ment to action research.

Nyhan (1998) outlines the lack of a researchtradition in VET (as opposed to general educa-tion). He reviews the approach of Kuhn (1996) inrooting rationality in particular political andcultural contexts, and Carr and Kemmis (1986) inaccepting that the social actors involved in trans-formation need to be engaged in a kind of actionresearch. In doing so, he points the Leonardo daVinci Programme firmly in the direction of actionresearch: ‘how can we change things at presentto ensure a better future?’

Gisela Shaw (1999) further elaborates theNyhan position: ‘This type of research (actionresearch) is based on two fundamental assump-

tions. The first is that social research, includingresearch in VET, is ill-advised, if it tries to copythe positivistic model of the natural sciences,relying primarily on quantitative and empiricalmethods and aiming at research results ofuniversal validity allowing scientific predictions;instead, social research ought to accept thathuman actions and behaviour can only be inter-preted and illuminated and that such interpreta-tion and illumination requires taking into consid-eration the social context in which they occur(Hans-Georg Gadamer’s hermeneutic/interpreta-tive approach; Gadamer, 1972). The second goesbeyond this position by aligning itself with a claimembedded in Jürgen Habermas’ critical theory:human science being rooted in the emancipatoryhuman knowledge interest (rather than in thetechnical or practical knowledge interest) cannotstop short at interpreting and illuminating humanactions and behaviour but ultimately aims toequip individuals with an understanding of howbest to pursue their objectives in a rational way(Habermas, 1972). Although acknowledging theproblem inherent in the notion of action research,i.e. a deliberate blurring of the boundary sepa-rating the researcher from the practitioner, Nyhannevertheless highlights the fruitful implementationof action research in educational and trainingprogrammes in a number of countries [...]’.

The action research strategy offers channelsfor continuous dialogue with the relevant stake-holders (policy-makers, actors) about the validityof implementation of the concepts being studied,throughout the project: ‘An action-researchproject attempts to build concurrent dissemina-tion activities into the research process’ (Nyhan,1998, p. 28).

This means that the process of undertaking anevaluation – i.e. the way in which it is negotiatedand managed and how it impacts on the subjectsof an evaluation – is itself seen as a legitimateobject of study. It follows that there is a thin linebetween evaluations, or action research, andquite specific consultant activity.

It follows also that the experience of the evalu-ators is key (as the OECD also suggests – seeSection 2.1). Long apprenticeships seem to benecessary, whereby the junior evaluator would tryto learn from the senior evaluator, thus goingthrough the Dreifuss & Dreifuss five-stageprocess from beginner to expert.

Steps of a policy evaluation process

(a) identification of stakeholders;

(b) holding a series of initial discussions;

(c) feedback;

(d) identification of areas of agreement or disagreement;

(e) use of other sources which could inform stakeholders;

(f) reaching the best possible consensus.

6.1. Evaluation criteria

In the introduction to our paper we havementioned evaluation criteria, such as relevance,effectiveness, efficiency, impact and sustain-ability, which are important to project evaluations.It is clear that, for the purpose of policy evalua-tions, we would have to alter or redefine the setof evaluation criteria.

The OECD (Windham, 2000), for example, hasused the following evaluation criteria for reviewsof national education policies/systems in south-east European countries:(a) appropriate political foundation and support

for the reform process;(b) equity in access, attainment and achievement;(c) flexibility in planning and implementation;(d) support for an interactive planning model

involving cooperation among national, local,and institutional levels;

(e) incorporation of intersectoral and intrasec-toral coordination;

(f) development of appropriate regional, nationaland international emphases;

(g) affordability;(h) sustainability;(i) efficiency;(j) encouragement of supplementary resource

mobilisation.To evaluate reform policies meant to aid

systemic change, as defined in Section 5.1, thefollowing criteria may be useful:(a) the political foundation and support for the

reform process; (b) the visions guiding the reforms; (c) the leadership driving them; (d) the structures and resources for planning and

implementation, including the knowledge andresearch foundation upon which they arebuilt;

(e) the opportunities for learning needed tosustain them;

(f) the mechanisms for communication used forintrasectoral and intersectoral coordination;

(g) the organisational arrangements designed tosupport them;

(h) the strategies used to implement them;(i) the structures and mechanisms used to

monitor and evaluate pilot implementation,and take corrective action, where necessary;

(j) the affordability/economic viability of inter-ventions, as well as the structures and mech-anisms used with a view to including innova-tive results in mainstream provision.

However, as illustrated above, a universalmanual is difficult to assemble; a list woulddepend on the scope and purpose of the evalua-tion exercise. We suggest that, following thedevelopment evaluation approach, these evalua-tion criteria were subject to negotiations with theactors concerned prior to the start of the evalua-tion exercise.

6.2. Similarities and differencesin VET reforms in westernEuropean countries

The same applies to the question of underlyingthemes and values. Evaluators often come withpreconceived notions: VET policies in transitioncountries tend to be evaluated against westernstandards and practices. Can we, indeed, identifysome common policy priorities (and indicators)which, sooner or later, lead to some convergenceof systems and which could, hence, facilitate VETpolicy evaluations in transition countries?

Over the past decade, the VET systems ofmany west European countries have come underreform pressures as a result of persistent or newproblems, such as:(a) the academic drift;(b) the poor motivation of students on VET

programmes; (c) the declining investment of time and money

of employers in VET (even in countries withstrong traditions of employer involvement,such as Denmark, Germany and Austria);

(d) global competitive pressures which areputting strains on the traditional social part-nership approaches;

6. Criteria, values and indicators underpinning VET policy evaluations

(e) vocational teacher training (polarised betweenGerman university studies obligatory for allteachers and the zero solution in the UK);

(f) the lack of parity of esteem between generaleducation and VET;

(g) the failure to provide future employees withnew types of knowledge and skills which theywill need in a fast changing global economy;

(h) the problem of organising learning environ-ments and learning processes which stimu-late context-free learning of generic skills; etc.

Lasonen and Manning (2001) have shown, inan interesting typology, that west European coun-tries have actually applied very different strate-gies to address these problems. They aresummarised as follows.

6.2.1. Improving progression from initial VETto higher education

There are two distinct approaches to improvingprogression from initial VET to higher education;the access approach and the diversificationapproach. Denmark, Austria and Finland, on theone hand, have emphasised the diversification ofhigher education and the upgrading of existingvocational higher education by creating a new tierof vocational higher education institutions, whereprogrammes have been extended in length. Manycountries, such as Denmark, Finland and Norwayhave combined the diversification of highereducation with creating opportunities for studentson initial vocational courses to progress to highereducation by obtaining dual qualifications.

England and Scotland, on the other hand, haveconcentrated on improving access to a singleunified university system. However, a clear hier-archy remains between the old or traditionaluniversities, which recruit almost entirely fromstudents with academic qualifications, and thenew universities which are more open to studentswith vocational qualifications. Local provision offurther education colleges, often in partnershipwith universities has been expanded.

Most countries want the proportion of studentsin higher education to be expanded to at least50 % of each cohort. In unified stratified systems,as found in England and Scotland, nearly allhigher education institutions have equivalentstatus as universities. In divided systems, such asthose emerging in Denmark, Austria and Finland,and already found in Germany, some higher

education institutions are specifically designed tohave a vocational role and not be universities.

The general conclusion is that the question ofparity of esteem of VET needs to focus onpost-compulsory education as a whole and notjust the provision for 16-19 year olds. As thenumbers opting for general education at uppersecondary level continue to rise, the parity ofesteem issue shifts towards higher education.

6.2.2. Improving links between vocationaleducation and employers

Major differences between the countries include: (a) those with a dual system of apprenticeship

based on social partnership (Denmark,Germany and Austria) in which employershave a direct role in decision making aboutVET, which is based initially on their recruit-ment of apprentices;

(b) the state-led systems of the Nordic countriesin which governments have actively involvedemployers in VET curriculum issues thoughconsultative committees at local and nationallevels;

(c) the more voluntarist systems, such asEngland and Scotland and, to a lesser extent,Spain, where governments have attempted toinvolve employers through work-basedtraining schemes, and through employers’membership in governing bodies of schoolsor colleges and regional training councils.

Voluntarist systems rely on creating the condi-tions for new types of learning partnershipsbetween employers and educational institutions.The problem is how to increase the involvementof small and medium enterprises with little sparestaff capacity for supporting partnerships.

A variety of ways for ensuring greater employerinvolvement in VET is being explored. In Finland,France, Norway and Spain a greater regionalemphasis is allowing individual institutions todevelop closer links with local employers. Workexperience inherent in curricula is seen, espe-cially in Finland, France and Spain, as an impor-tant driver for establishing new links withemployers and enhancing the future employabilityof young people.

6.2.3. Improving the status of VET teachersand trainers

Improving VET is intrinsically linked with thequality of teachers. Various patterns are emerging


in European countries. First, a much greateremphasis is being placed on improving thequality of vocational teachers rather than work-place instructors; only in countries with a dualsystem are serious steps being taken to upgradethe qualifications of trainers. In England andScotland, the focus on workplace trainers islimited to their (students’ performance) assess-ment role. Second, general subject teachers invocational schools are typically better qualifiedthan vocational subject teachers (the dualsystem-based countries are, again, exceptionshere). Most countries are trying to standardisequalifications and limit traditional regional differ-ences (as in Spain).

Finland is the only country with a statutoryin-service training requirement (five days a year).Most progress in terms of structural parity hasbeen achieved in the three countries with a dualsystem (Denmark, Germany and Austria) andmore recently in Norway, where the qualificationsfor VET and general subject teachers are equiva-lent and programmes are shared.

In England and Scotland, staff developmentand initial training of vocational teachers is left toindividual institutions. Despite an increase in thenumbers of qualified VET teachers, formal qualifi-cations are still not a requirement for vocationalsubject teachers in colleges, though they do haveto pass a test if they want to assess studentperformance rather than just teach certain typesof vocational courses. Steps are being taken toestablish national standards. One important stepwas not to introduce fees for the training of(would-be) vocational (or further) educationteachers, placing them in an equal position withgeneral subject teachers at primary andsecondary level.

6.2.4. Improving the vocational curriculumSchool-based systems, such as those in Franceand the Nordic countries, have increased thegeneral education component of vocationalcourses, as well as the choices available tostudents. In Finland, this increase is expressedthrough the concept of integrated learning,whereby:(a) schools can decide to increase the general

education content from one fifth to one third; (b) students can compose their personal study

plans; (c) all programmes include work placements;

(d) the vocational content has been strengthenedby integrating it with activities from workinglife.

Mixed systems which are becoming morelinked or unified, such as in England and Scot-land, give priority to generic skills such asnumeracy, literacy and team working, whichemployers claim are important, and to the mixingof general and vocational knowledge.

Systems with strong apprenticeship traditions,such as in Germany, try to enhance the voca-tional knowledge component of VET programmesthrough work process knowledge. Denmark isintroducing greater flexibility into the dual systemthough the overarching concept of pathways. TheDanish pathways approach stresses the supportof student choices through the development ofindividual course plans within a modularcurriculum and the need to complement voca-tional specialisation with broader-based studies.Vocational curricula in Spain seem to be devel-oping against the European trend with a reduc-tion in the amount of general education, shortercourses (more attractive to students andemployers) and, in line with other European coun-tries, more opportunities for local partnershipsbetween schools and employers.

6.2.5. SummaryWhat the above analysis illustrates is that manyof the west European countries have not tackledsome of the fundamental problems in VET them-selves; so there is no blueprint for a good model.EU experience has shown that neither thecommon factors pressing on national policyformation for education and training, nor thepriorities that most EU Member States share,have led to a uniform pattern of convergence(even if some common trans-European trendshave emerged). We are aware that the non-exis-tence of clear-cut policy priorities or indicatorsmay create a dilemma for the objective evaluator.A particular donor may impose them, but moreeasily on the evaluator than the specific govern-ment concerned. A possible way around thedilemma of preconceptions or nationally biasedunderstandings is to start an evaluation bypresenting the building blocks and formulatingideal practices against which existing practiceswould then be evaluated. Finally, the analysisshows that adopted strategies for VET systemrenewal or reform reflect the different educational


histories of each country rather than the prob-lems these strategies are designed to address.They cannot be detached from their institutionalcontexts, as we have also argued in Section 5.2.

This is confirmed by Young (2001) who anal-yses why some countries have so far remainedpartly immune from the pressures to developoutcome-based qualification frameworks of thekind found in the United Kingdom and otheranglophone countries. This is so despite the needto rethink the role of qualifications in light of globaleconomic changes and the related changes inskill and knowledge demands (Lasonen, 1996)(Lasonen and Young, 1998) (Young, 2001). Heshows that there are built-in contradictions withEU countries having quite divergent philosophiesand practices, making in particular a distinctionbetween an outcomes-based approach (English-speaking countries) and an institution or process-based approach (most notably those associatedwith the Germanic and Nordic traditions of educa-tion and training).

6.3. Potential and limits of thebenchmarking approach

Despite the diversity of systems at EU level,attempts have recently been made by individualcountries and at EU level to apply a bench-marking approach, first, to employment policyand subsequently to VET policy. What are thedefining characteristics of this tool, what areinitial results, and does benchmarking generallypoint the way forward in VET policy evaluations?

There are numerous definitions of bench-marking, the simplest being that benchmarking isthe process of learning by making comparisons.Although simple, it captures some of its essentialfeatures: namely, that it is deliberate learningprocess and that it involves comparison. Fromthe early days of benchmarking, companies haveused it as a way to compare their own processes,practices and performance with other similarcompanies with a view to making improvements.More specifically, benchmarking in its classicform is a process of studying, adopting andadapting successful processes and practices

from other organisations to improve processes,products, and services in one’s own organisation.

Since then benchmarking has become anumbrella concept used in modern management ofboth private and public organisations. Theincreasing use of benchmarking has also meantthat some of its basic features have, in some cases,been dropped, which makes it difficult to distin-guish it from comparison or comparative analysis.

To talk about benchmarking as a meaningfulactivity we need to view it as a formal and delib-erate process of comparison, aiming at detectingweakness and creating ideas for improvement.Benchmarking is normally distinguished frompurely analytical methods of comparison in thatspecific performance indicators must be devel-oped and units must be found which can beconsidered as best performers concerning thechosen indicators.

The comparison then aims at finding options tomake progress towards the position of the bestperformer (Grootings, 2000). Essential featuresinclude: (a) the need for improvement;(b) identification of a benchmark, i.e. agreeing on

a benchmark that can serve as a referencepoint for undertaking the comparisons;

(c) the translation of the analysis into concretesteps for improvement.

For several years the Danish Ministry ofFinance (1999) has undertaken benchmarkingexercises for various sectors to assess whereDenmark stands compared to other countrieswith respect to a number of issues that determineprosperity and welfare (8). A best practiceapproach was followed by choosing a fixedgroup of countries consisting of France,Germany, Great Britain, Japan, the Netherlands,Sweden and the US who are considered to beamong the world leaders in one or more relevantfields. Major areas of prosperity and welfare wereidentified, and each area compared, using anumber of indicators. A ranking between the twobest or the two worst scoring countries for eachindicator was judged as being above or belowaverage. In the area of education the indicatorsbelow were used, resulting in a ranking forDenmark which is shown on the left.


(8) Similar evaluations have been done for Finland: Benchmarking Finland – an evaluation of Finland’s competitive strengths andweaknesses and Sweden: Quality comparisons in education and the employment offices.

As the Danish example illustrates, bench-marking is used solely to compare educationalsystems on the basis of their results, whereas aqualitative analysis of the practices, processesand methods that have led to certain results isnot made. In the given example, no rationale wasoffered for why, for example, the reading skills ofnine-year-old students are poor. Also, it is takenfor granted that scoring high is good, while goodreading skills are no guarantee that they could bereadily applied to learning in everyday life. That iswhy the whole debate about which indicators to

choose for a benchmarking process is so impor-tant and needs to be considerably advanced (9).

To us it is doubtful whether the Danish bench-marking exercise was more than just an interna-tional comparison of selected educational indica-tors. It should be mentioned that the DanishMinistry of Finance clearly recognises that softereducational objectives, which are valued highly inthe Danish context, are not sufficiently coveredby the benchmarking exercise (10). However, theraking has been useful as a political exercise, asit triggered a national debate on education and


Figure 2: Denmark’s ranking – education sector

(9) The discussion on identification of relevant benchmarks is not new and not only linked to educational issues. The quote belowpoints to reservations and illustrates how sensitive to the whole benchmarking exercise the identification of relevant featuresand benchmarks is. ‘Companies often waste time benchmarking non-critical functions or struggling to raise the performancebar an insignificant amount. Does it really matter to your customers if you shrink your delivery time by two days? If you deliverpackages, yes. If you deliver antiques, maybe not. And how much does it cost to cut out those days?’ (Dahle, 1996).

(10) ‘In addition there are other important objectives. At primary and lower secondary school level one objective is to have pupilsdevelop their creativity and self-confidence, as well as interpersonal and communication skills, and for the school to preparepupils for life in a democratic society. These soft values need to be measured using new methods that are still in the develop-mental stage, though as yet there is still no sound body of data on which to make international comparisons. Indicators for thesofter values are not included below, therefore’. Strukturovervågning – International benchmarking af Danmark, Finansminis-teriet May 1999, section 14.

• Number of hours of teaching per teacher in basic vocational education

Above average • Pupil/teacher ratio in primary and secondary education

• Small proportion of young people who have been unemployed for at least one year

• Total expenditure per pupil in the educational system

• Annual number of hours of teaching in primary and lower secondary school – 12-year-olds

• Proportion of population with upper secondary education

Average• Proportion of population holding tertiary education qualification

• Maths skills

• Natural science skills – upper secondary education

• Reading skills of 14-year-olds

• Relative unemployment for 20 to 24-year-olds

• Number of hours’ teaching per teacher in primary and secondary (general and vocational) education

Below average • Annual number of hours’ teaching in primary and lower secondary schools – 14-year-olds

• Natural science skills after 8 years’ schooling

• Reading skills of nine-year-olds

Source: Danish Ministry of Finance, 1999.

helped identify areas for improvement or reori-enting the focus of reforms. The benchmarkingexercise was not used, therefore, to learn directlyfrom others, but to stimulate a national debate byletting the figures speak for themselves.

The example points to the complexity ofapplying benchmarking to education systems.The approach presupposes that it would bepossible to disaggregate a VET system intosmaller units and that a benchmark could besingled out that could then be adopted by theinstitutional frameworks of other VET systems.However, as we have seen before, the highlycontext-linked nature of VET systems presents amajor barrier for the benchmarking approach. Thequestion is whether the benchmarking approachcan be applied to social systems at all.

In a study for the European Commission on thepossibility of benchmarking for employment poli-cies (Schütz et al., 1998), a team of Germanresearchers makes a distinction between bench-marking as an analytical inventory and as a policytool for improvement. The first mainly implies achoice of benchmark areas for analysis andpolicy recommendations; the second is of ahigher order and involves the understanding ofperformance gaps and the implementation, moni-toring and evaluation of interventions. Bench-marking in the second sense has to be based onbenchmarking as an analytical inventory.

The authors argue that, in principle, bench-marking for (employment) policies is possible, if anumber of conditions are met:(a) a thorough understanding of the bench-

marking approach;(b) a wide-ranging database and multi-method

approaches;(c) adequate resources (time, money, personnel);(d) commitment of involved key actors to organi-

sational learning.They conclude that benchmarking is easier for

organisational entities (such as employmentoffices) than various (employment) policies,whose success depend in part on other policiesand institutions.

Grubb and Ryan (1999) see international orsystem level benchmarking as valuable butweakly experimental and consistent with fairly

weak research methods. Their comment is: ‘It ismore difficult to evaluate systems with theirmulti-dimensional goals and multiple intertwinedcharacteristics, than it is to evaluate specificpublic programmes with typically unidimensionalgoals and simple attributes. It is also the casethat no social experiment involving apprentice-ship has ever been conducted. Consequently,while quasi-experimental evidence remains valu-able, weakly experimental methods (notably inter-national comparisons) become more importantfor the evaluation of apprenticeship than for otherVET categories.’

The European Commission and the MemberStates’ attempts to develop benchmarking proce-dures, following the Amsterdam Treaty goal toharmonise VET policies, are still in a very earlystage. In employment policy, where the processis more advanced, the European Commission hasdefined benchmarking as: ‘a formalised processby which employment performances of differentcountries are compared with each other in orderto highlight the best performing ones, to setglobal targets for progress in the employmentsituation over the medium/long term and to iden-tify which policies have been most effective inraising the level of performance’ (EuropeanCommission, 1997).

The process implies:(a) setting verifiable objectives and deadlines,

both at European and national level;(b) agreeing on common performance and policy

indicators, based upon a country-comparablestatistical basis;

(c) undertaking peer reviews of major nationalemployment policies and programmes.

Experience with the benchmarking of VETsystems at EU level so far shows that the mainproblem is not so much the general acceptance,among Member States, of the approach, but thedevelopment of appropriate indicators andsecuring that data are available to measure these.However, there are also words of warning raisedagainst believing that strategies of the bestperformers can be easily copied under differentcontexts. We have been aware of this dangerduring our previous evaluation and policy advi-sory work.


7.1. Conclusions

There is evidence of a shift in international reviewpractice in VET from project-based to systemicapproaches. This shift is related to the growingawareness of the restricted impact of singleprojects and their sometimes adverse effects onthe system.

The aim of our study was to reach a betterunderstanding of the underlying concepts andmethods used for VET policy evaluations byexploring the contribution of research, as well asa number of examples from transition countries.

One of the central conclusions is that there isno holy grail in terms of conceptualisation ormethodology related to VET policy evaluations.The engineer’s toolbox is of limited use. Likewise,management approaches based on refinedanalytical frameworks often tend to obscurerather than illuminate VET, including reliance ontools derived from systems analysis. It appearsthat the only remedy is the evaluator’s broadunderstanding of the essential components ofVET, of their relationships, of the fundamentallogic between the system and its environmentand of strategic levers for change. This under-standing develops only through many years ofapprenticeship and first-hand experience of VETpolicy evaluations.

Using the hermeneutic approach, a particularsystem can be understood by alternatingbetween studying parts (system elements) andtotalities (the total system), as well as creatingcontinuous interactions, in our interpretationwork, between our preconception (Vor-Verstehen)and the new understandings worked out throughthe process. However, understanding alwaystakes place between two entities both placed in ahistorical and cultural context, and Gadamer(1972) insists that the interpreter cannot abstracthimself from personal history and culture. Whatthe evaluator must do is openly recognise andclearly articulate this tension.

According to Habermas (1972), human sciencecannot stop short at interpreting and illuminatinghuman actions and behaviour but, ultimately,

aims to equip individuals with an understandingof how best to pursue their objectives in arational way.

We explored the question of whether we canprovide a conceptual framework for grasping notonly the parts (building blocks) of VET, but also itshistoric roots and dynamic relationships and,thus, for managing the dialogue with key actorsfrom inside and outside the system. A number ofessential functions and processes of a successfulVET system and architectural elements weredescribed which are, initially, sufficiently simple toengage in the dialogue. However, for developinga better understanding of VET, the building-blocks approach has to be accompanied by adeeper insight into the systemic logic of the VETsystem.

A structural-historical approach (Strukturell-genetischer Zugang) could be used. Here, theanalysis would start with an overview of existingpractices, functions and structures, as theyappear to the (experienced) evaluator. Essentialstructures are then traced back to their origin(genesis), and an understanding/explanation ofthe historical context is reached in which thespecific phenomenon was born or established.

The phenomenological/hermeneutic method isprimarily used to achieve an understanding ratherthan for explanation, which is to state causesbehind the phenomenon observed. There is acontinuous interaction between understanding andexplanation; they feed each other. Understanding isthe point of departure for VET policy evaluations,and explanations are needed when we fail to under-stand. Useful explanatory frameworks include:(a) economic or labour market or sociological

laws and other forms of determination ofthe concrete conditions under which aphenomenon is active;

(b) functionalist explanations referring to thetotality in which the phenomenon to beexplained is placed and a description of theprecise function it serves;

(c) historical-cultural explanations behind theoccurrence of different institutions and prac-tices in contemporary VET systems;

7. Conclusions and needs for further research

(d) structural(ist) explanations of phenomenawhich can only be uncovered through an indepth analysis;

(e) system-analytical explanations unfolding howchanges in one component will have spill overeffects on other system components or onthe total system (including discussions oninternal and external consistency).

We conclude that, besides developing furtherthe building-blocks approach by including expla-nations, and in order to avoid hobbyhorse expla-nations, we have to cultivate, individually and in acommunity of VET evaluation practitioners, a newattitude towards argumentation, a more opendiscussion practice. This we term a discourse.

VET policy evaluation requires a systemicunderstanding of VET. But how can the VETsystem universe be delineated? What relation-ships exist between the units? Which matrix ofdominance patterns the interplay of units? Whatare the boundaries of the system? What relation-ship (metabolism) exists between the system andits environment? And what is the prime mover orstrategic lever for change?

VET as a subsystem can be seen as an opensystem in a steady state; it depends on self-regu-lating mechanisms to maintain its boundaries andits continued existence within these boundaries(meaning primarily boundaries which distinguishthe VET subsystem from other subsystems withthe same society). The steady state depends on abalance of inputs and outputs. The inputs aredemands and support: support makes the systemstrong enough to process demands and toproduce outputs in the form of qualifications. TheVET system sits in an environment – the totalsocial system – and there is a continuous feed-back into the VET subsystem, signalling if outputsproduce good or adverse effects in the environ-ment. A systemic approach to VET evaluationsfocuses on the analysis of relationships, ofcommunication channels, of responsiveness andadaptability, based on the fundamental under-standing that changes in one component lead tochanges in other components and in the systemas a whole. So, it is important to understand thespecific systemic logics of different VET systems.Hence, the interest in internal and external consis-tency, the preoccupation with systemic regulatoryframeworks, and the search for the prime mover orthe strategic lever related to VET system change.

Internal consistency indicates that improvingVET (including increasing their attractiveness forboth students and enterprises) can only be donethrough a system approach, encompassing allthe components of the education system. Thepolicy of a high degree of decentralisation andprivatisation, and the modularisation of curricula,were discussed as two examples bearing the riskof lacking internal consistency.

External consistency refers to the relationshipsbetween educational objectives and theeconomic and social context. There is a range ofpotential problems connected to that: theemployment system is not always able to articu-late its needs; there are tensions between thepedagogical logic of school systems and compe-tence needs of the employment system. Further-more, the drive to develop national qualificationsystems as a means to increase internal and indi-vidual learner consistency may conflict withderegulated labour markets and short-termrecruitment and training practices in companies.Finally, it is realised that education cannot byitself solve employment problems .

To balance system-inherent tensions, there is aneed for some form of regulatory framework thatwould allow flexibility and adaptability while satis-fying the need for high quality and social equality.Such a regulatory framework needs to guidereform efforts.

A systemic approach to VET evaluation, asdescribed above, is the way forward. The ultimateaim of all VET reform endeavours is to betterbalance tensions and ensure internal and externalconsistency of the system. But there is nouniform model that could be applied.

Apart from the systemic approach, the evaluatorneeds to develop an understanding of howsystems change in the interactions between actors,that innovations need to be taken forward not onlyto implementation but also to institutionalisation,and of the possible strategic levers for change.

Our experience with VET reforms in transitioncountries so far, as well as definitions of the termsystemic reform, demonstrate that two aspectsare key to the success of any systemic reformeffort in VET: (a) the professionalisation and further develop-

ment, in a structured and continuous way, ofteaching and other education staff;

(b) new school-university and local partnerships.


Change is a complex, multi-purpose, multi-layered, multi-actor, multi-stage process whoseoutcomes are partly unpredictable. System levelsaffected by each change comprise the primarylearning processes, the organisational level, theinstitutional frame, and the policy and legal frame.According to Grootings (2002b), reform isconstrained by the institutional context of thecountry, and there is an institutional logic that iscountry-dependent. Reform can only be under-taken through the mutation of historic institutionalstructures.

Another aspect that the evaluator must under-stand concerns suitable ways to gain under-standing and support from the actors necessaryfor implementation. Accomplishing change isabout reversing deeply embedded policies andstrongly held beliefs. It is about preventing resis-tance to change or attenuating its impact.

System change is likely to be achieved throughsmall, incremental change in narrow and targetedareas and only where there is equilibriumbetween radical change and traditional forces.Change requires, in a new political culture, a clearsense of public purpose, new partnerships andnew skills, and careful policy coordination,compensatory mechanisms and collaboration inadequate forums for consultation and joint deci-sion. The prevailing political culture will alsodetermine whether reforms are implementedthrough top-down or bottom-up approaches (or amixture thereof). Our experience has shown thatcontrasting organisational forms are required, i.e.a high degree of bureaucracy for structural reformand rather non-bureaucratic reforms for curricularinnovation and the learning process.

Owing to the multitude of factors impeding ona system that is in a steady state of developmentand the unpredictability in particular of humanbehaviour, planning system change must alwaysremain imprecise and its outcomes imperfectlydefined. As a consequence, it becomes essentialconsciously to build into the change processstrategic phases where social actors cometogether and reflect and learn with the ultimateaim of verifying or rectifying hypotheses as inputinto the next stage of the reform. This explainsthe need for evaluations not as one-off events,but as a continuous cycle.

Such evaluations are formative in nature, asthey have influence in forming or developing poli-

cies. They are constructivistic in that they denythat there is an objective knowledge about theworld (the VET system) and in that they put theactors at the centre of the evaluation rather thanits outcomes. Nyhan (1998) outlines the lack of aresearch tradition in VET (as opposed to generaleducation) and highlights the fruitful implementa-tion of action research in education and training,which engages the social actors involved intransformation. The action research strategy (hereused in its empowerment evaluation version)offers participants a voice in shaping the evalua-tion agenda, it involves an active feedback policy,and it sets a clear focus on utilisation and imple-mentation. Action research offers channels forcontinuous dialogue, with the relevant stake-holders, about the validity of implementation ofthe concepts being studied, throughout theproject.

This means that the process of undertaking anevaluation – i.e. the way in which it is negotiatedand managed and how it impacts on its subjects–is itself seen as a legitimate object of study. Italso follows that there is a thin line between eval-uation and consulting.

Evaluations call for substantial resources interms of staff, time and money. It is important toremember that, with continually evolving reformefforts, little time spent in the country and fewcases explored, evaluations represent only asnapshot and general conclusions must alwaysbe considered tentative.

Policy evaluations require different evaluationcriteria from project evaluation. These depend onthe scope and purpose of the evaluation andshould also be subject to negotiation with theactors involved.

There are no absolute standards for a goodVET system, which the evaluator could use.Concepts and practices of VET systems of westEuropean countries all have their positive andnegative aspects; they cannot be detached fromtheir institutional contexts. VET systems of westEuropean countries have come under reformpressures themselves, as many fundamentalproblems have not been tackled, and new chal-lenges have emerged. The adopted strategiesused by west European countries for VET systemrenewal or reform reflect the different educationalhistories of each country rather than the prob-lems these strategies are designed to address.


In these circumstances, a universal manual isdifficult to assemble. EU experience has shownthat neither the common factors pressing onnational policy formation for education andtraining, nor the priorities that most MemberStates share, are necessarily leading to a uniformpattern of convergence (even if some commontrans-European trends have emerged). Thenon-existence of clear-cut policy priorities or indi-cators may create a dilemma for the objectiveevaluator. A particular donor may impose them,but more easily on the evaluator than the specificgovernment concerned. A possible way aroundthe dilemma of preconceptions or nationallybiased understandings is to start an evaluation bypresenting the building blocks and formulatingideal practices against which actual practiceswould then be described and evaluated.

Benchmarking becomes a meaningful activity(or tool), when we view it as a formal and delib-erate process of comparison aimed at detectingweaknesses and creating ideas for improvement.Benchmarking is distinguished from purelyanalytical methods of comparison in that specificperformance indicators must be developed andunits must be found which can be considered asbest performers concerning the chosen indica-tors. The comparison then aims to find optionsto make progress towards the position of thebest performer. However, we have to take seri-ously the warning that strategies of the bestperformers cannot easily be copied underdifferent contexts.

A team of German researchers (Schütz et al.,1998) makes a distinction between benchmarkingas an analytical inventory and as a policy tool forimprovement. The first mainly implies a choice ofbenchmark areas for analysis and policy recom-mendations; the second is of a higher order andinvolves the understanding of performance gapsand the implementation, monitoring and evalua-tion of interventions. Benchmarking in the secondsense has to be based on benchmarking as ananalytical inventory. The latter is especially rele-vant for our purposes.

Essential conditions for using policy bench-marking include a thorough understanding of thebenchmarking approach; a wide-rangingdatabase and multi-method approaches;adequate resources (time, money, personnel);and commitment of involved key actors to organ-

isational learning. They conclude that bench-marking is easier for organisational entities thanfor policies, whose success depends on otherpolicies and institutions.

Member States have accepted the approach,but there are problems in developing appropriateindicators and secure that data are available tomeasure these. There are also words of warningagainst believing that strategies of the bestperformers can be easily copied under differentcontexts. We have been aware of this dangerduring our previous evaluation and policy advi-sory work.

7.2. Needs for further research

What is presented in this study are the firstresults of what has turned out to be ratherground-clearing work in VET policy evaluation.Thus, we see our study as a basis for discussionand further research work. Further analysis isneeded of the levers and critical factors forachieving genuine change, not only on paper butin the interactions between actors, which impliestaking innovations to institutionalisation.

There is a need to develop both formative eval-uation procedures and instruments to assistspecific systemic reform initiatives in evaluatingand monitoring progress towards goal achieve-ment, and to inform future planning. An evalua-tion model must take into account the creation ofnew organisational arrangements attempting tobring about simultaneous renewal in more thanone institution. Developing guidelines forlong-term documentation and evaluation, whichdefine needed data sources and indicators ofdevelopment and achievement, would serve thecontinuing planning and development of suchventures and assist in developing comparabledata to enhance our understanding of the limitsand potential of specific reform efforts, not leastthe cost-effectiveness of various strategies.Further research is also needed to examine thereal costs of education (or VET) reforms.

Given the importance of leadership in thesuccess of these endeavours, additional researchis needed to understand how leadership skillsdevelop, and the strategies for developing them.When a study finds leadership to be a criticalvariable, it presents a dilemma for policy-makers.


Can such leaders be selected and/or trained, ordo we have to wait for them to emerge? Moreresearch on the specific qualities of educationalentrepreneurship is needed, particularly with afocus on how individuals can be trained to leadcollaborative efforts.

Finally, more research is required into howbenchmarking could be used to improvesystems, which indicators should be selected forbenchmarking VET systems on an EU or interna-tional scale, and what can be considered as bestperformers concerning chosen indicators.


List of abbreviations

ETF European Training Foundation

NCEQ Centre for examination and qualifications

OECD Organisation for Economic Cooperation and Development.

PAS Pedagogical Academy in Sarajevo

TIHR Tavistock Institute of Human Relations

VET Vocational and educational training

Bengtsson, J. Labour markets of the future: thechallenge to education policy makers. Euro-pean Journal of Education, 1993, Vol. 28,Issue 2.

Bîrzea, C. The dilemmas of the reform of Romanianeducation: shock therapy, the infusion of inno-vation or cultural ‘decommunisation’? HigherEducation in Europe, 1997, Vol. 22, No 3,p. 321-328.

Boglia, L.; Parkes. Report on VET in Bosnia-Herze-govina. The World Bank/ETF – EuropeanTraining Foundation joint project, 1996[Unpublished draft].

Bryant, I. Action research and reflective practice. In:Scott, D.; Usher, R. (eds) Understanding educa-tional research. London: Routledge, 1996,p. 106-119.

Carr, W.; Kemmis, S. Becoming critical: educationalknowledge and action research. London:Falmer Press, 1986.

Challis, R. E.; Mason, C.; Parkes, D. YTS and thelocal authority. MSC Research and Develop-ment, Vol. 37, 1985.

Cort, P. Vocational education and training inDenmark: brief description. Luxembourg:Office for Official Publications of the EuropeanCommunities, 2002 (Cedefop Panorama,series, 49).

Dahle, C. Five uneasy pieces – Part 2: Bench-marking. CIO Magazine, 1 June 1996.

Danish Ministry of Finance. Strukturovervågning –International benchmarking af Danmark [Struc-tural monitoring – international benchmarking ofDenmark]. Copenhagen: Danish Ministry ofFinance, May 1999.

Davies, N. Europe: a history. Oxford: Oxford Univer-sity Press, 1996.

Descy, P.; Tessaring, M. (eds) Training and learningfor competence: second report on vocationaltraining research in Europe: executivesummary. Luxembourg: Office for Official Publi-cations of the European Communities, 2001.(Cedefop Reference Series, 11).

Dietzen, A. Fostering innovation by Europeanresearch in VET. In: Dietzen, A.; Kuhn, M. (eds)Building a European cooperative research tradi-

tion in vocational education and training. Berlin:BIBB – Bundesinstitut für Berufsbildung,1998b.

Dietzen, A; Sellin, B. The impact of survey and anal-ysis projects - towards a European dimensionof VET. In: Dietzen, A.; Kuhn, M. (eds) Building aEuropean cooperative research tradition invocational education and training. Berlin: BIBB– Bundesinstitut für Berufsbildung, 1998a.

Drucker, P. F. Management challenges for the21stcentury. NewYork: HarperCollins Publishers,1999.

Durand-Drouhin, M; Bertrand, O. Improving theresponsiveness of vocational education andtraining: flexibility and/or consistency? Paris:OECD, 1994 [Unpublished draft].

Dyankov, A. Current issues and trends in technicaland vocational education. Paris: Unesco, 1996.(Unevoc studies in technical and vocationaleducation series, No 8).

Elsenhans, H.; Nolke, A. International networks: thecase of the coordination of economic assis-tance to the CEEC and the NIS. Leipzig: Institutfuer Politikwissenschaft, 1996 (Working paper).

ETF. A cross-country analysis of curricular reform inVET in central and eastern Europe. Turin: ETF –European Training Foundation, 1999.

ETF. Baltic conference on VET: the importance ofEuropean cooperation, Tallinn, 22-23 April1996. Turin: ETF – European Training Founda-tion, 1996a.

ETF. Current constraints and challenges of the VETsystems in the Baltic States: the importance ofEuropean cooperation. Turin: ETF – EuropeanTraining Foundation, 1996b.

ETF. Transnational analysis of VET in the new inde-pendent States and Mongolia. Turin: ETF –European Training Foundation, 2000.

ETF. VET, a donor coordination framework for donorcoordination. Turin: ETF – European TrainingFoundation, 1997.

European Commission / SCR (The joint service for themanagement of Community aid to non-membercountries). Guidelines for: drawing up terms ofreference for evaluations, evaluation method-ology, criteria and suggested layout for evaluation

Bibliography and references

reports. Brussels: European Commission, 1999(SCR/F/5D(98)). Available from Internet:http://www.advd. unibe.ch/uls/evaluation/doku-mente/EU_Evaluation_Guidelines_021999.pdf [cited 9.6.2003].

European Commission, Directorate-General forEducation and Culture. European report on thequality of school education – Sixteen qualityindicators. Luxembourg: Office for OfficialPublications of the European Communities,2000.

European Commission, Directorate-General forEducation and Culture. A memorandum on life-long learning. Brussels: European Commission,2000 (SEC (2000) 1832). Available from Internet:http://europa.eu.int/comm/education/poli-cies/lll/life/memoen.pdf [cited 9.6.2003].

European Commission, Directorate-General forEmployment and Social Affairs. Commissiondraft for the joint employment report. Brussels:European Commission, 1997.

Fretwell, D. A framework for evaluating VET. Turin:ETF – European Training Foundation, 2000[Unpublished draft].

Gadamer, H. G. Truth and method. London: Sheedand Ward, 1981.

Gadamer, H. G. Wahrheit und Methode: Grundzügeeiner philosophischen Hermeneutik (3rd ed.).Tübingen: JCB Mohr (Paul Siebeck), 1972.

Gasskov, V. Managing vocational training systems:a handbook for senior administrators. Geneva:ILO, 2000.

Grootings, P. A comparative review of VET andlabour market developments in south easternEurope. Turin: ETF – European Training Foun-dation, 2001 (Working paper).

Grootings, P. A note on evaluation. Turin: ETF –European Training Foundation, 2000 [Unpub-lished draft].

Grootings, P. Advisory forum 2003 discussionpaper. Turin: ETF – European Training Founda-tion, 2002a (Internal paper).

Grootings, P. VET in transition: an overview ofchanges in three east European countries. Euro-pean Journal of Education, 1993, Vol. 28, No 2.

Grootings, P. VET reform as institutional change:looking at VET reform through the eyes of NewInstitutional Economics. Turin: ETF – EuropeanTraining Foundation, 2002b [Unpublished draft].

Group of 40 Key Actors: Green Paper: Strategy andpolicy for reform of VET in Bosnia-Herzegovina,

EU Phare VET programme, published locally,2000.

Grubb, N.; Ryan, P. The roles of evaluation for VET:plain talk on the field of dreams. Geneva:ILO – International Labour Organisation, 1999.

Habermas, J. Knowledge and human interest.London: Heinemann, 1972.

Hindle, T. What is a strategy? In: Economist Books– Pocket strategy. London: Penguin, 1994.

Hodgson, G. Socio-economic consequences of theadvance of complexity and knowledge. In:Future trends 7: an information base for scan-ning the future (CD-Rom). Paris: OECD,2001/02.

Howse, G. VET under review: the challenges incentral Asia. European journal of education,Vol. 36, No 1, 2001.

Jaworski, B.; Phillips, D. (eds) Comparing standardsinternationally: research and practice in mathe-matics and beyond. Oxford: Triangle Journals,1999, Vol. 9, No 1 (Oxford studies in compara-tive education series).

Kenneke, L. A guide for evaluation of technical andvocational education curricula. Bonn: Unesco/Unevoc, 1987 (Unevoc studies in technical andvocational education series, No 3).

King, K.; Buchert, L. Changing international aid toeducation: global patterns and nationalcontexts. Paris: Unesco /Norrag, 1999.

Köhler, W. Gestalt Psychology Today. AmericanPsychologist, 1959, Vol. 14, p. 727-734.

Kuhn, T. S. The structure of scientific revolutions.Chicago: University of Chicago Press, 1996(3rd ed.).

Lasonen, J. (ed.) Reforming upper secondaryeducation in Europe. Jyväskylä: Institute forEducational Research, 1996.

Lasonen, J.; Manning, S. How to improve thestanding of vocational compared to generaleducation: a collaborative investigation ofstrategies and qualifications across Europe. In:Descy, P.; Tessaring, M. (eds) Training in Europe:second report on vocational training research inEurope 2000: background report. Luxembourg:Office for Official Publications of the EuropeanCommunities, 2001, Vol. 1, Part 1, p. 115-167.(Cedefop Reference series, 3008).

Lasonen, J.; Young, M. (eds) Strategies forachieving parity of esteem in European uppersecondary education. Jyväskylä: Institute forEducational Research, 1998.


Luburic, V. et al. Teachers’ and Trainers´ Training inVET. Zagreb: Croatian National Observatory,2001.

Luhmann, N. Soziale Systeme. Grundriss einerallgemeinen Theorie. Frankfurt: Suhrkamp,2001.

McDonald, R. et al. New perspectives on assess-ment. Paris: Unesco 1995 (Unevoc studies intechnical and vocational education series,No 4).

McLeish, E. A.; Phillips, D. (eds) Processes of tran-sition in education systems. Oxford: TriangleJournals, 1998, Vol. 8, No 2 (Oxford studies incomparative education series).

Mebratu, T.; Crossley, M.; Johnson, D. (eds) Global-isation, educational transformation and soci-eties in transition. Oxford: Symposium Books,2000.

Miegel, M.; Nölke, A. Transformation strategies andthe role of economic assistance to central andeastern Europe. Leipzig: Institute for PoliticalScience, 1996.

Montenegrin National Observatory. Report on theVET system – Montenegro, Turin: ETF – Euro-pean Training Foundation, 2001.

Nielsen, S. Report on the evaluation of the EU Phare1995 VET reform programme in Romania. Turin:ETF – European Training Foundation, 1999a.

Nielsen, S. Report on the evaluation of the EU Phare1995 VET reform programme in Estonia. Turin:ETF – European Training Foundation, 1999b[Unpublished draft].

Nielsen, S. Report on the evaluation of the EU Phare1998 VET programme in Bosnia-Herzegovina.Turin: ETF – European Training Foundation,2001.

Nielsen, S. VET teachers’ and trainers’ training inCroatia: challenges, priorities and short-termassignments. Paper submitted to the workshopon ‘VET policies’ held in Split, Croatia, on 30January 2002. Turin: ETF – European TrainingFoundation, 2002 [Unpublished draft].

Nölke, A. Implementing economic assistance forcentral and east Europe: the problem of coordi-nation. In: Nölke, A.; Miegel, M. (ed.) Transfor-mation strategies and the role of economicassistance to central and eastern Europe.Leipzig: Leipziger Universitätsverlag, 1996,p. 171-192.

Nölke, A.; Miegel, M. (ed.) Transformation strategiesand the role of economic assistance to central

and eastern Europe. Leipzig: Leipziger Univer-sitätsverlag, 1996.

Nyhan, B. Promoting a European VET researchtradition: the role of the survey and analysismeasures of the Leonardo da Vinci Programme.In: Dietzen, A.; Kuhn, M. (eds) Building a Euro-pean co-operative research tradition in voca-tional education and training: the contribution ofthe Leonardo da Vinci programme’s surveysand analyses. Berlin: BIBB – Bundesinstitut fürBerufsbildung, 1998.

OECD. Development cooperation: efforts and poli-cies of the members of the development assis-tance committee. Paris: OECD, 1997.

OECD. Education and skills catalogue. Paris:OECD, 2001.

OECD. Education policy analysis. Paris: OECD,1999-2001.

OECD. Employment outlook. Paris: OECD, 2001.OECD. Evaluating and reforming education

systems. Paris: OECD, 1996.OECD. From initial education to working life: making

transition work. Paris: OECD, 1996.OECD. Future trends 7: an information base for

scanning the future (CD-Rom). Paris: OECD,2001/02.

OECD. Implementing the OECD job’s strategy:lessons from member countries’ experience.Paris: OECD, 1998.

OECD. Motivating students for lifelong learning.Paris: OECD, 2000c.

OECD. Reviews of national policies for education:Romania. Paris: OECD, 2000b.

OECD. The appraisal of investments in educationalfacilities. Paris: OECD, 2000a.

Paquet, G. The new governance, subsidiarity andthe strategic state. In: Future trends 7: an infor-mation base for scanning the future (CD-Rom).Paris: OECD, 2001/02.

Parkes, D. (ed.) VET under review (editorial). Euro-pean Journal of Education, 2001, Vol. 36, No 1.

Parkes, D. Comparative intervention in the reform ofVET. European Journal of Education, 1995a,Vol. 30, No 3.

Parkes, D. et al. A cross country analysis of curric-ular reform in vocational education and trainingin central and eastern Europe / EuropeanTraining Foundation. Luxembourg: Office forOfficial Publications of the European Commu-nities, 1999.


Parkes, D. Management of change. EuropeanJournal of Education, 1991, Vol. 26, No 1.

Parkes, D. Management of change. EuropeanJournal of Education, 1997, Vol. 32, No 3.

Parkes, D. Parity of esteem. European Journal ofEducation, 1993, Vol. 28, No 2.

Parkes, D. Recent trends in vocational educationand training. European Journal of Education,1995b, Vol. 30, No 3.

Parkes, D. The international perspective. In:Stanton, G.; Richardson, W. (eds) Qualificationsfor the future: a study of tripartite and other divi-sions in post-16 education and training.London: FEDA – Further Education Develop-ment Agency, 1998, p. 171-188 (FEDA reportseries, Vol. 2, No 5).

Parkes, D. VET in transition, renewal, reform andreconstruction. European Journal of Education,1999, Vol. 34, No 2.

Ryan, P. The school-to-work transition: problemsand indicators. Cambridge: Cambridge Univer-sity Press, 2001.

Schütz, H.; Speckesser, S.; Schmid, G. Bench-marking labour market performance and labourmarket policies: theoretical foundations andapplications. Berlin: WZB – Wissenschafts-zentrum für Sozialforschung, 1998 (DiscussionPaper FS198-205).

Sellin, B. et al. Scenarios and strategies for voca-tional education and training in Europe: Euro-pean synthesis report on phase I. Amsterdam:Max Goote Kenniscentrum, 2000.

Shaw, G. European standards in VET: what are theyand who wants them? European Journal ofEducation, 1999, Vol. 34, No 2.

Stanton, G.; Bailey, B. VET under review in England:trends and developments. European Journal ofEducation, Vol. 36, No 1, 2001.

Stern, E. Development evaluation: learning fromsocial and organisational innovations. London:TIHR – Tavistock Institute of Human Relations,1989.

Stern, E. Philosophies and types of evaluationresearch. In: Descy, P.; Tessaring, M. (eds) Thirdresearch report: background report. Luxem-bourg: Office for Official Publications of theEuropean Communities, 2003. (Cedefop Refer-ence series - forthcoming).

Stratmann, G. Help me! (But how?). Economicassistance to central eastern Europe in relation

to transformation theory. Leipzig: Institut fürPolitikwissenschaft, 1995 (Working paper).

Stratmann, G. Sectoral coordination of internationalassistance to central and eastern Europe1990/1996: the case of environment andeducation. Leipzig: Institut fuer Politikwis-senschaft, 1997 (Working paper).

Stratmann, G.; Nolke, A. The design and implemen-tation of economic assistance to central andeastern Europe: suffering from a lack of integra-tion and coordination. Leipzig: Institut fuer Poli-tikwissenschaft, 1996 (Working paper).

Teichler, U.; Gordon, J.; Maiworm, F.(eds) Socrates2000 evaluation study. Paris: EIESP – EuropeanInstitute of Education and Social Policy.

Tessaring, M. Training for a changing society: areport on current vocational education andtraining research in Europe (2nd ed.) Luxem-bourg: Office for Official Publications of theEuropean Communities, 1999. (Cedefop Refer-ence Series).

The World Bank. Hidden challenges to educationsystems in transition economies. Washington,DC: the World Bank, 2000.

The World Bank. Labor markets and social policy incentral and eastern Europe: the transition andbeyond. Oxford: Oxford University Press, 1994.

UNDP. Human development report: Bosnia andHerzegovina. New York: United Nations Devel-opment Programme, 1998.

Unesco. Technical and vocational education andtraining: a vision for the twenty-first century. Setof recommendations to the Director General ofUnesco at the Second International Congresson Technical and Vocational Education, Seoul,April 1999.

Unesco/Unevoc. Cooperation with the world ofwork in technical and vocational education.Berlin: Unesco/Unevoc, 1998.

Unesco/Unevoc. Vocational education and trainingin Europe on the threshold of the 21st century.Berlin: Unesco, 1999 (Unevoc documentNo ED/IUG/018).

US Department of Education. Background of thestudy: study aims and research questions. In:Systemic reform in the professionalism ofeducators. Washington, DC: US Department ofEducation, 1995. Available from Internet:http://www.ed.gov/pubs/SER/ProfEd/bkgd.html [cited 9.6.2003].


Van Wieringen, F. Scenario planning for vocationaland adult education. European Journal ofEducation, Vol. 34, Issue 2, 1999.

Viertel, E. Input into the report on the Peer review ofinitial VET in Croatia. Turin: ETF – EuropeanTraining Foundation, 2003 (forthcoming).

Viertel, E.; Grootings, P. Review and lessons learnedof Phare vocational education and trainingreform programmes 1993-1998. Turin: ETF –European Training Foundation, 2000.

von Bertalanffy, L. An outline of general systemstheory. British Journal for the Philosophy ofScience, 1950, Vol. 1, No 2.

Watson, K. (ed.) Doing comparative educationresearch: issues and problems. Oxford:Symposium Books, 2001.

Whitman, I. The OECD approach to educationpolicy reviews. Paper delivered to the workshopon ‘Methods and tools for the assessment ofVET systems’, ETF – European Training Foun-dation, Turin, 26-27 November 2002. Paris:OECD, 2001 [Unpublished draft].

Widmaier, A. L’évaluation de L’investissement enformation professionnelle. Turin: ETF – Euro-pean Training Foundation, 2000.

Willie, E.; Hodgson, P. Making change work.Cirencester, UK: Management Books 2000,2001.

Windham, D. M. The OECD thematic reviews ofeducation policy in south eastern Europe: aproposed outline for authors of country chap-ters. Paris: OECD, 2000 [Unpublished draft].

Wolf, A. Some final thoughts: vocational educationpolicy in a European context. European Journalof Education, 1993, No 2.

Wurzburg, G. A systemic approach to evaluating theimplementation of lifelong learning in OECDmember countries. Paper delivered to the work-shop on ‘Methods and tools for the assessmentof VET systems’, ETF – European TrainingFoundation, Turin, 26-27 November 2002.Paris: OECD, 2001 [Unpublished draft].

Young, M. Contrasting approaches to the role ofqualifications in the promotion of lifelonglearning. Paper delivered to the workshop on‘Methods and tools for the assessment of VETsystems’. ETF – European Training Foundation,Turin, 26-27 November 2002. London: Instituteof Education, 2001 [Unpublished draft].

Young, M. Improving vocational education:trans-European comparisons of developmentsin the late 1990s. In: Stenström, M.-L.;Lasonen, J. (eds) Strategies for reforming initialvocational education and training in Europe.Jyväskylä: Institute for Educational Research,2000.


Documents

From project to policy evaluation in vocational - Cedefop - Europa