UKFIET

INTERNATIONAL SUPPORT FOR TVETINTERVENTIONS IN DEVELOPING COUNTRIES:

APPROACHES, CONSEQUENCES AND IMPLICATIONS

 Mike Douse September 2015

EVALUATIONS OF TVET INTERVENTIONS

Recent evaluations of 31 donor-funded TVET projects and programmes  involved a range of Development Partners (UN agencies, WB, AsDB, EU, DfID, ILO, USAID, SDC and CIDA);

  addressed challenges in various Eastern European, Asian, African, Caribbean and Pacific nations;

  ranged , in terms of investment, from US$400,000 through US$85 million;

  had a median project period of just under 4 years, excluding extensions [11 of the 31 were extended, one by 42 months!].

Most evaluation reports were obtained from consultancy companies; some from fellow-consultants; 4 had been prepared by this author.  Not necessarily a statistically representative sample – for example, only evaluation reports in the English language were gathered.  Almost all of the evaluations occupied 3-6 weeks, including desk-based preparations along with drafting and finalising days. None of the 31 evaluations was ex post.

Anonymity was sometimes requested. And always respected.

SUCCESS AND ‘UN-SUCCESS’ The author’s judgement was applied in establishing, by inspection, to which of the following categories each of the 31 evaluated projects belonged: A: Highly successful: All or almost all key results achieved on schedule… B: Reasonably successful: Most key results achieved on or close to schedule… C: Not really successful: Some key results at least partially achieved… D: Unsuccessful/Failure: Few or no key results achieved… Less difficult than anticipated. Nearly all of the projects, based upon their evaluations, readily fell into particular categories. Of the 31: 3 were in the ‘highly successful’ category A;7 were in the ‘reasonably successful’ category B;11 were in the ‘not really successful’ category C; and10 were in the ‘unsuccessful/failure/cancelled’ category D.

 

TYPICAL COMMENTS BY EVALUATORS In relation to one of the three projects awarded an ‘A’, the evaluation stated that: “Overall, the project was successfully implemented and the planned results were actually achieved.”

However, a much higher proportion “…acknowledge the many shortfalls” or note that “…delays have been reflected in significant budget under-expenditure”.  One review of a category C project concluded that:“…despite some moderate successes, the Programme has been to a great extent a wasted opportunity… of the Financing Agreement allocation of €8 million, only €2.5 million has actually been spent [including over €1.2 million on Technical Assistance]… many of the LogFrame Indicators are not objectively verifiable but, of those that are reasonably SMART, only about a quarter have been achieved…”. while the evaluators of a category D intervention reported that “…we could find no hard evidence of any sustainable impact of this project: it is as if it never happened”.

OVERALL OBSERVATIONS Analyses suggest that: 

TVET projects tend to have been less successful than those in the broader educational and social sector areas generally;

the absence of up-to-date, research-based data, especially on the labour market and the careers of TVET graduates, hinders successful planning and management;

consequently, TVET is unresponsive to labour market demands; training programmes are supply-driven; non-targeted skills development is a widespread and major weaknesses;

the TVET sector’s low status limits its achievement; the TVET sector is underfunded and raising resources through training levies and/or from fees is not

straightforward; training methods tend to focus on theoretical approaches; in both teaching methodology and curricula, TVET lags behind contemporary technology; and very few of the projects had explicit exit strategies.

 

FACTORS ASSOCIATED WITH SUCCESS

The more successful projects appear, from the evaluations, to be those:

with good quality LogFrames, utilised as a genuine basis for planning;

involving participative planning with stakeholders, clear monitoring and financial arrangements;

where coherent TVET governance and a ‘National Skills Strategy’ already existed;

providing infrastructure and equipment (often a ‘quick win’, despite tendering and maintenance challenges); and

aimed at input/output targets in relation to disadvantaged groups (women, rural locations, school dropouts), although evidence at the outcome level was seldom sought, let alone established.

FACTORS ASSOCIATED WITH UNSUCCESS ‘Unsuccess’ tended to be linked with projects involving: 

poor-quality LogFrames – worse than no LogFrames at all; unattended ‘Risks and Assumptions’; too ambitious or insufficiently demanding Indicators; ‘Calls for Proposals’; private TVET providers; working with NGOs in relation to the informal sector; the ‘vocationalisation’ of secondary education;   Steering Committees (frequently “dysfunctional or delaying”); and inappropriate international specialists.

 

SEMINAL TRANSFORMATIONS 

While moves into, for example, Labour Market Information Systems, Competency Based Training, National Qualification Frameworks and TVET Sector-Wide Approaches

may offer substantial long-term benefits, these major transitions, unless carefully planned, seem to be associated with un-success. 

“The long and unanticipated delays in getting the qualifications framework ready for testing has held back other project components including curriculum development and lecturer upgrading.”

“A SWAp was forced upon the Ministry by development partners before anyone in government really knew what a TVET SWAp would involve so the first two years of the program were completely lost.”

“We were fully behind the idea of CBT but there were no curricula, no lecturers, no standards, no examiners so we just had to forget about it for this 3-year project and base everything on Results Based Training.” “There have been 3 projects since 2001 each aimed at setting up a LMIS but we still do not have one.”

FUNDING TVET

Across the evaluation reports there is a virtual consensus to the effect that:  The TVET sector is underfunded;

  High investment in TVET is vital for its competitiveness; and

  Raising resources through training levies and/or from fees is not always straightforward.  “The initial proposal of introducing a payroll levy was not followed up and no alternative mechanism for private sector contributions was put forward. In the two years following closure of the […] Credit, government funding for the sector was not channelled (as had been agreed) through the Fund which therefore ceased operation.” “Training institutions should be encouraged to be profit-oriented and to become active operators in the training market.” “Fees were possible only for in-service training in niche areas, when course times were adjusted to fit in with work commitments. For most TVET programmes the trainees simply had no cash and schemes such as bursaries and loans were administrative nightmares.”

LOGICAL FRAMEWORKS 27 of the 31 projects had some forms of logical frameworks but only 3 of those were of good quality and utilised as a genuine basis for planning, implementation and monitoring: these three projects were rated ‘A’, ‘A’ and ‘B’ respectively.

Lesser-quality LogFrames, perhaps produced at the last minute as a donor requirement, were not positively correlated with project success: indeed, there were indications that a poorly constructed LogFrame may be worse that no LogFrame at all.  The ‘Risks and Assumptions’ column seemed seldom to have been attended to during either implementation or M&E.

“… [the project] team did not really use the LogFrame as the management tool during the implementation process… they were not familiar with project cycle management (PCM) and results-based management (RBM)”  One piece of advice contained in an evaluation report was to the effect that:

“In LogFrame indicators and in M&E, link the provision of physical and technical support to TVET to the achievement of measurable results including attendance, completion, examination performance and subsequent careers.”

OECD/DAC conventions, while offering a coherent structure, in some instances, they inhibited the creativity of evaluators and in their focussing upon essentials related to each of their unique projects.

COMPETENT CONSULTANTS It is anecdotally apparent that effective TA and expertise is associated with successful projects – but also that there are instances of projects failing due to the inappropriateness of international specialists. This was addressed by more than one evaluation: “One consultant produced nothing in eleven months but was very popular.” 

“…wherever the performance of TA is demonstrably below the required standard, the responsible authorities should take appropriate action, jointly, thoughtfully, determinedly and lawfully, involving warnings, reprimands and dismissals as appropriate. The success of costly and vital projects and programmes should not be jeopardised by any unwillingness to rock boats or grieve nice but unproductive people.”

“Be aware that ineffective consultants are a function of weak project management. Some form of interview should occur before their appointment. All consultants should be regarded as on an initial one month’s trial. If performing poorly, they should be dismissed.”

It should be noted that this applies to national project officers also:

“Project Directors, finance and procurement managers, and others in key positions should be carefully selected and fully prepared for these responsibilities. [Country] officers need the confidence and competence to recognise if a project is off course, to determine why and take appropriate action, and to monitor and assess the contributions of consultants.”

  

 

Infrastructure and Equipment 

The provision of infrastructure and equipment was a quick win in a number of the 31 projects. Buildings were established and equipped more or less on schedule and ceremonially opened by Ministers and DP high officials with good publicity. However, some problems were reported:

“Tendering processes were byzantine.”

“All colleges are covering the maintenance cost themselves, however in some cases, they are left with rough or half-finished work (specially sealing the gaps between the new doors and windows and their surrounding wall frames).”

“…the equipment delivered has not been installed and was not operational because of a number of reasons, e.g. lack of appropriate facilities, absence of specialist for installation, contractor delays.”

“The lecturers do not know how to use some of the equipment so it is just lying idle.”

“The provision of IT equipment can be similarly problematical if there is limited electricity, no facilities for servicing or repair or no permanent staff member available to continue the tuition.” Inevitably, the interesting comments tend to be negative. However, of the 11 out of the 31 projects that included significant works components, it was those physical parts of the projects that were often those completed earliest and almost always those reported as being most appreciated by the beneficiaries.

Calls for Proposals 7 of the 31 projects involved Calls for Proposals. None of these processes occurred on schedule and 2 were cancelled, both for technical reasons related to tendering.  

Steering Committees

There were few favourable comments regarding the contributions of SCs and 4 evaluations spoke negatively, including

“…holding up decisions rather than smoothing the way…”  

“…different people came to each meeting, more and more junior each time, and knowing little about the project”.

“The Project Steering Committee was not used to give guidance to the project. As a result, PSC members knew very little about the project progress and results. This should not be the case.”

 “Much more attention should have been given to the [project] Steering Committee including TOR, membership, chairmanship, briefings, secretariat functions and heightened consciousness. They should have linked [project] with previous and future interventions, building upon the experiences (good and bad) of the past and drawing lessons for the projects that will follow. They did very little of this.”

 

TVET and Education Some signs in a few evaluations that the ‘vocationalisation’ of secondary education being resisted by teachers in those schools and by many parents:

“There remains a need to establish the organisational climate and culture in which the TVET learning process can develop with the respect and vitality it deserves in the overall social, economic and political development.”

“The move to ‘vocationalise’ secondary education, to help fill the skills gap, reduce the problem of youth unemployment and less dependence on foreign workers who generally repatriate their income to support family back in their home country, was still a focus of dispute six years on.”

“This project has involved assistance to the Ministry of Education to rewrite the Education Act to make the system more relevant to meeting the employment needs of [the country with particular reference to skills development for the employability of young people.”

One successful element of a particular category ‘A’ project involved establishing an integrated career guidance system in government secondary schools:

“…young people need to be allowed to make choices about educational and occupational paths as late as possible and education systems should avoid locking them into particular routes.”

   Status

 It is clear from some evaluations that TVET is still perceived as a poor relation of general education: “TVET has suffered from an international focus on basic, and especially primary education, which led to the neglect of post-basic education and training and their non-inclusion in the UN Millennium Development Goals.” and that moving from TVET into higher education is fraught with difficulties: “Another persistent problem is the historical difficulty of progressing to higher education from vocational pathways.”  Certainly, those projects aimed entirely or partially at taking the world of work into the classroom tended to be unsuccessful, not just due to teacher resistance and unpreparedness but also, in some instances, because of poor coordination – even animosity – between those government agencies responsible for general education on the one hand and TVET on the other.