Institutionalising tracer studies to assess the outcomes of workplace

based training: reflections on feasibility

INSTITUTIONALISING DESTINATION STUDIES IN THE POST-

SCHOOL EDUCATION AND TRAINING SECTOR WORKSHOP: 2B

TIME: 09H00 – 13H00 28 SEPTEMBER, BIRCHWOOD HOTEL AND CONFERENCE CENTRE

A new scoping study: Its intentions

• A scoping project embarked to determine:

– The extent to which past data problems have been addressed.

– What work-based skills programmes will be feasible to include in a set of institutionalised tracer studies?

– What is the current state of SETA population databases, for purposes of analysis of population trends?

– What is the current state of SETA databases in terms of the contact details of individual participants?

What can we learn from the literature?

• Internationally pathways approach used to measure progression (particularly ACER, NCVER, and OECD, measuring the processes and outcomes of transition at the level of the individual, aggregated to national level)

• These typical measures assume a linear progression: school -> learnership -> labour market

• BUT individuals have more complex trajectories or journeys, with multiple transitions, especially those who are most economically and socially vulnerable

school -> unemployed -> training -> unemployed -> learnership -> further study -> labour market • work of Raffe (2008) particularly, informed a focus on tracing the

transitions, trajectories, and destinations of young trainees to the labour market, mapping patterns of inclusion and exclusion.

• a methodology employed by Robinson (2009) to measure individual navigations through HE was adapted

Research Design LEARNERSHIPS Source APPRENTICESHIP Source

What kinds of

skills does the

pathway system

produce?

Population trends of

2005/6 and 2009/10

cohorts

DoL /

DHET

database

Population trends

2009/10

DoL /

DHET

database

Indlela

database

What are the

different

pathways in the

transition to

employment?

Follow up survey

tracking pathways of

2007 cohort

HSRC

2007

database

Survey tracking

pathways of 80% of

apprentices in 2009/10

New

database

with

assistance of

five SETAs

To what extent

does the

pathway system

build skills and

capabilities that

enhance

employment

and match firm

demand?

Survey tracking pathways

Three case studies: low, intermediate and high level

skills sectors

Interviews

with

employers

and training

providers,

sectoral data

Which data sources?

2006/07 Learnerships Baseline

2009/10 Learnerships 2009/10 Apprenticeships Baseline

2009/10 INDLELA

SETA data of all participants since inception until 28 May 2007 Mostly NSDS I 243 729

Cohort 1 Population Year 1 of NSDS II 53 644

*Survey of Year 1 6 815

NSDS Phase I (2001 to March 2005), NSDS Phase II (April 2005 to 2011) * Stratified random sampling by SETA, NQF level of Learnership (low, intermediate, high), employment status (18.1, 18.2) +Stratified random sampling by SETA INDLELA: Institute for the National Development of Learnerships Employment Skills and Labour Assessments

DHET data of all registered and

completed participants during 2009/10

DHET data of all registered and

completed participants during 2009/10

Cohort 2 Registered

43 569

Cohort 3 Completed

28 410

Cohort 4 Registered

9 316

Cohort 5 Completed

3 432

Po

pu

lati

on

dat

a Su

rvey

dat

a

Cohort 6 INDLELA data on all applicants for whom appointments were arranged – 5 608 (Certified competent 2303)

Used survey of Year 1 2 524 (37%)

SETA data CHIETA, ESETA, MERSETA,

SERVICES, TETA 10 153

+Survey 1 483 (15%)

Leave school

A W S W&S UOutcome of transition 1

Outcome of transition 2

Outcome of transition 3

ARecord transitions

until entered

Apprenticeship

A W S W&S U

A W S W&S U

Legend:

A = Entered Apprenticeship

W = Worked

S = Studied

W&S = Worked and studiedU = Did not work and did not study

A new survey methodology

Transition into apprenticeship

Respondents are not likely to enter into an apprenticeship directly after school. • Most individuals are likely to enter

after two or three transitions –roughly 85% of the sample.

• They are most likely to do one or two other activities (for instance work, or spend a period unemployed), before entry

• The most complex trajectories into apprenticeship, reflected in the highest number of transitions, is six, but such complex transitions (five and six) are least likely, involving only roughly 2% of the sample.

Transitions N % of sample

1 58 3.9

2 701 47.3

3 555 37.4

4 141 9.5

5 26 1.8

6 2 .1

Total 1483 100

Trajectories into apprenticeship

only 4.1% of the total sample entered an apprenticeship directly after school (A). Apprenticeship is clearly not a career option of choice for school leavers.

1 2 3 4 5 6 N % 1 2 3 4 5 6 N %

W A 316 21.3 S A 345 23.3

W S A 51 3.4 S W A 333 22.5

W S W A 20 1.3 S U A 52 3.5

W U A 14 0.9 S W U A 22 1.5

W B A 13 0.9 S U W A 16 1.1

W U W A 5 0.3 S W S A 12 0.8

W U S A 4 0.3 S U S A 11 0.7

W B W A 3 0.2 S B A 9 0.6

W S B A 2 0.1 S B W A 7 0.5

W S W U A 2 0.1 S W S W A 5 0.3

W U W 1 0.1 S W 2 0.1

W A S A 1 0.1 S W U S A 2 0.1

W U B A 1 0.1 S W B A 2 0.1

W S U W A 1 0.1 S A W A 1 0.1

W S W S A 1 0.1 S U S U A 1 0.1

W U W S A 1 0.1 S A U A 1 0.1

Subtotal 436 29.4 S U B A 1 0.1

U A 33 2.2 S U S W A 1 0.1

U W A 39 2.6 S W U W A 1 0.1

U S A 27 1.8 S U W S W A 1 0.1

U S W A 18 1.2 S W S W S A 1 0.1

U W S A 7 0.5 Subtotal 826 55.7

U W S W A 3 0.2 B A 12 0.8

U S U A 2 0.1 B W A 4 0.3

U W U A 2 0.1 B W 1 0.1

U B A 1 0.1 B U A 1 0.1

U W B A 1 0.1 B S W A 1 0.1

U S A U A 1 0.1 B W U A 1 0.1

U S U S A 1 0.1 B S W U A 1 0.1

U S U W A 1 0.1 Subtotal 21 1.4

U S W S A 1 0.1 A 58 3.9

U W B W A 1 0.1 A W A 3 0.2

U W U W A 1 0.1 Subtotal 61 4.1

Subtotal 139 9.4 Total 1483 100

Most common trajectories into apprenticeship

Race

Total

Trajectori

es

African Coloured Indian White Other

S-A 282 (82) 16 (5) 3 (1) 41 (12) 2 (1) 345 (100)

S-W-A 273 (82) 14 (4) 10 (3) 34 (10) 1 (0) 333 (100)

W-A 186 (60) 40 (13) 14 (5) 66 (21) 4 (1) 316 (100)

A 22 (38) 4 (7) 1 (2) 31 (53) 0 (0) 58 (100)

Insights possible through analysis of survey data

• Assessment of the labour market outcomes and links to educational outcomes of participants – Graduates from learnership programmes: 86% of those who

completed were employed, 79% immediately after completing – Graduates from artisanal programmes: 70% experienced a

smooth transition directly into employment (76% eventually) – Analyse the number of individual transitions (stability of

trajectory) and isolate most common trajectories of groups of individuals – Propensity for un/complicated trajectories and outcomes

informed by race, gender, age, socio-economic status, eg: • Africans over-represented and Whites under-represented, in the

learnership group that remains unemployed • White males more likely to move directly into an apprenticeship after

school

Insights possible through analysis of population data

Assessment of the size and shape of pathway systems • Apprenticeship: roughly 12 000 registered and completed = substantial incr of the size of the

system since the start of NSDSII. • Learnership: system comprises roughly 56 000 registered and completed = overall decrease of 19%

in the number of registrations • Both more successful in terms of reaching registration targets Assessment of the profile of groups participating in these pathway systems • Apprenticeship pathway:

– The majority comprised of black men, who are most likely young learners preparing for employment through indentured programmes (Section 13).

– Achievement of 88% of registration targets. Completion targets are not met as well (only 51%), although they are met more effectively (110%) for unemployed learners.

• Learnership pathway – Concentrated pattern in the provincial distribution, highest number of registrations in Gauteng

and the lowest in poorer and more rural provinces for learnership registration and completion. – Concentration within a few SETAs, a demographic profile that reflects registrations for the

most part, and evidence that in 2009/10, the learnership pathway system continued to provide education and training opportunities primarily for the unemployed.

– MERSETA, SASETA, W&RSETA, FASSET and MQA stand out as those producing the most learnership qualifications, together accounting for 54% of completed learnerships in Year 5.

Apprenticeships - A growth of roughly 20 000 over

5 years (167%)

M 69%

F 31%

B 86%

W 14%

Entered

M 83%

F 17%

B 78%

W 22%

Completed

32 085 CETA (29%) HWSETA (0.2%)

FP&M (0.2%) MERSETA (52%)

Learnerships - A growth of roughly 62 000 over 5 years

(111%)

M 47%

F 53%

B 94%

W 6%

F 51%

M 49%

B 89%

W 11%

Entered

Completed

118 348 WRSETA (12%) PSETA (0.5%)

HWSETA (11%) INSETA (2%)

Internships - A growth of roughly 10 600 over 5

years (267%)

Male 45%

Female

55%

Black 94%

White

6%

B 96%

W 4%

F 58%

M 42%

Entered

Completed

14 595 MICT (19%) SASETA (0.5%)

PSETA (19%) BANKSETA (0.3%)

Comparisons of tracer surveys

SETA POPULATION SAMPLE SIZE RESPONSE RATE TYPE OF DATA ANALYSIS CHALLENGES

FASSET 1126 261 23% Descriptive statistics • Poor response rate

• Incomplete responses

HWSETA 2274 852 37% Descriptive statistics

LGSETA 5255 1681 32%

• The survey data was quality assured and cleansed to ensure completeness of data thereafter; descriptive statistics were presented using the data. With regard to qualitative data, a simple coding process was used to identify themes

• Poor participation rate. Calls went to voicemail or was not answered • Incorrect information • Incorrect person/audience

MERSETA 1030

• Reliability of the database used • Unwillingness to participate by some learners • Some learners could not be traced • Some interpretations by the provider were not in line with merSETA’s expectations

Key questions to frame discussion

• Purpose & Scope

– What are we/ do we want to measure?

– Core research questions

– Target group – completed or registered

– What indicators?

• Integration

– Roles and responsibilities between DHET & SETA

• Sampling

– Generalisability

• Cost implication

– Who will do it (internal or external)

Emerging perspectives

• Seta’s have made efforts towards conducting internal tracer studies, however they acknowledge that they have limited internal capacity to carry out such in-depth studies.

• Seta’s recognise NB & value of conducting tracer studies - (1) to evaluate whether training is working, and (2) if the training allows individuals to enter jobs for which they study.

• Seta’s agree that tracer studies should be conducted on individuals who completed their training irrespective of employment status.

• Seta’s agree on regularly updating learner data and having a uniform/centralised database.