Embed Size (px)
Citation preview
EVALUATION OF THE JOB PLACEMENT PROGRAM
AND THE TRAINING FOR JOBS PROGRAM
SUMMARY
PREPARED BY:
PETER ADAMS AND CATHY TAIT
VICTORIA CONSULTING NETWORK LTD
PREPARED FOR:
EMPLOYMENT INITIATIVES BRANCH MINISTRY OF HUMAN RESOURCES
September 9, 2004
PREFACE Over the period 2002 to 2004 our firm, Victoria Consulting Network Ltd., was
retained by the Ministry of Human Resources to monitor and evaluate the success of two major employment program initiatives: the Job Placement Program and the Training for Jobs Program. Evaluations of both programs are now complete. This document summarises key features of the program, our approach to the evaluation and our findings. More detailed reports on methodology and analysis has been submitted to the Ministry.
Both employment programs are delivered by a range of external service providers. The primary focus of our evaluation has been on the success of the program not on the performance of individual contractors. Therefore, this report does not present information broken down by individual contractor.
TFJ was introduced in late 2002 and it is still relatively early to gauge the impact of the program. Therefore, some of our observations on TFJ must be considered preliminary rather than conclusive. In 2003, a special English as a Second Language component was introduced in TFJ. Our work does not include that component.
Acknowledgements
We would like to acknowledge the support and assistance of Ministry staff throughout this project. We are particularly indebted to Gloria Back, Katherine Rowe, Lindsay Kislock, Shelley Forrester, Bev Verboven, and Dan Blakey from the Employment Initiatives Branch. We are also grateful for the assistance provided by Ministry staff and contractors who have helped us pull together and analyse the data used in this report. Special thanks go to Rob Gage of Learning Works who helped us throughout with analysis of Ministry MIS data. We would also like to acknowledge the help provided by Rob Malatest of R.A. Malatest and Associates who conducted the client surveys, and the staff of Quartech Systems Ltd who provided us with data extracts for JP and TFJ. While we are grateful for the assistance provided, we remain responsible for any errors and omissions in the report.
TABLE OF CONTENTS CHAPTER ONE: BACKGROUND TO THE EVALUATION ............................................................... 1
AN OVERVIEW OF THE PROGRAMS ............................................................................................................... 1 KEY EVALUATION QUESTIONS...................................................................................................................... 2 THE RESEARCH COMPLETED........................................................................................................................ 3 ESTIMATING PROGRAM IMPACT ................................................................................................................... 4
CHAPTER TWO: THE JOB PLACEMENT PROGRAM ..................................................................... 5 REFERRALS ................................................................................................................................................. 5 CHARACTERISTICS OF PERSONS REFERRED.................................................................................................. 6 REFERRAL TARGETS .................................................................................................................................... 7 PERSONS ACCEPTED .................................................................................................................................... 7 PLACEMENT SERVICES PROVIDED................................................................................................................ 7 SUCCESS AT PLACING CLIENTS .................................................................................................................... 8 CHARACTERISTICS OF THE FIRST JOB......................................................................................................... 10 INDEPENDENCE......................................................................................................................................... 10 PROGRAM IMPACT..................................................................................................................................... 12 PROGRAM COST ........................................................................................................................................ 14
CHAPTER THREE: THE TRAINING FOR JOBS PROGRAM......................................................... 15 REFERRALS ............................................................................................................................................... 15 CHARACTERISTICS OF PERSONS REFERRED................................................................................................ 15 TRAINING.................................................................................................................................................. 16 JOB PLACEMENT AND INDEPENDENCE ....................................................................................................... 17 COMPARISON TO INDEPENDENCE TARGET.................................................................................................. 20 CHARACTERISTICS OF THE FIRST JOB......................................................................................................... 20 PROGRAM IMPACT..................................................................................................................................... 21 PROGRAM COSTS....................................................................................................................................... 22
CHAPTER FOUR: PRINCIPAL FINDINGS......................................................................................... 24 THE JOB PLACEMENT PROGRAM ................................................................................................................ 24 THE TRAINING FOR JOBS PROGRAM ........................................................................................................... 29
1
CHAPTER ONE: BACKGROUND TO THE EVALUATION
AN OVERVIEW OF THE PROGRAMS
The Job Placement Program
In January 2000, the Ministry of Human Resources introduced the Job Placement Program (JP). The objective of the program is to place eligible employment assistance clients in jobs so that they become, and remain, independent of BCEA. Private sector and non-profit contractors are engaged to assist clients find employment that will lead to sustained independence.
BCEA clients are referred to JP contractors by local Ministry offices. Contractors interview the clients referred and may accept or not accept them. Contractors then have 90 days in which to ‘place’ accepted clients into employment. Contractors assist clients in any way they consider necessary to help them find work. Typically they help clients improve their job search and interview skills and refer them to job openings. Once clients are placed, they must become independent of BCEA within a limited time or the contractor receives no payment for the service provided.
The contracts are ‘performance-based’: contractors are paid only when the clients become independent of BCEA. The amount paid for each client depends on the number of months of independence they achieve. Full payment is received only for clients who achieve 19 months of independence within a given period.
The program was first introduced as a two-year pilot (JP1) involving two contractors, which was subsequently extended for a further six months. In July 2002, the program moved out of the pilot stage and, after a competitive bidding process, the Ministry entered into one-year contracts (JP2) with the two original contractors and added two additional contractors in certain areas of the province. Effective July 2003, these contracts were extended for a further year (JPX).
The Training for Jobs Program
In December 2002, the Ministry introduced the Training for Jobs (TFJ) program. The objective of the program is to provide short-term, job-relevant training to BCEA clients and to place those clients into jobs after training is completed. The program, which is designed for clients who can become job-ready with short-term interventions, was introduced as a one year pilot. Program intake has been extended for a further year and currently runs until November 2004.
External contractors are engaged to deliver the program, which is divided into a training component and a job placement component. The type and length of training offered varies from contractor to contractor. The job placement component is more
2
standard and is modeled on the JP program. Each contractor is responsible for both components.
Following a call for proposals, the Ministry selected fifteen contractors offering differing programs of differing size to deliver the program. Given the pilot nature of the program, the Ministry encouraged different approaches among contractors. Most of the training courses are offered on a block basis with pre-determined start and end dates while others offer continuous entry. A few contractors offer personalized training courses where the programming is customized to individual clients. Four of the fifteen contractors were responsible for 74% of the expected number of clients.
Contractors are paid for each client accepted. For the training component, 75% of the payment is provided as a ‘fee for service’; the remaining 25% is ‘performance-based’ and is paid only if the client successfully completes the training. Payment for the job placement component is all performance-based: contractors are paid only if the client is placed into a job and becomes independent of BCEA with defined timelines. As in JP, the contractor receives a payment for each independence milestone achieved by the client. Although modelled on the JP program, some of the timeframes in the job placement component of TFJ are more challenging than those in JP1.
KEY EVALUATION QUESTIONS
At the start of this project we developed a set of key evaluation questions in conjunction with Ministry staff. The questions are similar for each of the programs under review, except we developed some additional questions for TFJ associated with the training component. While it has been possible to shed light on most of the many issues raised by these questions, it has not been possible to answer fully all the questions posed with the information available.
Four central evaluation questions were established for JP:
1. Does JP help BCEA Clients achieve independence through sustained employment (to a greater degree than would otherwise be possible)?
2. Is JP equally effective for all clients?
3. What characteristics of JP contribute to the success of the program?
4. What is the net cost (savings achieved less program cost) of the program to the Ministry?
In support of these main questions, a number of specific subsidiary questions and sub-questions were identified:
1 For example, TFJ contractors only have 6 months following placement for a client to achieve independence; in contrast, JP2 contractors had 12 months.
3
• Were a sufficient number of eligible clients referred to and accepted into the program?
• Did a high proportion of participants find work quickly because of the services provided and the help of contractors?
• Did a high proportion of persons who found work achieve and maintain independence from BCEA?
• What Ministry savings are attributable to the program?
In addition to the questions posed for JP, a number of additional questions were identified for TFJ:
• Did a high proportion of accepted clients successfully complete the training offered?
• Was the employment found linked to the training provided?
• How much did the training services cost?
THE RESEARCH COMPLETED
We have undertaken a variety of research tasks to assess the success of the programs to date. We examined the characteristics of the JP and TFJ clients groups2 and compared them to the characteristics of other BCEA ‘Expected to Work’ clients to see how representative program clients were of the ‘Expected to Work’ caseload. To follow the progress of program participants over time, we divided participants into monthly cohorts based on the month in which they were first referred to JP or TFJ. We have tracked these cohorts by monitoring their program and BCEA status in each of the months following the month of first referral.
For our analysis we relied on data from the information systems developed to support JP and TFJ as well as an extract from the Ministry’s principal BCEA database. These systems were designed primarily to meet the operational needs of Ministry staff. We have had to make a number of assumptions in using data from these sources because they were not always internally consistent. Even so, we are confident that the data used in the report is sufficiently reliable to support the conclusions drawn.
Our other research tasks included a telephone survey of over 1,800 program participants, conducted to assess client satisfaction with services received3. As well, we interviewed a sample of Ministry caseworkers about the referral process and their working relationship with contractors. Also, we interviewed contractor staff in various locations throughout the province about the services provided by them and asked them to
2 For example, family type, gender, age, location; attachment to BCEA 3 The survey was administered by a professional market research firm, R.A.Malatest and Associates Ltd.
4
comment on program design. Finally, we analysed data provided by JP contractors on the financial supports provided to clients and job turnover among their clients.
ESTIMATING PROGRAM IMPACT
The BCEA caseload is not static: every month, people leave the caseload and others join. Turnover is highest among clients in the Expected to Work category. We cannot assume, therefore, that everybody who achieves independence through JP or TFJ would have remained dependent on BCEA if they had not been accepted into an employment program. To measure a program’s real impact on BCEA clients we need to compare the experience of program participants to that of a carefully selected ‘control’ group.
By comparing the performance of program participants to that of a control group we can estimate the incremental benefit that is attributable to the program4. For this inference to be valid, however, the control group and the program participants should (ideally) be the same in every relevant way except for the fact that the control group did not attend the program. The probability that a person will leave BCEA is associated with their recent experience on BCEA and turnover tends to be higher for clients who have had less attachment to BCEA. Therefore, the design of the control group must take into account the recent history of program clients with BCEA.
In selecting a control group it is desirable to standardise for as many potential causes of bias as possible. In practice, no control group is perfect and, for this reason, we have used a combination of control groups to estimate program impact5. As a result, we are confident about the general magnitude of program impact presented in this report.
One of the difficulties associated with the two programs is the lag between the time the service is provided and the time that program benefits start to become clear. This is particularly difficult for TFJ which has been in place for only a relatively short time. While JP has been in place since 2000 both the program and clients referred to JP have changed significantly over the last four years. Therefore, conclusions drawn from JP1 may not be as relevant to today’s program as conclusions drawn from JP2 and JPX.
A further complication is the fact that the BCEA Expected to Work caseload has been declining sharply over the last two years because of changes in government policies with respect to eligibility and benefits. As part of our research, we have had to distinguish the impact of these policies from the impact of the programs.
4 For example, if we find that members of the control group averaged 3 months of independence from BCEA during a 12 month period, but program participants averaged 4 months of independence during the same period, we can infer that the extra 1 month of independence is due to participation in the program. 5 A Technical Appendix prepared for the Ministry presents a more complete discussion of control groups and the methods used to estimate program impact.
5
CHAPTER TWO: THE JOB PLACEMENT PROGRAM
REFERRALS
Almost 120,000 individuals have been referred to JP since the program’s inception in January 2000. Because individuals may be referred to different contractors at different points in time, the total count of referrals is greater than the number of individuals referred.
The target client group for JP was changed significantly starting in 2002. Under JP1, the program was targeted at persons with little attachment to BCEA who could be expected to make a rapid return to employment. Under JP2/X, the target group has been widened to all ETW clients. Despite this change, the average number of persons referred each month has been lower under JP2/X (2,210) than under JP1 (2,770).6 The decline is directly associated with the sharp decline in the ETW caseload that was triggered by changes in Ministry policy. Exhibit 2.1 graphically illustrates the steep decline in ETW cases that started in 2002. It also shows that the proportion of cases referred to JP has increased over time as a result of the maturing of the program, changes to the program target group and the reduction in ETW caseload. Currently, close to 70% of ETW cases have been referred at some time to JP.
EXHIBIT 2.1 PROPORTION OF EXPECTED TO WORK CASELOAD REFERRED TO JP
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
90,000
100,000
Jan-00 Jan-01 Jan-02 Jan-03 Jan-04
# Cas
es
ETW Referred to JP
6 The JP2/X average is composed of 600 persons per month who previously had been referred to JP1 and 1,610 persons per month who were referred for the first time under JP2 or JPX.
6
CHARACTERISTICS OF PERSONS REFERRED
The characteristics of persons referred to JP have been changing in recent years. Initially, under JP1, referrals were drawn largely from persons with little or no recent attachment to BCEA – in keeping with the policy of referring the most employable persons to JP. By the start of JP2, however, this pattern had changed for two reasons: the Ministry’s policy had changed to one of referring most ETW clients to JP; and, the ETW caseload mix itself had changed because the number of persons coming on to BCEA had fallen dramatically. As a result, a substantially greater proportion of individuals referred under JP2 had considerable attachment to BCEA than those referred under JP1. Nevertheless, on average, persons referred still had less attachment to BCEA than the ETW caseload as a whole. In 2003, the pattern began to change again -- both the ETW caseload and persons referred to JPX in 2003 were less attached to BCEA than had been the case in 2002. These changes are illustrated in Exhibit 2.2 which compares the ETW caseload and JP referrals by their recent attachment to BCEA.7
EXHIBIT 2.2 DISTRIBUTION OF ETW CASELOAD AND PERSONS REFERRED TO JP BY RECENT ATTACHMENT TO BCEA8
September 2000 Recent BCEA Attachment ETW JP1 Referrals
None in previous 12 months 4% 37% Intermittent 45% 43% Continuous over previous 12 months 50% 20% Total 100% 100%
September 2002 ETW JP2 Referrals
None in previous 12 months 3% 13% Intermittent 39% 41% Continuous over previous 12 months 58% 46% Total 100% 100%
September 2003 ETW JPX Referrals
None in previous 12 months 4% 22% Intermittent 45% 40% Continuous over previous 12 months 51% 38% Total 100% 100%
7 The figures in the Exhibit may understate the proportion of persons in the New category under JP1. Some of the individuals referred to JP1 never had any prior or subsequent association with BCEA and, for this reason, their BCEA experience could not be identified from Ministry records. 8 This Exhibit is based on persons referred for the first time and persons re-referred under an earlier JP contract. For example, persons referred to JPX include person previously referred under JP1 or JP2. In other reports we have used the terminology New, Intermittent and Persistent for the three categories.
7
REFERRAL TARGETS
The contracts with service providers establish referral targets that the Ministry strives to meet. However, with a declining caseload these targets have proven difficult to achieve. As a result, meeting referral targets have become a major administrative concern for the Ministry for two reasons. First, with the emphasis on employment assistance, the Ministry wants to be sure that all ETW clients are being referred to JP if there is any likelihood that they could benefit from the program. Second, having entered into contractual obligations with service providers, the Ministry wants to strive to meet those obligations.
While the Ministry’s focus on referral targets is understandable, it also leads to some undesirable consequences. For example, some clients are being referred to JP even when it is clear that they are not ready or suitable for the program. They are being referred just to make up the numbers. Also, referrals to some contractors may be interrupted so that the referral targets of other contractors can be met. As a result, contractors may not have sufficient clients available when employers contact them – a situation that can undermine their relationship with those employers.
PERSONS ACCEPTED
Under JP1, when the number of referrals was relatively high, contractors were quite selective about the clients chosen. Under JP2 and JPX, they have become less selective and have been accepting most of the persons interviewed Also, contractors and Ministry staff have made greater efforts under JP2/X to track down ‘no-shows’ with the assistance of Ministry staff. Under JP2/X, more than 70% of referred persons have been accepted by contractors – around 10-15% do not show for interview and contractors do not accept a further 10-15%.
PLACEMENT SERVICES PROVIDED
Once a client is accepted, the contractor can decide how best to assist that person achieve independence. They offer workshops on topics such as job search skills, provide one-on-one coaching and give practical assistance with résumé preparation. Contractors try to ‘match’ clients to known job openings but they also expect clients to undertake independent job search. Contractors also provide specific supports to clients who have job interviews (e.g. transportation, haircuts) or who have job offers (e.g. work clothing or work tools). On average, contractors are spending $100 per person on clients who receive specific supports.
The majority of clients surveyed expressed overall satisfaction with the services they received; however, quite a number expressed some level of dissatisfaction. Exhibit 2.3 indicates the satisfaction of placed and not placed clients, and the extent to which they feel the program helped them in specific ways. Not surprisingly, placed clients were more positive in their assessment of the value of the services they received.
8
EXHIBIT 2.3 CLIENT SURVEY: SATISFACTION AND VIEWS OF SERVICES RECEIVED9
Persons Placed Persons Not Placed
Strongly Agree or
Agree
Disagree or Strongly Disagree
Strongly Agree or
Agree
Disagree or Strongly Disagree
As a result of the service I feel I am more employable 49% 24% 40% 36%
The service taught me new job search skills 49% 30% 44% 35%
The job leads were helpful in getting a job 46% 34% 34% 44%
Overall I was satisfied with the service I received 72% 17% 55% 27%
Sample Size: placed = 566; not-placed = 387
Placed clients were also asked to rate the importance, in general, of the help they received from the contractor in finding the job they got. Forty percent said the program’s help was very or quite important, while 54% said the help was not very or not at all important.
SUCCESS AT PLACING CLIENTS
Over the period 2000-2003, almost 40,000 persons were placed by the program. Contractors are continuing to place as many people each month under JP2/X as they did under JP1 (allowing for seasonal differences) despite declining referrals and the changing characteristics of persons referred - as Exhibit 2.4 shows.
Exhibit 2.4 also compares the number of persons actually placed to the independent placement targets established for each contract period.10 It shows that the targets were substantially exceeded in JP1. The targets were increased under JP2 but were not met. Under JPX placement targets were reduced somewhat and these targets have been met. Overall, the average number of people placed per month since July 2002 (810) is higher than the monthly average before that date (730). Under JP2/X contractors have placed approximately 56% of persons accepted, a higher percentage than under JP1.
9 Percentages presented do not sum to 100% because neutral responses and non-“responses/”don’t know” are not shown in this table. The non-responses/”don’t know” category represented less than 5% of responses on most items. 10 The program does not establish placement targets but independent placement targets i.e. the number of clients that are placed by the contractors and achieve at least one month of independence. Since placement is the first step to independence, we have used the independent placement targets as the appropriate targets for placement.
9
EXHIBIT 2.4 EMPLOYMENT STARTS
Employment Starts by Job Start Date - Compared to Placement TargetsJob Placement Program
0
200
400
600
800
1000
1200
2000 2001 2002 2003 2004
JP1 JP2/X Target
This pattern of sustaining the number of placements has been possible despite the fact that clients referred to and accepted into the program have been less job ready than clients referred in the early years of the program. Contractors have adjusted their programs to assist a more challenging client group.
Half of those who find jobs are placed in under 6 weeks. Others take longer to place and the average time from referral to placement is 7.5 weeks. Nevertheless, a substantial number of people accepted by contractors are not placed by them in the time available. Contractors do not attribute this to a shortage of jobs available. They believe that it is difficult for clients who have been attached to BCEA for some time or who have significant personal barriers to employment (e.g. addictions; physical limitations; limited education or English language skills; lack of a social support network) to be placed quickly.
The clients who become placed tend to have less attachment to BCEA, on average, than all persons referred. However, contractors have demonstrated that they can help place clients with considerable attachment to BCEA. As contractors point out, motivation is the most important factor in determining whether a person will become employed. Single parents have had no more difficulty in finding employment than other
10
persons accepted. In fact, most contractors report that single parents typically tend to be a very enthusiastic and motivated client group.
CHARACTERISTICS OF THE FIRST JOB
The program is placing people in a wide range of jobs with employers large and small. Exhibit 2.5 summarises the average wage, average number of hours and average earnings of clients in the first job started under the program. The average wage is higher than the provincial minimum wage and that average earnings are larger than the benefits that would be paid by BCEA in most circumstances.11 Starting earnings are, on average, significantly lower for women than for men.
EXHIBIT 2.5 CHARACTERISTICS OF THE FIRST JOB
Women Men Both Hours Per Week 26.8 31.6 29.2 Hourly Wage Rate $9.63 $11.04 $10.32 Weekly Earnings $259 $356 $306
Some of the first jobs are part-time and more than 25% of the jobs pay less than $200 per week. Most contractors see part-time and temporary jobs as a stepping stone to better paying and longer-lasting jobs. Also, once ‘placed’ the client can remain with the contractor for a longer period of time. However, this may not necessarily be in the best interest of the client if the contractor does not help them move beyond part-time or temporary work.
Data provided by the contractors suggests that 45% placed people are still in their first job 18 months after they were referred. The client survey found slightly more turnover. Thirty eight percent of placed clients surveyed reported that they were still in the first job they found through the program.
INDEPENDENCE12
Overall around 90% of persons placed under JP2/X have achieved at least one month of independence within the time provided for in the contract. As time passes, most independent clients achieve additional months of independence. Those placed in the early months of JP2/X have earned, on average, more than 14 months of independence, which is 70% of the maximum possible since being placed. Within the average, there are significant differences among clients. Exhibit 2.6 shows the number of months of independence earned by clients who were placed in August 2002 and subsequently
11 38% of clients are paid the minimum wage. 12 Throughout, independence means one month or more of independence from BCEA.
11
became independent. One third has achieved 19 months or more of independence. In contrast, 10% have earned less than 6 months of independence.
EXHIBIT 2.6 MONTHS OF INDEPENDENCE EARNED BY PERSONS PLACED IN AUGUST 2002
Months of Independence
Per Cent of Persons Who
Achieve Independence
Less than 6 months 10% 6- 11 months 15% 12 – 18 months 42% 19 or more months 33% All 100%
Some of this variation in months of independence is attributable to differences in the time taken to achieve independence. The delay from placement to independence averages 11 weeks but can be much shorter and very much longer. Nevertheless, over 80% of placed people achieve their first month of independence within 15 weeks. The average delay is longer for women than men, and longest for women who are single parents.
Some placed clients cycle on and off of BCEA because employment is not sustained. However, continuity in independence is quite high overall: over 80% of persons who become independent remain so for three continuous months; over 60% remain independent for twelve continuous months.13
It is very important to the contractors that clients become independent and sustain independence because this is the only way in which they receive payment for the services they have provided. Therefore, contractors continue to assist clients with job retention and, if necessary, job search after they have been placed and become independent. Of the placed clients who experienced job loss, 41% reported that the contractor had helped them to look for a subsequent job.
Once placed, some clients achieve independence quickly and remain independent. Because payment to the contractor is linked to independence achieved by the clients, contractors may receive the largest payments for clients who were easiest to place and for whom they have had to provide the least support. In contrast, they may earn very little from providing a large amount of support to a client who finds it difficult to find a job or sustain independence. Under the performance-based contract, there is no direct
13 Based on JP2 clients who were placed in 2002.
12
relationship between the payment received for, and the effort expended on, individual clients.
PROGRAM IMPACT
It is clear that a large number of JP clients have achieved a substantial degree of independence. However, the fact that independence is being achieved does not, by itself, demonstrate that the program has had a positive incremental impact for clients or has resulted in lower BCEA payments for the Ministry.
There is always turnover in the BCEA caseload. This fact is illustrated by Exhibit 2.7 which tracks three cohorts of clients who were on the Expected to Work caseload at different dates.14 The cohorts tracked are persons who had been on the BCEA caseload continuously for at least 12 months on the dates selected. The Exhibit shows that, even among people with considerable attachment to BCEA, a significant proportion do not remain on the caseload. For example, of the cohort who were on the caseload in September 1998, 25% were no longer on the caseload 12 months later and 45% were no longer on the caseload 36 months later. In recent years, the pace at which people have left BCEA has increased as a result of changes in Ministry policy. Therefore, the September 2000 and September 2002 cohorts have been leaving the caseload faster than the September 1998 cohort.
EXHIBIT 2.7 TURNOVER IN PERSONS ON THE EXPECTED TO WORK CASELOAD WHO HAD CONTINUOUS ATTACHMENT OVER THE PREVIOUS 12 MONTHS
Percentage No longer on the Caseload After:
Attached ETW Cohort at:
12 months
24 Months
36 Months
September 1998 25% 37% 45% September 2000 23% 42% 55% September 2002 35%
Since there is always turnover in the BCEA caseload, it is possible that persons who achieve independence through JP are simply the people who would have achieved independence without the assistance of the program. We need to estimate whether these same people would have achieved more or less months of independence if they had not participated in the program.
To demonstrate the true impact of JP on independence we need to compare the performance of program participants to the performance of a suitable control group.
14 These cohorts include persons who were referred to JP and those who were not.
13
Exhibit 2.8 presents our estimate of the incremental impact of JP2.15 It compares the average number of months of independence earned by persons accepted into the program to the average months of independence earned by those who were referred but not accepted.16 The Exhibit shows a small but positive program impact for each accepted person of 0.4 months of incremental independence 21 months after referral. This level of impact would translate into cumulative savings of BCEA payments in the order of $5 million for JP2 after 21 months17. If the pattern to date is sustained, the cumulative savings attributable to the program will continue to increase slowly over time.
EXHIBIT 2.8 ESTIMATED IMPACT OF JP2 BASED ON ACCEPTED CLIENTS
Ave Months of Independence Achieved
After 21 Months
Persons Accepted 10.5 Months
Persons Referred but Not Accepted 10.1 Months
Incremental Months of Independence Attributable to the Program Per Accepted Person
0.4 Months
Note: The period of 21 months was chosen because it was the longest available for JP2 clients at the time
Clients who are placed but do not achieve even a single month of independence can still generate employment earnings that lead to lower BCEA payments than otherwise would be paid. Our tracking of these clients shows that the Ministry does receive a financial benefit that, on average, is equal to 10% of the BCEA payment made prior to participation in the program. However, the incremental benefit begins to diminish after six months and is not sustained over time.18
In addition to the financial benefits to the Ministry, there could be financial benefits to other government departments as a result of people being employed. Also, there may be real but less easily measured benefits to families and individuals that are not captured in the savings to government (e.g. quality of life, family stability, and feelings of self-worth). We do not understate the importance of these benefits but we have not tried to measure them as part of our study.
15 This estimate is based on a weighted average of all persons accepted and not-accepted in the period July 2002 to June 2003, adjusted for differences in prior attachment to BCEA. See Technical Appendix for a more detailed description of the methodology used to calculate incremental impacts. 16 The not-accepted group includes persons who did not show for an interview with the contractors. 17 Please refer to Technical Appendix for details of this calculation. 18 Some of these people also achieve independence from BCEA but not while part of the program.
14
PROGRAM COST
Exhibit 2.9 shows the maximum payment for each client that achieves independence through the JP2. These rates were established based on competitive bids from contractors. Contractors do not receive the maximum unless the client achieves 19 months of independence in the time allowed.19
EXHIBIT 2.9 MAXIMUM PAYMENTS PER INDEPENDENT CLIENT UNDER JP2
Type of Client Maximum JP2
Payments* Single Parents $3,904 Others $3,628
*The amounts vary by contractor and region. Figures shown are a weighted average based on the mix of actual payments to
date.
Total payments to contractors for JP2 alone are expected to reach $26 million over a four year period. This amount excludes other Ministry costs associated with program delivery (e.g. staff salaries, systems development and maintenance costs, etc.). If the impact of the program is sustained, the cumulative savings to the Ministry in BCEA savings will exceed the amount paid to contractors 6 to 7 years after the period of referral to JP2.
19 The figures presented are payments to contractors not the costs incurred by contractors. The Ministry does not ask contractors to provide information on their costs.
15
CHAPTER THREE: THE TRAINING FOR JOBS PROGRAM
REFERRALS
To date, 10,400 BCEA clients have been referred to the program. Of these, 8,570 were referred in the first program intake year that ran from December 2002 to November 2003. In the first program year the number of persons referred fell well below the referral target of 14,526. In the second program year, the referral target has been reduced and, thus far, is being met. The lower than expected number of referrals can be attributed to the decline in the ETW caseload and the Ministry’s policy of considering clients for JP before they are considered for TFJ.
CHARACTERISTICS OF PERSONS REFERRED
Persons referred to TFJ have had greater attachment to BCEA than those referred to JP2. Over 70% of those referred to TFJ have been on BCEA continuously for the previous 12 months. This compares to 46% for persons referred to JP2 and 58% for ETW clients as a whole – see Exhibit 3.1.
EXHIBIT 3.1 ATTACHMENT TO BCEA
Persons referred to:
Recent BCEA Attachment* ETW
(Sept. 2002) TFJ** JP2 None in previous 12 months 3% 2% 13% Intermittent 39% 26% 41% Continuous over previous 12 months 58% 72% 46% Total 100% 100% 100%
*In other reports we have used the terminology New, Intermittent and Persistent for the three categories presented here. **Based on the first program year
Also, persons referred to TFJ have more employment barriers than those referred to JP2. For example, around 15% of persons referred to TFJ clients had ESL needs, compared to 9% for JP2 referrals. Also, almost 50% of persons referred to TFJ had no work experience in the three years prior to screening, compared to 30% for JP2 referrals.20
20 Based on information provided by clients in answer to questions on the Ministry’s employability screening tool.
16
TRAINING
Not all of the persons referred have been accepted by contractors. Some clients do not show up at the contractors’ office; others are considered unsuitable for the program or for the specific training being offered. Individuals who are not accepted by a contractor may be referred to another contractor. Overall, 55% of persons referred to the program in the first intake year were accepted into training, compared to an expected 33%. Since approximately 15% of persons referred do not show for interview, contractors have accepted around 65% of persons who were interviewed by them.
The higher than expected acceptance rate has compensated for the lower than expected number of referrals. As a result, the total number of persons accepted into training in the first year of the program (4,759) was above the target number for that period (4,572). The acceptance rate has varied among contractors – from a low of 37% of referrals to a high of 73%. This variation is attributable in part to differences in the size of contractor. Some of the smaller contractors have had referral numbers closer to their target levels and, therefore, have been able to fill their training spaces with a lower acceptance rate.
In the first program year, the majority of training provided to accepted clients (68%) was provided on a ‘block’ basis with predetermined materials and prescheduled start dates; a further 12% allowed for ‘continuous’ entry. The balance (20%) provided training programs designed around the needs of the individual client as assessed by the contractors providing those services.
The majority of the training courses are between 10 or 16 weeks duration but some can be less than 4 weeks and some greater than 26 weeks. The personalized courses are more variable in length than the block or continuous courses. The vast majority of training provided falls into three of these categories: sales (46%), business (24%), and trades (18%). These are, however, very broad categories.
In addition to training for particular occupations, TFJ contractors provided various forms of other training including life skills, job search training and employability skills. In many cases the training program included a work experience component. Clients interviewed reported a good deal of satisfaction with the training they received - see Exhibit 3.2. As might be expected, placed clients were more satisfied than not placed clients. The majority of clients found the training easy to learn.
Three quarters of clients accepted by contractors in the first training year have completed the TFJ training component and, in the language of the program, have become ‘job ready’. The overall training success rate of 75% meets the program’s target. The rate varies among contractors: from a low of 59% to a high of 87%.
17
EXHIBIT 3.2 CLIENT SURVEY RESULTS: SATISFACTION WITH TRAINING COMPONENT
Placed Clients Not Placed Clients
Strongly Agree or
Agree
Disagree or Strongly Disagree
Strongly Agree or
Agree
Disagree or Strongly Disagree
I was very satisfied with the quality of the training 68% 18% 60% 24%
The training was easy to learn 74% 10% 69% 16% The work experience placement was a useful part of the training
55% 22% 42% 30%
Sample size: placed = 435; not placed = 417
For people who become ‘job ready’, the average time from original referral to job ready status has been 24 weeks.21 The time taken to achieve job ready status varies considerably among the contractors depending on the type of course offered and the scheduling of those courses. The average lag between referral and ‘job ready’ status has been longer for the personalized courses than the other types of courses.
JOB PLACEMENT AND INDEPENDENCE
After TFJ clients have reached ‘job ready’ status, contractors have four months to help them find employment and six months from the date of employment to achieve independence from BCEA. TFJ contractors offer many of the same services during the job search phase as JP contractors. The majority of TFJ clients surveyed expressed overall satisfaction with the job placement services they received. Exhibit 3.3 indicates the satisfaction of placed and not placed clients, and the extent to which they feel the program helped them in specific ways. As with JP clients, placed clients were more positive in their assessment of the value of the services they received. In general, TFJ clients expressed greater satisfaction than did JP clients.
21 This average includes the time taken in re-referral where applicable.
18
EXHIBIT 3.3 CLIENT SURVEY: TFJ SATISFACTION AND VIEWS OF JOB PLACEMENT SERVICES RECEIVED22
Placed Not Placed
Strongly Agree or
Agree
Disagree or
Strongly Disagree
Strongly Agree or
Agree
Disagree or
Strongly Disagree
As a result of the service I feel I am more employable 72% 14% 58% 22%
The service taught me new job search skills 65% 16% 54% 23%
The job leads were helpful in getting a job 55% 27% 30% 41%
Overall I was satisfied with the service I received 74% 12% 63% 22%
Sample size: placed = 435; not placed = 417
Placed clients were also asked to rate the importance, in general, of the help they received from the contractor in finding the job they got. Sixty-two percent said the program’s help was very or quite important, while 29% said the help was not very or not at all important. Nineteen percent of the clients who were not placed by the program went on to find paid employment after they left the program. Of those who did go on to employment, 67% said that the help they received through the program was very or somewhat important in helping them to get their job, while 33% said it was not very or not at all important. Again, these results are more positive than the results from the survey of JP clients.
Exhibit 3.4 shows the extent to which job ready clients have been placed and achieved independence as at the end of April 2004. As expected, those who became job ready recently are less likely to have been placed than those who became job ready in 2003.
22 Percentages presented do not sum to 100% because neutral responses and non-“responses/”don’t know” are not shown in this table. The non-responses/”don’t know” category represented less than 5% of responses on most items.
19
EXHIBIT 3.4 STATUS OF JOB READY CLIENTS BY DATE JOB READY
0
50
100
150
200
250
300
350
400
450
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4
2003 2004
Num
ber o
f Clie
nts
indepPlaced Not IndeNot Placed
According to program design, clients who became job ready before the end of 2003 should have been placed by the end of April 2004. Similarly, clients who became job ready before the end of June 2003 should have achieved independence before the end of April 2004. It is clear, therefore, that a significant proportion of job ready clients are not being placed and not becoming independent in the time recognized under the program.
The program is placing 68% of persons who become job ready and that 53% of job ready clients achieve at least one month of independence. The degree to which job ready clients achieve independence varies widely among contractors. However, most are clustered in the range of 45% to 70%.
The time from ‘job ready’ status to placement averages 5 weeks for TFJ clients, which is faster than the average delay between acceptance and placement under JP. The average time taken to find work varies by region and the lag tends to be longest in the North at seven weeks. The average lag is very slightly longer for women (5.3 weeks) than men (4.6 weeks). The average time taken between placement and independence for persons who have achieved independence is 13 weeks, which is two weeks longer than the average for JP. For TFJ clients the delay is slightly longer for women (13.5 weeks) than for men (12.5 weeks).
20
COMPARISON TO INDEPENDENCE TARGET
The target set at the start of the program was that 60% of accepted clients would achieve at least one month of independence. Performance to date suggests that the program is only achieving a 40% success rate.23 However, the original program target was ambitious when viewed against the experience of JP. Under JP2, only 51% of accepted persons become independent and the clients referred to JP have fewer barriers to employment than TFJ clients. In addition, JP contractors have a longer period over which placed persons can become independent – twelve months compared to six months for TFJ contractors.
It is understandable that the Ministry set a high target success rate for TFJ because TFJ contractors are paid for each person they accept and train. The Ministry did not want TFJ contractors to train large numbers of people who would not become employed quickly as a result of that training.24 In practice, many of the clients accepted into TFJ have faced a wide range of barriers to employment, not just lack of occupational skills. Although contractors have adapted their programs to respond to this challenge, it is not surprising that the program has not achieved the independence targets established when the program was conceived.
CHARACTERISTICS OF THE FIRST JOB
Exhibit 3.5 summarises the average wage, average number of hours and average earnings of clients in their first job under TFJ. It shows that the average wage is higher than the minimum wage and that average earnings are larger than the benefits that would be paid by BCEA in most circumstances. The Exhibit also shows that the starting weekly earnings are, on average, significantly lower for women than for men. The average starting wage rate, and average weekly earnings, for both men and women under TFJ is lower than for first jobs under JP. More than 20% of the first jobs pay less than $200 per week because they only provide part time hours. The percentage of part-time employment is much greater for women than for men.25
EXHIBIT 3.5 CHARACTERISTICS OF THE FIRST JOB
Women Men Both Hours Per Week 27.9 33.1 30.1 Hourly Wage Rate $9.18 $9.98 $9.52 Weekly Earnings $257 $336 $291
23 The 40% figure is derived from the fact that 75% of accepted persons become job ready and 53% of job ready clients achieve independence. 24 However, there is no financial penalty in the TFJ contract for training people who are not placed. 25 The male/female distinction is true for single persons and single parents.
21
The majority of jobs for women are in sales and business related occupations. For men, however, employment in trades is the largest grouping, with sales as the second largest group. For 54% of placed clients their first job obtained is in the same occupational category as the training provided.26 This figure is similar to the information generated through the client survey. Therefore, for many clients the occupational skills learned in the training period may not have been directly relevant to the first job obtained. Nevertheless, the training process may still have helped the client become more employable.
PROGRAM IMPACT
Like JP, we have chosen to estimate program impact by comparing the performance of persons accepted into the program to persons referred but not accepted. The two groups are roughly similar in size. The ‘not accepted’ group includes persons who did not show-up for an interview with a contractor.27
Exhibit 3.6 shows the extent to which accepted and not-accepted persons are still on BCEA in the months following the month of first referral. At first, accepted persons tend to be slower in leaving BCEA. This is to be expected because they are engaged in training. Over time, however, accepted persons begin to leave BCEA faster than those not accepted. After 14 months, 48% of accepted TFJ clients are still on BCEA compared to 57% of not-accepted clients.
Despite this trend, the not-accepted clients have accumulated more months of independence thus far. After 14 months, persons accepted into TFJ have achieved an average of 4.1 months of independence. In contrast, persons not-accepted have achieved an average of 4.4 months of independence. Therefore, to date the program has not generated incremental months of independence or savings in BCEA payments for the Ministry. However, if the trend shown in Exhibit 3.6 continues, we expect the program to begin to demonstrate a positive impact after 21 months (before payments to contractors and other Ministry costs are taken into account).
26 The 3 digit NOC categories used by the Ministry are very broad. Therefore, we cannot be more specific about the linkage between training and occupation. 27 We have developed impact estimates using other comparisons and have generated similar results.
22
EXHIBIT 3.6 COMPARISON OF INDEPENDENCE ACHIEVED BY ACCEPTED AND NOT-ACCEPTED CLIENTS
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14Months since Referral
% o
n B
CEA
Not Accepted
Accepted
PROGRAM COSTS
Payments to contractors are made up of a training payment and placement payment. Seventy five percent of the training payment is a fee for service and is paid to the contractor over the course of training. The balance of the training payment (25%) is performance-based and is paid only after a client successfully completes the training. The placement payment is all performance-based and is paid in full only if a client achieves 12 months of independence in the time allowed.
Both the payments vary from contractor to contractor depending on the length of training program and the cost of service proposed by the contractor. On average, the training payment is $4,332 for each client that successfully completes the training. The placement payment, on average, is $2,634 for each client that achieves 12 months of independence. Therefore, the maximum payment per fully independent client averages $7,000.
In addition to the payments for persons who achieve independence, the cost of the program includes payments made to contractors for clients who are trained but do not achieve independence and for people accepted that do not complete the training.
23
Together, these represent 35% of program costs. Therefore, the cost of the program to achieve a fully independent client is not $7,000 but is closer to $10,770 per independent client28. Total payments to the contractors for persons accepted in the first year of the program are expected to reach $25 million.
It is too early to forecast the longer term financial impact of TFJ on the Ministry with any certainty. However, at this time, it seems likely that it will take longer for the program to reach the break-even point (where savings associated with the program match program costs) than the 6 -7 years projected for JP2.
28 $10,770 = $7,000 (direct costs) + $3,770 (costs associated with training persons who do not become independent); $3,770 is 35% of $10,770.
24
CHAPTER FOUR: PRINCIPAL FINDINGS
THE JOB PLACEMENT PROGRAM
Introduction
The initial design of the Job Placement Program (JP) was based on three central assumptions:
• Many people on the BCEA caseload have the skills necessary to start and retain work. They do not lack occupational experience but are having difficulty finding work because they have exhausted their job leads, or are not being interviewed for job openings or are not being successful at those interviews.
• Many job opportunities are not advertised. Often, BCEA clients do not have the same access to these opportunities as other job seekers because they do not have a strong network of personal contacts, or they lack recent job experience or they face significant job search barriers such as lack of transportation or suitable work clothing.
• Having found a job, BCEA clients may find it difficult to keep the job because they do not have the personal support network and financial capacity to weather difficulties that arise at the job site or in their personal life.
Therefore, JP was designed to give ‘job ready’ BCEA clients the help of external job placement specialists who could help them find and keep a job. As noted earlier, when first introduced in January 2000, JP was targeted at clients who were new to or had very little attachment to BCEA. Since 2002, the target group has been widened to include all clients on the Expected to Work caseload. Not all the persons accepted into the program under JP2/X could be described as ‘job ready’.
We have examined JP to see whether it has helped its target client group and how it has adjusted to the changing characteristics of the persons referred. However, we have focused our analysis on the principal evaluation question, “Has JP helped BCEA clients find work faster than they otherwise would and to achieve a longer period of sustained independence from BCEA than they otherwise would?” This Chapter summarises our principal findings and makes a number of suggestions about how the program can be improved.
Strengths of the Program
JP has clearly demonstrated that many BCEA clients have the necessary skills and experience to find and keep work. To date, more than 40,000 persons have been placed in employment and achieved independence through the program. In addition, most of the
25
persons placed are achieving substantial months of independence. The program benefits a wide range of clients: single parents as much as single persons; and, persons located throughout the province. The majority of the clients interviewed through the study expressed satisfaction with the service provided by contractors including those who were not placed by the program.
The program has proven to be very adaptable:
• Contractors are paid on the basis of outcomes. How they achieve those outcomes is largely up to them. Each contractor has their own approach to service delivery. Moreover, contractors can and do tailor services to the needs of individual clients.
• Contractors provide supports (e.g. for transportation and work clothes) that reflect the specific needs of individuals and, because they are funded by the contractor, the contractor’s own assessment of the value of making the payment.
• Contractors have demonstrated an ability to modify their programming to match the changing profile of persons referred to the program. Even though contractors have been accepting a larger proportion of persons with a history of attachment to BCEA, the program has continued to place an average of 810 clients per month since mid 2002.
• Contractors have demonstrated that they can provide a ‘virtual’ service from a remote location in towns where it would not be efficient to have a service provider.
Overall, clients accepted into the program are achieving more months of independence than they otherwise would, had they not been in the program. The difference is not large: the average increment is less than half of one month of independence for each accepted person when measured 21 months after referral to the program. However, the increment is being sustained and is growing over time.
Weaknesses of the Program
A large numbers of people referred to JP do not benefit from the program. Approximately 25-30% people referred to the program either do not show up for interview or are not accepted by the contractors. More than 40% of people accepted by contractors do not achieve independence within the program timelines. Clearly, therefore, some ETW clients need more assistance than has been provided through this program.
While clients are satisfied overall with the service provided, less than half the clients interviewed agreed that the job leads provided were helpful in getting a job. This was true of persons placed by the program as well as those not placed. This finding reflects the fact that many of the people placed could have found work even without any help from the contractors. The contractors have a financial incentive to accept the most employable people even if they need very little assistance.
26
The process of referring persons to JP has been heavily influenced by the desire to meet the referral targets that are part of the contracts with service providers. As a result, some persons are being referred simply to ensure that referral targets are met, not because they are suitable for the program. This may be inefficient because clients could find work by themselves or it may be counter-productive because JP providers do not provide the full range of services needed by more challenged clients.
The program involves a number of time limits (e.g. how long a contractor can assist a client to become independent or earn months of independence) that are sensible from the point of view of overall program design but may not always serve the best interest of individual clients. While the Ministry can provide some limited flexibility in individual circumstances, the competition among contractors for clients requires the Ministry to be consistent and even-handed in its application of the program’s business rules.
There is duplication in the process of interviewing and assessing clients. Ministry staff interview and assess BCEA clients when they establish and update employment plans. A client may be interviewed and assessed by more than one JP contractor and if unsuccessful with JP may be interviewed and assessed by other program providers (e.g. TFJ). This duplication of effort stems, in part, from a lack of continuity in case management. When a client is accepted by a JP service provider, that contractor becomes the case manager but only for the period of time dictated by the program. They do not have any ongoing responsibility or obligation to the client. The Ministry has a continuing high level responsibility for case management but depends on the contractors to ensure that clients are fulfilling their job search obligations and are receiving the supports required to assist them with job search. It is not clear that having a multiplicity of case-managers serves the best interest of the Ministry or the client.
While there are clear benefits in having the program operated by external contractors who have close links with the employer community, the Ministry has to invest considerable staff time and resources in the management of JP contracts and responding to the concerns of contractors.
It is unlikely that the Ministry’s savings in BCEA payments will exceed the cost of the program for some time. In this respect, actual performance falls well below some of the more optimistic expectations for the program. However, actual performance of JP reflects the inherent difficulty in designing an employment program that would pay for itself. The difficulty is one of designing a process for identifying, in advance, the individuals who would benefit from the program and, thereby, not investing resources in persons who are unlikely to benefit.
The Design of the Performance-Based Contract
From the beginning, JP has been implemented through ‘performance-based’ contracts with external service providers. Contractors are paid for results not for service. In principle, this is a good idea and an improvement over paying contractors simply for providing a service, whether or not that service makes a difference. However, the current
27
design of the ‘performance-based’ contracts does not guarantee that the program will generate benefits for the clients or the Ministry.
Since contractors can select which clients to accept and can decide how best to help those clients, they have every incentive to target their efforts to the most employable client group. As a result:
• Contractors are not obliged to provide all the help needed to place everybody they accept.
• Contractors are paid even where clients need little or no assistance in finding and keeping work.
• The size of the performance payment is not linked to the difficulty of the task of placing a person and helping them stay employed.
• Contractors would still be paid even if the program showed no incremental benefit, on average, for persons accepted into the program.
Therefore, the ‘performance-based’ concept implemented through JP has not overcome the difficulty of designing a ‘self-financing’ employment program.
In addition, because the contractual arrangements focus on performance as measured by independence, they do not ensure that the Ministry receives information on other aspects of the work of contractors that would be useful to them in assessing the value of the program and the suitability of program design. For example, the Ministry is not regularly informed on: which types of service intervention work and which do not; the costs actually incurred by contractors in delivering service; and, the barriers to employment faced by clients who are accepted by contractors but not placed by them.29
Suggested Modifications
Overall, the program has shown that it can provide a benefit to some ETW clients and only the Ministry can decide whether those benefits are worth the financial cost. However, if the program is continued, we suggest that the following modifications be considered:
• The structure of the program should be redesigned to allow JP contractors to help a wider range of clients.30 Under today’s program, contractors have the discretion to invest modest amounts in training JP clients (e.g. introductory hospitality first aid programs). Contractors do not make significant training investments because they cannot recover large training costs from the current schedule of payments.
29 The Ministry has also not been receiving timely information on whether clients are meeting their employment obligations under BCEA (e.g. actively seeking work). However, changes are being made that will improve the flow of information. 30 In saying this we are not suggesting that the opportunity to provide service be limited to existing contractors.
28
Contractors should be able to access additional training funds where it can be demonstrated that clients would benefit from significant occupational or life-skill training. In this way, contractors would be more able to help persons facing significant barriers to employment and would be more adaptable to any changes in the characteristics of the caseload.
• If, as seems likely, the number of persons on the Expected to Work caseload continues to decline, it may be useful to reduce the number of contractors competing for referrals in some areas of the province and to have contractors accept and work with a wider range of clients. In the less populated areas, it may not be possible for contractors to sustain a viable service based on Job Placement alone.
• In keeping with the previous two suggestions, the Ministry should consider combining different types of assistance that are currently offered under different programs and by different contractors (see the later section on TFJ). Contractors would be expected to tailor the services provided to the needs of individual clients. They would be expected to work with those clients until they achieved independence or, in the case of more barriered clients, suitable employment goals.
• Since a number of BCEA clients can find work quickly on their own, the Ministry may wish to expect a longer period of independent job search before a person is referred to JP. This would allow the Ministry’s limited employment dollars to be focused on those who need assistance.
• The referral process should be changed to ensure that referrals are determined by the best interest of clients not the need to meet contractual referral obligations. Contractual referral targets could be replaced with a range of expected referrals. However, in locations where more than one contractor operates, the Ministry should continue to ensure that referrals are fairly allocated among the contractors.
• The concept of a performance based contract should be retained. However, the structure of the performance incentives in the contract should be fundamentally redesigned. Contractors are currently paid on the basis of independence achieved by some individuals. However, success of the program can only be measured by the incremental performance of all persons accepted into the program. The performance targets in the contract should match the performance objectives of the program. If this suggestion was adopted, a number of the time limits in the program could also be adjusted.
• The amount paid to contractors should reflect the difficulty of the task faced by contractors. For example, contractors should not be paid the same amount for persons with very different degrees of attachment to BCEA.
• Since it is difficult to design a performance-based payment system where clients have very different training needs, any training component added to JP would have to be a combination of a fee for service and a performance payment.
29
• The Ministry should consider paying contractors a fee for client assessment and case management services. This would provide some compensation to contractors for the time and effort invested even if a client is not successful in achieving independence. In return for this payment, contractors would be expected to consistently track the employment plan, the services provided and the progress of clients toward employment and independence. This information would be made available to the Ministry should the client prove not to be successful with the contractor.
• The Ministry should require contractors to provide regular monitoring reports on services provided and routine financial statements that clearly identify the costs associated with providing the service.
Since JP cannot be considered in isolation from other employment programs, the Ministry will have to decide which of these suggestions are most appropriate in the context of expected changes in the caseload and the role of other programs.
THE TRAINING FOR JOBS PROGRAM
Introduction
The Training for Jobs Program (TFJ) was based on the three principal assumptions:
• Some BCEA clients are not immediately job ready. However, relatively short periods of occupational training will substantially improve their prospects of job placement if the training provided is clearly associated with job openings in their community.
• To ensure a strong connection between training services and the job market, the contractors responsible for providing the training should also be responsible for helping clients find work.
• To minimise expenditure of training funds on persons who are not suitable for that training or are not likely to be successful in finding work using the skills learned, the contracts with service providers should have performance-based components.
As implemented, the program targets clients who have not been successful with JP or who are clearly not ready for JP. The service is provided by a wide range of external service providers offering a variety of training options. The job placement component of the program is very similar to JP, although the business rules are different. Compensation for the training component is a combination of a fee-for-service and a performance payment for successful completion. Compensation for the placement component is all performance-based.
Like our approach to JP, we have focused on the impact of the program on the client group and, in particular, on its success in helping those clients to achieve sustained independence more quickly than they otherwise would. Because TFJ was introduced
30
more recently than JP and because it takes some time for clients to complete their training and find work, there is less information about the impact of the program on independence than there is for JP. Therefore, the results of the evaluation cannot be considered definitive. Also, the program uses a number of different training models (e.g. block, continuous, personalised). However, based on the information available to date, it is not possible to conclude that one approach is more successful that the others.
Strengths
TFJ has demonstrated that selective short-term training can help clients with significantly greater attachment to BCEA than JP clients to achieve independence:
• In its first year TFJ, has achieved its training target despite lower than expected referrals. A total of 4,759 clients were accepted into training - and seventy five per cent of these people have completed their training.
• More than 50% of persons accepted into training have been placed into jobs.
• More than 40% of people accepted have achieved one or more months of independence in the time allowed under the program. While this success rate falls below the original program target, that target was very ambitious and the time allowed for contractors to assist clients become independent was short given the background of the clientele.
• On average, clients accepted into TFJ are beginning to achieve more months of independence than they otherwise would. In this respect, the program is starting to have a positive incremental impact.
A wide range of contractors have demonstrated that they can provide placement services as well as training services. Clients report a high level of satisfaction with the services received, including the placement services. Clients do not find the training too difficult and 54% of persons placed report that they are using the skills learned.
Weaknesses
One of the objectives of TFJ was to avoid some of the weaknesses of earlier training programs where people were trained but did not use that training to find work and become independent. By combining the responsibility for training and placement, and by having performance-related payments in both, TFJ was designed to minimise the amount of training provided for training sake. This has only been partially achieved:
• Given the lower than expected referrals and a client group with more barriers than anticipated when the program was designed, contractors have accepted clients who may not be very suitable for the training provided. Without these persons to help ‘make up the numbers’, some contractors would not have been able to offer training to the most suitable clients. However, it is clearly not in the interest of the Ministry or the client to spend money on training that is not beneficial.
31
• A significant number of people are being trained but are not being placed or are not becoming independent in the limited time available to contractors.
A number of contractors have found the program financially challenging. They report that they have had to invest additional resources in order to help a clientele more barriered than expected. Also, they were given less time than JP contractors to help clients become independent and thereby earn payments to cover placement costs.31
The Ministry has found the program administratively challenging given the pilot nature of the program, the large number of contractors involved and the difficulties faced by some of those contractors in sustaining service given the lower than expected number of referrals.
Like JP, it is unlikely that the Ministry’s savings in BCEA payments associated with TFJ will exceed the cost of the program for some time. Because of the larger investment needed, the break-even point will take longer to reach than under JP.
Suggested Modifications
Experience to date with TFJ suggests that it is a useful complement to JP for clients with greater attachment to BCEA. However, it also demonstrates that, even with limited training, the investment needed in more barriered clients can be substantial in order for them to become job ready.
We suggest that some modifications be considered if the program is retained:
• As noted under JP, there may be benefit in having a smaller number of contractors responsible for delivering both programs. The principal contractors would retain overall responsibility for the success of the clients but would have the flexibility to purchase training and/or placement services from sub-contractors as dictated by the needs of clients and the relative cost of sustaining their training and placement infrastructure.
• One of the difficulties with the program is the cost associated with training persons who do not achieve independence (currently 35% of total costs) or who did not need all elements of the training in order to find work. While it is impossible to avoid these costs completely, it may be possible to reduce those costs by changing some the performance payments in the contract to reflect the independence achievement of all persons accepted not just the more successful clients.
• The Ministry should monitor closely the longer term success rates associated with different styles of training as this information becomes available.
31 In the design of the program it was assumed that the clients would be able to be placed more quickly than JP clients because of the close linkages between TFJ contractors and employers and the direct relevance of the training provided to openings in the job market.
32
• The Ministry should require contractors to provide regular monitoring reports on services provided and routine financial statements that clearly identify the costs associated with providing the service.
