Demonstrating the Economic Value of Career Services Bryan Hiebert Vice-president, IAEVG Professor Emeritus, University of Calgary Adjunct Professor, Educational

Demonstrating the Economic Value of Career Services

Bryan HiebertVice-president, IAEVG

Professor Emeritus, University of Calgary

Adjunct Professor, Educational Psychology, University of Victoria

Docent of Education (Research), University of Jyväskyla

Member, Canadian Research Working Group on Evidence-based Practice in Career Development

[email protected]

1

What Counts:

2

Background and Rationale

A challenge by Canadian Policy Makers:

“You haven’t made the case for the impact and value

of career development services”

A research team formed in 2004 to follow-up• The Canadian Research Working Group for Evidence-

Based Practice in Career Development • 10 researchers from 7 universities & 1 foundation

3

State of Practice: Measuring Outcomes

84% of agencies report collecting data• Frequency counts, e.g., number of clients

served/month, number of clients who found employment, number of client action plans created, number of clients who completed programs

• Employment status

4

2005 Study: Agencies & practitioners

What are the 3 most important outcomes you report?1. Change in employment or educational status of

the clientand marginally2. Skill development; financial independence,

connectedness, self-confidence3. Number of clients served4. Client satisfaction5. Programs completion6. Service delivery7. Cost-benefit

5

What outcomes are you achieving that are going unreported or unmeasured?

(From 2005 CRWG State of Practice study)

• Client empowerment• Client skill development

• personal self-management skills• Client increased self-esteem • Client changes in attitudes

• about their future• about the nature of the workforce

• Client knowledge gains• Financial independence• Creation of support networks• More opportunities for clients

These are legitimate areas for

intervention6

Outcomes of Counselling

1. Client learning outcomes• Knowledge• Skills

2. Impact on client’s life• Client presenting problem• Economic factors• Third party factors

+ Precursors•Attitude•Motivation•Self-esteem•Stress•Internal locus of control•Belief that change is possible

Personal Attributes

Intervene between learning outcomes & impact outcomes7

Evidence-Based Outcome-Focused Practice

Input Process Outcome

Need to link process with outcome

8

Definitions

Outcome: Specific result or product of an intervention including changes in client competence, client situation and/or broader changes for the client and/or community

Input: Resources available for achieving outcomes Process: Activities engaged in to achieve

outcomes Intervention: Intentional activity implemented in the

hopes of fostering client change Output: Products produced during the intervention,

e.g., resume, sample cover letter, action plan

9

Outcome-Focused Evidence-Based Practice


Indicators of client change

1. Learning outcomes• Knowledge and skills linked to intervention

2. Personal attribute outcomes• Changes in attitudes• Intrapersonal variables (self-esteem, motivation, independence)

3. Impact outcomes • Impact of #1 & #2 on client’s life, e.g., employment status, enrolled in training• Societal, economic, relational impact

10



Activities that link to outputs or deliverablesGeneric interventions• Working alliance, microskills, etc.Specific interventions1. Interventions used by service providers

• Skills used by service providers• Home practice completed by students

2. Programs offered by school3. Involvement by 3rd parties4. Quality of service indicators

• Stakeholder satisfaction, including students 11



Specific interventions1. Career decision making2. Work-specific skills enhancement3. Work search4. Job maintenance5. Career-related personal development6. Other

12



Resources available1. Staff

• Number of staff, level of training, type of training2. Funding

• Budget3. Service guidelines

• Agency mandate4. Facilities5. Infrastructure6. Community resources

13

Intervention-Evaluation-Service Delivery: Merged Framework

Client Outcomes• Knowledge• Skills• Attributes• Impact

Context:Client Needs

Client Goals

Counsellor Actions

Client Actions

Outcomes

ProcessesInputs(Resources)

Service Delivery• Client flow• Accessibility• System factors• Client satisfaction

14

Quality Service Delivery1. Accessibility

• Regular hours• Extended hours• Physical accessibility• Resources in alternate format• Ease of access, who can access

2. Timeliness• % calls answered by 3rd ring• Wait time for appointment• Wait time in waiting room

3. System requirements• Adherence to mandate• Completion of paper work

4. Service standards• Staff credentials,

competencies, resources

5. Service delivery• Client volumes• Client presenting problems• Number of sessions

6. Responsiveness• Respect from staff• Courteous service• Clear communication

7. Overall satisfaction• % rating service good or

excellent• % referrals from other clients

15

Quality Service StandardsAre all components equally important?

Performance Management System (Ontario)

Three broad dimensions of service delivery success 1.Effectiveness (50%)

a. Participant Suitability (15%)

b. Service Impact (35%).

2.Customer Satisfaction (40%)

a. Customer Satisfaction (15%)

b. Service Coordination (25%)

3.Efficiency (10%)

a. Assisted Services Intake (5%)

b. Information Session/Workshop Activity (5%)

16

Outcome Focused Evidence-Based Practice

Quality Improvement

Resources Client change• Knowledge• Skill• Attribute• impact

Counsellor• Skills• Interventions• Programs

17

Outcome Focused Evidence-Based PracticeDynamic and Interactive

Inputs (Resources)

ProcessOutcome

Intervention, Linking process to outcome

Context:Client Needs

Client Goals

Counsellor Actions

Client Actions

• Knowledge

• Skills

• Attributes

• Impact18


Input Process OutcomeIntervention

= Process + Outcome

What will I do? + How is it working?

Professional Practitioner19

Professional Practitioner(Local Clinical Scientist)

Intervening in a systematic manner• Documenting what you did

Paying attention to what happened• Tracking the effects

Looking for associations between what you did & the effects that happened

Across time and across clients• Acquire ability to make predictions

linking interventions & outcomes Each client is a n = 1 experiment (investigation, exploration)

• Multiple replications provides predictability

20

Professional Practitioner(Local Clinical Scientist)

Approach your practice in a scientific manner• Be clear about the nature of the change clients desire• Be clear about what you will do to meet client goals• Document what you do• Document how well it works

Your own practice becomes your data source for predicting client outcomes

This is a viable, perhaps even preferable, alternative

to RCTs21



Need to link process with outcome

1. What will I do?

2. What are the expected client changes? What do I expect clients to learn? What sorts of personal attributes do I want my

clients to acquire? What will be the impact on their lives?

3. How will I tell?22

Evidence Policy makers can relate to

Concepts under development Return on investment Employment Equivalence

(Career Self-Sufficiency Index)

Food for thought and discussion

23

Return on investment: Community Agency

A community agency Career development services for welfare recipients to help them integrate into the labour marketGovernment investment was about $1,300 per client Return on investment came from two sources

• clients who found employment and were no longer on welfare, earned higher income, paid income tax

• service providers employed to deliver the program

Clients provided a copy of their pay stubs before and after the program Return on investment was between $1.14 and $1.46 for each $1.00 spent (times number of years employed)

24

Return on Investment for High School Career Education Programs

School funding is based on student enrolment (person-courses)

2 years after implementing career education program• Completion rates increased by 15%• number of students in their Registered Apprenticeship

Program increased

Increased funding provided • 1.5 additional staff (1 counsellor + .5 support staff)• more preparation time for teachers• perceived more positive work climate

25

Career Self-Sufficiency Index (Employment Equivalence)

Consider a client who receives careers guidance and• Decides to return to school so he can • Find a better job that pays more money and

less likelihood of unemployment Employment status does not change

• Considered a failure

26

Career Self-Sufficiency Index (Employment Equivalence)

Consider insteadIn Canada, men 30 years old are 34% more likely to be employed if they have high school education(compared to men with no high school diploma)

• Employment Equivalence (CSSI) for taking training is .34Consider also, men 30 years old who have high school education, earn on average $6,000 more money per year

• Return on investment = $6,000 times years worked, perhaps 30 years = $180,000

This is evidence of success

27

Return on Investment for Post Secondary Student Services

Post secondary leavers vs. completers 50% more likely to have difficulty keeping up with

the work load • CSSI = 0.50, for completing a study skills program

leavers reported being unsure of what they wanted to do,

#1 reason for leaving school was “lack of fit.” Completers were 45% more likely to report having

a career plan that was a good match for their program• CSSI = 0.45 for completing a program that helps

increase fit between career plans and program of study 28

Return on Investment for Post Secondary Student Services

Post secondary leavers vs. completers (continued…) PSE graduates earn on average $5,512 more than

those who do not graduate Return on investment for completing study skills

programs would be .50 x 5,512 = $2,756 per person per year

Return on investment for completing programs that promote congruence between students’ career plans and their course of study would be .45 x 5,512 = $2,480 per person per year

29

Applied Career Transitions Program(on-line program for unemployed university grads)

For Module 1

• All together there were 10 (items) x 29 (participants) = 290 ratings• Pre: 144 Unacceptable Ratings – Post: 3 Unacceptable Ratings • Unacceptable Ratings decreased from 50% to 14%• Pre: 6 Exceptional Ratings – Post: 130 Exceptional Ratings • Exceptional Ratings increased from 2 to 44% of the participants 30

Results: Impact Outcomes

Employment status• 27 out of 29 were employed• 90% employment rate

Quality of job• 13 of the jobs lined up well with career vision• 48% of jobs were a good fit with career vision

31

Attribution for ChangeTo what extent would you say that any changes in the ratings on the previous pages are a result of your participation in this research project, and to what extent were they a function of other factors in your life?

mostly other

factors

somewhat other

factorsuncertain

somewhat this

program

mostly this

program

ACT 0 0 0 10 19

LMI-Assisted 0 1 4 19 42

LMI -Independent 3 2 11 28 38

Program

32

Building cause and effect cases

We have data on the process used• Counsellor adherence to program• Client engagement in program

We have data on the outcomes• KSAs: Knowledge, Skills, Personal Attributes

We have data on the impact• Employment status

We have economic data• Career Self-Sufficiency Index (Employment Equivalence)

We have a clear link between process and outcome

33

Possible Career Self-Sufficiency Index (Employment Equivalence)

ElementEmployment equivalent

Take further training in institution with student counsellors on staff

0.34 for each year

Take further training in institution with no counsellors on staff

0.25

Complete career guidance program 0.45

Complete Job Finding Club 0.80

Complete ACT 0.9034

Future Possible Directions

35



Completes career program with modest self-confidence

0.60

Completes career program with good self-confidence

0.70

Completes ACT 0.90

36

Question to ponder

Is it logical that a Career Self-Sufficiency Index Employment Equivalence could be greater than 1?

If the goal is employment, job = 1 A good job with prospects for permanency and

advancement should contain a bonus Consult tables of labour turnover for various

occupations. • Turnover for labourer might happen every 6 months• For other categories it might be, say, 12 months• People getting low level jobs would get an equivalent

value of 1 and the latter an equivalent level of 2.

What do you think of this idea?37



Obtains job in firm with fewer than 20 workers

1.0

Obtains job in firm with more than 500 workers

1.25

Job obtained in unionized firm Add 0.25

38

The Problem

Agency managers and counsellors agree that evaluation of services is important

BUT Counsellors do not evaluate their work with

clients in a way that permits making a connection between • what counsellors do and • the client changes that take place.

Perhaps these ideas will help integrate evaluation into service delivery

39

Professional Identity: What we do defines who we are

Most practitioners define their job as delivering services• So … they do not evaluate

the impact of their services on clients

What is career development all about? The answer needs to include BOTH process and outcome

• What will I do to facilitate client change?

+• How well is it working?

Answers need to be a negotiated consensus between practitioners and clients

40

What have we learned?

From PractitionersStructure and checklists are foreign at first

• But later help them to be more focused

Service providers are willing research partners• Most said they would do it again if given the opportunity

Service providers are happy to follow procedures that result in meaningful evidence of client change

From ClientsStructure and timelines motivate action and a sense of progress

Giving clients hands-on tools is motivating

41

42

Demonstrating Value

It is really, really unfortunate when … There is an excellent program

That everyone knows is working Which is filling an important need

but The program is cancelled because

there is no evidence to support the positive claims

To demonstrate value, we need to develop

Culture of evaluation: We need to reach the state where• Identification of outcomes is an integrated part of

providing services Without efficacy data, career services are vulnerable It is in our best interest to gather evidence attesting to

the value of the services we provide• Measuring and reporting processes and outcomes is

integrated into practice• Outcome assessment is a prominent part of counsellor

education• Reporting processes and outcomes is a

policy (and funding) priority

This needs to be a priority in all sectors 43

Don’t worry about getting it right, just start and improve it as you use it

1. Small steps are OK

2. Several small steps = one BIG STEP

3. Share your success stories • with the people who need to hear them, • in language they can understand

4. Be persistent

5. Build support for yourself

44

45

Give Up

Don’t Ever

Discussion

1. What do you think of this idea?

2. Would general evaluation model work for you?

3. How could you use employment equivalence in your work?

4. Other … questions, comments, suggestions?

[email protected]

46

Demonstrating the Economic Value of Career Services

What Counts: Accountability, Evaluation, and Service Delivery

Intertwined

Bryan HiebertVice-president, IAEVG

Professor Emeritus, University of Calgary

Adjunct Professor of Educational Psychology, University of Victoria

Docent of Education (Research), University of Jyvaskyla

[email protected]

47

Documents

Demonstrating the Economic Value of Career Services Bryan Hiebert Vice-president, IAEVG Professor Emeritus, University of Calgary Adjunct Professor, Educational