Evaluation:A Necessity for the Profession,

an Essential Support for Practitioners

Linking Intervention with Outcome

Bryan HiebertUniversity of Victoria

University of Calgary

1

Overview: 4 main points

Re-conceptualizing professional practice

1. Expand the scope of accountability

2. A framework for demonstrating value

3. What are the legitimate outcomes resulting from career development services

4. Some specific examples

2

Focus on Accountability

Traditionally, focus was on outcomes• Are the outcomes (results, effects, etc.)

promised, achieved in practice?

Need to broaden the focus …• Are the outcomes achievable, realistic, and

relevant?• What factors contribute to the outcomes?• What interventions make the outcomes more

likely to be achieved?• How do we know that the outcomes are a result

of the interventions?3

4

Input Process Outcome

Framework developed by the Canadian Research Working Group

on Evidence-Based Practice in Career Development

Outcome Focused Evidence-Based Practice

Resources What I Do Effects of what I do

1. Comprehensive enough to include what is needed2. Simple enough for people to use3. The goal is to incorporate evaluation into service delivery

5

Outcome Focused Evidence-Based Practice

Indicators of client change

1. Learning outcomes

2. Personal attribute outcomes

3. Impact outcomes

Resources What I do Effects

6

Outcome Focused Evidence-Based Practice

Indicators of client change

1. Learning outcomes• Knowledge and skills linked to intervention

2. Personal attribute outcomes• Changes in attitudes,• Intrapersonal variables

(self-esteem, motivation, independence) 3. Impact outcomes

• Impact of #1 & #2 on client’s life, e.g., employment status, enrolled in training

• Societal and relational impact• Economic impact

Resources What I do Effects

7

Outcome Focused Evidence-Based Practice

Resources What I do Effects

Activities that link to outputs or deliverablesGeneric interventions• Working alliance, microskills, etc.Specific interventions1. Interventions used by service providers

• Skills used by service providers• Home practice completed by clients

2. Programs offered by agency• To reduce employment barriers• To enhance client employability

3. Involvement by 3rd parties4. Quality of service indicators

• Stakeholder satisfaction

8

Outcome Focused Evidence-Based Practice

Activities that link to outputs or deliverablesSpecific interventions1. Career decision making2. Work-specific skills enhancement3. Work search4. Job maintenance5. Career-related personal development6. Other

Resources What I do Effects

9

Outcome Focused Evidence-Based Practice

Resources What I do Effects

Resources available1. Staff

• Number of staff, level of training, type of training2. Funding

• Budget3. Service guidelines

• Agency mandate4. Facilities5. Infrastructure6. Community resources

10

Outcome Focused Evidence-Based Practice

Resources Client change• Knowledge• Skill• Attribute• impact

Counsellor• Skills•

Interventions

• Programs

Continuous Improvement Framework Quality Improvement of Services

Resources What I do Effects

11

Outcome-Focused Evidence-Based Practice

Resources What I do Effects

Intervention =

Process + Outcome

What will I do? + How is it working?

Professional Practitioner

12

Outcome Focused Evidence-Based Practice

Input Process Outcome

Need to link process with outcome

1. What will I do?

2. What are the expected client changes? What client learning will result? What sorts of personal attributes will the client acquire? What will be the impact on their lives?

3. How will I tell?

13

Evidence-based Practice

1. Research trials (traditional way in psychology)2. Professional Practitioner

Purposeful intervention Data (= evidence) on what was done Evidence on client change Look for patterns in data

linking intervention with outcome Develop scientific attitude toward practice

– Skepticism, curiosity, inquiry

A viable, perhaps even preferable, alternative scientific way to demonstrate effectiveness

Building cause and effect arguments

What counts? (from initial CRWG survey)• Career-life plan• Increased self-confidence • Greater capability to manage career-life plan• Clients taking action as set out in their career-life plan• Services delivered match client identified needs• Use of resources• How closely do service provider and client follow program• How often is career-life plan modified• Change in employment status• Socio-economic impact

How do we document these?14

Some of these require long term follow up

15

Assessment as Decision Making (vs. Judgement)

Please use a two-step process

1. Would you say that your level of mastery of the attribute under considerations is

2. Then assign the appropriate rating 0 = really quite poor 1 = just about OK, but not quite 2 = OK, but just barely 4 = really very good 3 = in between barely OK and really good

acceptableunacceptable

0 1 2 3 440

Problem with skill self-assessment

• Participants asked to rate their skill (or knowledge) before and after a program

• Often, pre-workshop scores are high and post-workshop scores are lowerPeople find out as a result of the workshop that they knew

less than they thought or had less skill than they thoughtBased on the new awareness, post-scores are lower

• People don’t know what they don’t know• How can we get around this problem?

17

Assessing Learning & Attribute Outcomes

Post-Pre Assessment

We would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now?

Please use a two-step process:Decide whether the characteristic in question is

acceptable or unacceptable, thenassign the appropriate rating

acceptableunacceptable

0 1 2 3 440

Promising Practices

1. Several projects have been able (or will be able) to provide a solid link between programs & services and client change

• Provide two examples

2. Encourage you to think about how these ideas could be applied in your workplace

18

Starting Small: An Example In Progress

• Client characteristicDependable

• Indicators (things that tell me a client is dependable)Punctual Follow through on a promise Takes personal responsibility Honest

• What I do to promote client dependabilityProvide focus for client Keep client on track Encourage/reinforce client to follow through

19

Tracking Process and Outcome

Maintaining Focus

Not done

Sort of done

Done well Dependable

Not really

Sort of

Definitely

Provide focus for client Punctual

Keep client on track

Follow through on a promise

Encourage client to follow through

Takes personal

responsibility

Honest

20

A sample checklist

OR use the 5 point decision-making scale described earlier

Applied Career Transitions Program(on-line program for unemployed university grads)

For Module 1

• All together there were 10 (items) x 29 (participants) = 290 ratings

• Pre: 144 Unacceptable Ratings – Post: 3 Unacceptable Ratings

• Unacceptable Ratings decreased from 50% to 14%

• Pre: 6 Exceptional Ratings – Post: 130 Exceptional Ratings

• Exceptional Ratings increased from 2 to 44% of the participants 21

Results: Impact outcomes

Module 123 out of 29 had found a job10 of the jobs lined up well with career vision

Module 24 out of 6 had found a job3 of the jobs lined up well with career vision

In total29 participants27 found jobs13 had jobs within their ideal career vision

22

23

Attribution for Change

To what extent would you say that the changes depicted above were the result of completing Module 1 of the ACT program, and to what extent were they a function of other factors in your life?

mostly other

factors

somewhat other

factorsuncertain

somewhat this

program

mostly this

program

0 0 0 10 19

Linking Process and Outcome

We have evidence that the facilitator followed the program

We have evidence about which participants followed the program more (vs. less) closelyWe could compare the results of those who follow the

program compared to those who did not

We have evidence of how much change took place

We have a connectionfacilitator process client engagement outcomes

Participants attribute change to the program

A solid link between program and outcome

Impact

Learning

Attribute

24

25

Summary

1. Counsellors and agencies think it is important to evaluate their work with clients • BUT they don’t do it

2. They need a framework for integrating evaluation and intervention

• What works and why

3. They need tools and resources for both short term and long term effects

4. We need an evaluation culture

A professional identity that defines our role as BOTH

delivering services & assessing outcomes