New Ways to Evaluate Counselling Effectiveness Bryan Hiebert
Adjunct Professor Department of Educational Psychology &
Leadership Studies University of Victoria Professor Emeritus in
Education University of Calgary 1
Slide 2
2 What is counselling all about? Counselling or Counselling
Psychology Clinical Psychology Social Work etc. 1.Write down the
words or phrases that come to your mind 1 minute
Slide 3
Overview 1.Background 2.An alternate to RTCs 3.A framework for
demonstrating value 4.Alternate evidence gathering tools Purpose
1.An alternate way of looking at demonstrating the value of human
services 2.Intended to stimulate discussion 3
Slide 4
4 Evaluation is important 1.But counsellors do not evaluate
their work with clients 2.1993 study 40% never evaluated their work
with clients 35% evaluated with client during interview 3.2005
study 33% did not answer 56% reported data Client flow, counsellor
time use, etc.
Slide 5
5 How to address the problem We need an approach that is:
1.Comprehensive enough to include what is needed 2.Simple enough
for people to use 3.Incorporates evaluation into standard practice
Your own practice becomes your data source for predicting client
outcomes Each client is a n = 1 experiment Across time and across
clients, counsellors acquire ability to make predictions linking
interventions & outcomes Local Clinical Scientist (Professional
Practitioner)
Slide 6
6 Evidence-Based Outcome-Focused Practice Input Process Outcome
Need to link process with outcome Resources Actions Client
change
Slide 7
7 Evidence-Based Outcome-Focused Practice Input Process Outcome
Indicators of client change 1. Learning outcomes Knowledge and
skills linked to intervention 2. Personal attribute outcomes
Changes in attitudes Intrapersonal variables (self-esteem,
motivation, independence) 3. Impact outcomes Impact of #1 & #2
on clients life, e.g., employment status, enrolled in training
Societal, economic, relational impact
Slide 8
8 Outcomes of Counselling 1.Client learning outcomes Knowledge
Skills 2.Impact on clients life Client presenting problem Economic
factors Third party factors + Precursors Personal Attributes
Slide 9
9 Precursors Intervene between learning outcomes & impact
outcomes 1.Attitude 2.Motivation 3.Self-esteem 4.Stress 5.Internal
locus of control 6.Belief that change is possible Changes
counsellors mentioned That were being observed but not
reported
Slide 10
10 Evidence-based Outcome-focused Practice Input Process
Outcome Activities that link to outputs or deliverables Generic
interventions Working alliance, microskills, etc. Specific
interventions 1. Interventions used by service providers Skills
used by service providers Home practice completed by students 2.
Programs offered by school 3. Involvement by 3 rd parties 4.
Quality of service indicators Stakeholder satisfaction, including
students
Slide 11
11 Evidence-based Outcome-focused Practice Input Process
Outcome Resources available 1. Staff Number of staff, level of
training, type of training 2. Funding Budget 3. Service guidelines
Agency mandate 4. Facilities 5. Infrastructure 6. Community
resources
Slide 12
12 Outcome Focused Evidence-Based Practice Input Process
Outcome Need to link process with outcome 1.What will I do? 2.What
are the expected client changes? What do I expect clients to learn?
What sorts of personal attributes do I want my clients to acquire?
What will be the impact on their lives? 3.How will I tell?
Slide 13
Outcome Focused Evidence-Based Practice 13 RESOURCES AVAILABLE
Context: Structure of opportunity Staff: Number of staff, level of
training, type of training Funding: Budget Service guidelines
Facilities Infrastructure Community resources ACTIVITIES THAT LINK
TO OUTCOMES OR DELIVERABLES Generic interventions Working alliance,
microskills, etc. Specific interventions Strategies linked to
specific client problems (stress, grief, depression, career, etc.)
Client home practice Other Programs & Workshops Facilitation
guides Intervention manuals External Referral INDICATORS OF CLIENT
(LEARNER) CHANGE 1.Learning outcomes Changes in knowledge and
skills linked to the program or intervention used Progress
Indicators End Result Indicators 2.Personal attribute outcomes
Changes in intrapersonal variables e.g., attitudes, self-esteem,
motivation, etc. Progress Indicators End Result Indicators 3.Impact
Outcomes Changes in the clients life resulting from application of
learning OUTCOMES INPUTS PROCESSES Client Context Needs Goals
Slide 14
14 Evidence gathering 1.A Framework for evaluation Learning
from the evaluation experts 2.Gathering evidence of our impact
Documenting what works and why 3.Developing evidence gathering
tools Need to use informal evidence Some examples
Slide 15
15 Assessment as Decision Making (vs. Judgement) Please use a
two-step process 1.Would you say that your level of mastery of the
attribute under considerations is 1.Then assign the appropriate
rating 0 = really quite poor 1 = just about OK, but not quite 2 =
OK, but just barely 4 = really very good 3 = in between barely OK
and really good acceptable unacceptable 0 123 4 4 0
Slide 16
Problem with skill self-assessment 1.Participants asked to rate
their skill (or knowledge) before and after a program 2.Often,
pre-workshop scores are high and post-workshop scores are lower
People find out as a result of the workshop that they knew less
than they thought or had less skill than they thought Based on the
new awareness, post-scores are lower 3.People dont know what they
dont know 4.How can we get around this problem? 16
Slide 17
17 Assessing Learning & Attribute Outcomes Post-Pre
Assessment 1.We would like you to compare yourself now and before
the workshop. Knowing what you know now, how would you rate
yourself before the workshop, and how would you rate yourself now?
2.Please use a two-step process: Decide whether the characteristic
in question is acceptable or unacceptable, then assign the
appropriate rating acceptable unacceptable 0 123 4 4 0
Slide 18
Applied Career Transitions Program (on-line program for
unemployed university grads) For Module 1 All together there were
10 (items) x 29 (participants) = 290 ratings Pre: 144 Unacceptable
Ratings Post: 3 Unacceptable Ratings Unacceptable Ratings decreased
from 50% to 14% Pre: 6 Exceptional Ratings Post: 130 Exceptional
Ratings Exceptional Ratings increased from 2 to 44% of the
participants 18
Slide 19
Results: Impact outcomes Module 1 23 out of 29 had found a job
10 of the jobs lined up well with career vision Module 2 4 out of 6
had found a job 3 of the jobs lined up well with career vision
19
Slide 20
Post-Pre Assessment: Bridging Health Care with Self-Care
1.Knowledge of stress and stress control 2.Knowledge of how to
manage personal change 3.Knowledge of nutrition and nutrition
control - - - - - - - 1.Level of stress 2.Level of nutrition
(high=healthy) 3.Level of fitness 4.Confidence in ability to manage
personal change 20
Slide 21
21 Participant Self-Assessed Change Stress & Stress Control
KnowledgeLevel Response categoryStartFinishStartFinish very
low17%0%4%9% somewhat low30%0%4%35% neither high nor low26%0%13%26%
somewhat high21%70%39%26% very high4%30%39%4%
Slide 22
Post-Pre Results Bridging Health Care with Self-Care Stress
& Stress Control End of ProgramStart of Program Vertical axis
indicates % of participants 22
Slide 23
Post-Pre Results Bridging Health Care with Self-Care Manage
Personal Change Start of ProgramEnd of Program Vertical axis
indicates % of participants 23
Slide 24
Attribution for Change 1.To what extent would you say that any
changes depicted above were a function of participating in our
program, and to what extent were they a function of other factors
(e.g., job, personal circumstances, etc.) mostly other factors
somewhat other factors uncertain somewhat this program mostly this
program 24 0% 9%26%57%
Slide 25
Participant Comments 1.This program made me realize that change
is possible for me to make, and that even a small effort can bring
significant benefits. When I joined the program, I thought that
only a superhuman effort would result in any positive benefits, and
that thought itself was a barrier to making a behavioural change.
Now I know that I can do it and that knowledge itself is half the
battle 2.This is a fantastic program. The biggest benefit for me is
that I am starting to see myself as being a more active person,
rather than a couch potato. I appreciated the focus of the program
being on life change and wise choices, rather than on diet and
weight loss. I 25
Slide 26
Tracking client goal attainment 1.Clients are different Live in
different contexts have different goals require different
interventions 2.How can we consolidate the results? 26
Slide 27
27 Goal Attainment Scaling 1.Much better than expected 1.A bit
better than expected 1.About what I expected 1.A bit worse than
expected 1.Much worse than expected Observable Indicators +2 +1 0
-2
Slide 28
Thought Listing Stimulus question 1.For Clients: What are the
main factors contributing to your presenting problem? What needs to
happen in order for you to achieve your goals? What are the
barriers you need to overcome? For counsellors: What is counselling
all about? What factors are responsible for client change? 2.Write
down all the words or phrases that come to mind in response to this
question. 28
Slide 29
29 Cognitive Mapping Task 1.Complete the thought listing task
2.Transfer each item to a Post-it sticker, 1 item per sticker.
3.Arrange the stickers on a large sheet of graph paper Place
related concepts close to each other 4.Draw connecting lines
between stickers that represent related concepts 5.Draw a circle
around any clusters of concepts label each circle
Slide 30
30 Cognitive Mapping-Example What is counselling all about?
Scoring scheme: http://cmap.ihmc.us
Slide 31
Relaxation Monitoring Sheet Date/Time Indicator
StartFinishDifference pulse rate (beats per minute) breathing rate
(beats per min.) finger temperature (degrees) Feelings, body
sensations, etc. Thoughts, perceptions, imagery, etc.. 31
Slide 32
Informal Evaluation of Headache, Pain, and Related Affective
States 32
Slide 33
Self-Monitoring Headache 0 -No headache 1 -Low level, only
enters awareness when you think about it 2 -Aware of headache most
of the time, but it can be ignored at times 3 -Painful headache,
but still able to continue job 4 -Severe headache, difficult to
concentrate with demanding tasks 5 -Intense incapacitating headache
33
Headache: Indicators of Success MeasureBeginningEnd Average
headache level4.10.6 Number of intense (level 4 or 5 ) headaches 30
Hours of intense (level 4 or 5 ) headache 130 Hours headache
free010 35
Slide 36
Self-Monitoring Stress 0 -No stress 1 -Low level, only enters
awareness when you think about it 2 -Aware of stress level most of
the time, but still able to stay focused on job 3 -Very stressful,
beginning to interfere with what you are doing 4 -Severe stress,
creates major interference with what you are doing 5 -The most
stress you ever feel 36
Slide 37
37 Self-Monitoring Confidence 0 - No confidence at all in your
ability to handle the situation 1 - Low confidence, but you might
get lucky and it will turn out OK 2 - Sort of confident, you think
you might be able to handle it, but probably not 3 - Pretty
confident, you think you have a chance of this working out OK 4 -
Very confident, youll probably be able to pull this off 5 -
Completely confident about what you are doing
Slide 38
38 Self-Monitoring Job Satisfaction 0 - No satisfaction at all
with your job 1 - Somewhat satisfied, there are some things about
your job that are OK 2 - Satisfied enough as jobs go, this one is
as good as any 3 - Mostly satisfied, occasionally there are things
about your job that you actually like 4 - Very satisfied, but there
are occasional things bother you 5 - Completely satisfied with your
job, feeling a sense of self-fulfillment
Slide 39
39 Self-Monitoring Motivation 5 -Very motivated-it's the most
important thing for me to do today 4 -Very motivated, but something
might come up to interfere 3 -Quite a bit motivated, I think I will
end up doing some of it 2 -I sort of care and I might get around to
doing some work today 1 -If I run out of things to do, I'll try
looking in some of my books 0 -I couldn't care less if I ever did
my school work
Slide 40
Using Self-Talk to Control Anger With cognitively-challenged
adults in residential settings Hiebert, B., & Malcolm, D.
(1988). Cognitive strategies for mentally handicapped clients. In
B. O'Byrne, (Ed.). Natcon-14 (pp. 289-298). Toronto, On: Ontario
College Counsellor's Association. (Reprinted in Resources In
Education, June 1989, ERIC Document Reproduction Service No. Ed
302800) 40
Slide 41
41 Picture Stimulus: Pushing
Slide 42
42 Picture Stimulus: Hitting
Slide 43
43 Picture Stimulus: Kicking
Slide 44
44 Interview Protocol for Assessment 1.I am going to show you 3
pictures and ask you some questions about each picture. 2.Look at
this picture. 3.Tell me how many people are in the picture. 4.What
are the people doing in the picture? 5.Point to the person that is
being pushed/kicked/hit 6.If you were the person being
pushed/kicked/hit, how would you feel? 7.If you were the person
being pushed/kicked/hit, what would you feel? 8.If you were the
person being pushed/kicked/hit, what would you think? 9.If you were
the person being pushed/kicked/hit, what would you do?
Slide 45
45 Interview Scoring Sheet StatementFeelingThinkingAction Anger
Non anger Non facilitative FacilitativeAggressive Non aggressive 1
5 6 how would you feel? 7 what would you feel? 8 what would you
think? 9 what would you do?
Slide 46
46 Field Test Results: Picture Stimuli Test time Group Response
Category FeelingThinkingAction Anger Non anger Facilitative Non
facilitative Aggressive Non aggressive Pre test Male10196889
Female13 63146 Total233212112215 Post test Male818150020
Female1016110215 Total1834260235
Slide 47
Frequency of Anger Outbursts 10 Mentally Handicapped Residents
47
Slide 48
48 Monitoring Your Self-Talk ConPro
Slide 49
49 Changing Your Self-Talk ConPro
Slide 50
50 Self-monitoring Stress Subjective Units of Disturbance
Slide 51
51 Self-monitoring Stress Subjective Units of Disturbance
Slide 52
52 Self-monitoring Stress Subjective Units of Disturbance
Slide 53
53 Emotional Thermometer
Slide 54
54 Outcome-Focused Evidence-Based Practice Outcomes
LearningPersonalImpact Processes Decision making Skill training
Stress control Self-management Other
Slide 55
Linking Process and Outcome We have evidence that the
facilitator followed the program We have evidence about which
participants followed the program more (vs. less) closely We could
compare the results of those who follow the program compared to
those who did not We have evidence of how much change took place We
have a connection counsellor process client engagement outcomes
Participants attribute change to the program A solid link between
program and outcome Impact Learning Attribute 55
Slide 56
Professional Identity: What we do defines who we are 1.Most
practitioners define their job as delivering services So they do
not evaluate the impact of their services on clients 2.What is
counselling all about? 3.Needs to include BOTH process and outcome
What will I do to facilitate client change? + How well is it
working? 56
Slide 57
57 To demonstrate value, we need to develop Culture of
evaluation: We need to reach the state where Identification of
outcomes is an integrated part of providing services Without
efficacy data, counselling services are vulnerable It is in our
best interest to gather evidence attesting to the value of the
services we provide Measuring & reporting outcomes is
integrated into practice Informal data are acceptable Reporting
outcomes is a policy priority Outcome assessment is a prominent
part of practitioner education This needs to be a priority in all
sectors
Slide 58
Multiple Effects of Relevant Evaluation Practices 1.On client
Increased motivation Increased awareness of effectiveness How am I
doing at this? 2.On counsellor Increased effectiveness Increased
confidence Increased motivation Increased job satisfaction 3.On
agency Value of services 58
Slide 59
59 What is counselling all about? Counselling or Counselling
Psychology Clinical Psychology Social Work etc. 1.Write down the
words or phrases that come to your mind 1 minute
Slide 60
New Ways to Evaluate Counselling Effectiveness Questions?
Comments? Bryan Hiebert Adjunct Professor Department of Educational
Psychology & Leadership Studies University of Victoria
Professor Emeritus in Education University of Calgary
http://www.bryanhiebert.ca 60
