Réginald Savard, Ph.D.Université de Sherbrooke

EVALUATION: A PROCESS SUPPORTING SKILLS AND INTERVENTION

DEVELOPMENT

Do you know of many studies aimed at determining… How interventions have been useful to clients? How interventions have contributed to

compensating for their difficulties? How interventions have helped to meet their

expectations and needs? How interventions have improved their future or

social trajectory?

2

Evaluation: The Research

3

1 •Evaluate for whom?

2 •Evaluate by whom?

3 •Why evaluate?

4 •Evaluate what?

5 •Evaluate when?

6 •Evaluate how?Eval

uatio

n

Evaluate for whom?

4

Intervention

EMPLOYER (manager)

COUNSELLOR

CLIENT

Evaluate for whom?Organizations Evaluate skills and performance Evaluate learnings post trainingClients Evaluate effects of interventions (on the basis of expectations and

needs) Evaluate level of satisfactionCounsellors Evaluate effectiveness of interventions Evaluate factors during the intervention process

5

Evaluate by whom?

6

The actors concerned, their motivations, their interests…

Evaluate by whom?

7

The actors concerned, their motivations, their interests…

Evaluate by whom?

8

The actors concerned, their motivations, their interests…

Evaluate by whom?

9

The actors concerned, their motivations, their interests…

Evaluation for CounsellorsA FEW ISSUES

Satisfaction with service offered Job enrichment Work satisfaction

Professional skills Self-esteem, humiliation, shame Feeling of personal efficacy Etc.

10

Who evaluates? Who evaluates whom?

Intervention

EMPLOYER (manager)

COUNSELLOR

CLIENT

11

External evaluator

3 perspectives Internal (self-evaluation)• Counsellors and clients Organizational External

12

Why evaluate? A phrase from the Viennese philosopher Ludwig

Wittgenstein. What he meant, so far as I understand him, was that

any thinking person sees the world like a fly looking out of his bottle.

That's to say that we see the world clearly, but we're limited by confines beyond our control: in the fly's case, the bottle.

In the human's case, our singular experience.

Why evaluate?

13

Why evaluate?

14

15

Why evaluate?To grasp the reality of individuals in constant

interaction with their environment and to understand the relationships within that environment.

To grasp the reality of individuals seeking to meet their needs and the ability of their environment to meet those needs. To evaluate factors related to the individual and those related

to the environment. To evaluate the individual’s ability to meet his/her needs in

his/her environment.

Why evaluate? Through evaluation, counsellors (or the organization)

learn about themselves (or the client), their functioning, the meaning of their results, their processes (Bouvier, 1998).

Evaluation practices encourage counsellors to seek to gain better self-knowledge, particularly through results analysis, by engaging in reflection in order to better understand the processes implemented.

Evaluation makes it possible to understand what they have done, what they are in the process of doing, what they plan to do, and what they will do.

16

Why evaluate?

To situate the person in his/her developmental trajectory in relation to the different spheres that help to explain his/her problem and adaptation, as well to identify his/her needs (Pronovost and Moreau, 2004).

Evaluation is necessary to gain a better knowledge and understanding of the problem situation and to make better decisions about the best intervention (Alfoldi, 1999).

17

Why evaluate? Evaluation reveals the person’s potential for change

and ability to adapt (Pronovost and Moreau, 2004). Evaluation may help to activate a change process with

clients Become aware of their abilities Become aware of their difficulties Formulate new ideas Propose methods never before contemplated Identify worthwhile projects

Why evaluate? Evaluation makes it possible to step back, which affords some

protection for counsellors who are very involved in the problem situation (Alfoldi, 1999).

Evaluation makes it possible to clarify the roles and responsibilities of each person, as well as the intervention targets (Alfoldi, 1999).

According to Alfoldi (1999), evaluation makes it possible to Have a different understanding of the person’s functioning (needs, abilities,

etc.) and sources of obstacles or dysfunction Plan the intervention Develop an action plan.

According to Pronovost and Moreau (2004), Determine the service to be offered.

Why evaluate? Evaluation enables the actors to clarify and

adjust objectives, verify practices, improve action, and help with decision making

Regulation of intervention practices Development of human resources Management of training plan Support for, or piloting of, quality initiatives Definition of an organization’s strategic directions

20

Why evaluate? Evaluation makes it possible to identify the client’s

initial functioning and determine whether that functioning improved following the intervention.

Evaluation is a way of taking a step back from a client’s problem that sometimes makes counsellors emotionally vulnerable.

21

Evaluation: a process

OBSERVER

MEASURER

ANALYZER

Intervention

22

Why evaluate?

This process should make it possible to plan, carry out, and re-evaluate the intervention (Hayes, Nelson and Jarret, 1987).

Evaluate what? Evaluation is a (dynamic) process through

which we prepare (identify), observe, and collect useful information (obtain) and synthesize it in the form of advice or judgements (reviews) for the purpose of disseminating it to others (counsellors, employers, decision makers, etc.). Oriented towards a specific goal (or goals), making it possible to judge possible decisions.

23Stufflebeam et al. 80

Evaluate what? Process (1) through which we define (2), obtain (3) and

provide (4) information (5) that is useful (6) and enables us to judge possible decisions (7)1. process = ongoing activity2. define = identify pertinent information3. obtain = collect, analyze, measure data4. provide = communicate those data5. information = facts to be interpreted6. useful information = that satisfies relevance criteria7. possible decisions = orientation actions, etc.

24Stufflebeam et al. 80

Evaluate what? Evaluating the quality of the product

delivered means asking questions about the criterion that defines that quality.

Quality is meeting client expectations and needs.

Evaluating the quality of the product delivered is based on the idea of querying resources about their ability to meet client expectations and needs.

25

Evaluate what? Relevance of interventions in relation to the client’s

expectations and needs and the organization’s priorities. Effectiveness, i.e., the degree to which interventions were

(or should be) carried out and their impacts Impact evaluation is a systematic identification of the positive or

negatives effects, intentional or not, on different elements generated by an activity, in this case a program or project.

Efficiency, i.e, the degree to which interventions were (or should be) carried out given the resources available.

26World Development Bank

Evaluate what? Evaluation requires selecting the subject of the evaluation

and the most appropriate tool. There are a wealth of tools available, and that should make it

possible to conduct an analysis that is as realistic and dynamic as possible of the person in terms of his/her career path (Birk and Cassereau, 2003).

Wealth of kaleidoscopic pictures (changing and moving).

Evaluation must be able to be argued, justified, and consequently based on a certain number of parameters or variables that can be identified (Steffatre and Pachoud, 2003).

Evaluate what? Combine the subjective and the objective to avoid a

distancing from the actual experience of the clients and the counsellor (Lecomte and Savard, 2003).

According to Patterson (2002), Evaluation forces us to specify the topic to be discussed to

avoid confusion. Evaluation requires systematic logic in the face of reality,

corroboration of intuition through observation. To intuition and logical discovery, add experience and

knowledge through empirical demonstration.

28

Evaluate what? Combine the subjective and the objective to avoid a

distancing from the actual experience of the clients and the counsellor (Lecomte and Savard, 2003).

Evaluation forces us to specify the topic to be discussed to avoid confusion.

Evaluation requires systematic logic in the face of reality, corroboration of intuition through observation.

What are the factors that help to explain the results obtained?

Contribution of counsellor’s characteristics Contribution of client’s characteristics Contribution of techniques

29

Heppner and Heppner, 2000; Swanson, 2002

30

Specific elements: 8%(Techniques)Specific interventions

General factors: 92%(Common factors) Counsellor variables Client variables Working alliance

Evaluate what?

Specific Genéral0

10

20

30

40

50

60

70

80

90

100

Effectiveness

Wampold, 2001

General

31

Evaluate what?

Client variables

Counsellor variables

RelationshipTechniques

Evaluation

of intervention

Evaluate when?

32

Time 1

Before

Time 2

During

Time 3

After

Time 4

3 months6 months12 monthsEtc.

INTERVENTION

Evaluate how?Qualitative approach: evaluation through a

combination of data collection and information processing techniques to determine significance

Quantitative approach: evaluation using standardized toolsSystematic observation gridsQuestionnairesStandardized tests

33

Evaluate how? Standarized tools would be required to guide the

intervention effectively. The criteria for selecting an evaluation instrument are

as follows:It must be flexible.It must not affect the relationship between the

counsellor and the client.It must take into account several factors in order to

reflect the complexity of the intervention.

Beutler and Martin, 2002

Evaluate for whom, by whom, why, what, when, and how?

Evaluation of intervention

InputWho?

ProcessWhat?

ResultsFor whom?

35

Intermediate results

Perhaps the essence of research can be summed up in this quotation from Schopenhauer:

“The task is not so much to see what no one has yet seen but to think what nobody has yet thought about that which everybody sees.”

36

Reginald.Savard@USherbrooke.ca

37