LIST OF TOOLS T€¦ · Creating a logbook 9. Road map 10. Timeline 11. Individual interviews 12. Focus group interviews 13. Consent form Unit 5: Analysing the data 14. Compilation

LIST OF TOOLS Unit 1: Deciding on evaluation 1. Winning conditions for evaluation 2. Who will do the evaluation?

Unit 2: Forming an evaluation committee 2. Who will do the evaluation? (same as unit 1)

Unit 3: Identifying what to evaluate 3. Developing a theory of action 4. Identifying the dimensions to evaluate from a theory of action 5. Choosing a pair of glasses for evaluation

Unit 4: Establishing an evaluation plan and collecting data 6. Establishing an evaluation plan 7. What do I see? 8. Creating a logbook 9. Road map 10. Timeline 11. Individual interviews 12. Focus group interviews 13. Consent form

Unit 5: Analysing the data 14. Compilation matrix 15. Taking stock

Unit 6: Summarizing and reporting the results 16. How to put together an evaluation report 17. How to write a press release

TOOLS

Winning conditions for evaluation

TOOL

1

5

TOOL 1: Winning conditions for evaluation

The winning conditions are in place. We can proceed with our evaluation.

Are we prepared to put time into the evaluation?

We cannot do the evaluation.

Is it possible to conduct an on-going evaluation as

part of our project?

We will have to do the evaluation at the

end of our project.

Do we have the necessary expertise to conduct a

participatory evaluation?

We will need to obtain the support of qualified individuals.

Is our project clearly defined, particularly with

regard to the action principles that guide us?

YES

YES

YES

YES

NO

NO

NO

NO

We will need to do a collective exercise before starting the

evaluation.

Our Answers

Are we ready to question our assumptions?

The evaluation will not be useful.

YES

NO

Tool 1

Who will do the evaluation?

TOOL

2

9

Tool 2

OUR ANSWERS

Do we want to have control over our evaluation?

YES NO

Do we want to conduct our evaluation ourselves? External evaluation: we can still

negotiate our role

Who will work on it? What type of help do we need?

List of evaluation group members

Do we need help collecting data?

Do the work according to the evaluation plan

Ask someone in the community (perhaps pay them) to collect data. We

then proceed with the data collection

NO YES

NO YES

Hire someone to do all the work

under the committee’s supervision

Ask for help (university,

master’s students, local actors, etc.)

TOOL 2: Who will do the evaluation?

Developing a theory of action

TOOL

3

13

A SHARED VISION FOR PLANNING AND EVALUATING COMMUNITY PROCESSES

Paule Simard in collaboration with Ginette Paré and Diane Champagne

Tool developed as part of the

Évaluation d’initiatives de développement avec les communautés 2004-2007 research project

conducted by Direction de santé publique de l’Abitibi-Témiscamingue, Institut national de santé publique, Réseau québécois de Villes et

Villages en santé, Université du Québec en Abitibi-Témiscamingue, and Université Laval, with funding from the

Lucie and André Chagnon Foundation

Tool 3

14

BACKGROUND After many years of supporting community development processes through various research approaches, it has become clear to us that those engaged in these processes often don’t take the time—at the start of or during their process—to develop a shared vision of what they want to accomplish, especially with regard to the values underlying their actions. Generally, they define a long-term goal and immediately identify how to reach it. Rarely do they stop to reflect on their assumptions about change or the principles that drive their actions. This finding led us to put together a workshop enabling groups engaged in community initiatives to clarify their members’ views and collectively develop a theory of action. OBJECTIVE OF THE ACTIVITY Enable a group to develop a shared vision of what it wants to achieve or clarify a project under way with a view to evaluation. WHAT IS A THEORY OF ACTION? Here, the term ―theory‖ means any intention behind the action (how do we understand it? How would we make it happen?). It is constructed from the working assumptions, strategies, and principles that guide the action. It is a combination of the beliefs, values, know-how, and behaviours of those working in the field. An action theory is not an action plan, i.e., an expression of the actions and methods used to achieve an objective, but rather the goals of a process and its underlying action principles. An action theory is developed by establishing common goals with a view to defining a shared vision of what you want to achieve together. WHO WILL FIND THIS USEFUL? Any group engaged in a collective process can define its theory of action. BENEFITS Defining a shared vision helps to

Develop a common language for the project/process and thereby define a team vision based on each member’s points of view

Clarify how you want to work together within the group (and with other partners, if applicable)

Determine the main steps of the collective process Identify means of intervention Provide an evaluation framework reflecting what the group really wants to achieve

ABOUT THEORIES OF ACTION

15

WHAT ARE THE STEPS IN THIS PROCESS? A theory of action is developed in three stages, to which a fourth can be added with a view to evaluation. Workshop 1: Goals of the process This step aims to establish the main goals of the process. The purpose here is not to identify objectives linked to concrete actions, but rather to get a clear overview of what the group wants to accomplish through its collective process. This step should make it possible to identify two or three main goals shared by the group. Workshop 2: Steps to achieve the goals Based on the goals defined in the first step, identify how the group intends to reach them, i.e., the anticipated means and steps the group thinks will be required to get there. This step should allow the group to develop its theory of action. Workshop 3: Validating the theory of action Based on the elements defined in the two previous steps, a graphic model of the group’s shared vision can be developed. In this third step, you can determine whether this model is suitable for your group and make any necessary adjustments. By the end of this step, your theory of action should be complete. However, ―complete‖ does not necessarily mean ―definitive.‖ During the process, different factors (political context, change in partners) may change the situation and impact the process. In this case, it may be best to validate the theory of action again and adjust it as needed. Workshop 4: Identifying the evaluation dimensions The action theory defined by your group provides a shared vision of a project or process. It can also help identify the dimensions you want to evaluate. To do this, draw on this shared vision to determine the elements that are most meaningful and that—if you want to evaluate the action principles—reflect the ways of doing things identified by the group.

16

Objective Define three underlying goals of the theory of action Duration Around two and a half hours Individual preparation Each participant should take some time before the workshop to reflect on his/her personal goals for the group process and list them on the sheet provided for this purpose (Exercise 1). Required material

* Small cards of the same colour, around 8½ x 5½ (plan three for each participant and a few extras)

* Three 8½ x 11 cards of the same colour, but not the same colour as the smaller ones * Markers (at least one per participant); markers will be used for the entire workshop, so

it’s best to use odourless ones * Reusable adhesive * A large, empty wall on which the cards can be affixed

Procedure The times shown here are for groups of ten or less. For larger groups, plan a little more time for steps 2 and 3. Step 1: Present the process or philosophy you want to implement (20 minutes) It may be useful to present or review the main ideas of the approach that drives the actions or that we want to implement (e.g., explain the Healthy Communities or community development approaches, the CSSS community approach, or any other approach favoured by the group). Step 2: Write the goals individually (15 minutes) At this step, all participants must have completed Exercise 1. If anyone has not had time to do so, give them a few minutes to identify their three goals. Next, ask all the participants to list their goals on a card (a small one). Step 3: Present the goals (15 minutes) Once the participants have written down their goals, the facilitator puts their cards up on the wall, then reads out each of the goals. Requests for clarification may be needed to understand a statement, but the discussion does not begin right away. The goals do not need to be identified by author; in some cases it may even be best that the goals remain anonymous.

WORKSHOP 1: GOALS OF THE PROCESS

17

Step 4: Group the ideas (30 minutes) The participants then try to group the goals that are similar or have common elements. The facilitator must make sure that no important elements are overlooked in this process, as sometimes the main idea of a goal is lost when it is grouped with others. Some goals may be very different from one another, in which case they are set aside. Goals can also be reclassified during the process. Break: 15 minutes Step 5: Define common goals (45 minutes) Generally, the preceding exercise sparks a discussion on goals. The objective here is to continue the discussion while at the same time trying to formulate a single goal for each group of goals. The idea is to arrive at a statement that everyone agrees on. Groups usually identify two or three goals, which are then written on the cards provided for this purpose (different colour than those used by the participants). Depending on how much time is allotted, it may be best to have three instead of two, since this enables deeper reflection. Goals that were initially set aside as being too different can be re-examined at this point to see if they should be kept. They may also be considered as action objectives that could be used in developing an action plan. The facilitator guides the discussion, but can also assist in reformulating goals when the group is not making progress. The facilitator’s neutral perspective often helps in reaching a consensus. Note: Workshops 1 and 2 can be held on the same day, depending on how much time the group has and how in-depth they want to go. However, experience has shown that it’s best to leave at least a few days between the two exercises, since this allows the participants to continue their reflection.

18

Objective Identify the steps leading to each of the goals defined in Workshop 1, i.e., the road map for achieving our goals. These road maps are central to the theory of action. Duration Minimum 2½ to 3 hours. Depending on how in-depth the group wants to go and the availability of its members, this exercise can be spread over several half days. Individual Preparation Reflect on the steps required to achieve each of the goals and write them down on a separate sheet for each goal (Exercise 2). Required Material

* Around 50 small cards of the same colour (8½ x 5½, and a different colour than the one used for the following three cards)

* The three cards on which the goals are written (Workshop 1) and a card of the same colour on which ―Starting Situation‖ is written

* Several odourless markers * Reusable adhesive * Large sheets of paper (brown packing paper will work) to cover the wall

Procedure Choose an initial goal to work with. Step 1: Define the steps individually (15 minutes) Check if the participants have filled out their sheets (Exercise 2) before starting. If everyone has completed it, go to Step 2. Otherwise give the participants more time to complete it. Step 2: Share individual points of view (10 minutes) Each participant presents the steps they deem important for achieving the first goal (as written down in Exercise 2). Step 3: Create a collective road map (30 to 45 minutes) Create a map comprising all the steps deemed necessary to achieve the goal. Start with one of the following questions: What is the most important thing to do to achieve this goal? What is the key step in reaching this goal? Write the answers on cards and place them on the wall in what seems to be the most appropriate spot on the road leading to the goal. Generally, the first statement is placed in the middle, but it can be moved after. Proceed with the other statements, placing them in relation to those already identified. As the map comes together, statements can be moved and rephrased.

WORKSHOP 2: STEPS TO ACHIEVE THE GOALS

19

The point of working on a paper-covered wall is to make connections between statements as the road map is created, so feel free to draw arrows or other links between items on the paper. Some statements may not necessarily fit on the route being mapped out, even though they are linked to another statement. In this case, put them to the side with an arrow linking them to the corresponding statement. Then repeat steps 1 to 3 for the two other goals. In some cases, the routes mapped out may not be parallel, but linked in different ways. Example of a theory of action

20

Objective Validate the theory of action defined by the group. The complexity of the theory as diagrammed will depend on the level of detail and depth reached by the group. The facilitator may be tasked with putting together the final version of the theory of action. In this case, even though the group has developed the theory, it is a good idea to meet in order to comment on and validate the final product. Duration 1 to 2 hours, depending on the group Individual Preparation Review the action theory diagram. Procedure Let the participants comment on the final diagram. Check whether any corrections are needed. This activity can be an opportunity to decide on follow up to the action theory development exercise: e.g., action plan, evaluation questions, etc.

WORKSHOP 3: VALIDATING THE THEORY OF ACTION

21

For a more detailed version of this workshop, see tool no 4 ―Identifying the dimensions to evaluate from a theory of action‖. Objective Identify the action theory dimensions you wish to evaluate. The goal is to select the action theory components that are of particular interest to the group, notably for the action principles. Duration 2 to 3 hours, depending on the group. It can be held immediately after Workshop 3 or at a later date. Individual Preparation Review the final version of the theory of action. Procedure Based on what has been said about ―how we work together,‖ the facilitator leads the group in identifying the dimensions it wants to observe. The goal is to identify the dimensions of the group’s work or process that participants consider most important and would like to document in the evaluation. List these dimensions on the board. Then select the dimensions to evaluate. To do this, you can group related ideas together and try to formulate a single statement for each group. This should result in one or two dimensions. If there are more than two, the group can vote to select one or two. The facilitator then goes over the dimensions one by one and asks each participant to specify which elements (or subdimensions) define them. Start by letting participants discuss freely while noting the proposed elements on the board. When there are no more new elements to add, the facilitator asks the participants to group and reword them. This rewording exercise should involve a fairly in-depth discussion on what these subdimensions mean to the group. Since data on these elements will be collected on a regular basis, the group must be sure they are well defined.

WORKSHOP 4: IDENTIFYING THE EVALUATION DIMENSIONS

ATTENTION! Experience has shown that data collection can be a real headache if the dimensions are not well defined.

ATTENTION! Experience has shown that even though some groups would like to observe multiple dimensions, it is best to choose one or two to facilitate data collection.

22

Exercise 1

DEVELOPING A SHARED VISION OF OUR PROCESS

I think the goals of our process should be… Careful! The objective here is to define the general goals of the project. These goals should identify the principles or values you feel underpin your actions, particularly with regard to the question ―How will we work together?‖

Goal 2:

Goal 1:

Goal 3:

23

Exercise 2

DEVELOPING A SHARED VISION OF OUR PROCESS

List the steps you think are required to achieve the goal—not necessarily the concrete methods for doing so, but in a sense, the best behaviours and practices for the job.

GOAL:

STARTING POINT

24

25

Identifying the evaluation dimensions from a theory of action

TOOL

4

26

27

TOOL 4: Identifying the evaluation dimensions from a theory of action

Objective1 Identify the evaluation dimensions and subdimensions from a theory of action or through any other exercise aimed at developing a shared vision of a project. Since this kind of evaluation is based on action principles, the specific goal is to select which principles to evaluate and define them so they can be observed. People involved Members of an evaluation committee or the project committee Duration About half a day

Step 1: Identify the action principles (dimensions to evaluate) From an action theory or shared vision, identify the action principles, i.e., the factors that influence ―how you work together.‖ If the committee has a diagram of its action theory that can be reproduced on a large poster and put up on the wall, proceed by circling the working-together dimensions deemed most important to the group. A text or any other document outlining the project vision can also serve as a starting point. The goal is to identify the dimensions of the group’s work or process that participants deem most important and would like to document in the evaluation. List these dimensions on the board or on a sheet of paper/posterboard affixed to the wall.

1 Tool developed by Paule Simard, Ginette Paré, and Diane Champagne as part of the Évaluation d’initiatives de développement avec les communautés. 2004-2007 research project conducted by Direction de santé publique de l’Abitibi-Témiscamingue, Institut national de santé publique, Réseau québécois de Villes et Villages en santé, Université du Québec en Abitibi-Témiscamingue and Université Laval, with funding from the Lucie and André Chagnon Foundation.

ABOUT THIS WORKSHOP

PROCEDURE

Tool 4

28

Then select the dimensions to evaluate. To do this, you can group related ideas together and try to formulate a single statement for each group. This should result in one or two dimensions (see Table 1). If there are more than two, the group can vote to select one or two. At this step, the dimensions are general (for example, goals may include working in an intersectoral capacity, mobilizing people, or promoting citizen participation). Step 2: Define the subdimensions in more detail The facilitator then goes over the dimensions one by one and asks each participant to specify which aspects define them. This yields a first level of detail, which we will refer to here as subdimensions. For this step, the group can be divided into two or three subgroups, with each subgroup working to define one dimension.

It is a good idea to let participants discuss freely while noting the proposed subdimensions on the board. At this stage, the aspects will still be relatively general. When there are no more aspects to add, the facilitator asks the participants to group and reword them. This rewording exercise should involve a fairly in-depth discussion on what these subdimensions mean to the group.

Step 3: Identify the elements to observe for each subdimension The next step is to define the elements for each subdimension on which data will be collected. To facilitate data collection, these elements must be as specific as possible and mean the same thing to the entire group. This exercise can be done in subgroups. To identify these elements, proceed as in Step 2. Have the group discuss each subdimension to define the concrete elements that could be used to document them. Each of these elements must be directly observable and easy to record. Once these three steps are complete, you should have the dimensions, secondary dimensions, and elements to evaluate. The following table shows how these fit together.


29

Table 1: Example of dimensions, subdimensions, and elements to observe in an evaluation

Primary dimensions Secondary dimensions Elements to observe

or data to collect

Cooperation within the committee

Changes in how the committee works

Member participation in discussions

Decision-making processes

Building of trust relationships between members

Listening and attention paid to the points of view of each member

Concrete actions and joint projects developed by the members

Changes in people’s sense of belonging

Newcomer participation in community life

Participation in local activities (specific to newcomers or general)

Level of use of community services/facilities

Newcomers’ sense of belonging to the community

Number of people in the community that newcomers can identify by name

Level of pride in living in this community

Participation of low income newcomers in improving their living conditions

Level of participation

Attendance at activities

Interest shown by newcomers in the proposed activities

Changes in the level of involvement

Changes in living conditions Improvement felt by those involved

30

Choosing a pair of glasses for evaluation

TOOL

5

33

WORKSHOP ON DEFINING

EVALUATION DIMENSIONS

Paule Simard, Ginette Paré, and Diane Champagne

Tool developed as part of the

Évaluation d’initiatives de développement avec les communautés 2004-2007 research project

conducted by Direction de santé publique de l’Abitibi-Témiscamingue, Institut national de santé publique, Réseau québécois de Villes et

Villages en santé, Université du Québec en Abitibi-Témiscamingue, and Université Laval, with funding from the

Lucie and André Chagnon Foundation

Tool 5

34

Objective Lead a group in defining the preferred action principles in their process for planning and evaluation purposes Participants Members of the committee responsible for the initiative or process to be evaluated (even if they are not necessarily the same as those on the evaluation committee) Required Material Large board and markers (several colors may be useful) Duration Two half days (ideally not consecutive) will be required to complete the four steps of this workshop.

Preliminary step (optional) Participants can be asked to complete a brief individual exercise prior to the workshop, which consists of reflecting on the project aspects they would like to evaluate, while focusing on the principles underpinning their action. Step 1: Tell me all about the project! Objective: Describe the process or project Participants explain their process or project, including what they do (activities, objectives, audience, etc.). It is important to let them discuss the projects while the facilitator notes what has been said on the board. Once everyone has had a turn to speak, go to Step 2. Step 2: A look in the mirror! Objective: Identify how the group works Once the project has been described, the facilitator introduces the following key phrase: How do you work together? The facilitator starts by letting the participants describe the different aspects of their group experience and how they work together. Make sure the group also discusses the action principles underpinning the process (e.g., participation of parents, collaboration with partners from other sectors, etc.). The facilitator notes what has been said on the board, either in the form of ideas or a diagram showing the relationships between partners, depending on how the discussion proceeds.

PROCEDURE

ABOUT THIS WORKSHOP

35

ATTENTION! Experience has shown that, even though some groups would like to observe multiple dimensions, it is best to choose one or two dimensions to facilitate data collection.

Step 3: Choose the right glasses Objective: Identify the evaluation dimension(s) Based on what has been said about ―how we work together,‖ the facilitator leads the group in determining which dimensions it wants to observe. The goal is to identify the dimensions of the group’s work or process that participants consider most important and would like to document in the evaluation. List these dimensions on the board. Then select the dimensions to evaluate. To do this, you can group related ideas together and try to formulate a single statement for each group. This should result in one or two dimensions. If there are more than two, the group can vote to select one or two. At this step, the dimensions are general (for example, goals may include working in an intersectoral capacity, mobilizing people, or promoting citizen participation). Step 4: Use bifocals Objective: Define in more detail the elements to observe for each dimension

The facilitator summarizes the dimensions already identified. If this step does not immediately follow the others, it’s also a good idea to briefly review the operating methods, main action principles, and chosen dimensions.

The facilitator then goes over the dimensions one by one and asks each participant to specify which aspects define them. This step should make it possible to identify the data elements to collect. It is a good idea to let participants discuss freely while noting the proposed elements on the board. When there are no more elements to add, the facilitator asks the participants to group and reword them. This rewording exercise should involve a fairly in-depth discussion on what these subdimensions mean to the group. Since data on these elements will be collected on a regular basis, the group must be sure they are well defined. After this fourth step, you should have the dimensions, secondary dimensions, and elements to evaluate. The following table shows how these levels fit together.


Ideally, Step 4 should be completed during the second work session.

36

Table 2: Example of Evaluation dimensions

Primary dimensions Secondary dimensions

Elements to observe or

data to collect






Listening and attention paid to the points of view of each member















37

Developing an evaluation plan

TOOL

6

38

39

TOOL 6: Developing an evaluation plan Usefulness of the tool An evaluation plan sums up the entire evaluation process at a glance. It explains which data you want to collect and how you plan to collect it. Development of the tool An evaluation plan normally includes the dimensions you have chosen to evaluate as well as information on where and how data will be collected, who will be responsible, and the timeframe. You can also add a section on how data will be analyzed and who will be in charge.

Primary and secondary dimensions chosen

Data to collect Data collection Data analysis

(optional)

Where and how? When? By who? How? By who?

Write the evaluation questions you have chosen

Specify the dimensions that will allow you to answer your questions

Identify the most appropriate and

realistic data collection methods

Indicate when the data should be

collected

Specify who will develop the tools

and who will collect the data

Indicate how the data will be

analyzed

Specify who will analyze the data

Tool 6

40

Sample evaluation plan

Primary dimensions

Secondary dimensions

Elements to observe Data collection Data analysis

(optional)

Methods Who? When? How? Who?




Personal mini-log Each committee member

Continuously for two years (each month at project committee meetings)

Annual assessment Compilation of all data by the evaluation committee First draft presented to the project committee for discussion

Preparation by the evaluation committee Analysis by the evaluation committee and then by the project committee Draw conclusions



Listening and attention paid to the points of view of each member Collective min-

log

Person in charge of data collection (committee chair)




Participation in local activities (specific to newcomers or general) Dashboard

Person in charge of the ―Welcome home!‖ project Continuously for

two years (depending on the activities carried out)




Survey and interviews

Survey evaluation team External resource (student): interviews





Dashboard Person in charge of activities

Continuously for two years (depending on the activities carried out)



Changes in living conditions

Improvement felt by those involved

Interviews External resource (student)

After 18 months

Survey Evaluation team

41

What do I see?

TOOL

7

42

43

EXAMPLE Original scenario A woman enters. Her shoulders are sagging, and she is sighing heavily. She seems to be carrying something rather large. She drags her feet. She puts her burden on the ground and takes something out of the load, shakes it and places it high up. After a few repetitions of this movement, she sits down and sighs. She holds her head.

TOOL 7: What do I see2? Learning to distinguish between what you see

and what you conclude from your observations

Objectives3 To help individuals who will be collecting data to distinguish between what they observe and how they interpret it. Participants A group of people who will be participating in data-collection, for example as part of a follow-up process for a project or an ongoing evaluation. Material needed Space to write (a large whiteboard divided in two or two flip-charts) and markers. Duration About one hour depending on the size of the group.

Step 1 One person (a participant or a member of the workshop team) acts out a situation for approximately two minutes. Any situation can be used, but it is important that when the story is recalled, it evokes a variety of emotions.

2 Outil utilisé dans le cadre de la recherche « Évaluation d’initiatives de développement avec les communautés. 2004-2007 » réalisée par la Direction de santé publique de l’Abitibi-Témiscamingue, l’Institut national de santé publique, le Réseau québécois de Villes et Villages en santé, l’Université du Québec en Abitibi-Témiscamingue et l’Université Laval à l’aide d’un financement de la Fondation Lucie et André Chagnon. 3 This workshop was adapted from an activity proposed by Jocelyne Lalonde as part of a training session on

Compassionate Communication held at Rouyn-Noranda in 2006.

ABOUT THIS WORKSHOP

STEPS

Tool 7

44

Step 2 The group leader asks participants one at a time to say what they saw. For each statement, the leader asks the group if it is a fact or an interpretation. Only the facts are written on the chart. As the participants learn to understand the distinction between fact and interpretation, they will be better able to add to what they have seen.

Step 3 Afterwards, the group briefly discusses the interpretations that they were able to construct from the facts they remembered. It soon becomes apparent that everyone doesn’t share the same interpretation of what they saw

EXAMPLE

What the participants say they saw: A woman hanging clothes on a clothes line She was dragging her feet She had a migraine headache She sighed heavily She was tired Etc.

What I saw She was dragging her feet She was sighing heavily She made repeated gestures She had sagging shoulders Etc.

My interpretation This is an exhausted/tired

woman, who finds her work very difficult

This is a tired old woman Etc.

45

Creating a logbook

TOOL

8

46

47

A tool for following the

phases of a community

project

Paule Simard with the collaboration of Ginette Paré and Diane Champagne A tool developed as part of the research project, ―Évaluation d’initiatives de développement avec les communautés, 2004-2007,‖ conducted by the Abitibi-Témiscamingue Department of Public Health, the Institut national de santé publique, the Réseau québécois de Villes et Villages en santé (Quebec Network of Healthy Cities and Towns), University of Quebec in Abitibi-Témiscamingue and Laval University with funding from the Lucie and André Gagnon Foundation.

Why a logbook?

Tool 8

48

ÉVALUATION D’INITIATIVES DE DÉVELOPPEMENT

AVEC LES COMMUNAUTÉS

(2004-2007)

Initiatives based on action principles

Community development approaches are based on the idea that local communities, territories, or regions are well positioned to improve the quality of life of their citizens because they constitute the decision-making level closest to individuals. However, due to the complexity of the problems these communities are facing, players from many sectors must be mobilized. Although community development approaches leave it up to the communities to decide what objectives to pursue, these approaches focus on the methods for attaining these objectives. The activity of developing communities is based on a number of action principles or principles about ―how we work together‖ that we believe can help make changes in the community. Despite the slight differences and specific features of each community development approach, all are structured around similar action principles that could be classified as follows: community participation, intersectoral action, and empowerment. Ultimately, what community development approaches have to offer are action principles, that is, ways of working together that can improve health and living conditions in local communities. However, these main action principles combine in different ways in local initiatives. In one, citizen participation may be emphasized, while in another there may be more concerted action with the various sectors. When it comes time to assess an initiative, it is therefore relevant to focus on ―how we work together.‖

Logbook Keeping a logbook helps follow the progress of a community development project (healthy communities, social development, sustainable development, etc.) and, as it evolves, stick to the principles that guide the action. Experience has shown that if we make the effort to think about the objectives regularly and systematically, as well as the methods we elaborated at the start, we are more effective, and it is easier to stay on course.

The logbook is built from the dimensions or questions chosen for the evaluation.4 To make the logbook as meaningful as possible for the project, it is important to the take time to design it well since the quality of the data collected depends on it.

4. This step is not described here, but is clarified elsewhere (see tools: ―Developing a theory of action: a shared vision for planning and evaluation‖ and ―Choosing your glasses for evaluation‖.

WHY A LOGBOOK ?

49

A model of a log is shown below. You can play with the different components to build the log that suits you best. The model below was designed to include the following elements:

The general dimension is stated on the first line Columns on the left: secondary dimensions; elements to observe Two columns on the right (space to write information): what we observed;

how we interpret it Possible additions: activities carried out; contextual elements

Building on the following example, you will see how a logbook might be designed (see logbook pages 5 and 6).

Principle

dimensions Secondary dimensions

Elements to observe

or

data to collect

How committee

meetings function

(*see example)

Development in procedures Members’ participation in discussions

Decision-making methods

Development of trust among members Respect for each member’s point of view

Development of action items and joint projects

Development in the

feeling of belonging

Participation of newcomers in organized

community life

Participation in local activities (both general and specific to newcomers)

Level of use of services/community facilities

Degree of attachment to the community Number of people who can be identified by name in the community

Level of community pride

Participation of low

income newcomers in

improving their living

conditions


Support needed for activities

Interest shown

Development in the level of commitment

Progress in living conditions Improvement felt by the people concerned

―The other advantage is to keep track…it requires a bit of work but most of our work should always be supported by written text. It is essential. We will be able to rely on this to move forward.‖

A Healthy Schools principal on the subject of the logbook

50

The mini-log

We can also build a logbook that is faster and easier to complete, a sort of mini-log. Since the information collected

in a mini-log is not elaborated in as much detail as the information in a logbook, it is better to reserve it for people

who, though not directly involved in the evaluation, have important viewpoints to record (members of the project

committee, leaders of specific activities, etc.). The mini-log can pertain to only one aspect of the dimensions being

evaluated in the project. It can also use visual elements for indicating satisfaction in addition to the space provided

for written text. See the example below.

Data collection

How and when the log is filled in must be established for each project. However, experience has shown that entries

should be made at least every 6 to 8 weeks, the ideal being about once a month. This depends on the frequency of

committee meetings. Here is a suggested calendar:#

JOURNAL S O N D J F M A M J

Selecting the elements to be followed in the logbook X

Logbook no 1 X

Logbook no 2 X

Logbook no 3 X

Logbook no 4 X

Logbook no 5 X

It is also important to decide who will be responsible for the log and who will help gather information.

Once a decision has been made about how often entries will be made and who will be responsible for

them, it is time to start collecting. It is extremely important to be regular and disciplined. This is

the price of high-quality data.

An example for

Dimension 1

51

Our logbook

DIMENSION 1: How the committee meetings function

ELEMENT TO FOLLOW IN THE LOG WHAT WE OBSERVED WHAT WE THINK ABOUT IT

Development of

procedures

Members’

participation in

discussions

This column is for the observed facts, what is happening.

This column is for comments, reflection, and analysis about the facts in the column on the left.

Decision-making

methods

52

DIMENSION 1: How the committee meetings function

ELEMENT TO FOLLOW IN THE LOG WHAT WE OBSERVED WHAT WE THINK ABOUT IT

Building

Development

of trust among

members

Respect for

each member’s

point of view

Development of

action items and

joint projects

53

My observation is about: a) How the committee works Trust among members b) Activities achieved What I saw

What I think about it What had an impact This is my observation: ______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

Signature: ___________________________________ Date of Observation: ___________________ -----------------------------------------------------------------------------------------------------

My observation is about: a) How the committee works Trust among members b) Activities achieved What I saw What I think about it What had an impact This is my observation: ______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________

Signature: ___________________________________ Date of Observation: _______________________

54

Road map

TOOL

9

57

TOOL 9: Road map A simple tool for compiling data on processes

The tool’s usefulness The road map5 serves to compile simple data related to the course of a project (activities, changes, results) based on the dimensions or evaluation questions defined at the start. It is a way to record information throughout the process. In fact, it is very much a kind of logbook that has the advantage of being easy to fill in quickly. The road map makes it easier to document observations while allowing comments to be recorded as well. The road map can be completed by a single person or by a group of people (evaluation committee members, the activity organizer, etc.). However, one person must still be responsible for ensuring that it is completed regularly. The key to collecting good data is to be organized and disciplined.

Constructing the tool The road map is built from the evaluation dimensions that were identified for the project. It can take different forms. It’s a matter of designing it to fit the needs of those who will be using it (see example).

5.

A tool developed by Paule Simard, Ginette Paré and Diane Champagne for the research project,―Évaluation d’initiatives de développement avec les communautés, 2004-2007,‖ conducted by the Abitibi-Témiscamingue Department of Public Health, the Institut national de santé publique, the Réseau québécois de Villes et Villages en santé (Quebec Network of Healthy Cities and Towns), Université du Québec en Abitibi-Témiscamingue and Université Laval with funding from the Lucie and André Gagnon Foundation.

Tool 9

58

Road map Activities completed

DATE DESCRIPTION

Subject— (The dimension or element you are documenting)

WHAT I SAW MY COMMENTS ON THIS SUBJECT





Name:

Date:

Page: #n#nn#o:

Time line

TOOL

10

61

TOOL 10: Timeline A simple tool for compiling data on processes

The tool’s usefulness The timeline6 serves to compile simple data related to the course of a project (activities, changes, results) based on the dimensions or evaluation questions defined at the start. Since data are recorded as the project evolves, what is happening can be monitored very simply, especially as it relates to action principles. The timeline can be completed by a single person or it can be accessible to as many people as possible so that everyone can contribute. However, one person must still be responsible for ensuring that it is completed regularly. The key to collecting good data is to be organized and disciplined. The drawback of this tool is that it does not allow for detailed information. However, this can be compensated for by recording more detailed comments or reflections in a note book. It is still useful to connect the events recorded on the timeline with the comments in the note book. Constructing the tool Simply draw a series of parallel lines on a large sheet of paper, cardboard or banner mounted on a wall. On the main line, benchmarks are drawn. These are small lines that mark the months, three-month terms or years, as the case may be. Data can then be recorded on each line, either directly on the mounted paper or on small sticky notes that can be added from time to time. Using the tool The main line indicates the time. Events related to the project can also be written down (important meetings, activities, arrival and departure of members, etc.). It is especially useful for recording facts about actions. The other parallel lines are used to record data concerning the dimensions to be evaluated. Each line is reserved for a specific dimension where observations about the various elements that define this dimension are written down.

6. A tool developed by Paule Simard, Ginette Paré and Diane Champagne for the research project,―Évaluation

d’initiatives de développement avec les communautés, 2004-2007,‖ conducted by the Abitibi-Témiscamingue Department of Public Health, the Institut national de santé publique, the Réseau québécois de Villes et Villages en santé (Quebec Network of Healthy Cities and Towns), Université du Québec en Abitibi-Témiscamingue and Université Laval with funding from the Lucie and André Gagnon Foundation.

Tool 10

62

Individual interview

TOOL

11

65

TOOL 11: Individual interviews7

The in-depth interview explores a topic in detail to deepen the interviewer’s knowledge of the topic. Open-ended refers to the fact that the interviewer is open to any and all relevant responses. There are no correct answers, and the interviewee is not asked to select from a series of alternative choices. Why do in-depth, opened-ended interviewing?

To explore a new, or little understood, topic To discover new information and expand existing understanding To get the perspective of one individual without interference from others

To do so, the interviewer must…

Keep in mind how the topic relates to and illuminates the larger questions asked by the study

Determine whether the person being interviewed is staying on topic, and if not, how to reintroduce the topic

Understand what logical connections the interview is making in the discussion when those connections are likely to be quite different from those of the interviewer

Decide whether or not pursue new ideas and directions Probe for the meaning of terms Recognize when the interviewee’s ideas are clearly expressed, and when they need to be

elaborated to make sure that they can be understood by everyone who reads the notes or transcripts

How do you conduct an individual interview?

Getting started:

Introduce yourself and the project, including the organizations sponsoring it. Ensure confidentiality and explain how you will protect privacy Tell interviewees that their views are very important to you and to the project, and

explain why. Ask permission to record interviews by tape-recording and in writing Make interviewees comfortable by asking how they are, how their day went, how their

family is, or some other culturally appropriate small talk.

7. Adapted from Margaret D. LeCompte, Jean J. Schensul and Stephen L. Schensul, Essential Ethnographic Methods, Ethnographer’s Toolkit, 1999, Altamira Press by Mary Richardson.

Tool 11

66

While the interview is being conducted, you can encourage additional information by:

Using neutral agreement Repeating what the person has said in a questioning way Asking for more information (What games do teens play in this neighbourhood? What

issues are people concerned with in this town?) Asking for clarification (what do you mean by...?) Asking for an opinion Ask for narratives of experience. (―Tell me about your last experience with a health

problem.‖)

If the person gets off-topic, you can:

Reintroduce the topic in a natural way Decide whether or not pursue new ideas and directions

How to Maintain a Positive Researcher-Respondent Relationship

Avoid offering opinions or making judgments about what the interviewee says, despite having strong feelings on the topic. Researcher judgments will influence respondents’ ideas and answers.

Avoid showing surprise, disgust, or other strong emotions, regardless of personal opinion. Accept hospitality when offered. Most people in most cultures offer food or drink when

they are interviewed, and refusing may insult your host. Unless the safety factor is critical or researchers have dietary or alcohol restrictions, they should accept and consume at least some of whatever food or drink is offered.

Be aware of the general condition of the informant. If he or she is ill, intoxicated, or emotionally upset, the information provided may not be of the highest quality.

What can create bias?

Asking leading questions Failing to follow up on or omitting topics that the interviewee introduces Redirecting the story or interrupting it Failing to recognize reactions of the interviewee to the interviewer’s personal

characteristics, including dress, age, race, gender, body size or apparent social status Asking questions that include or suggest the desired responses, i.e. be aware of how you

influence the respondent Using non-verbal cues to indicate the right answer to a question, or the response with

which the researcher agrees Stating opinions on an issue. Researchers should avoid volunteering opinions at all, and

if forced to do so, they should wait until after the interview is over

67

How to record your interviews There are 3 ways to record your interviews: written, taped and videotaped In writing:

ADVANTAGES: reminders of questions and answers – inexpensive – does not depend on electronic equipment which might fail – researcher controls the transcription process.

DISAVANTAGES: distracts the interviewer from the openness of direct communication with the interviewee; it is also difficult to write quickly – and legibly – enough to record everything an informant says.

Audio recording:

ADVANTAGES: accuracy of recording.

DISAVANTAGES: Audio recording interviews can be expensive, requiring high-quality recorders, working batteries or an available electrical outlet, and time and money for transcription. Audio-recorded interviews are often lengthy, and they will require as much as 3 or 4 hours of transcription time for every hour of taped interview. …noisy environment…. Quality of recording may be poor.

On videotape:

Usually is not used for in-depth interviewing because of its cost and the inconvenience of transcribing videotaped interviews. Confidentiality is more difficult to ensure with videotaped data because the faces and other physical characteristics of the interviewees are easy to distinguish. We do not recommend the use of videotaping for in-depth interviews unless the interviews are to be used in a video production on the research topic. Examples

Kinds of questions that can be used in an exploratory interview ―So you’ve just begun to implement a new social development curriculum in Hillside School. How is it going?‖ (Probes: who is using it, who is not, how is it different from the old one, what kinds of advantages does it have over the old one, and what problems). ―How are you introducing it to the teachers?‖ (how about the training process, any technical assistance, monitoring in the classroom, special help teachers need). ―Does it address some of the main social and health problems children face in this school system?‖ (Probes: What are the main problems children face? How are they incorporated into the curriculum? How will this new curriculum improve the chances for changing children’s behaviour or circumstances over the standard curriculum currently in use?)

68

The awareness of the interviewer An essential quality of an interviewer is to be aware of the interviewee’s feelings and to be as resourceful as possible to find the appropriate solution.

A researcher studying perceptions of body image among Puerto Rican adult women and young girls in Hartford, found that women were not comfortable when asked directly about their own bodies. She was able to initiate interviews by asking women to talk about their views of female body types as represented by line drawings depicting a range of women’s bodies in different shapes, heights, and sizes. It was also easier to do in-depth interviews with preadolescent girls about their attitudes toward alcohol use if they were allowed to draw a picture about the topic while they were talking about it. The children’s reservations at talking about these sensitive topics were allayed when they used the picture as a point of discussion.

69

Focus group interviews

TOOL

12

70

71

TOOL 12: Focus group interviews8 What is a focus group? A focus group is a qualitative social research method. The principle underlying focus groups is that each person, however ordinary, is an expert on his or her own experience. In a sense, a focus group is an interview. It is not a discussion, a problem-solving session or a decision-making group. The goal is not for the group to reach a consensus, but rather to encourage the expression of all opinions. It is a method that gives access to the perceptions, attitudes, beliefs and areas of resistance of target groups, in a context where the communication structure is egalitarian. Why use focus groups? Focus groups draw on the natural human tendency to discuss problems in groups. Focus groups emphasize the value of sharing viewpoints, discussing and resolving issues in groups. A considerable advantage of the method is that the information given by the participants is self-validated. The group dynamics help bring out the most important topics and maintain a friendly atmosphere. Focus groups are ideal for carrying out needs assessments, testing intervention possibilities, orienting policies, planning projects and evaluating the perceived impact of projects. Focus groups are most useful for gaining understanding, not testing a hypothesis.

History of focus groups

Focus groups were first used in 1926 by Bogardus, who used them to test a social distance scale (Frey and Fontana 1993). Many researchers have since used them, from the 1940s on.

Focus groups have probably been most used in social marketing. Researchers in the field of health have used focus groups, particularly in nursing. Focus groups are used in applied anthropology to improve development programs and

elaborate communication strategies that are well-adapted to different cultural contexts.

8. Adapted by Mary Richardson from Gisèle Simard (1989). La Méthode du "focus group". Mondia.

Tool 12

72

How do you plan a focus group? A representative number of groups are recruited, containing between 6 and 12 people with homogeneous characteristics. For example, depending on the theme to be discussed you may compose groups of the same gender (women or men), certain age groups, specific occupations, or people with similar life experiences. Open discussion is encouraged using a group interview outline on the theme of the study. A summary-analysis is then carried out identifying the key messages given by the participants, as well as the points of convergence and divergence between the groups in the sample. A successful focus group should respect the following 4 conditions:

The participants should belong to a homogeneous category: same social status, age group, level of education and income, and language.

The leader should respect group dynamics, while orienting the discussion around the themes in the interview outline.

The interview outline should meet the study objectives and be adapted to the study population: simple, clear language, open, non-biased questions.

The results should be systematically summarized, and include quantitative data (intergroup comparison and percentages).

A ―content saturation‖ point is usually reached after about 7 to 10 focus groups, that is, no new elements emerge. For the focus groups method to be valid and reliable and to obtain scientifically valid findings, it is important that the number of groups be sufficient, the sample be representative, the animation be unbiased and the analysis be objective. Recruiting participants How many participants?

Each focus group should have no less than 6 participants and no more than 12: 10 participants is ideal. A good idea is to ask 12-15 people to participate to allow for any who cannot come at the last minute.

How many groups?

What is important is reaching a content saturation point. The number of groups needed to do so could be between 10 and 150 depending on the magnitude of the research (local or national) and its complexity.

When to carry out focus groups?

The main consideration is people’s availability: avoid important events people are likely to be attending, depending on the local context.

73

Where to carry out focus groups?

Try to find a relatively neutral place, i.e. a non-religious, non-political building, in order to avoid being associated with a specific religion or political party or with customary vs. modern power, etc. The non-verbal message must be that all opinions are welcome and all participants are on an equal footing.

The best place may be a public school, a community association, a municipal building or town hall (provided there are no major political tensions).

How long should the focus group last?

Between 1 and 3 hours should be enough, depending on the number of themes to be discussed. For 2 or 3 themes, 1 hour might be sufficient, whereas more than 3 hours is too long and people will get tired.

Should men and women be mixed?

It all depends on the cultural context: if women are likely not to talk in the presence of men, or not to express their true opinions or not to contradict what the men say, they should be separate.

It also depends on the subject: some subjects may be more sensitive (sexuality, health issues, etc.) and people will feel more comfortable in a homogeneous group.

Who should participate?

In order to be representative of the population, the participants should be selected on the basis of age, sex, marital status, education, income, occupation, language, geographical territory, rural or urban concentrations.

Add on other criteria depending on the topic of the study (if the study is on prostitution, number of years practice, full-time or part-time, etc.).

How to recruit participants

Personal contact (by telephone, reference or chance encounters) Network (a number of people act as contact persons and recruit people in their

networks). The influence of a community leader who backs the research can be very important.

Should participants be paid?

Participants may be paid a small amount, at least to cover their transportation. A snack and drinks could be served after the end of the discussion.

Facilitating a focus group Greeting participants

It is very important to greet participants when they arrive to put them at ease and establish a warm atmosphere.

It is best to place chairs in a circle so that there is no subtle hierarchical relationship between people.

74

Presenting the project

Describe the project Emphasize the fact that the focus group is anonymous Explain why the discussion must be recorded

Participants

Participants often play different roles within the group: leader, counter-leader, associate leader, associate counter-leader, marginal, peace-keeper or consensus-seeker, passive or shy.

The facilitator’s role

The facilitator must create a group dynamic by putting participants at ease. The facilitator leads the discussion and keeps people on topic. He or she must give everyone a chance to speak, cutting off the leader and counter-

leader if they take up too much space, and encouraging shy or passive people to express themselves.

The facilitator can caricature certain ideas to make participants react and add nuances to their assertions.

If the participants hesitate too much or do not discuss, the facilitator may use scenarios and give every participant a role. This technique may in certain cases make it easier for the participants to speak.

The facilitator may have to call STOP when the discussion becomes too emotional, summarize the opinions expressed thus far and reorient the discussion with another question.

He or she must reformulate questions or sub-questions to encourage participants to speak freely about their experience, especially when no one has anything to say.

Facilitator’s attitudes

The facilitator has to act as described above and observe at the same time. He or she must observe non-verbal messages (body language), analyze the emerging group dynamics, and find ways to involve all participants.

The facilitator must be open and attentive, not take any position, remain neutral in both words and actions. He or she must direct the discussion without being authoritarian or rigid.

He or she must also try to ensure that the focus group is a positive experience of human relations, avoiding interpersonal conflicts and making sure that all individuals feel respected for who they are, whatever their views.

75

Consent form

TOOL

13

76

77

TOOL 13: Consent form Project title: Names of the researchers: Funding source: Project duration: PURPOSE OF THE RESEARCH … DESCRIPTION OF THE RESEARCH … Participation in the study Your participation in this study involves two interviews: one interview in a small group with your organization, and another in a large group. These interviews should not take longer than two (2:00) hours each. The subject will be... Disadvantages The only disadvantage to participating in this study is the time it takes for the interviews. Benefits You will have the opportunity to share your opinions, thus helping to develop... Confidentiality The information you give will be confidential and accessible only for research purposes. Only the research team will have access to the recordings of the interviews and will know the identity of the participants. They will be allowed to use interview excerpts in documents as long as these remain anonymous and the person speaking cannot be identified. The interview data (written and audio-recorded) will be kept confidentially in ... and destroyed after three years. Payment for participation You will not be paid for participating in this study. Sharing the results The results of the study will be shared with the participating organizations... A written report will be available in... You can ask for a copy of the document from the research coordinator (see below).

Tool 13

78

Consent I agree to participate in the study Title of study. I hereby consent to participate in this study. I understand that I am free to withdraw at any time without consequences of any kind. Signatures

Signature of the participant Signature of the research coordinator

_______________________________________

_____________________________________

Signed on ___________________________________, at _________________________________ For further information on your rights, please contact: Research coordinator, Contact information

79

Compilation matrix

TOOL

14

80

TOOL 14: Compilation matrix This matrix is useful to bring together all data collected over a long period of time with the same tool, for example a logbook. This is a reduced model: this tool works better on an 11 x 17 sheet.

Dimensions MINI-LOGS – YEAR ONE MINI-LOGS – YEAR TWO

1 2 3 4 5 6 7 8 9 10


Compilation of collective mini-log 1 on this dimension



… … … … … … …





… … … … … … …

Tool 14

Taking stock

TOOL

15

83

TOOL 15: Taking stock Extracting findings

Objectives of the workshop9 To summarize and extract findings from information that has accumulated over a period of time, with personal training for participants. Participants The members of an evaluation committee or the committee in charge of the project. Material needed

* Sheets of the same coloured cardboard * Markers * Sticky notes * Large wall to display the sheets of cardboard

Duration Approximately half a day.

Preparation One or two weeks before the workshop, a summary of the information collected can be given to participants. While this preparation is very useful, experience has shown that only a few committee members will get to know the data. If this is the case, the discussion can start with what the members remember and have experienced. This can be enriched by information collected systematically for the evaluation that the leader or other committee members more closely associated with the evaluation might be able to contribute.

9. A tool developed by Paule Simard, Ginette Paré and Diane Champagne for the research project, ―Évaluation

d’initiatives de développement avec les communautés, 2004-2007,‖ conducted by the Abitibi-Témiscamingue Department of Public Health, the Institut national de santé publique, the Réseau québécois de Villes et Villages en santé (Quebec Network of Healthy Cities and Towns), Université du Québec en Abitibi-Témiscamingue and Université Laval with funding from the Lucie and André Gagnon Foundation.

ABOUT THIS WORKSHOP

SEQUENCE OF STEPS

Tool 15

84

The leader of the workshop writes on the sheets of cardboard (8 ½ x 11) each of the dimensions or questions (sub-dimensions and sub-questions) that the group had initially decided to evaluate. Step 1 The leader mounts sheets of cardboard on a large wall (one beneath the other on the far right), on which the dimensions being evaluated by the group are written. One dimension is selected, and the leader asks participants if they have seen any progress, change or result in that dimension over the assessment period. Each statement is written down on a sheet of cardboard and mounted on the wall so that a road is built leading to the dimension in question. The first statements can be placed anywhere on the line, but as more elements are added and the sheets of cardboard are moved around, it becomes possible to piece together a certain logical sequence. Step 2 Repeat the same process for the other evaluation dimensions. Step 3 The statements are then gathered together as data and synthesized into a diagram. The discussions regarding each dimension are valuable lessons and make excellent exercises for analyzing data.

85

How to put together

an evaluation report

TOOL

16

86

87

ÉVALUATION D’INITIATIVES DE DÉVELOPPEMENT

AVEC LES COMMUNAUTÉS

(2004-2007)

A tool developed by Paule Simard, Ginette Paré and Diane Champagne for the research project, ―Évaluation d’initiatives de développement avec les communautés, 2004-2007,‖ conducted by the Abitibi-Témiscamingue Department of Public Health, the Institut national de santé publique, the Réseau québécois de Villes et Villages en santé (Quebec Network of Healthy Cities and Towns), Université du Québec en Abitibi-Témiscamingue and Université Laval with funding from the Lucie and André Gagnon Foundation.

Tool 16

88

SAMPLE OF A PLAN

Contents This tool uses a framework to structure an evaluation report based on the evaluation dimensions identified at the beginning of the process. It is structured in the same way as the logbook (see the tool, Creating a Logbook), but can be used even if you have not made a logbook.

Generally speaking, all the tools that are used to collect and organize data should have the same structure to facilitate seeing the development of the different evaluation dimensions. When you get to the evaluation report, the same initial structure serves to organize the text. Step 1 Build a framework from the dimensions selected for evaluation.

Step 2 The evaluation committee or the larger project committee completes the framework by drawing from the data that has been collected, connected and organized with the help of various tools (compilation matrices, flow charts, annual assessments, interview statements and other collected data. It is important, here, to be able to follow the development of the dimension and subdimension over time.

Step 3 Write the final report. The committee assigns one person to do it. If the framework has been completed with care, this will only involve transforming ideas into text. To facilitate matters and make sure that all points are considered, it is suggested that you write at least one paragraph (10 to 15 lines) for each element of the plan (that is, the lines of the framework). Use a plain language that will be meaningful for your team and your community.

This plan proposes the essential elements of an evaluation report… but it is important to adapt it to one’s own needs by adding or removing the elements appropriate to each situation. 1. The project

1.1. Presentation of the project brief history and current project the committee

o composition and evolution o way of operating

1.2. Presentation of the evaluation type of evaluation selected

o goal of the evaluation (Why was an evaluation made?)

o selection of the evaluation committee

methodology

2. Development of the dimensions

2.1. Identification of selected dimensions (those in the logbook)

2.2. Description of the development and analysis of the findings for each of the dimensions (and sub-dimensions)

dimension 1 dimension 2 etc.

2.3. Presentation of the findings for the dimensions as a whole (general facts)

2.4. Presentation of the effect of context on the project’s course

3. The course of the action

3.1. Paths taken regarding the operation of the committee

3.2. Paths taken regarding actions carried out in the project

4. In brief… What would you like to be retained about your evaluation?

HOW IT WORKS

ABOUT THIS WORKSHOP

89

1.1 Presentation of the project

Questions Progress observed through evaluation tools

Give the history of your project and explain what it currently involves

Explain how your project operates

Describe the form that your project committee takes (form, formal association to municipal council, mandate, etc.)

Name the members following the committee’s evolution

90

1.2 Presentation of the evaluation

Questions As defined in your evaluation plan Guidance for the

evaluation or future evaluation

Describe the type of evaluation you have chosen (goal, selection of committee)

Specify the methodology selected (form and frequency of data collection, methods of analyzing data, etc.)

91

2. Development of dimensions

Questions

Progress observed through evaluation tools

Guidance for monitoring the project

Identify the selected dimensions (indicate if they were modified along the way)

Summarize development of each dimension

Dimension 1 (write here)

Sub-dimension (write here)








92

2. Development of dimensions (continued)

Questions Progress observed through

evaluation tools

Guidance for monitoring the project

Summarize development of each dimension










Describe your general findings for the project as a whole

Describe the effect (or effects) of the different contexts (project, territory, municipality, etc.) on the course of your project

93

3. The course of action…


List the paths taken regarding the operation of the committee

List the paths taken as to the actions carried out in the project

94

4. Summary…


State the main points you would like to be retained regarding the findings of your evaluation

1

2

3

4

How to write a press release

TOOL

17

97

TOOL 17: How to write a press release Press releases can be used to inform the population on an event, a situation, a program. Here is a visual example followed by explanations. Your name/Name of the association/logo Address/contact information PRESS RELEASE For immediate release TITLE LONDON – May 15th, 2000 – Introduction Body/ Conclusion/contact information for the readers

###

Contact information for the publisher

On the top left, write who the press release is from (name, association, logo, address).

Under it, centered in capital letters bigger than the rest: PRESS RELEASE and ―For immediate release‖.

Under it, centered in capital letters: The TITLE. It must be short, to the point and make

people want to read more. Usually, it is composed of a SUBJECT (your organization’s name), a VERB (the action) and the OBJECT (what).

Introduction, all (what, when, why, who) must be said in the first 25-30 words. Start with

PLACE – Date – and with a catchy phrase or a question. At the end of the introduction, add a short sentence to tell people why they should read more.

The body must have a maximum of 250 words, about 3 to 4 paragraphs of about 3

phrases. It includes the introduction and the conclusion. Keep it simple and attractive. Avoid the use of too many adjectives or fancy words.

Take a pause and read it again. Each time you find it boring, add a ―!‖ or a question.

Adapt your wording to the people you want to talk to. You can’t talk to everybody at

once. Who do you think will relate to your message? Adding an eye-catching graphic or a good quality black and white picture can be a good

idea if it adds to your text. Be sure that the media can use the image (free or with rights). If

the image is yours, don’t forget to write this under it so the media can use it for free. Provide contact information: contact name, address, phone, email, web site.

End the press release with ### centered (press release norms).

When you send the press release, send it to someone in particular and add a personalized

memo to the person.

Tool 17

LIST OF HEATHY PINE VALLEY’S TOOLS Unit 1: Deciding on evaluation Winning conditions for an evaluation in Healthy Pine Valley Who will do the Heathy Pine Valley’s evaluation?

Unit 2: Forming an evaluation committee Unit 3: Identifying what to evaluate Heathy Pine Valley’s action theory Dimensions selected by Heathy Pine Valley

Unit 4: Establishing an evaluation plan and collecting data Heathy Pine Valley’s evaluation plan Heathy Pine Valley’s personal mini-log Heathy Pine Valley’s collective log

Unit 5: Analysing the data Heathy Pine Valley’s compilation matrix Heathy Pine Valley’s flow chart

Unit 6: Summarizing and reporting the results

TOOLS

Healthy Pine Valley

101

Winning conditions for an evaluation in Healthy Pine Valley

Are we prepared to put time into the evaluation?

We cannot do the evaluation.

Is it possible to conduct an on-going evaluation as part

of our project?

We will have to do the evaluation at the end

of our project.

Do we have the necessary expertise to conduct a

participatory evaluation?

We will need to obtain the support of qualified individuals.

Is our project clearly defined, particularly with regard to the action principles that

guide us?

YES

YES

YES

YES

NO

NO

NO

NO

We will need to do a collective exercise before starting the

evaluation.

YES, but not if it has a negative impact on

action

Our Answers

YES

NO, we will need someone to help us

No, we never did this kind of exercise

Are we ready to question how we do things?

The evaluation will not be useful.

YES

NO YES, if it’s done with

respect.

102

OUR ANSWERS

Do we want to have control over our evaluation?

YES NO

Do we want to conduct our evaluation ourselves? External evaluation: we can still

negotiate our role

Who will work on it? What type of help do we need?

List of evaluation group members

Do we need help collecting data?

Do the work according to the evaluation plan Ask someone in the

community (perhaps pay them) to collect

data. We then proceed with the data collection

NO YES

NO YES

Hire someone to do all the work under

the committee’s supervision

Ask for help (university,

master’s students, local actors, etc.)

Yes

Yes

A 3 member committee is formed:

Julie, George and Norman (community

worker)

Yes

Who will do Healthy Pine Valley’s evaluation?

Julie contacts professor André, a University

researcher, who is willing to help the committee if needed. A student could also give a hand for data

collection.

103

CREATE A RICH, SUSTAINABLE, AND

ACTIVE PARTNERSHIP IN THE COMMUNITY

DEVELOP A SENSE OF BELONGING TO THE

MUNICIPALITY, ESPECIALLY AMONG

NEWCOMERS

Identify the partners to

contact

Convince the partners to

participate in the process

Define working rules that allow everyone

to participate

Build relationships of trust

Get to know one another

Develop actions that meet the interests of all

parties

Identify newcomers

Define what is meant by ―sense of

belonging‖ and how to identify it

Reach out to newcomers Develop welcome

activities

Make established residents aware of the need to

properly welcome newcomers

Organize activities to facilitate contacts between

all citizens

Develop a vision of a livable municipality

Carry out a municipal revitalization project

EMPOWER LOW INCOME NEWCOMERS TO PARTICIPATE IN

IMPROVING THEIR LIVING CONDITIONS IN

THE COMMUNITY

Reach out to newcomer

tenants

Organize activities to break the

isolation of these newcomers Ensure continuity in

carrying out these activities

Have newcomers identify activities that meet their

needs

Obtain support from CSSS and the municipality Develop their

capacity to initiate and

manage their activities

themselves

Healthy Pine Valley’s action theory

104

Dimensions selected by Healthy Pine Valley

Primary dimensions Secondary dimensions Elements to observe






Respect for the points of view of each member

Development of actions and joint projects





Sense of belonging to the community

Number of people in the community who can be identified by their name





Interest shown



105

Healthy Pine Valley’s evaluation plan

Primary dimensions

Secondary dimensions Elements to observe Data collection

methods Who? When?



Member participation in discussions Personal mini-log

Each member of the Healthy Pine Valley Committee

Continuously for two years (each month at Healthy Pine Valley Committee meetings)

Decision-making processes Building of trust relationships between members

Respect for the points of view of each member

Collective mini-log Person in charge of collection (committee chair) Development of actions and

joint projects



Participation in local activities (specific to newcomers or general) Dashboard

Person in charge of the ―Welcome home!‖ project Continuously for

two years (depending on the activities carried out)


Sense of belonging to the community

Number of people in the community who can be identified by name

Survey and interviews

Survey evaluation team External resource (student): interviews

Level of pride in living in the community




Dashboard

Person in charge of activities

Continuously for two years (depending on the activities carried out)

Interest shown Person in charge of activities

Continuously for two years Changes in the level of

involvement

Changes in living conditions

Improvement felt by those involved

Interviews External resource (student) After 18 months

Survey Evaluation team

106

Healthy Pine Valley’s personal mini-log

Regarding changes in how the Healthy Pine Valley committee works Member participation in discussions (give examples)

WHAT I SAW MY COMMENTS

I noticed that John from the food bank participated more actively at the last meeting. Michelle and Robert, two citizen representatives, have not spoken during the last three meetings.

I think he’s starting (after three months of observation) to better understand how we operate. Maybe he didn’t have all the information… Something to think about for the new members. The participation of ―ordinary‖ citizens is adversely affected by the presence of two institutional representatives, the directors of the CSSS and CLE, even though they’re both very open minded.

Decision making


There was a bit of confusion around the decision to invite a low income resident to join the committee. Nathalie didn’t seem comfortable with the idea (her body language spoke volumes) but she didn’t say anything.

Maybe this needs more discussion… Would a value-sharing exercise be relevant?

Regarding trust relationships between members Respect for points of view


See above: Nathalie’s reluctance to share her point of view…

Regarding Nathalie: Does she perceive a lack of respect for opinions that differ from the majority?


WHAT I SAW MY COMMENTS Peter (from ―The Link,‖ an organization that supports low income families) agreed to lead the community kitchen project. He will put a few ideas down on paper and present them at the next meeting.

This is the Healthy Pine Valley Committee’s first project of this magnitude. People are enthusiastic, even if the project is big and requires a lot of time and energy.

Name: Julie Date: November 24, 20…

107

Healthy Pine Valley’s collective log

What the members observed What the members thought

Reg

ard

ing

cha

nges

in h

ow t

he H

ealt

hy P

ine

Va

lley

com

mit

tee

wor

ks


Compilation of comments by various people: Julie… George… Jean-Pierre… Etc.

Compilation of comments by various people: Julie…

George… Jean-Pierre… Etc.





108

What the Members Observed What the Members Thought

Reg

ard

ing

trus

t re

lati

onsh

ips b

etw

een

mem

ber

s

Respect for points of view







109

Healthy Pine Valley’s compilation matrix (over two years)

Dimension 1: Changes in cooperation within the healthy pine valley committee (reduced model: this tool works better on an 11 x 17 sheet)

Dimensions MINI-LOGS – YEAR ONE MINI-LOGS – YEAR TWO

1 2 3 4 5 6 7 8 9 10





… … … … … … …





… … … … … … …

110

CREATE A RICH, SUSTAINABLE,

AND ACTIVE PARTNERSHIP IN

THE COMMUNITY

Meeting of all potential

partners to present the

idea -

5 of 7 agree to get involved

At the initial meetings,

attendees get to know each other and decide on an operating method

In February, CSSS withdraws from the

project due to a lack of resources

- Demobilization

- Meeting with CSSS

management

An initial project emerges (citizen advocacy) and

seems to rally the committee

CSSS returns in September

- The group is very

motivated -

The workshop on sense of

belonging has energized members

Discussion on how the committee works

- Positive, but two points come up:

presence of directors alongside citizens,

and the lack of/need for coordination

DEVELOP THE SENSE OF BELONGING TO THE

MUNICIPALITY, ESPECIALLY AMONG

THE NEWCOMERS

Questioning about how to strengthen the sense

of belonging –

Decision to conduct a survey based on the dimensions identified

during the activity -

Holiday party ¾ of newcomers attend

The survey indicates that newcomers’

sense of belonging seems

stronger when they know people in the community

Decision to set up the citizen

advocacy project in

September

EMPOWER LOW INCOME NEWCOMERS

TO PARTICIPATE IN IMPROVING THEIR

LIVING CONDITIONS IN

THE COMMUNITY

Healthy Pine Valley’s flow chart (over two years)

June

2008

December

2006

June

2007

December

2007

Reflective activity on the sense of

belonging

It is noted that several newcomers are low

income. -

A CSSS social worker is contacted,

especially since CSSS has just left the

committee

3 coffee meetings are held with the

newcomers identified with help of CSSS

- Interest in a community

kitchen

Decision to explore the idea of a

community kitchen in the months ahead

Documents

LIST OF TOOLS T€¦ · Creating a logbook 9. Road map 10. Timeline 11. Individual interviews 12. Focus group interviews 13. Consent form Unit 5: Analysing the data 14. Compilation