1

MISSION POSSIBLEEvaluating Stakeholder

Consultation Mechanisms

Presentation by:

David Taylor and Unnati Vasavada

Program Evaluation Directorate

Transport Canada

Canadian Evaluation Society Conference 2003

Vancouver, B.C.

June 2, 2003

2

Presentation Overview

1. Evaluating Consultation: How?

2. Case Study: Safety and Security Group, Transport Canada

Evaluation ApproachKey Study FindingsWhat Went WellChallengesLessons Learned

3

How to evaluate consultation?*

*Privy Council Office. Consulting and Engaging Canadians: Evaluation Guidelines

Stage in the Consultationor Engagement

The Value Added of Evaluation

1. Preparation Stage Setting clear, measurable and realistic objectives

2. Design Stage Mapping the process: identif ying f actors that canimpact on the success of a process andstrategies/ tools f or collecting inf ormation

3. Implementation Evaluating progress: issues/ questions andstrategies/ tools f or collecting inf ormation

4. Feedback and Follow- up Evaluating accountability: issues/ questions andstrategies/ tools f or collecting inf ormation

5. Final Evaluation Evaluating the process and outcomes, and sharinglessons learned.

4

Case Study

Evaluation of consultation mechanisms used

by directorates in Transport Canada

responsible for safety regulation in aviation,

marine, rail and road modes of

transportation.

5

Evaluation Approach

Consultation defined:

“Consultation” involves two-way exchanges of ideas and information before a decision is made.

6

Evaluation Approach

Scope of evaluation determined: formal consultative committees stakeholder meetings direct mailings Internet discussion sites

7

Evaluation Approach

11 assessment criteria established: Clarity and Achievement of Purpose Stakeholders Selection and Structure of Mechanisms Decision-Making Process Roles, Responsibilities and

Accountabilities of Decision-Makers and Stakeholders

8

Evaluation Approach

Assessment criteria (cont’d) Resources Time Frame Information Compatible with Context Open and Credible to Those Being

Consulted Ongoing Review

9

Evaluation Approach

Stakeholders selected for interview random sampling targeted sampling population

Interviews (mainly by telephone) conducted

10

Key Study Findings Need to: Improve direct representation of general

public in consultations Provide regular and timely feedback and

decision rationale. Improve length of decision-making process Maintain face-to-face meetings

11

Key Study Findings (cont’d)Need to: Articulate and communicate consultation

objectives, participants’ roles and responsibilities

Conduct periodic assessments of consultation exercises

Explore benefits of direct citizen engagement techniques

12

What Went Well?

Clear definition of consultation Assessment criteria Stakeholders appreciated evaluation exercise

13

Challenges Low or non response rates Poorly maintained stakeholder lists Varying levels of consultation experience

among respondents Length of time lapsed since participating in

consultation Managing qualitative data

14

Lessons Learned Consider carefully who should evaluate:

consultants vs. in-house experts Plan carefully study timing and timelines Use focused and pragmatic evaluation approach Use mix of qualitative and quantitative research

methods. Be thorough in attending to administrative details Consider informal means of evaluating

consultation

15

Discussion

Questions?

Comments?