Cheryl Poth, Faculty of Education, University of Alberta Teresa Broers, QUIPPED project manager, Queen’s University

Jennifer Medves, Margo Paterson, Cori Schroder, QUIPPED principal investigators, Queen’s University

Sarita Verma, QUIPPED principal investigator, U of Toronto

1 QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Canadian Evaluation Society, May 31 – June 3, 2009, Ottawa, Ontario

  Introductions ◦ Evaluation context: Stakeholders & Evaluator ◦ Developmental evaluation approach

 Developmental evaluation process ◦ Strategies & critical issues  Beginning, throughout, wrap up  Project outcomes

  Lessons Learned ◦ Evaluator & organizational perspectives

 Concluding thoughts & questions

2

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Project Queen’s University Inter-Professional

Patient-Centred Education Direction

 Goal: to promote interprofessional learning in the Faculty of Health Sciences at Queen’s University ◦  funded by Health Canada for 3 years to an inter-disciplinary team of PIs ◦ Operated as a dynamic organization within a dynamic context

3

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Dynamic organization Dynamic context   Characterized by ◦  distributed control ◦  ability to adapt to changes

in their environment

  Characterized by ◦  Influenced by

interconnections

4

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

•  Evidence •  Nature of organizational members’ interactions

•  shared responsibilities/roles •  Influence of political and cultural institutional aspects •  Influence of internal refocusing of vision

Evaluation

  Purposes: ◦ Primary: Accountability   Interim and final reporting to funders

◦ Emergent Secondary: Program Development  Ongoing use to inform decisions

5

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Dynamic organization Dynamic context

  has the potential to place unprecedented demands on the evaluation process

  may require the evaluator to maintain close contact to monitor and accommodate the changes within the evaluation process.

6

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

  Who are we? ◦ Organizational members: Principal investigators,

project managers, research assistants, administrative assistants. ◦  Changes in personnel during the project

  What was our focus? ◦ Meeting the accountability requirement of funder

  How did we interact with evaluator? ◦ Over 24 months:

  Interviews: 23 individual, 18 small group, 6 large group   306 emails, phone calls, and informal face-to-face exchanges

7

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

 How did I become the evaluator? ◦ AEG & luck

 How did I integrate my experiences into my evaluation approach? ◦ A decade of experience  Primarily accountability-focused  Experienced frustration with lack of use ◦ Increasingly participative to enhance use  Experienced satisfaction  Challenge: an evaluation context where the program outdistancing our initial evaluation design

8

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Unique characteristics: Represents a radical shift from traditional approach.

(a)  not specifying a time frame in which the evaluation is to be conducted,

(b)  not predetermining the evaluation purpose, and (c)  not describing the role of the evaluator as

objective and detached.

According to Patton (2006) relies on “a long-term partnering relationship between evaluators and

those engaged in innovative initiatives and development” to create and keep relevant the

focus for evaluative inquiry (p. 28).

9

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

 Beginning ◦ Ask specific questions, listen attentively & keep within time allotted  Open dialogue at initial meeting

◦ Create multiple opportunities for interactions & spend more time listening than talking  Respect for expertise & experience

◦ Distribute summaries following interactions  Member checks build confidence that voices are heard

◦ Pay attention to those that do not participate   Invest time for later involvement

10

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Evaluator Organization

  Upfront about approach

  Make involvement expectations clear

  Show responsiveness   Build confidence in

ability to meet needs

  Building trust   Communicating ◦  Voice expectations for

the evaluation ◦  Bring forward

contextual understandings

  Make known view of evaluator as external and “arms length”

11

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Throughout

 Maintain frequent interactions ◦  Increased use of email

 Create ongoing involvement opportunities ◦ Planning, tool development, analysis

 Use of an emergent design ◦ Responsive to opportunities

 Ongoing data sharing ◦ Access is maintained

 Communicate next steps ◦ Trust is maintained

12

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Evaluator Organization

  Needs are being met   Data accuracy   Interpretations from

everyone is heard   Consistent

involvement in all aspects

  Trust & communication are maintained

  Evaluation process adapts to contextual changes

  View of evaluator and role evolves

13

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Wrap up   Integrate opportunities for contributing to

development of report early ◦  Involvement in data interpretation, drafts

 Maintain focus on evolving needs of stakeholders ◦ Adapt to emerging thinking

 Create opportunities for reflection and future planning ◦ Thinking beyond the report

14

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Evaluator Organization

  Reports reflect data & shared

interpretations   Looking forward

and building on understandings of context & process

  No surprises in report   Report meets funder

timelines and requirements

15

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

From the final evaluation report:

 Overall, it has become clear that ◦ To become sustainable and to remain relevant to the needs of learners, faculty, and community clinicians and health care consumers, interprofessional education (IPE) requires a commitment of time, resources, and personnel.

16

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Three distinct impacts described the effects of the QUIPPED project in the final evaluation report:

 A creator of IP momentum.

 A developer of IP curriculum.

 A contributor to IP scholarship.

17

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

Key understandings from report:

  Participants should be knowledgeable about their own discipline-specific scopes of practice prior to engaging in IP opportunities that are of interest to them.

  Time and facilitation should be dedicated at the beginning of the IP opportunity for developing the small IP team.

  Feedback related to participants’ experiences should be solicited with the aim of informing subsequent implementations.

  The institutional context in which the IP opportunities take place should value IPE was an integral part of the curriculum and consequently should provide time, resources, and personnel to support IPE endeavours.

18

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

 Refocused organizational vision

  Increased communication

 Documented project development

  Integrated data into evidence-based decision making

 Broadened conceptions of use

 Responded to evolving understanding of evaluator role

19

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

 Build & nurture trust

◦  foster participatory environments

◦ acknowledge the working constraints of stakeholders

◦ be attentive to expectations of participation

◦ be open to adopting a role within the organizational structure that is mutually negotiated.

20

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

  Promote & maintain collaborative opportunities

◦  individual gain an in-depth understanding of their program and of their experiences.

◦ monitor and verify evolving understandings of the program and the influence of dynamic contextual elements.

21

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

 Maintain a focus on use

◦ Reflection is key for evaluator learning

◦ use a design that satisfies the evaluation purpose and remains relevant to evolving needs

◦ be mindful of individuals’ receptiveness of use beyond primary evaluation focus

22

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

 Understand the complexities of IPE

◦ Need for an evaluation approach that fit the organization and context

◦ Need for evaluator to undertake a role beyond a typical evaluator to fully understand the context

◦ Need for evaluator to listen, learn, and verify understandings

23

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

 Build and maintain trust & avenues for communication

◦ Needed to overcome previous experiences

◦ Use of listening and focus on ongoing communication

24

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

  Involvement of evaluator as an adjunct organizational member created new learning opportunities for everyone.

  Further empirical examples from both perspectives related to how evaluators establish credibility and acceptance in this role are required

25

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction

  Your thoughts? experiences?

Cheryl Poth cpoth@ualberta

26

QUIPPED Queen’s University Inter-Professional Patient-Centred Education Direction