Embed Size (px)
Citation preview
We Implemented a Great
Performance Management
Framework Yet Something
was Missing
1
Presenters
Annette Sonneveld
Jocelyn Edwards
Afroz Sajwani
• None of the presenters at this session have received
financial support or in-kind support from a commercial
sponsor
• None of the presenters have potential conflicts of interest
to declare
Disclosure of Commercial Support
2
Today’s Agenda
3
Items Time
Welcome 5 minutes
Performance Management Framework and
Systems
20 minutes
The Missing Piece 45 minutes
Summary of Workshop and Questions 15 minutes
TOPHC Evaluation of the Workshop 5 minutes
Introduction of the Presenters and
Toronto Public Health
4
• Toronto Public Health (TPH) is responsible for the health and well-
being of all 2.9 million residents
• TPH has focused on protecting and promoting the health of Toronto
residents since 1883
Workshop Objectives
1. To discuss the approach to developing a performance
management system at Toronto Public Health
2. To explain the importance of developing a theory of
change for a performance management system
3. To gain tools to support the development and
integration of a performance management system
5
How do we know how well we are
Performing?
We live in the world of information overload
6
What is Performance Management?
Performance Management
The ongoing process of setting
performance standards,
systematically measuring
performance over time, and
making improvements based on
data
Performance Measurement
The development and use of
performance measures to assess
achievement of performance
7
Performance Management is Embedded
in our Work
1. Ontario Public Health Standards 2018
2. City of Toronto Strategic Plan 2013-2018
3. TPH Strategic Plan 2015-2019
8
High Functioning Performance
Management Systems
• Are integrated into daily practice at all levels
• Have committed leadership
• Have engaged staff
• Are customer focussed
9
Discussion Question
What does performance
management look like in your
organization?
Please share an example with your neighbour
10
TPH’s Performance Management
Framework
11
Component One: Planning & Reviewing
Standards
12
Steps
1. Examine standards and
mandated performance
measures
2. Develop a logic model
Logic Model Template
13
Component Two: Developing
Performance Measures
14
Steps
1. Identify or develop all performance
measures
2. Narrow down your list of
performance measures to the
critical few measures by assessing
the quality of performance
measures
3. Document details for each
measure in a data dictionary
4. Set performance targets, as
needed
Developing Performance Measures
Using your logic model, convert:
• Activities and outputs output measures
• Process outcomes process/efficiency measures
• Short, intermediate and long term outcomes
outcome/effectiveness measures
Examples:
15
Logic Model Element Performance Measure
Activity:
“conduct workshops“
Output measure:
"number of workshops conducted"
Process outcome:
"client satisfaction”
Process measure:
"percent of satisfied clients"
Short term outcome:
"increased awareness of
community resources"
Outcome measure:
"percent of clients with increased
awareness of community resources"
Data Dictionary Template
16
A Guide for Assessing the Quality of
Performance Measures
17
A Guide on Target Setting
18
Component Three: Collecting, Analyzing, &
Reporting Performance Measures
19
Steps
1. Develop a data collection plan
2. Develop or modify data collection
tools
3. Collect data on your performance
measures
4. Clean, analyze, display and
interpret your data
5. Report your findings and plan for
action
Data Collection Plan Template
20
Component Four: Improving
Performance
21
Steps
1. Identify appropriate
improvement strategies
2. Update work plan
3. Do the improvement work
Types of Improvement Strategies
• Informal solutions
• Obvious, readily identified and easily implemented solutions
• Program Evaluation
• Conduct a systematic, individual study to produce information
to assess how well, and why, a program is working
• Quality Improvement
• Identify areas for improvement and implement solutions
through the use of deliberate and defined improvement
processes
Document your improvement activities in your work plan!22
Operationalization of the
Framework
23
1. Trainings and workshop
(certification)
2. Guidebook
3. Internet page
4. Guidelines and procedures
5. Consultative support, as
requested
6. Lead QI and evaluations
projects, as requested
Why Engage in Performance
Management?
• Better alignment of public health programs
• Better measurement for improvement
• Emphasis on quality rather than quantity
• Greater accountability and transparency for stakeholders
• Better return on dollars invested in public health
• Better outcomes of public health programs
24
Something was Missing
25
We implemented a great Performance Management
Framework yet something was missing...
…Theory of Change (TOC) was added
The link between
what we are doing
and why we are
doing itA formal method for
the collection and
use of evidence
Addition of Theory of Change
26
• Added to component one
of the framework
because it is an
important planning
method
• Added before logic
model because it informs
the development of a
logic model
What is a Theory of Change?
A TOC is a narrative that provides an overall explanation
of the relationship between the planned program and the
desired results
27
Why Develop a Theory of Change?
• Helps you describe the need you are trying to address
• Explains why you think your interventions will lead to the
outcomes you want
• Explores whether your plans are supported by evidence
• Makes your assumptions explicit and therefore testable
• Helps you consider and articulate the conditions or resources
that you need
• Helps you articulate and scrutinize your ideas
A Theory of Change helps to articulate the
reasoning behind a program
28
When to Develop a Theory of
Change?
• When planning for a new program or project
• When revisiting or redesigning an existing program
29
Logic Model and Theory of Change
I already have a logic model… how is this different?
30
A Logic Model…
• Tells us how activities lead to
outcomes
• Represents intention
• Tells us what we are doing
• Lists program activities
• Lists anticipated outcomes
A TOC…
• Tells us why we believe activities
will lead to outcomes
• Explains intention
• Tells us if we are doing the right
thing
• Explains the rationale behind the
choice of program activities
• Spells out the assumptions that
underlie the intended outcomes
31
Logic Model and Theory of Change
Applying the Concepts of Logic Model
and Theory of Change - Activity
Which is Which?
32
Theory of Change Elements
• Context
• Precursors
• Solutions
• Assumptions
• Enablers
• Evidence
33
Theory of Change Elements
Context
• Problem: The problem being addressed and its root causes
• Target population: Who is affected by the problem?
• Ultimate outcome: What is the ultimate change you want to
see?
34
Theory of Change Elements
Precursors (also known as preconditions)
Conditions that must exist in order for the next precursor in
the pathway to be achieved
35
Theory of Change Elements
36Outcomes Chain Example from The Community
Builder’s Approach to Theory of Change, page 4.
Ask
yourself:
What has to
happen in
order for
this to be
achieved?
Theory of Change Elements
37
PRECURSORS
What results for the target population?
SHORT
Learning
Changes in
• Awareness
• Knowledge
• Attitudes
• Skills
• Opinion
• Aspirations
• Motivation
• Behavioral intent
INTERMEDIATE
Action
Changes in
• Behaviour
• Decision-making
• Policies
• Social action
LONG-TERM
Conditions
Changes in
• Conditions
• Social (well-being)
• Health
• Economic
• Civic
• Environmental
Theory of Change Elements
Solution
• Specifies what can be done to solve or reduce an
identified problem
• Also known as “intervention” or “strategy”
38
Children meet their optimal health and
developmental potential
Children have families
and caregivers that
provide safe, nurturing
and positive environments
Child-friendly healthy public policies
are in place
Families and children
have social networks
that provide safe,
nurturing and positive
environments
Families and children
live in
neighbourhoods that
are safe, nurturing,
positive and inclusive
Parents and
caregivers
have well-
being
Health
promoting
infrastructu
re is in place
(e.g., parks,
playgrounds)
Families have
knowledge and
skills to
positively
influence HCD
EY staff are
knowledgeable and
skillful in providing
inclusive services
related to HCD
Families and
caregivers have
appropriate
financial supports
to positively
influence HCD
Policy makers are aware and
accept the relevant policy needs
Families access
inclusive services
and resources
related to HCD
Families are aware
of services and
resources related to
parent/ caregiver
well-being
Families are aware
of inclusive services
and resources
related to HCD
Community partners,
agencies, academic
institutions and the
general public are aware
of and advocate for
child-friendly health public
policy
EY advocates and
develops policies
in collaboration with
other TPH
directorates (HPP)
Families access
services and resources
that enhance the well-
being of parents and
caregivers
EY trains its
staffEY provides
referrals to
services that
enhance
parent/caregiver
well-being
EY supports
community
agencies to
provide HCD
services
EY provides
direct
services and
resources
related to HCD
Child
care is
safe,
affordable
and
effective
Streets
are free
of
violence
and crime
Stores
provide
healthy
food at
affordabl
e cost
Houses
are safe,
affordable
and
adequate
39
40
Children have families and caregivers
that provide safe, nurturing and positive
environments
Parents and
caregivers have
well-being
Families have
knowledge and
skills to positively
influence HCD
Families and caregivers
have appropriate financial
supports to positively
influence HCD
Families are aware of
services and resources
related to parent/ caregiver
well-being
Families access services
and resources that
enhance the well-being of
parents and caregivers
EY provides
referrals to services
that enhance
parent/caregiver
well-being
Theory of Change Elements
Assumptions
• The underlying beliefs about how a program will work, the
people involved, and the context
• We often make assumptions about:
• The context within which a program operates
• What kind of strategies are best
• How to best deliver interventions
• How strategies will lead to precursors
• Assumptions should be backed up with evidence41
Theory of Change Elements
Enablers
Conditions or resources that need to be present to allow a
program to succeed and that you believe already exist and will
not be problematic
• Internal enablers: need to exist within your organization for
the TOC to work, and are mostly within your control
• External enablers: need to exist in the external environment
for a TOC to work, and are often beyond your immediate
control
42
Theory of Change Elements
Evidence
• Available body of facts or information
• Provides models of action and best practices to draw on
• Think about the evidence that contradicts your assumptions
and modify what you will do to reflect what the evidence is
telling you
• Possible sources of evidence: Published scientific literature,
grey literature, population health surveillance data, program
evaluation, Quality Improvement project, stakeholder
perspectives and experiences
43
Applying the Theory of Change
Elements - Activity
Supervised Injection Services (SIS)* Now known as Consumption and Treatment Services *
44
Working through the Example
Context
• Problem
• Individuals who are not currently engaged with other addiction
services and are more likely to be homeless, engage in public
drug use, and are at high risk for overdoses and blood-borne
infections
• Target Population
• People who inject drugs, in particular people who are at high
risk of harm due to social location or other circumstances
• Ultimate Outcome
• The SIS aims to reduce overdoses and blood-borne infections
45
Working through the Example
Precursors
• SIS services aim to reduce clients' behaviour, such as
sharing injection supplies and not using sterile water for
injection
• The SIS aims to reduce the transmission of blood-borne
infections such as hepatitis and HIV
46
Working through the Example
Solution
• The SIS will provide a range of health services on site
• The SIS will provide safer injection education on site
• Clients will be provided with naloxone and overdose
training for offsite administration
• The SIS will include a chill-out room for clients after
injection
47
Working through the Example
Assumptions
• People will practice safer injection techniques because
we've provided them with training
• Providing safe injection services offers people who use
drugs a stable and controlled environment where other
activities can be offered, thereby enabling us to address
multiple needs
• With this type of support, individuals can be equipped to
practice safer drug use
48
Working through the Example
Enablers
• Staff who work in the SIS will need to value the service
and have a positive attitude toward its clients.
• Success of SIS depends on attracting people who inject
drugs. We need to get them in the door.
49
Working through the Example
Evidence
• It has been found that SISs reduce public drug use, lessen
publicly discarded injection supplies, and reduce crime in the
surrounding area
• Research on the SIS in Sydney, Australia showed that 75% of
the people registered at the SIS had not previously used any
of the local health services
• Three cross-sectional surveys conducted in Europe between
1990 and 2001 found an increase in condom use and
decreases in the levels of acceptance of sharing injection
supplies and the actual sharing of supplies among those
frequenting an SIS50
Developing a Theory of Change
• Identify a workgroup who will consult on the development of
a TOC
• Identify a project lead who will gather and synthesize
evidence, facilitate the meetings, aggregate findings from the
meetings, and write the TOC
• Use the TOC Template in the Workbook to guide you as you
write narrative
• Use the TOC Checklist in the Workbook to review your TOC
after you have written it
• Don’t forget to get approvals and buy-in from your
stakeholders
51
Challenges to Theory of Change
Implementation
• There is very little literature on TOC in general and
specifically in relations to health
• TOC is an evolving practice and is not yet prescriptive
• Addition of TOC came after the implementation of the
framework – staff needed time to understand and adopt
• Confusion regarding the difference between a logic
model and TOC
• TOC development requires time and resource
commitment
• TOC development requires expertise
52
Challenges to Performance Management
Implementation at TPH
• Different uptake of the framework among
program teams
• Clarifying roles regarding the development and
implementation of Performance Management
Systems
• Lack of dedicated staff
• Competing priorities
53
Conclusion
• Entire PM framework supports and uses
evidence to inform public health practice
• Documents created by the program in the
four components of the framework need to
be reviewed regularly as new information
and learnings occur
• TOC is a valuable method that can tell you
why you are doing what you are doing
Addition of TOC to our Performance
Management Framework formalized the
collection and use of evidence54
![Magnetic Contactors & Thermal Overload Relays...Structure of Thermal Overload Relay. Overview . 1] Performance . Compact size & Easy connection . Improved Arc Protection from sealed](https://img.pdfslide.us/doc/110x75/5e7c728c896df074b5229847/magnetic-contactors-thermal-overload-relays-structure-of-thermal-overload.jpg)
