Upload
jackie72
View
8.197
Download
0
Embed Size (px)
Citation preview
KPIDesigning Key Performance Indicators
© 2006 Dr Josephine Palermo & Martin CarrollTraining Module 03 v8
A Ministry of Higher Education &Oman Accreditation Council joint initiative
Module ObjectivesModule ObjectivesBy the end of this module, participants should:By the end of this module, participants should:1.1. Understand the role of KPI in a quality Understand the role of KPI in a quality
assurance system.assurance system.2.2. Understand the attributes of well designed KPI.Understand the attributes of well designed KPI.3.3. Recognise different types of KPI and the Recognise different types of KPI and the
consequences of using different types.consequences of using different types.4.4. Develop KPI for given goals.Develop KPI for given goals.5.5. Be able to constructively critique a KPI against Be able to constructively critique a KPI against
its intended purpose.its intended purpose.
Session OutlineSession Outline1.1. A model for designing good KPI:A model for designing good KPI:
– KPI in the planning hierarchyKPI in the planning hierarchy– Objectives, indicators, measures, targets and resultsObjectives, indicators, measures, targets and results
2.2. Conceptually understanding KPI:Conceptually understanding KPI:– Why use KPI?Why use KPI?– ADRI and KPIADRI and KPI– Lag and Lead indicatorsLag and Lead indicators
3.3. Tools for critically analysing the design of KPI:Tools for critically analysing the design of KPI:– SMARTER KPISMARTER KPI– Tips for good KPITips for good KPI
4.4. WorkshopsWorkshops
1.1.A model forA model for
designing good KPIdesigning good KPI
KPI in the Planning HierarchyKPI in the Planning HierarchyMission
Goal
Objective
Indicator
Measure
Target
Result
Strategy
A one-to-one relationship (e.g. one measure has one target)
A one-to-many relationship (e.g. one goal may have many strategies)
KPI in the Planning HierarchyKPI in the Planning HierarchyMission
Goal
Objective
Indicator
Measure
Target
Result
Strategy
KPI may be thought of as essential elements of the overall planning and monitoring system.
They are derived from the College’s Mission and help track performance towards accomplishing the Mission.
A KPI is not a single component, but an integrated collection of integrated collection of componentscomponents.
Indicator
Measure
Target
Result
Designing Good KPIsDesigning Good KPIs• Objectives: Objectives: what are we what are we trying to achievetrying to achieve??
– May be more than one indicator for each objectiveMay be more than one indicator for each objective– Each objective will have strategies on how to achieve themEach objective will have strategies on how to achieve them
• Indicators:Indicators: whatwhat are you going to measure? are you going to measure? – used to assess the present state of progress and to suggest used to assess the present state of progress and to suggest
an appropriate course of action. an appropriate course of action.
• Measures:Measures: howhow are you going to measure it? are you going to measure it?– can be qualitative or quantitative data related to inputs, can be qualitative or quantitative data related to inputs,
processes or outputs.processes or outputs.
• Targets:Targets: what is the what is the desired resultdesired result??– can be minima targets, stretch targets or a combination.can be minima targets, stretch targets or a combination.
• Results:Results: what have you what have you actually achievedactually achieved??
• ObjectiveObjective– High Quality TeachingHigh Quality Teaching
• Indicator (one of several)Indicator (one of several)– Student satisfaction with the teaching they experienceStudent satisfaction with the teaching they experience
• MeasureMeasure– mean student response, per class, to the question, mean student response, per class, to the question,
“overall, how satisfied are you with this teacher?”“overall, how satisfied are you with this teacher?”– on a 1 to 5 Likert-type scaleon a 1 to 5 Likert-type scale
• TargetsTargets– At least 3.5 on a 1 to 5 scale (where 5 is the best): ORAt least 3.5 on a 1 to 5 scale (where 5 is the best): OR– Best in class compared with benchmark partnersBest in class compared with benchmark partners
Constructing Good KPIsConstructing Good KPIs
2.2.ConceptuallyConceptually
understanding KPIunderstanding KPI
Why use KPIs?Why use KPIs?• Key Performance Indicators help us to know Key Performance Indicators help us to know
whether we are succeeding in our mission. whether we are succeeding in our mission.
• They are an important management tool for They are an important management tool for tracking progress against strategic goals.tracking progress against strategic goals.
• Working with KPIs encourages system thinking.Working with KPIs encourages system thinking.– in most systems 85% of problems can usually be in most systems 85% of problems can usually be
attributed to the system and 15% to the individual attributed to the system and 15% to the individual (Juran)(Juran)
• KPIs direct and prioritise behavior towards KPIs direct and prioritise behavior towards achievement of the vision, goals and objectives.achievement of the vision, goals and objectives.
APPROACH
RESULTSDEPLOYMENT
IMPROVEMENT
ADRI and KPIsADRI and KPIs• Each phase of the ADRI process can involve KPIs.Each phase of the ADRI process can involve KPIs.
• Examples of ‘Approach’ KPIs include:Examples of ‘Approach’ KPIs include:– Staff training program participation; Student recruitment by targetStaff training program participation; Student recruitment by target
• Examples of ‘Deployment’ KPIs include:Examples of ‘Deployment’ KPIs include:– Internet uptime; Teaching space usage; Student class attendanceInternet uptime; Teaching space usage; Student class attendance
• Examples of ‘Results’ KPIs include:Examples of ‘Results’ KPIs include:– Student pass rates; Program accreditationStudent pass rates; Program accreditation
• Examples of ‘Improvement’ KPIs include:Examples of ‘Improvement’ KPIs include:– Number of review recommendations resolved on time; Policy Number of review recommendations resolved on time; Policy
updatesupdates
Balanced Scorecard KPIsBalanced Scorecard KPIs• Kaplan & Norton designed a ‘Balanced Scorecard’, Kaplan & Norton designed a ‘Balanced Scorecard’,
with ‘lag’ and ‘lead’ KPIs. Useful for managing large with ‘lag’ and ‘lead’ KPIs. Useful for managing large numbers of KPIs in a complex organisation.numbers of KPIs in a complex organisation.
• ‘‘Lag’ indicators tell how well a system performed. Lag’ indicators tell how well a system performed. Mostly used in ADRI ‘Results’ and ‘Deployment’ Mostly used in ADRI ‘Results’ and ‘Deployment’ phases. Examples include:phases. Examples include:
– Student employment rates; research publications; number of prizesStudent employment rates; research publications; number of prizes
• ‘‘Lead’ indicators help predict system performance. Lead’ indicators help predict system performance. Mostly used in ADRI ‘Approach’ and ‘Improvement’ Mostly used in ADRI ‘Approach’ and ‘Improvement’ phases. Examples include:phases. Examples include:
– Staff training; Investment in advertising; Student-computer ratios Staff training; Investment in advertising; Student-computer ratios
Lag, Lead & the Planning HierarchyLag, Lead & the Planning HierarchyMission
Goal
Objective
Indicator
Measure
Target
Result
Strategy
Put another way, ‘lag’ KPI focus on the extent to which outcomes-based objectives have been achieved.
‘Lead’ KPI focus on the extent to which strategies for achieving the objectives are being successfully implemented.
Therefore we can structurally separate lag and lead KPI in the planning hierarchy.
Lag, Lead & the Planning HierarchyLag, Lead & the Planning Hierarchy
Strategy
Indicator
Measure
Target
Result
Lag KPI setLead KPI set
There may be separate KPI for objectives and their associated strategies.
Provided that objectives are outcomes-based,and strategies support objectives, then:
Mission
Goal
Objective
Indicator
Measure
Target
Result
3.3.Tools for criticallyTools for critically
analysing KPIanalysing KPI
SSMMAARRTTEERR
pecificeasurablelignedealisticimelythicalecorded
Other Tips for Good KPIsOther Tips for Good KPIs1.1. Have them!Have them!2.2. Align them to standards.Align them to standards.3.3. Test them for validity and reliability.Test them for validity and reliability.4.4. Discuss and review them. Are they really Discuss and review them. Are they really keykey? ?
Don’t overestimate their importance!Don’t overestimate their importance!5.5. Differentiate between lag and lead indicators.Differentiate between lag and lead indicators.6.6. Benchmark them.Benchmark them.7.7. Do something with them. KPI should lead to change.Do something with them. KPI should lead to change.8.8. Keep them simple – but not too simple!Keep them simple – but not too simple!9.9. Balance focus on positive and negative news (i.e. Balance focus on positive and negative news (i.e.
opportunities for improvement).opportunities for improvement).
Humanise the ProcessHumanise the Process• Address staff concerns and ideas about Address staff concerns and ideas about
performance management.performance management.• Address concerns about the uncertainty of the data.Address concerns about the uncertainty of the data.• Regardless of the topic, keep performance review Regardless of the topic, keep performance review
processes fair and transparent.processes fair and transparent.• Place performance review within a context of Place performance review within a context of
learning.learning.• Enable a two way dialogue about performance.Enable a two way dialogue about performance.• Remember to highlight good news!Remember to highlight good news!
Workshop ExampleWorkshop ExampleIndicators for Learning and TeachingIndicators for Learning and Teaching
Objectives
1. Students will learn to solve problems.
2. The program will be viable.
3. The program will be relevant to industry and employers.
ExampleExample• For each of these three
objectives, strategies are provided.
• KPIs are then constructed using the model:indicator → measure → target
• KPIs are then tested using SMARTER
Workshop Example #1Workshop Example #1Indicators for Learning and TeachingIndicators for Learning and Teaching
Objective Strategy KPIs
Students learn to solve problems
Design and deliver problem based curriculum
Indicator: courses include learning outcomes that relate to problem solving
Measure: number of courses each semester with problem solving learning outcomes included in institutional course guides
Target: 100% of new courses; 100% of all courses by 2010
Design and implement problem based assessment of student learning
Indicator: courses include PBL assessment exercises
Measure: student pass rate/semesterTarget: 80% by 2007
SPECIFIC?Yes. The indicator is tightly focused on the
single issue of courses design.
MEASURABLE?Yes. The number of courses that comply
can easily be counted.
ALIGNED?Yes. The measure
relates directly to the indicator; the target
relates directly to the measure.
REALISTIC?Yes. The KPI has been made
realistic by separating new courses, which can implement
the strategy directly, from existing courses, which will
need to be changed.
TIMELY?Yes. The measure is
each semester, allowing for current
data and trend data. The target is time-
specific. ETHICAL?There are no obvious ethical concerns with collecting and using
this information.
RECORDED?The target implies that trend data will
need to be kept and analysed until at
least 2010.
TIMELY?Yes. The measure is
each semester, allowing for current
data and trend data. The target is time-
specific.
REALISTIC?Yes. The target does
not appear unachievable.
SPECIFIC?Yes. The indicator is tightly focused on the
single issue of assessing student’s problem solving
abilities.
ALIGNED?Yes. The measure
relates directly to the indicator; the target
relates directly to the measure.
MEASURABLE?Yes. Student pass rates are
easily measured.
RECORDED?The target implies that trend data will need to be kept and analysed
until at least 2008.
ETHICAL?There are no obvious ethical concerns with collecting and using
this aggregated information.
These are SMARTER KPIs
These are SMARTER KPIs
Objective Strategy KPIs
The program will be viable.
Attract the best students by direct marketing of programs to school leavers.
Indicator: Student satisfaction.
Measure: (a) Student pass rates(b) Excellent program
Target:(a) 100%(b) All staff qualified
Workshop Example #2 Workshop Example #2 Indicators for Learning and TeachingIndicators for Learning and Teaching
TIMELY?No. There is no time definition
provided for this KPI
REALISTIC?No. Any national or
international benchmark will show that the target
of 100% pass rates is not realistic.
SPECIFIC?No. ‘Student
satisfaction’ is very broad. Satisfaction with
exactly what?
ALIGNED?No. Student satisfaction and student pass rates are not the same thing.MEASURABLE?
Yes. Student pass rates are
easily measured.
RECORDED?There is no indication on how this KPI will be
recorded, reported and stored.
ETHICAL?Maybe not. The
target of 100% may encourage staff to
mark too softly.
TIMELY?No. There is no time definition
provided for this KPI
REALISTIC?No. If the measures and
targets cannot be measured and do not
align, then how can they be realistic?
SPECIFIC?No. ‘Student
satisfaction’ is very broad. Satisfaction with
exactly what?
ALIGNED?No. Staff qualifications
is not necessarily related to whether the program is excellent.
MEASURABLE?No. What is the
means for measuring that the
program is ‘excellent’?
RECORDED?There is no indication on how this KPI will be
recorded, reported and stored.
ETHICAL?No. This KPI implies that staff
can be held accountable for ‘student satisfaction’ and the ‘excellence’ of the program
and by virtue of their qualifications. The case for
this is not justified here.
These are not SMARTER KPIs
These are not SMARTER KPIsLet’s improve them…Let’s improve them…
Workshop Example #2 Workshop Example #2 Indicators for Learning and TeachingIndicators for Learning and Teaching
Goal Strategy KPIsThe program will be viable.
Attract the best students by direct marketing of programs to school leavers.
Indicator: Student program preferences by academic performance.
Measure: (a) Percentage of enrolled students who
listed the program as their 1st preference (via Admission Centre)
(b) Proportion of 1st preference students in national top 10% by academic rank
Target:(a) More than 50%(b) At least 20% and up to 40%
(assuming four competitor programs in Oman)
Workshop Example #3 Workshop Example #3 Indicators for Learning and TeachingIndicators for Learning and Teaching
Goal Strategy KPIs
The program will be relevant to industry and employers.
Establish Industry Advisory Boards (IABs) for each program. Invite key employers to be members.
Indicator: Graduate employability rateMeasure: % graduates in full time
employment in the industryTarget: above national averageIndicator: Acceptance of program by
industryMeasure: (a) Professional accreditation of
program by 2009(b) Positive feedback from employers
and industryTarget: (a) Professional accreditation passed(b) Positive IAB Program Review
1.1. Teaching and Learning (done)Teaching and Learning (done)
2. Student Outcomes
3. Educational Support Resources
4. Research and Research Training
5. Community Service / Engagement
6. Organisational Leadership
7. Staffing
8. Institutional Infrastructure
9. Information and Communication
Workshop Instructions Workshop Instructions Select your topicSelect your topic
ActivityActivity• Select an area (from the list) that you
are currently working on.• What are two or three main objectives
for that area?• What strategies could you have in
place to achieve each objective?• Construct appropriate KPIs, including
Indicators → Measures → Targets.• Check them. Are your KPI SMARTER?• Are they Lag or Lead KPI?• When you have finished (and ONLY
then!), try clicking each area for further examples of good KPI.
• When you are ready you may EXIT
Workshop ActivitiesWorkshop ActivitiesStudent Outcomes (a)Student Outcomes (a)
Objective Strategy KPI
Students will achieve high rates of success in their learning
Progress Scheme identifies ‘at risk’ students, by Department, and invites them to participate in student learning support services.
Indicator: Student progress rates
Measure: Percentage of students progressing to the next level of study without having to repeat any units; differentiated by those who participated in the Progress Scheme and those who didn’t.
Target: Improvement trends over three years for both cohorts, but steeper for Progress Scheme students.
Workshop ActivitiesWorkshop ActivitiesStudent Outcomes (b)Student Outcomes (b)
Objective Strategy KPI
Students will be highly satisfied with their career choices
Academic Advisors are available for career planning
Indicator: Graduate recruitment by employment preference
Measure: Percentage of graduates in preferred employment within 12 months of graduating
Target: At least 90%
Returnto list
Workshop ActivitiesWorkshop ActivitiesEducational Support ResourcesEducational Support Resources
Objective Strategy KPI
The library supports learning outcomes
The library has developed a ‘one-stop-shop’ help desk for increased support to academic staff and students
Indicator: Staff and student satisfaction with the help desk
Measure: Library User Survey asks staff & students to rate their satisfaction with and relevance of helpdesk features. Gap measure is developed.
Target: The features with highest relevance also have equally high satisfaction
IT services support learning outcomes
Actively communicate with College departments to identify IT requirements and problems with IT services
Indicator: Client (Dept) satisfactionMeasure: Academic staff satisfaction
results on customer feedback surveys
Target: Average satisfaction is above 4 on a 1-5 likert type scale
Returnto list
Workshop ActivitiesWorkshop ActivitiesResearch and Research Training (a)Research and Research Training (a)
Objective Strategy KPI
We will improve the level of research income via external grants
Staff can access institutional seed grants money to encourage external applications for research funding.
Indicator: External grant applications
Measure: (a) Proportion of FTE Teaching+
Research staff submitting an external grant application (Lead)
(b) Proportion of applications successful (Lag)
Target: (a) 50% of all FTE staff submitting an
external grant by 2010.(b) 20% success rate
Workshop ActivitiesWorkshop ActivitiesResearch and Research Training (b)Research and Research Training (b)
Objective Strategy KPI
We will improve the postgraduate research experience
System of bi-annual progress reporting for HDR students that includes follow up and evaluation processes
Indicator: Students successfully completing postgraduate research degrees in a timely fashion
Measure: Proportion of HDR students graduating each year against total number of HDR students scheduled to complete each year.
Target:(a) at least 30 by 2010(b) Over 75% scheduled students
graduate
Returnto list
Workshop ActivitiesWorkshop ActivitiesCommunity Service / Engagement (a)Community Service / Engagement (a)
Objective Strategy KPI
We will provide leadership to our local community
Staff are encouraged to nominate for community level boards and committees
Indicator: Participation on community boards / committees
Measure: Proportion of FTE staff (academic + non academic) who hold positions on boards and committees)
Target: At least 50%
Click here to return to list
Returnto list
Workshop ActivitiesWorkshop ActivitiesCommunity Service / Engagement (b)Community Service / Engagement (b)
Objective Strategy KPI
We will increase the number of new enrolments from students in the local region
Schools liaison program is conducted with schools in the College’s catchment area.
Indicator: Number of commencing students from our region
Measure: (a) Proportion of local region new
enrolments against College’s total new enrolments
(b) Proportion of local region new enrolments against total local region school leavers
Target: (a) At least 30% in 2007; with an
increasing trend of 3% every year for next three years.
(b) At least 50%; and more than any other local College.
Returnto list
Workshop Activities Workshop Activities Organizational LeadershipOrganizational Leadership
Objective Strategy KPI
Strategic planning is practiced across all boards and committees
Every Board and Committee is required to develop a Plan that is aligned to the College Strategic Plan and includes performance indicators
Indicator: Board and Committee Plans
Measure: Proportion of Boards and Committees with a current Plan that complies with the College’s template
Target: 100%We will create and maintain a positive organisational climate
The leadership team meets monthly to identify, discuss and respond to any issues raised by staff about the organisational climate, and to review progress on previous responses.
Indicator: Staff satisfaction with their organisational climate.
Measure: Mean staff responses to questions about climate.
Target: Mean satisfaction is at least 4 on a 1-5 Likert-type scale.
Returnto list
Workshop ActivitiesWorkshop ActivitiesStaffing (a)Staffing (a)
Objective Strategy KPI
People are placed in meaningful jobs
Staff are recruited into positions that are related to goals in the College Strategic Plan.Exit interviews are conducted with staff who leave the organisation.
Indicator: Staff turnover due to poor recruitment practices
Measure: (a) expenditure related to staff
turnover (eg. Recruitment expenses into recurrent jobs, retrenchment pay)
(b) Number of staff who resign each year
Target: (a) No more than 0.5% total
annual budget(b) Reduce by 3% per year
Returnto list
Workshop ActivitiesWorkshop ActivitiesStaffing (b)Staffing (b)
Objective Strategy KPI
Staff skills and competencies are developed on the job
Professional development is a high priority within the organisation. All staff have access to a professional development program.
Indicator: Staff development activity
Measure: Annual percentage of all staff who participated in professional development
Target: At least 60%
Returnto list
Workshop ActivitiesWorkshop ActivitiesInstitutional InfrastructureInstitutional Infrastructure
Objective Strategy KPIsSound financial management is practiced
All budget centre managers are trained in the College’s financial management processes.
Indicator: Expenditure against budget.
Measure: Monthly variance of expenditure against budget; reported by account code.
Target: Expenditure no more than 5% over or 10% under budget.
Effective space management supports learning outcomes
Space planning is incorporated into the Colleges Academic Plan
Indicator: Occupancy Adjusted for Usage Measure: The total amount of people using
the room over the total capacity of the teaching space.
Target: Usage at least 60% aggregated across all teaching spaces
Returnto list
Workshop ActivitiesWorkshop ActivitiesInformation and CommunicationInformation and Communication
Objective Strategy KPIEffective marketing maintains the high profile of the institution
Media and Marketing Strategy clearly targets media outlets
Indicator: Level of media awareness of the institution
Measure: (a) Number of ‘hits’ in targeted media
outlets (eg newspapers, radio)(b) Ratio of positive to negative ‘hits’Target: (a) increasing over time(b) Greater than 60%
Messages about quality assurance from management are consistently deployed to staff
Heads of Depts hold regular staff meetings where QA issues are discussed
Indicator: Evidence of discussions about QA policies and strategies in staff meeting minutes
Measure: Number of times QA issues are addressed in minutes
Target: At least once per month
Returnto list
KPIDesigning Key Performance Indicators
Congratulations on completing thistraining module.