The Pathfinder Project Identifying Outcomes

PATHFINDER

Guidance on Outcome Focused Management

Building Block 1 : Identifying Outcomes

Version 2.1, July 2003

This document provides guidance for agencies integrating outcome information into their decision-making processes. The document was produced by New Zealand’s Pathfinder Project.

More Pathfinder guidance documents are available on http://io.ssc.govt.nz/pathfinder.

We hope other outcome-based initiatives continue to develop the material presented in this suite.

The Pathfinder Project Page 1 Identifying Outcomes

Building Block 1 : Identifying Outcomes

Agencies’ use of inconsistent definitions for their programs’ measures could hamper decision-makers’ use of data collected from those measures in planning, comparing performance,

and reporting on performance achieved.

Implementing Results-Based Management: Lessons from the Literature. Office of the Auditor General of Canada, 2001

Purpose Statement

1 This Building Block covers the process of identifying outcomes linked to your actions, refining definitions, and defining measures. This is the first of five Pathfinder Building Blocks that provide managerial and technical information for the development of outcome based management in government agencies. The Building Block covers the first step of the continuous improvement cycle (below).

2 The advice is intended mainly for managers and staff responsible for design and implementation. It includes a management overview for senior executives. This chapter covers the identification of the ‘vital few’ outcomes. This is the essential first step that must be completed before a department sets about developing its outcome indicators.

3 Once management have agreed the ‘vital few’ outcomes for their agency or, in the case of a diversified agency their business units, then they are ready to proceed to identifying outcome measures. Good definitions of the vital few outcomes drive the usefulness of all subsequent work. A poorly defined or inappropriate outcome is of little use in decision-making, and at worse could result in decisions that have an adverse impact on the target group(s) or entities.

4 A well defined outcome is tightly linked to the mission and actions of the organisation, and guides decisions on future action.

Management Overview

5 Outcomes are defined and measured to enhance management decision-making. The first step in developing an outcome based decision-making system is the identification of the ‘vital few’ outcomes. At best these are outcomes that:

• are well aligned with the agency’s (or business unit’s) mission or purpose;

• are tangibly linked to outputs and inputs (the things an agency manages);

Continuous Improvement

Where are we? (Did we get there?) Where do we want to go?

How did we do? How do we get there?

Identify Outcomes;

Environmental Scan

Measure Outcome Indicators

ID Major Areas for Change (& Goals)

Assess Outputs

& Impact

Identify Options; Deliver ‘Best’ Interventions

The Pathfinder Project Page 2 Identifying Outcomes

• measure the benefits experienced by clients or entities the agency manages;

• support critical business decisions, including resource decisions; and

• can be measured using reliable data, obtainable at a reasonable cost.

Identifying The Vital Few Outcomes

6 Managing for results may require an agency or sector to measure and manage different outcomes in different business units. This may mean defining a hierarchy of outcomes spanning inter-agency, agency and sub-unit outcomes. So a first step is to identify the level(s) of Government or level(s) within the agency to which outcome information should relate (Exhibit 1).

Exhibit 1. Finding Your place in the Hierarchy

7 The agency’s mission and services should focus management’s attention on a small number of crucial outcomes. At any level of organisation these outcomes should specify the set of results expected from core services, and identify the target group or entity those services are designed to benefit. The US General Accounting Office refers to these outcomes as the ‘vital few’. Major business units may need to identify how their ‘vital few’ outcomes drive the agency’s ‘vital few’ outcomes.

8 Selecting the ‘vital few’ outcomes underpins all later steps in establishing a results-driven organisation. While compromises can be made to make outcomes measurable, improve timeliness or work around data shortfalls, they should still include the ‘vital few’ outcomes that are difficult to measure. Unmeasured outcomes can be used in robust conceptual (ex ante) processes, e.g. identifying good outputs.

9 The ‘vital few’ outcomes can be developed from first principles, but can also be found on many web sites1 and in research literature. Using external sources has advantages through:

• saving work and reducing risk of unproductive work;

• providing measurable definitions, assurance that measurement is feasible and, optionally, benchmarking information; and

• identifying agencies that could provide useful advice on measurement issues and management applications.

1 e.g. the Social Report (http://socialreport.msd.govt.nz/2003/index/shtml), Oregon Shines II

(http://www.econ.state.or.us/OPB/index.htm), and performance targets in UK Public Service Agreements.

Sector Outcomes

Sector Outcomes

DepartmentOutcomes

Department Outcomes

Business Unit Outcomes

Business Unit Outcomes

Whole of Government

The Pathfinder Project Page 3 Identifying Outcomes

10 Concentrating on the ‘vital few’ outcomes reduces the agency’s risks of information overload. Most agencies contribute to many outcomes, and will also monitor input, output and quality information. Given that multiple measures may be required for each outcome selected, and the need to know outcomes for different demographic groups (or entities), business units and treatments, it is important to focus on what matters most.

11 From a management perspective, the outcomes that matter most are strongly attributable to the actions or services of your agency and business units. It is likely that other agencies will also contribute to these outcomes. They are ‘vital’ but not always exclusive.

Getting Stakeholder “Buy-in”

12 It is important to ensure the outcome information will be valued by internal users, as well as useful for external reporting purposes. This will provide incentives for the agency to continue measuring and using outcome information. It will also foster a support base within the agency as measurement problems and outcome management issues are encountered and worked through.

13 Internal and external stakeholders must accept the validity of an agency’s outcome measures, and have confidence in the integrity of the underlying data, before they will have confidence in decisions made on the basis of outcome measures. Agencies must engage with ministers as they work through which outcomes will be used as the ‘vital few’ outcomes. The concept of the ‘vital few’ is a prioritisation tool. Do not ignore the important. Engagement is the key to stakeholder acceptance.

Steps for Developing an Outcome Definition

14 Once an agency has agreed its vital few outcomes it will need to produce a clear definition of its outcome(s). The purpose of defining outcomes is to ultimately produce “feedback” data that informs management of the impact (or effectiveness) of the agencies outputs. To achieve this a clear set of outcome definition(s) is the essential first step. An outcome definition that is ambiguous is unlikely to provide feedback information that can be useful to managers.

15 Exhibit 2 tabulates the process for defining good outcome definitions. The process starts with a values-based outcome statement that is then refined to a clear outcome definition. This is then followed by defining outcome indicator(s) (see Building Block 2) and finally impact measures (covered in Building Block 4).

The Pathfinder Project Page Identifying Outcomes 4

Exhibit 2. Steps to Identify Measures of the ‘Vital Few’ Outcomes

Step Name and Description Attributes Use of Information Limitations

1. Values-based Outcome Statement: Description of outcome desired by community (e.g. reduced re-offending, health, safety, well being), without clear causal link to government outputs

• General or conceptual statement about results of value to New Zealanders

• Unspecified link (causality) between outputs or activities, and high-level outcome

• Public discussion or consumption • General communication • Conceptually driven decisions

• Many interpretations of benefit • Poor link to outputs • Minimal value in evidence-based decision-

making

2. Outcome Definition: Written definition of outcomes related to outputs of one or more government agencies (e.g. imprisonment for offences committed within 12 months of release from prison)

Precise, written definition of changes attributable to the activities of one or more agencies which:

• only defines how performance will be measured; • clearly relates to the consequence of the agency’s

outputs or activities for the community; • is focused on target groups for services; and • is linked to accessible source data

• Precise, common understanding of the primary results expected of the agency

• Basis for building ‘hard’, evidence-based decision-making systems

• Defines core performance measure(s) for agency and/or business units

• Project specification for IT-based reporting systems

• Must be translated into Outcome Indicators or Impact Measures before performance feedback can be used in evidence-based decision-making

3. Outcome Indicator: Measure of the outcome experienced by a group or entity, based on the Outcome Definition (e.g. 27% of European inmates imprisoned for offences committed within 12 months of release)

Measure of prevailing situation which is:

• clearly linked to agency services, and groups or entities receiving (or candidates for) services;

• reported in aggregate terms, and separately for population groups, regions, business units, etc;

• measured in absolute terms (i.e. without comparison with a reference or control group); and

• statistically robust

Evidence-based decisions. Used for:

• strategic/Environmental scans; • goal and target setting; • benchmarking studies; and • broad priority setting

• Cannot distinguish between the impact of agency activities and external factors beyond the agency’s influence or control

• Timeliness of feedback • Measurement errors • Some agencies may have difficulty measuring

indicators for ‘at risk’ population groups (e.g. Customs)

4. Impact Measure: Outcome as defined in the PFA. Measure of an intervention(s) effect on one or more Indicator (e.g. programme reduces offending interventions in changing an Outcome leading to imprisonment by 12%)

Measure of ‘treatment’ effect which:

• compares outcomes for the ‘treated’ group relative to the ‘untreated’ reference group or control group (e.g. the difference in Outcome Indicators between groups).

• is statistically robust; • is reported for the individual intervention or mix, and

valid only for the target group that was ‘treated’; and • if used to compare outcomes for different

interventions or groups, outcomes must be measured using identical methods

Evidence-based decisions. Used for:

• strategic planning; • evaluating individual interventions; • benefit-cost analysis in business cases; • comparing interventions; • comparing target group options; • evaluating service delivery units; • prioritising intervention option/target

groups; and • optimising the intervention mix

• Robustness of evaluation design • Ethical constraints in building reference groups • Timeliness of feedback • Measurement errors • Risks in assuming impact will be sustained for an

untested group • Ability to measure impacts for different

interventions and target groups in comparable terms

The Pathfinder Project Page Identifying Outcomes 5

Criteria for Testing Outcome Systems

16 The National Audit Office provides the useful FABRIC set of criteria (see exhibit 3) for testing an agency’s outcome measurement system.

Exhibit 3. The FABRIC Criteria for an Effective Outcome Measurement System

Criteria Explanation

Focused on the agency’s mission, aims and objectives;

Appropriate to, and useful for, the stakeholders who are likely to use it;

Balanced giving a picture of what the agency is doing, and covering all significant areas of work;

Robust measures of outcomes that have a strong link to core activities, based on accurate observation and calculation, which are independent of the way a service is delivered;

Integrated into the agency, being part of the business planning, and

Cost-effective balancing the benefits of the information against the costs of measurement and management responses.

After: Measuring the Performance of Government Departments. National Audit Office (UK), March 2001, p.20

Examples of Outcome Definitions and Indicators

17 Examples of outcome definitions that are measurable are shown in Exhibit 4. Not all outcomes will be as measurable as these outcomes.

Exhibit 4. Some High-level Outcomes for Different Justice Sector Agencies Ministry of Justice Victimisation rates per 1,000 population per year, by crime type

Percent citizens perceiving they are safe and secure from crime

Police Reported crime rates per 1,000 population per year, by crime type

Resolution rates per 1,000 reported crimes, by crime type

Corrections Recidivism rate per 1,000 offenders, measured over 2 years

Incident rate per 1,000 offenders per year (assaults, escapes, etc) Social cost avoided per offender given a rehabilitative programme

After: Managing for Outcomes: a Guide to Outcome Statements. Queensland Treasury, Australia, 1997

Defining Outcomes: The Next Steps

18 The vital few outcome definitions must be clear enough to enable management to identify outcome indicators (Building Block 2) that provide useful feedback on the effectiveness of the agencies activities. Outcome definitions that are not well defined or not aligned to the agency’s mission as perceived by stakeholders cannot provide useful feedback information to help management make better decisions. Agencies must spend time defining and testing their vital few outcomes with stakeholders before proceeding with the next steps.