Measuring Performance (July 2008)€¦ · Measuring Performance . ii . ... We should always remind ourselves to measure the right things, and not just the easy ones. ... ofers a good

Measuring Performance

July 2008

EFFICIENCY UNIT VISION AND MISSION

Vision stateMent

To be the preferred consulting partner for all government bureaux and departments and to advance the delivery of world-class public services to the people of Hong Kong

Mission stateMent

To provide strategic and implementable solutions to all our clients as they seek to deliver people-based government services We do this by combining our extensive understanding of policies our specialised knowledge and our broad contacts and linkages throughout the Government and the private sector In doing this we join our clients in contributing to the advancement of the community while also providing a fulfilling career for all members of our team

This brief was researched and authored by the Research Division Institute of Public Administration Ireland (wwwipaieresearch) The Research Division provides applied research services for policy makers in a wide range of public service organisations drawing on an extensive network of contacts and experience gained over more than thirty years

other efficiency Unit DocUMents The Efficiency Unit has produced a number of guides on good practice on a wide range of areas including outsourcing and contract management These may be found on the Efficiency Unit website at wwweugovhk

Foreword

We have all heard of the consultant jargon which states that lsquowhat gets measured gets donersquo But quite often establishing meaningful performance targets and measures is not a straightforward task As public managers we believe we are diligently measuring the performance of our organisations every day Oversight agencies also keep us on our toes

But how good are we at this task

Performance measurement is an integral part of the process of delivering services to the community We have been practising the target-based performance measurement system for over a decade and we hold ourselves accountable over the use of financial resources through the Controlling Officersrsquo Report

But overseasrsquo literature and experience show that performance measurement is an ongoing pursuit not only by the private sector but also by governments To ensure that we are in tune with the public at all times it is of paramount importance that we continue to adapt and perfect our performance measures so as to reflect the desired social outcomes that we pledge to achieve

We know that performance measurement in the public sector is a much more complex task than that in the private sector This report resonates with many of our beliefs It also recognises the fact that it takes time and resources to compile the necessary indicators to construe whether a social outcome is achieved or not

Nevertheless governments in different parts of the world are fine-tuning their systems to make them more outcomeoutput-based than input-based Setting appropriate performance measures establishing challenging and yet achievable targets and defining results ndash these are effective means of focusing our combined efforts to adhere to social objectives when delivering public services

We should always remind ourselves to measure the right things and not just the easy ones

Increasingly governments worldwide are using performance measures and targets following consultation with departmental stakeholders Experience overseas has yielded mixed results I hope that this report will assist departments in identifying the most appropriate opportunities for improving their own regimes

Comments feedback and sharing are most welcome

Head Efficiency Unit July 2008

contents 1

execUtiVe sUMMary 2

1 PerforMance MeasUreMent Key challenges 6

2 iMProVing PerforMance MeasUreMent in BUsiness Planning 7

3 enhancing the MeasUreMent of Policy oUtcoMes 19

4 MeasUring PUBlic sector ProDUctiVity 27

5 ensUring that PerforMance MeasUres are UseD anD UsefUl 31

enDnotes 42

references 44

Executive Summary

A key challenge nowadays is to focus measurement systems on results Government policies and programmes deliver two main types of results outputs (the direct products and services produced) and outcomes (the consequences of those outputs for society) In terms of this overall focus on results four main measurement challenges are addressed in this report These are improving performance measurement in business planning enhancing the measurement of policy outcomes measuring public sector productivity and ensuring that performance measures are used and useful

iMProVing PerforMance MeasUreMent in BUsiness Planning

Nowadays business planning is a vital element in most public service organisations An important first step in developing performance measures for business plans is to get a clear understanding of the range of performance issues associated with the area under scrutiny Public organisations in many countries are now using the logic model approach to help structure thinking about developing a range of performance measures covering inputs outputs and outcomes

By focusing particular attention on outcomes measures are more likely to be developed that address the needs

and interests of clients customers and citizens Business plans should not be internal-looking documents that focus solely on what managers and staff see as important Measures should be developed that address the critical issues surfaced by citizens as part of the planning process

Having determined a set of measures for inclusion in the business planning process it is useful to check the quality of the proposed performance measures Good quality measures are needed if they are to be useful Various sets of criteria have been developed for assessing the quality of performance measures

Business plans and associated performance measures are an important part of the public accountability process for the use of public funds In this accountability context some measures may be used properly for individual or programme accountability purposes while other measures may be misleading if used for such accountability purposes but nevertheless be important for the overall management of the programme or activity Both intermediate and final outcome measures for example are beyond the direct control of staff working in the area Nevertheless these outcomes should be reported as part of the business planning process But they should be seen as contributing to giving an account on performance rather than being used to hold staff to account

EXECUTIVE SUMMARY

enhancing the MeasUreMent of Policy oUtcoMes

Whole-of-government and crossshy cutting policy outcome measures

Some governments at both national and local levels have developed measures that have a high-level focus on the final outcomes of policies concerning social economic and environmental issues These measures do not enable tracking of individual agency performance but they do provide greater clarity on desired high-level outcomes and set a context within which individual agency performance measures are developed

At both the whole-of-government and sectoral levels there is increasing attention on the need to develop measures that cut across organisational boundaries and focus on the social and economic outcomes desired by citizens The interest here is on measures that track and encourage joined-up thinking and practice on the part of government organisations For example policy units in government departments often have responsibility for developing and outlining the intermediate and final outcome objectives for the areas where they have policy responsibility While it may be the task of other delivery agencies to develop measures and collect information on performance against the objectives the department should report on performance against these measures as part of its overall policy responsibility

addressing time-lag and attribution issues

Time-lag and attribution problems are significant when developing outcome-oriented performance measures Time-lag issues refer to the fact that it can often be many years before the full effects of a policy are seen in practice Attribution issues refer to the difficulty in attributing causation with regard to policy outcomes to particular interventions Changes in infant mortality rates to take one example are influenced by a myriad of factors some associated with health-care practices some with social and economic programmes and some with educational programmes Identifying the precise contribution of a particular programme or agency to such final outcomes is often in practice impossible There are no easy answers to these challenges But the separation of outcomes into intermediate and final outcomes and the measurement of each offers one way forward for managers Another approach to dealing with attribution is to use impact evaluation

EXECUTIVE SUMMARY

MeasUring PUBlic sector ProDUctiVity

The evidence from studies of public sector productivity measurement is that productivity measurement is still in its early stages Despite efforts going back to the 1980s the productivity measures being produced need to be interpreted cautiously There is also the danger that over-simplistic use of the measures can lead to perverse consequences It is clear that no single figure of productivity can be used for public sector activities unless there is clear and widespread agreement that it is an appropriate measure A range of supporting information is needed to measure productivity change

ensUring that PerforMance MeasUres are UseD anD UsefUl

Performance measures need to be seen in a wider management context if they are to be used and useful Among the main issues that need to be addressed when placing performance measures in this wider context are linking measures and targets engaging citizens in the process of performance measurement and using incentives and sanctions to encourage the use of performance measures

linking measures and targets

Targets are an important element in making performance measures useful Good targets can lead to enhanced performance Challenging and specific targets improve

the performance of employees and are associated in many instances with higher levels of performance than either no or general lsquodo your bestrsquo targets But how do we know when a lsquogoodrsquo target has been set What might a challenging yet realistic target look like The identification and use of baseline data can be very helpful in this context when setting targets

engaging citizens in the process of performance measurement

It is vitally important to develop measures of issues that citizens want measured in performance measurement systems The question is not whether the customer should be engaged in performance measurement but how best to engage the public Two issues of central importance are means of engaging the public and the reporting of performance measures to the public

Methods of engaging the public include focus groups neighbourhood meetings citizen satisfaction surveys report cards for programme users web-based discussion forums and web-based surveys Each method of engagement has advantages and disadvantages and may vary in terms of cost The use of a diversity of approaches offers a good way forward

Also the way in which performance measures are reported back to citizens affects the degree of engagement of the public and ultimately the usefulness of many performance measures Issues such as the provision of measures on

EXECUTIVE SUMMARY

a neighbourhood basis and the use of comparative performance benchmarks can be helpful here

Using incentives and sanctions to encourage the use of performance measures

Performance measures are rarely used simply on the basis that they have been produced Performance measurement is not just a technical exercise it has strong cultural and political components In such an environment incentives and sanctions (carrots and sticks) can play an important role in encouraging the use of performance measures The commonly used incentives and sanctions are legislation review awardrecognition assurance and involvementfeedback

1 Performance Measurement Key Challenges

A key challenge nowadays for managers is to focus measurement systems on results The public and politicians are increasingly concerned with what is being delivered for the public money being used to fund government programmes They want to know that the money is being spent to achieve the purposes for which it was allocated but also and equally importantly that the money is being spent wisely Hence the concern with results Government policies and programmes deliver two main types of results outputs (the direct products and services produced) and outcomes (the consequences of those outputs for society)

But a focus on outcomes while very important can also be very difficult It is often hard to connect the performance of programmes and managers with distant and complex outcomes There is a risk of measurement becoming a paper-based exercise that does not really change practice People may fall back on measuring what is easiest to measure which may not be the most important issue from the viewpoint of the citizens Goal displacement may occur where measures themselves become the objectives taking attention away from what the programme should be delivering There is also often a tendency to fall back on input measures as these are familiar even though they tell us only about efficiency not effectiveness

To overcome such difficulties a results and measurement-oriented culture is needed involving managers and staff throughout government In a supportive culture performance measures1 can help tell the performance story of a programme or organisation

In terms of this overall focus on results and to address the issues raised above four main measurement challenges are addressed in this report These are

Improving performance measurement in business planning Enhancing the measurement of policy outcomes Measuring public sector productivity Ensuring that performance measures are used and useful

A particular emphasis is placed on addressing the needs of the citizens Ultimately we need to be sure that we are measuring the results that the citizens want to have measured

2 Improving Performance Measurement in Business Planning

Public sector organisations are increasingly using performance measures as part of their business planning process Divisional and work unit business plans are now a common feature in many government agencies In this section issues that are of particular relevance to the development of performance measures for business planning are outlined First steps to be taken and issues to consider in the development of performance measures for business plans are presented Second methods of assuring the quality of performance measures are reviewed Third the use of measures for management and accountability purposes is discussed And finally a specific current issue for the day-to-day business of many organisations ndash assessment of the performance ofgovernment websites ndash is examined

DeVeloPing PerforMance MeasUres for BUsiness Plans

Using the logic model approach to identify performance measures

An important first step in developing performance measures is to get a structured understanding of the range of performance issues associated with the programme2 under scrutiny Public organisations in many countries are now using the logic model approach to help structure thinking about developing a range of performance measures covering inputs outputs and outcomes Figure 1 shows the results spectrum produced from a logic model This is sometimes referred to

as the results chain At the start of the chain are the outputs results that managers can largely control eg drafting of legislation or giving of grants At the end of the chain are the final outcomes the end results sought eg improvements to health the economy or the environment And in-between is a sequence of intermediate outcomes that are intended to lead to the final outcomes eg changes in peoplersquos attitudes and behaviours

The United Way of America (1996) notes ldquoA programme logic model is a description of how the programme theoretically works to achieve benefits for participants It is the lsquoif-thenrsquo sequence of changes that the programme intends to set in motion through its inputs activities and outputs Logic models are useful frameworks for examining outcomes They help you think through the steps of participantsrsquo progress and develop a realistic picture of what your programme can expect to accomplish for participants They also help you in identifying the key programme components that must be tracked to assess the programmersquos effectiveness

figUre 1 logic MoDel resUlts chain

Outputs (goods and services produced by programme)

IMPROVING PERFORMANCE MEASUREMENT IN BUSINESS PLANNING

Intermediate outcomes (immediate effects of the outputs)

Intermediate outcomes (medium-term effects of the outputs)

Final outcomes (the final or long-term consequences)

An illustrative example of measures derived from using a logic model approach is given in Figure 2 ndash evidential breath testing of drivers to improve road safety

choosing measures that meet stakeholder needs particularly citizensrsquo needs

An important issue when developing performance measures for business planning is to consider the needs of a range of stakeholders when selecting measures for business plans For example politicians central finance officials staff

and management of the organisation and service users all have their own information needs The incentive to develop a balanced set of performance measures incorporating both financial and non-financial measures stems from trying to meet the needs of these different stakeholders Initiatives such as the logic model and the balanced scorecard (Kaplan and Norton 1993) have spread through many private and public organisations in an effort to ensure that performance measures meet the needs of various stakeholders

figUre 2 Using the logic MoDel to DeVeloP PerforMance MeasUres eViDential Breath testing

input objective To ensure that programme and administrative expenditure levels are contained as agreed

input measures

bullbull

bull bull

Programme expenditure Recurrent Capital

Administrative expenditure Pay related Non-pay related

output objective To test drivers for evidence of alcohol levels

output measures Number of drivers tested

intermediate outcome objectives To change the attitudes and behaviour of drivers towards alcohol consumption To reduce numbers tested who are over the legal limit To reduce the number of convictions for drink driving offences

intermediate outcome measures

Survey findings of attitude and behaviour change Numbers tested over the legal limit Number of convictions for drink driving offences

final outcome objectives To reduce alcohol-related road accidents To improve road safety

final outcome measures

Number of alcohol-related road accidents Number of deaths and injuries on the roads linked to alcohol

A key stakeholder challenge for managers is to select performance measures for their business plans that measure the issues and results that service users and citizens want to see measured Ho (2007 http wwwbusinessofgovernmentorgpdfs HoReportpdf ) suggests that this citizen focus is often missing in the traditional approach to performance measurement development In the traditional approach managers start by asking themselves and their staff what their vision and objectives are They then develop performance measures on the basis of this work Ho identifies four main problems with this approach

It assumes managers are capable of establishing programme vision and objectives that reflect public concerns and priorities Managers may be trapped by their own blind spots and fail to see beyond what they routinely do in programme delivery and planning And the public may view the measures as a tool to showcase the achievements of the administration The traditional approach assumes that performance measures by their own weight can influence how elected officials think about effectiveness and resource allocation But in practice politicians may not pay much attention to measures if they do not believe that the measures reflect their constituenciesrsquo concerns and priorities Managers may focus on their own priorities rather than on how they can best collaborate with other agencies and the public to deliver results This can emphasise a lsquosilorsquo mentality rather than a lsquojoined uprsquo perspective

What is needed is some means of involving citizens in the selection of the measures used in business plans For example the Oregon Progress Board established an eight-member advisory committee to give state agencies a better sense of how citizens view the statersquos current performance measurement system To do so four state agencies offered their most recent annual performance measure reports for members of the advisory committee to read and respond to This led to initiatives such as more use of comparative data to benchmark performance measures (See httpwwworegongovDASOPB docskpmNCCINCCIfinaldoc for more details)

A more detailed case of good practice in citizen involvement is given in the case study of the Des Moines (USA) citizen-initiated performance assessment project

The Des Moines project was considered a success as it led to the development of performance measures that were actively used by officials but also were of direct interest to citizens The measures addressed issues that were of concern to citizens and were identified by them as very important But there are limitations to this partnership approach In the Des Moines case the president of Des Moines Neighbors (an umbrella organisation for neighbourhood groups) and the city manager both stepped down within a short period of each other With the loss of two key supporters the project lost momentum in 2005 An annual performance report continues to be issued but the civic engagement process has not been maintained This illustrates that maintaining such a partnership approach requires continuity of commitment and that the loss of key individuals who drive the process may break the continuity

case stUDy

Des Moines citizen-initiateD PerforMance assessMent Project

Des Moines Iowa has a population of about 200000 people Between 2001 and 2004 the city implemented a citizen-initiated performance assessment (CIPA) project Public officials and neighbourhood representatives jointly developed performance measures The process went through several stages

Topic selection ndash Participants wanted to measure the quality of life in the city and what the city had done to improve the well-being of residents Under the broad heading of nuisance control a small number of key issues emerged from the process eg odour control noise pollution traffic law enforcement and clearance of abandoned housing

Measure selection ndash For each topic critical issues were identified and measures developed to track these critical issues as illustrated below

toPic critical issUe PerforMance MeasUres

odour control

Where are the problems Number of complaints received - by neighbourhood

Are citizens satisfied with the air quality and the current level of odour control

Satisfaction level of surveyed citizens - by neighbourhood

Satisfaction of city response after a complaint is filed

Level of satisfaction with the service received after a complaint was made a month previously

Data collection ndash Public officials tracked nuisance complaints over time and by location They also conducted sample surveys of residentsrsquo satisfaction with city services and citizensrsquo perception of quality of life in their residential area Citizens also directly participated in digital surveys in selected neighbourhoods using handheld electronic devices to document problems such as potholes and graffiti

Engagement of citizens and other stakeholders in reporting ndash Performance measurement results were actively disseminated to the city council and directly to the public

Source Ho 2007 p18-23 Performance reports can be accessed at httpwwwcides-moinesiaus performancereporthtm

More generally while having significant benefits involving citizens in the development of performance measures can be a challenging activity and a number of issues need to be borne in mind if taking this approach

Managers must be committed to the process and provide leadership to work with citizen groups If this leadership is not there the process will not be taken seriously There are risks involved in engaging citizens in performance measurement Measures may reveal problems that present challenges to managers and elected officials There is a risk that citizen groups consulted may not be representative of the wider citizenry Steps must be taken to ensure that citizen engagement is comprehensive and inclusive Citizen engagement is resource-intensive particularly in terms of the time commitment required of management and staff It is also demanding of citizens themselves There is no single citizen view of performance More likely there will be a variety of views expressed as to the importance of measures selected and what they mean Balancing and interpreting these divergent views presents challenges to managers

None of this is to suggest that citizen engagement in performance measurement is not important But an awareness of the challenges and limitations can facilitate a more effective approach to such engagement

assUring the QUality of PerforMance MeasUres UseD in BUsiness Planning

Having determined a set of measures for inclusion in the business planning process it is useful to check the quality of the proposed performance measures Good quality measures are needed if they are to be useful Various sets of criteria have been developed for assessing the quality of performance measures Perhaps the most common is to check measures against the SMART criteria (specific measurable achievable relevant and time-bound) An illustrative example of this process is given in Table 1 In the Irish Department of Agriculture and Food performance measures proposed by divisions for their business plans have been subject to a quality analysis using SMART criteria from within the economic and planning unit The benefits of this process in terms of improving the specificity of the measures and ensuring there is a link between measures and targets can clearly be seen from this example

taBle 1 exaMPles of PerforMance MeasUres analyseD Using the sMart criteria 13

Measure Specific Measurable Achievable Relevant Time-bound

Level of animal diseases

Relates to a specific item (animal disease levels)

In terms of number of disease levels

What is identified here is an indicator Target not identified therefore achievability cannot be commented on

Relevant to several objectives in the statement of strategy

Not identified but perhaps not applicable

Percentage of payments within protocol deadlines

Relates to a specific item (payment deadline protocols)

In terms of percentage within deadline

As above As above Time criteria identified (protocol deadlines)

Using MeasUres in BUsiness Planning DistingUishing BetWeen ManageMent anD accoUntaBility PUrPoses

Business planning is intended as part of a process of improved management within public service organisations Business plans are meant to lsquostep downrsquo to the divisional level the high-level goals and objectives contained in strategic plans As such business plans and consequently the performance measures contained in

business plans are an important part of the public accountability process for the use of public funds In this accountability context it is important to recognise that measures may play different roles depending on the aspect of performance being measured Some measures may be used properly for individual or programme accountability purposes Other measures may be misleading if used for such accountability purposes but nevertheless be important for the overall management of the programme or activity This issue is

best illustrated by way of example as set out in Figure 3

Figure 3 sets out performance objectives and measures used to assess an anti-smoking television advertisement campaign Input activity and output measures in this example can be used to help assess individual and team performance of staff working in this area It is the responsibility of the staff working in the area to ensure that the campaign material is designed tested and runs on the television within agreed resource allocations Staff can be held to account for this work

The intermediate outcome objectives and measures ndash assessing whether people see the advertisements and if their attitudes and behaviour are affected ndash are clearly beyond the direct responsibility of people working in the area But these measures provide information about the results of the programme that can readily be directly attributed to the programme It is possible to judge if the advertising campaign has been used and if it has had an effect on peoplersquos attitudes and behaviour As such these measures can be used to assess programme performance and hence programme accountability

The final outcome measures ndash less smoking and lower incidence of smoking-related diseases ndash are affected by many other factors apart from the campaign itself These measures are clearly relevant to judgements about programme performance but it is not possible to directly attribute

changes in these measures directly to the programme

Both intermediate and final outcome indicators are beyond the direct control of staff working in the area But as Schacter (2002) notes lsquoThey are nevertheless things that you must monitor because they have profound relevance to the design and the implementation of your program These are the results that you are managing for even if you canrsquot control themrsquo These outcomes should be reported on as part of the business planning process But they should be seen as contributing to giving an account on performance rather than being used to hold staff to account Including outcome indicators such as these is important for the successful management of the programme

figUre 3 Using MeasUres for Different accoUntaBility PUrPoses

inputs $$$ Full Time Equivalent Staff

activities

Design and test campaign material

(Measure material produced by ddmmyyyy)

Produce campaign material

outputs Advertisements run on television

(Measure number of schools receiving campaign material)

Fair reflection of individualperform

intermediate outcomes

People see the advertisements

(Measure number of viewers)

Peoplesrsquo attitudes affected

(Measure degree of attitude change)

Behaviour affected

(Measure degree of behaviour change)

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

(Measure level of smoking)

Lower incidence of smoking-related diseases

(Measure incidence of smoking-related diseases)

Relevant to progr am

perfor mance

Source adapted from Schacter (2002)

assessing the PerforMance of goVernMent WeBsites

A specific measurement issue yet one of growing prominence for many organisations is the measurement of the performance of government websites This issue is often dealt with in the context of business planning for an organisation when looking ahead to challenges to be addressed in the coming year Websites have become a key channel of communication between governments and citizens As such they must address the needs of very wide-ranging and differing audiences Usability is crucial But simply tracking measures such as the number of hits can be misleading A site may get a lot of hits yet visitors may stay on the site for only a few seconds suggesting they did not find it useful once there A site with a smaller number of hits may have people staying on for longer and returning frequently suggesting that it is meeting a need

What then is to be done to allow a more sophisticated measurement of website performance A first step is to be clear about the purposes and objectives of the website Is it there to enable people to fill out a survey use interactive applications built into the site download important information and so on In the language of the logic model what are the main inputs outputs and outcomes required of the website Measures can then be developed to address these issues (see Table 2 taken from Stowers 2004 httpwwwbusinessofgovernment orgpdfs8493_Stowers_Repor t pdf

The case study on page 18 highlights two initiatives to measure website performance one from Australia and one from the USA Each develops a wide variety of measures against which website performance is judged Comparative assessment of the websites of different government agencies is encouraged

TABLE 2 POTENTIAL WEBSITE PERFORMANCE MEASURES

Input Measures Output Measures Outcome Measures

Application development and hardware set-up

Staff costs Other development costs Other vendor costs Staff time f or application development Other development time Vendor time for development purposes

Number of hits or user contact sessions Number of downloads of documents Time users spend on a site Number of times transactions completed or the times online forms have been accessed and completed Monetary amounts processed through each site

Accessibility of services y Number of site pages

meeting accessibility criteria Accuracy of the assistance or information as measured by percentage accuracy rates in random fact checking Adequacy of information as measured by staff and citizen surveys Ease of use as measured by pop-up or other surveys Citizen satisfaction with site

Service qualityPercentage of time when website is down and not available

Webpage errors Efficiency

Cost per transaction Total cost per user session

Final outcomes Cost savings from e-government Staff time sa vings from e-government

case stUDy

MethoDs for MeasUring WeBsite PerforMance

assessing federal government websites ndash australia The Australian federal government commissioned a private company UsabilityOne to conduct an assessment of ten federal government websites The company developed a usability compliance audit for evaluating websites The audit involves 151 criteria based on acknowledged usability principles and extensive user-testing experience The main areas covered in the audit include navigation content and content writing design and graphics search error prevention and recovery trust internationalisation window titles news and press releases plus branding and company information

Two analysts independently reviewed each website against the criteria and arrived at an overall compliance score for each website Full details of the project can be accessed at httpwwwusabilityonecom

the american customer satisfaction index (acsi) methodology The ACSI was developed by researchers at the University of Michigan to evaluate customer satisfaction generally and in 2001 ForeSee Results began to use the methodology to assess customer satisfaction with federal agency websites ACSI which uses a set of well-researched and benchmarked questions and research processes uses a causal statistical model to predict customer behaviour such as the probability of return visits to the website Satisfaction with websites is examined using pop-up surveys

These results are then analysed using the model which includes measures both of customer satisfaction and of customer loyalty Results for individual websites can be compared to a nationally developed index based on more than 200 organisations For more details see httpwwwforeseeresultscom and Stowers 2004 httpwwwbusinessofgovernmentorgpdfs8493_Stowers_Reportpdf p26

3 Enhancing the Measurement of Policy Outcomes

The use of performance measures to facilitate judgements about the efficiency and effectiveness of policy work presents significant challenges As Perrin (1998) notes a requirement to produce more policy-focused measures can lead to situations where there is the widespread development of easy-to-count measures which have little or no relationship to what the policy is supposed to be achieving Peripheral activities that are easy to quantify become the focus of reports that nobody uses

Such actions as described above arise in part because of cultural and lsquopoliticalrsquo reasons but there are also inherent technical difficulties in lsquomeasuringrsquo policy work There are a number of particular issues

Whole-of-government challenges such as multiple high-level outcomes which must be pursued simultaneously and that sometimes are in conflict with each other or where policy is initiated in one organisation but responsibility for delivering on outcomes rests with other agencies Time-lag problems associated with the fact that many years may elapse between the initiation of a policy and its implementation Attribution problems where it is impossible to disentangle the impact of a particular policy on final outcomes because these outcomes are also affected by other policies and influences

These examples are given not to suggest that measures have no role in policy work but rather to caution about raising expectations too highly in terms of the role that measures may play As part of broader efforts to improve evidence on performance measures can be helpful for managers and staff engaged in policy work

Whole-of-goVernMent anD cross-cUtting Policy oUtcoMe MeasUres

Some governments at both national and local levels have developed measures that have a high-level focus on the final outcomes of policies concerning social economic and environmental issues These measures do not enable tracking of individual agency performance but they do provide greater clarity on desired high-level outcomes and set a context within which individual agency performance measures are developed Such high-level measurement systems have been developed because of a recognition of the limitation of relying on economic measures

ldquoEconomic indicators have traditionally been used to assess the economic lsquostate of the statersquo Strong economic growth low inflation and unemployment were regarded as indicative of a healthy economic climate and believed to result in prosperity for citizens However citizens have become increasingly concerned about their relative quality of life expressed in terms such as quality of education and

ENHANCING THE MEASUREMENT OF POLICY OUTCOMES

health care availability of recreational cultural opportunities clean environment and safety from crime Accounting and economic based measurement systems were not designed to address these issues thus governments have introduced new systems for measuring progress including policy outcome based performance measurement (Ogata and Goodkey 1998)rdquo

Two longstanding and relatively successful whole-of-government measurement systems are Oregon Shines and Albertarsquos Measuring Up examined in the case study below

case stUDy

Whole-of-goVernMent oUtcoMe MeasUreMent

oregon shines The state of Oregon USA adopted a development strategy in 1989 called Oregon Shines The strategy addressed quality-of-life issues as well as economic diversification Oregon Shines II was adopted in 1997 as a successor strategy and Oregon Shines III is planned for 2009 Progress against the strategy is tracked through Oregon Benchmarks an annual report which tracks over ninety measures against benchmark targets in seven categories economy education civic engagement social support public safety community development and environment State agencies link their key performance measures to these benchmarks where it makes sense to do so For details go to httpwwworegon govDASOPBdocsobmNew_Benchmark_Numbersdoc

Measuring Up alberta In 1993 the province of Alberta Canada started a three-year strategic planning and performance measurement system In 1995 the first annual Measuring Up report was published Measuring Up contains sixty-five core performance measures related to fourteen government goals These measures focus on issues such as infrastructure capacity literacy and numeracy rates crime rate and water quality The intention is to develop outcome-based measures to provide information on progress towards longshyterm targets The targets are selected and driven by political leadership and through public consultation State agencies link their business plan objectives to relevant priorities For details go to httpwwwtreasuryboardgovabca1089cfm

At both the whole-of-government and sectoral levels there is increasing attention on the need to develop measures that cut across organisational boundaries and that focus on the social and economic outcomes desired by citizens The interest here is on measures that track and encourage joined-up thinking and practice on the part of government organisations

The UK government is acknowledged as a leader in the development of cross-cutting targets and measures in part through the development of cross-cutting Performance Service Agreements (PSAs) (see case study below) It is too early as yet to judge the success of this initiative But it has been welcomed by the Select Committee on Treasury though the Committee cautions that the cross-departmental nature of the new PSAs poses a challenge for a system of accountability based on departmental reporting and the work of departmental select committees They also stress that the new agreements to be effective must genuinely reduce the number of measures in use and not simply bring together diverse topics within a PSA and that there must be corresponding reductions in measures down the delivery chain (see httpwwwparliamentthe-stationeryshyofficecoukpacm200607cmselect cmtreasy27927909htm and httpwww parliamentthe-stationery-officecoukpa cm200708cmselectcmtreasy555507 htm)

Additionally in a critique of earlier PSAs Neely and Micheli (2004) note that there was little use in the public sector of what has become best practice in the private sector ndash developing success or strategy mapswhen designing measurement systems Successstrategy maps are intended to help people identify which are the critical measures and can be helpful when rationalising the number of measures in use

Another common cross-cutting situation is that of policy-oriented units operating in government departments who have overall responsibility for policy but where implementation of that policy is through agencies operating under their control In this instance policy units may sometimes not develop measures themselves saying that it is the responsibility of the agencies to develop performance measures The argument used here following on from the logic model approach is that policy units should be responsible for developing and outlining the programme theory including the intermediate and final outcome objectives While it may then be the task of the relevant agencies to develop measures and collect information on performance against the objectives the department should report on performance against these measures as part of its overall policy responsibility

case stUDy

UK cross-DePartMental PUBlic serVice agreeMents

In 1998 in the first Comprehensive Spending Review the UK government introduced a new performance measurement framework based on Public Service Agreements (PSAs) While generally welcomed these PSAs suffered from a number of drawbacks The system was seen as top-heavy and unwieldy with too many measures created at local level to feed into the system Particular difficulties arose from targets held jointly between departments and some issues that involved more than one department were not adequately captured

In the 2007 Comprehensive Spending review the government announced a smaller suite of thirty new PSAs that outline the governmentrsquos highest priorities and span departmental boundaries The agreements are cross-departmental although there is a lead department for each agreement Departments are required to produce a cross-departmental Delivery Agreement for each PSA informed by consultation with the delivery chain These Delivery Agreements set out how each PSA is to be delivered and the measures used to track performance against PSAs and their associated delivery Agreements can be accessed via httpwwwhm-treasurygovukpbr_csrpsapbr_ csr07_psaindexcfm

An example of this approach is set out in Table 3 Here in an Irish example the focus is on the Local Development Social Inclusion Programme (LDSIP) overseen by the Department of Community Rural and Gaeltacht Affairs Implementation and administration of the programme is overseen by an agency ADM Ltd on behalf of the Department An intermediate outcome objective for the LDSIP is that local groups develop innovative plans and programmes to improve the local social and economic infrastructure Readily applied measures here are the

number of plans developed the number of actions implemented from the plans and the quality of the plans (for example the percentage of plans inspected that fall below a determined quality standard) In this case the programme logic is that as a result of developing high-quality plans and the subsequent implementation of these plans local groups contribute to sustained social and economic development in their communities which is one of the final outcome objectives of the community and local development programme

With regard to this final outcome objective of sustained social and economic development in communities Table 3 gives examples of measures gathered by ADM Ltd that can be used to track progress While ADM Ltd has responsibility for developing and gathering the data on these measures as the policy body with

overall responsibility for the programme it is important that the Department use these measures in its own reporting procedures In this way an overview is provided of the outcomes of the community and local development programme for public accountability purposes

taBle 3 coMMUnity anD local DeVeloPMent MeasUres

interMeDiate oUtcoMes

objective Development by local groups of innovative plans and programmes to improve the local social and economic infrastructure

Measures Number of plans developed Quality of plans produced Number of actions implemented

final oUtcoMes

objective Sustained social and economic development in communities with a special focus on areas of disadvantage

Measures Number of people placed into jobs Number of people supported into self-employment

aDDressing tiMe-lag anD attriBUtion issUes

Time-lag and attribution problems are significant when developing outcome-oriented performance measures Time-lag issues refer to the fact that it can often be many years before the full effects of a policy are seen in practice Attribution issues refer to the difficulty in attributing causation with regard to policy outcomes to particular interventions Changes in infant mortality rates to take one example are influenced by a myriad of factors some associated with health-care practices some with social and economic programmes and some with educational programmes To identify the precise contribution of a particular programme or agency to such final outcomes is often in practice impossible

There are no easy answers to these challenges But the separation of outcomes into intermediate and final outcomes and the measurement of each offers one way forward for managers Take as an illustrative example the case of a research and development (RampD) grant scheme aimed at encouraging innovation and the development of new products Final outcome measures such as number and type of new products on the market and turnover attributed to new products and services may be influenced by factors other than grants given to firms to develop their RampD capabilities The national and international economic situation local educational initiatives and so on also have an impact It is not possible to examine the

success of the programme solely on the basis of final outcome measures though clearly they provide vital information to inform an overall judgement Similarly the outputs of the programme ndash the number and value of grants issued ndash do not on their own give a satisfactory picture of performance

This is where intermediate outcome measures can help The programme logic is that as a result of receiving supports firms develop their research capability and improve their links with third-level research institutes thus contributing to the final outcome objectives Measures such as level of research skills before and after receiving the grant (assessed by survey) and type and quality of links with third-level research institutes can track these intermediate outcome objectives Measuring the intermediate outcomes can give a better sense of programme performance And if the programme logic is demonstrated to be faulty a new theory can be developed to address the issue The intermediate outcome measures help give a picture of the performance of the RampD grant scheme Neither the output nor the final outcome measures alone could give such a picture Intermediate outcome measures help flesh out the performance story

Using impact evaluation

Another systematic means of developing performance measures that address attribution and time-lag issues and that may be useful in certain circumstances is impact evaluation Impact evaluation is defined by the World Bank (2006) as an assessment of the impact of an intervention on final outcomes It assesses the changes arising from an intervention that can be attributed to a particular project programme or policy

(see case study below on increasing tax revenue in Pennsylvania) By knowing the actual impact of intervention towards the final outcomes appropriate performance measures can be designed

case stUDy

increasing tax reVenUe in PennsylVania

The Pennsylvania state government commissioned a large randomised control trial to evaluate the effectiveness of various approaches to improving tax compliance by businesses that were late in paying their sales taxes The trial randomly assigned 7000 such businesses to receive one of seven letters ranging from threatening to pleading and made use of outcome data that the state had already collected for other purposes ndash namely whether the businesses paid their taxes

The trial found that a letter containing a short (one-third page) statement that tax is due and that the business is liable produced significantly more tax revenue than the statersquos existing letter (full-page detailed letter with boxes that the businesses check to indicate why they have not paid the tax) The trial results indicated that the statersquos use of the short letter for all late-paying businesses could generate US$6 million annually in increased revenue

Source httpwwwwhitehousegovombpart2004_program_evalpdf p12

A key element in impact evaluation is identifying the lsquocounterfactualrsquo What would have happened had the intervention not taken place This involves the identification of a comparison group who are as alike as possible to the group who receive the intervention but who were not subject to the intervention The most frequently used methods of identifying the counterfactual are the following

Randomised control trials (RCTs) ndash An RCT is a study that measures aninterventionrsquos effect by randomly assigning individuals (or other units such as schools or hospitals) into an intervention group which receives the intervention and into a control group which does not Following the intervention measurements are taken to establish the difference between the intervention group and the control group Because the control group simulates what would have happened if there were no intervention the difference in outcomes between the groups demonstrates the final outcome or impact one would expect for the intervention There are however many programmes for which it would not be possible to conduct an RCT To carry out an RCT there must be a possibility of creating a control group who will not receive the intervention For practical legal and ethical reasons this may not always be possible

Quasi-Experiments ndash Like RCTs these evaluations assess the differences that result from an intervention and the result that would have occurred without the intervention However the control group is not randomly assigned Instead it is designed on the basis that the evaluator judges how to minimise any differences between the two groups or it may be a pre-existing group Use of comparison group studies does increase the risk of misleading results because of the difficulty in eliminating bias in the selection of the control group

The benefits and challenges of impact evaluation are well described by the World Bank Independent Evaluation Group (httpwwwworldbankorgiegecd)

4 Measuring Public Sector Productivity

A number of initiatives have taken place in several countries in recent years aimed specifically at improving the measurement of public sector productivity The experience of a small number of countries at the forefront of attempts to improve the measurement of productivity is outlined here the UK Finland Sweden and Australia

MeasUring PUBlic sector ProDUctiVity in the UK

Since 1988 the Office for National Statistics has been progressively moving away from the output=input approach to productivity and incorporating direct measures of the volume of government output in the national accounts By 2005 these direct output estimates accounted for two-thirds of general government final consumption In the context of this focus on output measurement the UK government commissioned Sir Tony Atkinson to undertake a review of the measurement of government output in the national accounts This review (Atkinson 2005) provides a comprehensive overview of developments and recommendations for future progress

The UK government accepted the findings and recommendations of the Atkinson review and the Office for National Statistics (ONS) has the lead role in taking forward the recommendations To this end the ONS has set up the UK Centre for the Measurement of Government Activity (UKCeMGA) The UKCeMGA has issued a number of interesting reports and research

papers on public sector productivity measurement eg education and health sector productivity measurement (see httpwwwstatisticsgovukaboutdata methodologyspecificPublicSector outputdefaultasp for details)

MeasUring PUBlic sector ProDUctiVity in finlanD

Finland along with the UK is widely regarded as a world leader in public sector productivity measurement In 1995 a project was established to measure public sector productivity in Finland It was located in Statistics Finland the national statistics office The aim of the project was lsquoto develop a measurement and monitoring system for government sector production and productivity by using an output indicator method to measure the volume of outputrsquo (Niemi 1998) In 1997 the scope of the project was expanded to include the measurement of the productivity of local government services

Under the terms of the project for central government services the final output and the output measures are specified by the agencies themselves Examples of output measures are given in Table 4 The agencies for which input and output data are gathered cover about 80 per cent of the compensation of employees in central government Initial results show growth rates of output and productivity varying extensively

MEASURING PUBLIC SECTOR PRODUCTIVITY

TABLE 4

EXAMPLES OF OUTPUT INDICATORS IN THE CENTRAL GOVERNMENT IN FINLAND

Consumer Ombudsmanrsquos Office

number of petitions to market court

marketing instructions

(number of ) contractual terms negotiated

statements on legislative initiatives

cases solved individually

replies to written enquiries

Courts number of cases settled

National Food Administration number of letters guiding supervision

number of administrative decisions and memos

number of publications

number of statements

number of training events

new instruction materials

State Audit Office number of annual audits

supplementary audits

international audits

expertise activities statements

Universities number of degrees completed (generally separated into graduate and postgraduate degrees)

adult education and continuing education measured for example in days or number of courses (depending on the university)

number of publications (research)

Source Niemi 1998

MeasUring PUBlic sector ProDUctiVity in sWeDen

Sweden has been measuring public sector productivity since the mid-1980s An Expert Group on Public Finances (a subcommittee under the Ministry of Finance) established a steering group to conduct the work The steering group was supported by Statistics Sweden the national statistics office

Particular focus is given to the development of output measures for services These include items such as the number of admitted patients for in-patient medical care the number of learning hours for education traffic volume as measured by vehicle kilometres for public roads number of flying hours for the air force Attempts are made to adjust the quantity of outputs for quality variations where data are available A number of lessons are drawn based on the experience of producing annual productivity measures over a period of time

Productivity varies greatly from year to year To assess any given year a time series of several yearsrsquo information is needed Most agencies have several categories of output Attributing the same weight to all categories may yield deceptive results In the enforcement service for example counting cases dealt with regardless of category gives a 2 per cent decrease in output from 1981 to 1992 If however different weights are assigned to different categories of output to take into account differences in composition of the cases the result is

a 20 per cent increase in output To measure the productivity of an agency as a whole means overheads must be taken into account Results differ depending on the kind of price index chosen (Ministry of Finance 1997)

MeasUring PUBlic sector

ProDUctiVity in aUstralia

In 1993 the Council of Australian Governments established the Review of Government Service Provision to provide information on the effectiveness and efficiency of government services (Australian Productivity Commission 2006) The review is conducted annually overseen by a steering committee of senior representatives from the central agencies of all the state governments with the assistance of a secretariat provided by the Productivity Commission Performance information is provided on fourteen service areas covering six main government functions education justice emergency management health community services and housing

The report includes performance comparisons across jurisdictions for the services using a common method Both outputs and outcomes are measured as well as efficiency effectiveness and equity Government funding per unit of output delivered is typically used as a measure of technical efficiency eg recurrent funding per annual curriculum hour for vocational education and training Where there are shortcomings in the data other measures

of efficiency are used (including partial productivity ratios such as staff level per student in government schools staff per prisoner in corrective services and administrative costs as a proportion of total expenditure in services for people with a disability) (Australian Productivity Commission 2006)

learning the lessons froM PUBlic sector ProDUctiVity MeasUreMent initiatiVes

The evidence from studies of public sector productivity measurement is that the latter is still in its early stages Despite efforts going back to the 1980s the productivity measures being produced need to be interpreted cautiously There is also the danger that an over-simplistic use of the measures could lead to perverse consequences It is clear that no single figure of productivity can be used for public sector activities unless there is clear and widespread agreement that it is an appropriate measure The Atkinson (2005) recommendation that a range of supporting information is needed to measure productivity change and not to rely solely on a single productivity measure is one that should be applied generally

There is some evidence of output measures produced under the various initiatives being used in a limited manner in resource allocation decisions The OECD (2007) has reviewed the use of output measures in the budgetary process It finds that in most sectors and cases performance measures are loosely connected to decisions in the budgetary process Output measures are often used by ministries of finance in the budgetary process along with other information on performance and on political priorities to inform budget allocations But the OECD warns that a direct linkage where results determine funding creates incentives for gaming the system such as manipulating the data Ministries of finance have taken a cautious approach to using performance information to financially punish or reward agencies When programmes show poor performance against outputs the most common course of action is that resources are held constant and the programme reviewed during the course of the year

5 Ensuring that Performance Measures are Used and Useful

Performance measures are not always used or used in ways that their designers intended Van Thiel and Leeuw (2002) coined the phrase the performance paradox The performance paradox refers to the situation where there is a weak correlation between performance measures and performance itself Measures lose their value over time and can no longer discriminate between good and bad performance As a result the relationship between actual and reported performance declines

Behn (2008) too makes the point that performance measures do not always of themselves guarantee good performance He examines the use of performance measures in structured settings which he refers to as PerformanceStat PerformanceStat is Behnrsquos label for the approach initiated in 1994 by the New York Police Department when they created CompStat a strategy designed to reduce the cityrsquos crime rate The approach was adopted by Baltimore and applied to the whole of city government where it was labelled CitiStat3 The approach has subsequently been applied in a number of other US cities and in Scotland Behn uses the term PerformanceStat as a generic title for such an approach which he defines as follows

ldquoA jurisdiction or agency is employing a PerformanceStat performance strategy if it holds an ongoing series of regular frequent periodic integrated meetings during which the chief executive andor the principal members of the chief executiversquos leadership team plus the individual director (and the top managers) of different sub-units use data to analyse the unitrsquos past performance to follow-up on previous decisions and commitments to improve performance to establish its next performance objectives and to examine the effectiveness of its overall performance strategies (Behn 2008 p2)rdquo

Behn identifies seven big mistakes that organisations often make in applying PerformanceStat approaches These are outlined in Table 5

The key message emerging from both Van Thiel and Leeuw and Behnrsquos analyses is that performance measures need to be seen in a wider management context if they are to be used and useful Some of the main issues that need to be addressed when placing performance measures in this wider context are linking measures and targets performance measurement and citizen engagement and using incentives and sanctions

ENSURING THAT PERFORMANCE MEASURES ARE USED AND USEFUL

taBle 5

seVen Big errors of the PerforMancestat aPProach to the

aPPlication of PerforMance MeasUreMent

Error No 1 No clear purpose In this case the approach is applied as the latest lsquofadrsquo with little or no thought as to what results the organisation is trying to achieve or what better performance might look like

Error No 2 No one has specific responsibilities You need to be able to answer the question of who will be responsible for what

Error No 3 The meetings are held irregularly infrequently or randomly If meetings are not held regularly and frequently they cannot provide good feedback on successes and failures

Error No 4 No one person authorised to run the meetings One person with clear authority needs to consistently run the meetings if they are to have an impact Ideally this should be the chief executive

Error No 5 No dedicated analytic staff PerformanceStat requires data for the measures used For PerformanceStat to provide meaningful results it needs a few analytical people working on it full-time

Error No 6 No follow-up Issues identified by the measures should be followed up from one meeting to the next and not start over each time

Error No 7 No balance between the brutal and the bland Baltimorersquos application of CitiStat is known for being tough and uncompromising with poor performers Sometimes overly so Others have tried to avoid this but in the process have achieved little more than presenting a picture of accomplishments Balance is needed

Source Behn 2008 httpwwwhksharvardeduthebehnreportBehn207PerformanceStatErrorspdf

linking Measures and targets

Targets are an important element in making performance measures useful Good targets can lead to enhanced performance (see local education authorities case study below) In an extensive review of the literature on motivating employees Perry Mesch and Paarlberg (2006) identify three important propositions supported by evidence provided by the literature with regard to targets

1 Challenging and specific targets improvethe performance of employees Target setting increases individual group and work unit performance In many instances specific and challenging targets are associated with higher levels of performance than either no or general lsquodo your bestrsquo targets

2 Setting learning targets as opposed tomerely difficult-to-attain targets may be most effective when tasks are complex When tasks are complex challenging targets may be less helpful In such situations targets that encourage employees to explore strategies to tackle the task may improve performance

3 The target-performance relationship isstrongest when employees are committed to their targets and receive incentives (monetary or otherwise) input and feedback related to the achievement of targets A manager who wants to ensure that challenging targets are met should try to improve the ability of employees to meet these targets and provide feedback on the results of their efforts

case stUDy

target setting for local eDUcation aUthorities

Theories suggest that organisational performance improves if targets for future achievement are used Boyne and Chen (2006) explored this theory using panel data for 147 English local education authorities between 1998 and 2003 The dependent variables in the analysis are exam results for school pupils The authors found that controlling for other variables the extent of performance improvement is influenced positively by the presence of a target The results are consistent with the view that clear and quantified strategic priority targets lead to better organisational outcomes

setting good targets ndash using baselines

Knowing that linking performance measures to targets can enhance performance is useful but not enough on its own How do we know when a lsquogoodrsquo target has been set What might a challenging yet realistic target look like

The World Bank suggests that the identification and use of baseline data can be very helpful when setting targets Kusek and Rist (2004) note that targets are the quantifiable level of a performance measure that a country or organisation wants to achieve by a given time They suggest that one method to establish targets is to start with a baseline indicator level and include the desired level of improvement (taking into consideration available resources over a specific time period) to arrive at the performance target (see Table 6 for a worked example)

Sometimes a target range rather than a specific figure may be appropriate especially where the measures are new and untested However there is a need to guard against the games people sometimes play when setting targets Organisations may set such modest targets that these are achieved quite easily Conversely unachievable targets may be set that just end up demoralising staff Use of the baseline measures can be important here especially if trend data are available to show what a realistic but challenging target might be Similarly external scrutiny of targets by key stakeholders or audit institutions can help in ensuring that over-comfortable targets are not set Some examples of good and bad targets are outlined in Table 7

taBle 6

DeVeloPing targets Using Baseline Data

Outcomes Measures Baselines Targets

Nationrsquos children have better access to preschool programmes

1 Percentage ofeligible urban children enrolled in preschool education 2 Percentageof eligible rural children enrolled in preschool education

1 In 2003 75 percent of children ages 3-5

1 By 2010 85 percent of children ages 3-5

Primary school learning outcomes for children are improved

Percentage of Grade 6 students scoring 70 per cent or better on standardised maths and science tests

In 2005 75 per cent scored 70 per cent or better in maths and 61 per cent scored 70 per cent or better in science

By 2010 80 per cent scoring 70 per cent or better in maths and 67 per cent scoring 70 per cent or better in science

Source adapted from Kusek and Rist 2004 p95

bullbull

bullbullbullbull

taBle 7

exaMPles of gooD anD BaD targets 35

Examples of good targets We will reduce the number of missed household collections by 10 per cent by next year We will aim to collect 95 per cent of the council tax that is due next year We will increase the number of visits to local libraries by 20 per cent before the end of 2009 We will cut the number of unfilled places in primary schools by 10 per cent by 31 December 2010

Examples of poor targets bull We will improve the way we handle complaints

We will buy as many books for the school as possibleWe aim to have the best bus service in the regionWe aim to increase co-operation between school and police authoritiesWe will answer 75 per cent of all letters within 5 daysa poor target if the remaining 25 per cent take 3 months to answer)

Source adapted from UK Audit Commission 2000 p24

PerforMance MeasUreMent anD citizen engageMent

The importance of including measures that citizens want measured in performance measurement systems was discussed in Section 2 Generally there are a number of reasons why the public should be engaged in performance measurement and the reporting of performance

As funders of the service citizensrsquoviews on how well services are provided are an important direct consideration for the staff of public organisations The direct involvement of the public may help legitimise the decisions and priorities of public service organisations

Performance measures developed with itizen input are likely to have more upport from politicians overseeing he work of public organisations

Citizensrsquo views may provide the opportunity and stimulus for innovation suggesting new ways of doing business

The question is not whether the customer should be engaged in performance measurement but how best to engage the public Two issues of central importance here are means of engaging the public and the reporting of performance measures to the public

Means of engaging the public in performance measurement

Ho (2007) suggests a portfolio approachto engaging the public in performancemeasurement Among the methods ofengaging the public are the following

Focus groups Neighbourhood meetings Citizen satisfaction surveys Report cards for programme users Web-based discussion forums Web-based surveys

Each method of engagement has advantages and disadvantages and may vary in terms of cost Ho argues for a diversified portfolio approach that contains multiple strategies to receive diverse and balanced input from citizens from all walks of life

Two of the strategies noted above have received particular attention in recent years ndash report cards and citizen surveys Experience in using each is discussed briefly below to illustrate some of the benefits and challenges of involving the public in performance measurement

Citizen report cards

Citizen report cards are participatory services that provide quantitative feedback on user perception of the quality adequacy and efficiency of public services (see http webworldbankorgWBSITEEXTERNAL TOPICSEXTSOCIALDEVELOPMENTEXTPC ENG0contentMDK20507680~pagePK1 48956~piPK216618~theSitePK41030600 html for details of the report card approach

provided by the World Bank) The use of citizen report cards began in Bangalore4 India and the practice has since spread

to many countries including Canada Denmark the Philippines Sweden Ukraine and the UK The case study below gives

details of the Filipino report card

Three main types of institutional arrangements for report cards are possible Report card by civil society organisation This was the model adopted in Bangalore where the Public Affairs Centre developed the report card in response to anecdotal evidence of customer dissatisfaction with municipal services The report card gives citizen feedback on both quantitative and qualitative dimensions of the selected public services The strength of this approach is the independence and credibility attached to the performance measures The limitations of the approach relate to the fact that not many civil society organisations have the technical capacity and willingness to take on such a job

Report card by government service provider agency This approach has been used in Canada and the UK Here a government agency takes responsibility for the report card production although the actual survey and draft report preparation is often contracted out to a commercial organisation A strength of this approach is the ownership of the exercise by the public agency This is also a potential limitation as the public and politicians may question the independence and objectivity of the findings

37 case stUDy

institUtionalising the filiPino rePort carD

The Department of Budget and Management (DBM) in the Philippines initiated a programme on the development of performance measures for public agencies so as to assess their effectiveness and guide future budget allocations As part of this process they decided to pilot the use of the report card as a means of getting regular user feedback on key public services The incentive for service providers to respond to customer feedback is enhanced knowing that they will be regularly monitored by DBM Analysis and report preparation is contracted out to a credible and independent civil society organisation An advisory panel comprising representatives of service providers other government agencies the private sector sectoral groups and prominent experts has been established to provide advice and guidance on the report card process and its integration with the budgetary process

Some specific lessons emerged from piloting the report card

Focus on key performance measures The first report card spread the net wide and tried to cover many aspects of service delivery Subsequent rounds are to be limited to a few key performance measures The intention is that the measures selected will have a significant overlap with those used by DBM in monitoring outputs and processes facilitating the comparison of results to provide a comprehensive picture of performance

Revising the questionnaire Overlaps in survey questions were identified Greater clarity of wording was identified as needed in some questions Focusing of the questionnaire on the key performance measures common with those used for monitoring outputs and processes by DBM was identified as a significant issue

Improving cost-effectiveness Limiting questions and refining the questionnaire should bring about considerable savings Also limiting the scope to a few principal and common performance measures is intended to focus the attention of public service providers and policy makers and result in concrete actions and follow-up

Source httpsiteresourcesworldbankorgINTPCENG1143333-1116505690049 20509283Filipino+Report+Card+on+Pro-chapter7-report-cardspdf

uuml Report card by government oversight agency This approach has been used by the US and the Philippines It involves a government co-ordinating agency engaging an independent civil society organisation to undertake the design and preparation of the report card in consultation with the public service provider agencies This approach is the most comprehensive both in terms of process and product Similar limitations exist to the civil society organisation approach in terms of the availability of the technical capacity to undertake the work

Citizen surveys Report cards can be seen as a specific type of citizen survey Citizen surveys are generally and widely used to develop measures to assess customer satisfaction with services But there are concerns about the representativeness and validity of citizen surveys How do we know if the right questions are being asked Are they an accurate indication of performance Is the cost of obtaining a sample large enough to be representative really justified

In terms of the benefits of citizen surveys work at the City University of New York provides some interesting pointers (Van Ryzin and Immerwahr 2007 and Van Ryzin Immerwahr and Altman 2008) Using data from New York Cityrsquos street cleanliness scorecard combined with responses to a citizen survey the researchers found a clear and consistent correlation between the scorecard results and citizen rating of street cleanliness The results suggest that citizen judgements about government

performance can correspond with more lsquoobjectivelyrsquo measured outcomes supporting the use of citizen surveys as a measure of government performance

However the researchers urge caution in interpreting their findings Street cleanliness may be an issue that lends itself to measurement through a citizen survey compared to other more complex or less visible services But the findings suggest that in the right circumstances citizen surveys can be used to judge and report on the quality of government services and may be particularly useful where it is too complex or costly to gather data in other ways

The researchers also note that in analysing citizen surveys it can sometimes be difficult to make sense of the ratings citizens give to various services and to use these ratings to meaningfully compare performance They advocate the use of importance-performance analysis (see Table 8) This charts the relationship between service performance ratings and service importance as seen by the customer Efforts should be focused on improving the high importance-low performance sector

Two alternative means of measuring the importance of a service are possible stated importance in response to a survey question that explicitly asks respondents to rate importance and statistically derived importance based on a regression model of overall customer satisfaction in which the services are the independent variables On the basis of comparative analysis the researchers recommend the use of

derived importance because although more analytically complex it has a better predictive ability with regard to consumer behaviour

reporting of performance measures to the public

The way in which performance measures are reported back to citizens affects the degree of engagement of the public and ultimately the usefulness of many performance measures Ho (2007) after examining a wide variety of performance reporting practice identifies nine lessons for reporting performance measurement results to citizens

1 Focus on outcomes and intermediateoutcome measures in public reportingCitizens are primarily interested in theresults delivered by programmes

taBle 8

the iMPortance-PerforMance griD

Low importance High importance

High performance

Potential overkill in terms of effort and possibility of slack resources

Doing a good job providing services that citizens value

Low performance

From a citizen satisfaction perspective services that could be given a lower priority

Services that citizens value but which suffer from poor performance ratings

2 Make the design of publicperformance reports citizen friendlyThe use of figures graphs and chartsto replacecomplementsupplementtables of numbers and text can makea big difference

3 Provide geographic segregationof data Many citizens wantperformance measures to be brokendown by neighbourhood ratherthan be presented for the wholegeographical area covered bythe programme For example theBoston Indicators project websitehas a feature known as lsquoGeographyat a Glancersquo which highlightssome of the key socio-economicmeasures by neighbourhoods (see httpwwwbostonindicatorsorgIndicatorsProject)

4 Provide comparable performancebenchmarks Citizens like to comparetheir neighbourhood with thatof other equivalent communitiesThis helps citizens contextualiseperformance measures

5 Provide stories to explain and elaboratethe data Qualitative descriptionsof specific events can complementquantitative measures Stories ofsuccesses and failures illustratemeasures and make them moremeaningful

6 Organise performance informationby community concerns not bydepartmental or agency structure

Citizensrsquo views of services and programmes cut across organisational and other boundaries They want to know about the issues that concern them not the part of an issue that falls under one particular level of jurisdiction Reporting of performance measures should be organised by topics that citizens are concerned about

7 Use web-based reporting to keep citizensmore informed Web-based reportingcan be updated regularly and cost-effectively It also facilitates userengagement For example visitorsto the online version of Oregonrsquos2007 Benchmark Report (httpbenchmarksoregongov) can selectthe benchmarks that interest themand generate a customised report

8 Blend e-reporting and paper copies Paper copies of reports are stillimportant It is not an eitherorsituation with regard to e-reportingPaper reports for example can bestructured to lead people to thewebsite later

9 Guarantee data accuracy and reliability Public trust and credibility withregard to the performance measuresreported is vital Actions such asconducting performance audits toverify the reliability and validity of thedata behind the measures can helpsecure public trust So can balancedreporting of both successes andfailures

Using incentiVes anD sanctions to encoUrage the Use of PerforMance MeasUres

Performance measures are rarely used simply on the basis that they have been produced Performance measurement is not just a technical exercise it has strong cultural and political components In such an environment incentives and sanctions (carrots and sticks) can play an important role in encouraging the use of performance measures Amongst frequently used incentives and sanctions are the following

Legislation Making performance measurement and reporting a legislative requirement can help create a performance culture in an organisation Measuring performance is more likely to be seen as an important activity worthy of senior management attention Canadarsquos Revenue Agency for example has a legislative requirement to produce a performance report and has stated that this has had a positive impact

Review Internal and external reviews of performance measures and reports can help ensure that measurement is taken seriously The Office of the Auditor General in Canada has regularly reviewed departmental reporting of performance measures

AwardRecognition In 2006 the New South Wales Public Bodies Review Committee in conjunction with the Public Accounts Committee hosted the first Premierrsquos

Annual Reports Award Such initiatives recognise excellence in the reporting of performance measures They can be used to highlight and spread good practice

Assurance Four levels of assurance can be identified (CCAF-FCVI 2007 p38) Assurance is inferred if the user must assess the report content to determine how much confidence to place in the performance measures Assurance is affirmed if management affirms its legal responsibility for reporting Assurance is described if management describes the basis for its judgements the steps it has taken to validate measures and the limitations of the data Assurance is corroborated if a third party (such as an auditor) examines the report and adds assurance (or delivers cautions)

Involvementfeedback Many studies show that if measures are to take root in organisations staff must have an opportunity at some stage to influence the choice of measures used Staff as well as customers should also be given feedback on performance rather than simply feeding information up the line Examples of ways of encouraging participation include developing mixed task teams to develop measures and regular meetings between managers and staff to discuss results

Endnotes

1 The term performance measure as used here includes indicators In the literaturemeasures and indicators are sometimes differentiated with measures referringto quantitative assessment of normative performance and indicators giving aquantitative or qualitative assessment of relative performance

2 The term programme as used here is shorthand for a variety of objects of performancemeasurement including programmes projects operations administrative systemsetc

3 CompStat originated in the New York City Police Department in 1994 under leadershipof Police Commissioner William Bratton and his Deputy Commissioner Jack MapleOn a weekly basis personnel from each of the Departmentrsquos 76 Precincts 9 PoliceService Areas and 12 Transit Districts compile a statistical summary of the weekrsquoscrime complaint arrest and summons activity as well as a written recapitulation ofsignificant cases crime patterns and police activities These data are forwarded tothe Chief of Departmentrsquos CompStat Unit where they are collated and loaded intoa citywide database The data are analysed by computer and a weekly CompStatReport is generated The CompStat Report captures crime complaint and arrestactivity at the precinct patrol borough and citywide levels and presents a concisesummary of these and other important performance measures Comparisonsto previous yearsrsquo activity are shown Precinct commanders and members of theagencyrsquos top management can see emerging and established crime trends as wellas deviations and anomalies and can make comparisons between commands

Also several American cities inspired by its success in Baltimore are engaging with a technology-enabled management approach called CitiStat The mayor of the city has a database of indicators established that tracks performance against his or her political priorities such as reducing lead levels in childrenrsquos blood from traffic fumes and also internal management issues such as overtime and absenteeism The mayor then holds regular meetings with city managers to discuss these data and their trends The mayor persistently questions managers about problems and strategies to deal with these problems Specific targets assignments and deadlines are given to address issues raised The data are actively used to try to improve performance (see Henderson 2003 httpwwwbusinessofgovernmentorgpdfs HendersonReportpdf )

ENDNOTES

4 Inspired by a private sector practice of conducting client satisfaction surveys a groupof people in Bangalore India concerned about the cityrsquos worsening standards of publicservices started an exercise in 1993 to collect feedback from public service users Userviews of the quality efficiency and adequacy of the various services were aggregatedto produce a lsquoreport cardrsquo that rated the performance of public service providers inthe city The findings received media attention which helped mobilise citizen andgovernment support for reform and encouraged the rated agencies to improve theirstandards The exercise was repeated in 1999 Report cards provide a useful meansby which citizens can signal to public service agencies their level of satisfaction ordissatisfaction with services and provide pressure for change For further details seehttpsiteresourcesworldbankorgINTPCENG1143333-111650569004920509275makingpdf

References

Atkinson AB (2005) Atkinson Review Final Report ndash Measurement of Government Output and Productivity for the National Accounts Final report Basingstoke Palgrave Macmillan

UK Audit Commission (2000) On Target The Practice of Performance Indicators London Audit Commission

Australian Productivity Commission (2006) Report on Government Services 2006 Report of the Steering Committee for the Review of Government Service Provision Canberra Australian Productivity Commission (httpwwwpcgovau)

Behn B (2008) The Seven Big Errors of PerformanceStat Policy Briefs John F Kennedy School of Government Harvard University (httpwwwhksharvardeduthebehnreportBehn207PerformanceStatErrorspdf )

Boyne GA and AA Chen (2006) lsquoPerformance targets and public service improvementrsquo Journal of Public Administration Research and Theory Vol17 No3 pp455-477

Boyle R (2005) Civil Service Performance Indicators Committee for Public Management Research Discussion Paper No29 Dublin Institute of Public Administration (httpwwwcpmrgoviepublicationsdiscussion-papers)

CCAF-FCVI Inc (2007) What Can We Learn from Effective Public Performance Reporting Ottawa CCAF-FCVI (httpwwwperformancereportingcadocumentsPPR-Good_Practicespdf )

REFERENCES

Efficiency Unit (2000) Step-by-Step Guide to Performance Measurement Efficiency Unit The Government of the Hong Kong Special Administrative Region (httpwwweugovhkenglishpublicationfilesstep_by_step_guide_to_pmpdf )

Henderson LJ (2003) The Baltimore CitiStat Program Washington DC IBM Center for the Business of Government (httpwwwbusinessofgovernmentorgpdfsHendersonReportpdf )

Ho AT (2007) Engaging Citizens in Measuring and Reporting Community Conditions A Managerrsquos Guide Washington DC IBM Center for the Business of Government (httpwwwbusinessofgovernmentorgpdfsHoReportpdf )

Kaplan RS and DP Norton (1993) lsquoPutting the balanced scorecard to workrsquo Harvard Business Review September-October pp134-147

Kusek JZ and RC Rist (2004) Ten Steps to a Results-Based Monitoring and Evaluation System Washington DC The World Bank

Ministry of Finance (1997) Public Sector Productivity in Sweden Stockholm Ministry of Finance

Neely A and P Micheli (2004) Performance measurement in the UKrsquos public sector linking the national to the local agenda CIMA New Public Sector Seminar Edinburgh 28-29 October 2004 (httpwwwsomcranfieldacuksomresearchcentrescbpdownloads2004BAM_ PMpaperpdf )

REFERENCES

Niemi M (1998) lsquoMeasuring government sector output and productivity in Finland ndash application of the output indicator methodrsquo Agenda item 1 OECD meeting of national accounts experts Statistics Directorate STD NA(98)4 22-25 September Paris OECD

OECD (2007) lsquoTowards Better Measurement of Governmentrsquo OECD Working Papers on Public Governance 20071 Paris OECD Publishing doi101787301575636734 (httpwwwoecdorgdataoecd11613813 4037pdf )

Ogata K and R Goodkey (1998) lsquoRedefining government performancersquo Alberta Finance Canada (httpwwwfinancegovabcapublicationsmeasuringcambridge_paperhtml)

Perrin B (1998) lsquoEffective use and misuse of performance measurementrsquo American Journal of Evaluation Vol19 No3 pp 367-379

Perry JL D Mesch and L Paarlberg (2006) lsquoMotivating employees in a new governance era the performance paradigm revisitedrsquo Public Administration Review Vol 66 No 4 pp505-513

Pollitt C S Harrison R Bal G Doswell and S Jerak (2007) lsquoConceptualising the development of performance measurement systemsrsquo paper presented at European Study Group on Performance and Quality conference Madrid 19-22 September (httpsockuleuvenbeioegpaqualmadridpaperspaper20Pollitt20et20aldoc)

REFERENCES

Schacter M (2002) Not a Tool Kit Practitioners Guide to Measuring the Performance of Public Programs Ottawa Institute on Governance

Stowers G (2004) Measuring the Performance of E-Government Washington DC IBM Center for the Business of Government (httpwwwbusinessofgovernmentorgpdfs8493_Stowers_Reportpdf )

United Way of America (1996) Measuring Program Outcomes A Practical Approach Alexandra Virginia United Way of America

Van Ryzin GG and S Immerwahr (2007) lsquoImportance-performance analysis of citizen satisfaction surveysrsquo Public Administration Vol 85 No 1 pp215-226

Van Ryzin GG S Immerwahr and S Altman (2008) lsquoMeasuring street cleanliness a comparison of New York Cityrsquos scorecard and results from a citizen surveyrsquo Public Administration Review Vol 68 No 2 pp295-303

Van Thiel S and FL Leeuw (2002) lsquoThe performance paradox in the public sectorrsquo Public Performance and Management Review Vol 25 No 3 pp267-281

World Bank (2006) Impact Evaluation ndash The Experience of the Independent Evaluation Group of the World Bank Washington DC The World Bank Independent Evaluation Group (httplnweb18worldbankorgoedoeddoclibnsfDocUNIDViewForJavaSearch35BC4 20995BF58F8852571E00068C6BD$fileimpact_evaluationpdf )

EfficiEncy unit 13fWest Wing central Government Offices 11 ice House Street central Hong Kong

Email euwmeugovhk tel 2165 7255 fax 2524 7267 Website wwweugovhk

Measuring Performance (July 2008)

Measuring Performance July 2008


VISION STATEMENT

MISSION STATEMENT

OTHER EFFICIENCY UNIT DOCUMENTS

Foreword

contents

Executive Summary





1Performance Measurement Key Challenges

2Improving Performance Measurement in Business Planning

DEVELOPING PERFORMANCE MEASURES FOR BUSINESS PLANS

FIGURE 1 LOGIC MODEL RESULTS CHAIN

FIGURE 2USING THE LOGIC MODEL TO DEVELOP PERFORMANCE MEASURES EVIDENTIAL BREATH TESTING

DES MOINES CITIZEN-INITIATED PERFORMANCE ASSESSMENT PROJECT

ASSURING THE QUALITY OF PERFORMANCE MEASURES USED IN BUSINESS PLANNING

TABLE 1 EXAMPLES OF PERFORMANCE MEASURES ANALYSED USING THE SMART CRITERIA

USING MEASURES IN BUSINESS PLANNING DISTINGUISHING BETWEEN MANAGEMENT AND ACCOUNTABILITY PURPOSES

FIGURE 3 USING MEASURES FOR DIFFERENT ACCOUNTABILITY PURPOSES

ASSESSING THE PERFORMANCE OF GOVERNMENT WEBSITES


METHODS FOR MEASURING WEBSITE PERFORMANCE

3Enhancing the Measurement of Policy Outcomes

WHOLE-OF-GOVERNMENT AND CROSS-CUTTING POLICY OUTCOME MEASURES

WHOLE-OF-GOVERNMENT OUTCOME MEASUREMENT

UK CROSS-DEPARTMENTAL PUBLIC SERVICE AGREEMENTS

TABLE 3 COMMUNITY AND LOCAL DEVELOPMENT MEASURES

ADDRESSING TIME-LAG AND ATTRIBUTION ISSUES

INCREASING TAX REVENUE IN PENNSYLVANIA


MEASURING PUBLIC SECTOR PRODUCTIVITY IN THE UK

MEASURING PUBLIC SECTOR PRODUCTIVITY IN FINLAND

TABLE 4 EXAMPLES OF OUTPUT INDICATORS IN THE CENTRAL GOVERNMENT IN FINLAND

MEASURING PUBLIC SECTOR PRODUCTIVITY IN SWEDEN

MEASURING PUBLIC SECTOR PRODUCTIVITY IN AUSTRALIA

LEARNING THE LESSONS FROM PUBLIC SECTOR PRODUCTIVITY MEASUREMENT INITIATIVES


TABLE 5 SEVEN BIG ERRORS OF THE PERFORMANCESTAT APPROACH TO THE APPLICATION OF PERFORMANCE MEASUREMENT

TARGET SETTING FOR LOCAL EDUCATION AUTHORITIES

TABLE 6 DEVELOPING TARGETS USING BASELINE DATA

TABLE 7 EXAMPLES OF GOOD AND BAD TARGETS

PERFORMANCE MEASUREMENT AND CITIZEN ENGAGEMENT

INSTITUTIONALISING THE FILIPINO REPORT CARD

TABLE 8 THE IMPORTANCE-PERFORMANCE GRID

USING INCENTIVES AND SANCTIONS TO ENCOURAGE THE USE OF PERFORMANCE MEASURES

Endnotes

References

Vision stateMent

To be the preferred consulting partner for all government bureaux and departments and to advance the delivery of world-class public services to the people of Hong Kong

Mission stateMent

To provide strategic and implementable solutions to all our clients as they seek to deliver people-based government services We do this by combining our extensive understanding of policies our specialised knowledge and our broad contacts and linkages throughout the Government and the private sector In doing this we join our clients in contributing to the advancement of the community while also providing a fulfilling career for all members of our team

This brief was researched and authored by the Research Division Institute of Public Administration Ireland (wwwipaieresearch) The Research Division provides applied research services for policy makers in a wide range of public service organisations drawing on an extensive network of contacts and experience gained over more than thirty years

other efficiency Unit DocUMents The Efficiency Unit has produced a number of guides on good practice on a wide range of areas including outsourcing and contract management These may be found on the Efficiency Unit website at wwweugovhk

Foreword

contents 1

execUtiVe sUMMary 2

enDnotes 42

references 44

Executive Summary

EXECUTIVE SUMMARY

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

Foreword

contents 1

execUtiVe sUMMary 2

enDnotes 42

references 44

Executive Summary

EXECUTIVE SUMMARY

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

contents 1

execUtiVe sUMMary 2

enDnotes 42

references 44

Executive Summary

EXECUTIVE SUMMARY

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

Executive Summary

EXECUTIVE SUMMARY

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

EXECUTIVE SUMMARY

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

EXECUTIVE SUMMARY

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

EXECUTIVE SUMMARY

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

input measures

bullbull

bull bull

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

odour control

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

activities

Behaviour affected

of progr amm

eFair reflection

p erform ance

final outcomes

Less smoking

perfor mance

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

final oUtcoMes

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

TABLE 4

Source Niemi 1998

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

taBle 5

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

case stUDy

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

taBle 6

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

bullbull

bullbullbullbull

taBle 7

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

37 case stUDy

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

taBle 8

High performance

Low performance

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

Endnotes

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

ENDNOTES

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

REFERENCES



VISION STATEMENT

MISSION STATEMENT


Foreword

contents

Executive Summary









































Endnotes

References

Documents

Measuring Performance (July 2008)€¦ · Measuring Performance . ii . ... We should always remind ourselves to measure the right things, and not just the easy ones. ... ofers a good