Measuring Public Policy - NCSL

Measuring Measuring Public Public Policy Policy

John Turcotte

DirectorProgram Evaluation Division

North Carolina General Assembly

National Association of Legislative Fiscal Officers

Portland, Maine

September 6, 2007

22

John Turcotte, NC General Assembly Program Evaluation Division, September 6, 2007

Presentation OverviewPresentation OverviewImportance of insisting that agency executives focus on what the public wants--not so much on how much money and effort goes into programsHow to decode “accountababel” by differentiating between planning, organization, system, and relative quality termsGetting better outcome measures by using the “So What?” testGetting better utility or “scalability” from unit cost measuresKey questions about performance measures as a whole

33


Public Policy Debate Over Means Often Public Policy Debate Over Means Often Clouds the EndsClouds the Ends

How will we know that we have what we want?

Whatdo we want?

What do we need to do to get what we want?

What resources are necessary?

Who will supply the resources?

In order to comply with state law and to accomplish our mission, we need continued financial support.

We believe we do a good job and could do more if we only had more resources. The public demands this.

Inflation is sapping our ability to serve the people.

Our employees have gone two budget cycles without raises.

Do you have any questions?

In order to comply with state law and to accomplish our mission, we need continued financial support.

We believe we do a good job and could do more if we only had more resources. The public demands this.

Inflation is sapping our ability to serve the people.

Our employees have gone two budget cycles without raises.

Do you have any questions?

44


Accountability is Improved by a Accountability is Improved by a Right to Left ViewRight to Left View

How will we know that we have what we want?

Whatdo we want?

What dowe need to do to get what we want?

What resources are necessary?

Who will supply the resources?

55


AccountababelAccountababel——A Confusing Blend A Confusing Blend ofof Organizational, Planning, Systems Organizational, Planning, Systems and Relative Quality Terms and Relative Quality Terms

Last year, the enforcement program of the Department of Justice adhered to our mission, achieved goals, met objectives, improved services and activities and were

efficient, effective, and economical. We stayed on task, increased outputs and achieved outcomes while

holding inputs constant.

66


Accountability Oversight: Accountability Oversight: How to DeHow to De--Code the Terms Code the Terms Used in the “Discipline”Used in the “Discipline”

PlanningPlanning Terms: Used for setting a Terms: Used for setting a program in motion and measuring progressprogram in motion and measuring progressOrganizationOrganization Terms: Used for sorting out Terms: Used for sorting out where work is donewhere work is doneSystemSystem Terms: Used for modeling the logic Terms: Used for modeling the logic of all parts of a programof all parts of a programRelative QualityRelative Quality Terms: Used for analysis Terms: Used for analysis of alternativesof alternatives

77


Planning TermsPlanning Terms

PremisePremiseGoalsGoalsObjectivesObjectivesTasksTasks

Free Market

• Compelling reason for governmental intervention because the market is not working

• Intervention consists of a scheme that is supposed to produce results

We spend entirely too much time on program and not nearly enough on premise.

Former US Senator Daniel Patrick Moynihan, N.Y.

PremisePremise

Premise

Mission

Goals

Objectives

Tasks

Objectives and Tasks Should be Measurable:

1. Active Verb (To arrest)

2. Quantifier (90%)

3. Object (of suspects)

4. Time Frame (within 72 hours after arrest warrant is issued)

Tasks are components of and action steps required to accomplish an objective

Planning TermsPlanning Terms

Organization Organization TermsTerms

System TermsSystem Terms

Source: Adaptation by Turcotte of W.K. Kellogg Foundation: Using Logic Models to Bring Together Planning, Evaluation, and Action:Logic Model Development Guide, 2004, p. 1.

Planned Work Intended Result

Performance MEASURES

Performance INDICATORS

Financial Audit Performance Audit Program Evaluation

1212


What Should Outcome What Should Outcome Measures Measure?Measures Measure?

Rates of occurrence compared to previous year, to a “stretch” standard previously set, and if possible to comparable jurisdictions

Negatives: inmate idleness, escapes, assaults on fellow inmates and staff, repeat offenders, substance abuse during incarceration, inmate disease and debilitating conditionsPositives: inmate completion of education programs, post incarceration employment and earnings, victim repayment

1313


Use the Use the So WhatSo What? Test to Differentiate ? Test to Differentiate Outputs from Outcomes and Improve Outputs from Outcomes and Improve

Outcome MeasuresOutcome Measures

We examined 6,800 or 94% of inmates for substance abuse in FY 2005. “So what?”We discovered that 20% of them were substance abusers. “So what?”We provided 100% of them with treatment regimens including counseling and appropriate therapies. “So what?”64% of the inmates we have treated to date have completed the program. “So what?”70% of completers were tested substance free after 60 days. “So what?”50% of completers were tested substance free after three years, and 70% of those remained substance free during parole and 90% of these were substance free after two years after completing parole. Our completion and substance-free rates were the highest among all US prison populations. “That’s better.”

1414


Relative Quality TermsRelative Quality Terms

Economical

EfficientEffective

1515


EfficientEfficientThe degree to which a system or The degree to which a system or

component performs its designated component performs its designated functions with minimum consumption functions with minimum consumption of resourcesof resources .. Deploying the minimum Deploying the minimum number of guards needed to secure a number of guards needed to secure a corrections facility.corrections facility.

1616


EffectiveEffectiveAchieves Achieves

resultsresults. E.g. . E.g. provides provides highest level of highest level of security to security to prevent prevent escapes and escapes and assaults on assaults on inmates and inmates and guards. guards.

"Supermax" Penitentiary in Florence, Colorado, (Photo: DLR Group)

1717


EconomicalEconomicalA lower cost

alternative. A house arrest program cost less per inmate than a minimum security facility.

1818


The Holy Grail: Cost The Holy Grail: Cost EffectivenessEffectiveness

Economical

EfficientEffective

1919


ScalabilityScalability is very important!is very important!Could the Legislature build a Could the Legislature build a totaltotal

budget for an activity by multiplying budget for an activity by multiplying each unit cost (cost per output) by each unit cost (cost per output) by the number of expected outputs?the number of expected outputs?

If not, the “unit cost” number is If not, the “unit cost” number is inaccurate and of little use.inaccurate and of little use.

Ditto for time duration per output. Ditto for time duration per output. Should be scalable for computing Should be scalable for computing staff positions needed.staff positions needed.

Key Issue: Scalability of Unit Key Issue: Scalability of Unit Cost and Time Duration Cost and Time Duration

Efficiency MeasuresEfficiency Measures

2020


Key Questions About Key Questions About Performance Performance Measures as a Whole Measures as a Whole

Is the logic model and premise clear and understood?Is the logic model and premise clear and understood?Are measures unequivocal? (No terms such as most, sufficient) Are measures unequivocal? (No terms such as most, sufficient) Are supporting numbers accurate and verifiable?Are supporting numbers accurate and verifiable?Are measures properly classified? (Output or Outcome)Are measures properly classified? (Output or Outcome)Are measures Are measures comprehensivecomprehensive? Are all important or more costly ? Are all important or more costly activities measured?activities measured?Are measures properly constructed? (Appropriate metrics)Are measures properly constructed? (Appropriate metrics)Are there Are there quality quality standards for outputs and outcomes?standards for outputs and outcomes?What are the expected What are the expected quantityquantity standards for outputs and outcomes? standards for outputs and outcomes? Are these a “stretch”?Are these a “stretch”?Are measures portrayed graphically or in easyAre measures portrayed graphically or in easy--toto--use format?use format?

2121


John TurcotteJohn TurcotteDirectorDirector

North Carolina General AssemblyNorth Carolina General AssemblyProgram Evaluation DivisionProgram Evaluation Division

[email protected]@ncleg.net

Documents

Measuring Public Policy - NCSL