Abt Associates |pg 1

Performance Management Systems and Evaluation:Towards a Mutually Reinforcing Relationship

Jacob Alex Klerman (Abt Associates)

APPAM/HSE Conference “Improving the Quality of Public Services”Moscow, June 2011

Abt Associates |pg 2

Performance Management Systems and Evaluation

Performance ManagementEvaluation

Abt Associates |pg 3

Performance Management Systems and Evaluation

Performance Management The need is clear

– “What gets measured gets done”– If you know what you want done; you need to manage against it– To manage against it, you need to measure it

Evaluation

Abt Associates |pg 4

Performance Management Systems and Evaluation

Performance Management The need is clear

– “What gets measured gets done”– If you know what you want done; you need to manage against it– To manage against it, you need to measure it

Evaluation But what do you want done?

– Are you sure?

That’s the role of rigorous impact evaluation– Dirty little secret: much of what we do—much of what seems

“plausible”—has minimal impact (or even hurts)

Abt Associates |pg 5

Everyone wants better program outcomes– We might even be willing to spend more if we could prove

better outcomes

Proving “better outcomes” requires rigorous impact evaluation– Many apparently plausible programs (and program

innovations) don’t work– Naive evaluation methods give the wrong answer

Rigorous impact evaluation is challenging– Requiring large samples– And, the smaller the projected incremental impact,

the larger the required samples

Rigorous Impact Evaluation Is Crucial

Abt Associates |pg 6

Current Practice

A Better Way

Closing Thoughts

Outline

Abt Associates |pg 7

Current Evaluation Practice Isn’t Very Useful

Asks the wrong questions: Does the program “work”?– i.e., Should we shut the program down?– Big programs address major social problems– The programs aren’t going away

Abt Associates |pg 8

Current Evaluation Practice Isn’t Very Useful

Asks the wrong questions: Does the program “work”?– i.e., Should we shut the program down?– Big programs address major social problems– The programs aren’t going away

The right question is often:How can we make the program better?– Which program model works better?– Would some minor—and affordable—change in program

design help?– For which subgroups does our program work?

Target the program at them

Abt Associates |pg 9

Answering up/down evaluation question requires (relatively) small samples– For a training program, perhaps 500-2,000 case

Answering practitioners’ questions requires much large samples– For a training program, perhaps 10,000+ cases

At current evaluation cost—$1,000+ per case—we can’t afford to answer practitioner’s questions– Especially if the change in outcomes will be at best small– And that’s a big problem because CQI/kaizen suggests that major

improvement often comes from lots of small improvements

The Realities of Sample Size and Cost

Abt Associates |pg 10

Answering up/down evaluation question requires (relatively) small samples– For a training program, perhaps 500-2,000 case

Answering practitioners’ questions requires much large samples– For a training program, perhaps 10,000+ cases

At current evaluation cost—$1,000+ per case—we can’t afford to answer practitioner’s questions– Especially if the change in outcomes will be at best small– And that’s a big problem because CQI/kaizen suggests that major

improvement often comes from lots of small improvements

The Realities of Sample Size and Cost

To answer practitioner’s questions, we’re going to need to get the cost way down

Abt Associates |pg 11

Negotiate access to sites, including convincing them to deny service to some applicants – <time consuming and expensive>

Detailed process analysis at each site – <expensive>

Detailed survey follow-up – <expensive for each case; and there are a lot of cases>

Steps in a Current Evaluation

Abt Associates |pg 12

Negotiate access to sites, including convincing them to deny service to some applicants – <time consuming and expensive>

Detailed process analysis at each site – <expensive>

Detailed survey follow-up – <expensive for each case; and there are a lot of cases>

Steps in a Current Evaluation

Is there another way? Sometimes, yes …

Abt Associates |pg 13

Current Practice

A Better Way

Closing Thoughts

Outline

Abt Associates |pg 14

We have just argued that collecting information on outcomes drives costs

Performance measurement systems already collect information on outcomes– Presumably on the key outcomes

So, when we can measure outcomes through the performance measurement system– Costs will be much, much lower– Allowing large samples– A key requirement for evaluating incremental changes

Will only work when both treatment and control are “in the system” (e.g., incremental changes)

At the Back End—Leverage Ongoing Performance Management Systems

Abt Associates |pg 15

Currently research is “top down”– Someone outside the system decides to evaluate X– Then, evaluator tries to convince sites to adopt X;

and to deny all services to a control group

At the Front End—A Learning Organization

Abt Associates |pg 16

Currently research is “top down”– Someone outside the system decides to evaluate X– Then, evaluator tries to convince sites to adopt X;

and to deny all services to a control group

Alternative is “bottom up”– Ask sites to suggest policy/program changes– Form a committee—site representatives, central program staff,

substance experts, evaluation experts– Ask them to make a short list from among the suggestions– Ask sites to volunteer to implement the selected suggestions– Randomize at the site level:

control condition is “current practice”, not “no service”(no one is denied service)

At the Front End—A Learning Organization

Cutting time and costs

Abt Associates |pg 17

In Summary

Now Better

Negotiate access to sites <expensive and time consuming>

They volunteer

Detailed process analysis <expensive, not useful>

Skip this

Collect detailed survey outcome data <very, very expensive>

Use Performance Management System data

And when your costs drop sharply, CQI is now feasible; i.e., you can evaluate small changes

Abt Associates |pg 18

Current Practice

A Better Way

Closing Thoughts

Outline

Abt Associates |pg 19

A True Learning Organization

Performance measurement is an ongoing task CQI/Continuous Quality Improvement; i.e.,

– Proposing small changes to SOP/Standard Operating Procedures– Rigorously evaluating those small changes– Adopting those that can be shown to “help”

… Should also be an ongoing task The key insight of “kaizen” is that improved outcomes arise

from the accumulation of lots of such small changes

Data already collected as part of Performance Management Systems makes such CQI feasible

Abt Associates |pg 20

Site level randomization needs lots (50-200) of, relatively similar, sites– Sites can be small (15-100 cases)

Central organization controls resources– Much easier to get volunteers, when volunteering is the

only way to get more resources

When Will this Work?

We’re looking for test cases. Any volunteers?

Performance Management Systems and Evaluation:Towards a Mutually Reinforcing Relationship

Jacob Alex Klerman

APPAM/HSE Conference “Improving the Quality of Public Services”Moscow, June 2011