Upload
juliet-ellis
View
222
Download
0
Tags:
Embed Size (px)
Citation preview
Developing Public Health Performance Measures: Benefits and Challenges
Mare Schumacher
Maricopa County Department of Public Health
APHA Meetings October 2001
About Us
Maricopa County Department of Public Health in Phoenix, AZ
440 employees
47 distinct programs (from childhood immunizations to HIV education)
85% grant funded
Developing Measures
History
Pre-1998: Collected some metrics (mostly grants)
1998-99: Dept. leaders request collection of measures for all programs
2000: Countywide initiative
2000-01: Complete “family” of measures for all 47 programs
Developing Measures
One employee dedicated to process coordination (about ½ time)
Trained managers/supervisors on:Types of pms (e.g. output, outcome)Proper collection methodsProper documentation
Developing Measures
Supervisors sat down with staff to determine goals
When needed, coordinator worked individually with programs
Biggest challenge: the psychology…
Stages of PM Process*
* Apologies to Elisabeth Kubler-Ross
Denial: “This is ‘flavor of the month’. If I ignore it, it will go away shortly.”
Anger: “I haven’t got time for this!”
Panic/Freeze: PMA (Performance Measure Anxiety)
Depression: “I can’t wait to retire.”
Acceptance: “We have to do this.”
Two Sales OptionsCheerleader Model:
Telling employees how super-fantastic these performance measures will be!!!
Empathy (I Feel Your Pain) Model:
Acknowledging hard work that will be required by employees. Helping them through difficulties.
Collection and Maintenance
Maintain PMs in central document/ electronic location w/gatekeeper
Important to keep narrative explaining significant shifts in numbers
Updated semi-annually, now quarterly
Example: Immunizations
Result: Coverage level - % of clients receiving entire series
Output: # of immunizations, # of clients, # of clients 0-2 years old
Efficiency: Cost perimmunization
Example: Nutrition Education
Output: # of students
Efficiency: Cost per student
Result: Increase in knowledge, attitudes, behavioral intent from pre- to post-test
Using and Abusing Measures
Philosophy
Set Goals
Run ProgramFunding Decisions (“Accountability”)
Measure and Analyze Program Results
Benefits
Long term resource planningStaffBudget
A goal to “shoot for”, rally aroundSticking with priorities – are we using our energies for our top priority?
Do we really measure performance?
Performance Measurement Experiment
County XMaricopa County
Goal for Fiscal Year 01:
Actual Performance:
Goal for Fiscal Year 01:
25 25
Actual Performance:
Performance Measurement Experiment
County XMaricopa County
Goal for Fiscal Year 01:
Actual Performance:
Goal for Fiscal Year 01:
25 25
Actual Performance:
At this point in the presentation, two people – representing each of the counties – must fill a cup with marbles. Maricopa County uses a spoon and County X has to use chopsticks. The result is that County X scores lower on its performance measure. Without knowing that County X had chopsticks, we might conclude that it exhibits “poor performance.”
The “Chopstick Effect”
Funding
Staff shortages
Change in grant requirements
Marlboro has big ad/promotion
And so on…
Issues
May not measure program effort or potential effectiveness
“Widget Factory” model often not applicable in PH
Cost of collecting data some-times not worth it
Goal setting as art
Alternatives
Process evaluation“Up front” research (usually from Academia)Statistical modelingStandard research practices:Test/Control, etc.National Public Health Performance Standards (self-assessment)
ConclusionsYou can develop measures for your organization…But don’t forget the psychological component.Use performance measures wisely – better for planning than evaluating.Let’s find other ways to accurately determine success (for true “accountability”).
InformationMare SchumacherMaricopa County Department of PH1845 E. Roosevelt, Phoenix, AZ [email protected]: “Taking Stock of Performance Measurement: Information Resources for Public Managers”, Government Finance Review, April 2001Resource: “Performance Measures Basics” (presentation for program staff). Send e-mail to Mare Schumacher for copy.
The following is a presentation I used to educate program staff on performance measures at our Department. (It was not presented at the APHA.) Feel free to use it or alter it for your own purposes.I ask nothing in return except that you think of me for a moment every time you use it…
Mare Schumacher
Department of Public Health
Maricopa County, Arizona (Phoenix)
Performance Measures
Presentation to Public Health
Strategic Planning Team
March 1999
The Past
The night before due, make up #s to put in as performance measuresNo one looks at themNo one checks to see if you measuredNo one thinks about it for 11 months and 27 daysNext year, repeat process
The Future
County Administration will be looking closely at performance measures
Depts./Programs will be judged on performance measures
Budget may be tied to performance measures
Making the Best of It
The night before due, make up #s
Live with consequences
Come up with performance measures that are meaningful to County and you
We’ve got two choices:
Suggested Process
May be other options...
Meet with Staff
In a meeting with all or many staffHelps set realistic goalsEmployees know what is
expected of them
“What gets measured gets done.”-- Rose Howe, December 1998
Decide Program Mission
Why do we exist?
What contribution to we make to the people of MC?
NOT What do we do?
Example: HIV Planning
What we do:
Have meetings
Contact coalition members
Prepare reports
Get trained in how to build teams
Why we exist
To create a comprehensive HIV prevention plan that will reduce the spread of HIV
Just talk this
We’re not measuring what we do
We are measuring our contribution to the public health
Doesn’t mean what we do is not important, just not the thing we’re measuring
Create Output Measurement
Output measurement:
Activities
Things we do
Examples:Number of applications processedNumber of clientsNumber of immunizationsNumber of coalition meetings
Example: Immunizations*
Conducted 7500 immunizations in 97/98, etc.
Expect to administer 9500 in 99/00
Reasons: Population growing by 5%, will have increase in staff in 99/00
* All numbers made up -- demonstration purposes only
Create Efficiency Measurement
Efficiency measurement:
Cost per output
Resources used to produce the activity
Examples:
Cost per client
Time spent per client
Cost per client compared to certain standard(s)
Example: Immunizations
Clients per hour = 4 in 97/98
Expect 5 by 2000/01 due to new methods
Cost per client = $1.22
Expect cost = $1.45 by 00/01 due to increase cost of vaccine.
Create Impact (Outcome) Measurement
Impact (outcome) measurement:
Indicate the impact of service on the community
Should flow directly from mission
Examples:
Percentage of clients rehabilitated
Reduction in incidence of disease
Example: Immunizations
90% coverage of school-aged children by 00/01
57% coverage of clients at public health immunization sites
Possibly: 400 cases of measles avoided
All Measures
What data will we use to measure it?
If none available now, how will we get it?
Goals (Future Years)
Make them specific
Make them realistic - how much control do we have over that measure?
What are the reasons for expecting the goal to be higher, lower, etc.
Document why you picked that goal? How did you calculate it?
Timing
Develop measures with staff
Draft meeting with John/Mare
Final draft meeting
Final measures due
End of March/April
End of April (sign-up sheet)
May 11 SP Meeting
June 1