Performance Measurement Series:Defining Performance Indicators and Measures
May 1, 2012
YWCA Great Lakes Alliance Region
Three Part Series:
1. Establishing the Basis for Program Performance Tracking and Measurement
April 17, 2012 9am CT
2. Defining Performance Indicators and Measures May 1, 2012 9am CT
3. Analyzing, Using and Reporting DataMay 15, 2012 9am CT
SESSION 1 Review: Laying the groundwork for
measurement….
SESSION 1 Review: Laying the groundwork for
measurement….
Organizational measurement systems and the role of program performance measurement
The Importance of program logic in establishing the basis for measurement
Outputs vs. outcomes Mission-driven chain of results Selecting outcomes to measure Resources and examples
Organizational measurement systems and the role of program performance measurement
The Importance of program logic in establishing the basis for measurement
Outputs vs. outcomes Mission-driven chain of results Selecting outcomes to measure Resources and examples
4
Steps for Measuring Outcomes
1. Identify desired outcomes and impact. Distinguish outcomes from inputs and activities/outputs
2. Determine outcomes sequence
3. Select outcomes to measure
4. Identify indicators (evidence) and performance standards (achievement levels)
5. Determine data collection approaches
6. Collect data
7. Use and learn from data
SESSION 2 TOPICSMaking measurement real….
SESSION 2 TOPICSMaking measurement real….
Framing indicators Sources of data Validity, reliability and other criteria for
establishing effective measures Measurement problems and challenges Resources and examples
Framing indicators Sources of data Validity, reliability and other criteria for
establishing effective measures Measurement problems and challenges Resources and examples
6
Assess Mission Related Situation
Define Intended Impact and Outcomes to improve the situation
Define Programming/ Service Methods and Approaches to achieve intended impact
Determine Feasibility – Resources Needed and Available to carry out methods and approaches; Environmental Factors that could help or interfere
Establish Systems: Marketing, Performance Measurement, Improvement
Program Planning Steps
Mission
Vision
Strategic Goals & Objectives
Performance Measures
Performance Measures
Performance Measures
Example: Racial Justice Education Program
Short Termknowledge
Intermediatebehavior
Long Termcondition
Increased understanding about racism, privilege and their impact
Desire to work against racism
Increased interaction among people of different racial heritage groups
Commitment to action and action on commitment
Increased relationships between whites, people of color
More integrated community
Decreased disparities based on race
SITUATION INPUTS ACTIVITIES and OUTPUTS
OUTCOMES
Short Term Interm
Long Term
Women represent 15% of elected leadership positions in the community
Recent report indicates few women choose to run
Policy climate not favorable to women; back burner, erosion of choice
Women with experience in political leadership
Funds to support program coordination
Curriculum, trainers
Series of political education workshops reach 100 women
Women in leadership mentor those who complete the series of workshops
Women increase their understan-ding of political leadership process, opportuni-ties
Women believe they can run and win
More women run for office
Women elected to 50% of all positions
Policies more favorable to women
Measuring Outcomes: Framing Indicators
Indicators: the specific, observable, measurable changes that indicate the outcome has been achieved.
The “evidence”….
Begin with: The number of…. The percentage of …. The proportion of…. The change in…..
The # and % of parents who participate in the Family Center Parenting Program who report that their skills have increased in at least 4 of the 5 areas of nurturing.”
Identifying Indicators for Outcomes we will Measure
More about Indicators
Observable and measurable “milestones” toward an outcome.
What you would see, hear, read, etc., that would indicate whether you're making any progress toward your outcome target or not
Example: The number and percent of teen participants who quit
smoking right after the program and six months after the program.
ExampleProgram: Financial Empowerment
Outcome: Women who participate in the financial
management series will increase their financial management knowledge and skills.
Indicators: # and % of women who complete the series that report
increased knowledge at the end of the session. Of those who complete, # and % of women who develop
budgets and financial goals.
Developing Outcome Indicators
Be clear about the outcome Reflect on the problem—especially root
causes--driving the program What can be observed? Identify what evidence will indicate whether
the outcome is achieved Use specific wording
The Importance of Specificity
What will we accomplish with this portion of our population within this period of time
In contrast to mission, vision, goals and objectives, which tend to be multiple and broad, outcome targets are narrow and focused. They speak to specific benefits for specific customers.
Outcome: Girls who complete the intensive math program will increase their proficiency in math.
INSUFFICIENT INDICATOR
Girls who demonstrate improved skills on the post-program test.
BETTER INDICATORS
The # and % of girls who complete the program, who:Demonstrate improved skills on the post-program test;
and achieve grade B or better in math by the end of the school year.
Example
Program: Transitional Housing
Outcome: Women secure safe and affordable housing.
Indicators: Number and percent of women who move to long term
housing within 24 months of entry. Number and percent of women who maintain safe and
permanent housing for six months or more. Number and percent of clients who do not re-enter the
homeless system within one year of obtaining permanent housing.
Example
Program: Racial Justice Education
Outcome: Participants increase their understanding of racial justice.
Indicators: Number and percent of participants who complete the
program and report their understanding of racism and white privilege increased because of their participation in the program.
Number and percent of completers who are able to identify examples of racism and white privilege.
Outcome Indicators Checklist Is there at least one indicator for each outcome? Relevant to the outcome? Important to the outcome? Observable and measurable? Sufficiently specific? Unambiguous? Understandable to information users?
Outcome Indicators Checklist
Feasible to collect reasonably valid data? Is there is sufficient basis for numerical targets
if they are included? Is a timeframe for achievement specified?
How much service is required before a participant will have reasonable potential for achieving the outcome?
SITUATION INPUTS ACTIVITIES and OUTPUTS
OUTCOMES
Short Term Interm.
Long Term
Women represent 15% of elected leadership positions in the community
Recent report indicates few women choose to run
Policy climate not favorable to women; back burner, erosion of choice
Women with experience in political leadership
Funds to support program coordination
Curriculum, trainers
Series of political education workshops reach 100 women
Women in leadership group mentors 25
Women’s leadership summit defines women’s leadership
Women increase their understanding of political campaign processes, opportunities
Women believe they can run and win
More women run for office
Women elected to 50% of all positions
Policies more favorable to women
Example
Program: Women’s Political Leadership
Outcome: Women increase their understanding of political campaign processes, opportunities
Indicators: Number and percent of women who complete the program
and report their understanding of campaign processes has increased as a result of their participation.
Number and percent of completers who are able to identify opportunities for engagement in political leadership.
Measuring Outcomes:
Collecting Data on Outcome Indicators
Challenges to Measuring Outcomes
Aren’t there some things that we just can’t measure?
Pre-and post tests are too cumbersome. What can we really influence? Where do we begin? How can racial justice be measured?
Determining Data Collection: Considerations
Where will information for indicators come from or be recorded?
Does the data already exist?
If not, can we modify an existing tool?
How often should the data be collected?
Do we collect data for all customers or a sample?
Examples: Program records, tests, surveys, other reports, documented observation.
“Parents who participated in the program will be asked to complete an assessment of their parenting skill changes at the end of the program.”
25
Sources of Indicator Data
Examine outcomes and indicators of similar programs/ services.
Talk to program staff. Talk to current and former customers. Find research about the problem and proven
solutions. Use situation/problem as the basis.
Sources and Methods of Data Collection
Sources of informationParticipantsParentsTeachersStaff, others
Method of collecting the informationRecordsSurveyInterviewObservationEnd-of-program questionnaire/ testFocus group
Organizational Records
Your Program RecordsWhat data do you already collect? Service engagement Program completion Test scores
65% of adults who complete the GED series will achieve the scores needed to complete their GED.
Organizational Records
Records from other Organizations Grades Achievements Student test scores
Of the girls who regularly attend the math maniacs series, 75% will achieve a grade of B or higher in mathematics courses.
Organizational Records
ADVANTAGES Readily available/ low
cost Staff familiarity,
control
DISADVANTAGES Records seldom
adequate Records from other
organizations may be difficult to secure
Trained Observer Ratings
Reporting observable changes in behavior, skill, condition
Utilize trained observers Carefully constructed ratings tool
Youth who complete the leadership development series will demonstrate the 4 leadership skills based on documented review of their presentations and interactions by program staff.
Trained Observer Ratings
ADVANTAGES Availability of
professional perspectives
Good for populations that cannot complete surveys
DISADVANTAGES Observer time Potential for bias Potential for
unreliable observations
Customer Surveys
Changes reported by customers due to program engagement Self-reports post program Pre and post
Of those who complete the racial justice education series, 80% will report on the post-program assessment that their understanding of racism has increased as a result of their participation.
Information that can be gained through Customer Surveys
Customer knowledge and attitudes after receiving services
Customer action or behavior after receiving services
Customer attribution of changes to program participation
Demographic information about participants
Customer Surveys
ADVANTAGES Fairly feasible Provide direct input
from participants
DISADVANTAGES Require level of
expertise Potential time and
cost Reliability of
participant responses
Gathering data from customers: 4 Levels
1. Reactions and feelings (feelings are often poor indicators that your service made lasting impact)
2. Learning (enhanced attitudes, perceptions or knowledge)
3. Changes in skills (applied the learning to enhance behaviors)
4. Changes in behavior (improved performance)
Developing Data Collection Approaches
Seek advice from university, consultant Look for existing tested approaches
programs Pilot test Confidentiality Customer Consent
Selecting Data Collection Methods: Additional Considerations
1. Of this information, how much can be collected and analyzed in a low-cost and practical manner, e.g., using questionnaires, surveys and checklists?
2. How accurate will the information be?3. Will the methods get all of the needed information?4. What additional methods should and could be used if
additional information is needed?5. Will the information appear as credible to decision makers,
e.g., to funders or top management?6. Will the nature of the audience conform to the methods, e.g.,
will they fill out questionnaires carefully?7. Who can administer the methods now or is training required?8. How can the information be analyzed?
Validity and Reliability
The perfect performance indicator may never exist because there is always the possibility of some error in the measurement process. Poister 2003
“Good” data provides: unbiased indicators that are appropriate measures of performanceA reasonable level of objective statistical reliability
Valid Indicators and Measures
Provide fair, unbiased data Are directly related to and represent what is
to be measured
Problem: Reported crime rates tend to underestimate
the number of crimes committed due to underreporting.
Reliable Indicators and Measures
Yield objective, precise, dependable data Yield consistent responses
Problem: Trained observers report significantly
different results for the same observation.
Factors that can Jeopardize Validity and Reliability
Poor instrument design Desire to “look good” Survey, observer, interviewer bias Systematic patterns of incomplete data
collection Inconsistent data collection, scoring, etc.
Validity, like reliability, is still a matter
of degree. Poister, 2003
Next Session:Analyzing, Using and Reporting
Outcomes Data
May 15, 2012 10am ET/ 9am CT Benchmarking Data trends Data management systems Role of leadership Resources and examples
Resources to Assist with Measuring Outcomes
Urban Institute www.urban.org Has developed “outcomes taxonomy” for fourteen
service areas
Performance Measurement: Getting Results by Harry Hatry, 2nd ed. 2006 Offers tips on indicator development, sources of data,
customer survey design, trained observer ratings design, etc.
Measuring Performance in Public and Nonprofit Organizations by Theodore Poister, 2003
Resources for Measuring Outcomes
The Grantsmanship Center www.tgci.com Offers guides to program evaluation
United Way www.unitedway.org/outcomes Offers outcome measurement resources, guides
W.K. Kellog Foundation www.wkkf.org Publishes Logic Model Development Guide and Evaluation
Handbook
Performance Management Network, inc. www.pmn.net search: logic models Instruction on logic models framework and development