Performance Measurement Series:Defining Performance Indicators and Measures

May 1, 2012

YWCA Great Lakes Alliance Region

Three Part Series:

1. Establishing the Basis for Program Performance Tracking and Measurement

April 17, 2012 9am CT

2. Defining Performance Indicators and Measures May 1, 2012 9am CT

3. Analyzing, Using and Reporting DataMay 15, 2012 9am CT

SESSION 1 Review: Laying the groundwork for

measurement….

SESSION 1 Review: Laying the groundwork for

measurement….

Organizational measurement systems and the role of program performance measurement

The Importance of program logic in establishing the basis for measurement

Outputs vs. outcomes Mission-driven chain of results Selecting outcomes to measure Resources and examples

Organizational measurement systems and the role of program performance measurement

The Importance of program logic in establishing the basis for measurement

Outputs vs. outcomes Mission-driven chain of results Selecting outcomes to measure Resources and examples

4

Steps for Measuring Outcomes

1. Identify desired outcomes and impact. Distinguish outcomes from inputs and activities/outputs

2. Determine outcomes sequence

3. Select outcomes to measure

4. Identify indicators (evidence) and performance standards (achievement levels)

5. Determine data collection approaches

6. Collect data

7. Use and learn from data

SESSION 2 TOPICSMaking measurement real….

SESSION 2 TOPICSMaking measurement real….

Framing indicators Sources of data Validity, reliability and other criteria for

establishing effective measures Measurement problems and challenges Resources and examples

Framing indicators Sources of data Validity, reliability and other criteria for

establishing effective measures Measurement problems and challenges Resources and examples

6

Assess Mission Related Situation

Define Intended Impact and Outcomes to improve the situation

Define Programming/ Service Methods and Approaches to achieve intended impact

Determine Feasibility – Resources Needed and Available to carry out methods and approaches; Environmental Factors that could help or interfere

Establish Systems: Marketing, Performance Measurement, Improvement

Program Planning Steps

Mission

Vision

Strategic Goals & Objectives

Performance Measures

Performance Measures

Performance Measures

Example: Racial Justice Education Program

Short Termknowledge

Intermediatebehavior

Long Termcondition

Increased understanding about racism, privilege and their impact

Desire to work against racism

Increased interaction among people of different racial heritage groups

Commitment to action and action on commitment

Increased relationships between whites, people of color

More integrated community

Decreased disparities based on race

SITUATION INPUTS ACTIVITIES and OUTPUTS

OUTCOMES

Short Term Interm

Long Term

Women represent 15% of elected leadership positions in the community

Recent report indicates few women choose to run

Policy climate not favorable to women; back burner, erosion of choice

Women with experience in political leadership

Funds to support program coordination

Curriculum, trainers

Series of political education workshops reach 100 women

Women in leadership mentor those who complete the series of workshops

Women increase their understan-ding of political leadership process, opportuni-ties

Women believe they can run and win

More women run for office

Women elected to 50% of all positions

Policies more favorable to women

Measuring Outcomes: Framing Indicators

Indicators: the specific, observable, measurable changes that indicate the outcome has been achieved.

The “evidence”….

Begin with: The number of…. The percentage of …. The proportion of…. The change in…..

The # and % of parents who participate in the Family Center Parenting Program who report that their skills have increased in at least 4 of the 5 areas of nurturing.”

Identifying Indicators for Outcomes we will Measure

More about Indicators

Observable and measurable “milestones” toward an outcome.

What you would see, hear, read, etc., that would indicate whether you're making any progress toward your outcome target or not

Example: The number and percent of teen participants who quit

smoking right after the program and six months after the program.

ExampleProgram: Financial Empowerment

Outcome: Women who participate in the financial

management series will increase their financial management knowledge and skills.

Indicators: # and % of women who complete the series that report

increased knowledge at the end of the session. Of those who complete, # and % of women who develop

budgets and financial goals.

Developing Outcome Indicators

Be clear about the outcome Reflect on the problem—especially root

causes--driving the program What can be observed? Identify what evidence will indicate whether

the outcome is achieved Use specific wording

The Importance of Specificity

What will we accomplish with this portion of our population within this period of time

In contrast to mission, vision, goals and objectives, which tend to be multiple and broad, outcome targets are narrow and focused. They speak to specific benefits for specific customers.

Outcome: Girls who complete the intensive math program will increase their proficiency in math.

INSUFFICIENT INDICATOR

Girls who demonstrate improved skills on the post-program test.

BETTER INDICATORS

The # and % of girls who complete the program, who:Demonstrate improved skills on the post-program test;

and achieve grade B or better in math by the end of the school year.

Example

Program: Transitional Housing

Outcome: Women secure safe and affordable housing.

Indicators: Number and percent of women who move to long term

housing within 24 months of entry. Number and percent of women who maintain safe and

permanent housing for six months or more. Number and percent of clients who do not re-enter the

homeless system within one year of obtaining permanent housing.

Example

Program: Racial Justice Education

Outcome: Participants increase their understanding of racial justice.

Indicators: Number and percent of participants who complete the

program and report their understanding of racism and white privilege increased because of their participation in the program.

Number and percent of completers who are able to identify examples of racism and white privilege.

Outcome Indicators Checklist Is there at least one indicator for each outcome? Relevant to the outcome? Important to the outcome? Observable and measurable? Sufficiently specific? Unambiguous? Understandable to information users?

Outcome Indicators Checklist

Feasible to collect reasonably valid data? Is there is sufficient basis for numerical targets

if they are included? Is a timeframe for achievement specified?

How much service is required before a participant will have reasonable potential for achieving the outcome?

SITUATION INPUTS ACTIVITIES and OUTPUTS

OUTCOMES

Short Term Interm.

Long Term

Women represent 15% of elected leadership positions in the community

Recent report indicates few women choose to run

Policy climate not favorable to women; back burner, erosion of choice

Women with experience in political leadership

Funds to support program coordination

Curriculum, trainers

Series of political education workshops reach 100 women

Women in leadership group mentors 25

Women’s leadership summit defines women’s leadership

Women increase their understanding of political campaign processes, opportunities

Women believe they can run and win

More women run for office

Women elected to 50% of all positions

Policies more favorable to women

Example

Program: Women’s Political Leadership

Outcome: Women increase their understanding of political campaign processes, opportunities

Indicators: Number and percent of women who complete the program

and report their understanding of campaign processes has increased as a result of their participation.

Number and percent of completers who are able to identify opportunities for engagement in political leadership.

Measuring Outcomes:

Collecting Data on Outcome Indicators

Challenges to Measuring Outcomes

Aren’t there some things that we just can’t measure?

Pre-and post tests are too cumbersome. What can we really influence? Where do we begin? How can racial justice be measured?

Determining Data Collection: Considerations

Where will information for indicators come from or be recorded?

Does the data already exist?

If not, can we modify an existing tool?

How often should the data be collected?

Do we collect data for all customers or a sample?

Examples: Program records, tests, surveys, other reports, documented observation.

“Parents who participated in the program will be asked to complete an assessment of their parenting skill changes at the end of the program.”

25

Sources of Indicator Data

Examine outcomes and indicators of similar programs/ services.

Talk to program staff. Talk to current and former customers. Find research about the problem and proven

solutions. Use situation/problem as the basis.

Sources and Methods of Data Collection

Sources of informationParticipantsParentsTeachersStaff, others

Method of collecting the informationRecordsSurveyInterviewObservationEnd-of-program questionnaire/ testFocus group

Organizational Records

Your Program RecordsWhat data do you already collect? Service engagement Program completion Test scores

65% of adults who complete the GED series will achieve the scores needed to complete their GED.

Organizational Records

Records from other Organizations Grades Achievements Student test scores

Of the girls who regularly attend the math maniacs series, 75% will achieve a grade of B or higher in mathematics courses.

Organizational Records

ADVANTAGES Readily available/ low

cost Staff familiarity,

control

DISADVANTAGES Records seldom

adequate Records from other

organizations may be difficult to secure

Trained Observer Ratings

Reporting observable changes in behavior, skill, condition

Utilize trained observers Carefully constructed ratings tool

Youth who complete the leadership development series will demonstrate the 4 leadership skills based on documented review of their presentations and interactions by program staff.

Trained Observer Ratings

ADVANTAGES Availability of

professional perspectives

Good for populations that cannot complete surveys

DISADVANTAGES Observer time Potential for bias Potential for

unreliable observations

Customer Surveys

Changes reported by customers due to program engagement Self-reports post program Pre and post

Of those who complete the racial justice education series, 80% will report on the post-program assessment that their understanding of racism has increased as a result of their participation.

Information that can be gained through Customer Surveys

Customer knowledge and attitudes after receiving services

Customer action or behavior after receiving services

Customer attribution of changes to program participation

Demographic information about participants

Customer Surveys

ADVANTAGES Fairly feasible Provide direct input

from participants

DISADVANTAGES Require level of

expertise Potential time and

cost Reliability of

participant responses

Gathering data from customers: 4 Levels

1. Reactions and feelings (feelings are often poor indicators that your service made lasting impact)

2. Learning (enhanced attitudes, perceptions or knowledge)

3. Changes in skills (applied the learning to enhance behaviors)

4. Changes in behavior (improved performance)

Developing Data Collection Approaches

Seek advice from university, consultant Look for existing tested approaches

programs Pilot test Confidentiality Customer Consent

Selecting Data Collection Methods: Additional Considerations

1. Of this information, how much can be collected and analyzed in a low-cost and practical manner, e.g., using questionnaires, surveys and checklists?

2. How accurate will the information be?3. Will the methods get all of the needed information?4. What additional methods should and could be used if

additional information is needed?5. Will the information appear as credible to decision makers,

e.g., to funders or top management?6. Will the nature of the audience conform to the methods, e.g.,

will they fill out questionnaires carefully?7. Who can administer the methods now or is training required?8. How can the information be analyzed?

Validity and Reliability

The perfect performance indicator may never exist because there is always the possibility of some error in the measurement process. Poister 2003

“Good” data provides: unbiased indicators that are appropriate measures of performanceA reasonable level of objective statistical reliability

Valid Indicators and Measures

Provide fair, unbiased data Are directly related to and represent what is

to be measured

Problem: Reported crime rates tend to underestimate

the number of crimes committed due to underreporting.

Reliable Indicators and Measures

Yield objective, precise, dependable data Yield consistent responses

Problem: Trained observers report significantly

different results for the same observation.

Factors that can Jeopardize Validity and Reliability

Poor instrument design Desire to “look good” Survey, observer, interviewer bias Systematic patterns of incomplete data

collection Inconsistent data collection, scoring, etc.

Validity, like reliability, is still a matter

of degree. Poister, 2003

Next Session:Analyzing, Using and Reporting

Outcomes Data

May 15, 2012 10am ET/ 9am CT Benchmarking Data trends Data management systems Role of leadership Resources and examples

Resources to Assist with Measuring Outcomes

Urban Institute www.urban.org Has developed “outcomes taxonomy” for fourteen

service areas

Performance Measurement: Getting Results by Harry Hatry, 2nd ed. 2006 Offers tips on indicator development, sources of data,

customer survey design, trained observer ratings design, etc.

Measuring Performance in Public and Nonprofit Organizations by Theodore Poister, 2003

Resources for Measuring Outcomes

The Grantsmanship Center www.tgci.com Offers guides to program evaluation

United Way www.unitedway.org/outcomes Offers outcome measurement resources, guides

W.K. Kellog Foundation www.wkkf.org Publishes Logic Model Development Guide and Evaluation

Handbook

Performance Management Network, inc. www.pmn.net search: logic models Instruction on logic models framework and development