Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement...

Evaluating Ongoing Programs:

A Chronological Perspective to Include Performance Measurement

Summarized from Berk & Rossi’s Thinking About Program Evaluation, Sage, 1990; Martin & Kettner’s Measuring the Performance of Human Service Programs, Sage, 1996

Stages of Assessment

• Stage 1: Determining Whether the Program is Reaching the Appropriate Beneficiaries.

• Stage 2: Making Sure the Program is Being Properly Delivered

• Stage 3: Ensuring Funds Are Being Used Appropriately

• Stage 4: Ensuring Effectiveness can be Estimated• Stage 5: Determining Whether the Program

Works• Stage 6: Determining Program Worth

Stage One: Program Impact

Program Impact Research is designed to identify who is actually served by a program to determine the number being served that meet program service criteria and those that are being served that do not meet service criteria.

Stage Two: Program Integrity

Program Integrity Research analyzes the essentials of program delivery such as:

• personnel qualifications & skill assessment

• consistency of program services with program mission

• targeting & marketing of services

• service coordination

Stage Three: Fiscal Accountability

Accountant Perspective:

• Out of Pocket Expenses

• Historical Costs

• Depreciation

• Current & Anticipated Revenues

• Product Inventory

• Income & Outgo of Funds

Stage Three: Fiscal Accountability

Economist Perspective & Opportunity Costs

• Opportunity Costs May be Considered as What was Given Up to Direct Resources in a Particular Direction

• Opportunity Costs May Also Be Construed as the “Next Best Use” of Resources

Stage Four: Evaluability

Criteria for Evaluability

• Clarifying Goals

• Specifying Program Goals

• Determining Possible Outcomes in the Absence of the Program

Stage Five: Program Effectiveness

• Comparisons Across Subjects

• Comparison Across Settings

• Comparison Across Time

• Comparison Across Criterion

• Pooled Comparisons

Research Designs for Estimating Effectiveness

• Random Assignment: Comparing Mean Outcomes of Control & Experimental S’s

• Interrupted Time Series: (Pre- and Post-Assessment Model)

• Cross Sectional Designs: Comparisons of Different Types of Units (e.g. comparing smaller & larger cities) with Comparisons Occurring at Only One Point In Time

Research Designs for Estimating Effectiveness

• Regression Time Series: Assignment of S’s by variables (Criterion Based). Objectives:– To provide estimates of values of the dependent

variable (outcome variable) from values of the independent variable (assignment Variable)

– To obtain measures of the error involved in using the regression line as a basis of estimation (I.E. Standard Error of Estimate)

– To obtain a measure of the degree of association or correlation between the two variable

Research Designs for Estimating Effectiveness

• Pooled Cross Sectional & Time Series: – Randomized Experiments &– Regression Designs

May Be Compared

– Across Units (Cross Sectional) &– Across Time (Time Series)

Stage Six: Cost Effectiveness

Ongoing Versus New Programs:

• Ongoing Programs Have Historical Data to work With.

• New programs lack such historical data from which to determine cost effectiveness

Performance Measurement

Defining Performance Measurement

The regular collection of and reporting of information about the efficiency, quality, and effectiveness of human service programs. (Urban Institute, 1980)

Perspectives of Performance Measurement

• Efficiency Perspective

• Quality Perspective

• Effectiveness Perspective

Systems Model Essentials

• Inputs: Includes anything used by a system to achieve its purpose

• Process: Involves the treatment or delivery process in which inputs are consumed to produce outputs

• Outputs: That which is produced

• Feedback: System information reintroduced into the process to improve quality, efficiency & effectiveness

Efficiency Perspective

• Productivity = ratio of outputs to inputs

• Efficiency = maximizing outputs to inputs– Efficiency can not reflect whether program

goals are being met – Inefficiency is how many programs are

regarded by the public - often in the absence of a full understanding of the goals, mission, clientele, resources, and services of the agency

Quality Perspective

• Typically involves benchmarking against standards and criteria of excellence (as in TQM, or Total Quality Management)

• TQM now defines productivity as the ratio of outputs that meet a specified quality standard

Effectiveness Perspective

• Focuses on outcomes such as the results, impacts and accomplishments of programs

• Effectiveness is the highest form or performance accountability

• Focuses upon which intervention works in which settings

• Effectiveness accountability is primarily concerned with ratios of outcomes to inputs.

Reasons for Adopting Performance Measurement

• Performance measurement has the potential to improve the management of human service programs

• Performance measurement has the potential to affect the allocation of resources to human service programs

• Performance measurement may be a forced choice for many, if not, most human service programs

Key Questions in Performance Measurement

• Who are the clients?

• What are their demographic characteristics?

• What are their social or presenting problems?

• What services are they receiving?

• In what amounts?

• What is the level of service quality?

• What results are being achieved?

• At what costs?

Performance Measurement as a Management Tool

• Performance Measurement promotes client centered approaches to service delivery

• Provides a shared language for comparing human service programs for quality, efficiency, & effectiveness

• Allows administrators to continuously monitor programs to identify areas for improvement

• Provides direct feedback to personnel, allowing them to improve their service provision

Performance Measurement Programs

Government Performance & Results Act (1993)

• Effective 1998, all federal agencies must begin reporting effectiveness data for their services & products

• This requirement will be passed on to agency contractors & subcontractors

• Increasingly Federal block - grant programs also have this requirement

National Performance Review

• Refers to governmental efforts at instituting program effectiveness, efficiency, and quality, to implement the 1992 report on government practices entitled Reinventing Government (Osborn & Gaebler, 1992)

Total Quality Management Movement

• National Movement to Improve Quality

• Focuses upon:– consumer satisfaction– outputs as measured against a quality standard

Managed Care

• Emanates from health care

• Promotes efficiency to assist health care industry shift from cost-based to capitated reimbursement

(SEA) Service Efforts and Accomplishments Reporting

• Standard introduced by the Governmental Accounting Standards Board (GASB)

• SEA is GASB’s term for performance measurement

SEA Reporting Model

• Built upon an expanded system model including:

• inputs

• outputs

• quality outputs, &

• outcomes BUT

• Excludes Process

SEA’s Lack Of Emphasis Upon Process

Absence of the Process component reflects SEA’s primary emphasis upon performance & performance cost considerations

SEA Reporting Elements

• Service Efforts

• Service Accomplishments

• Measures or Ratios Relating Service Efforts to Service Accomplishments

Service Efforts

Service Efforts are inputs utilized in a human service program, which are measured by the GASB in terms of

• Total Program Costs

• Total Full Time Equivalent Staff (FTE)

• Total Number of Employee Hours

Service Accomplishments

Outputs:Total Volume of Total Service Provided

Proportion of Total Service Volume Meeting Quality Standard

Outcomes:

Measures of results, accomplishments, impacts

Service Accomplishment Ratios

• Efficiency (output measures) cost per unit of service– cost per FTE– cost per service completion– service completions per FTE

• Effectiveness (outcome measures)– cost per outcome– outcome per FTE

Output Performance Measures

• Intermediate: – episode or contact unit of service– material unit of service

• Final: Service completions

Outcome Performance Measures

• Intermediate:– numeric counts,– standardized measures– level of functioning scales– client satisfaction

• Ultimate:– numeric counts– standardized measures– level of functioning scales