Performance Evaluation in the Healthcare

Heesuk Yun

Korea Development Institute

March 17, 2004

Need for Performance Evaluation in the Health Field

Expenditure for public health is rapidly increasing in Korea

Targets of public health actions is expanding beyond infectious diseases

Increasing emphasis on prevention, but efficacy  is hard to prove

– Pressure of cost containment and Importance of accountability of resources invested in health field

– The need for rigorous evaluation will increase

This paper attempts to

Summarize the essential elements and special characteristics of performance evaluation in the health field

Introduce the U.S. experience of devising indicators in the health field

Review standards for effective evaluation of health interventions

Extract  implications for establishing a performance evaluation in the health field in Korea

Objectives of Public Health Intervention

Multiplicity: objectives range from the most general such as reducing mortality to the very specific such as reading a health pamphlet.

Objectives and sub-objectives can be hypothesized corresponding to the steps or actions of a program.

A continuous series of events → divided into a hierarchy of sub-goals for evaluation purposes Each is  the result of successfully achieving the preceding goal, and a precondition to the next higher goal.

Cumulative chain of objectives

Reduction in morbidity or mortalityReduction in morbidity or mortality

Number or proportion of persons who are meeting prescribed and accepted standardsNumber or proportion of persons who are meeting prescribed and accepted standards

Number of persons who change behavior pattern in accordance with the new knowledge Number of persons who change behavior pattern in accordance with the new knowledge

Number or proportion of persons who change opinions or attitudes from the new knowledgeNumber or proportion of persons who change opinions or attitudes from the new knowledge

Number or proportion of persons who learn the facts contained thereinNumber or proportion of persons who learn the facts contained therein

Number or proportion of persons who glance at or read itNumber or proportion of persons who glance at or read it

Number or proportion of persons who see the materialNumber or proportion of persons who see the material

Number or proportion of persons who receive the materialNumber or proportion of persons who receive the material

Number of requests received for the material, or number distributedNumber of requests received for the material, or number distributed

Number of pieces of literature available for distributionNumber of pieces of literature available for distribution

Pretesting of literature by special readability formulasPretesting of literature by special readability formulas

Needs to prove the intervening assumptions through careful research

An assumption needed  whenever one moves from a higher-order objective to a lower one. Every lower-level objective must assume all of the assumptions made for all of the objectives above it.

Any program based on a false set of assumptions cannot be justified, even if sound evaluations are completed for each of the other objectives individually.

Special Characteristics of Performance Evaluation in the Health Field: Scientific rigor

Intervention logic: an explanation of what the public action is supposed to achieve and how it is supposed to achieve it

Causal assumptions are hidden in the logic

Involves scientific theory  

Identifying the hidden assumptions and investigating their uncertainties are more complicating

Smoking and Low Birth Weight

Older Than 17 Years

Older Than 17 Years

MedicalRisks

MedicalRisks

EconomicStress

EconomicStress

No SocialSupport

No SocialSupport

FamilyStress

FamilyStress

AddictiveBehaviorAddictiveBehavior

LowBirth Weight

LowBirth Weight

.75

.28

.32

.19

.89

.41.90

.78

.42

.55

.30

.35

.35

-.13

Addictive behaviors such as smoking and alcohol consumption have direct effect on low birth weight

Economic stress has direct effects on family stress and social support

Family stress and social support have direct effects on addictive behavior

The three stress variables have indirect effects on low birth weight

The standardized regression coefficients indicate the size of the change from various factors

Special Characteristics of Performance Evaluation in the Health Field: Contextual Complexity

Complicated contexts including historical, geographical settings, political, social and economic conditions, and influences of related or competing organizations

Need to consider the needs of the target population and the particular problems

Difficult to identify what effects are genuinely caused by a program and to separate these effects from other influences on the socio-economic problems

– Needs to understand these intertwined factors, to design a context-sensitive evaluation, to interpret findings accurately and to assess the generalizability of the findings.

Political Context:

Health Care interventions are often created to reduce health disparities across socio-economic groups

– Evaluations must consider political aspects among income groups, politicians, and interest groups

– Evaluations must be conducted in a political context where groups compete in their own interests.

Methods in Performance Evaluation for Health Care Intervention: Customizing Tools for Evaluation

Health care's  special features due to scientific hypotheses or the contextual particularity

– Need to understand the intervention specific components and process

Golden rule: there is no single evaluation methodology which is universally applicable for the entire health field. The choice of method determined by the particular evaluation problems

Prerequisites for Starting: Evaluation Infrastructure

(1) Construction of Representative Frameworks for Evaluating Public Healthcare Intervention

Standardized framework supports a practical approach based on steps and standards applicable in public health settings.

Provides a guide for designing and conducting specific evaluation projects

Can be used as a template to create or enhance program-specific evaluation guidelines that further operationalize the step sand standards in ways that are appropriate for each intervention

Guided by the steps and standards in the framework:

Program planning will also evolve

Integrated information systems will support a more systematic measurement

Lessons learned from evaluations can be used more effectively to guide changes in public health strategies.

General framework can encourage evaluation research to be integrated with routine program operations.

Prerequisites for Starting:

(2) Constructing Indicators for Major Healthcare Evaluation Types

System of criteria to measure the effects of an intervention

Where intervention logic is grounded on scientific knowledge and all the political, social, and economic interests, Indicators should function as the guiding rules

Should translate general concepts regarding the intervention, its context, and its expected effects into specific measures that can be interpreted, while providing a basis for collecting evidence

Developing indicators should be based on intervention logic

For each step of the model, qualitative and quantitative indicators  developed to suit the concept in question, linking assumption, the information available, and the planned usage of data.

By relating indicators to intervention logic,

Detect changes in performance faster than when relying on a single outcome as the only performance measure

Lines of responsibility and accountability are clarified as the measures are aligned with each step of the program strategy

Detect the consequences of intermediate effects on health outcomes of the program.

Prerequisites for Starting:

(3) Preparing Date Sources for Evaluation

When an indicator is proposed for use in performance monitoring, appropriate data must be available to support an indicator

Data should be collected from the specific population of interest, within the relevant time frame, using valid, reliable, and responsive measures

Often, performance evaluations have to rely on data collected for another purpose

– Evaluators must understand limitations on the applicability of the data.

Examining Indicators for Healthcare Evaluation: Experience of U.S.

Panel on Performance Measures and Data for Public Health Performance Partnership Grants

Recommend performance measures in ten areas: chronic disease; sexually transmitted disease (STD), human immunodeficiency virus (HIV) infection, and tuberculosis; mental health; immunization; substance abuse;  sexual assault, disabilities, and emergency medical services

Panel's Framework for Assessing Suggested Indicators

Health outcome measures are widely used but insufficient

Many measures of health outcomes are affected by various factors that are not under the health intervention's control → changes in outcomes cannot be attributed only to specific program effectiveness.

Many important public health objectives, such as lowering the incidence of cancer and HIV infection, cannot be achieved over short periods of time to derive an outcome measure.

Health outcome: change in the health of a defined population related to an intervention.

Risk status (intermediate outcome): change in the risk demonstrated or assumed to be associated with health status.

Process: what is done for the defined groups as part of the delivery of services, such as performing a test or procedure or offering an educational service.

Capacity: the ability to provide specific services, such as clinical screening and disease surveillance

Example: CHRONIC DISEALSE

Prevention of chronic disease is the primary goal of many health programs,

But chronic disease incidence and mortality data are not useful because

– The expected time period between  prevention activities and the effect exceeds the time that health departments are generally willing to wait to assess the intervention's effectiveness

– Potential chronic disease measures are focused on risk reduction and screening, and process measures.

Potential Risk Status Measure

Percentage of (a) persons aged 18-24 and (b) persons aged 25 and older currently smoking tobacco

Who eat five or more servings of fruits and vegetables per day

Who do not engage in physical activity or exercise

Who had their blood pressure checked within past 2 years

Examples of Process Measure

Nutrition Program Strategy: Percentage of schools with menus that meet dietary guidelines

Physical Activity Program Strategy: Percentage of worksites with worksite wellness programs

Smoking Program Strategy: Percentage of vendors who illegally sell smoking tobacco to minors

Screening Program Strategy: Percentage of persons with diabetes receiving diabetes health education

Examples of Capacity Measures

Resources: Number of full-time health department employees for chronic disease prevention

Proficiencies: Number of key surveillance systems and data sets that are establish and maintained

Planning: Percentage of population served by systematic community planning process, with leadership provided by the official health agency

Community Involvement: Proportion of health department programs that operate within the framework of a community coalition or have a community advisory group

Summary

Performance evaluation in the health field is an increasingly important area

Evaluation of public health action: use of rigorous scientific methods and contextual complexity requiring socio-economic and political awareness.

Setting up an evaluation system requires building infrastructure

1) Framework for evaluating public health intervention needs to be devised 

2) Indicators need to be collected and assessed for future use

3) Appropriate data sources for each indicator should be gathered and organized. and following up data collection should be made within a reasonable time.