Operations Research Mini Course

University of Washington Center for AIDS Research

Scientific Program on Health Services and Strategies Research

July 31, 2009

An Introduction to Operations An Introduction to Operations ResearchResearch

------- or -------------- or -------How can I make my health How can I make my health

program better?program better?

Mark Micek, MD, MPHMark Micek, MD, MPH

The IssueThe Issue

It is difficult to…It is difficult to………effectively deliver scientifically proven health effectively deliver scientifically proven health

interventions in the “real world”interventions in the “real world”……translate research into health program settingstranslate research into health program settings

Why?Why? Research generally occurs in controlled settingsResearch generally occurs in controlled settings

Homogenous sample, controlled settingHomogenous sample, controlled setting Health programs exist in a complex settingHealth programs exist in a complex setting

Heterogeneous clients, multiple settingsHeterogeneous clients, multiple settings Dependent on external context of care-delivery systemDependent on external context of care-delivery system

i.e. policy, resource availability ($ and personnel), i.e. policy, resource availability ($ and personnel), community perceptionscommunity perceptions

Example of difficulty translating Example of difficulty translating proven treatment into practiceproven treatment into practice

HAART reduces mortality among patients with HIVHAART reduces mortality among patients with HIV BUTBUT many eligible HIV-positive people don’t start HAART many eligible HIV-positive people don’t start HAART

Monthly flow through the HIV care system in Beira and Chimoio, Mozambique, Jun 04 - Sept 05

HIV+

Undergo CD4 testing (78%)

Enroll at HIV clinic (59%)

Eligible for HAART (48%)

Start HAART (46%)

0

100

200

300

400

500

600

700

Ave

rag

e p

atie

nts

per

mo

nth

Example of difficulty translating Example of difficulty translating proven treatment into practiceproven treatment into practice

Short-course Short-course ARV therapy can ARV therapy can decrease decrease mother-to-child mother-to-child HIV transmissionHIV transmission BUTBUT few HIV+ few HIV+

women receive women receive treatmenttreatment

Data from: Evaluation of United Nations-Supported Pilot Projects for Data from: Evaluation of United Nations-Supported Pilot Projects for the Prevention of Mother-to-Child Transmission of HIV: Overview the Prevention of Mother-to-Child Transmission of HIV: Overview of Findings. UNICEF, New York, 2003.of Findings. UNICEF, New York, 2003.

Women tested for HIV and receiving ARV prophylaxis in 11 pilot pMTCT sites, Jan00-Jun02

Attend ANC at pMTCT site

Counseled for HIV (71%)

Tested for HIV (70%)

HIV-positive (9%) Received ARVs

(49%)0

100000

200000

300000

400000

500000

600000

Nu

mb

er o

f p

reg

nan

t w

om

en

How can we improve the How can we improve the performance of our performance of our

programs in an evidence-programs in an evidence-based way?based way?

Use the principles of Operations Use the principles of Operations ResearchResearch

What is OR?What is OR?Generic definitionGeneric definition

““Use of systematic research techniques for program Use of systematic research techniques for program decision-making to achieve a specified outcome.” decision-making to achieve a specified outcome.”

[Population Council, 2006][Population Council, 2006]

Goal = find a best possible solution to improve Goal = find a best possible solution to improve performance of the organizationperformance of the organization Early examples in military, businessEarly examples in military, business Use data, statistics, mathematical modelingUse data, statistics, mathematical modeling

Goal (health care) = “to increase the efficiency, Goal (health care) = “to increase the efficiency, effectiveness, and quality of services delivered by effectiveness, and quality of services delivered by providers, and the availability, accessibility, and providers, and the availability, accessibility, and acceptability of services desired by users”acceptability of services desired by users”

[Population Council, 2002][Population Council, 2002]

3 Core Principles of OR3 Core Principles of OR

1.1. Study health programsStudy health programs

2.2. Actively try to make the program betterActively try to make the program better

3.3. Use results to improve the programUse results to improve the program

Corollary: requires collaboration between Corollary: requires collaboration between managers and researchersmanagers and researchers

Core principle of OR #1:Core principle of OR #1:Study health programsStudy health programs

Health program is key to health care deliveryHealth program is key to health care delivery OR usually focuses on existing programOR usually focuses on existing program

Research problem = Research problem = programprogram problem problem Research intervention = Research intervention = programprogram solution solution

Feasible within context of entire systemFeasible within context of entire system Assures problems and solutions are defined Assures problems and solutions are defined

by realities of the health program systemby realities of the health program system

Without involvement of a health program, it’s Without involvement of a health program, it’s not ORnot OR

Health program as systemHealth program as system

Outputs

Raw Materials:

•Finances

•Costs

•Staff

•Facilities

Activities:

•Trainings

•Supervision

•Logistics

•Reporting

•Record keeping

•Patient flow strategies

Products of program activities:

•# condoms distributed

•# people tested for HIV

•# enrolling for HIV care

Effect on knowledge or behavior:

•# youth using condoms

•# women using pMTCT services

•# starting ART

•% adhering to ART

Outcomes Impact

Change in health/well-being

•HIV/STD incidence

•# deaths

Health ProgramUnder managers’ control

Inputs Process

Effects of Health Program

Integrated Health Network Model

Information, Education and Communication at all levels

VCT Centers

1-2 / 100,000 I nhabitants

National TB Program

Maternity(Niverapine)

Internal MedicinePediatrics

STD Consults

Blood Bank

•Home based care•Community DOT•PLWHA groups•Nutrition support

Day Hospital1-2 doctors, PA’s,

Counselors•OI treatment•CD4 testing•HAART•Adherence support

Health Centers

TARV

Mental Health

MTCT

Health programs are complex systemsHealth programs are complex systems

Adhere to HAART

Test for HIV

Enroll in HIV clinic

CD4 testing

Start HAART (if eligible)

Schedule appt with clinician

Return for appt with clinician

Time and drop-off

Time and drop-off

Time and drop-off

Pt returns to get blood

drawn

Clinician orders CD4

Time and drop-off

Return for appointment with clinician

Time and drop-off

Blood drawn

Schedule appt for results

Time and drop-off

Enroll in HIV clinic

Workflow model: obtaining a CD4Workflow model: obtaining a CD4

HIV Positive Patient comes

to Clinic

Enrollment Process with RN

Patient scheduled to see MD

Registration Process with Receptionist

Patient returns to clinic for

appointment

Patient registers

Pt has access to

ARVs?

Intervention:· Counselling · CD4 test not

ordered

Intervention:· Counselling· CD4 testing

ordered

Lab open?

Yes

NoNo

Yes

Patient scheduled

for CD4 count

Blood for CD4 count drawn

Patient schedules

appointment to review results of

CD4 count

Patient returns to lab

for appointment

Patient returns to clinic for

appointment

Patient registers

Patient seen by Physician

Patient seen by Physician

Treatment plan is

developed.

CD4 count reviewed with patient, and significance explained.

Patient Leaves Clinic

Patient Leaves Clinic

Patient Leaves Clinic

Complex and interdependent

Core principle of OR #2:Core principle of OR #2:Actively try to make the program Actively try to make the program betterbetter

Better “understanding” of situation is not enoughBetter “understanding” of situation is not enough Better can mean…Better can mean…

Improve access to servicesImprove access to services Improve qualityImprove quality Limit costs (find cost-effective strategies)Limit costs (find cost-effective strategies) Improve healthImprove health

Without actively trying to improve the program, Without actively trying to improve the program, it’s not ORit’s not OR

Core principle of OR #3:Core principle of OR #3:Use results to improve programUse results to improve program

Using results can mean…Using results can mean… Implement new strategy on local / national scaleImplement new strategy on local / national scale Influence national / international policyInfluence national / international policy Dissemination of results, develop “best practices”Dissemination of results, develop “best practices”

OR successful only if results used to improve OR successful only if results used to improve the programthe program Published papers are NOT a valid indicator of OR Published papers are NOT a valid indicator of OR

successsuccess

Corollary: Corollary: OR requires collaboration between OR requires collaboration between

managers and researchersmanagers and researchers

Program managers/policy-makersProgram managers/policy-makers Should be involved in ALL ASPECTS of research process Should be involved in ALL ASPECTS of research process

Understand health care systemUnderstand health care system Help ensure problem is important, solution is feasibleHelp ensure problem is important, solution is feasible Help ensure results will be implementedHelp ensure results will be implemented

ResearchersResearchers Understand of research methodologyUnderstand of research methodology Responsible of recommending and implementing Responsible of recommending and implementing

appropriate research techniquesappropriate research techniques

Can be the same personCan be the same person

How is OR different from other How is OR different from other types of research?types of research?

All types of health research try to improve All types of health research try to improve healthhealth

All types of research can use similar All types of research can use similar methodologiesmethodologies Quantitative, qualitative, surveys, Quantitative, qualitative, surveys,

experiments, focus groups, simulationsexperiments, focus groups, simulations

Difference = focus and goalsDifference = focus and goals

Broad methodologies of ORBroad methodologies of OR

Modeling (classic)Modeling (classic) Develop mathematical model to mimic health care Develop mathematical model to mimic health care

systemsystem Manipulate to find the best possible “solution”Manipulate to find the best possible “solution”

Optimize efficiencyOptimize efficiency Maximize Y given constraints XMaximize Y given constraints X

Intervention-based (Population Council)Intervention-based (Population Council) Design/test best way to deliver servicesDesign/test best way to deliver services Similar to quality improvement (IHI/WHO)Similar to quality improvement (IHI/WHO)

Example of OR modeling in HIVExample of OR modeling in HIV

SIMCLIN computer model*SIMCLIN computer model* Consortium for Strategic HIV Operations Research (Clinton Consortium for Strategic HIV Operations Research (Clinton

AIDS Initiative)AIDS Initiative) GoalsGoals

Plan best allocation of current resourcesPlan best allocation of current resources Forecast future needsForecast future needs Predict impact of program changesPredict impact of program changes

Examples of use:Examples of use: Given X nurses and Y doctors, how many patients can be Given X nurses and Y doctors, how many patients can be

treated in 1 year with ARVs?treated in 1 year with ARVs? How many nurses and doctors are needed to treat 1,000 How many nurses and doctors are needed to treat 1,000

patients per year?patients per year? How will spacing CD4 schedule affect resource needs?How will spacing CD4 schedule affect resource needs?

* * Clinton Foundation HIV/AIDS Initiative, Consortium for Strategic HIV Operations Research (CSHOR): SIMCLIN- An HIV Service Delivery Resource Clinton Foundation HIV/AIDS Initiative, Consortium for Strategic HIV Operations Research (CSHOR): SIMCLIN- An HIV Service Delivery Resource

Planning Tool Technical Overview, available online at Planning Tool Technical Overview, available online at http://www.cshor.org/SIMCLINTechnicalOverview.dochttp://www.cshor.org/SIMCLINTechnicalOverview.doc

Inputs Inputs to to

modelmodel

Inputs Inputs to to

modelmodel

Outputs of modelOutputs of model

Other uses of modelingOther uses of modeling

Planning resource allocation given certain Planning resource allocation given certain constraintsconstraints Where to place HIV treatment facilities, given HIV Where to place HIV treatment facilities, given HIV

prevalence and infrastructure, human resource, and prevalence and infrastructure, human resource, and transportation constraintstransportation constraints

Where to locate laboratory facilities, given technology Where to locate laboratory facilities, given technology and transportation constraintsand transportation constraints

How to improve clinic efficiency, given human How to improve clinic efficiency, given human resource and infrastructural constraintsresource and infrastructural constraints

Intervention-based ORIntervention-based OR

Population CouncilPopulation Council IHI CollaborativeIHI Collaborative

LinearLinear CyclicalCyclical

1. Identify program problem1. Identify program problem 1. 1. PPlanlan

2. Generate program solution2. Generate program solution 2. 2. DDoo

3. Test program solution3. Test program solution 3. 3. SStudytudy

4. Use/disseminate results4. Use/disseminate results 4. 4. AActct

Act Plan

DoStudy

Real-Time Interactive Operations Real-Time Interactive Operations Research (IHI/WHO)Research (IHI/WHO)

From: An Approach to Rapid Scale-up: Using HIV/AIDS From: An Approach to Rapid Scale-up: Using HIV/AIDS Treatment and Care as an Example. WHO, 2004.Treatment and Care as an Example. WHO, 2004.

Steps in intervention-based ORSteps in intervention-based OR

1.1. Identify program problemIdentify program problem• Usually determined in ongoing programUsually determined in ongoing program• Routine data (M&E, surveillance) vs. program evaluationRoutine data (M&E, surveillance) vs. program evaluation• Under control of program managerUnder control of program manager

2.2. Generate program solutionGenerate program solution• Review workflow, talk to staff/clientsReview workflow, talk to staff/clients• Consider exploratory study if causes/solution unknownConsider exploratory study if causes/solution unknown• Must be feasibleMust be feasible

3.3. Test program solutionTest program solution• Level of intervention: facility vs. individualLevel of intervention: facility vs. individual• Data measurement: routine vs. added proceduresData measurement: routine vs. added procedures• Allocation: non-randomized (quasi-experimental) vs. randomized Allocation: non-randomized (quasi-experimental) vs. randomized

(experimental)(experimental)

4.4. Use/disseminate resultsUse/disseminate results• Continue/expand successful interventionsContinue/expand successful interventions• Influence national/international policyInfluence national/international policy• OR not typically “generalizable” but can be relevant for similar programs OR not typically “generalizable” but can be relevant for similar programs

(“best practices”)(“best practices”)

Common OR study designsCommon OR study designs(Experimental/quasi-experimental)(Experimental/quasi-experimental)

Pre-post control /non-equivalent control groupPre-post control /non-equivalent control group

Simple time-series (some control for time)Simple time-series (some control for time)

Step-wedge time-series (better control for time)Step-wedge time-series (better control for time)

Time

Exp group O O O X O O O

Time

Exp group 1 O X O O O

RA/non-RA Exp group 2 O O X O O

Exp group 3 O O O X O

Time

Exp group O1 X O2RA/non-RA

Control group O3 O4

Example 1: HIV testing in pMTCT Example 1: HIV testing in pMTCT program in Rwandaprogram in Rwanda

ProblemProblem: 2-3 day : 2-3 day delay in getting HIV delay in getting HIV test results test results 18% 18% did not return for did not return for resultsresults

Potential solutionPotential solution: : Same-day resultsSame-day results

Test of solutionTest of solution: Time : Time series, dropout ~0 series, dropout ~0 post-interventionpost-intervention

Source: An Approach to Rapid Scale-up: Using HIV/AIDS Treatment and Care as an Example. WHO. Geneva, 2004.Source: An Approach to Rapid Scale-up: Using HIV/AIDS Treatment and Care as an Example. WHO. Geneva, 2004.

Example 2Example 2: How to increase the : How to increase the number of patients who start HAART?number of patients who start HAART?

0

20

40

60

80

100

120

Q1 2004 Q2 2004 Q3 2004 Q4 2004 Q1 2005 Q2 2005 Q3 2005

Beira

Chimoio

National ARV Program

Identify steps required to start ARTIdentify steps required to start ART

HIV testing centers VCT

Home-based Care

Pregnant

Day Hospital Clinical evaluation (CD4)

Start HAART in

eligible patients

Adherence to ARV

Treatment Adherence to

Care

Youth

Community

TB patients

Ill/Hospitalized Hospital

Youth VCT

pMTCT

STEP 1 HIV Testing

STEP 2 Arrival to

Day Hospital

STEP 3 CD4

Testing

STEP 4 Start

HAART

Using programmatic data:Using programmatic data:Where are patients lost?Where are patients lost?

Monthly flow through the HIV care system in Beira and Chimoio, Mozambique, Jun 04 - Sept 05

HIV+

Undergo CD4 testing (78%)

Enroll at HIV clinic (59%)

Eligible for HAART (48%)

Start HAART (46%)

0

100

200

300

400

500

600

700

Ave

rag

e p

atie

nts

per

mo

nth

Step 1

Step 2 Step

3

Step 4

Using programmatic data:Using programmatic data:What are priorities to address?What are priorities to address?

Step 1: Tested for HIV Step 1: Tested for HIV1229 931

HIV+ HIV+440 23036% 25%

Step 2: Enroll in clinic Extra if fixed Step 2: Enroll in clinic Extra if fixed240 to 100% 150 to 100%54% 30 65% 15

Adults Adults230 14496% 96%

Step 3: Obtain CD4 Extra if fixed Step 3: Obtain CD4 Extra if fixed191 to 100% 102 to 100%83% 7 71% 12

Eligible Eligible86 55

45% 54%

Step 4: Start HAART Extra if fixed Step 4: Start HAART Extra if fixed36 to 100% 29 to 100%

42% 50 52% 27

Beira Chimoio

By City

Why do HAART-eligible patients Why do HAART-eligible patients not start ARVs (step 4)?not start ARVs (step 4)?

Poor follow-up also reported as reason for not starting HAART in other studiesPoor follow-up also reported as reason for not starting HAART in other studies Giordano TP et al, Factors Associated with the Use of Highly Active Antiretroviral Therapy in Patients Newly Entering Care in an Urban Giordano TP et al, Factors Associated with the Use of Highly Active Antiretroviral Therapy in Patients Newly Entering Care in an Urban

Clinic. JAIDS, 32:399-405.Clinic. JAIDS, 32:399-405.

Follow-up of HAART eligible patients (CD4<200) that do and do not start HAART

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

FU>0d FU>30d FU>60d FU>90d

Pro

po

rtio

n w

ith

fo

llow

-up

bey

on

d t

imep

oin

t

HAART

No HAART

Improving rates of starting ARVs Improving rates of starting ARVs in HAART-eligible patientsin HAART-eligible patients

Reasons for poor Reasons for poor follow-upfollow-up Pre-HAART procedure Pre-HAART procedure

too cumbersome too cumbersome Dissatisfaction with Dissatisfaction with

servicesservices Trouble paying Trouble paying

transportation coststransportation costs Poor understanding of Poor understanding of

clinic proceduresclinic procedures Stigma of going to HIV Stigma of going to HIV

clinicclinic DeathDeath

Potential solutionsPotential solutions Change workflow around Change workflow around

HAART-eligible patientsHAART-eligible patients Improve counselingImprove counseling Improve relationship Improve relationship

between patients and between patients and health care workershealth care workers

Decentralize ARV Decentralize ARV servicesservices

Number of HIV+ pregnant women enrolled at Number of HIV+ pregnant women enrolled at ART site <30 days after HIV testingART site <30 days after HIV testing

On-site ART vs. Off-site ART clinic: OR 7.2 (CI 5.9-8.8, p<0.001)

0

500

1000

1500

2000

2500

Beira Chimoio Nhamatanda Catandica

Off-site ART clinic On-site ART clinic

Total HIV +

Enrolled

34%

22% 76% 74%

30% 75%

ART-eligible starting ART (Total and ART-eligible starting ART (Total and <<90 90 days), Sofala and Manica, 2004-2007days), Sofala and Manica, 2004-2007

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Beira

Nham

atan

da

Caia

Ponta

Gea

Mar

rom

eu

Chimoio

Catan

dica

Man

ica

Espun

gabe

ra

Gondo

la

1o d

e M

aio

Chitob

e

ART TotalART <90days

Vertical VerticalIntegrated Integrated

Total stated on ART:

Vertical 50% vs. Integrated 65% (p<0.001)

ART <90 days:

Vertical 37% vs. Integrated 59% (p<0.001)

N=9,193

Other examples of OROther examples of OR

Program problem Potential OR studies

(Pre-post control, time series, step-wedge)

Poor adherence to HAART Study strategies to improve adherence (DOT, counseling, community-based treatment partners,

food/transport subsidies)

Low rates of HIV treatment among TB patients, inpatients

Study integration of HIV services into TB programs or hospital services

Underdiagnosis of TB among HIV+ patients Study strategies to increase TB screening/diagnosis (training/mentorship, standardized care algorithms,

screen at VCT)

Low rates of HIV treatment among pregnant women

Study strategies to improve retention into care (decentralize HIV services, improve counseling,

involvement of peer counselors)

Poor provider adherence to HIV care protocols Study strategies to improve adherence to protocols (training/mentorship, performance reviews,

involvement of lower-level HCWs)

Low level of prevention behaviors among HIV+ people

Study strategies to increase prevention (couples counseling/care, intensify prevention counseling in

testing and care centers)

Funding for ORFunding for OR

Research-directed fundingResearch-directed funding Government:Government:

USAID/PEPFAR TE, now TASCIIIBUSAID/PEPFAR TE, now TASCIIIB CDC (ASPH partnerships– i.e. UW-malaria)CDC (ASPH partnerships– i.e. UW-malaria) NIH?NIH?

Foundations:Foundations: Doris Duke Foundation Doris Duke Foundation

Operations Research for AIDS Care and Treatment in Africa Operations Research for AIDS Care and Treatment in Africa (ORACTA)(ORACTA)

20 2-year grants awarded, $100,000/year20 2-year grants awarded, $100,000/year African Health Initiative: $20 million over 5-7yrAfrican Health Initiative: $20 million over 5-7yr

Program-directed fundingProgram-directed funding Multilaterals: WHO (Global Fund), World Bank (TAP)Multilaterals: WHO (Global Fund), World Bank (TAP) Ministries of HeathMinistries of Heath

OR resourcesOR resources

Designing HIV/AIDS Intervention Studies: An Operations Research Designing HIV/AIDS Intervention Studies: An Operations Research Handbook. Andrew A. Fisher and James Foreit. The Population Council, Handbook. Andrew A. Fisher and James Foreit. The Population Council, New York, 2002. Available at: New York, 2002. Available at: http://www.popcouncil.org/pdfs/horizons/orhivaidshndbk.pdf

An Approach to Rapid Scale-up: Using HIV/AIDS Treatment and Care as an An Approach to Rapid Scale-up: Using HIV/AIDS Treatment and Care as an Example. World Health Organization, Geneva, 2004. Available at: Example. World Health Organization, Geneva, 2004. Available at: http://www.who.int/entity/hiv/pub/prev_care/en/rapidscale_up.pdf

The Breakthrough Series: IHI’s Collaborative Model for Achieving The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Improvement. Institute for Health Care Improvement. Breakthrough Improvement. Institute for Health Care Improvement. Cambridge, MA, 2003. Available at Cambridge, MA, 2003. Available at http://www.ihi.org/NR/rdonlyres/BCA88D8F-35EE-4251-BB93-E2252619A06D/0/BreakthroughSeriesWhitePaper2003.pdf

Population Council / Horizons program on HIV/AIDS OR: Population Council / Horizons program on HIV/AIDS OR: http://www.popcouncil.org/horizons/