Using indicators to measure country pharmaceutical
situations
Dr. Edelisa D. CarandangMedicines Policy & Supply Management
Department of Technical Cooperation for Essential Drugs and Traditional Medicines (TCM)
World Health Organization, Geneva
Sept 2006
Using indicators to measure country pharmaceutical situations
Concepts on pharmaceutical monitoring and assessment
The WHO process on assessing and monitoring pharmaceutical situation
Snapshots of current global pharmaceutical situations using Level I and Level II indicator survey- the fact book
New approaches in access indicators
Pharmaceutical monitoring and assessment
Monitoring Review of the progress toward completing program
activities and achieving objectives Allows for corrective action during program
implementation Focus on inputs and outputs -integral part of
management. Common methods in fully developed monitoring systems
• Supervisory visits
• Routine reporting of selected data
• Sentinel sites
• Special studies
Pharmaceutical monitoring and assessment
Assessment Discussed along with monitoring as part of overall
evaluation Analysis of progress and meeting objectives Types of evaluations
• Needs assessment (situation analysis, • Formative evaluation (midterm review)• Summative evaluation (final evaluation)
Field surveys using standard pharmaceutical assessment indicators & ongoing monitoring system, document review
Strategies for monitoring and assessment developed in parallel for comprehensive unified strategy
Principles in designing monitoring and evaluation system
Focus on key monitoring questions and indicators
Minimum data collection Develop practical procedures Consider comparisons between facilities
and performance over time. Plan how information will be available for
timely feedback and follow-up action Phase the design and implementation Design based on the current system
Who can use the results from assessment and monitoring?
Countries - focus action, prioritize, measure achievement
National policy-makers• synchronise policies • data and information to donors and other governmental agencies
International agencies• to assess the structure and capability of countries, assess the
progress, accomplishment and impact of aid Professional groups, NGOs and academia
• to focus advocacy activities and information campaigns
Health facilities to be aware of institutional problems & improve situations
Pharmaceutical indicators
Variables that measure situations and change (numerical, binomial)
Useful tools to track the performance of particular aspects or activities of the pharmaceutical system
Linked to an important input, process, or outcome Well-established indicators can be adapted/ modified
to reflect the realities Field test
Summarizing indicator measures Percentage: yes or no over total Measures of central tendency
• Mean: average value, sensitive to outliers, weighed toward skewed value, best summary of normally distributed values
• Median: middle value, resistant to outliers, good summary of any distribution
• Equivalent if data are normally distributed
Measure of variation• 25th and 75th percentiles:
boundaries of middle half of values, good summary of the overall spread of values, better summary of skewed data
Indicator measure: Ideal/logical values
Ideal value • 100%-adequate labelling, meds dispensed,
adherence to STG, availability of medicines, generic, adequacy of storage
• 0 days- of stock out Set by countries
• % affordability Logical value (% use of antibiotics,
%injection, ave drug)• complex
• can be calculated empirically
Antibiotics
51.7 50.0
36.746.3 43.3
60.0
0
20
40
60
80
Low Middle
Country Income Level
% of
patie
nts
25%ile Median 75%ile
Indicator measure: group norm
% patients receiving an antibiotic - distribution of results
0%
20%
40%
60%
80%
100%
facility
medianvalue
•Easy for region/facilities to relate to peers•Norms may be wrong
Why is it important to use indicators:(1)Setting target
% availability of key drugs in public sector
46%
78%73% 72%
25%
55%
75%
15%
0%
20%
40%
60%
80%
100%
Rural 1 Rural 2 Rural 3 Kampala
Health Facility
Warehouse
Ministry Target =
90%
Why is it important to use indicators:(2) For comparison
% Availability at public and private sector (2002)
0%
20%
40%
60%
80%
100%
Tanzania Mali Ghana
Public facility pharmacy
Private pharmacy
Why is it important to use indicators: (3) Seeing trends overtime
Comparing 1995-2002 key indicators shows progress in some areas but that enhanced efforts needed in others
0%
20%
40%
60%
80%
100%
Availability of keydrugs
% of presc. drug inEDL
% presc. withinjection
Availability of keydrugs
% of presc. drug inEDL
% presc. withantibiotics
% patient withadequate
knowledge
% presc. withinjection
1995 2002
Bulgaria Philippines
Background on WHO work on indicators and pharmaceutical assessment &monitoring
• Indicator and monitoring tools• Indicator for Monitoring National Drug Policy (94 & 99 Ed• WHO Operational package for Monitoring & Assessing
Country Pharmaceutical Situation
• Publications• 1988 World Drug Situation• 2003 World Medicines Situation• 2006 Using indicators to measure country pharmaceutical
situation: Fact book on WHO Level I and Level II monitoring indicators
• WHO medicines data base
Develop Develop implementation implementation plans and identify plans and identify strategies & strategies & interventions interventions based on based on data/information data/information on: on: availability, availability, affordability, affordability, pricing, drug use pricing, drug use and regulatory and regulatory profile, TRIPS, drug profile, TRIPS, drug management management
situation.situation.
Support implementation of activities and advise in the execution of work plans
Indicator-based tools to evaluate structures, Indicator-based tools to evaluate structures, processes, outcomes of in countriesprocesses, outcomes of in countries
WHO Evidence-Based Planning and InterventionsGuiding Country Works in Medicines
WHO Evidence-Based Planning and InterventionsGuiding Country Works in Medicines
Why WHO is monitoring and assess pharmaceutical situation using indicators?
Objectives prioritized pharmaceutical policy objectives
based on evidence determine effects, impact of policy
implementation over time establish situation evidence for advocacy
Indicators for evidenced based policy Provides evidence for planning, prioritising
and identifying interventions Systematic data gathering Allows comparisons & set target Measuring trends
1. Assess and Monitor
2. Plan3. Implement
Access & rational useof quality medicines
WHO hierarchical approach to monitoring and assessing pharmaceutical situations
Level I Core structure
& process indicators
Level II Core outcome/impact indicators
& household survey
Level III Indicator tools for specific components of the pharmaceutical sector
●Pricing ●Traditional medicine●Paediatrics ●Assessing regulatory capacity●TRIPS ● Supply & Procurement
Systematic survey
Questionnaire (Health Officials)
Level I•Questionnaire/rapid assessment/checklist•Arrays achievement & weaknessess, illustrate sectoral approaches
Level II•Comprehensive monitoring of pharmaceutical strategy outcome and impact•Measures attainment of objectives
Level II •More detailed îndicators for monitoring and evaluating specifc areas/components
Level I indicators: structure and process indicators
Regular survey questionnaire
Advantages• Cheap way to get information across countries
• Can be done repeatedly/regular period
• Automated questionnaire and data encoding processing
Limitations• Validation of data
• Accuracy of responses
• Missing data
Level II indicators: WHO Operational Package for Monitoring and Assessing county Pharmaceutical Situations
Procedure and guidelines Administrative preparation:
• Coordinating with WHO, ministry/department of health, public health facilities, private drug outlets, warehouses
• Making logistic arrangements and budget allocations Technical requirements:
• Selecting geographic areas and facilities • Identifying country-specific items of the survey forms,
e.g. key basket of medicines, treatment guidelines, etc.• Training data collectors to carry out the survey and use
the survey and summary forms• Analyzing and computing the data• Preparing a report and using result
Indicators in the Level II survey
15 survey forms-public health facilities, public pharmacy/dispensary, private pharmacy, warehouses
indicators• on availability, stock out, record keeping and expiry of
key drugs
• conservation conditions and handling of medicines
• affordability (child and adult moderate pneumonia and option for other disease condition
• drug prescribing, dispensing,patient knowledge
Sampling for systematic survey
Follow specific procedures • to minimize selection bias
• study population is representative of the reference population
A balance between what is desirable and what is feasible- smallest one with a degree of precision
Training data collector for the survey
Important for consistency in data gathering process, validity & accuracy of data
Who can be trained?• Physicians, nurses, pharmacists or paramedical staff
• Health ministry/department staff and temporary employees (health related background and experience)
• data collectors from different parts of the country (language differences) • Fiji -used students from B.Pharm Y3/Y4,Improve FSm/MoH
relations
• Solomon Islands-pharmacy officers and pharmacists who are not from their own province
Level II Indicator (advantages & limitations)
Advantages Practical survey based on small samples Covers impact and outcome in one comprehensive survey Efficient in terms of resource requirement owing to
standardized process of training, data gathering and time allocation
Limitations Requires more resources for training and conducting
systematic survey Limitation in terms number of countries that can be
covered
Level III Indicators
Systematic survey and monitoring• Drug price survey and monitoring
Rapid assessment• Global survey on Paediatrics medicines
• Questionnaire on public sector medicines procurement and supply management systems in countries
• Assessment of regulatory capacity
Fact book on Level I & Level II indicator result
Product of years’ of work on developing and improving data-gathering tools, systematic collection of information
Overview of pharmaceutical sector components current status & impact
Countries grouped low, middle or high-income. Data and information in tables and graphs
• current situation – 2003 Level I survey
• progress - comparing Level I 1999 and 2003 surveys
• impact and outcome measures - Level I & Level II. Country data in CD ROM
Percentage of Countries with Official National Medicines
Policy (NMP) in 1999 and 2003
0
10
20
30
40
50
60
70
80
High Medium Low
1999
2003
Key pharmaceutical sector legislation
Country Income Level
Low Middle High
Policy area covered:Number
%of Countries
Number %
of Countries
Number %
of Countries
Establishment of regulatory authority 53 98.2 52 89.7 17 94.4
Manufacturing of medicines 48 90.6 50 86.2 16 88.9
Distribution of medicines 54 98.2 50 89.3 17 94.4
Promotion & advertising of medicines 47 88.7 50 87.7 16 88.9
Importation of medicines 53 98.2 53 89.8 17 94.4
Empowerment to enter premises and collect samples and documentation
47 90.4 49 89.1 17 100
Generic prescribing and substitution regulations in 1999 and 2003
0102030405060708090
100
Low Middle High
1999 Public
2003 Public
1999 Private
2003 Private
Generic Prescribing
Samples Collected for Regulatory Purposes, Tested, and
Failing Tests in Previous Year
Country Income Level
Low Middle High
Median [25th,75th] Median [25th,75th] Median [25th,75th]
# of samples tested 763[172,2202 923[433,3382] 409[53, 744]
(%) of samples that failed test
(11%) [3, 108] (3.9%) [4, 106] (<1%) [1, 17]
Storage and handling conditionsStorage and handling conditions in public pharmacies and warehouses
Public Health Facilities Warehouses
76.580.7
75.0
83.287.5 89.290.5
94.4
81.8
70.1
78.1
87.8
60
70
80
90
100
Low (n=11)
Middle(n=10)
Low (n=11)
Middle(n=10)
Country Income Level
% o
f m
axim
um
sto
rag
e
an
d h
an
dlin
g s
co
re
25%ile Median 75%ile
Countries with insurance that covers medicines
0
10
20
30
40
50
60
70
80
90
Low Med. High
1999
2003
0102030405060708090
100
Low Med. High
1999
2003
Public insurance Private insurance
Medicines Pricing Policies
Country Income Level
Low Middle High
No. % No. % No. %
Pricing policy public sector
29 53.7 32 60.4 8 66.7
Pricing policy private sector
22 46.8 30 65.2 3 33.3
Pricing policy NGO 7 17.5 15 27.8 2 28.6
Proportion of prescribed medicines dispensed
% medicines dispensed
0
20
40
60
80
100
Low income countries
Per
cen
tag
e d
isp
ense
d
Cambodia
Cameroon
Ethiopia
Kenya
Lao PDR
Mali
Nepal
Rwanda
Senegal
Tanzania
Uganda
0
20
40
60
80
100
Middle incomecountries
Per
cen
tag
e d
isp
ense
d
Brazil
Bulgaria
China
Colombia
Guatemala
Indonesia
Iran
Malaysia
Oman
Philippines
Affordability of pneumonia treatment in children
Children
Private Pharmacies Public Facilities
1.91.40.2
0.9
3.5
5.0
0
2
4
6
8
Low LowCountry Income Level
No
. of
da
ys
' wa
ge
s*
to p
ay
fo
r tr
ea
tme
nt
25%ile Median 75%ile
Rational Use of Medicines : Specific Conditions
Country Income Level
Low (n=12) Middle (n=9)
Median % [25th, 75th]
Median % [25th, 75th]
Children with diarrhea receiving ORS
80.0 [71.9, 87.5] 95.0 [90, 100] (Min. Max)
Children with diarrhea receiving antidiarhea
0 [0,40] (Min. Max) 5 [0,5] (Min. Max)
Children with pneumonia receiving first line antibiotic
90.0 [80, 100] 85.0 [70, 100] (Min. Max)
Children with pneumonia receiving +1 antibiotic
0 [0,25] (Min. Max) 0 [0,0] (Min. Max)
ARI patients receiving antibiotic treatment
90.0 [70, 98.8] 50 [20, 100] (Min. Max)
Measuring access to essential medicines
Level of Access to Essential Medicines Within One Hour Walking Distance*
Very low access (<50%)
n=18
Low to medium (50%-80%)n=46
Medium to high (81%-95%)n=20
Very high access (>95%)
n=20
WHO Region:No. %of Countries
No. %of Countries
No. %of Countries
No. %of Countries
Africa 10 55.6 19 41.3 4 20.0 0 0.0
Americas 2 11.1 10 21.7 1 5.0 7 35.0
East. Mediterranean 1 5.6 2 4.3 4 20.0 1 5.0
European 3 16.7 6 13.0 2 10.0 9 45.0
South-East Asia 0 0.0 4 8.7 3 15.0 0 0.0
Western Pacific 2 11.1 5 10.9 6 30.0 3 15.0
*Estimate
Measuring access to essential medicines ( Household Survey)
Level I and Level II surveys do not measure access from the patient/consumer perspective.
Only household surveys can provide population-based information about how pharmaceutical policies affect the well-being of individuals.
Household surveys were tested on a pilot basis
With Level II surveys to maximize resources
Measuring access (Composite indicators)
Measure policy structures supporting access preconditions)
Steps used in the current processOperational• Obtain complete data for candidate variables• Validate data accuracyStatistical• Formal statistical analysis to obtain appropriate
factor weights• Validate relationship between Level I structure and
process and Level II outcomes indicators
4. Reliable
systems
5.Affordable
prices
3. Sustainable
financing
2. ACCESS TO
ESSENTIAL MEDICINES
1. Rational
selection
National drug policy process
Formulation
Identify problems Define objectives Develop strategies
Monitoring & evaluation
Develop system Identify tools Use results
Implementation
Develop and execute action plan based on available resources Prioritize and implement
strategies
Medicine Supply Management
Management Support• Organization• Financing• Info. Mangnt.• HR
Selection
Distribution
Procurement/Use
Monitoring
MonitoringMonitoring
Monitoring
Improved drug supply management: Guide selection, procurement, distribution, use
The way forward on country monitoring
Evidence through systematic but feasible data collection process is necessary in policy making and activity implementation
Should demonstrate that in the long run regular monitoring is not difficult and can be done in a cost efficient manner
Portion of country support budget and project grants should be allotted to monitoring and evaluation using indicators
Information and data sharing is important
THANK YOU