Upload
austin-blevins
View
221
Download
1
Tags:
Embed Size (px)
Citation preview
WHO VIP Webinar 20111
Evaluation Considerations: Measures & Methods
Shrikant I. Bangdiwala, PhD
Professor of Research in BiostatisticsInjury Prevention Research CenterUniversity of North Carolina at Chapel Hill, USA
presented at
WHO VIP Webinar 20112
Content
Purpose of evaluation Cycle of program planning & evaluation Indicators Study designs Statistical modeling Challenges
Safety 2010 London3
What are we ‘evaluating’?
Actions, programs, activities Conducted in a community setting, over a
period of time Aimed at reducing deaths, injuries, and/or
events and behaviors that cause injuries
WHO VIP Webinar 20113
WHO VIP Webinar 20114
Example: Suwon, South Korea
area of ‘safety promotion’
http://www.phs.ki.se/csp/safecom/suwon2.htm
WHO VIP Webinar 20115
Why do we ‘evaluate’?
To know ourselves what works and if we are doing some goodIn performing some activityIn the communityIn the country
To convince funders and supporters that their investment is worthwhile
To convince the community about the benefits of the multiple activities and actions carried out
WHO VIP Webinar 20116 NSC Chicago 2010 6
Main purposes of evaluation
Evaluation helps determine:How well a program/policy works relative to its
goals & objectivesWhy a program/policy did or didn’t work, relative
to planned processHow to restructure a program/policy to make it
work, or work betterWhether to change funding for a program
WHO VIP Webinar 20117
Methodological complications
Multiplicities Multiple components of a programMultiple populations at riskMultiple study designsMultiple types of effects/impacts/outcomes &
severitiesMultiple audiences/objectives of ‘evaluation’Multiple methods for conducting evaluation
WHO VIP Webinar 20118 8
When should evaluation be considered? Evaluation needs to begin in, and be part of,
the planning process… Otherwise, “if you do not know where you are
going, it does not matter which way you go, and you will never know if you got there or not!”
Lewis Carroll (1872)
Alice in Wonderland
Adapted from M. Garrettson
WHO VIP Webinar 20119 NSC Chicago 2010 9
ProgramPlanningPhase
Formative EvaluationHow can the program activities be improved before
implementation?
Process EvaluationHow is/was the program (being) implemented?
PostProgramPhase
Impact / OutcomeDid the program succeed in achieving the intended
impact or outcome?
Program Implementation
Phase
Types of evaluation depending on program phase
WHO VIP Webinar 201110
Cycle of program planning and evaluation
Adapted from C Runyan
WHO VIP Webinar 201111 11
Identify population & problem Surveillance data Other needs assessment strategies
key informant interviewsfocus groupssurveys evaluations of past programs literatureconsultation with peersother info…
WHO VIP Webinar 201112 NSC Chicago 2010 12
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
WHO VIP Webinar 201113 13
Define target audience
To whom is the program directed?Whose injuries need to be reduced?Who is the target of the program?
• at risk persons• care givers (e.g. parents)• general public• media• decision makers
WHO VIP Webinar 201114 14
Understand target audience
What are their characteristics?Special needs (e.g. literacy)Interests, concerns, prioritiesAttitudes & beliefs re: problem & solutions to
problem Cultural issues
WHO VIP Webinar 201115 15
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
WHO VIP Webinar 201116 16
Identify resources
Community partnersinterest in topicworking on similar projects
On-going activities Sources of financial support Interests in community
WHO VIP Webinar 201117 17
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
WHO VIP Webinar 201118 18
Set goals & objectives
Goalbroad statement of what program is trying to
accomplish
ObjectivesSpecificMeasurableTime-framed
WHO VIP Webinar 201119 19
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
WHO VIP Webinar 201120 20
Choose Strategies
Identify existing strategies/programsLiterature: evidence based? promising practice?WHO manualsSuccesses from other communities-regions-
countries Develop new strategies:
Logic model (how would it work)Haddon matrix
WHO VIP Webinar 201121 21
Haddon Matrix
Person Vehicle/vector
Physical Environ.
Social Environ.
Pre-event
Event
Post-event
Haddon 1970 Am J Public Health
WHO VIP Webinar 201122 22
Pre-Event Other??
Decision CriteriaDecision Criteria
PhasesPhases
3-dimensional Haddon Matrix
P
erso
n
Veh
icle
/V
ecto
r)
P
hysi
cal
Env
iron.
Soc
ial
Env
iron.
FactorsFactors
Event
Post-event
FeasibilityPreferences
StigmatizationEquity
FreedomCost
Effectiveness
Runyan 1998 Injury Prevention
WHO VIP Webinar 201123 NSC Chicago 2010 23
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
WHO VIP Webinar 201124 24
Formative Evaluation
What is the best way to influence the target population?
Will the activities reach the people intended, be understood and accepted by target population?
How can activities be improved?
Improves (pilot-tests) program activities before full-scale implementation
May increase likelihood program or policy will succeed
May help stretch resources
Why it’s usefulQuestions it answers
* Modified from Thompson & McClintock, 2000
WHO VIP Webinar 201125 NSC Chicago 2010 25
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
WHO VIP Webinar 201126 26
Implementation
As planned, with attention to detail
Documented clearly so others can replicate if appropriate
WHO VIP Webinar 201127 NSC Chicago 2010 27
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
WHO VIP Webinar 201128 28
Process evaluation
Purpose is to address:What was done?How was it implemented?How well was it implemented?Was it implemented as planned?
WHO VIP Webinar 201129 29
Process evaluation – examples of questions
• Who carried out intervention?• Was this the appropriate person/group?• Who supported and opposed intervention?• What methods/activities were used?
WHO VIP Webinar 201130 30
Process evaluation - why is it useful?
• Allows replication of programs that work.• Helps understand why programs fail.
* Modified from Thompson & McClintock, 2000
WHO VIP Webinar 201131 31
The intervention cannot be a black box…
OutcomeIdea ?
It must be clearly understood
WHO VIP Webinar 201132 NSC Chicago 2010 32
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
WHO VIP Webinar 201133 33
Impact evaluation
Purpose is to address changes in: knowledge attitudes beliefs/ values skills behaviors / practices
WHO VIP Webinar 201134 34
Using impact measures for
Establishing effectivenessSuppose we have a public safety campaign as our
strategyNeed to show Campaign Behavior Outcome
If we already have demonstrated that Behavior Outcome
We simply need to show Campaign Behavior
WHO VIP Webinar 201135 35
Outcome evaluation
Purpose is to address changes in:injury events (e.g. frequency, type, pattern) morbidity (e.g. frequency, severity, type)mortality (e.g. frequency, time to death)cost (e.g. direct and indirect)
WHO VIP Webinar 201136 NSC Chicago 2010 36
OutcomesImpacts Intervention
Physician counseling
parents
Enforcement of helmet law
Media campaign
Parental attitudes
toward child helmet use
Purchase of helmets
Use of helmets by children
Head injury in bike crashes
Deaths from head injury in crashes
Example: Bike helmets
WHO VIP Webinar 201137 37
Evaluation examples of questions for local policy of smoke alarms
Did the local policy of smoke alarms in apartments… Get passed Where people aware of it? Did people have access to smoke alarms? Did people get them installed properly? Do people keep them maintained? Lead to a reduction in the number or rates of:
events (e.g. apartment fires) injuries deaths costs (e.g. burn center costs, family burden, property loss)
WHO VIP Webinar 2011383838
Evaluation – selection of measures‘Quantitative Indicators’
Process Impact Outcome
Health related Financial
WHO VIP Webinar 201139
Choice of measure or indicator
We need to choose appropriate impact and outcome measures ‘Soft’ (more difficult to measure) outcomes –
Perceptions constructs: fear, insecurity, wellbeing, quality of life Knowledge, attitudes and behaviors constructs
Hard outcomes – Deaths, hospitalizations, disabilities due to injuries and violence Societal impacts – local development indicators
Economics outcomes – Direct $/€/£/¥, indirect DALYs, QALYs, opportunities lost, burdens
WHO VIP Webinar 201140
Evidence of effectiveness
Obtain qualitative ‘evidence’ to complement the quantitative ‘evidence’Ex. Are “multisectorial collaborations and
partnerships” friendly and functioning well?Ex. Is “community participation” optimal?
Incorporate process indicators Incorporate narratives & testimonials
WHO VIP Webinar 201141 NSC Chicago 2010 41
Test & refineimplementation
Test, Refine,Implement
Choose strategies
Set goals/objectives
Disseminate
Identifyproblem
& population
Define target audience
Evaluation: •Formative
Implement
Identify resources
Evaluation:•Process•Impact•Outcome
WHO VIP Webinar 201142 42
Dissemination
Dissemination not done wellNot attemptedNot based on research about how to disseminate
information to intended audience Dissemination done well
Defining audienceHow to access audienceHow best to communicate change message to themPresentation of clear, straightforward messages
WHO VIP Webinar 2011434343
Evaluation measures
Lead to evidence of effectiveness But only if the research and study
methodologies, and the statistical analyses methodologies, are appropriate to convince the funders and supporters, the skeptics, the stakeholders, the communityand understandable
WHO VIP Webinar 201144
Research methodology approach:Evidence of effectiveness
Obtain quantitative ‘evidence’ that favors the hypothesis that the intervention is effective as opposed to the (null) hypothesis that the intervention is not effective.
How? Experimental study designs - randomized clinical trials,
grouped randomized experiments, community-randomized studies
Quasi-experimental study designs - non-randomized comparative studies, before-after studies
Observational studies - cohort studies, case-control studies and comparative cross-sectional studies
WHO VIP Webinar 2011454545
Randomized controlled trial (RCT) / Experiment
‘strongest’ evidence
Randomize
InterventionGroup
Control Group
O X O
O OX’
WHO VIP Webinar 2011464646
Quasi-experimental designs
‘qualified’ evidence
Intervention Group
Comparison Group
O X O
O O
WHO VIP Webinar 2011474747
One group pre/post
‘weak’ evidence
InterventionGroup
O X O
WHO VIP Webinar 2011484848
One group – multiple pre / multiple post
better ‘weak’ evidenceIntervention
GroupO O O X O O O O O
WHO VIP Webinar 2011494949
One group, post only
‘basically ignorable’ evidence
Intervention Group
X O
Safety 2010 London505050
Observational designs- cohort study
evidence?Self-chosen Intervention Group
Self-chosen Non-intervention Group
X X X
O O O
WHO VIP Webinar 201150
Safety 2010 London515151
Observational designs- case-control study
evidence? Cases
Controls
X
O
X
O
WHO VIP Webinar 201151
Safety 2010 London525252
Observational designs- cross-sectional study
evidence?
InjuredX X O XO
O X OO O
Non-injured
WHO VIP Webinar 201152
WHO VIP Webinar 201153
Statistical analysis methodologies
Choice - often guided by what has been done previously, or what is feasible to do, or easy to explain
Choice should be tailored to the audience & their ability to understand results; but also on the ability of the presenter to explain the methodologies
WHO VIP Webinar 201154
Statistical analysis Determined by research question(s) Guided by study design – experimental or observational
Group randomized controlled experiment Non-randomized comparison study Single site pre/post; surveillance study Retrospective or cross-sectional
Guided by whether outcome is studied at a single time point or multiple time points Time series analyses
Guided by audience Visual and descriptive appreciation
WHO VIP Webinar 201155
Visual and descriptive analysis – longitudinal time series
Example:
Espitia et al (2008) Salud Pública Mexico
Safety 2010 London56
Visual and descriptive analysis – comparisons over time
Example:
www.gapminder.org
WHO VIP Webinar 201156
WHO VIP Webinar 201157
Statistical analysis - challenge
But what we as a field have not done as well as other fields, is to draw strength from numbers develop collective evidence
Combine results from multiple studiesSystematic reviews (of observational studies)Meta analysis (of experimental & observational studies)Meta regression (of heterogeneous studies)Mixed treatment meta regression (for indirect
comparisons)
WHO VIP Webinar 201158
Systematic reviews
A protocol driven comprehensive review and synthesis of data focusing on a topic or on related key questions formulate specific key questions developing a protocol refining the questions of interest conducting a literature search for evidence selecting studies that meet the inclusion criteria appraising the studies critically synthesizing and interpreting the results
WHO VIP Webinar 201159
Example – Systematic review
Shults et al (2001) Amer J Prev Med
WHO VIP Webinar 201160
Systematic reviews
Of particular value in bringing together a number of separately conducted studies, sometimes with conflicting findings, and synthesizing their results.
To this end, systematic reviews may or may not include a statistical synthesis called meta-analysis, depending on whether the studies are similar enough so that combining their results is meaningful Green (2005) Singapore Medical Journal
Zaza et al (2001) Amer. J Preventive Medicine – motor vehicle
WHO VIP Webinar 201161
Meta analysis
A method of combining the results of studies quantitatively to obtain a summary estimate of the effect of an interventionOften restricted to randomized controlled trialsRecently, the Cochrane Collaboration is
‘branching out’ to include both experimental and observational studies in meta analyses
WHO VIP Webinar 201162
Meta analysis
e.g. Liu et al (2008) Cochrane Collaboration
WHO VIP Webinar 201163
Meta analysis
The combining of results should take into account: the ‘quality’ of the studies
• Assessed by the reciprocal of the variance the ‘heterogeneity’ among the studies
• Assessed by the variance between studies
WHO VIP Webinar 201164
Meta analysis – estimation of effect
The estimate is a weighted average, where the weight of a study is the reciprocal of its variance
In order to calculate the variance of a study, one can use either a ‘fixed’ effects model or a ‘mixed’/’random’ effects model Fixed effects model:
utilizes no information from other studies Random effects model:
considers variance among and within studies
*2
2
)var(
)var(
iYi
iii
VY
eY
Borenstein et al (2009) Introduction to Meta Analysis
iYii VeY )var()var(
WHO VIP Webinar 201165
Meta analysis & meta regression
Dealing with ‘heterogeneity’ among the studies - 2 Decompose the total variance into among and within
components using mixed effects models for getting a more precise estimate of the intervention effect
If there is still residual heterogeneityExpand the mixed effects model to include study-level
covariates that may explain some of the residual variability among studies meta regression
WHO VIP Webinar 201166
Meta regression
e.g.
iiiii eXXY 2211
Overall mean
X1 study variable – EU/USA
X2 study variable – population type
study random effect
random error
WHO VIP Webinar 201167
Meta analysis
Standard meta-analytical methods are typically restricted to comparisons of 2 interventions using direct, head-to-head evidence alone.
So, for example, if we are interested in the Intervention A vs Intervention B comparison, we would include only studies that compare Intervention A versus Intervention B directly.
Many times we have multiple types of interventions for the same type of problem, and we hardly have head-to-head comparisons
We may also have multiple component interventions
WHO VIP Webinar 201168
Mixed treatment meta analysis
Let the outcome variable be a binary response 1 = positive response 0 = negative response
We can calculate the binomial counts out of a total number at risk on the kth intervention in the jth study
We can then calculate
the estimated probability of the outcome (risk of response) for the kth intervention in the jth study
kj
kjkj n
rp
:
::
kjr :kjn :
Welton et al 2009 Amer J Epid
WHO VIP Webinar 201169
Mixed treatment meta analysis
Let each study have a reference ‘‘standard’’ intervention arm, sj, with study-specific ‘‘standard’’ log odds of outcome, j .
The log odds ratio, j:k, of outcome for intervention k, relative to standard sj, is assumed to come from a random effects model with mean log odds ratio , and between-study standard deviation
where dk is the mean log odds ratio of outcome for intervention k relative to control (so that d1 = 0).
)(jsk dd
Welton et al 2009 Amer J Epid
WHO VIP Webinar 201170
Mixed treatment meta analysis
This leads to the following logistic regression model:
where ]),[(~: jskkj ddN
k
s
p
p
kjj
jj
kj
kj
int
int
1ln
::
:
Welton et al 2009 Amer J Epid
WHO VIP Webinar 201171
Mixed treatment meta analysis- multiple-methods interventions
If we have multiple methods in the ith intervention
Plus we have multiple times when the outcome is assessed
itiititiit eXMMtY 423121
,...,, 321 iii MMM
Study effect
Time effect
Components 1 & 2 effects
Study covariable
Error term
WHO VIP Webinar 201172
Statistical analysis
Methodology does exist for developing stronger collective evidence, evaluating the effectiveness of community based interventions, using different types of study designs and interventions
Developing “practice-based evidence”
Safety 2010 London73
Dissemination
We should not stop at developing the evidence We must work alongside economists in developing
ways to effectively communicate ‘what works’ methodology and cost models do exist for estimating the “return on investment”
Money talks !!
WHO VIP Webinar 201173
WHO VIP Webinar 201174
Challenges – Evaluation requires
Integration of evaluation from the beginning Appropriate measures, possible to be collected objectively,
unbiasedly, easily and with completeness Appropriate qualitative and process information, to complement
the quantitative information Concrete and convincing evidence of what aspects work in
individual communities Formal methodological statistical evaluation of specific elements
of programs Collective evidence of what common elements of programs work Effective dissemination strategies – “return on investment”
WHO VIP Webinar 201175 75