Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6

HENMethodology

Step by step

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Reviewing evidence

• Systematic reviews are increasingly common

• Cochrane, Campbell collaboration, DARE, HTA…

• These are often done by experts• Can be slow (6 months plus) • Expensive: $130,000 average• Probably needs 2+ people per review

• We don’t have time!• We don’t have the resources!

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6

P rim ary research S econdary synthesis or in tegration

1. C ontro lled c lin ical tria ls

2 . Longitud inal observational stud ies

3 . C ross-over observational stud ies

4 . C lin ical series

5 . Surveys

1 . C lin ical data-bases

2 . Ep idem io log ical data-bases

3 . A dm in istrative data-bases

4 . Econom ic data-bases

5 . C ensus

6. Patterns o f use

7 . R eg isters, records

1 . M eta-analysis

2 . C lin ical decis ion analysis

3 . Econom ic analysis

4 . Q ualitative data analysis

Types of methods to produce evidence

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6The starting point for HEN

is the policy makers’ question

Policy makers’ question

Synthesisof

evidence

10 page reports&

1 page summary

Final Version

POLICY QUESTION

HEN – TUs

Refine questions

Identify authors

HEN

Possible reject

Drafting Report

Initial Review

Author

External / Internal Reviewers

TUs

Peer Review

Draft revision

Possible reject

HEN Quality Control

Possible rejectAuthor

Final Synthesis

HEN

Communication

INFORMATION NEED

Identify sources

Description

Mapping

Documents/DB Selection

Edit

Resources

HEN

Consultant

HEN

Freelance editor

HEN

Member States’ Evidence Needs

Synthesis

Update

AuthorsRe-phrase question

Immediate answers

Resources

HEN

Copy-editor

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Collection of questions

A) Proactive• Call for topics once a year

- Ministries of health (using WHO channels) - Technical Units of WHO- Network Members

• Systematically review the work of HEN Members

B) Reactive• HEN electronic mailbox • Phone questions from policy makers • Questions or policy concerns identified by WHO

Technical Units• Issues raised at regional committees and ministerial

conferences

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Prioritization of the policy concerns

• Highest priority: questions from Ministries of Health

• Priority areas of WHO Europe – health systems– mental health– child health– environment– HIV/AIDS – nutrition– non-communicable diseases– ageing– poverty

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Selection criteria

• Availability of evidence (after preliminary literature searches, and discussions with technical staff)

• Feasibility (is it practically possible to produce a synthesis report within a reasonable time and budget frame?)

• Relevance for audience • Coverage (whether the proposed public

health questions are of interest to a number of Member States or only one)

• Timeliness (how long it will take to produce the answer)

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Selection of authors

• Proven ability to undertake a systematic review (participation in systematic reviews conducted by the Cochrane Collaboration, HTA agencies)

• Proven international record of publication in the field of public health (international scientific papers, indexed in Medline/PubMed or another scientific bibliographical database)

• Proven record of communication with policy makers (indicated by the topics of the experts publications, conference presentations or teaching areas) (Medline, CV)

• Availability and possibility to produce a paper in a given time table

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Standard structure

for synthesis reports

The report:

• Introduction

• Findings from health related research

• Other knowledge

• Current debate

• Discussion

• Conclusions

• References

The summary:

• Issue

• Findings

• Policy considerations

• Type of evidence

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Quality Control

1) Author has conducted a proper and systematic search and is transparent with the search strategy

2) Inclusion/exclusion criteria and the methods of analysis are described properly

3) List of references, adding those relevant to the paper (and to our Member States)

4) Findings/results reported critically but objectively

5) Style suitable for policy makers (not too clinical or technical)

6) Paper and summary follow the standard format7) Policy considerations are clear, based on

findings and provide concrete support for decision making or action

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Grading of evidence and strength of policy options

Strong evidence – consistent findings in two or several scientific studies of high quality

Moderate evidence – consistent findings in two or several scientific studies of acceptable quality

Limited evidence – only one study available or inconsistent findings in several studies No evidence – no study of acceptable scientific quality available

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6

Methodological appropriatenessdifferent types of questions best answered by

different types of studyResearch question Qualitative

ResearchSurveys Case control

studiesCohortStudies

RCTs

EffectivenessDoes this work?

+ ++

Effectiveness ofservice delivery:How does it work ?

++ +

SalienceDoes it matter ?

++ ++

AcceptabilityWill children/parentswant to use it?

++ + +

AppropriatenessIs this the right servicefor these children?

++ ++

Muir Gray, 1997 (on social interventions in children)

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6(Randomised) Controlled Trials

• Was the assignment to the treatment groups random? • Was relatively complete follow-up achieved? • Were the outcomes of people who withdrew

described and included in the analysis? • Were the control and intervention groups similar at

the start of the study? • Were the groups treated identically (other than the

intervention/s of interest?

• How big is the study? How big is the effect? Do the numbers add up?

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Cohort Studies (that follow cohorts

of people over time)

• Is the sample representative?

• What else happened? (What factors other than the intervention may have affected the outcome, and were the cohorts being compared comparable on these important confounding factors?

• Are the outcomes meaningful?

• Was there adequate statistical adjustment, or matching for the effects of these confounding variables?

• How big is it? How big is the effect? Do the numbers add up?

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Surveys (cross-sectional - one

point in time)

• Is the study based on a representative (random) sample?

• Was follow-up long enough for important events to occur?

• Are the measures meaningful?• Do we know how people got into the survey?• Do we know how many were surveyed and how

many refused?• How big is the study? (Big surveys are not

necessarily better)• Who dropped out?• Surveys: gold mines for data-dredgers

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Appraisal of qualitative studies

• How credible are the findings?• How clear is the basis for evaluation?• How defensible is the research design?• How well was the data collection carried

out?• Hoe well has diversity of perspective and

content been explored?• How adequately has the research

process been documented?• Etc.

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Assessing the validity of literature

reviews

Look for sources of bias. The main ones are:

•A poorly defined question •A limited search for literature•Poorly defined inclusion/exclusion

criteria•Lack of assessment of the validity

of the included studies•Lack of investigation of

heterogeneity•Inappropriate pooling of studies

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6The “systematic-ish” review

• Looks a bit systematic...• Includes details of some databases that

were searched...• Uses the right jargon...• Doesn’t appraise the included studies...• The conclusions are not consistent with

the results of the studies...

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6

Reading other people’s reviews• Are they answering the right question?• Are they using the right evidence to answer that

question?• Is it likely that they missed relevant evidence? • Do you suspect that they are using evidence

selectively?• Have they paid attention to the quality of that

evidence? E.g. Its methodological soundness, and relevance

• Who pays them? Who paid for the review? Are these, or other sources of funding likely to introduce bias?

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6How to be more evidence-based:

Presentation is everything

• Be clear about what questions you are answering

• Be clear about what sort of evidence you believe is admissible, and why

• Be clear about what you have excluded/included

• Reference well-conducted research, in preference to opinion pieces, editorials, general reviews, general WHO or World Bank commentaries

• Reference relevantly• Show them 1. methods and 2. the evidence

Work

sh

op

on

VH

L a

nd

HEN

, S

ao P

au

lo,

10

-11

Ap

ril 2

00

6Always check the evidence -

whatever the source

• “Those who can make you believe absurdities...can make you commit atrocities” Voltaire 1694-1778