Evidence-Based Practice – Basics and Global Trends –

Tomoya Masaki,

Senior Advisor, Evaluation Department

Japan International Cooperation Agency

13th ODA Evaluation Workshop 2015, Tokyo

Outline

Background of evidence-based practice (EBP)

1. Impact evaluations (IEs)

2. Evidence-based medicine (EBM) as a basis for EBP

3. Why comparative designs are applied?

• Key messages

• References & Web resources

Box: What does “statistical significance” mean?

2

Background • Social movements in UK & US

– First published randomized controlled trial (RCT) in UK, 1948

– Wide-spread of RCTs from early 1990s globally

– Government performance and results act in US, 1993

– Result based management; Effective & efficient use of resource for public health; Outcome/decision research; Managed care etc.

– By the late 20th century, RCTs were recognized as the standard method for "rational therapeutics" in medicine.

3 *: International initiative for impact evaluation

• Infrastructure of EBP (systematic reviews of RCTs) Two global networks for dissemination

– CC: The Cochrane Collaboration (1992-)

– C2: Campbell Collaboration (2000-)

• 3ie* had joined to C2 as international development group (2009-, nearly 2,400 IEs had been registered)

Experi-mental design

w/ RCTs or

cluster RCTs

Impact evaluation (IE)

1. Impact evaluation (IE)

• Global trends

• Image of impact evaluation

– Measure impact of intervention

– All project can be recognized as an intervention

4

IE EBM

Evidence based practice (EBP)

EBP w/RCT Non-experimental

design (no comparison group; pre-post comparison)

Quasi-Experimental design (e.g. matching, differencing, instrumental variables and the pipeline approach etc.)

No RCTs design (e.g. case report, case series, open labeled etc.)

Impact evaluations published per year (1981–2012).

5 Cameron DB, Mishra A, Brown AN. The growth of impact evaluation for international development: how much have we learned? Journal of Development Effectiveness 2015:1-21.

Impact evaluations published by source (2000–2012).

6 Cameron DB, Mishra A, Brown AN. The growth of impact evaluation for international development: how much have we learned? Journal of Development Effectiveness 2015:1-21.

Impact evaluations published by major sector (2000–2012).

7 Cameron DB, Mishra A, Brown AN. The growth of impact evaluation for international

development: how much have we learned? Journal of Development Effectiveness 2015:1-21.

Source: JICA operations Evaluation System. http://www.jica.go.jp/english/our_work/evaluation/reports/2014/c8h0vm00009ga6st-att/part1_2014.pdf

Conceptual diagram of the impact evaluation: Comparison of situation actually observed and counterfactual situation.

8

2. Evidence based medicine (EBM) as a basis for impact evaluations

• EBM→EBP as a basis for IEs

• Evidence flow: generate, communicate* and utilize

• Wide-spread of randomized controlled trials (RCTs) in healthcare and practice

• From evidence to recommendation

• Evidence as an element in original conceptual model of EBM

9

*: Both of the Cochrane Collaboration (CC) and Campbell Collaboration (C2) are growing as the infrastructure for evidence communication.

Evidence based clinical decisions

• 1996 “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients”

• 2000 “the integration of best research evidence with clinical expertise and patient values”

Haynes RB, Devereaux PJ, Guyatt GH. Clinical expertise in the era of evidence-based medicine and patient choice. EBM 2002;7:36-8.

Clinical expertise

Research evidence

Patient preferences

Clinical state and circumstances

Patients’ preferences and actions

Research evidence

Clinical expertise

Early model, 1996 Updated model, 2000

10

3. Why comparative designs are applied?

• A conceptual figure with natural course fluctuations

– Really good intervention but recognized as bad!

– Really bad intervention but recognized as good!

– How can we identify the causality between an intervention and outcome?

11

Issue points in evaluation

Time

Posi

tive

alt

erat

ion

Intervention

1. Timing of evaluation

2. Data availability 3. Compare to the base-line or group

Solved by … Study design (e.g. RCT, target population & sample size)

12

4. Counter factual

5. Group comparison

Natural course dilemma A

Intervention A

Time Disaster (or catch cold)

Observed indicator will be recognized as aggravation after “A” even though the “A” is a really good intervention.

Agg

rava

tio

n

(e.g

. ob

s.#

of

dia

rrh

ea, f

ever

)

13

Ideal natural trend (invisible but real trend)

observed

Natural course dilemma B A

ggra

vati

on

(e

.g. o

bs.

# o

f d

iarr

hea

, fev

er)

Intervention B

Disaster (or catch cold)

Intervention “B” is really harmful but the observed indicator shows improve.

14

Time

Natural course dilemma A

lter

atio

n

intervention To avoid such before-after observational misunderstanding, design of RCT is useful.

15

Time Disaster or catch cold

Ensuring generalizability through random sampling and allocation

16 Source: Tsutani K. Activities of Cochrane Collaboration (in Japanese), 1994 (accessed 25 Nov, 2015) http://www.lifescience.co.jp/yk/jpt_online/cochrane/index_cochrane1.html

external validity internal validity

17

Intervention (e.g. special education)

No intervention (do nothing or usual practice)

With v.s. without comparison

Random allocation for internal validity

Method of RCT

Group A

Group B

Outcome A

Outcome B

Evaluation (e.g. test score)

Before-after d

iffe

ren

ce

Cluster RCTs as a common method for international development projects

Cluster RCTs

RCTs

Research Ethics (subject protection, integrity, equipoise)

18

Consideration: Are the cluster RCTs for research or development cooperation (non-research practice)?

•Area, Government Districts •Institute, School, Hospital ward

•Family, Class, Teacher, Doctor

•Individual subjects

Allocation unit

0

20

40

60

80

100

120

140

19

76

19

80

19

81

19

82

19

83

19

84

19

85

19

86

19

87

19

88

19

89

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

20

11

20

12

20

13

20

14

20

15

N.B. The numbers do not indicate the number of trials of cluster randomization. Query: cluster[All Fields] AND ("random allocation"[MeSH Terms] OR ("random"[All Fields] AND "allocation"[All Fields]) OR "random allocation"[All Fields] OR "randomization"[All Fields])

Total: 1,149 hits, as of 20 Nov 2015

Number of publications in PubMed through searching the term “Cluster Randomization”

Nu

mb

er o

f re

po

rts

19

Publication year

Key messages • EBP is increasingly applied in many areas including

IEs for international development projects.

• The original concept of EBM implies that how decision making by IE results should be -- cooperative decision making by experts and participants with each other in a given circumstance is crucial.

• Why method of randomized controlled trials (RCTs) are applied? Because RCTs are a type of impact evaluation which will be able to limit bias and generating an internally valid impact estimate.

• Application of cluster RCTs is rapidly becoming a common practice for IEs of international development projects.

20

Key messages (cont’d)

• Statistical significance can be led by “effort” and “negligence” (pls see Box).

• To avoid such unexpected meaningless statistical significance by chance, appropriate designing of IE in line with the study is quite important to create a meaningful evidence for decision making.

21

Reference 1. Textbook

Gertler P J, Martinez S., Premand P., Rawlings L. B., Vermeersch C. M. J. (2010). Impact Evaluation in Practice. The World Bank. DOI: 10.1596/978-0-8213-8541-8

http://www.worldbank.org/pdt

2. Global trend of impact evaluation Cameron, D. B., Mishra, A., & Brown, A. N. (2015). The growth of impact evaluation for international development: how much have we learned? Journal of Development Effectiveness, 1-21.

3. Online database of impact evaluation 3ie: international initiative for impact evaluation

http://www.3ieimpact.org/

4. Current topics in 3ie journal Special Issue: Improving lives through better evidence: an acknowledgement of the contribution of Howard White to international development. Journal of Development Effectiveness Volume 7 2015, Issue 3 2015, 267-392 [TOC]

22

Box: What does the statistical significance mean?

• Statistical significance is no more/less than itself – it can be understood as rare situation had happened by compared to a set probability known as α error (threshold of specific risk or misunderstanding).

• Expressed by stars such as *: p<0.10, **: p<0.05, ***: p<0.01” in figures and tables.

• These signs are independent from meaningful difference of project activities or magnitude of interventions (just express “statistical significance”).

• A meaningful difference (or not) should be evaluated by decision-makers who have adequate knowledge and experiences in the given area.

23

Box: What does the statistical significance mean?

We should be fully aware of:

• Statistical significance can be led by “Effort” & “Negligence” by chance.

“Effort” –– stratification, multiple testing, large sample size, biased/intentional data cleaning

“Negligence” –– loose check, accept errors and/or misunderstanding

24

Box: Relationship between sample size and Effect size d (α error=.05 with power 0.8)

25 Effect size d (ranged 0-1) can be defined as normalized strength of intervention by standard deviation. It can be understood like as signal/noise ratio (S/N).

Calculated by G*Power 3.1.9.2

THANK YOU!

Tomoya MASAKi, MPH, PhD, CAE Senior Advisor, Evaluation Department, Japan International Cooperation Agency

Evidence-Based Practice: – Basics and Global Trends –

13th ODA Evaluation Workshop 2015, Tokyo

26