On snow, chocolate and repair shops

J. C. Villasboas

Larry Young

To shun fear and learn to love forest plots

To review principles of when it is appropriate to combine

studies

Interpret the results of a meta-analysis including forest plots

Understand what systematic reviews can and cannot do for you

Demonstrate teaching techniques and identify resources to

assist

Teaching Objectives

Extra-Special Teaching Objective

CERTIFIED to be JARGON FREE

Need a volunteer

Are Systematic Reviews Scary? Raise your hand if you…

Always skip the methods section on Systematic

Reviews because it sounds like Gibberish

Make your colleagues teach the systematic review

session so you don’t look like an idiot

Think a “Forest Plot” might be a good buy given

the current housing market

How to decrease fear

What strategies have you seen this week (small or

large group) to decrease fear in the learning

environment?

Teaching Time Out

Simplify concepts

Avoid numbers and Jargon

There are tools to assist your teaching

TIPS Series

Gleeful to Grief-stricken Scale

Validated instrument for assessing levels of comfort (glee)

or fear (grief) in EBM workshop participants

Glee (0) Indifference (5) Grief (10)

Reactions of people on hearing that they need to look at a meta-analysis:

Ridicularia Exemplarum…

The best way to understand just about anything in

medicine is to reflect on a patient…or other life

dilemmas

The following tale is one such dilemma…

Disclaimer

No conflict of interests for presenters

All characters and events in this presentation are fictional

Any use of names (or initials) is purely unintentional and any

similarities to actual persons, in this room or elsewhere, are

purely coincidence

We cannot emphasize enough how coincidental similarities

of our fictional entities to any real life individual, entity, or

character are

Winds of change

Winds of change blow over Miami

It’s 2pm and another hard day of work is over

LY drives back home one breezy Floridian afternoon

He thinks about life

He thinks about his friend and mentor SK

Fleetwood Mac bursts through his stereo

Winds of change

Nostalgia

Nostalgia

Nostalgia

Nostalgia

Nostalgia

LY understands

He decides to buy her a farewell surprise gift

He wants something meaningful.

He decides to call Manny

Manny is smart and lives in Cleveland, OH

He may know one or two things about cold weather

Nostalgia

Finding Manny

Manny has left Cleveland

He succumbed to Winter Blues

He returned to Brazil

He now runs a repair shop

He still does some EBM on the side

Seasonal Affective Disorder - SAD

Manny vs. S.A.D.

LY worries for SK

After so many years in Miami it

may be hard for her to adapt

He asks for Manny’s help

Manny vs. S.A.D.

Manny vs. S.A.D.

Manny tried several interventions

Lights, exercise, binge drinking

Nothing seemed to work

He is convinced prevention is key

He thinks he has found the answer

Chocolate vs. S.A.D.

After an extensive search, he has

found 4 RCTs on chocolate for

S.A.D. prevention

But he knows LY wants the best

evidence

He is not awfully busy lately

He could put them together for LY

The Systematic Review

Somewhere in those 4 studies.

The Evidence Pyramid (Of Truth)

Unsystematic Clinical Experience

Case-Control

Case Series

Observational Studies

(Cohort)

RCT

Systematic Review/Meta-analysis

Let’s help Manny summarize each study

P I C O F/U Cearras et al

Ribeiro et al

Pastorini et al

Brown et al

P I C O F/U Cearras et al

Ribeiro et al

Pastorini et al

Brown et al

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al

Pastorini et al

Brown et al

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al Adults at risk

of S.A.D. Hot chocolate Placebo

Incidence of

S.A.D. 2y

Pastorini et al

Brown et al

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al Adults at risk

of S.A.D. Hot chocolate Placebo

Incidence of

S.A.D. 2y

Pastorini et al Adults at risk

of S.A.D. Chocolate bar Cinnamon rolls

Incidence of

S.A.D. 2y

Brown et al

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al Adults at risk

of S.A.D. Hot chocolate Placebo

Incidence of

S.A.D. 2y

Pastorini et al Adults at risk

of S.A.D. Chocolate bar Cinnamon rolls

Incidence of

S.A.D. 2y

Brown et al Healthy pedia

tric volunteers Chocolate bar Placebo

Change in GGs

Scale 2y

Next Exercise

Instructions to be given by a visiting celebrity scholar….

Go to video and then do what the video tells you to do….

(use discretion in your interpretation)

Celebrity Scholar

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al Adults at risk

of S.A.D. Hot chocolate Placebo

Incidence of

S.A.D. 2y

Pastorini et al Adults at risk

of S.A.D. Chocolate bar Cinnamon rolls

Incidence of

S.A.D. 2y

Brown et al Healthy pedia

tric volunteers Chocolate bar Placebo

Change in GGs

Scale 2y

One of these things is not like the other things…

One of these things is not like the other things…

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al Adults at risk

of S.A.D. Hot chocolate Placebo

Incidence of

S.A.D. 2y

Pastorini et al Adults at risk

of S.A.D. Chocolate bar Cinnamon rolls

Incidence of

S.A.D. 2y

Brown et al Healthy pedia

tric volunteers Chocolate bar Placebo

Change in GGs

Scale 2y

One of these things is not like the other things…

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al Adults at risk

of S.A.D. Hot chocolate Placebo

Incidence of

S.A.D. 2y

Pastorini et al Adults at risk

of S.A.D. Chocolate bar Cinnamon rolls

Incidence of

S.A.D. 2y

Brown et al Healthy pedia

tric volunteers Chocolate bar Placebo

Change in GGs

Scale 2y

One of these things is not like the other things…

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al Adults at risk

of S.A.D. Hot chocolate Placebo

Incidence of

S.A.D. 2y

Pastorini et al Adults at risk

of S.A.D. Chocolate bar Cinnamon rolls

Incidence of

S.A.D. 2y

Brown et al Healthy pedia

tric volunteers Chocolate bar Placebo

Change in GGs

Scale 2y

One of these things is not like the other things…

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al Adults at risk

of S.A.D. Hot chocolate Placebo

Incidence of

S.A.D. 2y

Pastorini et al Adults at risk

of S.A.D. Chocolate bar Cinnamon rolls

Incidence of

S.A.D. 2y

Brown et al Healthy pedia

tric volunteers Chocolate bar Placebo

Change in GGs

Scale 2y

One of these things is not like the other things…

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al Adults at risk

of S.A.D. Hot chocolate Placebo

Incidence of

S.A.D. 2y

Pastorini et al Adults at risk

of S.A.D. Chocolate bar Cinnamon rolls

Incidence of

S.A.D. 2y

Brown et al Healthy pedia

tric volunteers Chocolate bar Placebo

Change in GGs

Scale 2y

One of these things is not like the other things…

Is pooling sensible?

A priori determination of whether it makes sense

to combine on a principled basis

You have not looked at the data yet…

Is pooling sensible?

MA of all kinds of treatments for all cancers?

MA of steroid treatments for COPD?

MA of studies looking at CT-angiogram for

diagnosis of pulmonary embolism?

P I C O F/U Cearras et al

Adults at risk

of S.A.D. Chocolate bar Placebo

Incidence of

S.A.D. 2y

Ribeiro et al Adults at risk

of S.A.D. Hot chocolate Placebo

Incidence of

S.A.D. 2y

Pastorini et al Adults at risk

of S.A.D. Chocolate bar Cinnamon rolls

Incidence of

S.A.D. 2y

Brown et al Healthy pedia

tric volunteers Chocolate bar Placebo

Change in GGs

Scale 2y

Does combining RESULTS make sense?

Comes from a review of dots and lines

What are the dots and lines in a MA?

Favors New Treatment Favors Control No Difference

Result 1

Eye-ball test

Favors New Treatment Favors Control No Difference

Can we combine?

Result 2

Favors New Treatment Favors Control No Difference

Eye-ball test

Can we combine?

Favors New Treatment Favors Control No Difference

Result 3

Favors New Treatment Favors Control No Difference

Eye-ball test

Can we combine?

Favors New Treatment Favors Control No Difference

Okay: What is the Eye-ball test?

Eye-ball test

Visual impression of degree of overlap of each

study’s results (find a common neighborhood of

truth)

Visual impression of where the estimates of study

“effects” themselves lie (where are the dots)

Favors New Treatment Favors Control No Difference

Back to result 3: if not, why not?

Favors New Treatment Favors Control No Difference

Why might things be different?

Different age groups

Chocolate vehicle

Active control / placebo

incidence of SAD vs. GGsS

Let’s help Manny build a forest plot

1 2 5 10 0.5 .2 .1

Cearras

Pastorini

Ribeiro

Brown

RR Cocoa product better Cocoa product worse

To Combine Or Not To Combine?

Systematic review vs. meta-analysis

Disbelievers be comforted…

1 2 5 10 0.5 .2 .1

Cearras

Pastorini

Ribeiro

Brown

RR Cocoa product better Cocoa product worse

We Don’t Have To Combine

1 2 5 10 0.5 .2 .1

Cearras

Pastorini

Ribeiro

RR Cocoa product better Cocoa product worse

Brown

Or we can try to explain the differences in Brown’s trial (pediatric and GGsS) and find a more similar grouping

Back To Our Scenario

I guess it works!

Who cares about calories?

LY will make sure she takes enough chocolate

with her

Tips for minimizing learner fear

Use non-medical examples

Use common sense approach explanations rather

than statistical ones

Now to a real life example Right after this commercial message

(5 min bathroom break)

Case Scenario: Take Tai Chi twice a week

80yo aunt calls

She was urged by her PCP to start exercising

He is trying to decrease her fall risk

He recommends she starts Tai Chi classes

PCP’s wife owns the only Tai Chi gym in town

She is suspicious and wants my advice

Being the EBM nerd that you are, you reply, “I’ll find out and

get back to you!”

What’s our first step?

Case Scenario: Take Tai Chi twice a week

Those 6 Letters Again

elderly adults

exercise [Tai Chi]

no intervention

falls

Treatment Question

RCT or Meta-analysis of well done RCTs

Success!

Let’s appraise the article together… in 5 minutes?

RAME IT

Did the authors define (frame) a sensible clinical question using

PICO framework?

IND IT

Did the authors do a detailed and exhaustive search (find) that

is unlikely to systematically miss important relevant studies?

Words that start with the letter “F”

Words that start with the letter “F”

ILTER AND ETCH

Did authors select studies based on inclusion criteria, assess

risk of bias, collect data from individual studies in a

reproducible manner?

INDINGS AND OREST PLOTS

Review findings of a systematic review and the forest plots or

summary statistics if graphs are not provided for meta-analysis.

Success… Appraisal

Words that start with the letter “F”

RAME IT

IND IT

ILTER AND ETCH (AND IGHT)

INDINGS OREST PLOTS

What are the results?

Everyone: look at Figure 2 (page 820)

Describe the dots and lines

Exercise or Physical Therapy Interventions

Favors Control Favors Intervention

I2 what?

From 0 to 100

The variability that is due to a real difference

I2 = 0

I2 = 100

What is the range of possible values for I2

Explain in a sentence?

Is the variability between studies due ONLY to chance?

Is the variability between studies due ONLY to a real difference in treatment?

Favors New Treatment Favors Control No Difference

Result 1: what is I2?

I2= ____ 10%

Result 2: what is I2?

Favors New Treatment Favors Control No Difference

I2= ____ 5 %

Result 3: what is I2?

Favors New Treatment Favors Control No Difference

I2= ____ 90%

Exercise or Physical Therapy Interventions

Favors Control Favors Intervention

Single Clinical Intervention: Vitamin D

Favors Control Favors Intervention

Back to your aunt…

Tai Chi classes are reasonable fall prevention strategies

Certainly other exercises are equally reasonable

Vitamin D supplementation can also be considered

What Systematic Reviews Can Do for You

Save time

Increase power to detect rare events

Obviate need for large, expensive trials

Detect harm

Increase the precision of the estimate of effect

Enhance generalizability of the results if samples from

different populations are included

What Systematic Reviews Can’t Do for You

Allow you to determine “equivalency” of different

interventions

“Get rid of” bias from individual studies

To shun fear and learn to love forest plots

To review principles of when it is appropriate to combine

studies

Interpret the results of a meta-analysis including forest plots

Understand what systematic reviews can and cannot do for you

Demonstrate teaching techniques and identify resources to

assist

Teaching Objectives

Extra-Special Teaching Objective

JARGON FREE (at least for

the first half…)

Avoided words

Point Estimate

Confidence Interval

Summary Statistic

Heterogeneity

I2

Take Home Points

A well done meta-analysis seeks to get us closer to the

truth by combining results of well done individual trials

Sometimes it’s okay to combine the results of different

trials… sometimes it’s not

Differences between trials can be assessed using

different methods

Take Home Points

I2 = 0 is EXCELLENT

I2 = 100 is AWFUL

We can all still learn a lot from Cookie Monster

Gleeful to Grief-stricken Scale

Validated instrument for assessing levels of comfort (glee)

or fear (grief) in EBM workshop participants

Glee (0) Indifference (5) Grief (10)

We Hope It Wasn’t So Scary After All