Dr. Erica Roebbelen

January 16, 2015

Banda Aceh

CRITICAL APPRAISAL

The process of systematically examining research evidence to assess validity, results and relevance before using it to inform a decision

Why do you think this is important?

WHAT IS CRITICAL APPRAISAL?

Research articlesExpert opinion – from our seniors, from staff

physiciansCase reportsAnecdotes or storiesPopular media

WHAT TYPES OF EVIDENCE DO WE ENCOUNTER?

Case-control study

REVIEW TYPES OF STUDY DESIGN

• Cohort Study• Randomized Control Trial

• Meta-analysis• Qualitative

Studies

Observational or experimental

What types of studies give us the strongest evidence?

ASSESSING EVIDENCE

ASSESSING EVIDENCE

When reading a study, always ask yourself these 4 questions:

Who is the POPULATION or PATIENT CHARACTERISTICS?

What is the INTERVENTION?What is the COMPARISON or CONTROL group?What is the OUTCOME being measured?

Remember… PICOThese are also the steps to take to formulate a

research question

APPROACH TO CRITICAL APPRAISAL

Once you know the basics of the study (the PICO), you need to ask these 3 questions:

1. Are the results of the study valid?

2. What are the results?

3. Will the results help me in caring for my patients?

APPROACH TO CRITICAL APPRAISAL

What is validity? The degree to which the outcome observed in the study can be attributed to the intervention.

ASSESSING VALIDITY

5 questions to ask about the validity of primary studies:1. Were the patients randomized?2. Was the follow-up of patients sufficiently

long enough and complete?3. Were all patients analyzed in the groups to

which they were randomized? (intention to treat analysis)

4. Were the groups treated equally except for the intervention?

5. Were the patients and clinicians kept blind to treatment? (was the study “double blinded”?)

ASSESSING VALIDITY

Per-Protocol Analysis (PP)Strategy of analysis in which only patients who complete the entire study are counted towards the results

Intention to Treat Analysis (ITT)When groups are analyzed exactly as they existed upon randomization (i.e. using data from all patients, including those who did not complete the study)

ANALYSIS

Some more questions to consider:

Were the groups similar at the start of the trial? (think about gender, comorbidities, age, etc.)

Were the appropriate and valid exposure and outcome measures obtained?

Were outcome assessors aware of group allocation?

Was the study conducted in an ethical way?Who funded the study?Do the authors have any conflicts of interest?

ASSESSING VALIDITY

Internal Validity the degree to which the findings of the sample truly represent the findings in the study population

External Validitydegree to which the results of the study can be generalized to other situations or populations

ASSESSING VALIDITY

Once you know the basics of the study (the PICO), you need to ask these 3 questions:

1. Are the results of the study valid?

2. What are the results?

3. Will the results help me in caring for my patients?

APPROACH TO CRITICAL APPRAISAL

What was the impact of the treatment results?

How precise was the estimate of the treatment effect?

What were the confidence intervals and power of the study?

CRITICAL APPRAISAL OF RESULTS

Confidence Interval:provides a range of values within which the true population result (e.g. the mean) lies

frequently reported as 95% CI (e.g. one can be 95% certain that the true value is within this data range)

bounded by the upper and lower confidence limits

A wider confidence interval implies more variance than a tighter confidence interval

A VERY QUICK STATISTICS REVIEW

Power the probability of finding a specified difference to be statistically significant at a given p-value

power increases with an increase in sample size

the probability of a true positive result

A VERY QUICK STATISTICS REVIEW

Relative Risk Reduction (RRR)Absolute Risk Reduction (ARR)Number Needed to Treat (NNT)Number Needed to Harm (NNH)

EFFECTIVENESS OF INTERVENTIONS

EFFECTIVENESS OF A TEST

• SPIN: Specificity – use a SPecific test to rule IN a hypothesis. Specific tests have very few false positives.

• SNOUT: Sensitivity – use a SENsitive test to rule OUT a hypothesis. Sensitive tests have few false negatives.

Once you know the basics of the study (the PICO), you need to ask these 3 questions:

1. Are the results of the study valid?

2. What are the results?

3. Will the results help me in caring for my patients?

APPROACH TO CRITICAL APPRAISAL

are the results clinically significant?can I apply the results to my patient

population?were all clinically important outcomes

considered?are the likely treatment benefits worth the

potential harm and costs?

WILL THE RESULTS HELP ME IN CARING FOR MY PATIENTS?