00crit+Appr+Ppt+Ss

8/13/2019 00crit+Appr+Ppt+Ss

1/64

Introduction to

Critical appraisals of the

medical literature

Partini Pudjiastuti, Sudigdo Sastroasmoro

Department of Child HealthMedical School University of Indonesia


2/64

Evidence based medicine 5 steps

Formulate question

Efficiently track

down bestavailable

evidence

Critically review thevalidity and usefulness

of the evidence

Implement

changes in

clinical practice

Evaluate

performance


3/64

>25,000 journals worldwide

>2 million published articles per year

Many published articles have methodological(including statistical) flawseven in mostrespected journals

Not all results can be applied due to many

reasons, a.o. dissimilarities of study subjectswith our patients

Limited time of physicians: focus on articlesrelevant to your clinical practice

Rationale


4/64

Errors and inappropriateness may occur

in any part of medical research:

in choosing appropriate design

in choosing population

in selecting study subjects

in the details of the design

in intervention and measurements

in use of statistical methods in analysis

in interpreting statistical analysis

and in writing research report


5/64

The effect of dietary habit on

calcium level in pediatric patientswith nephrotic syndrome

J Agric Soc Sci 2006;2

r = - 0.1213, p = 0.015

There was a significant inverse relationshipbetween the age and the total calcium level


6/64

IMPORTANT!!!

Statistical significance vs.clinical

importance

Negligible clinical difference may bestatistically very significant if the number of

subjects >>>. e.g., difference in reduction of

cholesterol level of 3 mg/dl, n1=n

2= 10,000; p

= 0.00002

Large clinical difference may be statistically

non- significant if the no of subjects


7/64

R

300

mg/dl

300

mg/dl

Standard

treatment

New

treatment

Cholesterol level,mg/dl

t = df = 9998 p = 0.00002

200

197

Clinical

Statistical

Clinical importance vs. statistical significan

n=10000

n=10000


8/64

Cured Died

Standard Rx 0 10 (100%)

New Rx 3 7 (70%)

Fischer exact test: p = 0.211

Clinical importance vs. statistical significan

Absolute risk reduction = 30% Clinical

Statistical


9/64

Critical appraisal

(Making Reading More Worthwhile)

What is Critical Appraisal?

1. Critical appraisal = quality assessment

2. .process of weighing up evidence to see

how useful it is in decision making

3. .a process of assessing the validity,

reliability and usefulness of evidence4. ..is about considering, evaluating and

interpreting information in a systematic and

objective way


10/64

Critically Appraise What You Read.

Separating the wheat from the chaff.

Time is limitedyou should aim to quicklystop reading the dross.

Others contain useful information mixed

with rubbish. Simple checklists enable the useful

information to be identified.


11/64

Critical AppraisalCritical Thinking

Appraising the available evidence to

construct clinical reasoning and to make

decisions

Finding strengths and limitations of

written evidence Deciding what evidence to pay attention

to versus what to ignore


12/64

Why critically appraise?

Supports sound decision making based

on best available evidence

Helps us determine (three Rs):

How rigorousa piece of research is -

Validity What the resultsare telling us -

Importance

How relevantit is to our patient -A licabilit


13/64

Value =

Mortality

Morbidity

QoLPatientSatisfaction HealthStatus

Resources used

Quality

Cost


14/64

What is Evidence?

People disagree on what constitutes

evidence

Evidencegenerally = scientific fact

Evidence - a combination of information

obtained from 3 sources: research,clinical experience, and client preferences

(Kitson, Harvey, & McCormack, 1998)


15/64

Why do we need evidence?

Resources should be allocated to things

that are EFFECTIVE The only way of judging effectiveness is

EVIDENCE

In God we trust all others needevidence


16/64

Sources of Evidence

Primary sources

Based on experiments and published

research Secondary sources

Systematic reviews

Clinical guidelines Journals of secondary publication e.g.

Evidence Based Medicine


17/64

Levels of evidence

1. Syst reviews of RCTs and high quality

RCTs

2. Syst reviews of cohort studies, lower

quality RCTs, outcomes research

3. Syst reviews of case controls, case

control studies

4. Case Series

5. Expert opinion

www.cebm.net


18/64

Types of EvidenceQuestion Types

Type of Question Best Evidence

Health care interventions:

treatment, prevention

Quantitative: Systematic Review of

RCTs or RCT

Harm or Etiology Quantitative: Observational Study -Cohort or Case Control

Prognosis Quantitative: Observational Study -

Cohort, Case Control

Diagnosis or Assessment Quantitative: Comparison to Gold

StandardEconomics Quantitative: Cost-effectiveness Study

Meaning Qualitative: case study, ethnography,

grounded theory, phenomenologic

approach


19/64

Key quality parameters

Validity

Reliability

Importance


20/64

Validity

Internal

Is the study designed in such a way that I

Can trust the findings?

External

Is the study designed in such a way that I

Can generalize the findings?


21/64

Reliability

If the study was conducted again,

would the results be the same?

Usually interpreted as the accuracy

of measurement.


22/64

Importance

What was the effect size

or magnitude of effect?

Clinical vs. statistical significance.


23/64

Tools for Critical Appraisal

Are the results valid?

What are the results?

Will the results help me in patient care?

EBM simplified approach:

V

I

A


24/64

Format of research reports

Title

Authors and Institutions

Abstract & keywords

IMRAD

Introduction - why did I start?

Methods - what did I do?

Results - what did I find?

Discussion - what does it mean?

References


25/64

Check list for medical literature

1. Title2. Authors & Institutions

3. Abstract: Structured? Informative? Abbreviation?

4. Introduction: Length? Relevant ref? Objective?

5. Methods: Design, time and place

Inclusion criteria

Exclusion criteria

Sample size, sampling method

Randomization technique Intervention: masking?

Outcome measurement: blinding?

Primary outcome: type of variable

Secondary outcome: type of variable

Analysis: Clinical, statistical


26/64

6. Results Baseline characteristics

Main outcome

Secondary outcome

7. Discussion General Strength and weakness

Conclusions

8. References

Vancouver style Constant

9. Acknowledgments

10. Ethics approval

11. Conflict of interest

Check list for medical literature


27/64

What to assess?

(in study of cause-effect relationship)

A. General description

Type of design

Target population, source population,sample

Sampling method

Dependent and independent variables Main results?


28/64

B. Internal validity, non-causal relationship

Influence of bias

Influence of chance

Influence of confounders

What to assess?



29/64

Bias

What is a bias?A process that tends to produceresults that depart systematically from the true valuesexisting in the study population

Types of bias1. Sample (subject selection) biases, which may result

in the subjects in the sample being unrepresentativeof the population which you are interested in

2. Measurement (detection) biases, which includeissues related to how the outcome of interest wasmeasured

3. Intervention (performance) biases, which involve how

the treatment itself was carried out.


30/64

C. Internal validity, causal relationship Temporality (cause precedes effect)

Strength of association (large difference, RR, OR, etc) orsmall p value or narrow confidence interval

Biological gradient (dose dependence)

Consistency among studies (diff. populations or designs)

Specificity (certain factor results in certain effect) Coherence (does not conflict with current knowledge)

Biological plausibility: can be explained with current

knowledge (at least in part)

What to assess?



31/64

D. External validity

Applicable to study subjects

Applicable to source population

Applicable to target population

What to assess?



32/64

11 items, each with 3 sections

1. Can you find this information in thepaper?

2. Is the way this was done a problem?

3. Does this problem threaten the validity of

the study?


33/64

11 items

1. What is the research question?

2. What is the study type?3. What are the outcome factors and how are they measured?

4. What are the study factors and how are they measured?

5. What important confounders are considered?

6. What are the sampling frame and sampling method?7. In an experimental study, how were the subjects assigned to

groups? In a longitudinal study, how many reached final

follow-up? In a case control study, are the controls

appropriate? (Etc)8. Are statistical tests considered?

9. Are the results clinically/socially significant?

10. Is the study ethical?

11. What conclusions did the authors reach about the question?


34/64

1. What is the research question?

(Is the way this was done a problem?)

Is it concerned with the impact of an

intervention, causality or determining themagnitude of a health problem?

(Does this problem threaten the validity of the study?)

Is it a well stated research

question/hypothesis?


35/64

2. What is the study type?


Is the study type appropriate to the researchquestion?

(Does this problem threaten the validity of the study?)

If not, how useful are the results produced by

this type of study?


36/64

3. What are the outcome factors and

how are they measured?


a) are all relevant outcomes assessed

b) is there measurement error?

(Does this problem threaten the validity of

the study?)

a) how important are omitted outcomes

b) is measurement error an important

source of bias?


37/64

4. What are the study factors and how

are the measured?


Is there measurement error?


the study?)

Is measurement error an important

source of bias?


38/64

5. What important potential

confounders are considered?


Are potential confounders examined andcontrolled for?

(Does this problem threaten the validity ofthe study?)

Is confounding an important source of

bias?


39/64

6. What are the sampling frame and

sampling method?


Is there selection bias?

(Does this problem threaten the validity

of the study?) Does this threaten the external

validity of the study?


40/64

7. Questions of internal validity


In an experimental study, how were thesubjects assigned to groups?

In a longitudinal study, how many reachedfollow-up?

In a case control study, are the controlsappropriate? of the study?


41/64

8. Are statistical tests considered?


Were the tests appropriate for the data?

Are confidence intervals given?

Is the power given if a null result?

In a trial, are results presented as

absolute risk reduction as well asrelative risk reduction?

How useful are the results?


42/64

9. Are the results clinically/socially

significant?


Was the sample size adequate to detecta clinically/socially significant result?

Are the results presented in a way to

help in health policy decisions? (Does this problem threaten the validity of

the study?)

Is the study useful?


43/64

10. Are ethical issues considered?


Does the paper indicate ethics

approval?

Can you identify potential ethical

issues?


the study?)


44/64

11. What conclusions did the authors

reach about the study question?


Do the results apply to the population inwhich you are interested?

(Does this problem threaten the validity ofthe study?)

Will you use the results of the study?


45/64

Appraisal Tools

Tools from the Critical Appraisal Skills Programme

(CASP)

Systematic Reviews

Randomised Controlled Trials

Qualitative Research Studies

Cohort Studies

Case-Control Studies Diagnostic Test Studies

Economic Evaluation Studies

Available
http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm


46/64

Study Designs Recap

Effectiveness of

Therapy

Risk Factors /Prognosis

Diagnosis

Attitudes & Beliefs

Randomised Controlled

Trial

Cohort Study

Survey using Gold Standard

Qualitative (Interviews,

Observations, etc)


47/64

Critical appraisal

- Valid

- Important

- Applicable

MethodsResults

Discussion


48/64

50 massa pemuda dan mahasiswa berunjuk

rasa memprotes pelecehan seksual yang makin

marak di depan Mabes PolriMenyadari pentingnya Panduan Pelayanan

Medis (PPM), dibentuklah Panitia Penyusunan

SPM di RSCM

Sekuens dan hubungan

subyek-predikat


49/64

Dalam pertemuan ilmiah yang diselenggarakan

setiap tahun yang merupakan ajang untuk

menyajikan perkembangan mutakhir dalam

bidang ilmu penyakit dalam di tanah air tersebutmenyimpulkan bahwa pertemuan tersebut

disamping dilakukan oleh Universitas

selayaknya juga dilakukan oleh cabang-cabang

PAPDI di setiap propinsi, bahkan kalau mungkindisetiap kabupaten


50/64

Plagiarisme adalah tindakan yang dapat diartikan

sebagai pencurian ide atau hasil pemikiran dan

tulisan orang lain yang digunakan dalam tulisan

seolah-olah ide atau tulisan orang lain tersebut

adalah ide atau hasil tulisannya sendiri untuk

keuntungannya sendiri sehingga merugikan oranglain baik materiil maupun non-materiil, atau

plagiarisme dapat berupa pencurian sebuah kata,

frase, kalimat, atau alinea, atau bahkan pencuriansuatu bab dari sebuah tulisan atau buku

seseorang, tanpa menyebut sumber yang dicuri.

(Draft SK Rektor UI)


51/64

RCT (Pragmatic trials): Validity

Were the study participants randomized?Was the randomization technique described?

Was the randomization table concealed?

Were the characteristics of the subjectssimilar at the start of the intervention?

Were all participants given equal treatment

apart from the intervention?

Were all relevant outcomes considered?

Were the results analyzed correctly?


52/64

RCT (Pragmatic trials): Importance

Calculate: EER, CER, RRR, ARR, NNT

a b

c d

E

C

Success Failure

EER = a/(a+b)

CER = c/(c+d)

RRR = (CER-EER)/CER

ARR = CER-EER

NNT = 1/ARR


53/64

RCT: Applicability

Were the participations similar to your

patients?

May be intuitively concluded or use f(factor indicating how much severe your

patient compared to the study participation

in terms of prognostic factor)


54/64

DIAGNOSTIC TEST: Validity

Was independent and blind comparison

to gold standard applied?Was the diagnostic test include

spectrum of disease similar to your real

practice?Was the gold standard applied

regardless of the diagnostic result?


55/64

Diagnostic Test: Importance

Calculate: Sensitivity, specificity, predicitivevalues, likelihood ratios

a b

c d

+

-

+ -

Se = a/(a+c)

Sp = d/(b+d)

PPV = a/(a+b)

NPV = d/(c+d)

LR+ = se/(1-sp)

LR - = (1-se) /sp

Posttest odds = Pretest odds x LR+

Test


56/64

Diagnostic Test: Applicability

Were the participations similar to your

patients?

Is the diagnostic test applicable,acceptable, and affordable in your setting?

Will the result of the test help your

patient?


57/64

Cohort Studies: Validity

Was the inception cohort assembled in usualpoint of course of the disease?

Was the follow-up sufficient & complete?

Were outcome criteria applied in blind fashion?

Was there any validation in other group ofpatients?

Was subgroup analysis performed afteradjustment for prognostic factors?


58/64

Cohort studies: Importance

Calculate: Relative risk (RR) and 95% CI

a b

c d

+

-

+ -

Exposure

RR = a/(a+b) : c/(c+d)RR > 1: risk factor

RR < 1 : protective factor

RR = 1 : not a risk factor


59/64


60/64


61/64

Case control studies: Importance

Calculate: Odds ratio (OR) and 95%CI

a b

c d

+

-

+ -

Exposure

OR = a/b : c/dOR > 1 : risk factor

OR < 1 : protective factor

OR = 1 : not a risk factor


62/64

Case-control studies:

Applicability Were the patients similar to yours?

Wil the evidence make a clinically

important impact for overall care

(diagnosis, treatment, prognosis) of your

future patients?b


63/64

From Data to Wisdom

Dataare what researchers collect

Informationresults when data is analyzed

and interpreted (EVIDENCE) Knowledgeresults when information is

shared, acquired, and used

Wisdomis the ability to make the right useof knowledge


64/64

THANKS

Documents

00crit+Appr+Ppt+Ss