CRITICAL APPRAISAL OF THE LITERATURE

Prof AE Zemlin

Division of Chemical Pathology

Tygerberg Hospital

Introduction

• Part of EBM

• Research should be reported transparently:

• What was planned?

• What was done?

• What was found?

• What conclusions?

Evidence-based decisions

Part of EBLM

ASK

ACQUIRE

APPRAISE

ANALYSE

APPLY

ASSESS

EVIDENCE IN LABORATORY MEDICINE

Systematic review

• Benefits:• Multiple studies

• Meets statistical power

• Can identify variations in practice

• Limitations:• Often few studies meet inclusion and quality criteria

• Require identical outcome measures

Randomized control trial*

• Benefits:• Offers lowest risk of bias

• Limitations:• Assessment of test and treat intervention

• Generalizability to entire population for test complicated

• Blinding difficult

Observational study, e.g. audit

• Benefits:• Most representative of routine practice

• Useful for audit

• Good for assessing care processes, e.g. laboratory practice guidelines

• Limitations:• Intervention and control groups may not be rigorously

matched

• Open to bias

Before and after trial

• Benefits:• Practical design

• Limitations:• Risk of operator or incorporation bias

Case-control study

• Benefits:• Adds insight for exploratory questions

• Limitations:• Exploratory information only

• May not hold in prospective studies

Case study

• Benefits:• Provides information for infrequently encountered

patients

• Limitations:• Limited information and likely not evidence based as

underpowered

Expert opinion

• Benefits:• Insight into how knowledgeable experts approach

questions / problems

• Limitations:• Biases of expert

BIAS – DEPARTURE FROM TRUENESS

Spectrum bias

• Performance of test may differ in different patient populations

• Found often in case-control studies – 2 populations with different characteristics

• Extreme – healthy vs disease

• Greatest effect on results

• Also called selection / recruitment bias

Underpowered study bias

• Power calculations important part of study design

• More for RCT on therapeutic interventions

• Consequence – no insight into variability of measure

Attrition bias

• Participants lost to follow-up

• Must be accounted for

Verification bias

• Work-up bias of referral bias

• Results of index test influence choice of reference test

Observer / operator bias

• Person using test is aware of other info about patient or other investigations

• Review bias – can lead to overestimation of effect

• Can be mitigated by incorporating blinding into study

Disease progression bias

• Long delay between index and reference test being performed

• Change in disease status

Performance bias

• Change in performance

Test reproducibility bias

• Test reproducibility – impact on clinical performance and effectiveness

• Technical performance of analytical methods needs to be described in detail

Reporting bias

• Prevent poor diagnostic studies being reported

• STARD - checklistsm

Publication bias

• Reluctance to publish negative data

• Meta-analysis may not be representative

Conducting a critical appraisal

• Checklist

• Identify problem

• PICO

• EQUATOR – international initiative seeks to improve the value of published health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines

EQUATOR – Enhancing the Quality and Transparency Of health Research

• Diagnostic / prognostic studies – STARD

• Randomised trials – CONSORT

• Observational studies – STROBE

• Systematic reviews – PRISMA

• Case reports – CARE

• Quality improvement studies – SQUIRE

Minimum information that should be included

Standards for Reporting of Diagnostic Accuracy Studies (STARD)

• Completeness and transparency

• Minimum essential items

• Judge for potential bias

Title / Abstract

• Identification as a study of diagnostic accuracy using at least one measure of accuracy

• Structured summary of study design, methods, results and conclusions*

Introduction

• Scientific / clinical background including intended use and clinical role of index test *

• Study objectives and hypothesis *

Methods – study design

• Was data collection planned before index test and reference standard were performed (prospective) of after (retrospective)?

Methods - participants

• Eligibility criteria

• How were they identified?

• Where and when were potentially eligible participants identified?

• Did participants form a consecutive, random or convenience series?

Methods – test methods

• Index test

• Reference tests

• Rationale for choosing reference standard

• Definition and rationale for test positivity cut-offs or result categories of reference standard

• Definition and rationale for test positivity cut-offs or result categories of index test

• Whether clinical information and reference standard results were available to the performers / readers of the index test

• Whether clinical information and index test results were available to the assessors of reference standard

Methods -analysis

• Methods for estimating or comparing measures of diagnostic accuracy

• How indeterminate index test or reference standard results were handled

• How missing data on the index test or reference standard were handled

• Any analyses of variability in diagnostic accuracy, distinguishing pre-specified from exploratory

• Intended sample size and how was determined *

Results - participants

• Flow diagram of participants

• Baseline demographic and clinical characteristics of participants

• Distribution of severity of disease in those with target condition

• Distribution of alternative diagnoses in those without target condition

• Time interval and any clinical interventions between index test and reference standard

Results – test results

• Cross tabulation of the index test results (or their distribution) by the results of the reference standard

• Estimates of diagnostic accuracy and their precision (e.g. 95% CI)

• Any adverse events from performing index test or reference standard

Discussion

• Study limitations, including sources of potential bias, statistical uncertainty, and generalisability *

• Implications for practice, including the intended use and clinical role of the index test *

Other information

• Registration number and name of registry *

• Where full study protocol can be assessed *

• Sources of funding and other support; role of funders *

Conclusion

• Necessary to assess quality and relevance of evidence to support clinical decision and policymaking

• Number of checklists available depending on type of study being appraised

• All assess:1. Question being addressed

2. Quality of methodology used

3. Quality of results

4. Applicability to the problem and potential consequences of using this evidence

• Failure to adequately appraise may lead to inappropriate testing

QUESTIONS??