Critical Appraisal of Health Care Intervention Studies

Dr. Maureen Markle-Reid, RN, MScN, PhDAssociate Professor and Acting Assistant Dean (Research),

School of Nursing, Associate Member, Clinical Epidemiology and Biostatistics

Nursing Graduate SeminarsOctober 4, 2011

Objectives Define evidence-informed decision-making and

describe its importance to nursing practice Identify the parts of a relevant, answerable

question Describe the different types of questions

relevant to nursing practice Define the term critical appraisal and discuss its

relevance to nursing practice, policy and research

Identify criteria appraisal criteria for health care intervention studies

Develop skill in applying the criteria for critical appraisal of an intervention study to determine the quality and applicability of the research.

What is evidence-informed decision-making?

“The conscientious, explicit and judicious use of the current best evidence in making decisions about the care of individual patients”

(Sackett et al, 1996)

Health care resources

Patient preferences and actions

Research evidence

Clinical state, setting, and circumstances

Clinical Expertise

Clinical Decision

A Model for Evidence-Based Clinical Decisions

[Haynes, R.B., Devereaux, P.J., & Guyatt, G.H. (2002). Clinical expertise in the era of evidence-based medicine and patient choice. ACP J Club, 136, A11-14]

Why is evidence-based practice important?

Demonstrate that nursing actions and decisions are clinically appropriate and result in positive outcomes for clients, their families and health care system as a whole

Demonstrate professional accountability to clients Provide evidence for the cost-effectiveness of

nursing care

Steps to Evidence-Informed Decision-Making:

1.1. Define: Define: Formulate a focused answerable question Formulate a focused answerable question from the practice situation based on your information from the practice situation based on your information needs needs

2.2. Search:Search: Efficiently search for research Efficiently search for research

3.3. Appraise:Appraise: Critically and efficiently appraise the Critically and efficiently appraise the research sourcesresearch sources

4.4. Synthesize:Synthesize: Interpret/form recommendations for Interpret/form recommendations for practice based on the literaturepractice based on the literature

5.5. Apply:Apply: Apply the results to the client/population Apply the results to the client/population

6.6. Implement: Implement: Decide whether (and plan how) to Decide whether (and plan how) to implement the adapted evidence into practiceimplement the adapted evidence into practice

7.7. Evaluate: Evaluate: Evaluate the effectiveness of Evaluate the effectiveness of implementation effortsimplementation efforts

Formulate a Focused Answerable QuestionFormulate a Focused Answerable Question

P opulation / Situation

I ntervention / Exposure

C ounter intervention

O utcome T imeframe

Population/Situation:

Client group or clinical scenario of interest Single patient or group of patients with a

particular condition or health care problem, e.g.,: A person with a health condition People with hypertension (a group of people

with a particular condition Primary health care for the elderly (an aspect

of health care delivery)

Intervention/Exposure:

Interventions can be: Therapeutic Preventive Diagnostic Organizational

If exploring the meaning of a phenomena, the question may involve a situation rather than an intervention

Counter Intervention: Clinical decisions involve choosing between alternative

courses of action (or no action)

Outcome:Outcome: What is the outcome, or the effect we are hoping to

achieve by the using the intervention? May be more than one outcome that is important to the

question

Time Frame:

What is the period of time over which the question occurs?

What is the optimal time to measure a change in the outcome(s)?

Step 1:Formulate a focused answerable questionStep 1:Formulate a focused answerable question

Formulating Answerable Clinical Questions (Centre for Evidence- (Centre for Evidence-Based Medicine, Mount Sinai Hospital)Based Medicine, Mount Sinai Hospital)

The Well-Built Clinical Question (Duke University Medical Center (Duke University Medical Center Library and Health Sciences Library, UNC-Chapel Hill) Library and Health Sciences Library, UNC-Chapel Hill)

Formulating Patient Centered Questions (University Library, (University Library, University of Illinois at Chicago)University of Illinois at Chicago)

Asking Focused Questions (Centre for Evidence-Based Medicine, (Centre for Evidence-Based Medicine, Oxford) Oxford)

Types of questions relevant to nursing practice Therapy/Health care interventions: What is the effectiveness

of different interventions? Causation and harm: What might be causing disease/ill

health/adverse effects? Diagnosis or assessment: Does a diagnostic test differentiate

between people with and without a condition? Prognosis: What are potential future outcomes of a condition? Economic evaluation: What is the cost-effectiveness of

different interventions? Meaning: Describing, exploring and explaining aspects of health

and illness.

Nursing Graduate Seminar 200916

Different clinical questions require Different clinical questions require evidence from different research evidence from different research

designsdesignsType of Question Suggested Best Type of Study Design

Therapy/Health care Therapy/Health care intervention intervention

RCT>cohort>case control>case seriesRCT>cohort>case control>case series

DiagnosisDiagnosis Prospective, blind comparison to a gold standardProspective, blind comparison to a gold standard

PrognosisPrognosis Cohort study>case control>case seriesCohort study>case control>case series

Harm (Causation or Harm (Causation or etiology)etiology)

RCT>cohort>case control>case seriesRCT>cohort>case control>case series

Economic evaluationEconomic evaluation Economic analysisEconomic analysis

Meaning of illnessMeaning of illness QualitativeQualitative

Qualitative Survey(cross sectional)

(Randomised)crossover

(Randomised)Parallel group

Cohort study

Cross-sectional(analytic)

Case-control study

Q1

Q3

Q2

All studies

Experimental

Analytic

(PICO or PECO) Descriptive

(PO) Theoretical

(PS)

Observational

analytic

What type of research design will you use?

[Glasziou, P., & Heneghan, C. (2009). A spotter’s guide to study designs. EBN Notebook, 12, 71-72]

Step 2: Collect the best evidence: Hierarchy of Pre-Processed Evidence

[Adapted from Haynes, R.B. (2007). Of studies, summaries, synopses, and systems: The “5S” evolution of information services for evidence-based healthcare decisions. Evidence-Based

Nursing, 10, 6-7]

EXAMPLES: SYSTEMS:

Computerized decision support

SUMMARIES::Evidence-Based Guidelines

~ Evidence-Based Texts  

SYNOPSES OF SYNTHESES:DARE ~ Evidence-Based

Abstract Journals  

SYNTHESES (Systematic Reviews): Cochrane Database

of Systematic Reviews SYNOPSES OF SINGLE

STUDIES:Evidence-Based Abstract

Journals  

SINGLE STUDIES:Clinical Queries

Step 3: Critically appraise the literature for Step 3: Critically appraise the literature for

validity and applicabilityvalidity and applicability

What is critical appraisal?What is critical appraisal?

Critical appraisal is the process of assessing and Critical appraisal is the process of assessing and interpreting evidence by systematically considering its interpreting evidence by systematically considering its validity, results and relevance to an individual's work.validity, results and relevance to an individual's work.

Relevance of Critical Appraisal to Nursing Relevance of Critical Appraisal to Nursing

Practice, Policy and ResearchPractice, Policy and Research

Use the literature more effectively in Use the literature more effectively in answering clinical questions to guide answering clinical questions to guide clinical practiceclinical practice

Distinguish stronger evidence from Distinguish stronger evidence from weaker evidence – identify high quality weaker evidence – identify high quality research research

Identify the methodological strengths and Identify the methodological strengths and limitations, results and relevance of the limitations, results and relevance of the studies in answering a clinical questionstudies in answering a clinical question

Three Steps in Using an Article from the

Health Care Literature

Are the results valid?

What are the results?

Are the results applicable (and useful) to my client care population?

[DiCenso, A., Guyatt, G., & Ciliska, D. (2005). Evidence-based nursing: A guide to clinical practice]

Are the results valid?Are the results valid?

Are the study methods sufficiently Are the study methods sufficiently rigorous to ensure that the study results rigorous to ensure that the study results represent an unbiased estimate of the represent an unbiased estimate of the true effect?true effect?

OROR

Are the study methods sufficiently biased Are the study methods sufficiently biased to lead to a false conclusion?to lead to a false conclusion?

Are the results valid?Are the results valid?

Final assessment of validity is never Final assessment of validity is never a yes/no decisiona yes/no decision

Validity as a continuum ranging Validity as a continuum ranging from strong studies to weak studiesfrom strong studies to weak studies

Evaluation of the validity of a study Evaluation of the validity of a study involves some subjectivity involves some subjectivity

What are the results?What are the results?

Size and precision of the estimate of Size and precision of the estimate of effecteffect

Are the results applicable (and useful) to my client care population?

Can you apply the results to patients Can you apply the results to patients in your clinical setting?in your clinical setting?

Were all important outcomes Were all important outcomes considered? considered?

Are the likely intervention benefits Are the likely intervention benefits worth the potential harm and costs?worth the potential harm and costs?

Applying appropriate criteriaApplying appropriate criteria

User Guides to Evidence-Based PracticeUser Guides to Evidence-Based Practice

1993-2000: Evidence based medicine working group: “User guides 1993-2000: Evidence based medicine working group: “User guides to the Medical Literature” in JAMAto the Medical Literature” in JAMA

2002: Guyatt et al. “User guides to the medical literature: A manual 2002: Guyatt et al. “User guides to the medical literature: A manual for evidence based clinical practice”for evidence based clinical practice”

2005: DiCenso, Guyatt, & Ciliska. “Evidence-based nursing: A 2005: DiCenso, Guyatt, & Ciliska. “Evidence-based nursing: A guide to clinical practice”guide to clinical practice”

Applying appropriate criteriaApplying appropriate criteria

Critical Appraisal Forms:Critical Appraisal Forms: Critical Appraisal of Articles on Critical Appraisal of Articles on CAUSATIONCAUSATION Critical Appraisal of Articles on Critical Appraisal of Articles on

THERAPY/INTERVENTIONSTHERAPY/INTERVENTIONS Critical Appraisal of Critical Appraisal of SYSTEMATIC REVIEWSSYSTEMATIC REVIEWS Critical Appraisal of Articles on Critical Appraisal of Articles on PREVALENCE AND PREVALENCE AND

INCIDENCEINCIDENCE Critical Appraisal of Critical Appraisal of QUALITATIVE RESEARCHQUALITATIVE RESEARCH Critical Appraisal of Articles on Critical Appraisal of Articles on PROGNOSISPROGNOSIS

http://www.cche.net/usersguides/prognosis.asphttp://www.cche.net/usersguides/prognosis.asp Critical Appraisal of Critical Appraisal of GUIDELINES GUIDELINES [AGREE: Appraisal of [AGREE: Appraisal of

Guidelines Research and Evaluation Instrument] Guidelines Research and Evaluation Instrument] http://www.agreecollaboration.org/pdf/agreeinstrumentfinalhttp://www.agreecollaboration.org/pdf/agreeinstrumentfinal.pdf.pdf

Example: The Effectiveness of a Nurse-Led Example: The Effectiveness of a Nurse-Led Interprofessional Team Approach to Fall Prevention Interprofessional Team Approach to Fall Prevention

in Older Home Care Clients at Risk of Fallingin Older Home Care Clients at Risk of Falling

Markle-Reid, M., Browne, G., Gafni, A., Roberts, J., Weir, R., Markle-Reid, M., Browne, G., Gafni, A., Roberts, J., Weir, R., Thabane, L., Miles, M., Vaitonis, V., Hecimovich, C., Baxter, P., & Thabane, L., Miles, M., Vaitonis, V., Hecimovich, C., Baxter, P., & Henderson, S. (2010). The effects and costs of a multifactorial and Henderson, S. (2010). The effects and costs of a multifactorial and interdisciplinary team approach to fall prevention for older home interdisciplinary team approach to fall prevention for older home care clients “at risk” for falling: A randomized controlled trial. care clients “at risk” for falling: A randomized controlled trial. Canadian Journal on Aging, 29Canadian Journal on Aging, 29(1), 139-161(1), 139-161

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention StudiesScreening Questions:Screening Questions:

1.1. Did the study ask a clearly focused question? Did the study ask a clearly focused question?

Consider if the question is ‘focused’ in terms of:Consider if the question is ‘focused’ in terms of: The population studiedThe population studied The intervention givenThe intervention given The outcomes consideredThe outcomes considered

Research Question:Research Question:

`What is the effectiveness of a 6-month nurse-led `What is the effectiveness of a 6-month nurse-led interprofessional team approach to fall prevention compared interprofessional team approach to fall prevention compared with usual home care services in older home care clients “at with usual home care services in older home care clients “at risk” of falling with respect to the number of falls and fall risk” of falling with respect to the number of falls and fall risk factors (slip or trip, health-related quality of life and risk factors (slip or trip, health-related quality of life and function, depressive symptoms, nutritional status, gait and function, depressive symptoms, nutritional status, gait and balance, cognitive function, fear of falling)? balance, cognitive function, fear of falling)?

PP Frail older adults using home care services at risk for Frail older adults using home care services at risk for fallingfalling

II Nurse-led interprofessional team approach to fall Nurse-led interprofessional team approach to fall preventionprevention

CC Usual home care servicesUsual home care services OO Number of falls, slip or trip, Number of falls, slip or trip, health-related quality health-related quality

of life and function, depressive symptoms, nutritional of life and function, depressive symptoms, nutritional status, gait and balance, cognitive function, fear of falling status, gait and balance, cognitive function, fear of falling

TT 6 months6 months

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies

Screening Questions:Screening Questions:2.2. Was this a randomized controlled trial (RCT) and was it Was this a randomized controlled trial (RCT) and was it

appropriately so?appropriately so?

Consider:Consider: Why this study was carried out as an RCTWhy this study was carried out as an RCT If this was the right research approach for the question If this was the right research approach for the question

being askedbeing asked

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies

Is it worth continuing? Is it worth continuing?

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies3.3. How were participants allocated to How were participants allocated to

intervention and control groups? intervention and control groups?

Consider:Consider: How participants were allocated to intervention and How participants were allocated to intervention and

control groups. Was the process truly random?control groups. Was the process truly random? Whether the method of allocation was describedWhether the method of allocation was described How the randomization schedule was generatedHow the randomization schedule was generated Were the groups comparable at baseline on Were the groups comparable at baseline on

characteristics that might explain the outcome? characteristics that might explain the outcome?

Population Population

Eligible SampleBaseline MeasuresEligible Sample

Baseline Measures

IneligibleIneligible

R n=

Treatment Control

Follow-up Measures Follow-up Measures

StrengthsStrengths

Random allocation of participants to Random allocation of participants to groups ensures that groups are similar in groups ensures that groups are similar in all respects except exposure to the all respects except exposure to the outcome; outcome;

Prospective design ensures that exposure Prospective design ensures that exposure to the intervention precedes the to the intervention precedes the development of the outcome;development of the outcome;

Greater likelihood that participants, health Greater likelihood that participants, health care providers, and outcome assessors care providers, and outcome assessors can be blinded. can be blinded.

LimitationsLimitations

May not be ethical or feasible;May not be ethical or feasible; Volunteer bias limits generalizability; Volunteer bias limits generalizability; Problems associated with a Problems associated with a

longitudinal design:longitudinal design: TimeTime ExpenseExpense DropoutsDropouts

Final results may not be available for Final results may not be available for several years. several years.

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies4.4. Were participants, staff and study Were participants, staff and study

personnel ‘blind’ to participants’ study personnel ‘blind’ to participants’ study group?group?

Consider:Consider: The fact that blinding is not always possibleThe fact that blinding is not always possible If every effort was made to achieve blindingIf every effort was made to achieve blinding If you think this matters to the studyIf you think this matters to the study The fact that we are looking for ‘observer The fact that we are looking for ‘observer

bias’bias’

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies5.5. Was follow-up complete? Was follow-up complete?

Consider:Consider: If any intervention-group participants got a control If any intervention-group participants got a control

group option or vice versagroup option or vice versa If all participants were followed up in each study groupIf all participants were followed up in each study group Was there loss to follow-up bias?Was there loss to follow-up bias? If all of the participants’ outcomes were analysed by the If all of the participants’ outcomes were analysed by the

groups to which they were originally allocated (intention groups to which they were originally allocated (intention to treat analysis)to treat analysis)

Study FlowStudy Flow

R

Eligible Clientsn=267

Informed consent from 40.8% of eligible applicants

n=109

Refused 141Unable to Contact 13

Language 4158

Group 1n=54

Group 2 n=55

n=5 (9.3%)

n=49 (90.7%)

Interprofessional Falls Prevention Team

Usual Home Care Services

n=12 (21.8%)

n=43 (78.2%)

Group Size

Interventions

Dropouts @ 6 month follow-up: 17 clients (15.6%)

Analyzed @ 6 month follow-up: 92 clients (84.4%)

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies6.6. Were participants in both groups followed up and data Were participants in both groups followed up and data

collected in the same way?collected in the same way?

Consider:Consider: Was data collected in the same way and at the same Was data collected in the same way and at the same

time interval for both groups? time interval for both groups?

Data Collection Data Collection

Multiple sources of data:Multiple sources of data: In-person interviewsIn-person interviews CCAC dataCCAC data RAI-HC dataRAI-HC data Data from service provider agenciesData from service provider agencies

Measurement of Clinical Outcomes: Measurement of Clinical Outcomes: Baseline and 6-monthsBaseline and 6-months

OutcomesOutcomesEffectsEffects::Frequency and Context of Falls (Falls Surveillance Report)Frequency and Context of Falls (Falls Surveillance Report)

Functional Health Status and Quality of Life (SF-36 Health Functional Health Status and Quality of Life (SF-36 Health Survey)Survey)

Cognitive Status (SMMSE)Cognitive Status (SMMSE)

Depressive Symptoms (CES-D)Depressive Symptoms (CES-D)

Gait and Balance (POMA)Gait and Balance (POMA)

Nutritional Status (SCREEN II)Nutritional Status (SCREEN II)

Environmental Safety (HOME FAST) Environmental Safety (HOME FAST)

Perceived Self-Efficacy (MFES)Perceived Self-Efficacy (MFES)

CostsCosts::Health Services Utilization (HSSU and CCAC Utilization Health Services Utilization (HSSU and CCAC Utilization Data) Data)

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies7.7. How are the results presented and what is the main How are the results presented and what is the main

result?result?

Consider:Consider: What are the results?What are the results? How large this size of result and how meaningful it isHow large this size of result and how meaningful it is How you would sum up the bottom-line result of the trial How you would sum up the bottom-line result of the trial

in a few sentencesin a few sentences

ResultsResults At 6 months, there was noAt 6 months, there was no difference in the mean difference in the mean

number of falls between groups;number of falls between groups;

Subgroup analyses showed that the intervention Subgroup analyses showed that the intervention was effective in reducing falls was effective in reducing falls in menin men, 75-84 , 75-84 years years of age, with a fear of falling, or a negative history of age, with a fear of falling, or a negative history of fallsof falls;;

Greater reduction in number of slips and trips in Greater reduction in number of slips and trips in the intervention group; the intervention group;

Greater improvement in role functioning related to Greater improvement in role functioning related to emotional health in the intervention group; emotional health in the intervention group;

No additional cost from a societal perspective, e.g. No additional cost from a societal perspective, e.g. both interventions cost the same. both interventions cost the same.

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies8.8. How precise are these results?How precise are these results?

Consider:Consider: If the result is precise enough to make a decisionIf the result is precise enough to make a decision If a confidence interval were reported. Would your If a confidence interval were reported. Would your

decision about whether or not to use this intervention decision about whether or not to use this intervention be the same at the upper confidence limit as at the be the same at the upper confidence limit as at the lower confidence limit?lower confidence limit?

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies9.9. Were all important outcomes considered so the results Were all important outcomes considered so the results

can be applied?can be applied?

Consider whether:Consider whether: How the participants could be different from your population in How the participants could be different from your population in

ways that would produce different resultsways that would produce different results Your local setting differs from that of the trialYour local setting differs from that of the trial Is the intervention feasible in your setting?Is the intervention feasible in your setting?

Consider outcomes from the point of view of the:Consider outcomes from the point of view of the: IndividualIndividual Policy maker and professionalPolicy maker and professional Family caregiversFamily caregivers Wider community Wider community

Critical Appraisal Criteria Critical Appraisal Criteria for Health Care for Health Care Intervention StudiesIntervention Studies9.9. Were all important outcomes considered so the results Were all important outcomes considered so the results

can be applied?can be applied?

Consider whether:Consider whether: Any benefit reported outweighs any harm and/or costAny benefit reported outweighs any harm and/or cost Policy or practice should change as a result of the Policy or practice should change as a result of the

results of this trialresults of this trial

Methodological Strengths

Study design: randomized controlled trialStudy design: randomized controlled trial Follow-up period was appropriateFollow-up period was appropriate Follow-up was complete (<20% loss to Follow-up was complete (<20% loss to

follow-up)follow-up) Groups comparable at baselineGroups comparable at baseline

Methodological Limitations

Small sample sizeSmall sample size

Volunteer biasVolunteer bias

Potential for recall bias related to reporting fallsPotential for recall bias related to reporting falls

Strategies for collecting data regarding the number of Strategies for collecting data regarding the number of falls falls

Other Resources Evidence-Informed Decision-Making Resource Modules posted on Avenue to Learn Bandolier Evidence-Based Health Care http://www.medicine.ox.ac.uk/bandolier/ Tutorials:

Introduction to Evidence-Based Medicine (Duke University Medical Center Library) http://www.hsl.unc.edu/services/tutorials/ebm/index.htm

Directories: Evidence Based Medicine (EBM) Tookit (New York University School of Medicine.

The Frederick L. Ehrman Medical Library): www.urmc.rochester.edu/hslt/miner/digital_library/evidence_based_resources.cfm Resource Guide for Evidence-Based Practice (University of Alberta Libraries)

http://www.library.ualberta.ca/subject/evidence/guide/index.cfm McKibbon, A. (1999). PDQ: Evidence-based principles and practice. Hamilton, Ontario:

B.C. Decker Inc. Evidence Based Practice Tips http://www.ebmtips.net/risk001.asp