Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
26th November 2012
Critical Paper Review
Dr Dave Brodbelt MA VetMB PhD DVA DipECVAA MRCVS Senior Lecturer, Companion Animal Epidemiology VEPH Group, Royal Veterinary College, London [email protected]
Learning Outcome
• Identify the key elements of a critical appraisal
• Distinguish validity from importance
• Evaluate limitations including
– Selection / allocation of patients
– Outcomes assessment
– Dealing with bias and confounding
– Study power
Review: Types of Study Design
Descriptive
Studies
Case Series Case Reports Surveys
Analytical
Studies
Intervention Observational
Cross
Sectional
Studies
Cohort
Studies
Case-
Control
Studies
Descriptive Studies
• Describe events
• Generate hypotheses for further evaluation
• Particularly valuable when little is known about the
condition
– E.g. Preliminary observations of a new disease
• Types of descriptive studies
– Case reports
– Case series
– Surveys
Analytical Studies
• Build on descriptive studies
• Explicit comparison between groups
• Test hypotheses
• Identify ‘determinants’ of ‘disease’ – risk factors
• Explore / identify causal relationships
• Types of analytical studies – Observational studies
– Intervention studies
Hierarchy of Evidence
Systematic reviews / Meta
analyses
Randomised Controlled
Trial
Cohort study
Case-control study
Cross sectional study
Case series
Single case reports
Editorials, Opinion
Increasing Strength
of Evidence
Weaker
Evidence
Critical Appraisal of Studies • Validity of the results – alternative explanation?
– Chance?
– Bias?
– Confounding?
• Importance of results
– What was the magnitude of the treatment effect?
– How precise are the estimates of treatment effects?
Bias – Systematic Error
• Selection bias
– Systematic error in the identification of the study
population
– Patients selected differ systematically from those not
selected
– Selection based on exposure / outcome depends on the
other
• E.g. allocation of dogs with worst ataxia to spinal
surgery and others to conservative treatment in a
non-randomised clinical trial
Bias – Systematic Error
• Information bias
– Information obtained differently between study
groups
• Systematically incorrect measurements
• Systematically incorrect recording of exposure
or outcome
– Recall Bias – patients in one group more likely to
remember exposure / risk factor
Bias – Systematic Error
• Information bias
– Observer Bias –
• Investigators assessing exposure do so with
differing accuracy for cases and controls
• E.g. interviewer bias – systematic differences
in soliciting, recording or interpreting
information from study participants
Potential confounders identified and adjusted for?
– Effect of extraneous variable wholly or partially accounts for apparent association between exposure and disease
– Health status as a confounder
• ‘Isoflurane associated with anaesthetic-death’
• Health status = confounder
Health status
Isoflurane Death
Confounders • Potential confounders identified and adjusted for
– Confounders need to be measured
– Adjust for
• Match for the confounder in the study design
• Adjust for in the analysis
– Stratify
– Multivariable regression
Importance of results
• What was the magnitude of the treatment
effect? – Greater likelihood of real effect
– More relevant to clinical practice
• How precise are the estimates of treatment
effects?
Importance of results • How precise are the estimates of treatment effects?
– The level of UNCERTAINTY in the measured results
– Confidence Intervals (CI) - 95% CI
• The range in which the true treatment effect is likely to
lie with a given degree of confidence
• “95% sure that true value in lies within the interval”
– 100 repeated samples taken from same
population → 95% of confidence intervals
calculated will include true population estimate
• Narrow CI means greater precision
Association and Causation
• Questions we should ask:
– For a study are the associations valid?
– Given the totality of evidence are the associations
causal?
• Studies often report a valid association of exposure
and outcome
• Association NOT necessarily causation
– Association – statistical dependence
– Cause – factor contributes to outcome
Association and Causation • Assess causation based on body of
evidence
– Type of study
– Strength of association
– Consistency with other work
– Specificity of association
– Temporal relationship
A Bradford Hill 1965
Association and Causation • Assess causation based on body of evidence
– Biological gradient / dose response
– Biological plausibility
– Coherence with known biology of the disease
– Supporting experimental evidence
– Analogy of a similar condition
AB Hill 1965
Appraisal Check List
• Study – original?
– There may be previous studies
– Does the study ADD to the literature?
• Who was the study about?
– How were subjects recruited?
– Who was included in the study?
– Who was excluded from the study?
Appraisal Check List
• Study Design – sensible?
– Objectives clear?
– Study design clear AND appropriate?
• What intervention or other process studied?
• Who compared to?
– What outcome measured?
Appraisal Check List
• Systemic bias avoided?
– Clinical trials – random allocation of subject?
• RCT
• Non-randomised clinical trials?
– Observational studies
• Case definitions
• Selection of control group
• Follow-up sufficiently long and complete
• Adjusted for confounders in the design
Appraisal Check List
• Was assessment blind?
– Assessment of outcome blind?
• Were preliminary statistical questions answered?
– Sample size based on pre-study power calculation?
– Study large enough?
– Sufficiently long follow-up?
Appraisal Check List
• Study Conduct
– Good response rate?
– Sufficient sample size?
– Losses to follow-up?
• Minimal
• Attempt to evaluate any major losses to follow-up
Appraisal Check List
• Analysis and presentation
– Statistical methods clear AND appropriate?
– Presentation of data appropriate?
– Sufficient analyses presented?
– Adjusting for confounders if appropriate?
– Precision / confidence intervals for main results?
• Overall assessment
– Limitations acknowledged?
– Conclusions justified?
‘Controlled trial of behavioural therapy for separation-related disorders in dogs’
• Vet Rec (2006) 158, 551-554
• Method – 55 dogs assigned to 3 groups – 2 treatment, 1 control
– Underwent behavioural therapy, or no treatment
– Outcome
• Owner assessed change in behaviour
• Video footage for duration of 6 behaviour types
• Study design?
• Appraisal?
‘Controlled trial of behavioural therapy for separation-related disorders in dogs’
• VR (2006) 158, 551-554
• Method – 55 dogs assigned to 3 groups – 2 treatment, 1 control
• Study design? Non-randomised trial
• Appraisal? – Pre-study power?
– Random allocation?(no)
– Clear outcome defined and assessed?
– Owner and investigator assessments blinded?
Cystic ovarian disease Comparison of synchronization of ovulation with timed insemination and exogenous
progesterone
The objective of this study was to compare the effectiveness of the Ovsynch and
controlled internal drug releasing (CIDR) protocols under commercial conditions
for the treatment of cystic ovarian disease in dairy cattle. A total of 401 lactating
dairy cows with ovarian cysts were alternatively allocated to two treatment
groups on the day of diagnosis. Cows in the Ovsynch group were treated with
GnRH on Day 0, PGF2alpha on Day 7, GnRH on Day 9,
• Study design?
• Limitations?
Cystic ovarian disease Comparison of synchronization of ovulation with timed insemination and exogenous
progesterone
The objective of this study was to compare the effectiveness of the Ovsynch and
controlled internal drug releasing (CIDR) protocols under commercial conditions
for the treatment of cystic ovarian disease in dairy cattle. A total of 401 lactating
dairy cows with ovarian cysts were alternatively allocated to two treatment
groups on the day of diagnosis. Cows in the Ovsynch group were treated with
GnRH on Day 0, PGF2alpha on Day 7, GnRH on Day 9, • Study design? Non-random allocation - alternate
• Limitations
• Sampling method? How selected cows?
• Clear outcome?
• Sufficiently long follow-up?
• Median survival?
• Limitations?
Outcomes over time • Percent survival at a particular time
– 1 or 2 year survival
• Median survival time
– Time at which 50% of patients have developed the outcome
• Survival curves
– Each point on the curve
represents the proportion of
patients that have
NOT developed the outcome
– Probability of surviving
– Kaplan Meier curve
• Appraisal
• Study group – all similar at start of study?
• Clear outcome?
• Blindly assessed?
• Follow-up length and completeness?
• Adjusted for co-treatments?
‘Improved survival in a retrospective cohort of 28 dogs with insulinoma’
• JSAP (2007) 48, 151-156
• Method
– 28 dogs identified with insulinoma
– Record clinical signs, lab results, clinical staging
– Outcome – survival time, complications
• Study design?
• Appraisal?
‘Improved survival in a retrospective cohort of 28 dogs with insulinoma’
• JSAP (2007) 48, 151-156
– 28 dogs identified with insulinoma
– Outcome – survival time, complications
• Study Design - Cohort
• Appraisal?
– Representative sample?
– Clear definition of exposure / risk factors?
– Objective outcome applied blindly?
– Follow-up sufficiently long and complete?
– Adjust for confounders?
Cohort Study • Group of subjects with a common defined characteristic, i.e.
exposure / risk factor status
• Prospective
– Study starts BEFORE outcome of interest occurred
– Preferred – temporal sequence ensured, data required is recorded
• Retrospective
– Study begins AFTER
outcome occurred
– Relies on
pre-existing records
often not recorded
with study in mind
OUTCOMEEXPOSURE
Yes
No
Yes
No
Time
PROSPECTIVE
starts study here
RETROSPECTIVE
starts study here
‘Risk factors for anaesthetic death in Rabbits’
• D Brodbelt, PhD thesis (2006)
• Method – Study animals
• 70 deaths, peri-operative
• 70 non-deaths – anaesthetised before or soon after each death at the same centre
– Record patient, procedure, anaesthetic management, personnel and recovery details
– Outcome – anaesthetic related death
• Study design?
• Limitations?
• Precision of estimates?
‘Risk factors for anaesthetic death in Rabbits’ • D Brodbelt, PhD thesis (2006)
– CASE – CONTROL STUDY
• 70 deaths = CASES
• 70 non-deaths requested – before or soon after each death at the same centre = MATCHED CONTROLS
– Record patient, procedure, anaesthetic management, personnel and recovery details
– Outcome – anaesthetic related death
• Limitations? – Study size?
– Selection of controls?
– Clear outcome?
– Adjusted for confounders?
• Precision of estimates?
Case-Control Study
• Identify patients with disease (Cases) then
compare to non-diseased patients (Controls)
• Look back at their exposures / risk factors
• Identify risk factors for disease
• Well suited for RARE diseases
OUTCOMEEXPOSURE
Yes
No
TIME
Study starts
here
Yes / Case
No / Control
Rabbit Multivariable
Model Risk Factor OR (95% CI) LRT
P value
Sick/Healthy
(ASA 3-5/1-2)
28.7 (2.9 – 283.7) 0.001
Duration intended
30 min plus /<30 min
4.5 (0.9-21.5) 0.034
Major /Minor Procedures 5.8 (0.9 – 39.2) 0.048
Vet: Very Familiar / Familiar or
Unfamiliar
0.14 (0.02-0.96) 0.025
What is likely to be the major weakness of this study?
‘…Investigators randomly allocated 20 patients to two
analgesic groups … postoperative analgesia was
scored.
Results. No difference detected between analgesics.
Conclusions. The new analgesic was as good as the
current standard…’
1. One – non-random allocation
2. Two – investigators were not
blinded to groups
3. Three – conclusion over-stated
Conclusions
• Study appraisal – consider main elements – to
assess validity and importance
• Major issues
– Original?
– Whom does it concern? Generalisability?
– Study design sensible?
– Systematic bias avoided / minimised?
– Assessments undertaken blindly?
– Assess preliminary statistical questions?
Further Reading
• Cockcroft and Holmes (2003) Handbook of Evidence-Based Veterinary Medicine
• Cockcroft and Holmes (2004) EBVM 1-3 In Practice
• Dohoo, I., W. Martin, et al. (2009). Veterinary Epidemiologic Research. Charlottetown, AVC Inc. 2nd edn
• Greenhalgh, T (1997) How to read a paper: Assessing the methodological quality of published papers. BMJ, 7103, 315
• Greenhalgh T (2010) How to read a paper, the basics of evidence based medicine. 4th edn. BMJ Books.
• Pfeiffer (2010) Veterinary Epidemiology: An Introduction
• Straus et al (2005) Evidence-Based Medicine, How to Practice and Teach EBM. London, Elsevier.