Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
MRC/CSO Social and Public Health Sciences Unit
Different types of review and applying the principles of systematic review
Matt Egan
MRC/CSO Social and Public Health Sciences Unit
MRC|CSO Social and Public Health Sciences Unit
Outline
• Different types of review & systematic review
• Challenges for systematic reviews
• Applying systematic review principles to non-systematic reviews
MRC|CSO Social and Public Health Sciences Unit
Different types of review
• Systematic reviews of
• Intervention outcomes
• Intervention process
• Epidemiology
• Qualitative research
• Methods
• Rapid Review
• Scoping review
• Traditional review
•Conceptual/theoretical review •Realist review •Critical review •Expert review •“State of the art” review •Review of reviews
MRC|CSO Social and Public Health Sciences Unit
Not clear cut definitions
• Conceptual/theoretical review •Review of ideas, models, debates
• Realist review • Review of studies to identify generalisable theories
• Critical review
• Assess a theory by a less formalised critical assessment of studies
• Expert review • Review by acknowledged expert(s)
• “State of the art” review • Recent literature to bring readers up to date about research on specific subject
MRC|CSO Social and Public Health Sciences Unit
Use of “systematic review” label
• „Traditional review‟ is a term used by systematic reviewers to describe the kind of reviews they do not do
• Use of multiple labels can be unnecessarily complicated
• Don‟t waste time getting confused by “clever” labels
• Essential items of systematic review are: • Clear question
• Record of search sources & strategy
• Clear inclusion/exclusion criteria
• Assessment of study quality
• Description of synthesis methods: meta-analysis or narrative
• Any review type can use above principles of systematic review, e.g. expert review may have clear question etc
MRC|CSO Social and Public Health Sciences Unit
Most “systematic reviews” aren‟t systematic
• The validity of 480 systematic review and meta-analyses using a simple scale was assessed
• Thorough search
• Appraisal of study quality
• Assessment of heterogeneity
• 26% met all three the criteria
Petticrew M, Song F, Wilson P, Wright K International Journal of Technology Assessment in Health Care 2000; 15(4)671-8.
MRC|CSO Social and Public Health Sciences Unit
Marks of a good systematic review
• Essential characteristics- transparent & replicable
• Clear & focussed review question
• Search strategy described
• Comprehensive searching of all included sources
• Two reviewers to select included studies
• Two reviewers to assess quality of included studies
• Justification for method of synthesis- meta-analysis or narrative
• Synthesis reflects quality of included studies- less weight given to studies with high levels of bias
• Results & characteristics of individual studies to accompany final conclusions of synthesis- tables, forest plots etc
MRC|CSO Social and Public Health Sciences Unit
Making non-systematic reviews more systematic
Steps & components of systematic review
• Define question
• Write protocol
• Develop & conduct searches
• Screen for eligible studies according to clear incl/excl criteria
• Assess study quality
• Extract findings
• Synthesise data taking into consideration study quality
• Clear links between data and conclusions- good tables
Including any of these
stages will make a review „more systematic‟:
The more the better!
MRC|CSO Social and Public Health Sciences Unit
Different types of questions & data for systematic reviews
MRC|CSO Social and Public Health Sciences Unit
Policy relevant questions
• How, why and when do Community Based Disaster Risk Management initiatives impact on social and economic costs of disasters? Too many questions
• Are interventions to reduce the impact of arsenic contamination of groundwater on human health and crop productivity in developing countries effective? Effective to achieve what...
• Does private involvement (including public private partnerships) in delivery of water, information and communication technology and electricity services lead to improved access and quality of service in developing countries? Intervention not defined- very broad
Does communication for development contribute to the diffusion of information that results in behaviour change around key social issues such as HIV, domestic violence etc? Intervention not defined- very broad
MRC|CSO Social and Public Health Sciences Unit
What kind of questions can a systematic review address?
• What works?
• How does it work?
• Will it do more harm than good?
• Is it worth the money?
• Acceptability and appropriateness to certain groups
• Is there an association between A & B?
• How strong it the association?
• What affects the association?
• How was intervention implemented?
MRC|CSO Social and Public Health Sciences Unit
What type of research evidence can be included?
MRC|CSO Social and Public Health Sciences Unit
Methodological aptness: choose data that can answer the question Research question Qualitative
Research Surveys Case control
studies Cohort Studies
RCTs Sys Reviews
Effectiveness Does this work?
+ ++
+++
Effectiveness of service delivery: How does it work ?
++
+
+++
Salience Does it matter ?
++
++
+++
Safety: Will it do more harm than good?
+ + + ++ +++
Acceptability Will children/parents want to use it?
++
+
+
+++
Cost-effectiveness: Is it worth buying?
++ +++
Satisfaction with service
++ ++ + + +
Appropriateness Is this the right service
++
++
++
(Muir Gray, 1997 & Petticrew, 2003)
MRC|CSO Social and Public Health Sciences Unit
Challenges for systematic reviews
• Large reviews can take a long time and are expensive to conduct
• This can be harmful...
• Requires considerable expertise
• This is a particular problem for reviews of complex social interventions
• Difficult to identify literature
• Requires broad search strategies
• Includes diverse study designs, interventions & outcomes etc
• Multiple relevant questions- not just what works?
MRC|CSO Social and Public Health Sciences Unit
Delaying the publication of findings can cause harm
MRC|CSO Social and Public Health Sciences Unit
Rapid reviews
• Large team of reviewers to conduct large review more quickly • Still costly
• Narrow focus of review to conduct review more quickly
• Limit review criteria
• Sources
• PICOCS
• Assessment of study quality • Advantage: quicker & cheaper
• Disadvantage: Limited review criteria
• Sometimes this is the most appropriate use of resources
– Can be useful to establish nature of available research for broad policy questions
MRC|CSO Social and Public Health Sciences Unit
Rapid reviews: maintain systematic principles
• Important to maintain transparent methods
• Keep record of selection of included studies with justification
• Consider study quality in the synthesis
• Is the included evidence strong enough to answer the questions?
• Consider how the limits of the review might affect your conclusions?
• E.g. Only included data from Sri Lanka, only included RCTs, only included since 2000 etc etc
MRC|CSO Social and Public Health Sciences Unit
Making non-systematic reviews more systematic: use principles of systematic review
Steps & components of systematic review
• Define question
• Write protocol
• Develop & conduct searches
• Screen for eligible studies according to clear incl/excl criteria
• Assess study quality
• Extract findings
• Synthesise data taking into consideration study quality
• Clear links between data and conclusions- good tables
Including any of these
stages will make a review „more systematic‟:
The more the better!
MRC|CSO Social and Public Health Sciences Unit
MRC|CSO Social and Public Health Sciences Unit
Making non-systematic reviews more systematic
Example 2: theory review
Why do one?
• Bigger picture: whilst empirical research often focuses on single interventions or risk factors, theory reviews identify the underlying assumptions across a range of research.
• Understanding theories underpinning interventions can aid transferablity
Tips
• Manual searches likely to be particularly important.
• Defining a „theory‟ is tricky. Input mechanism outcome
Drawback
• No systematic way of critically appraising theories
• Not always obvious what to conclude or recommend
MRC|CSO Social and Public Health Sciences Unit
Example of theory review
MRC|CSO Social and Public Health Sciences Unit
MRC|CSO Social and Public Health Sciences Unit
Making non-systematic reviews more systematic
Example 4: scoping reviews
Why do one?
• Quick (rapid review)
• Prepare the ground for a more systematic review
• If the field is under researched you can use a broad question to produce a review that is both systematic and scoping
Tips
• Not too broad: eg. Similar interventions, different outcomes;
• Focus on how to deal with heterogeneity: e.g. categorise findings.
• May need to restrict search to stop it becoming unmanageable.
Drawback
• Too big?
• Too heterogeneous?
MRC|CSO Social and Public Health Sciences Unit
Examples of scoping systematic reviews
New Roads
•Similar interventions •Different study designs •Different outcome measures (Egan et al, AJPH, 2003)
Modal Shift
•Different interventions •Different study designs •Similar outcomes measures (Ogilvie et al, BMJ, 2004)
MRC|CSO Social and Public Health Sciences Unit
Examples of scoping systematic reviews
New Roads
•Similar interventions •Different study designs •Different outcome measures (Egan et al, AJPH, 2003)
Modal Shift
•Different interventions •Different study designs •Similar outcomes measures (Ogilvie et al, BMJ, 2004)
MRC|CSO Social and Public Health Sciences Unit
Different types of review
• Systematic reviews of
• Intervention outcomes
• Intervention process
• Epidemiology
• Qualitative research
• Methods
• Rapid Review
• Scoping review
• Traditional review
•Conceptual/theoretical review •Realist review •Critical review •Expert review •“State of the art” review •Traditional review •Review of (systematic) reviews
MRC|CSO Social and Public Health Sciences Unit
Why do reviewers care about implementation process, mechanisms, context, etc?
• So we can learn how to improve intervention delivery
• To help us assess the extent to which findings are generalisable and transferable (relationship between context, implementation and outcomes).
• So we can distinguish between theory failure and implementation failure.
MRC|CSO Social and Public Health Sciences Unit
Including process, context and mechanisms in systematic reviews of complex interventions
MRC|CSO Social and Public Health Sciences Unit
Including process, context and mechanisms in systematic reviews of complex interventions
Theme Does the study describe….. (yes/no)
1. Motivation
Why the management decided to do the intervention? Was concern about stress/health a factor?
2. Theory of change
A theory of change describing pathway from implementation to health outcome?
3. Implementation context
Political, economic, managerial or other contextual factors relevant to implementation?
4. Experience
Whether those implementing the intervention had appropriate experience/training to implement intervention?
5. Planning consultations
Consultation between managers, employees and any other relevant parties during planning stage?
MRC|CSO Social and Public Health Sciences Unit
Including process, context and mechanisms in systematic reviews of complex interventions
Theme Does the study describe….. (yes/no)
6. Delivery collaborations
Consultation between managers, employees and any other relevant parties during delivery stage?
7. Manager support
Were on-site managers/supervisors supportive of the intervention?
8. Employee support
Were employees supportive of the intervention?
9. Resources
resources required in implementing the intervention (eg time, money, people, equipment)?
10. Differential effects Who benefited and who didn't?
MRC|CSO Social and Public Health Sciences Unit
Specific problems with our checklist
• Ignorance: No well established hierarchy for implementation quality. We didn‟t know what a well implemented workplace reorganisation intervention looked like.
• Uninformative: unclear if we are reviewing implementation or reporting of implementation.
• Simplistic: not clear which implementation issues are most important. So there‟s a danger of implying (falsely) that they are of equal weight
• There wasn’t much to appraise.
(average checklist score was 3 out of 10).
MRC|CSO Social and Public Health Sciences Unit
Problem not just about quantity of reporting – also quality
• General reliance on anecdotal evidence rather than a planned study of implementation.
• Sometimes appeared to privilege one view point – e.g. either managers or employees.
• Few studies discussed how implementation influenced outcomes.
• Quality of implementation reporting did not vary much by type or date of publication, or methodological quality, or health outcome.
MRC|CSO Social and Public Health Sciences Unit
Qualitative version of the checklist (no score)
MRC|CSO Social and Public Health Sciences Unit
This more qualitative approach produced more useful findings
• Context: Participation interventions don‟t work in downsizing organisations. Task restructuring interventions could be harmful when driven by financial rather than health considerations.
• Mechanism(?): Findings often supportive of psychosocial „demand, control, support‟ models explaining workplace health – especially control.
• Differential effects: Some evidence that the positive and negative health effects of organisational changes to the psychosocial work environment are felt more by men and lower socio-economic groups.
MRC|CSO Social and Public Health Sciences Unit
Conclusion Do you like your water clear or muddy?
A purist approach to systematic reviews leads to high quality, comprehensive reviews that reduce the risk of erroneous conclusions.
A less pure approach may have advantages related to speed, efficiency, pragmatism, innovation and ability to review a wider range of topics.