Presentation s rs

Systematic ReviewsWhat are they, and how do you

conduct one?

Julia MuellerSchool of Nursing, Midwifery and Social Work

School of Computer Science

2

Structure1) Definition 2) The Cochrane Collaboration3) Review protocol 4) Conducting the systematic review5) Conclusions

3

Definition

4

Definition

The Cochrane Collaboration

The Review protocol

Conducting the systematic review

Interpreting results

Conclusions

What are systematic reviews?

Cook et al., 1997

Systematic Review

• A literature reviewing technique typically used in evidence-based medicine, but applicable to other fields of study

• Gathering, critically appraising and summarising all relevant studies that address a certain research question

Basis for decision making

5

Definition


The Review protocol



Conclusions

Why do we need systematic reviews?

• The problem:

ww

w.b

vallc

.

com

- Too many papers, too little time

- Conflicting information and recommendations from different papers

Summaries of all literature on a given topic are needed so that the evidence is easily and quickly accessible

6

Definition


The Review protocol



Conclusions

Why is it important to be systematic?

– Concise and rigorous literature search– Explicit search strategy– Explicit inclusion and exclusion criteria– Explicit methods for extracting and

summarising data from primary studies

– Overall: COMPREHENSIVE, EXPLICIT, REPRODUCIBLE, RATIONAL

Ensures reproducibility,

reduces bias

7

Definition


The Review protocol



Conclusions

Systematic vs. traditional reviews

Traditional Systematic

Research question

Broad Specific

Search strategy, data collection and interpretation

Informal, subjective, not explicit

Explicit, clear, with rationale

Critical appraisal of included studies

May or may not take place, not using specific tools

Systematic using standardised tools

Cipriani and Geddes, 2003

8

Definition


The Review protocol



Conclusions

Different forms of literature reviews

Literature reviews

Systematic reviews

Traditional reviewsMeta-

analyses

9

Definition


The Review protocol



Conclusions

What is a meta-analysis?= “the use of statistical techniques to integrate and

summarise the results of included studies”

• Combining information from several studies larger sample size higher precision higher statistical power (= the probability that the statistical test will detect an effect that is really there)

Liberati et al., 2009

10

Definition


The Review protocol



Conclusions

What is a meta-analysis?• Whether you can conduct a meta-analysis depends

on:– No. of available studies– Variability of methods and outcome measures

• If a meta-analysis is not possible, a narrative/qualitative systematic review can be conducted

11

Definition


The Review protocol



Conclusions

Who can conduct a systematic review? • The review team should have skills in

– Systematic review methods– Information retrieval– The relevant topic area– Statistics– Qualitative research methods

• It is good practice to have at least 2 researchers involved minimise bias and error

12

13

Definition


The Review protocol



Conclusions

= an international network

• named after Archie Cochrane (1909-1988), a British epidemiologist, who advocated evidence-based medicine

• Aim: Provide resources to enable well-informed decisions about health care

• http://www.cochrane.org/

http://www.cochrane.org/

14

Definition


The Review protocol



Conclusions

Cochrane reviews

= Systematic reviews and meta-analyses published by the Cochrane Collaboration in the Cochrane Library

• Specific methods, structure and format: Cochrane Handbook for Systematic Reviews of Interventions

• “internationally recognised as the highest standard in evidence-based health care”

• Existing Cochrane Reviews are updated regularly as new information becomes available

Can result in changed conclusions

http://www.thecochranelibrary.com/view/0/index.html

http://handbook.cochrane.org/

http://handbook.cochrane.org/

15

Cochrane Reviews

over 5,000 Cochrane Reviews currently available in The Cochrane Library (November 2013)

16

Definition


The Review protocol



Conclusions

Cochrane reviews in Manchester• Cochrane groups with editorial base in Manchester

• Cochrane Oral Health Group, School of Dentistry, The University of Manchester

• The Cochrane Bone, Joint and Muscle Trauma Group (BJMTG)

17

The Review Protocol

18

Definition


The Review protocol



Conclusions

Review Protocol• Describes the methods to be used in the review

– Background– Review question– Inclusion/Exclusion criteria– Search strategy– Study selection– Quality assessment– Data extraction– Data synthesis– Plans for dissemination

• Specifying the methods in advance reduces the risk of bias• If modifications are required, they should be documented

and justified

19

Definition


The Review protocol



Conclusions

Background

• Key contextual factors relevant to the review question• Explain why the review is required

20

Definition


The Review protocol



Conclusions

Background: CS-related exampleExample Kitchenham et al. (2007): “A Systematic Review

of Cross- vs. Within-Company Cost Estimation Studies.”

• Two different types of cost estimation models• Within-company models developed using one company’s

data• Cross-company models developed using datasets from

several companies

• Several studies have compared prediction accuracies with contradictory results

• Important for small companies who do not have their own project data

Systematic review to determine factors that influence the outcome of studies comparing within and cross-company models

21

Definition


The Review protocol



Conclusions

Inclusion criteria• Language: The ideal is to include all available relevant

evidence, but this often not feasible Language bias

• Publication type/status– Full papers in peer-reviewed journals

“Publication bias”: When studies with significantresults are more likely to get published

– Contact authors– Ongoing studies partially published as conference abstracts– Reports and discussion papers– Book chapters– Theses– …

22

Definition


The Review protocol



Conclusions

Inclusion criteria• Use the “PICOS” elements:

Population: Patients or Users (disease, age, gender, disability), software, type of technology, websites …

Interventions: Type, main features …, e.g. assistive technology for disabled users…

Comparison: No treatment, treatment as usual; between or within organisations …

Outcomes: e.g. survival, disease remission, interactions with the keyboard, commands executed, …

Study design: Randomised controlled trials, cohort studies, case-control studies…

Use this to define your question

23

Definition


The Review protocol



Conclusions

Review question• Clear, as specific as possible

• Example 1: Are psychological interventions effective in reducing chronic headache in children and adolescents?

• Example 2: Are cross-company models significantly worse than within-company estimation models for predicting effort for software/web projects?

24

Definition


The Review protocol



Conclusions

Building the search strategy1. Describe all relevant PICOS elements (you might not

want to include all!)2. Organise them into concepts3. Find all related words and synonyms4. Search all synonyms for one concept with “OR”5. Combine all concepts with “AND”

25

Definition


The Review protocol



Conclusions

Search: Health-related exampleExample: Are psychological interventions effective inreducing chronic headache in children and adolescents?

Population1. Concept: “Children and adolescents” Child* OR adolescen* OR juvenile OR paediatric2. Concept: “Headache”Headache OR migraine OR tension headacheInterventions3. Concept: “Psychological interventions”psycholog* OR psychotherapy OR biofeedbackOR relaxation OR cognitive OR behavio#ral1. AND 2. AND 3.

26

Definition


The Review protocol



Conclusions

Search: CS-related example• Kitchenham et al. (2007). A Systematic Review of Cross- vs. Within-

Company Cost Estimation Studies.

1. Population: software OR application OR product OR Web OR WWW OR Internet OR World-Wide Web OR project OR development

2. Intervention: cross company OR cross organisation OR cross organization OR multiple-organizational OR multiple-organisational model OR modeling OR modelling effort OR cost OR resource estimation OR prediction OR assessment

3. Comparison: within-organisation OR within-organization OR within-organizational OR within-organisational OR single company OR single organisation

4. Outcome: Accuracy OR Mean Magnitude Relative Error

• 1. AND 2. AND 3. AND 4.

27

Conducting a systematic review

28

Definition


The Review protocol



Conclusions

Conducting a literature search

Search electronic databases, e.g. ACM Digital Library, CiteSeer, INSPEC, …

Visually scan reference lists from identified studies

Handsearch key journals and conference proceedings

Contact study authors, experts, manufacturers, other organisations …

…

29

Definition


The Review protocol



Conclusions

Conducting a literature search# records identified through

database screening# of records identified through other sources

# of records after duplicates removed

# of records screened (title, abstracts)

# of full-text articles assessed for eligibility

# of studies included in data synthesis

# of full-text articles excluded, with reasons

# of records excluded

iden

tifica

tion

scre

enin

gel

igib

ility

incl

uded

Liberati et al., 2009: The PRISMA statement for reporting systematic reviews

30

Definition


The Review protocol



Conclusions

Quality assessment• A review should be based on the best quality evidence

available• The quality of the included studies will impact on the

reliability of the results of the review

• Useful tools for quality assessment– CASP tools (Critical Appraisal Skills Programme), e.g.

CASP checklist for randomised controlled trials– JADAD scale– …

31

Definition


The Review protocol



Conclusions

Quality assessment: CSKitchenham et al. (2007)

• 6 questions to assess the quality of included studies, developed by the researchers themselves

• Examples:

– Is the analysis process description complete?– Is it clear how accuracy was measured?– Were all model construction methods fully defined

(tools and methods used)?

32

Definition


The Review protocol



Conclusions

Data extraction

• What information will be extracted from studies?• Data extraction sheet

33

Definition


The Review protocol



Conclusions

Data synthesis• If a meta-analysis is possible, data can be summed up

statistically using quantitative methods

• If not, data can be synthesised narratively

• Narrative data synthesis usually ocurrs in form of table(s) and textual description

34

Narrative data synthesis: Health-related example

Author, date and location

Participant demographics, attrition

Intervention length, content and groups

Measures and follow-up

Reported results

Author’s conclucions

Segatto etal., 2010.Brazil

N= 17590.3% maleAge: 21.8 (2.6)16 – 25 yearsMostly Caucasian(71%)Patients in emergency roomAttrition: 14,9% ,no differencesbtw. completersand dropoutsfound

Motivational Interview (MI): person-centredtechniques, also included EB (45 min.)Educational brochure (EB): 3 page informativegeneral guidance on risks of alcohol (5 min.)

Alcohol ConsumptionQuestionnaire (ACQ), RutgersAlcohol Problems Index(RAPI), Alcohol ConsumptionRisk Questionnaire (ACRQ),Measured at baseline and 3months

Time effect for: days of alcoholuse, days with moderate use, dayswith heavy use, negativeconsequences, being in actionstageNo sig. differences in long-termchange in risk-perceptionNo group x time effects

MI alone not able to promote sig. and lastingchangesLimitations: brief interventions usually targetrisk populations at an early stage, too shortobservation time

… … … … … …

35

Narrative data synthesis: CS-related example

36

Definition


The Review protocol



Conclusions

Interpreting results• Was there consensus across all studies?• Where did studies differ? What might be the reason?

Did these studies differ in participant populations, methods/measures used …?

• Did it make sense to pool the results? (heterogeneity/homogeneity)

• Discuss the quality of the included studies: Can the results of the review be valid?

• Discuss possible sources of bias: Publication bias, language bias…

37

Conclusions

38

Definition


The Review protocol



Conclusions

Conclusions

• Systematic Reviewing is designed to ensure a comprehensive, reproducible and critical review of all the available literature on a given topic

• Not all techniques shown here will be equally applicable to CS

• Other methods may be legitimate as long as they are stated explicitly and a rationale is provided

39

Thanks for listening. Any questions?

Julia Mueller

[email protected]://jnmueller.wordpress.com/

School of Computer Science,LF1 Kilburn Building,

Oxford Road, MANCHESTER,M13 9PL, UK

http://jnmueller.wordpress.com/

40

Definition


The Review protocol



Conclusions

References2009. Systematic Reviews: CRD's guidance for undertaking

reviews in health care. CRD, University of York.CIPRIANI, A. & GEDDES, J. 2003. Comparison of systematic

and narrative reviews: the example of the atypical antipsychotics. Epidemiol Psichiatr Soc, 12, 146-53.

COOK, D. J., MULROW, C. D. & HAYNES, R. B. 1997. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med, 126, 376-80.

LIBERATI, A., ALTMAN, D. G., TETZLAFF, J., MULROW, C., GOTZSCHE, P. C., IOANNIDIS, J. P., CLARKE, M., DEVEREAUX, P. J., KLEIJNEN, J. & MOHER, D. 2009. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ, 339, b2700.

41

Definition


The Review protocol



Conclusions

References• Here’s an example of a well-conducted systematic

review and meta-analysis:

COVENTRY, P. A. & HIND, D. 2007. Comprehensive pulmonary rehabilitation for anxiety and depression in adults with chronic obstructive pulmonary disease: Systematic review and meta-analysis. J Psychosom Res, 63, 551-65.

Cochrane Handbook for systematic reviews for interventions: http://handbook.cochrane.org/

42

Definition


The Review protocol



Conclusions

References• This is a systematic review in the area of CS:Kitchenham, B., Mendes, E., Travassos, G.H. (2007) A

Systematic Review of Cross- vs. Within-Company Cost Estimation Studies, IEEE Trans on SE, 33 (5), pp 316-329.

• Guidelines for performing Systematic Literature Reviews in Software Engineering

http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CEQQFjAC&url=http://cdn.elsevier.com/promis_misc/525444systematicreviewsguide.pdf&ei=Pz2fUtzjL-PN0QXXhoHoBw&usg=AFQjCNEYFGDiS8kGHzCpf8sbvkZZumuI9Q&bvm=bv.57155469,d.d2k

http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CEQQFjAC&url=http://cdn.elsevier.com/promis_misc/525444systematicreviewsguide.pdf&ei=Pz2fUtzjL-PN0QXXhoHoBw&usg=AFQjCNEYFGDiS8kGHzCpf8sbvkZZumuI9Q&bvm=bv.57155469,d.d2k