Journal club 2015 07-15 tom

Brighton and Sussex NHS LKS is part of Brighton and Sussex University Hospitals NHS Trust

Librarian co-authors correlated with higher quality reported search strategies in generalinternal medicine systematic

reviews

Tom RoperJournal Club, 15 July 2015

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in generalinternal medicine systematic reviews. J Clin Epidemiol. 2015 Jun;68(6):617-26. doi: 10.1016/j.jclinepi.2014.11.025

Systematic review reporting standards • 1999: QUORUM (QUality Of Reporting Of Meta-

analyses)• 2000: MOOSE (Meta-analysis Of Observational

Studies in Epidemiology)• 2009:PRISMA (Preferred Reporting Items of

Systematic reviews and Meta-Analyses)• 2007: AMSTAR (Assessment of Multiple Systematic

Reviews)• 2011: IOM Finding What Works in Health Care• 1996 onwards: Cochrane Handbook Chapter 6 (last

revision 2011)• 2012: Methodological Expectations of Cochrane

Intervention Reviews standards (MECIR)

What did the authors want to do?• SR searches poorly reported, if at all, and

rarely reproducible

• The research question: The purpose of this study was to determine whether LIS authorship was associated with better SR search quality and reporting

How did they do it?

• Took highest impact journals publishing SRs: Annals of Internal Medicine, BMJ, JAMA, Lancet, and PLoS Medicine (not NEJM)

• All SRs published between 2008 and 2012 identified:

• (search*[tiab] OR meta-analysis[Publication Type] OR meta-analysis[tiab] OR MEDLINE[tiab] OR (systematic[tiab] AND review[tiab])) OR systematic [sb]) AND (‘‘Lancet’’[Journal] OR ‘‘JAMA’’[Jour- nal] OR ‘‘Ann Intern Med’’[Journal] OR ‘‘PLoS Med’’[Journal] OR ‘‘BMJ’’[Journal])

One of three reviewers scrutinised papers Criteria: – article states it is a SR in ti or ab OR– article states it is a meta-analysis and methods

include literature search OR– methods include reported extensive search of the

literature and list inclusion and exclusion criteria

Short form to record LIS involvement– no or unclear LIS participation– mentioned or acknowledged LIS– LIS author

• Reproducibility: – At least one complete strategy including Boolean

combinations

• Checklist based on the 15 IOM Recommended Standards for the Search Process and

• Modified version of the Peer Review of Electronic Search Strategies (PRESS) instrument

• Blinding: strategies stripped and loaded into a Google doc

Flow diagram. IOM, Institute of Medicine; PRESS, Peer Review of Electronic Search Strategies.

Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviewsRethlefsen, Melissa L., Journal of Clinical Epidemiology, Volume 68, Issue 6, 617-626

Copyright © 2015 Elsevier Inc.

Results

Results

Systematic reviews and searches

None or unclear

Mentioned or acknowledged

Author

Number of systematic reviews (%)

438 (69.5) 147 (23.3) 45 (7.1)

Number of reproducible searches (% of searches reproducible)

164 (37.4) 82 (55.8) 29 (64.4)

Analysis of means for proportions of search reproducibility by levels of librarians and information specialists (LIS) involvement. Search reproducibility proportions significantly differ from the mean proportion of reproducibility in all three levels of LIS participation. Where the incident rate lines cross the decision limits, significant difference from the mean is demonstrated.



• Level of LIS participation significantly associated with search reproducibilty (P < 0.0001)

Box plot of Peer Review of Electronic Search Strategies score by level of librarians and information specialists involvement.



• Overall, LIS co-authored SRs had significantly higher odds of meeting 8 of the 13 analyzed IOM Standards than SRs with no LIS

Discussion

• LIS co-authored SRs– much higher quality, both in terms of the search strategy

itself and search strategy reporting, than SRs with a mentioned LIS

• more likely to hand-search• search for non-English studies• search citation indexes• subject databases• regional databases• gray [sic] literature

– better use of natural language terms – better translation of question into search strategy

• one negative correlation: updating (IOM 3.1.7): update the search at intervals appropriate to the pace of generation of new information for the research question being addressed

• a large number of SRs are published without a strategy, much less a reproducible one.

• notable number of PubMed search strategies only use keywords with no truncation or field codes — completely eliminates reproducibility

• peer reviewers and journal editors don’t review published search strategies carefully

Recommendations

• Search strategy peer review• Narrative explanations of search choices and

construction • Journal editors to consider having search strategies

reviewed for SRs • Editors to suggest SR authors receive and

acknowledge LIS help with SR searches and use the presence of LIS involvement as quality criterion

• NB Five previously reproducible search strategies disappeared from publisher Web sites and other locations during study

Limitations

• only fve journals

• what’s an SR?

• what’s a librarian?

• no assessment of precision and recall of SR strategies

• disagreement between IOM and AHRQ on standards:– AHRQ said no evidence for LIS involvement in

review of search strategy

Why does this matter?

• How many SRs do we do? Search for ‘systematic review’ in title of evidence search in KS = 17 hits

• Can we extrapolate from SR searches to other searches?

See also…

Koffel JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS One. 2015 May 4;10(5):e0125931. doi: 10.1371/journal.pone.0125931.

Health & Medicine

Journal club 2015 07-15 tom