Effective Search Strategies for Systematic Reviews of Medical Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules

Effective Search Strategies for Systematic Reviews of Medical

TestsPrepared for:

The Agency for Healthcare Research and Quality (AHRQ)

Training Modules for Medical Test Reviews Methods Guide

www.ahrq.gov

Overview of a Medical Test Review

Prepare Topic

•Develop the Topic and Structure the Review

•Choose the Important Outcomes

Research Sources

Search for and Select Studies for Inclusion

•Search for Studies

Extract Data From Studies

Analyze and Synthesize Studies

•Assess Risk of Bias as a Domain of Quality

•Assess Applicability

•Grade the Body of Evidence

•Meta-analysis of Test Performance Evidence With a “Gold Standard” — or —

•Meta-analysis of Test Performance Evidence With an Imperfect Reference Standard

•Decision Modeling

Report Medical Test

ReviewRelevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

Recognize the importance of a multiple-approach method of searching

Understand the current state of indexing and reporting and its effect on the most pragmatic methods of conducting searches

Learning Objectives

Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

The overall goal is to locate all published studies relevant to key questions.

General guidance on searching for systematic reviews also applies to reviews of medical tests.

A balance must be achieved between recall and precision: Recall = how much of the relevant literature is retrieved. Precision = how much of the retrieved literature is relevant.

An optimal balance depends on the context. For comparative effectiveness reviews, the goal is to conduct a comprehensive and exhaustive search while minimizing the resources needed to review retrieved citations.

General Search Guidance (1 of 3)

Relevo R. Effective search strategies for systematic reviews of medical tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews.Available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

Develop the search strategy for the study with the research team according to the analytic framework and the inclusion and exclusion criteria.

Search electronic databases for applicable studies: MEDLINE and the Cochrane Central Register of Controlled

Trials should always be included in the search strategy. Other useful databases include EMBASE , CINAHL, and

PsycINFO. Tailor the search to each database. Use both controlled (i.e., MeSH subject headings) and

uncontrolled vocabulary. Use delimiters such as language only if a specific case can be

made.



Identifying additional studies: References from studies included for review Citation databases (e.g., SciVerse Scopus or Web of

Science ) Unpublished or “grey” literature, such as:

Clinical trial registries (e.g., ClinicalTrials.gov and the World Health Organization International Clinical Trials Registry Platform)

Conference proceedings/abstracts (e.g., ProQuest COS Conference Papers Index)



Searches are required on each relevant test strategy being considered.

One of two approaches is used to conduct the search: Search all possible tests used to evaluate the disease.

This requires knowledge of all possible test strategies available.

Search on the disease or condition, and then filter for the medical test.

Searching is often an iterative process, requiring refinements to search criteria.

Challenges Specific to Searches forStudies of Medical Tests (1 of 3)


The first approach for identifying studies of medical tests is to search for specific named tests.Benefits:

It is straightforward; test names are used to locate studies. A specific search for the concept of diagnosis/screening/

prognosis may not be necessary.Drawbacks:

New approaches may be missed because of evolving test strategies.

Using the test name alone may be insufficient, especially for gene product tests that are associated with multiple diseases.

Challenges Specific to Searching for Studies of Medical Tests (2 of 3)


The second approach for identifying studies of medical tests is to search for a specific disease or condition. Benefits:

This approach is preferred if specific medical tests are unknown.

It can be used in combination with searches for known test names.

Drawback: Searches are broad and increase the amount of work needed to filter down to the relevant studies.

This approach is used more often by reviewers.

Challenges Specific to Searching for Studies of Medical Tests (3 of 3)


1. Do not rely on search filters alone.

2. Do not rely on controlled vocabulary (subject headings) alone.

3. Search in multiple locations.

Principles of Searching for Studies of Medical Tests


Search filters (or “hedges”): Are pre-prepared and tested searches. Can be combined with searches on a particular disease or

condition. Most search filters have been developed for MEDLINE. Additional search filters have been developed

specifically for diagnostic imaging and EMBASE . Currently, search filters are inappropriate for systematic

reviews because they: May miss relevant studies. Do not perform as well for studies published before 1990

because of nonstandardized reporting and indexing of medical test studies, which may improve in the future.

Search Principle 1: Do Not Rely onSearch Filters Alone (1 of 2)


Category

Optimization of Search

Sensitivity/Specificity of Search

PubMed Search String

Diagnosis

BroadSensitivity/breadth(of “Diagnosis” search)

98%/74% (sensitiv*[Title/Abstract] OR sensitivity and specificity[MeSH Terms] OR diagnos*[Title/Abstract] OR diagnosis[MeSH:noexp] OR diagnostic* [MeSH:noexp] OR diagnosis,differential[MeSH:noexp] OR diagnosis[Subheading:noexp])

NarrowSpecificity/narrowness(of “Diagnosis” search)

64%/98% (specificity[Title/Abstract])

Search Principle 1: Do Not Rely onSearch Filters Alone (2 of 2)

Example: Filter used by PubMed Clinical Queries Category selected: “Diagnosis” Two choices of scope to optimize search: “Broad” or “Narrow”


It is important to use all known variants of a test name, as in the examples below that refer to hematocrit: Abbreviations (Hct, Crit, PCV) Generic names (hematocrit, packed cell volume) Proprietary names (e.g., LighTouch® HCT) International terms/spellings (haematocrit)

Controlled vocabulary may not contain all variants. Controlled vocabulary is unreliable due to nonstandardized

reporting and indexing. Textwords help identify tests not yet indexed or indexed

improperly. Filters may suggest appropriate textwords.

Search Principle 2: Do Not Rely on Controlled Vocabulary (Subject Headings) Alone


Searching more than one database and tailoring searches to each is always advisable. Otherwise, there is a risk of bias due to little overlap between the many databases.

Until reporting/indexing is standardized, the best method is to combine highly sensitive searches and manual article screening. Articles can still miss relevant articles, so additional sources of

information are necessary. Other sources of information:

Additional bibliographic databases Citation tracking (i.e., reading through references of relevant

articles) Regulatory documents

Search for diagnostic documents at Devices@FDA, a U.S. Food and Drug Administration Web site.

Search Principle 3: Search in Multiple Locations (1 of 4)


Open-Access Databases

Database URL Topic CoverageIBIDS (International Bibliographic Informationon Dietary Supplements)

www.ods.od.nih.gov/Health_ Information/IBIDS.aspx

• Dietary supplements

PubMed www.ncbi.nlm.nih.gov/pubmed

• Biology• Health sciences

TOXNET (Toxicology Data Network)

www.toxnet.nlm.nih.gov • Toxicology• Environmental health

adverse effectsWorld Health Organization Global Health Library

www.who.int/ghl/medicus/en/ • International biomedical topics

• Global Index Medicus

Examples of specialized databases that have free access:



Subscription DatabasesDatabase URL Topic CoverageCINAHL (Cumulative Indexto Nursing and Allied Health)

www.ebscohost.com/cinahl • Nursing• Allied health

EMBASE www.embase.com • Biomedical with emphases on drugs and pharmaceuticals

• More non–U.S. coverage than MEDLINE

IPA (International Pharmaceutical Abstracts)

www.csa.com/factsheets/ipa-set-c.php

• Drugs and pharmaceuticals

MANTIS (Manual Alternative and Natural Therapy Index System)

www.healthindex.com/MANTIS.aspx • Osteopathy• Chiropractic• Alternative medicine

PsycINFO www.apa.org/pubs/databases/psycinfo/index.aspx

• Psychological literature

Examples of specialized databases that require a subscription:



Citation-Tracking Databases

Database URL Access

Google Scholar www.scholar.google.com Free

PubFocus www.pubfocus.com Free

PubReMiner bioinfo.amc.uva.nl/human-genetics/pubreminer Free

SciVerse Scopus

www.info.scopus.com Subscription required

Web of Science

thomsonreuters.com/products_services/science/ science_products/a-z/web_of_science

Subscription required

Examples of citation-tracking databases:



AHRQ Evidence Report/Technology Assessment No. 142: Testing for BNP and NT-proBNP in the Diagnosis and Prognosis of Heart Failure All medical tests were known. The search consisted of all possible variations on test names. Using a search string to capture the diagnostic testing concept

was not necessary. AHRQ Comparative Effectiveness Review No. 2: Effectiveness

of Noninvasive Diagnostic Tests for Breast Abnormalities All medical tests were not known. The PubMed search string relied on textwords used to capture

the diagnostic testing concept:

“diagnosis OR diagnose OR diagnostic OR di[sh] OR “gold standard” OR “ROC” OR “receiver operating characteristic” OR sensitivity and specificity[mh] OR likelihood OR “false positive” OR “false negative” OR “true positive” OR “true negative” OR “predictive value” OR accuracy OR precision”

Examples of Search Strategies


Diagnostic search filters cannot be depended on exclusively.

When the full range of tests is known, searching for a specific test using all variant names may be sufficient.

Combining highly sensitive database searches (using textwords), hand searching, and searching lists of cited references is currently the best way to identify relevant studies.

Do not rely on controlled vocabulary alone. Search multiple locations and databases.

Be sure to check the Devices@FDA Web site, which is a useful source.

Key Messages


1. Generally, when designing a literature search, higher precision is desirable relative to recall.

a. True

b. False

Practice Question 1 (1 of 2)

Explanation of Question 1:

The statement is false. Maintaining balance between recall and precision is usually desirable.


2. Search filters are inappropriate for systematic reviews.

a. True

b. False



The statement is true. Search filters may miss relevant studies and do not perform as well for studies published before 1990 because of nonstandardized reporting and indexing.


3. Which of the following are true statements regarding controlled vocabulary:

3. A controlled vocabulary may not contain all variants.

4. A controlled vocabulary is reliable thanks to standardized reporting and indexing.

5. Textwords help identify tests not yet indexed or indexed improperly.

Choices:

i. a and b

ii. a and c

iii. b and c

iv. All three statements are true.



The correct answer is b. Controlled vocabulary is actually unreliable for identifying variants of a test name because of nonstandardized reporting and indexing.


4. Which of the following is not a principle of searching for studies outlined in this module:

< Do not rely on search filters alone.

< Do not rely on controlled vocabulary alone.

< Search in multiple locations.

< Avoid using Google Scholar as a search tool.



The correct answer is d. Google Scholar can be useful for grey literature searches.


This presentation was prepared by Brooke Heidenfelder, Rachael Posey, Lorraine Sease, Remy Coeytaux, Gillian Sanders, and Alex Vaz, members of the Duke University Evidence-based Practice Center.

The module is based on Chapter 4, Effective Search Strategies for Systematic Reviews of Medical Tests. In: Methods Guide for Medical Test Reviews. AHRQ Publication No. 12-EC017. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. www.effectivehealthcare.ahrq.gov/medtestsguide.cfm

Authors

Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Review. Rockville, MD: Agency for Healthcare Research and Quality; April 2012. AHRQ Publication No. 10(12)-EHC063-EF. Chapters available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

Astin MP, Brazzelli MG, Fraser CM, et al. Developing a sensitive search strategy in MEDLINE to retrieve studies on assessment of the diagnostic performance of imaging techniques. Radiology. 2008 May;247(2):365-73. PMID: 18372447.

Bachmann LM, Coray R, Estermann P, et al. Identifying diagnostic studies in MEDLINE: reducing the number needed to read. J Am Med Inform Assoc. 2002 Nov-Dec;9(6):653-8. PMID: 12386115.

Bachmann LM, Estermann P, Kornenberg C, et al. Identifying diagnostic accuracy studies in EMBASE. J Med Libr Assoc. 2003 Jul;91(3):341-6. PMID: 12883560.

References (1 of 6)

Balion C, Santaguida PL, Hill S, et al. Testing for BNP and NT-proBNP in the Diagnosis and Prognosis of Heart Failure. Evidence Report/Technology Assessment No. 142 (Prepared by the McMaster University Evidence-based Practice Center under Contract No. 290-02-0020). Rockville, MD: Agency for Healthcare Research and Quality; September 2006. AHRQ Publication No. 06-E014. Available at www.ncbi.nlm.nih.gov/books/NBK38136.

Betran AP, Say L, Gulmezoglu AM, et al. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005 Jan 28;5(1):6. PMID: 15679886.

Bruening W, Launders J, Pinkney N, et al. Effectiveness of Noninvasive Diagnostic Tests for Breast Abnormalities. Comparative Effectiveness Review No. 2 (Prepared by the ECRI Evidence-based Practice Center under Contract No. 290-02-0019). Rockville, MD: Agency for Healthcare Research and Quality; February 2006. AHRQ Publication No. 06-EHC005-EF. www.effectivehealthcare.ahrq.gov/repFiles/BrCADx%20Final%20Report.pdf.

References (2 of 6)

Conn VS, Isaramalai SA, Rath S, et al. Beyond MEDLINE for literature searches. J Nurs Scholarsh. 2003;35(2):177-82. PMID: 12854300.

Deville WL, Bezemer PD, Bouter LM. Publications on diagnostic test evaluation in family medicine journals: an optimal search strategy. J Clin Epidemiol. 2000 Jan;53(1):65-9. PMID: 10693905.

Doust JA, Pietrzak E, Sanders S, et al. Identifying studies for systematic reviews of diagnostic tests was difficult due to the poor sensitivity and precision of methodologic filters and the lack of information in the abstract. J Clin Epidemiol. 2005 May;58(5):444-9. PMID: 15845330.

Haynes RB, Wilczynski NL. Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey. MBJ. 2004 May 1;328(7447):1040. PMID: 15073027.

Haynes RB, Wilczynski N, McKibbon KA, et al. Developing optimal search strategies for detecting clinically sound studies in MEDLINE. J Am Med Inform Assoc. 1994 Nov-Dec;1(6):447-58. PMID: 7850570.

References (3 of 6)

Honest H, Bachmann LM, Khan K. Electronic searching of the literature for systematic reviews of screening and diagnostic tests for preterm birth. Eur J Obstet Gynecol Reprod Biol. 2003 Mar 26;107(1):19-23. PMID: 12593888.

Leeflang MM, Scholten RJ, Rutjes AW, et al. Use of methodological search filters to identify diagnostic accuracy studies can lead to the omission of relevant studies. J Clin Epidemiol. 2006 Mar;59(3):234-40. PMID: 16488353.

Relevo R. Effective search strategies for systematic reviews of medical tests. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 4.1-4.9. AHRQ Publication No. 12-EHC017. www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

Ritchie G, Glanville J, Lefebvre C. Do published search filters to identify diagnostic test accuracy studies perform adequately? Health Info Libr J. 2007 Sep;24(3):188-92. PMID: 17714173.

References (4 of 6)

Sampson M, Barrowman NJ, Moher D, et al. Should meta-analysts search EMBASE in addition to MEDLINE? J Clin Epidemiol. 2003 Oct; 56(10):943-55. PMID: 14568625.

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complement Ther Med. 2004 Dec;12(4):228-32. PMID: 15649836.

Suarez-Almazor ME, Belseck E, Homik J, et al. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000 Oct;21(5):476-87. PMID: 11018564.

van der Weijden T, Ijzermans CJ, Dinant GJ, et al. Identifying relevant diagnostic studies in MEDLINE. The diagnostic value of the erythrocyte sedimentation rate (ESR) and dipstick as an example. Fam Pract. 1997 Jun;14(3):204-8. PMID: 9201493.

References (5 of 6)

Vincent S, Greenley S, Beaven O. Clinical evidence diagnosis: developing a sensitive search strategy to retrieve diagnostic studies on deep vein thrombosis: a pragmatic approach. Health Info Libr J. 2003 Sep;20(3):150-9. PMID: 12919278.

Whiting P, Westwood M, Beynon R, et al. Inclusion of methodological filters in searches for diagnostic test accuracy studies misses relevant studies. J Clin Epidemiol. 2011 Jun;64(6):602-7. PMID: 21075596.

Whiting P, Westwood M, Burke M, et al. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol. 2008 Apr;61(4):357-64. PMID: 18313560.

Wilczynski NL, Haynes RB. EMBASE search strategies for identifying methodologically sound diagnostic studies for use by clinicians and researchers. BMC. Med 2005 Mar;3:7. PMID: 15796772.

Zheng MH, Zhang X, Ye Q, et al. Searching additional databases except PubMed are necessary for a systematic review. Stroke. 2008 Aug;39(8):e139. PMID: 18583557.

References (6 of 6)

Documents

Effective Search Strategies for Systematic Reviews of Medical Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules