Searching for Grey

Literature Alison Bethel (Information Specialist, PenCLAHRC) Simon Briscoe (Information Specialist, PenCLAHRC) Morwenna Rogers (Information Specialist, PenCLAHRC)

Overview of presentation

Grey literature: definitions and challenges

Identifying grey literature using web searching (SB)

Identifying grey literature using web-based catalogues and

databases (AB)

Identifying grey literature using bibliographic databases (MR)

Reporting grey literature searching

Discussion

Definition(s) of grey literature

“That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers”.

(The ‘Luxembourg’ definition)

Fourth International Conference on Grey Literature: New Frontiers in Grey Literature. GreyNet, Grey Literature Network Service. Washington D.C. USA, 4-5 October 1999.

Some other definitions:

“…not available through standard distribution means, no standard bibliographic controls, not peer-reviewed, ephemeral and historically difficult to find.”

“…anything that won’t stand up on a shelf on its own.”

“…material that is difficult to catalogue.”

Mahood, Q. et al. (2014) Searching for grey literature for systematic reviews. Research Synthesis Methods. 5:221-34

Types of grey literature

Grey literature

Reports-including preprints; preliminary progress and advanced reports [including ongoing trials reports]; institutional, internal, technical, and statistical reports; research memoranda…;

Theses;

Conference proceedings;

Technical specifications and standards;

Translations (not distributed commercially);

Bibliographies;

Technical and commercial documentation;

Official documents (issued in limited numbers).

Alberani, V et al. (1990) The use of grey literature in health sciences: a preliminary survey. Bulletin of the Medical Library Association. 78(4): 358-63.

What are the challenges when searching for grey literature?

1. There is no ‘main’ database for grey literature, e.g. there is no equivalent to MEDLINE,

EMBASE or PsycINFO.

2. Grey literature databases often have fewer bibliographic fields to search in than published

literature databases, e.g. may not have abstract or index term fields.

3. Grey literature resources (e.g. databases, websites, catalogues) often lack advanced search

features.

4. Identifying grey literature resources can be time consuming, e.g. might entail extensive web

searches for relevant websites before you start the formal search process.

5. Grey literature resources often lack export features.

6. Content in grey literature resources might frequently change.

1. Identifying grey literature using web searching (SB)

Definition of web searching

Searching using search engines and websites which are not dedicated web-based literature

resources.

(Web searching is generally used to identify grey literature not indexed in databases, though it

can also be used for identifying published studies).

Search engines

Search engines are often the best place to start, e.g. for browsing the web to identify

potential websites to search.

There are many choices of search engine and they each offer different approaches to

searching the web (and return different results).

Two reasons why search results might differ are:

The algorithms used to retrieve and rank results vary between search engines.

Some search engines use the internet protocol (IP) address of a user to tailor the search results to

the user’s search history.

It is worth experimenting with different search engines to see how the results vary.

Search engines

Some examples of search engines:

Google Search https://www.google.com/

(See http://www.googleguide.com/ for Google search tips)

DuckDuckGo https://duckduckgo.com/

DogPile http://www.dogpile.com/

The Deep web

Graphic downloaded from http://www.slideshare.net/

Websites

Which websites to search will be determined by the review topic and type:

Pharmaceutical companies (for company trials registries);

Medical device manufacturers (for regulatory documentation);

Medicines regulatory bodies e.g. US Food and Drug Administration (FDA) and European

Medicines Agency (EMA) (for regulatory documentation);

Professional societies (for professional guidelines, reports);

National health departments of UK and other countries (for govt. guidelines, reports);

Charities and think tanks, e.g. Kings Fund, Nuffield Trust (for reports).

Grey Matters is a useful list of websites which index health related grey literature, produced by

the Canadian Agency for Drugs and Technologies in Health (CADTH).

https://www.cadth.ca/resources/finding-evidence/grey-matters

Systematic and pragmatic approaches to web searching

Systematic approaches:

Keep search terms similar to the bibliographic database search terms.

Use search terms consistently between different resources.

Document the search process so can be reported in the review if required.

Pragmatic approaches:

Use an iterative approach when searching simple interfaces.

Screen ‘on screen’ if no export features (difficult to double screen).

When using search engines limit the screening process to a set number of pages e.g. first 100

results.

Limit the search to a specified domain name, file type, or to webpages where the search

terms appear in the title.

Identifying grey literature using web-based catalogues and databases (AB)

EThOS by the British Library

What:

Aims to demonstrate the quality of UK research by making UK doctoral theses discoverable and openly available online. It supports the UK Government’s open access principle that publications resulting from publicly-funded research should be made freely available for all researchers, providing opportunities for further research.

http://ethos.bl.uk/Home.do

Pros:

• Catalogued over 300,000 UK theses

• Access to over 80,000 full text

• Does have an advanced search

Cons:

• Don’t use truncation (*)

• Difficult to download bibliographic results • Try using the firefox plugin to Zotero

Explore at the British Library

What

Main catalogue for the British Library for millions of records of books, journals, report, sound archives etc in the British Library’s collection.

http://explore.bl.uk/primo_library/libweb/action/search.do?vid=BLVU1

Pros:

• Can restrict to report literature (or theses)

• Advanced search

• Truncation search (*)

• Download to Endnote

Cons:

• ?

OpenGrey

What System for Information on Grey Literature in Europe, is your open access to 700,000 bibliographical references of grey literature (paper) produced in Europe and allows you to export records and locate the documents. Includes technical or research reports, doctoral dissertations, some conference papers, some official publications, and other types of grey literature.

http://www.opengrey.eu/

Pros:

• European wide

• Can use Boolean searching in the search box

• Shows results by discipline

• Can use truncation (*)

Cons:

• Not easy to download bibliographic record • Can use the firefox plugin for Zotero

• No advanced search

• Not obvious where it’s searching

Grey Literature Report

What A bimonthly publication of The New York Academy of Medicine alerting readers to new grey literature publications in health services research and selected urban health topics. Grey Literature resources are cataloged and indexed using MeSH.

http://greylit.org/

Pros:

• Health services subject specific

• Indexed using MeSH

• Can’t use truncation

• Can use Boolean search

• Contains a summary/abstract

• Can export to Endnote

Cons:

• No advanced search

• US bias

Clinical trials

Clinicaltrials.gov

ClinicalTrials.gov is a registry and results database of publicly and privately supported clinical studies of human participants conducted around the world. Currently lists 224,838 studies with locations in 50 US states and in 192 countries

https://clinicaltrials.gov/

ICTRP:

The main aim of the WHO ICTRP is to facilitate the prospective registration of the WHO Trial Registration Data Set on all clinical trials, and the public accessibility of that information

http://apps.who.int/trialsearch/Default.aspx

General points

• Overlap between the resources

• Downloading can tricky

• Keep your search simple

• Record what terms you’re searching

• Record where you’re searching

Identifying grey literature using bibliographic databases (MR)

Google Scholar for Grey Literature

• Free to use • Articles, theses, books, abstracts from academic

publishers, professional societies, online repositories, universities and other web sites.

• Identify grey literature using related documents

BUT • Cannot search systematically • No human input; results unreliable • Poor export facilities

Finding grey literature by forward citation chasing: an example

What Is the Impact of Using Outdoor Spaces Such as Gardens on the Physical and Mental Well-Being of Those With Dementia? A Systematic Review of Quantitative and Qualitative Evidence.

Whear R, Coon JT, Bethel A, Abbott R, Stein K, Garside R. Journal of the American Medical Directors Association 2014; 15(10):697-705.

Google Scholar

Google Scholar FCC results

Document type Number

Books/Chapters 5

Editorials 1

Primary Studies 3 (1 qual, 2 quant including 1 RCT)

Theses 6

General Review papers 2

Websites 1

Systematic reviews 2

Policy documents 1

News/opinion piece 1

Bibliographic databases for grey literature

Advantages

• Can search systematically

• Easy to run/download results

• Easy to report (transparency/replicability)

Disadvantages

• Difficult to identify among published literature (unless specific database e.g. for theses)

• Less likely to be an abstract, therefore less likely to come up in a search

• Fields may be empty; incompatible with reference management software

Databases for conference abstracts

DATABASE

SUBJECT

RECOMMENDED PROVIDER

Conference Proceedings Citation Index - Science (CPCI – S) Conference Proceedings Citation Index – Social Science & Humanities (CPCI – SSH)

All science (Acoustics, Agriculture, Anatomy, Astronomy….) E.g. Anthropology, Archaeology, Architecture, Art

Web of Science

ERIC Education ProQuest

PsycINFO Psychology and related fields

OvidSp

CINAHL Plus with Full Text Nursing EBSCOhost

EMBASE Medicine OvidSp

Databases for dissertations

DATABASE

SUBJECT

RECOMMENDED PROVIDER

ProQuest Dissertation & Theses Global (PQDT Global)

ALL ProQuest

ProQuest Dissertations &Theses – UK and Ireland (Index to Theses)*

ALL

ProQuest

PsycINFO (content from Dissertation Abstracts International)

Psychology and related fields

OvidSp

CINAHL Plus with Full Text Nursing EBSCOhost

*Included in PQDT Global

HMIC (Health Management Information Consortium)

Subject coverage: • Health management • Health service policy • Health service quality • Planning, design, construction and

maintenance of health service buildings

• Occupational health • Control and regulation of medicines • Medical equipment and supplies • Social care and social services

Types of document: • Official publications • Pamphlets • Government reports

Content from the UK Department of Health and the King’s Fund Information and Library Service from early 1980s. 174,000+ records

Social Policy and Practice Content supplied by SCIE, Centre for Policy on Ageing, IDOX, Social Care Institute for Excellence since 1890s

Subject coverage:

• Public and social policy

• Public health

• Social care

• Community development

• Mental and community health

• Homelessness

• Housing

• Crime

• Equalities

• Law and order

• Children and families

• Older people

Types of document: • Books • Pamphlets • Semi-published reports • Surveys • Statistical reports

Subject specific databases covering grey literature

• ADOLEC (adolescent health)

• Africa-Wide Information (Africa)

• BDENF (nursing)

• CEHA Database (environmental health & sanitation)

• CINAHL Plus (nursing and allied health)

• DESASTRES (disasters and emergencies)

• ELDIS (international development)

• Global Health (international public health)

• GreenFILE (green issues)

• HEER (higher education research)

• HISA (public health)

• HMIC (health management)

• LEYES (Latin Americas and Caribbean, health-related legislation)

• Library, Information Science & Technology Abstracts (library studies)

• LILACS (South America) • MedCarib (Caribbean) • National Criminal Justice Reference

Service • NBER Working Papers (economics) • Popline (reproductive health) • PsycEXTRA (psychiatry, psychology,

mental health) • ReliefWeb (humanitarian disaster

relief) • REPIDISCA (sanitation and

environmental sciences) • Social Policy & Practice (Social Policy) • Teaching Reference Center (teaching

and learning) • TRIP Database (clinical medicine) • WHO Reproductive Health Library

(reproductive health) • WPRIM (WHO Western Pacific region)

Reporting web searching

Web searching is typically not reported transparently, e.g.

Fig 1. Details reported about searching websites*

Though there are problems with reproducibility…

*Figure taken from: Briscoe, S. Web searching for systematic reviews: a case study of reporting standards in the UK Health Technology Assessment programme. BMC Research Notes 2015; 8:153.

Reporting web searching*

In order to facilitate the transparency and reproducibility of searches, for each website or

search engine document and report the:

- resource name and URL;

- the dates searched;

- the search terms;

- the citation details of any includable literature.

Keep a copy of any useful information found on a webpage in case the content is

subsequently changed or deleted.

*Text adapted from: Briscoe S (2015). Web searching for systematic reviews: how to develop the current guidance on methods and reporting standards. 23rd Cochrane Colloquium. Vienna. 3rd Oct - 7th Dec 2015.

Reporting web searching*

The results of web searches will not be reproducible to the same extent as bibliographic

database searches because web content and search algorithms often change. However, it is

still important to report the search process in order to ensure the methods used are

transparent.

*Text taken from: Briscoe S (2015). Web searching for systematic reviews: how to develop the current guidance on methods and reporting standards. 23rd Cochrane Colloquium. Vienna. 3rd Oct - 7th Dec 2015.

Search summary table

Effectiveness of mindfulness-based stress reduction and mindfulness-based cognitive therapy in vascular disease: a systematic review and meta-analysis of randomised controlled trials

Included references Format Database searches (run Jan 2013) Supplementary seaches

AMED BNI CINAHL Cochrane Embase HIMC Medline PsycINFO SPP WoS fcs bcs hs wss

Blom 2012 jnl x x

De la Fuente 2010 jnl x x

Hartmann 2012 jnl x x x x x

Hughes 2013 jnl x x

Johansson 2012 jnl x x x x

Robert McComb 2004 jnl x x x

Tacon 2003 jnl x x x x x x

Nyklicek 2014 jnl x

Van Son 2013 jnl x

No. included refs 1 1 3 3 5 0 4 2 0 5 2 0 0 0

No. unique refs 0 0 0 0 0 0 0 0 0 0 2 0 0 0

Total no. refs downloaded

21 7 103 67 773 39 343 278 nd 376

No. refs screened 21 4 34 4 533 36 339 201 2 172

Sensitivity 11% 11% 33% 33% 56% 0% 44% 22% 0% 56%

Precision 5% 14.3% 2.9% 4.5% 0.60% 0% 1.2% 1.2% nd 1.3%

No. database searches carried out = 10

Total no. refs found from searching = 2038

No. refs screened at Ti&Ab = 1346

No. of included refs from searching = 7

Total no. of included refs = 9

Discussion

• Examples where identified grey literature contributed to the findings of a review?

• Any examples where including grey literature has changed the conclusion of a review?

• Do you have any useful resources to share?

• Do you have any tips on searching/finding/managing grey lit to share?

• Who searches for the grey lit in your reviews?

• Do you and should you double screen grey lit?