Knowing how good our searches are

Knowing how good our searches are: an approach derived from search filter development methodology

Sarah Hayman

8th International Evidence Based Library and Information Practice ConferenceBrisbane, July 2015

Smart Searching

• Introduction• Search Filters at CareSearch and

Flinders Filters• Developing the learning modules• Evaluation of the learning modules

The Importance of Searching

• Need for evidence• Impact of missing evidence• Volume of information• Complexity of sources

Search Filters• CareSearch• Flinders Filters• Some Search filters worldwide

– ISSG Search Filters Resource– BMJ Clinical Evidence– Canadian Agency for Drugs and Technologies in Health (CA

DTH)– HL WIKI International– McMaster University. Health Information Research Unit– Scottish Intercollegiate Guideline Network (SIGN)

A search filter …• is a validated search strategy of known performance

effectiveness• is designed for and built in a particular bibliographic

database• can be methodology- or subject-based• can be expressed as a URL and embedded in a web

page for quick reliable access to evidence• may be sensitive or specific

Using Search Filters• Understand the nature and purpose of the search

filter• Critically appraise the search filter (tools available at

e.g. ISSG Search Filters Resource) • Look at the terms in the search filter and use as a

starting point for your search• Make search filters available to your users as useful

tools for accessing evidence reliably• Note that search filters may not be appropriate for the

comprehensive searching required for systematic reviews

Example: the Heart Failure Search Filter

As a URL:http://www.ncbi.nlm.nih.gov/pubmed?term=((heart+failure[tw]+OR+ventricular+dysfunction,+left[mh:noexp]+OR+cardiomyopathy[tw]+OR+left+ventricular+ejection+fraction[tw])+AND+Medline[sb])+OR+((heart+failure[tw]+OR+left+ventricular+dysfunction[tw]+OR+cardiomyopathy[tw]+OR+left+ventricular+ejection+fraction[tw]+OR+cardiac+resynchronization[tw]+OR+LV+dysfunction[tw]+OR+left+ventricular+systolic+dysfunction[tw]+OR+left+ventricular+diastolic+dysfunction[tw]+OR+cardiac+failure[tw])+NOT+medline[sb])+AND+english[la]

As a PubMed query:

Smart SearchingExpert Advisory Group (EAG)

Subject expertsGold Standard Set

Sample setTerm identification

Term identificationValidation

Testing

https://sites.google.com/site/smartsearchinglogical/home

The Modules

Usage and Evaluation of the Smart Searching modules

• Direct feedback sought from site• Web usage statistics (Google Analytics)• User survey conducted April, 2015

Google Analytics Report: Audience Overview (June 19, 2015)

Google Analytics Report: Location(June 19, 2015)

User Survey (April, 2015) - 50 responses

Occupation

Would you recommend this site to a colleague?

User Survey (April, 2015) - 50 responses

Do you think you would use this approach for testing?

Have you applied any of these techniques in your searching practice?

User Survey (April, 2015) - 50 responses

Some Representative Comments

Functioning of the SiteSystematic and methodical approach

Clear and easy to use (13 others similar to this)

Template applicable across disciplines

While the content is useful- the constant arrow moving would not be appealing to busy clinicians or medical librarians

Time constraintsTime is the major factor, followed closely by access to the subject experts

it can possibly save me some time as I spend a lot of time in my job training and assisting health researchers in building effective literature searches

Time restraints, level of information need does not usually require that level of sensitivity (several like this)

Very useful, however not sure if I would have time to test every search in a real life work situation.

User Survey (April, 2015) - 50 responsesSome Representative Comments

Value of the SiteReassessing the way I approach things

The more knowledge/ideas we share about improving search techniques the more beneficial it is to the profession

I can see the value in being able to 'qualify' and measure my searching outcomes

Informative...I bet there are other librarians who, just like me, are not utilizing these techniques properly.

Adding those extra dimensions increases the robustness of our searching and helps to systematise the things we do

I tend to be more intuitive than systematic with my searches […] Reporting would force me to ensure consistency!

Seems great as a refresher for me but will also be really useful for staff training purposes

I don't think that librarians test their search strategy and I feel it is an important tool to argue our competence and relevancy, especially in private enterprise

It gives a measure of effectiveness that speaks for itself...numbers are extremely hard to dispute!

Quality of the information and instruction; Quiz to reinforce what just learnt

User Survey (April, 2015) - 50 responses

Two Interesting Comments

Keep this aimed at professional librarians.  I've never seen a profession so bent on becoming obsolete.  Professional librarians/expert searchers have unique skills that we should be promoting to incoming librarians and teaching the new generation of searchers. 

I was not happy to see that you state that librarians are not subject experts. After many years of searching on specialist topics I would consider myself to have more knowledge about the thesaurus terms used by databases than most subject 'experts'.

Conclusion• We developed this resource because we believe that searches are

important and should be accorded the respect of a scientific approach.• The resource appears to be well used (4,484 unique users in first year,

from 72 countries)• 29.55% of total sessions were from Australia in the first year• Top six countries represent approx. 85% of all users in the first year

(Australia, US, UK, Canada, Ireland, NZ)• Bounce rate lower and session times longer amongst Australian users• Responses to the survey were positive (81.63% would recommend to a

colleague)• Need more information about reasons for using – can largely only

speculate beyond what the survey responses suggested• We will maintain and aim to improve the resource – all suggestions and

feedback welcomed

AcknowledgementsThank you to:

• Colleagues at CareSearch and Flinders Filters (Raechel Damarell, Mikaela Lawrence, Yasmine Shaheem, Jennifer Tieman)

• Members of the Smart Searching Advisory Group• Health Libraries Australia and Medical Director (formerly HCN)

for the Health Informatics Innovation Award 2012 that supported the development of Smart Searching

Sarah HaymanFlinders Filters, Flinders University

sarah.hayman@flinders.edu.au