Assessing an adaptive, profile-based PubMed search tool

Assessing an adaptive,profile-based PubMed

search toolMarisa Conte

meese@umich.eduNLM Associate Fellow

University of Michigan Health Sciences Libraries

MiSearch Adaptive biomedical literature search tool Utilizes implicit relevance feedback Builds statistical profile to predict which results will be most

relevant to user

Objective To compare the performance of a locally-

developed third-party PubMed search toolto PubMed in terms of user satisfaction,search success and efficiency

Collaboration

Methodology N = 11 librarians (12 recruited) Tasks derived from mediated search files Each subject completed 2 tasks with each

search tool Each task completed by 3 subjects with

each search tool Pre-training conducted by tool developer

Data collection Web-based surveys

Demographic survey Task-specific survey

PubMed search histories MiSearch data logfiles

Sample questions In patients with diabetes mellitus, does

increasing hemoglobin A1c reduce the riskof amputation and kidney failure?

What are the attitudes, beliefs and socio-cultural factors that influence cervicalcancer screening in the US? What aboutspecifically among immigrant groups? AndI’m only interested in research from 1990 –present.

Limitations Convenience sample Unclear survey questions Study tasks too complicated Problems with data collection

Sample results – Query 2

1 = low21.7Completeness21.3Accuracy

1.71.3Speed1.71.3Tool1.71Search activity

PubMedMiSearchAveragesatisfaction

Overall satisfaction with searches

0

10

20

30

40

50

60

% r

esp

on

ses

High Fair Low

User satisfaction

PubMed

Misearch

Overall satisfaction with tools

0

10

20

30

40

50

60

70

% r

esp

on

ses

High Fair Low

User satisfaction

PubMed

MiSearch

Suggestions Improve database response time Add more filters Add a “reset” function Make search details more transparent

Next steps

Librarians suggested changes which: were easy to implement dramatically improved the tool’s functionality

First usability test – helped establish formalusability protocols and heuristics

New roles for librarians

New roles for librarians Usability experts

Heuristic evaluation of tools Portal development Formal usability studies Weblog analyses

Education, promotion, dissemination Research collaborations

Resources MiSearch: http://misearch.ncibi.org/ NCIBI: https://portal.ncibi.org/gateway/

States, D.J., Ade, A.S., Wright, Z.C., Bookvich,A.V. & Athey, B.D. (2008) Misearch AdaptivePubMed Search Tool. Bioinformatics.PMID 18326507

Acknowledgments Jean Song, Nirit Glazer, Barbara Mirel, David States Librarians from UM’s Health Sciences and Shapiro

Science Libraries National Center for Integrative Biomedical Informatics

(NIH grant #U54DA021519) National Library of Medicine

This research was supported in part by an appointment to the NLMAssociate Fellowship Program sponsored by the National Library ofMedicine and administered by the Oak Ridge Institute for Science andEducation.