Embed Size (px)
DESCRIPTION
Traditional metrics, such as the h-index and journal impact factors, are used to measure the scholarly impact of research. However, in the current climate of accountability by funding providers, fund recipients would benefit from a more comprehensive impact management system (IMS) to facilitate the capture and reporting of narratives (including metrics) about research impact in the academy, on social policy, in industry, and ultimately with the public. Librarians have always been good at telling and facilitating stories. Research support librarians can use their storytelling skills to contribute to the implementation and administration of an impact management system. Being able to translate research impact into harvestable and reportable metadata is the key.
Citation preview
Impact beyond metrics: Telling your research impact story
Research Support Community DayBrisbane, Australia, 11 February 2013
Pat LoriaResearch LibrarianUniversity of Southern Queensland
Limitations of traditional metrics
Scholarly Public
Recommended Faculty of 1000 Popular press
Cited Traditional citations Wikipedia
Discussed Scholarly blogs Facebook, Twitter
Saved Mendeley, CiteULike Delicious
Viewed PDF views HTML views
Adapted from Jason Priem, Altmetrics and Revolutions: https://docs.google.com/presentation/pub?id=1Y4JnchsmHHiOQdJsEpQr33qmMWqhZJrPTDAg1cZoCcI&start=false&loop=false&delayms=3000#slide=id.i0
Limitations of traditional metrics
Scholarly Public
Recommended Faculty of 1000 Popular press
Cited Traditional citations Wikipedia
Discussed Scholarly blogs Facebook, Twitter
Saved Mendeley, CiteULike Delicious
Viewed PDF views HTML views
Adapted from Jason Priem, Altmetrics and Revolutions: https://docs.google.com/presentation/pub?id=1Y4JnchsmHHiOQdJsEpQr33qmMWqhZJrPTDAg1cZoCcI&start=false&loop=false&delayms=3000#slide=id.i0
Three generations of impact metrics
Journal-level
metrics
Author-level
metrics
Article-level
metrics
Different levels of metrics
Journal-level metrics Journal impact factor SJR & SNIP
Author-level metrics Citations per Paper (CPP) H-index
Article-level metrics Articles, datasets, blogs, code, artistic creations Citations, mentions, views, downloads, etc. Altmetric bookmarklet, ImpactStory, etc.
USQ and the NCP calculator
NCP: Normalized Citations per Paper Developed by USQ (Library and Systems) Uses Scopus RDCP data to normalize Normalization creates a level playing field Citations in low visibility/low citing fields
normalized up Citations in high visibility/high citing fields
normalized down Source code for NCP calculator to be released
soon on GitHub
NCP calculator
Citations: 2010; Papers: 2007-2009
Measuring impact beyond academia
…impact is defined in a similar way as for the UK REF, i.e. “an effect on, change, benefit to the economy, society, culture, public policy or services, health, the environment or quality of life beyond academia”.
ATN and Go8, Guidelines for completion of case studies in ATN/Go8 EIA Impact Assessment Trial: June – August 2012, http://www.atn.edu.au/eia/Docs/EIA_Trial_Guidelines_FINAL.pdf
Measuring impact beyond academia
Impact Case Study (ATN & Go8, similar to REF) Institution SEO codes Title of Case Study Context Summary of Case Study Impact Details of Impact Research underpinning Impact Research outputs from research underpinning impact Additional information:
Validation of impact People Investment income
http://www.atn.edu.au/eia/Docs/EIA_Trial_Guidelines_FINAL.pdf
• Overheads?• Data sources?
Alternative metrics or altmetrics
[A]ltmetrics is the creation and study of new metrics based on the Social Web for analyzing, and informing scholarship.
http://altmetrics.org/about/
Altmetric bookmarklet
http://www.altmetric.com/bookmarklet.php
Altmetric bookmarklet
Free bookmark for Chrome, Firefox and Safari Click on while viewing paper for impact data Altmetric API can be embedded into second
and third-party platforms, apps and mashups Only works on PubMed, arXiv or DOI Only supports publishers who embed Google
Scholar friendly citation metadata Twitter mentions are only available for articles
published since July 2011
Example: Oxidants, antioxidants and the current incurability of metastatic cancers
Altmetric bookmarklet
Counts Disadvantages / Bugs
Altmetric score Difficult to explain
Tweeters
Facebook users
News outlets
Science blogs
Google+ users
Mendeley readers
CiteULike readers
Connotea readers
Bookmarklet only works on PubMed, arXiv or pages containing a DOI
Only supports publishers who embed Scholar friendly citation metadata
Twitter mentions are only available for articles published since July 2011
http://www.altmetric.com/bookmarklet.php
ImpactStory.orgArtefact type Counts Known Bugs 12/12
Journal article Topsy Twitter coverage weak
Dataset Science Seeker Different results for DOI and URL searchesSoftware PubMed
Slides Dryad Missing metrics
Generic SlideShare Metrics too low
Wikipedia
Figshare
PLoS Search
PLoS ALM
Delicious
Facebook No link to mentions
Scopus
CiteULike
GitHub
Mendeley
ImpactStory.org
PlumAnalytics.com
Usage - Downloads, views, book holdings, ILL, document delivery
Captures - Favourites, bookmarks, saves, readers, groups, watchers
Mentions - blog posts, news stories, Wikipedia articles, comments, reviews
Social media - Tweets, +1's, likes, shares, ratings
Citations - Web of Science, Scopus, Google Scholar, Microsoft Academic Search
PlumAnalytics.com
articles book chapters books clinical trials datasets figures grants patents
presentations source code videos greatest sources researcher graph group metrics paid service no API available
Reasons to use altmetrics
Same reasons as using traditional metrics: Grants, promotions, staff/program review Review research dissemination strategy
Measure of influence and reach of output Publishers can add value for authors/readers Informed decisions by research managers Comprehensive view of impact for funders “Citation graph data is like Chekhov’s gun:
once on stage, it has to be fired.” (Peter Vinkler, cited by Jason Priem, Altmetrics and Revolutions)
Article Level Metrics (ALMs)
“Article-Level Metrics are a comprehensive set of impact indicators that enable numerous ways to assess and navigate research most relevant to the field” (http://article-level-metrics.plos.org/alm-info/)
PLoS ALMs
http://article-level-metrics.plos.org/alm-info/
PLoS example
Source: http://article-level-metrics.plos.org/researchers/
Research profile systems
For example, Symplectic Elements:
The great objection!
It’s not real impact!= It’s not scholarly impact.
Research Impact Story
Scholarly Public
Government Industry
What is USQ’s research impact story?
Impact management system (IMS)
IMS
Impact Metrics
Curriculum Vitae
Esteem Measures
Professional Activities
Industry Impact
Policy Impact
Community Impact
Research Income
Human Resources
Research Outputs
Flexible Reportable
Metadata is key!
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
Impact metadata
Academic Citation Publication Teaching Supervision Research project Research income
Government Application Consultation Social policy
Industry Consultation Partnership Patent
Social Altmetric
Facebook Twitter Mendeley Wikipedia
Engagement Media coverage
Design according to reporting needs
Discussions with Symplectic
Librarian as storyteller
Image: http://kassonpubliclibrary.blogspot.com.au/2010/03/spring-planting-at-library-storytime.html
Advantages of IMS
Facilitates storytelling of research impact Saves time for researchers and managers Enables internal and external reporting Evidence base for individual/program review Accommodates government requirements Data for case studies and grant applications Impact captured for longitudinal analysis Best IMS support transfer of data to other IMS Managing impact: a new research literacy?
Take-aways
Holistic impact monitoring and reporting Make it easy for academics and managers Do you need an impact management system? What is your research impact story?
http://jeps.efpsa.org/blog/2012/06/20/maximizing-research-impact/
Research impact discussion
Image: http://www.duffysrehab.com/blog/want-to-make-an-impact-consider-betty-ford
Pat LoriaEmail: [email protected]: @pat_loria
