Benchmarking Research Performance using Incites Citation-based intelligence for research performance evaluation

Subhasree Nag, PhDSenior Solutions Consultant

2

Speaker Profile

Dr. Subhasree Nag is a senior solution consultant for the scientific and scholarly research division at Clarivate Analytics. She completed her M.Pharm from ICT, Mumbai, her PhD from Texas Tech University Health Sciences Center, USA and her post-doctoral training from Pacific Northwest National Laboratory, USA where she worked on anticancer drug discovery and pharmacokinetic modeling. She has 28 peer reviewed publications and more than 600 citations. As part of Clarivate Analytics, she is instrumental in carrying out author workshops and designing research capacity building solutions in different institutions all over the country. Her other interests include scientometrics and bibliometric evaluation

Trends in scholarly research

Efforts at objective approaches to allocation of credit

Changing nature of scholarly journal publishing

Global competition in the Sciences – BIG SCIENCE

Reputation management & demonstration of achievement

Funding pressures

A need for information tools to advance informed decision makingResearch Analytics– using metrics to understand research activities– addresses the need for administrators to answer the following:

• How does my institution compare to peer institutions (down to the paper and individual level)?

• What are the strongest fields at my institution? Which ones need improvement?• How many papers did my institution produce?• Which authors are rising stars?• Is my institution's research focus changing?• Who is collaborating with whom inside and outside the institution? How often? And where

is it most impactful?

What is INCITES?

InCites™ is a citation-based research evaluation tool on the Web that enables you to analyze institutional productivityand benchmark your output against peers worldwide.

This comprehensive resource supplies all the data and tools you need to easily produce targeted, customized reports...all in one place. You can conduct in-depth analyses of your institution's role in research, as well as produce focused snapshots that showcase particular aspects of research performance

How Clarivate Analytics can helpWith InCites Benchmarking & Analytics, you can:• Analyze institutional productivity and benchmark your output against peers worldwide.• Pinpoint influential researchers and emerging research trends.• Showcase strengths and identify potential areas for growth. • Monitor collaboration activity and track new collaboration opportunities. • Support accreditation activity, funding proposals, legislative agendas, alumni appeals, faculty and student

recruitment.• Plan a research strategy with metrics that can be tracked over time.• Pull multiple datasets (of 50,000) from the Web of Science Core Collection for deeper analysis.

Basic indicators – Type of Citation Metrics used in Incites

6

Productivity # papers

Total influence

# citations

H-index

Efficiency

Avg. citation rate

Percent of papers cited

Relative Impact/Bench-marking

Journal actual/expected citation rateCategory actual/ expected citation ratePercentile in categoryand mean percentile% papers in top 10% of theirfield% papers in top 1% of theirfield

Aggregated PerformanceIndicator

Specialization Disciplinarity index Interdisciplinarity index

Can be applied to an institution,a researcher, a research group,etc.

Key Metrics used in Incites

METRIC MEASURE IDENTIFYTimes Cited Total cites to an authors papers Authors with highest /lowest total cites to their papers

WOS documents Total number of papers by an author indataset Authors with highest/ lowest numberof publications

Average cites per document

Efficiency (or average impact) of author papers Authors with highest/lowestaverage impact

h-index An authors research performance. Publications are ranked in descending order by the times cited. The value of h is equal to the number of papers (N) in the list that have N or more citations

Authors with highest impact and quantity of publications in a single indicator

JournalActual/Expected Citations

Average ratio for authors papers. Ratio is relationship between actual citations to each paper to what is expected for papers in same journal/ publication year and document type

Authors who’s papers perform above (1) or below what is expect in their respective journals. Useful when comparing authors in different fields/ career length

Category Actual/Expected Citations (Category Normalized Citation Impact)

Average ratio for authors papers. Ratio is relationship between actual citations to each paper to what is expected for papers in same category/ publicationyear and document type

Authors who’s papers perform above (1) or below what is expected in their respective subject categories. Useful when comparing authors in differentfields/ career length

Average percentile Average Percentile for set of authors papers. Percentile is assigned to a paper within a set of papers from same subject category/year/ document type ordered most cited (0%) to least cited (100%)

Authors who’s papers are performing at the top or bottom of their respectivefields

7

Web of Science Core Collection: The InCites Data Source

Web of Science Core Collection: The InCites Data Source

• Cited references for all papers, back to 1900

• All author names and addresses captured for all papers

• Funding data from 2008-present

For many years Clarivate Analytics has provided a wide range oftools and services supporting accurate and effective research evaluation.

Our specialists work with Web of Science data and ensure maximum standardization and unification before delivery to customers.

Clarivate Analytics offers not just simple counts and averages, butreal “metrics” founded on baselines for comparison and normalized statistics.

Clarivate AnalyticsExpertise and Processing

Research Analytics Resources

Data

Address UnificationData Cleansing

& Standardization

Normalization and Baselines

Web of Science

Web of Science Core Collection Journal Selection Policy

Journals are selected by experts with:

150 years of experience

Fluency in 12 languages

0 affiliations with any journals or publishers

Publishing Standards

Editorial Content

International Focus

Citation Analysis

Four Points of Evaluation

• A complex process: no one factor is considered in isolation.

• Each journal is evaluated on its own merits, with an objective, unbiased approach.

• Covered titles are monitored to ensure that they maintain performance, and de-selected .

More information available in the Journal Selection Process Essay

InCites & the Web of Science Core Collection

INCITES PLATFORMWEB OF SCIENCE PLATFORM

Journal Citation Reports• Impact Factor - uses data from most

recent JCR data year + prior 2 years

Essential Science Indicators• Data from most recent 10 year period• Hot & Highly Cited papers in 22

categories

Benchmarking & Analytics• Data from most recent 35 year period

Web of Science Core Collection• Science Citation Index Expanded • Social Sciences Citation Index• Arts & Humanities Citation Index• Conference Proceedings Citation Index• Book Citation Index

• Emerging Sources Citation Index

Web of Science Core CollectionThe world’s most trusted citation index.

Master Journal List | Conferences List | Book List

Powerful bibliographic and cited reference search capabilities, together with the benefits of cited reference linking and navigation.

Cover-to-cover indexing of over 18,000 journals

Over 12,000 conferences covered, annually

Over 80,000 scholarly books

Science Citation

Index Expanded

Social Sciences Citation

Index

Arts and Humanities

Citation Index

Emerging Sources Citation

Index

Conference Proceedings

Citation Index

Book Citation

Index

Multidisciplinary International Influential

Save to InCites from Web of Science Core Collection

— See normalized metrics for each paper in your results set to put citation counts into context by field & journal

— Identify experts & centers of excellence on your topic of interest— View publication trends by country to understand the global research

landscape specific to your area of interest— Datasets: 50,000 documents (search results) or 50,000 documents from

Marked List.• 50,000 counted against your sort on the WoS results page • Content restricted to the Core Collection 1980 to present

20Custom datasets

per account

InCites Benchmarking & Analytics Current Dataset (February 2018)The data included in InCites Benchmarking & Analytics:

• Data updated monthly

• Web of Science Core Collection content indexed through the week of December 31, 2017

• Standard Address Unification (Organization-Enhanced)

Parameters Values

Source Editions SCI-E, SSCI, A&H, CPCI, CPCI-H, BKCI, BKCI-H

Citing Editions ALL

Document Types ALL

Source & Citing Years(publication date) 1980-2017

17

Which academic, government and industry groups produce the most impactful work in my specialty?

Research Impact Analysis & Visualization

Top 10 organizations publishing in immuno-oncology by Category Normalized Citation Impact (CNCI)

This graph shows organizations that published at least 50 oncology papers during the past 10 years (2007-2018) ranked by Category Normalized Citation Impact. All are producing research cited a rate far above average. Research from Bristol Meyers Squibb has been cited at over 11 times the average expected rate.1.0 = average impact in

field

18

Where are the centers of excellence in a particular region?Research Impact Analysis & Visualization

Top 10 Chinese universities publishing in nanomaterials by Category Normalized Citation Impact (CNCI)

This tree map shows organizations in China that published at least 200 nanomaterials papers during the past 10 years (2007-2018) ranked by Category Normalized Citation Impact.

You can use this information to target the top institutions in any region, globally, for partnership.

19

Which organizations produce the hottest research? Who is funding the hottest research?

Research Impact Analysis & Visualization

Hot Papers are papers published in the last two years, that have accrued the most citations in the past two months–they represent the top 0.1% of recently cited papers.

You can use this data to understand where the leading research in your area is being produced, and who is funding it.

Organizations publishinghot papers in immuno-oncology

Number of Hot Papers

Harvard University 7UNICANCER 7Yale University 6Gustave Roussy 6

VA Boston Healthcare System 6

University of California Los Angeles 5

Dana-Farber Cancer Institute 5Memorial Sloan Kettering Cancer Center 5

University of Chicago 4University of North Carolina Chapel Hill 4

Vanderbilt University 4

UTMD Anderson Cancer Center 4

Merck & Company 4

Organizations funding*hot papers in immuno-oncology

Number of Hot Papers

National Institutes of Health (NIH) - USA 14NIH National Cancer Institute (NCI) 8Merck & Company 3KWF Kankerbestrijding 2European Research Council (ERC) 2

*Funding data is derived from the funding acknowledgments in the journal articles.

20

Which KoLs have produced the most impactful work?KoL Assessment & Reporting

Top 10 Global Authors publishing on fuel efficiency by CNCI

1.0 = average impact in field

Rank Name Affiliation CountryWeb of Science

Documents

Category Normalized

Citation Impact

H-Index

1 Yang, Peidong Lawrence Berkeley National Laboratory USA 13 13.23 10

2 Baek, Jong-Beom Ulsan National Institute of Science & Technology SOUTH KOREA 11 12.3 9

3 Yang, Peidong University of California Berkeley USA 15 12.11 114 Dai, Liming Case Western Reserve University USA 27 10.63 205 Qiao, Shi Zhang University of Adelaide AUSTRALIA 11 10.52 106 Qu, Liangti Beijing Institute of Technology CHINA 12 9.55 97 Yu, Shu-Hong University of Science & Technology of China CHINA 17 8.86 138 Klemes, Jiri Jaromir University of Pannonia HUNGARY 10 8.62 79 Lewis, Nathan S. California Institute of Technology USA 27 8.09 1410 Jaroniec, Mietek Kent State University USA 12 8.06 11

This table shows authors that have published at least 10 papers on fuel efficiency between 2008-2017, ranked by Category Normalized Citation Impact. All are producing research cited a rate far above average.

You can use this data to inform: Recruitment Strategic partnerships

21

Who are the KoLs in a particular region?KoL Assessment & Reporting

Quickly locate experts publishing on any topic.

You can focus your search to any country or region.

22

What is the funding landscape for my specialty?Research Funding Analysis & Visualization

*Data derived from the funding acknowledgments in the journal articles

Top 10 organizations funding* immuno-oncology research by number of funded documents

You can instantly see which agencies, foundations and corporations fund work in your area.

Over 900 unified funders are currently available for analysis, with more added monthly.

Funding details are available for material from 2008-present.

23

Which institutions and authors are benefitting from which agency?Research Funding Analysis & Visualization

*Data derived from the funding acknowledgments in the journal articles

CNCI of Top 10 organizations funding* immuno-oncology research

BMS-Funded Institutions (Top 10 by output)

Web of Science

DocumentsMemorial Sloan Kettering Cancer Center 52

Harvard University 41Bristol Myers Squibb 40Johns Hopkins University 35VA Boston Healthcare System 34

Dana-Farber Cancer Institute 31Johns Hopkins Oncology Center 30

H Lee Moffitt Cancer Center & Research Institute 23

UTMD Anderson Cancer Center 22

Cornell University 201.0 = average impact in field

24Collaboration Analysis & Visualization

Industry Academia Collaboration-Which collaborations have the most impact?

BASF• Collaborating organizations

defined as co-authorship• Ranked clockwise by # co-

authored papers• Displaying CNCI

Top BASF collaborations in nanomaterials researchby number of co-authored documents

Quickly visualize your existing partnerships & determine which have the highest impact in science.

25Collaboration Analysis & Visualization

Industry Academia Collaboration-Which academic institutions is my competitor working with? Which collaborations have the most impact?

DuPont• Collaborating organizations defined

as co-authorship• Ranked clockwise by # co-authored

papers• Displaying CNCI

Top Dupont collaborations in nanomaterials researchby number of co-authored documents

Are you missing out on expertise that your competitors are exploiting?

26Collaboration Analysis & Visualization

Which corporations is a university working with? Which collaborations have the most impact?

Johns Hopkins Oncology Center • Collaborating organizations defined as co-

authorship• Ranked clockwise by # co-authored papers• Displaying CNCI

Top Johns Hopkins Oncology Center Collaborations in immuno-oncology by number of co-authored documents

View the extent of a university’s existing corporate partnerships to gauge opportunity.

Understand how your organization fits in with other research activities at the target.

27

InCites Examples with Screenshots

You can easily visualize what filter you have applied to your analysis

You can clear your filter in one click

All filter sections are expandable

You can choose two different metrics to:- Rank your results- Generate the visualization

Use this to e.g. Avoid having to use a thresholdAn item can be searched in the

list.Use this to search your

institution

Time period selector is a slider.Use this to avoid “unfinished” years

such as 2019

Define a threshold of number of WoS Documents or Times cited.

Very useful to avoid meaningless reports based on too few documents/citations

You can of course combine filters to demonstrate the infinite amount of angles one can use

I want to find all German institutions, NOT

corporateand THE ranked, that have

published at least 30Karticles and reviews in the

past five years

Remove any item from a list, just tick the box next to it and select “exclude from results”

This is particularly useful when one of the institution listed is part of the analyzed institution (e.g. University of California System)

For example, you can analyze the journals where an institution has published.

Instead of typing its name in the “Journals” section, you can simply click on its name in the “organizations” section, then refocus on “Journals”.

Demonstrating how you can bring a selection of documents to another section can be very impressive.

In this list of five institutions that have published at least 200K papers in the last

ten years, which one has the highest Normalized citation Impact?

In this list of five institutions that have published at least 200K papers in the last

ten years, which one has the highest proportion of Cited Documents?

Make sure you click on this to access a multitude of additional indicators

Click on “Browse Indicators” to add indicators to your report

In this list of five institutions that have published at least 200K papers in the

last ten years, which one has the biggest proportion of international

collaborations?

In this list of five institutions that have published at least 200K

papers in the last ten years, which one has the biggest H Index?

Filtering by research areas is now possible with a multitude of schema

In this list of five institutions that have published at least 200K papers in the last ten years,

which one is the most influential, in Physics, as defined by ESI?

!You need to “PIN TO TOP” these 5 institutions before setting the document

threshold back to zero!

Which one has the biggest Open Access output?

Which one has the most papers in

specific journals?

Which one has published articles

with one of the three 2013

chemistry Nobel prices?

Which one gets the biggest impact

when collaborating with

India?

you have to enter its name in the “Collaborations with Organizations”

section

OR

You can simply click on the name of the institution in a given list, then refocus the

analysis using “collaborating organizations”.

To study collaborations with a specific institution

!! When you use “Refocus”, try to do it from a list coming from a small set of filters, otherwise the refocus will take too long and/or not work at all !!

The German and UK institutions that collaborated with CNRS in the past 10 years

From this list, I can generate reports on as much as 51 indicators

The 10 journals in which University of California

Berkeley has published the most.

How much of these papers got cited?

If Tetrahedron decided to not

publish any author from a German

institution, what would happen?

What happened when Tetrahedron published a Nobel

Prize winner’s articles?

Where did this person publish in the last 10

years?

What are the journals classified

in the FAPESP Chemistry

category, and what are their impact performances?

The 10 countries publishing the most of Physics Condensed

Matter papers.

Are these papers coming from collaborations with industry

sector?

The 5 countries publishing the most of Physics

Condensed Matter papers.

Do they publish a lot of Highly Cited Papers?

What are the countries

publishing in Nature?

What are the countries

collaborating with Chinese

institutions?What are the

performances of these

collaborations?

Does UC Berkeley collaborate

internationally?Does it help in

terms of impact?

Are these 6 countries

publishing in this OECD subject

category?

The 10 disciplines (SCADC) in which Chinese institutions

publish the most.

What is their performance in terms of percentiles?

The 10 disciplines (SCADC) in which Chinese institutions publish the most.

Did they recently produce a lot of Hot Papers?

Did the world publish more in

Organic or Inorganic

Chemistry?Which subject get

more impact?

In which disciplines did this group (department?)

publish the most?

In Which OECD category can we

find UPMC’s publications?

Any strength or weakness?

In which subject did these 4 major players publish at least 300K papers

in the last 10 years?

Subhasree Nag, Sr. Solutions Consultant| subhasree.nag@clarivate.com | clarivate.com