David A. PendleburyAnalystInstitute for Scientific Information

What it Really Means to be Highly Cited and its Impact

2

Our Guests: Professors Katsuhiko Ariga and Ruth Feldman

Katsuhiko Ariga, Ph.D. World Premier International (WPI)

Research Center for Materials Nanoarchitectonics (MANA), National Institute for Materials Science, Tsukuba, Japan

Department of Advanced Materials Science, Graduate School of Frontier Sciences, University of Tokyo, Japan

Ruth Feldman, Ph.D. Center for Developmental,

Social, and Relationship Neuroscience, Interdisciplinary Center, Herzliya, Israel

Yale University Child Study Center, New Haven, CT, USA

3

David Pendlebury,Institute for Scientific Information, Clarivate Analytics:

Analysis of Research Structure and Dynamics,Citation Analysis in Evaluation

I. Introduction• What it means to be highly cited• The skewed distribution of citations in the literature• What this implies about talent and competition

5Information retrieval:Eugene Garfield, “Citation indexes for science: a new dimension in documentation through association of ideas,” Science, 122 (3159): 108-11, 15 July 1955

History and sociology of science:Eugene Garfield, “Citation indexes in sociological and historical research,” American Documentation, 14 (4): 289-291, 1963

Eugene Garfield, “Citation indexing for studying science,” Nature, 227 (5259): 669-671, 1970

Father of Citation Indexing and Analysis

Structure and dynamics of science:Eugene Garfield, M.V. Malin, and H. Small, "A system for automatic classification of scientific literature," Journal of the Indian Institute of Science, 57 (2): 61-74, 1975

Evaluation of research:Eugene Garfield, “Is citation analysis a legitimate evaluation tool?,” Scientometrics, 1 (4): 359-375, 1979

6

Theories of Citation and the Normative School Robert K. Merton, (1910-2003),

sociologist of science, Columbia University. Normative theory.

Citations as currency used to repay intellectual debts. Those with many citations have gained “credits” from their peers.

The formal nature of publication and the moral imperative to cite.

Other theories, including citations as rhetorical devices, constructivist theories.

Known for coining the concepts and phrases: “self-fulfilling prophecy,” “role model,” “focus group,” “unanticipated consequences,” and “Matthew effect.”

7

Citations Distributions are Skewed: People and PapersFirst Name Last Name CitationsGeorge M. Sheldrick 52551Michael Graetzel 37409Rodney S. Ruoff 35159Omar M. Yaghi 24221SonBinh T. Nguyen 23119Donald G. Truhlar 22600Stefan Grimme 20082Mohammad Khaja Nazeeruddin 18578Jeffrey R. Long 18327Richard D. Piner 16480Avelino Corma 16245Yan Zhao 16080Hong-Cai Zhou 15730Sasha Stankovich 15586Michael O'Keeffe 15578Hui-Ming Cheng 15455Hongjie Dai 15385Dmitriy A. Dikin 15142Joseph T. Hupp 14806Jin-Quan Yu 14087Prashant V. Kamat 13926Younan Xia 13650Jiaguo Yu 13228Jean-Marie Tarascon 12680Gao Qing (Max) Lu 12576

Citations are created in a socio-cognitive context, in which contributions of value are recognized and those recognized then tend to attract more citations: cumulative advantage, preferential attachment, success breeds success, and Merton’s Matthew effect

Power law probability distribution: Pareto’s description of income: 80/20

8

Many Papers Published But Very Few Are Highly Cited(Web of Science, 1970-2017, Articles and Proceedings only)

Citations Number in Range Cumulative Count

100,000 - 331,679 3 350,000 - 99,999 9 1210,000 - 49,999 188 2005,000 - 9,999 543 7433,000 - 4,999 1,214 1,9572,000 - 2,999 2,366 4,3231,000 - 1,999 12,215 16,538500 - 999 48,243 64,7810 - 499 45,538,186 45,602,967

Only 2,000 Papers Cited 3,000 or more timesOnly 4,300 Cited 2,000 or more times

.01%

9

10

Successful Forecasts: 50 Citation Laureates Nobel Laureates2002: Daniel Kahneman, Economics (2002)2003: Robert F. Engle and Clive W.J. Granger, Economics (2003)2005: Robert H. Grubbs, Chemistry (2003)2007: Mario Capecchi, Martin J. Evans, and Oliver Smithies, Medicine (2006); Albert Fert and Peter Grünberg, Physics (2006)2008: Paul Krugman, Economics (2006); Roger Y. Tsien, Chemistry (2008)2009: Elizabeth H. Blackburn, Carol W. Greider, and Jack W. Szostak, Medicine (2009); Oliver E. Williamson, Economics (2006)2010: Andre K. Geim and Konstantin Novoselov, Physics (2008)2011: Bruce A. Beutler and Jules A. Hoffmann (2008), and Ralph M. Steinman, Medicine (2010); Saul Perlmutter, Adam G. Riess, and Brian P. Schmidt, Physics (2010); Dan Shechtman, Chemistry (selected for Physics in 2008); Thomas J. Sargent and Christopher A. Sims, Economics (2008)2012: Shinya Yamanaka, Medicine (2010)2013: James E. Rothman and Randy W. Schekman, Medicine (2009); Francois Englert and Peter W. Higgs, Physics (2013); Martin Karplus, Chemistry (2011); Eugene F. Fama (2002) and Lars Peter Hansen (2008) and Robert J. Shiller (2012), Economics2014: Shuji Nakamura, Physics (2002); Jean Tirole, Economics (2007)2015: Arthur B. McDonald, Physics (2007); Angus Deaton, Economics (2012)2016: Yoshinori Ohsumi, Medicine (2013); J. Fraser Stoddart , Chemistry (2002); Oliver D. Hart and Bengt R. Holmström, Economics (2006)2017: Rainer Weiss and Kip S. Thorne, Physics (2016); Richard H. Thaler, Economics (2002)2018: James P. Allison and Tasuku Honjo, Medicine (2016); Paul Romer, Economics (2005) and William D. Nordhaus, Economics (2009)

Year in parenthesis is year individual was named a Clarivate Analytics Citation Laureate and forecasted to win a Nobel Prize. Orange same year; blue within 2 years; green within 3 years

11

Sir Francis Galton, 1822 - 1911Statistician, Explorer, Sociologist, Inventor,

Meteorologist, Pioneer Geneticist, Psychometrician,

…Polymath

Known for: Nature vs. Nurture, standard

deviation, regression to the mean, twin studies, fingerprint patterns, etc.

English Men of Science: Their Nature and Nurture (1874)

“It is of interest to know the ratio which the numbers of the leading scientific men bear to the population of England generally….one in 10,000 [i.e., .01%]”

12

Powers of Ten: A Hierarchy of Achievement and Status

0.01% Eminent, Illustrious, Nobel Prize recipients

0.1% Distinguished, Esteemed, members of national academies of sciences

1% Exceptional, Gifted, recognized leaders in specialty areas

10% Talented, Accomplished, up-and-coming juniors and established seniors

100% Professional, Qualified, academic faculty, and full-time equivalents in gov't labs and industry

13

Talent and Competition Citation data demonstrate the rarity

of top papers and top people Bidding for researchers who are leaders

in their specialties – especially in the biomedical sciences – has increased sharply since the late 1980s and early 1990s

New Public Management, introduced in universities in the United States about the same time, has employed business models into academia and emphasized performance indicators including publication and citation measures

Scientists, who previously rejected evaluation by outsiders, have gradually accepted bibliometric assessments, because opportunities and rewards tied to such assessments have become institutionalized. Some researchers now list citation indicators on their CVs and websites, such as the h index

• History of Highly Cited Researchers

• Methodology• Used in Shanghai Ranking

(ARWU), Consulted for Recruitment, Promotions

II. Highly CitedResearchers, 2018

15History of Identification of Highly Cited Researchers Hundreds of essays and articles by company founder Eugene Garfield

highlighting most cited researchers and Nobel Prize winners: http://www.garfield.library.upenn.edu/

List created in 2001 and updated in 2003 including about 300 authors in each of 21 broad fields (similar to those used in Essential Science Indicators). Method: the period surveyed was 1981 – 2001 (2003) and authors were ranked and selected by total citations. This list favored senior researchers with many years of publications and citations.

In 2014 we released a new list of Highly Cited Researchers using a different methodology.

Period surveyed: 11 years (initially 2002-2012)Fields analyzed: 21 broad fields used in Essential Science IndicatorsSelection method: those with greatest number of highly cited papers in field, top 1% in citation distribution for papers of same field and publication year.Number selected: Square root of number of publishing authors in field (= elite). This method focused on more contemporary impact, admitted younger as well as senior researchers, and was normalized for field and age as well as field size to obtain an equal top cut of top performers 2015 (2003-2013), 2016 (2004-2014), 2017 (2005-2015), 2018 (2006-2016) In 2018, addition of Cross-Field category, with increase of 50% named

researchers.

16

6,077 Researchers Selected as Highly Cited for 2018

17Nobel Laureates and Citation Laureates in the 2018 Highly Cited Researchers ListNobel Laureates:James P. Allison, David Baltimore, Bruce Beutler, Albert Fert, Andre K. Geim, James J. Heckman, Allan J. Heeger, Brian K. Kobilka, Robert J. Lefkowitz, Edward I. Moser, May-Britt Moser, William Nordhaus, Kostya S. Novoselov, Elinor Ostrom, J. Fraser Stoddart, Jean Tirole, Susumu Tonegawa

Citation Laureates:Shizuo Akira, Phaedon Avouris, David Awschalom, Jeffrey A. Bluestone, Louis E. Brus, Emmanuelle Charpentier, George M. Church, Juan Ignacio Cirac, Marvin L. Cohen, Francis S. Collins, Cees Dekker, Avinash K. Dixit, Jennifer A. Doudna, Gordon J. Freeman, Karl J. Friston, Fred H. Gage, Yury Gogotsi, John B. Goodenough, Jeffrey I. Gordon, Michael Graetzel, Hideo Hosono, Deborah S. Jin, Charles L. Kane, Susumu Kitagawa, Geoffrey W. Marcy, Tobin J. Marks, Joan Massague, Paul L. McEuen, Chad A. Mirkin, Mizushima Noboru, John B. Pendry, M. Hashem Pesaran, Richard Peto, Ramesh Ramamoorthy, Alexander Y. Rudensky, Rodney S. Ruoff, David M. Sabatini, Shimon Sakaguchi, Charles L. Sawyers, Stephen W. Scherer, Arlene H. Sharpe, Henry J. Snaith, Craig B. Thompson, Yoshinori Tokura, Bert Vogelstein, George M. Whitesides, Omar M. Yaghi, Shoucheng Zhang, Peter Zoller

18Highly Cited Researcher Data Used in Academic Ranking of World Universities (Shanghai Ranking): 20% Weighting

Criteria Indicator Weight

Quality of Education Alumni of an institution winning Nobel Prizes and Fields Medals 10%

Quality of Faculty

Staff of an institution winning Nobel Prizes and Fields Medals 20%

Highly cited researchers in 21 broad subject categories 20%

Research Output

Papers published in Nature and Science* 20%

Papers indexed in Science Citation Index-expanded and Social Science Citation Index 20%

Per Capita Performance Per capita academic performance of an institution 10%

Total 100%

III. Interview:Professor Katsuhiko Ariga

20

Highly Cited Researcher in Materials Science (2014-2018) and in Chemistry (2018)

Year Chemistry Materials Science2007 12008 120092010 2 12011 12012 4 32013 2 22014 3 12015 1 12016 4 4

21

III. Interview:Professor Katsuhiko Ariga

IV. New for Highly Cited Researchers 2018:Identification of Researchers with Cross-Field Impact

24

From the 2017 Highly Cited Researcher Website’s Frequently Asked Questions:

25

Cross-field Impact: Calculating an Equivalent Impact to That in One or More ESI Fields

The challenge: To find a method that took account of the different threshold number of highly cited papers in each field so that those contributing papers in several fields could be compared in an equal manner with those selected in one or more ESI fields. The solution chosen was to fractionate the credit for each highly cited paper such that a paper in a field with a high threshold number of papers was weighted less than a paper in a field with a lower threshold number of papers.

26

Who is Joseph Savant?

David BaltimoreRobert Andrews Millikan Professor of Biology, CaltechRecipient 1975 Nobel Prize in Physiology or MedicineRecipient 1999 U.S. National Medal of SciencePresident of Caltech, 1997-2005

27

The UNESCO World Science Report 2015 estimated the number of full-time researchers at 7.8 million: .1% of the world population. The 2018 Highly Cited Researchers, numbering 6,077, is (by rounding up from .08%) about .1% of the 7.8 million.

ESI FieldHighly Cited Researchers

Agricultural Sciences 158Biology & Biochemistry 254Chemistry 261Clinical Medicine 497Computer Science 95Economics & Business 96Engineering 204Environment/Ecology 185Geosciences 184Immunology 146Materials Science 208Mathematics 90Microbiology 148Molecular Biology & Genetics 249Neuroscience & Behavior 197Pharmacology/Toxicology 161Physics 211Plant & Animal Sciences 223Psychiatry/Psychology 157Social Sciences, General 211Space Science 122Total 4057Cross-Field 2020Grand Total 6077

Addition of Highly Cited Researchers with Cross-Field Impact Increases Selectee Group by 50%

The characteristics of the new Cross-Field Highly Cited Researchers closely resemble those chosen in specific fields

V. Interview:Professor Ruth Feldman

29

A highly interdisciplinary, cross-field research program: brain science, childhood development, immunology, psychology, epidemiology, etc.

30

Feldman’s Highly Cited Papers Classified as Neuroscience & Behavior

Article is classified as Neuroscience based on analysis of cited references

31

Feldman’s Highly Cited Papers Classified as Psychiatry/Psychology

V. Interview:Professor Ruth Feldman

VI. Use of Citation Data in the Evaluation of Researchers: Some Words of Caution and of Benefits

34

On the Use of Bibliometric Data in Identifying Research Excellence, Especially in Terms of Individuals

There is no unique indicator or universally agreed concept of what constitutes exceptional research performance and elite status in the sciences and social sciences.

Consequently, no quantitative method will point to a list that satisfies all expectations or requirements. Moreover, a different basis or formula for selection could generate a different – though likely overlapping – list of names.

Thus, the absence of a name on our list cannot be interpreted as inferior performance or stature in comparison to those selected.

The absence of evidence is not evidence of absence

35Indicators of Activity vs. Activity Driven by Indicators

Formulaic application of citation data is not good practice

Such evaluation – particularly when linked to rewards – may change behaviors, producing the negative unintended consequence that the scores or ranks become a goal instead of real improvement in activity

Goodhart’s Law: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”

Campbell’s Law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

The concern: that number of HCRs at a university is the only use of the data, and that HCR status becomes a goal (and subject to gaming) rather than a consequence of research excellence

36Citation Analysis, Carefully Applied, Adds an Expanded Dimension to Research Evaluation of Individuals, an

Important Check on Too-Narrow or Biased Peer Review Citation data at high frequency provide strong evidence of exceptional research impact.

But use of the data or the designation of Highly Cited requires more investigation of a researcher’s work.

Publication and citation data demand informed and usually limited application, and only as a complement to peer review. One should NOT rely on publication and citation data as a substitute for reading and assessing a researcher’s contribution and for human judgment.

Single measure indicators, such as the h index, are inadequate and unidimensional. In particular, false precision – making distinctions without a difference – should should be avoided.

Quantitative assessment, as a supplement to peer review, sometimes increases fairness.

“When used properly, citation analyses can help members of the faculty evaluation committee develop better informed opinions about candidates. Objective information about the usefulness of a scientist’s work to the scientific community is to be welcomed. Careful analysis of the candidate’s publications can confirm or contradict the conclusions of a faculty review committee.” – Eugene Garfield, Institute for Scientific Information