Thomson Reuters and Research Evaluation in the Humanities
James Pringle, VP, Development
Academic & Government, Thomson Reuters Scientific
September 19, 2008
Citations in the 21st Century
• The “User-Generated Content” of formal scholarly communications
• Fundamental to effective evaluation & management
• Frequently mis-used and misunderstood
• May have a place in humanities research evaluation
3
Changing Scholarly Communications Value Chain
A Contentious Issue
“…impact factors and citations are dominating minds, distorting behaviour and determining careers….
…The main villains are fashion, the management cult, and the politics of our time, all of which favour numerical evaluation of ‘performance’ and reward compliance.”
--Lawrence, P. A. (2007).
The Management Challenge
• “Metrics democratization” leads to confusion
• Reasonable efforts by university administrators to derive objective measures for promotion and tenure decisions free of bias.
• Pressures on these administrators to upgrade the reputation and quality of their programs to compete in global academic markets.
• Global competition in scholarship, leading to national policies designed to increase global competitiveness and demonstrate national achievement.
Some Rules for Good Use of Citation Data
• Consider whether available data can address question
• Compare like with like
• Use relative measures, not just absolute counts
• Recognize skewed nature of citation data
• Confirm data collected are relevant to question
• Ask whether the results are reasonable
---Using Bibliometrics in Evaluating Research
http://scientific.thomsonreuters.com/bibliometrics2/
7
Institutional Research Evaluation Workflow
Supporting evaluation workflow:
Tools
Content
Workflow Services
8
Government agencies/funding organizations
IndividualsFaculty, staff, students
University DepartmentsInstitutional research, academic
affairs, tech transfer, etc.
We want to know how the funds that we have granted have been effective in promoting research.
External Entities
University ManagementManagement, including committees,
provost, vice provosts
I’m responsible for seeing how we’re doing overall in performing in terms of bringing in research dollars in overall publications.”
For those aspirational peers...I want to go in and evaluate the faculty and those departments to give me a direct comparison, then I get every single faculty member and run them through the citation index”
“We have built a research information system which is fully integrated with all of the systems—library, research office, grants, and the university -- it’s CV driven, and so every researcher in the institution can access a web based, online view and over time create a research profile for themselves containing up-to-date publications”
“Every author should have a unique identifier in order to retrieve unequivocally his/her articles.
Citation Metrics in Research Evaluation
9
US, NSF: biennial Science & Engineering Indicators report (1974 - )
European Union: EC’s DG XII (Research Directorate)
UK: Office of Science & Technology; Higher Education Funding Council
Canada: NSERC, FRSQ (Quebec), Alberta Research Council
France: Min. de la Recherche, OST - Paris, CNRS
Germany: Max Planck Society, several gov’t labs, DKFZ, MDC
Italy: CRUI (University Rectors) MURST (Ministry of Research, CNR
Spain: CSIC (Spanish Science Agency), CIRIT (Catalonia)
Japan: National Institute of Informatics, Ministry of Education, Ministry of Economy, Trade & Industry
People’s Republic of China: Chinese Academy of Science
Korea: Korea Research Foundation, Korea Advanced Inst. Of S&T
Australia: Australian Academy of Science, gov’t lab CSIRO
Turkey: ULAKBIM
Others include: New Zealand, S. Africa, Portugal, Ireland, Switzerland, Austria, Poland, Czech Republic, Singapore, Malaysia, Thailand, Sweden, Norway, Denmark, Finland, Mexico, Brazil, Chile, Argentina, Uruguay, Russia… and more!
Nations and Agencies using TS Evaluative Reports
Citations and Evaluation in the Humanities• Medium of Scholarly Communication
– Importance of monograph (e.g. Cronin & La Barre, 2004)
• Disciplinary Role of Citations– Intellectual debt or literary artifact?
• Usefulness of journal citations in humanities evaluation– High level of citations to non-journal literature (e.g. Larivere, 2006)
– Useful in aggregate for specific disciplines? (e.g. Oppenheim, 2008)
– Useful as indicators of non-journal literature? (e.g. Butler, 2006)
• Characteristics of the ISI Citation Databases– Regional coverage issues (Archimbault, 2006; Nederhof, 2006)
– Increasing Coverage in ISI Citation Databases
10
Growth of Journal Coverage in WoS: 2002-2008
Web of Science
2002 2008 Growth Variance
Total Journal Coverage
8,607 10,185 1,578 +18%
11
Growth of Journal Coverage in WoS Focus: Regions
Web of Science: Regions
2002 2008 Growth Variance
Asia Pacific 497 767 270 +54%
European Union
1684 2155 471 +28%
Middle East/Africa
54 109 55 +102%
Latin America
63 160 97 +154%
12
Growth of Journals in Arts & Humanities 2002-2008
2002 2008 Growth Variance
A&H Coverage 1122 1238 116 10%
Overall Growth:
Regional Growth:Region 2002 2008 Growth Variance
Asia Pacific 27 40 13 48%
European Union
532 626 94 18%
Latin America 7 16 9 129%
Middle East/Africa
8 15 7 88%
North America 548 541 -7 -1%
Growth of Papers in Humanities Fields1981-2007
15
ERIH Category # ERIH Titles (Total)
# Shared w/ WoS
Percent in WoS
Anthropology, Evolutionary 36 27 75%Anthropology, Social 195 54 28%Archaeology 390 50 13%Art, Architectural & Design history 418 76 18%Classical Studies 208 41 20%Gender Studies 108 42 39%History 757 192 25%History & Philosophy of Science 146 55 38%Linguistics 542 149 27%Literature 728 215 30%Musicology 160 58 36%Pedagogical & Educational Research
434 151 35%
Philosophy 246 89 36%Psychology 532 503 95%Religious Studies_Theology 305 58 19%Total 5205 1760 34%
Comparison with European Reference Index for the Humanities (ERIH) Journal List
European Reference Index for the Humanities (ERIH), http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih/erih-initial-lists.html
16
ERIH Category ERIH "A List" Titles
Included in WoS
Percent of "A List" in WoS
Anthropology, Evolutionary 8 8 100%Anthropology, Social 29 19 66%Archaeology 69 31 45%Art, Architectural & Design history
101 47 47%
Classical Studies 47 16 34%Gender Studies 28 21 75%History 98 72 73%History & Philosophy of Science
25 20 80%
Linguistics 83 55 66%Literature 170 93 55%Musicology 20 16 80%Pedagogical & Educational Research
73 50 68%
Philosophy 23 19 83%Psychology 184 184 100%Religious Studies_Theology 53 20 38%Total 1011 671 66%
Comparison with ERIH “A” List
17
ERIH Category ERIH "B List"
"B List" in WoS
Percent of "B List" in WoS
Anthropology, Evolutionary 20 17 85%Anthropology, Social 87 32 37%Archaeology 159 14 9%Art, Architectural & Design history
192 24 13%
Classical Studies 86 20 23%Gender Studies 54 19 35%History 303 98 32%History & Philosophy of Science
80 31 39%
Linguistics 195 69 35%Literature 343 104 30%Musicology 83 27 33%Pedagogical & Educational Research
274 97 35%
Philosophy 104 42 40%Psychology 264 260 98%Religious Studies_Theology 168 35 21%
Total 2412 889 37%
Comparison with ERIH “B” List
References• Archambault, E., Vignola-Gagne, E., Côté, G., Larivière, Gingras, Y., “Benchmarking Scientific Output in
the Social Sciences and Humanities: The Limits of Existing Databases,” Scientometrics, 68(3) 329-342, 2006.
• Butler, L., Visser, M., “Extending Citation Analysis to Non-Source Items,” Scientometrics, 66(2): 327-343, 2006.
• Cronin, B., and La Barre, K., “Mickey Mouse and Milton: Book Publishing in the Humanities”, Learned Publishing, 17(2): 85-98, 2004.
• European Reference Index for the Humanities (ERIH), http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih/erih-initial-lists.html
• Larivière, V., Archambault, E., Gingras, Y, Vignola-Gagné, “The Place of Serials in Referencing Practices: Comparing Natural Sciences and Engineering with Social Sciences and Humanities,” Journal of the American Society for Information Science and Technology, 57(8): 997-1004, 2006.
• Lawrence, P. A. (2007). "The Mismeasurement of Science." Current Biology 17(15): R583-585.
• Nederhof, A., “Bibliometric Monitoring of Research Performance in the Social Science and Humanities: A Review,” Scientometrics, 66(1) 81-100, 2006.
• Oppenheim, C., Summers, M., “Citation Counts and the Research Assessment Exercise, Part VI: Unit of Assessment 67 (Music),” Information Research—An International Electronic Journal, 13(2): 342, June 2008.
• Using Bibliometrics in Evaluating Research, http://scientific.thomsonreuters.com/bibliometrics2/
18
19
Thank You!