View
214
Download
0
Category
Preview:
Citation preview
The Multi‐Dimensional Research Assessment Matrix
Henk F. Moed
Elsevier, Amsterdam, NetherlandsLecture at ESSS Conference, Vienna, 12 Sept. 2011
A framework to characterize and position bibliometric indicators,
methods and products
A framework to characterize and position bibliometric methods and products
• The multi‐dimensional research assessment matrix
• Top‐down versus bottom‐up delimitation methods
• Three generations of bibliometric indicators
The future of research assessment exercises lies in the
intelligent combination of metrics and peer review
The Multi‐Dimensional Research Assessment Matrix
Expert Group on the Assessment of University-Based Research (AUBR, 2010)
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometricindicators
Other indicators
Individual Allocate resources
Researchproductivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journalcitation impact
Patents, licences, spin offs
Department Increase multi‐discipl.research
Innovation and social benefit
Actualcitation impact
Invitationsfor conferences
Institution Increaseregional engagement
Sustainabi‐lity & Scale
Internat. co‐authorship
Externalresearch income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com‐pletion rates
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometricindicators
Other indicators
Individual Allocate resources
Researchproductivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journalcitation impact
Patents, licences, spin offs
Department Increase multi‐discipl.research
Innovation and social benefit
Actualcitation impact
Invitationsfor conferences
Institution Increaseregional engagement
Sustainabi‐lity & Scale
Internat. co‐authorship
Externalresearch income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com‐pletion rates
Read column‐wise
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometricindicators
Other indicators
Individual Allocate resources
Researchproductivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journalcitation impact
Patents, licences, spin offs
Department Increase multi‐discipl.research
Innovation and social benefit
Actualcitation impact
Invitationsfor conferences
Institution Increaseregional engagement
Sustainabi‐lity & Scale
Internat. co‐authorship
Externalresearch income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com‐pletion rates
Types of research outputs
Impacts Publication/text Non-publication
Scientific-scholarly
Journal paper; book chapter; monograph
Research data file; video of experiment
Educational Teaching course book; syllabus
Skilled researchers
Economic Patent Product; process; device; design; image; spin off
Cultural Newspaper article Interviews; events; performances; exhibits
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometricindicators
Other indicators
Individual Allocate resources
Researchproductivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journalcitation impact
Patents, licences, spin offs
Department Increase multi‐discipl.research
Innovation and social benefit
Actualcitation impact
Invitationsfor conferences
Institution Increaseregional engagement
Sustainabi‐lity & Scale
Internat. co‐authorship
Externalresearch income
Research field
Promotion& hiring
Research infrastruct.
citation ‘prestige’
PhD com‐pletion rates
MD‐RAM: Example 1
Individual Hiring/promotionProductivity &
impactPhD date, place,
supervisor;Invitations for
conferences
Publications in international jrnls;
Actual citation impact
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometricindicators
Other indicators
Individual Allocate resources
Researchproductivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journalcitation impact
Patents, licences, spin offs
Department Increase multi‐discipl.research
Innovation and social benefit
Actualcitation impact
Invitationsfor conferences
Institution Increaseregional engagement
Sustainabi‐lity & Scale
Internat. co‐authorship
Externalresearch income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com‐pletion rates
MD‐RAM: Example 2
Research group Allocate resourcesRes. productivity &
impactCompetitive
research income;Ratio research
active/total staff
Publications in international jrnls;
Actual citation impact
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometricindicators
Other indicators
Individual Allocate resources
Researchproductivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journalcitation impact
Patents, licences, spin offs
Depart‐ment
Increase multi‐discipl.research
Innovationand social benefit
Actualcitation impact
Invitationsfor conferences
Institution Increaseregional engagemnt
Sustainabi‐lity & Scale
Internat. co‐authorship
Externalresearch income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com‐pletion rates
MD‐RAM: Example 3
DepartmentIncrease regional
engagementSocial benefits &
innovation% Funding from
endusers;Licenses, spin
offs
% Industry co-authored papers;
PatentsRes. reports
Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment
Purpose Output dimensions
Bibliometricindicators
Other indicators
Individual Allocate resources
Researchproductivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journalcitation impact
Patents, licences, spin offs
Department Increase multi‐disc. research
Innovation and social benefit
Actualcitation impact
Invitationsfor conferences
Institution Increaseregional engagement
Sustainabi‐lity & Scale
Internat. co‐authorship
Externalresearch income
Research field
Promotion, hiring
Research infrastruct.
citation ‘prestige’
PhD com‐pletion rates
MD‐RAM: Example 4
InstitutionIncrease multi-
disciplinary researchRes. productivity &
impactInter-discipl.
teaching progr;Funding from
multi-disc. res. progr
% Papers in multi-disciplinary fields;
Co-authorships from different discipl
Positioning bibliometric products ‐ 1
Unit of assessment
Purpose Aspect Product
Researchgroup
Improve performance;Allocate resources
Productivity/ impact A
Department; institution
Allocate resources; Stimulate multi-disciplinary research
Researchinfrastructure and potential
B
Institution Stimulate multi-disciplinary research;Improve performance;
Social benefits; sustainability;Prod./impact
C
Group, Dept, Institution
Improve performance& regional engagement;Allocate resources;
Productivity/ impact; social impact; infrastr.
D
Top-down vs. bottom-up institutional delimitation
Top‐down institutional analysis
Select an institution’s papers using author affiliations (incl. verification)
Categorize articles intoresearch fields
Calculate indicators
Compare with benchmarks
Bottom‐up institutional analysis
Compile a list of researchers
Compile a list of publications per
researcher (incl. verification)
Aggregate researchers into groups, departments, fields, etc.
Calculate indicators;
compare with benchmarks
159 NL Academic Chemistry Groups
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
0 200 400 600 800
Total Publications
Nor
mal
ised
Cita
tion
Impa
ct
1.0 =World Average
Total Articles (10 yrs)
Normalised
Citation Im
pact
Top-down vs. bottom-up subject field delimitation
Top‐down field analysis
(Pre‐)define a field categorization
(Pre‐)allocate papers (journals)
to categories
Calculate indicators
per category
Bottom‐up field analysis
Create a (large) dataset
of publications
Cluster papers on the basis of
(co‐) citation or word analysis
Name the clusters
Calculate indicators by cluster
ST
RA
TE
GIC
DIA
GR
AM
Co‐Word Analysis (COPALRED, prof. R. Bailon, Univ Granada)
Library Catalog AnalysisLibrary Catalog Analysis2. RESULTS Library Catalog Analysis
Positioning bibliometric products – 2
Institutional delimitation
Semantic/field delimitation
Product or service
Top‐down Top‐down A
Top‐down Bottom‐up B
Bottom‐up Top‐down C
Bottom‐up Bottom‐up D
Other aspects
• Which are the principal user groups?
• Relationship to peer review
• Time window covered
• Focus on top vs. bottom of the quality distribution
Three generations of bibliometric indicators
Three generations of bibliometric indicators
Generation Indicators (Examples)
1 Basic No. publications No. CitationsCites per paper (e.g., Jrnl Impact Factor)
2 Normalized Relative Citation Rate No. publ. in top citation percentiles
3 Contextual Entire citation network based (e.g., Eigenfactor, SJR, Jrnl Prestige);
Citation context based
Differences in citation potential between fields
Molecular Biology Mathematics
0
10
20
30
40
50
0 1 2 3 4 5 6 70
10
20
30
40
50
0 1 2 3 4 5 6 7
Number of received citations
%
Papers
Refe-rencelists
Relative Citation Rate (RCR)
The average citation rate of a unit’s papers
÷world citation average in the subfields in
which the unit is active
Corrects for differences in citation practices among fields, publication years and type of article
GINI Index of Concentration (for countries)
Do concentrated national academic systems performbetter than more evenly distributed ones?
China
USA
Source:SCImago/ Scopus
Smaller Western European countries
Anglo‐Saxon Countries
HK
TW
16 broad disciplines
(A&H) Arts & Humanities GEO Geosciences
APC Appl Phys & Chem MATH Mathematics
BIOL-A&P
Biol Sci ~ Anim & Plants MOLB Mol Biol & Biochem
BIOL-HU
Biol Sci ~ Humans (MULTI) Multi-disciplinary
CHEM Chemistry PHYS Physics & Astron
CLM Clinical Medicine PSY Psychol & Psychiat
ECON Economics & Business (SOC) Social Sci
ENG Engineering SOC-MED Social Sci ~ Medicine
General European Univ
TOP 25%BOTTOM 25%
TOP 25%
BOTTOM 25%
Impact
Publications
Among top 25 % in publication output and citation impact
‘Top’ research university
(SOC)
SOC-MEDPSY
PHYS (MULTI)MOLB
MATH
GEO
ENG ECON
CLM
CHEM
BIOL-HUBIOL-AP
APC
(A&H)
0
25
50
75
1000255075100
PUBLICATION RANK PTCL
CIT
ATI
ON
I PA
CT
RA
NK
PC
TL University has a top position
in each discipline
Indicators are becoming more ‘informative’
Feature Example
Put numbers in context Field‐normalized citation measures
“who” is citing Citations weighted with “prestige” of citing source
“Distance” between citing and cited authors
Impact outside the own niche; multi‐disciplinarity; bridging paradigms
“Why” is it cited Classification of citation context (e.g., organic vs. perfunctory)
Thank you for your attention!
Recommended