The Multi Research Assessment Matrix Moed ESSS Vienna 12 Sept... · The Multi‐Dimensional Research Assessment Matrix Henk F. Moed Elsevier, Amsterdam, Netherlands. Lecture at ESSS

The Multi‐Dimensional Research Assessment Matrix

Henk F. Moed

Elsevier, Amsterdam, NetherlandsLecture at ESSS Conference, Vienna, 12 Sept. 2011

A framework to characterize and position bibliometric indicators,

methods and products

A framework to characterize and position bibliometric methods and products

• The multi‐dimensional research assessment matrix

• Top‐down versus bottom‐up delimitation methods

• Three generations of bibliometric indicators

The future of research assessment exercises lies in the

intelligent combination of metrics and peer review

The Multi‐Dimensional Research Assessment Matrix

Expert Group on the Assessment of University-Based Research (AUBR, 2010)

Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment

Purpose Output dimensions

Bibliometricindicators

Other indicators

Individual Allocate resources

Researchproductivity

Publications Peer review

Research group

Improve performance

Quality, scholarly impact

Journalcitation impact

Patents, licences, spin offs

Department Increase multi‐discipl.research

Innovation and social benefit

Actualcitation impact

Invitationsfor conferences

Institution Increaseregional engagement

Sustainabi‐lity & Scale

Internat. co‐authorship

Externalresearch income

Research field

Promotion, hiring

Research infrastruct.

citation ‘prestige’

PhD com‐pletion rates




Other indicators




Research group

Improve performance












Research field

Promotion, hiring




Read column‐wise




Other indicators




Research group

Improve performance












Research field

Promotion, hiring




Types of research outputs

Impacts Publication/text Non-publication

Scientific-scholarly

Journal paper; book chapter; monograph

Research data file; video of experiment

Educational Teaching course book; syllabus

Skilled researchers

Economic Patent Product; process; device; design; image; spin off

Cultural Newspaper article Interviews; events; performances; exhibits




Other indicators




Research group

Improve performance












Research field

Promotion& hiring




MD‐RAM: Example 1

Individual Hiring/promotionProductivity &

impactPhD date, place,

supervisor;Invitations for

conferences

Publications in international jrnls;

Actual citation impact




Other indicators




Research group

Improve performance












Research field

Promotion, hiring




MD‐RAM: Example 2

Research group Allocate resourcesRes. productivity &

impactCompetitive

research income;Ratio research

active/total staff

Publications in international jrnls;

Actual citation impact




Other indicators




Research group

Improve performance




Depart‐ment

Increase multi‐discipl.research

Innovationand social benefit



Institution Increaseregional engagemnt




Research field

Promotion, hiring




MD‐RAM: Example 3

DepartmentIncrease regional

engagementSocial benefits &

innovation% Funding from

endusers;Licenses, spin

offs

% Industry co-authored papers;

PatentsRes. reports




Other indicators




Research group

Improve performance




Department Increase multi‐disc. research








Research field

Promotion, hiring




MD‐RAM: Example 4

InstitutionIncrease multi-

disciplinary researchRes. productivity &

impactInter-discipl.

teaching progr;Funding from

multi-disc. res. progr

% Papers in multi-disciplinary fields;

Co-authorships from different discipl

Positioning bibliometric products ‐ 1

Unit of assessment

Purpose Aspect Product

Researchgroup

Improve performance;Allocate resources

Productivity/ impact A

Department; institution

Allocate resources; Stimulate multi-disciplinary research

Researchinfrastructure and potential

B

Institution Stimulate multi-disciplinary research;Improve performance;

Social benefits; sustainability;Prod./impact

C

Group, Dept, Institution

Improve performance& regional engagement;Allocate resources;

Productivity/ impact; social impact; infrastr.

D

Top-down vs. bottom-up institutional delimitation

Top‐down institutional analysis

Select an institution’s papers using author affiliations (incl. verification)

Categorize articles intoresearch fields

Calculate indicators

Compare with benchmarks

Bottom‐up institutional analysis

Compile a list of researchers

Compile a list of publications per

researcher (incl. verification)

Aggregate researchers into groups, departments, fields, etc.

Calculate indicators;

compare with benchmarks

159 NL Academic Chemistry Groups

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

0 200 400 600 800

Total Publications

Nor

mal

ised

Cita

tion

Impa

ct

1.0 =World Average

Total Articles (10 yrs)

Normalised

Citation Im

pact

Top-down vs. bottom-up subject field delimitation

Top‐down field analysis

(Pre‐)define a field categorization

(Pre‐)allocate papers (journals)

to categories

Calculate indicators

per category

Bottom‐up field analysis

Create a (large) dataset

of publications

Cluster papers on the basis of

(co‐) citation or word analysis

Name the clusters

Calculate indicators by cluster

ST

RA

TE

GIC

DIA

GR

AM

Co‐Word Analysis (COPALRED, prof. R. Bailon, Univ Granada)

Library Catalog AnalysisLibrary Catalog Analysis2. RESULTS Library Catalog Analysis

Positioning bibliometric products – 2

Institutional delimitation

Semantic/field delimitation

Product or service

Top‐down Top‐down A

Top‐down Bottom‐up B

Bottom‐up Top‐down C

Bottom‐up Bottom‐up D

Other aspects

• Which are the principal user groups?

• Relationship to peer review

• Time window covered

• Focus on top vs. bottom of the quality distribution

Three generations of bibliometric indicators

Three generations of bibliometric indicators

Generation Indicators (Examples)

1 Basic No. publications No. CitationsCites per paper (e.g., Jrnl Impact Factor)

2 Normalized Relative Citation Rate No. publ. in top citation percentiles

3 Contextual Entire citation network based (e.g., Eigenfactor, SJR, Jrnl Prestige);

Citation context based

Differences in citation potential between fields

Molecular Biology Mathematics

0

10

20

30

40

50

0 1 2 3 4 5 6 70

10

20

30

40

50

0 1 2 3 4 5 6 7

Number of received citations

%

Papers

Refe-rencelists

Relative Citation Rate (RCR)

The average citation rate of a unit’s papers

÷world citation average in the subfields in

which the unit is active

Corrects for differences in citation practices among fields, publication years and type of article

GINI Index of Concentration (for countries)

Do concentrated national academic systems performbetter than more evenly distributed ones?

China

USA

Source:SCImago/ Scopus

Smaller Western European countries

Anglo‐Saxon Countries

HK

TW

16 broad disciplines

(A&H) Arts & Humanities GEO Geosciences

APC Appl Phys & Chem MATH Mathematics

BIOL-A&P

Biol Sci ~ Anim & Plants MOLB Mol Biol & Biochem

BIOL-HU

Biol Sci ~ Humans (MULTI) Multi-disciplinary

CHEM Chemistry PHYS Physics & Astron

CLM Clinical Medicine PSY Psychol & Psychiat

ECON Economics & Business (SOC) Social Sci

ENG Engineering SOC-MED Social Sci ~ Medicine

General European Univ

TOP 25%BOTTOM 25%

TOP 25%

BOTTOM 25%

Impact

Publications

Among top 25 % in publication output and citation impact

‘Top’ research university

(SOC)

SOC-MEDPSY

PHYS (MULTI)MOLB

MATH

GEO

ENG ECON

CLM

CHEM

BIOL-HUBIOL-AP

APC

(A&H)

0

25

50

75

1000255075100

PUBLICATION RANK PTCL

CIT

ATI

ON

I PA

CT

RA

NK

PC

TL University has a top position

in each discipline

Indicators are becoming more ‘informative’

Feature Example

Put numbers in context Field‐normalized citation measures

“who” is citing Citations weighted with “prestige” of citing source

“Distance” between citing and cited authors

Impact outside the own niche; multi‐disciplinarity; bridging paradigms

“Why” is it cited Classification of citation context (e.g., organic vs. perfunctory)

Thank you for your attention!

Documents

The Multi Research Assessment Matrix Moed ESSS Vienna 12 Sept... · The Multi‐Dimensional Research Assessment Matrix Henk F. Moed Elsevier, Amsterdam, Netherlands. Lecture at ESSS