Rafols - Towards more inclusive STI indicators

Towards more inclusive STI indicators

Ismael Ràfols1,2 and Jordi Molas-Gallart1, Richard Woolley1 and Diego Chavarro2

1Ingenio (CSIC-UPV), Universitat Politècnica de València2 SPRU (Science Policy Research Unit), University of Sussex, Brighton

Gent, OECD Blue Sky 2016

Re-shaping design and use for inclusion

• Indicators may be harming – via exclusion Current indicators are only (partially) appropriate for some types of science and

innovation. Biases against and potential suppression of creative and valuable types of

research (agro-, health,…). Threat to diversity. May count as positive harmful forms of innovation (Soete)

• Not only more, but other types of indicators needed Making visible “other” contributions and other types of research and innovation

(e.g. action research, co-creation) Enhancing visualisation for “opening up” perspectives rather than facilitating

“closing down”

• Understanding uses of indicators Better embedding in policy, evaluation, context Indicators used to pluralise debate, as tools for interpretation and

deliberation, not a substitute for judgement (Barré)

Uses of indicators: Pressing demands of research policy, management and evaluation --- Can indicators help?

Yes, indicators can help make decisions… Reduce time and costs Increase transparency and sense of objectivity Reduce complexity, accessible to managersbut do they lead to the “right” decisions?

Evaluation gap (Wouters):“discrepancy between [the] criteria [implicit in indicators] and the social and

economic functions of science”

*Academia – “excellence” *Innovation – economic “growth”

Missions not well covered: agriculture, public health, defence,development, social inclusion,…

Often related to marginalised / “neglected” populations

Space of problems

Space of research

Researchwell illuminatedby indicators

Problems, research, indicators and marginalisation

Problems, research, indicators and marginalisation

Multiple types of space:

STI Peripheries:research spaces notwell capturedby indicators


Cognitive: SSH, engineering

Linguistic: non-English

Sectoral: low-tech, agriculture, creative ind.

Social: gender, minorities

Geographical: regional, “South”

Coverage: Knowledge production on rice by country

India China Japan USA0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%2000-2009

CABI

Scopus

WoS

Publ

icati

ons

Rafols, Ciarli and Chavarro (in preparation)

Coverage: Knowledge production on rice by topic

Rafols, Ciarli and Chavarro (in preparation)

2002-2012

Streetlight effect in indicators: mistaking light with “problems”

Space of problems

Space of research


Streetlight effect in indicators: mistaking light with “problems”

Space of problems

Space of research

Space of problemsSpace of research

Space of problems

Hypothesis: reduced indicator coverage may contract research space

Space of research

Space of STIindicators

The societal needs dealt by research that is under the streetlight effect, will be better rewarded.

Reduced diversity ofresearch efforts...

…reduced coverageof societal needs

Space of problems

Demands for expanding role of science in society…

Space of research


Space of problems

Demands for expanding role of science in society…

Space of research


Space of problems

…may require expanded sets of data and indicators

Space of research


Space of problems

…may require expanded sets of data and indicators

Space of research


• Comprehensive databases

• Better normalisationby context (e.g. citation bias)

• Indicators of process or interactions

• New data sources

narrow

broad

closing-down opening-up

range of appraisal's inputs(issues, perspectives, scenarios, methods)

effect of appraisal ‘outputs’ on decision-making

Leach et al. 2010

Appraisal: Broadening out vs. Opening up

narrow

broad


range of appraisal’s inputs(issues, perspectives, scenarios, methods)


Most conventionalS&T indicators??


narrow

broad


range of appraisal’s inputs(issues, perspectives, scenarios, methods)


ConventionalS&T indicators??

Broadening out

Incorporation multiple analytical dimensions:

New analytical inputs: media, blogsphere.


narrow

broad


range of appraisals inputs(issues, perspectives, scenarios, methods)


Journal rankings

University rankings Unitary measuresthat may be translated into prescriptionComposite

European InnovationScoreboard


narrow

broad


range of appraisals inputs(issues, perspectives, scenarios, methods)



ConventionalS&T Indicators??

opening-up

Making explicit underlying conceptualisations and contextcreating tools to facilitate exploration

NOT about the uniquely best methodOr about the unitary best explanationOr the single best prediction

From S&T indicators for justification and disciplining…… towards S&T indicators as tools for deliberation

Model 2: Plural and conditionalExploring diverse choices Facilitating options/choices in landscapes

Model 1: Unique and prescriptiveProposing “best choices”Rankings -- ranking list of preferences

An agenda for more inclusive metrics• Inclusiveness in the inputs

Broadening out: Create more diverse indicators – Indicators of open science, RRI, hidden, social innovation– Improve representation of SSH scholarship, languages other than

English, the “South”,…

• Inclusiveness in the outputs Opening up: develop toolkits that allow exploration of choices. New

ways of presenting indicators– Multi-ranking tools– Interactive visualisations

• Inclusiveness in the policy process (??) Develop new social processes on use of indicators

– STI indicators as tools for interpretation and deliberation (R. Barré)

Backup slides

24

Rice Varieties Classic Genetics

TransgenicsMol. Biology Genomics

PestsPlant protection

Weeds Plant protection

Plant nutrition

Production & socioeconomic issues

ConsumptionH. nutrition, food techs

What are the “options” in rice research?

Not a fined-grain perspective

25

US, 2000-12

How research priorities differ by country

26

India 2000-12


27

Thailand 2000-12


S&T Indicators Conference – First week

21st Science and Technology Indicators ConferenceValència, 14-16 September 2016, www.sti2016.org

The problem:• Some STI activities and their outcome are badly captured by indicators• Disincentive to carry out certain types of activity at certain spaces• Empirically: problem more acute at peripheral spaces in science • Potential effects: Reduction of diversity –particularly locally relevant?

Bias in citation within a subdiscipline

Good

Average

Bad

Van Eck, Waltman et al. (2013)

More basic

More applied

Clinical neurologyIs basic always better than applied?

Citations: not stable to changes in classification and granularity (Zitt et al., 2005; Adams et al., 2008).

Measures of “scientific excellence”

ISSTI SPRU MIoIR Imperial WBS LBS0

0.5

1

1.5

2

2.5

3

3.5

4AB

S Ra

nk


1

2

3

4

Cita

tions

/pub

Jo

urna

l-fiel

d N

orm

alis

ed


0.05

0.1

0.15

0.2

Cita

tions

/pub

Citi

ng-p

aper

Nor

mal

ised

Which one is more meaningful??


1

2

3

4

Jour

nal I

mpa

ct F

acto

r

Rafols et al. (2012, Research Policy)

Data & Analytics

Rafols - Towards more inclusive STI indicators