UCount: A community-driven approach for measuring Scientific Reputation

Cristhian ParraUniversity of Trento, Italy

parra@disi.unitn.it

Altmetrics Workshop / websci2011

Context

http://beta.kspaces.net/ic/ http://reseseval.org/http://liquidjournal.org/

What is Scientific Reputation?

Scientific Reputation is the social evaluation (opinion) by the scientific community of a

researcher or its contributions given a certain criterion (scientific impact)

Main Goal

How, Why ?

To understand the way reputation is formed within and across scientific communitiesunderstand

[Insert footer]http://reseval.org/survey

DatasetTop H-Index (>200)

79 total Replies8 Online Surveys

ICWE (18)BPM (20)

VLDB (15)...

http://www.cs.ucla.edu/~palsberg/h-number.html

Experiment #1: LiquidReputation Surveys

Results published in ISSI2011 and SEBD2011

1

23

# Publications (DBLP)

H-Index (Palsberg)

H-Index (Script)

Correlation Results

MIUR* CNRS**

Researchers 664 >1000

Areas 12 45

Output Winner/Loser Pairs Selected ResearcherRanked Waiting Lists

# Rankings 333 pairs (with H-Index >= 0)208 pairs (with H-Index > 0)

196 Rankings of 5 researchers in average

(*) http://reclutamento.murst.it/ (**) http://intersection.dsi.cnrs.fr/intersection/resultats- cc- en.do

Experiment #2: Position Contests Analysis

Results

• Surveys:– Correlation between bibliometric indicators and

reputation is always in the rank of (-0.5:0.5)• Research Position Contests– CNRS dataset: same result as in surveys– Italian dataset: around 50% of effectiveness in

predictions for all metrics

Bibliometrics are not a good describer of real reputation

UCount Methodology

UCount Sci. ExcellenceUCount Reviewer Score

UCount

Eliciting Reputation

Community oriented Surveys

Peer Review based assessment(Research Position Contests)

SurveysBeen there

Peer Review Feedback

UCount Surveys

DBLP Coauthorship Graph

ICST

Palsberg

http://www.cs.ucla.edu/~palsberg/h-number.htmlhttp://icst.org/icst-transactions/

Editorial Boards

Top H Researchers

AffinityShortest Path +

Jaccard

List of Candidates

http://icst.org/UCount-Survey/

UCount

Surveys Results

Derive Reputation Functions

Peer Review Feedback

UCount Scientific Impact

UCount Reviewer Score

Reverse Engineering of Reputation

# Real reputation

5.5 Alon Halevy

4.7 Stefano Ceri

3.9 Tim Berners Lee

3 Jim Gray

# Estimated Rep.

5.1 Alon Halevy

4.2 Tim Berners Lee

3.7 Stefano Ceri

2.9 Jim Gray

Features

H-Index

Affiliation

Citations

Readership

Combine

Minimum Distance

Other Features?

UCount

Surveys Results

Derive Reputation Functions

Peer Review Feedback

UCount Scientific Impact

UCount Reviewer Score

Community Reputation

Functions Library

Reverse Engineering Approaches

• Decision Trees– No tree with more than 60% of accuracy

• Unsupervised Methods– Genetic algorithms applied on CNRS Dataset improved

correlation in an average of 15% (running only for 5 minutes)– Highly improved correlation for fields Research Management

and Politics. • Next

– Applying Machine Learning techniques– Explore other techniques (e.g. neural networks)– Obtain other types of features (e.g. keynotes, advisory

networks)– http://code.google.com/p/revengrep/ – https://github.com/cdparra/melquiades/

Where are we now?

Thanks!

Questions?

Ideas?