September 9th 2010
Mit
glie
d d
er
Helm
holt
z-G
em
ein
sch
aft
Journal Evaluation and Science 2.0Using Social Bookmarks to Analyze Reader Perception
| Stefanie Haustein, Evgeni Golov, Kathleen Luckanus, Sabrina Reher & Jens Terliesner
Outline
1. Introduction
Multidimensional Journal Evaluation
Journal Usage
Social Bookmarking in STM
2. Retrieval of Bookmarking Data
3. Journal EvaluationUsage Indicators
Tags
4. Conclusions
5. Literature
Introduction: Multidimensional Journal Evaluation
one indicator cannot measure true influence of periodicals
five dimensions of journal evaluation are identified
test set: 45 solid state physics
journals all publications from
2004 to 2008 bibliographic data for
168,109 documents from Web of Science
MULTIDIMENSIONALJOURNAL
EVALUATION
Introduction: Journal Perception
measuring problematic despite standards only available for usage at own institutions publishers shut global statistics away alternative needed to measure worldwide journal usage
bookmarking articles on social bookmarking platforms indicates journal usage
Introduction: Social Bookmarking in STM
modeled on delicious
allow academic users to store references online
indexing resources with tags sharing literature with
other users finding new content
through other users
Introduction: Social Bookmarking in STM
Introduction: Social Bookmarking in STM
Introduction: Social Bookmarking in STM
Introduction: Social Bookmarking in STM
CiteULike Connotea BibSonomy 2collab
Launch 2004 2004 2006 2007
Responsibility Oversity Ltd. sponsored by Springer
Nature Publishing Group
Universityof Kassel
Elsevier B.V.
Unique visitors1 360,000 380,000 350,000 40,000
Total visits2 480,000 690,000 620,000 44,000
Page views3 2,100,000 3,100,000 2,800,000 180,000
1 Approximate number of estimated cookies on a site for April 2010 (Google, Inc., 2010)2 Estimated number of times a site is accessed by unique visitors in April 2010 (Google, Inc., 2010).3 Total estimated number of times pages on a site have been accessed by users (Google, Inc., 2010)
Introduction: Social Bookmarking in STM
CiteULike Connotea BibSonomy 2collab
Launch 2004 2004 2006 2007
Responsibility Oversity Ltd. sponsored by Springer
Nature Publishing Group
Universityof Kassel
Elsevier B.V.
Unique visitors1 360,000 380,000 350,000 40,000
Total visits2 480,000 690,000 620,000 44,000
Page views3 2,100,000 3,100,000 2,800,000 180,000
1 Approximate number of estimated cookies on a site for April 2010 (Google, Inc., 2010)2 Estimated number of times a site is accessed by unique visitors in April 2010 (Google, Inc., 2010).3 Total estimated number of times pages on a site have been accessed by users (Google, Inc., 2010)
Retrieval of Bookmarking Data
bookmarks from BibSonomy, CiteULike and Connotea to articles published in 45 Physics journals from 2004 to 2008
search strategies applied to compensate incomplete and erroneous bookmarking data journal title abbreviations ISSNs DOIs of articles
download of bookmarks via API or by parsing
Retrieval of Bookmarking Data
completeness / correctness of bookmarking data
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
journal title DOI ISSN journal abbreviation
Perc
enta
ge o
f boo
kmar
ks f
ound
by
sear
ch st
rate
gy
Search stragey
all services BibSonomy CiteULike Connotea
Retrieval of Bookmarking Data
completeness / correctness of bookmarking data
26.82%
65.11%
21.19%
38.61%
58.67%
32.34%
60.70%
60.26%
14.37%1.81%
18.00%
1.13%0.14% 0.74% 0.11%
all services BibSonomy CiteULike Connotea
all fields (journal title, abbreviation, ISSN, DOI)
three fields
two fields
one field
Retrieval of Bookmarking Data
download was checked and limited to bookmarks to documents published in 45 journals from 2004 to 2008
13,760 correct bookmarks retrieved
940
10778
2042
BibSonomy
CiteULike
Connotea
Number of bookmarks retrieved
Retrieval of Bookmarking Data
missing DOIs were replaced by CrossRef and manually bookmarks were matched to the bibliographic entries from
Web of Science via DOI 98.9% of bookmarks matched
2.8% doubles in user names between services
results were combined
145 users
2054 users
313 users
802 articles
8082 articles
1995 articles
BibSonomy
CiteULike
Connotea
Number of unique users and articles
Retrieval of Bookmarking Data
13,608 bookmarks were matched to 10,280 articles 2,441 unique users
0
50
100
150
200
250
300
350satoshi (322 posts)
bronckobuster (238 posts)rice (234 posts)
Num
ber o
f bo
okm
arks
per
use
r
Users from BibSonomy, CiteULike and Connotea
1,179 users posted one article 75% of content is created by 21%
of users 8,511 articles were only
bookmarked once
Journal Evaluation: Usage Indicators
bookmarking an article indicates journal usage
Usage ratio journal articles with bookmarks compared to all publications
0
500
1000
1500
2000
2500
0%
10%
20%
30%
40%
50%
60%
70%
RE
V M
OD
PH
YS
RE
P P
RO
G P
HY
SP
HY
S R
EP
NE
W J
PH
YS
PH
YS
RE
V E
SO
FT
MA
TT
ER
J S
TAT
ME
CH
PH
YS
RE
V A
EU
R P
HY
S J
EE
PL
AC
T C
RY
ST
A
AP
PL
PH
YS
LE
TP
HY
S F
LU
IDS
EU
R P
HY
S J
B
J S
TAT
PH
YS
AN
N P
HY
SJ
PH
YS
CO
ND
EN
S M
AT
TE
RN
AN
OT
EC
HN
OL
J AP
PL
PH
YS
J P
HY
S A
J P
HY
S D
IEE
E N
AN
OT
EC
HN
OL
CO
MP
UT
MA
TE
R S
CI
AP
PL
PH
YS
AJ
LO
W T
EM
P P
HY
S
PH
YS
STA
T S
OL
BN
UC
L I
NS
TR
UM
ME
TH
AP
HY
S L
ET
T A
PH
YS
STA
T S
OL
AS
UP
ER
CO
ND
SC
I T
EC
HN
OL
PH
YS
SC
RN
UC
L I
NS
TR
UM
ME
TH
B
PH
YS
TO
DA
YJE
TP
LE
TT
INT
J T
HE
RM
OP
HY
S
AC
T C
RY
ST
BJ
VA
C S
CI
TE
CH
NO
L A
J M
AG
N M
AG
N M
AT
ER
PH
YS
ICA
C
J R
HE
OL
PH
YS
ICA
BP
RA
MA
NA
HY
PE
RF
INE
IN
TE
RA
CT
SO
LID
STA
TE
IO
NP
HY
S S
OL
ID S
TAT
E
Num
ber o
f bo
okm
arke
d ar
tice
ls
Usa
ge ra
tio
articles bookmarked
share of articles bookmarked
Journal Evaluation: Usage Indicators
Usage diffusion number of unique users per journal depicts the diffusion of journal content into the community
0
100
200
300
400
500
600
700
800
900
PH
YS
RE
V E
AP
PL
PH
YS
LE
TJ A
PP
L P
HY
SP
HY
S R
EV
AJ
PH
YS
CO
ND
EN
S M
AT
TE
RR
EV
MO
D P
HY
SN
EW
J P
HY
SE
PL
J P
HY
S A
NA
NO
TE
CH
NO
LP
HY
S R
EP
EU
R P
HY
S J
BJ
STA
T M
EC
HR
EP
PR
OG
PH
YS
J P
HY
S D
PH
YS
FL
UID
SP
HY
S L
ET
T A
NU
CL
IN
ST
RU
M M
ET
H A
AP
PL
PH
YS
AJ
MA
GN
MA
GN
MA
TE
RE
UR
PH
YS
J E
PH
YS
STA
T S
OL
BJ
STA
T P
HY
SN
UC
L I
NS
TR
UM
ME
TH
BS
OF
T M
AT
TE
RP
HY
S S
TAT
SO
L A
PH
YS
ICA
BP
HY
S S
CR
PH
YS
TO
DA
YC
OM
PU
T M
AT
ER
SC
IS
UP
ER
CO
ND
SC
I T
EC
HN
OL
PH
YS
ICA
CA
CT
CR
YS
T A
JET
P L
ET
TJ
LO
W T
EM
P P
HY
SA
NN
PH
YS
J V
AC
SC
I T
EC
HN
OL
AIE
EE
NA
NO
TE
CH
NO
LP
RA
MA
NA
SO
LID
STA
TE
IO
NIN
T J
TH
ER
MO
PH
YS
AC
T C
RY
ST
BH
YP
ER
FIN
E I
NT
ER
AC
TP
HY
S S
OL
ID S
TAT
EJ
RH
EO
L
Num
ber o
f un
ique
use
rs
Journal Evaluation: Usage Indicators
Article Usage Intensity average number of bookmarks per article shows how often articles are read
0
1000
2000
3000
0.0
1.5
3.0
4.5
RE
V M
OD
PH
YS
PH
YS
RE
P
RE
P P
RO
G P
HY
S
J S
TAT
ME
CH
EU
R P
HY
S J
B
PH
YS
RE
V E
IEE
E N
AN
OT
EC
HN
OL
NE
W J
PH
YS
AC
T C
RY
ST
A
J P
HY
S A
EU
R P
HY
S J
E
EP
L
J P
HY
S C
ON
DE
NS
MA
TT
ER
PH
YS
RE
V A
J R
HE
OL
SU
PE
RC
ON
D S
CI
TE
CH
NO
L
PH
YS
TO
DA
Y
PH
YS
ICA
C
J S
TAT
PH
YS
PH
YS
FL
UID
S
SO
FT
MA
TT
ER
AP
PL
PH
YS
LE
T
AN
N P
HY
S
PH
YS
LE
TT
A
J M
AG
N M
AG
N M
AT
ER
NA
NO
TE
CH
NO
L
J P
HY
S D
AC
T C
RY
ST
B
JET
P L
ET
T
PH
YS
STA
T S
OL
A
J AP
PL
PH
YS
NU
CL
IN
ST
RU
M M
ET
H A
CO
MP
UT
MA
TE
R S
CI
INT
J T
HE
RM
OP
HY
S
AP
PL
PH
YS
A
PH
YS
STA
T S
OL
B
NU
CL
IN
ST
RU
M M
ET
H B
J L
OW
TE
MP
PH
YS
PH
YS
SC
R
J V
AC
SC
I T
EC
HN
OL
A
PH
YS
ICA
B
HY
PE
RF
INE
IN
TE
RA
CT
PH
YS
SO
LID
STA
TE
PR
AM
AN
A
SO
LID
STA
TE
IO
N
Num
ber o
f bo
okm
arks
Art
icle
usa
ge in
tens
ity
number of bookmarks
article usage intensity
Journal Evaluation: Usage Indicators
Journal Usage Intensity mean number of articles per user measures how intensive the average user reads the journal
0
100
200
300
400
500
600
700
800
0.00
1.00
2.00
3.00
4.00
5.00
PH
YS
RE
V A
AP
PL
PH
YS
LE
T
J AP
PL
PH
YS
NU
CL
IN
ST
RU
M M
ET
H A
NU
CL
IN
ST
RU
M M
ET
H B
PH
YS
RE
V E
PH
YS
FL
UID
S
PH
YS
ICA
C
SO
FT
MA
TT
ER
J L
OW
TE
MP
PH
YS
J P
HY
S C
ON
DE
NS
MA
TT
ER
INT
J T
HE
RM
OP
HY
S
PH
YS
ICA
B
J V
AC
SC
I T
EC
HN
OL
A
NA
NO
TE
CH
NO
L
J M
AG
N M
AG
N M
AT
ER
HY
PE
RF
INE
IN
TE
RA
CT
CO
MP
UT
MA
TE
R S
CI
PH
YS
SC
R
EP
L
J P
HY
S D
EU
R P
HY
S J
E
PH
YS
LE
TT
A
AC
T C
RY
ST
A
PH
YS
STA
T S
OL
B
JET
P L
ET
T
J R
HE
OL
SO
LID
STA
TE
IO
N
PH
YS
STA
T S
OL
A
J S
TAT
PH
YS
NE
W J
PH
YS
IEE
E N
AN
OT
EC
HN
OL
J P
HY
S A
AP
PL
PH
YS
A
SU
PE
RC
ON
D S
CI
TE
CH
NO
L
AC
T C
RY
ST
B
PR
AM
AN
A
AN
N P
HY
S
PH
YS
SO
LID
STA
TE
J S
TAT
ME
CH
PH
YS
TO
DA
Y
EU
R P
HY
S J
B
RE
P P
RO
G P
HY
S
PH
YS
RE
P
RE
V M
OD
PH
YS
Usa
ge d
iffu
sion
Jour
nal u
sage
inte
nsit
y
usage diffusion
journal usage intensity
Journal Evaluation: Usage Indicators
Pearson correlation
Usage ratio
Usage diffusion
Article usage
intensity
Journal usage
intensityP 5-year IF h-index
Cited Half-Life
Usage ratio
0.33 0.90 -0.22 -0.09 0.86 0.50 0.02
Usage diffusion
0.33 0.27 0.45 0.72 0.14 0.78 -0.24
Article usage intensity
0.90 0.27 -0.36 -0.16 0.88 0.46 0.09
Journal usage intensity
-0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10
P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17
5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21
h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22
Cited Half-Life
0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
Journal Evaluation: Usage Indicators
Pearson correlation
Usage ratio
Usage diffusion
Article usage
intensity
Journal usage
intensityP 5-year IF h-index
Cited Half-Life
Usage ratio
0.33 0.90 -0.22 -0.09 0.86 0.50 0.02
Usage diffusion
0.33 0.27 0.45 0.72 0.14 0.78 -0.24
Article usage intensity 0.90 0.27 -0.36 -0.16 0.88 0.46 0.09
Journal usage intensity
-0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10
P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17
5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21
h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22
Cited Half-Life
0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
Journal Evaluation: Usage Indicators
Pearson correlation
Usage ratio
Usage diffusion
Article usage
intensity
Journal usage
intensityP 5-year IF h-index
Cited Half-Life
Usage ratio 0.33 0.90 -0.22 -0.09 0.86 0.50 0.02
Usage diffusion 0.33 0.27 0.45 0.72 0.14 0.78 -0.24
Article usage intensity
0.90 0.27 -0.36 -0.16 0.88 0.46 0.09
Journal usage intensity -0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10
P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17
5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21
h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22
Cited Half-Life
0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
Journal Evaluation: Usage Indicators
Pearson correlation
Usage ratio
Usage diffusion
Article usage
intensity
Journal usage
intensityP 5-year IF h-index
Cited Half-Life
Usage ratio
0.33 0.90 -0.22 -0.09 0.86 0.50 0.02
Usage diffusion
0.33 0.27 0.45 0.72 0.14 0.78 -0.24
Article usage intensity
0.90 0.27 -0.36 -0.16 0.88 0.46 0.09
Journal usage intensity
-0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10
P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17
5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21
h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22
Cited Half-Life
0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22
Journal Evaluation: Tags
88.4% of all bookmarks contained tags 8,208 unique tags assigned 38,241 times
1
10
100
1000
10000
1 10 100 1000 10000
freq
uenc
y
tags
tagcloud: all tags assigned at least 50 times
Journal Evaluation: Tags
2,535
2,139
1,453
1,227
657
711
546
637
564
494
0 2000 4000 6000 8000 10000
PHYS REV E
APPL PHYS LET
PHYS REV A
J APPL PHYS
J PHYS CONDENS MATTER
NEW J PHYS
REV MOD PHYS
EPL
NANOTECHNOL
J PHYS A
tag frequency
2004
2005
2006
2007
2008number of unique tags
2004
2005
2006
2007
2008
Number of tags per publication year for all bookmarked articles
Journal Evaluation: Tags
tags assigned to journal articles depict the users’ perspective on content
analysis over time can reveal shifts in thematic focus areas
tags assigned to articles published in J Phys Condens Matter in 2004
tags assigned to articles published in J Phys Condens Matter in 2008
Conclusions
data from social bookmarking services can be used in journal evaluation
number of bookmarks can be used alternatively to measure worldwide usage of journals
tags reveal the users’ perspective on journal content
poor quality of bookmarking data makes retrieval and analysis complex
social bookmarking in STM is still in its infancy and services are competing for a critical mass
Literature
BERNIUS, S., HANAUSKE, M. AND DUGALL, B. (2009). Von traditioneller wissenschaftlicher Kommunikation zu "Science 2.0". ABI-Technik 29, 214-226.
HAMMOND, T., HANNAY, T., LUND, B. AND SCOTT, J. (2005). Social Bookmarking Tools (I). D-Lib Magazine 11.
HAUSTEIN, S. (2010). Multidimensional Journal Evaluation. Proceedings of STI 2010, Leiden.
JUCHEM, K., SCHLOEGL, C. AND STOCK, W.G. (2006). Dimensionen der Zeitschriftenszientometrie am Beispiel von "Buch und Bibliothek". Information Wissenschaft & Praxis 57, 31-37.
PROJECT COUNTER (2008). The COUNTER Code of Practice. Journals and Databases Release 3, in: http://www.projectcounter.org/r3/Release3D9.pdf.
REHER, S. AND HAUSTEIN, S. (to be published). Social Bookmarking in STM: Putting Services to the Acid Test. ONLINE.
ROUSSEAU, R. (2002). Journal Evaluation: Technical and Practical Issues. Library Trends 50, 418-439.
SHNEIDERMAN, B. (2008). Computer Science - Science 2.0. Science 319, 1349-1350.
Thank you for your kind attention!