Embed Size (px)
Citation preview
IRF Report 2010-00002
Date: 2010-07-23
CLEF-IP 2009: Evaluation Summary –
Part Two
Florina Piroi, Giovanna Roda, Veronika Zenz
ABSTRACT: This document presents a summary of the evaluation activities that
were carried out for CLEF-IP 2009 track, with respect to 12 specific topics. The data
from the 70 submitted runs was preprocessed and grouped into bundles according to
experiment size and task. For each experiment we computed precision and recall at
various levels, and map measures.
Clef–Ip 2009: Evaluation Summary – Part Two
Florina [email protected]
Giovanna [email protected]
Veronika [email protected]
July 23, 2010
Abstract
This document presents a summary of the evaluation activities that were carried out forClef–Ip 2009 track, with respect to 12 specific topics. The data from the 70 submittedruns was preprocessed and grouped into bundles according to experiment size and task. Foreach experiment we computed precision and recall at various levels, and map measures.
1 Further Evaluations at Clef–Ip 2009
In this report we present the results of run evaluations for a set of 12 topics. The 70 runs used forthese evaluations were submitted by fourteen different teams corresponding to fifteen differentparticipating institutions (see Table 1).
For each experiment we computed 9 standard IR measures:; Precision, Precision@5, Precision@10, Precision@100; Recall, Recall@5, Recall@10, Recall@100; MAPThe values shown in this evaluation report are computed with trec eval 1.
Table 1: List of active participants and runs submitted
ID Institution Tasks Sizes RunsTUD Tech. Univ. Darmstadt, Dept. of CS,
Ubiquitous Knowledge Processing LabDE Main, EN,
DE, FRS(4),M(4),L(4), XL(4)
16
UniNE Univ. Neuchatel - Computer Science CH Main S(7), XL(1) 8uscom Santiago de Compostela Univ. - Dept.
Electronica y ComputacionES Main S(8) 8
UTASICS University of Tampere - Info Studies& Interactive Media and Swedish In-stitute of Computer Science
FISE
Main XL(8) 8
clefip-ug Glasgow Univ. - IR Group Keith UK Main M(4),XL(1) 5clefip-unige
Geneva Univ. - Centre Universitaired’Informatique
CH Main XL(5) 5
cwi Centrum Wiskunde & Informatica -Interactive Information Access
NL Main M(1),XL(4) 4
hcuge Geneva Univ. Hospitals - Service ofMedical Informatics
CH Main, EN,DE, FR
M(3),XL(1) 4
humb Humboldt Univ. - Dept. of GermanLanguage and Linguistics
DE Main, EN,DE, FR
XL(4) 4
clefip-dcu Dublin City Univ. - School ofComputing
IR Main XL(3) 3
clefip-run Radboud Univ. Nijmegen - Centrefor Language Studies & SpeechTechnologies
NL Main, EN S(2) 2
Continued on next page
1http://trec.nist.gov/trec eval
1
Evaluation Summary - Part Two 2
Table 1: List of active participants and runs submitted
ID Institution Tasks Sizes RunsHildesheim Hildesheim Univ. - Inst. of Informa-
tion Systems & Nat. Lang. ProcessingDE Main S(1) 1
NLEL Technical Univ. Valencia - NaturalLangugage Engineering
ES Main S(1) 1
UAIC AI. I. Cuza University of Iasi - NaturalLanguage Processing
RO EN S(1) 1
2 Selection of Topic Patents, Data Preprocessing
For this round of evaluations we have been collaborating with volunteering experts in intelectualproperty. Each patent expert aggreed to assess retrieval results for up to four patents in theirown area of expertise. Each expertise area was communicated to us as an Ipc code. Therefore,the twelve selected patents (see Table 2) are not randomly extracted from the Clef–Ip topicsets (Small, Medium, Large, eXtra Large), but selected depending on their Ipc code, with apreference for those with a higher number of recorded citations.
Table 2: Topics manually assessed by patent experts
ID Patent IPC codes Citations In Topic Set1 EP1421078 A61K, C07D, A61P 12 S2 EP1278338 H04L, G05B, H04Q 5 S3 EP1133104 G06F, H04L, H04N, H04B, H04M 8 S4 EP1236569 B41N, C08L, B41C, C08K, G03F 3 S5 EP1333887 A61K, A61P 4 S6 EP1314753 B32B, C08K, E04B, C08J, C09J 3 S7 EP1130071 C08L, C08F, C09J 12 M8 EP1255138 G02B 4 M9 EP1297752 A23D, C11B, A21D, A23L, C11C 3 Xl10 EP1175873 A61C 3 Xl11 EP1149535 A23F 3 Xl12 EP1333726 A23F, A23L 5 Xl
Before computing retrieval efficiency measures, we filtered the submitted experiments tocontain only the results for the 12 topics. Then, using a similar process to the one described in[1], we’ve created three evaluation bundles: one for experiments that contain retrieve results fortopics 1 to 6; a second one for experiments with results for topics 1 to 8; and a third one forthe experiments with retrieval results for all the 12 topics. Not only for convenience, we use thesame S, M, and XL bundle names as in the first evaluation summary [1].2 The list of (filtered)experiments in the resulting bundles are the same as in [1], Section 2, and we do not reproducethem here.
Section 4 shows the evaluation results grouped by S, M, and XL bundles of runs.
3 Assessments
3.1 Manual Relevance Assessments
To record the assessments done by the volunteering patent experts, we have used a customisedversion of the web plartform system used by Trec ’Legal Track’ since 20073. Upon login, thesystem allowed an IP expert to select the patent topic to assess, to inspect each retrieved patent,and mark its relevancy. However, only a few patent experts used the provided systems. Most of
2The ‘S’, ‘M’, ‘XL’ bundle names also reflect the fact that topics 1 to 6 are to be found in the Small topic set,topics 1 to 8 in the Medium topic set, and topics 1 to 12 in the eXtra Large topic set.
3The web platform has been originally desigend by Ian Soboroff, at NIST.
Evaluation Summary - Part Two 3
them preferred using their own system for doing assessments. Therefore, we have provided themwith text files of the documents to assess, and they sent us back the results of their assessments.
In average, each expert had to assess 264 results per topic, which correspond to a pooling upto rank 20 of the experimental data. The original relevance levels an assessor could assign were:
Figure 1: Assessment system after login.
‘unsure’, ‘not relevant’, ‘A’ (background information), ‘X’ (invalidating one or more claims), or’Y’ (combined with other references, invalidating one ore more claims). As, according to the Ipexperts, a decision wether a document is ‘X’ or ‘Y’-relevant for a given topic is time consuming,we have later aggreed that they would use only ‘not relevant’ and ‘A’ (or ‘relevant’) for theassessments.
Evaluation Summary - Part Two 4
Figure 2: Assessing one topic patent.
4 Measurements
In this section we present the values of the measures mentioned in Section 1 for all run filesgrouped by task and size.
4.1 Main Task
Table 3: Measures for bundle S, task Main
run id P R MAPclefip-dcu baseline 0.0092 0.5957 0.0615clefip-dcu Filtered2 0.0085 0.579 0.0729clefip-dcu Filtered 0.0082 0.5594 0.0501
clefip-run ClaimsBOW 0.0217 0.1971 0.0174clefip-ug bm25medstandard 0. 0. 0.
clefip-ug infdocfreqBM25EnglishTerms 0. 0. 0.clefip-ug infdocfreqBM25FrenchTermsNew 0. 0. 0.
clefip-ug infdocfreqBM25GermanTerms 0. 0. 0.clefip-ug infdocfreqCosEnglishTerms 0.0075 0.383 0.0193
clefip-unige RUN1 0.0075 0.5028 0.0934clefip-unige RUN2 0.0082 0.5646 0.0936clefip-unige RUN3 0.0075 0.5028 0.0934clefip-unige RUN4 0.0077 0.5123 0.0413clefip-unige RUN5 0.0045 0.351 0.0395
cwi bm25 0.0072 0.5104 0.0208cwi boolean 0.0059 0.3108 0.0031
cwi categorybm25 0.0058 0.381 0.0066cwi category 0.0062 0.4744 0.0197hcuge BiTeM 0.008 0.5204 0.0319
Hildesheim MethodAnew 0. 0. 0.humb 1 0.0079 0.4025 0.1631
NLEL MethodA 0.0058 0.37 0.0574TUD 800noTitle 0.0088 0.5699 0.0466
UAIC MethodAnew 0.0002 0.0128 0.0026UniNE strat1 0.005 0.3655 0.0346UniNE strat2 0.0073 0.5533 0.0779UniNE strat3 0.0098 0.5681 0.075UniNE strat4 0.0089 0.585 0.0714UniNE strat5 0.0065 0.4544 0.0197UniNE strat6 0.0096 0.6411 0.062
run id P R MAPContinued on next page
Evaluation Summary - Part Two 5
Table 3: Measures for bundle S, task Main, continued
run id P R MAPUniNE strat7 0.0097 0.6514 0.0199UniNE strat8 0.0082 0.5595 0.0189uscom BM25a 0.0084 0.5402 0.1165uscom BM25at 0.0088 0.586 0.1222uscom BM25b 0.0096 0.6012 0.0416uscom BM25bt 0.0099 0.6039 0.0503uscom BM25c 0.0094 0.5883 0.032uscom BM25ct 0.0096 0.5899 0.0372uscom BM25d 0.0092 0.5755 0.0315uscom BM25dt 0.0096 0.5899 0.0371
UTASICS abs-des-ratf-GT 0.0093 0.6541 0.0758UTASICS abs-des-ratf 0.009 0.6118 0.0447
UTASICS abs-tit-cla-ratf-GT-ipcr 0.0087 0.6746 0.0642UTASICS abs-tit-cla-ratf-ipcr 0.0087 0.641 0.0701
UTASICS all-ratf-GT-ipcr 0.0093 0.675 0.0953UTASICS all-ratf-ipcr 0.0092 0.6621 0.0853
UTASICS all-tf-idf-GT-ipcr 0.0082 0.6172 0.1102UTASICS all-tf-idf-ipcr 0.0087 0.6353 0.0981
run id P R MAP
Table 4: Measures for bundle M, task Main
run id P R MAPclefip-dcu baseline 0.0081 0.5975 0.055clefip-dcu Filtered2 0.0081 0.5997 0.0672clefip-dcu Filtered 0.0081 0.5924 0.0484
clefip-ug bm25medstandard 0.0009 0.0257 0.0002clefip-ug infdocfreqBM25EnglishTerms 0. 0. 0.clefip-ug infdocfreqBM25FrenchTerms 0. 0. 0.
clefip-ug infdocfreqBM25GermanTerms 0. 0. 0.clefip-ug infdocfreqCosEnglishTerms 0.0097 0.4932 0.0323
clefip-unige RUN1 0.0079 0.5573 0.0998clefip-unige RUN2 0.0086 0.6109 0.0934clefip-unige RUN3 0.0079 0.5573 0.0998clefip-unige RUN4 0.0076 0.5533 0.0563clefip-unige RUN5 0.0058 0.4471 0.0462
cwi bm25 0.0065 0.5298 0.0471cwi boolean 0.0186 0.3201 0.0036
cwi categorybm25 0.0067 0.4659 0.0164cwi category 0.0058 0.5029 0.0164hcuge BiTeM 0.0086 0.5668 0.0348
humb 1 0.0099 0.4784 0.149TUD 800noTitle 0.0095 0.626 0.0574
UniNE strat8 0.0088 0.6108 0.0261UTASICS abs-des-ratf-GT 0.0101 0.6965 0.071
UTASICS abs-des-ratf 0.01 0.6684 0.0677UTASICS abs-tit-cla-ratf-GT-ipcr 0.0093 0.7008 0.0539
UTASICS abs-tit-cla-ratf-ipcr 0.0093 0.6756 0.0583UTASICS all-ratf-GT-ipcr 0.0106 0.7268 0.0851
UTASICS all-ratf-ipcr 0.0105 0.7172 0.0717UTASICS all-tf-idf-GT-ipcr 0.0098 0.6835 0.0954
UTASICS all-tf-idf-ipcr 0.0099 0.6897 0.0792
run id P R MAP
Evaluation Summary - Part Two 6
Table 5: Measures for bundle XL, task Main
run id P R MAPclefip-dcu baseline 0.0112 0.6599 0.084clefip-dcu Filtered2 0.0118 0.625 0.0935clefip-dcu Filtered 0.0118 0.6201 0.0759
clefip-ug infdocfreqCosEnglishTerms 0.0131 0.5452 0.0614clefip-unige RUN1 0.0102 0.5857 0.1156clefip-unige RUN2 0.0103 0.6013 0.1077clefip-unige RUN3 0.0102 0.5857 0.1157clefip-unige RUN4 0.0107 0.617 0.0837clefip-unige RUN5 0.0077 0.4688 0.0481
cwi bm25 0.0098 0.6063 0.0736cwi boolean 0.0528 0.3659 0.0258cwi category 0.0131 0.5396 0.0407hcuge BiTeM 0.0267 0.5788 0.0748
humb 1 0.0316 0.5089 0.1664TUD 800noTitle 0.0129 0.6486 0.0685
UniNE strat8 0.0088 0.5376 0.0337UTASICS abs-des-ratf-GT 0.0124 0.7142 0.098
UTASICS abs-des-ratf 0.0124 0.703 0.1034UTASICS abs-tit-cla-ratf-GT-ipcr 0.0124 0.7642 0.0956
UTASICS abs-tit-cla-ratf-ipcr 0.0121 0.7292 0.0874UTASICS all-ratf-GT-ipcr 0.0136 0.783 0.1245
UTASICS all-ratf-ipcr 0.0131 0.7616 0.099UTASICS all-tf-idf-GT-ipcr 0.0126 0.7378 0.122
UTASICS all-tf-idf-ipcr 0.0124 0.733 0.105
run id P R MAP
Evaluation Summary - Part Two 7
Tab
le6:
Measuresforbundle
S,task
Main
run
idP
P5
P10
P100
RR5
R10
R100
MAP
clefi
p-dcu
baseline
0.009
20.1
0.13
330.04
830.59
570.026
90.07
060.289
0.0615
clefi
p-dcu
Filtered2
0.008
50.13
330.13
330.04
170.57
90.057
0.10
230.2697
0.0729
clefi
p-dcu
Filtered
0.008
20.1
0.1
0.03
830.55
940.032
50.06
230.2471
0.0501
clefi
p-runClaim
sBOW
0.021
70.
0.03
330.02
170.19
710.
0.03
130.1971
0.0174
clefi
p-ugbm25
med
stan
dard
0.0.
0.0.
0.0.
0.
0.0.
clefi
p-uginfdocfreqBM25E
nglishTerms
0.0.
0.0.
0.0.
0.
0.0.
clefi
p-uginfdocfreqBM25F
renchTermsN
ew0.
0.0.
0.0.
0.
0.
0.0.
clefi
p-uginfdocfreqBM25
GermanTerms
0.0.
0.0.
0.0.
0.
0.0.
clefi
p-uginfdocfreqCosEnglishTerms
0.007
50.06
670.05
0.02
170.38
30.022
70.02
690.1316
0.0193
clefi
p-unigeRUN1
0.007
50.2
0.11
670.03
670.50
280.088
30.10
110.2759
0.0934
clefi
p-unigeRUN2
0.008
20.16
670.13
330.03
670.56
460.069
80.11
090.2872
0.0936
clefi
p-unigeRUN3
0.007
50.2
0.11
670.03
670.50
280.088
30.10
110.2759
0.0934
clefi
p-unigeRUN4
0.007
70.1
0.08
330.03
170.51
230.026
90.04
950.246
0.0413
clefi
p-unigeRUN5
0.004
50.1
0.06
670.01
670.35
10.049
90.06
270.1625
0.0395
cwibm25
0.007
20.06
670.05
0.01
830.51
040.008
30.02
120.0944
0.0208
cwiboolean
0.005
90.
0.0.00
670.31
080.
0.
0.0423
0.0031
cwicatego
rybm25
0.005
80.
0.01
670.01
0.38
10.
0.00
420.048
0.0066
cwicategory
0.006
20.03
330.05
0.02
50.47
440.004
20.02
120.1569
0.0197
hcu
geBiTeM
0.008
0.03
330.06
670.03
330.52
040.009
80.04
230.2698
0.0319
Hildesheim
MethodAnew
0.0.
0.0.
0.0.
0.
0.0.
humb1
0.007
90.23
330.15
0.03
330.40
250.161
20.18
910.3177
0.1631
NLELMethodA
0.005
80.1
0.13
330.03
170.37
0.029
80.09
660.2114
0.0574
TUD
800
noT
itle
0.008
80.1
0.06
670.04
170.56
990.021
20.02
530.2755
0.0466
UAIC
MethodAnew
0.000
20.03
330.01
670.00
170.01
280.012
80.01
280.0128
0.0026
UniN
Estrat1
0.005
0.06
670.06
670.01
830.36
550.028
30.05
10.1201
0.0346
UniN
Estrat2
0.007
30.13
330.08
330.03
50.55
330.068
40.07
820.3001
0.0779
UniN
Estrat3
0.009
80.1
0.08
330.04
170.56
810.041
20.06
380.3336
0.075
UniN
Estrat4
0.008
90.06
670.08
330.03
50.58
50.037
0.06
680.3084
0.0714
UniN
Estrat5
0.006
50.
0.03
330.03
170.45
440.
0.02
830.248
0.0197
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Con
tinued
onnextpage
Evaluation Summary - Part Two 8
Table
6:Measuresforbundle
S,task
Main,continued
run
idP
P5
P10
P100
RR5
R10
R100
MAP
UniN
Estrat6
0.009
60.13
330.06
670.03
330.64
110.051
0.05
10.2353
0.062
UniN
Estrat7
0.009
70.
0.03
330.02
830.65
140.
0.01
70.1738
0.0199
UniN
Estrat8
0.008
20.
0.05
0.02
330.55
950.
0.04
650.1758
0.0189
uscom
BM25
a0.008
40.16
670.13
330.03
330.54
020.075
50.12
540.3076
0.1165
uscom
BM25
at0.008
80.16
670.15
0.04
0.58
60.075
50.13
820.3386
0.1222
uscom
BM25b
0.009
60.1
0.05
0.03
330.60
120.026
80.02
680.2902
0.0416
uscom
BM25b
t0.009
90.1
0.08
330.03
330.60
390.029
80.05
240.2932
0.0503
uscom
BM25c
0.009
40.06
670.03
330.03
0.58
830.014
0.01
40.2732
0.032
uscom
BM25ct
0.009
60.03
330.06
670.03
170.58
990.004
20.03
960.2803
0.0372
uscom
BM25d
0.009
20.06
670.05
0.02
830.57
550.014
0.03
250.2603
0.0315
uscom
BM25d
t0.009
60.06
670.08
330.03
170.58
990.017
0.05
810.2803
0.0371
UTASIC
Sabs-des-ratf-GT
0.009
30.1
0.1
0.03
50.65
410.041
20.08
230.3084
0.0758
UTASIC
Sabs-des-ratf
0.009
0.03
330.03
330.03
330.61
180.018
50.03
130.3042
0.0447
UTASIC
Sab
s-tit-cla-ratf-G
T-ipcr
0.008
70.13
330.11
670.04
0.67
460.059
70.09
220.2765
0.0642
UTASIC
Sabs-tit-cla-ratf-ipcr
0.008
70.2
0.13
330.03
830.64
10.073
60.11
60.2958
0.0701
UTASIC
Sall-ratf-G
T-ipcr
0.009
30.23
330.15
0.04
170.67
50.092
20.11
480.3798
0.0953
UTASIC
Sall-ratf-ipcr
0.009
20.23
330.16
670.04
50.66
210.092
20.11
890.3846
0.0853
UTASIC
Sall-tf-idf-GT-ipcr
0.008
20.23
330.13
330.04
0.61
720.100
80.11
060.2863
0.1102
UTASIC
Sall-tf-idf-ipcr
0.008
70.23
330.13
330.04
670.63
530.100
80.11
060.3745
0.0981
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Table
7:Measuresforbundle
M,task
Main
run
idP
P5
P10
P100
RR5
R10
R100
MAP
clefi
p-dcu
baseline
0.00
810.07
50.1
0.04
370.59
750.020
10.053
0.3528
0.055
clefi
p-dcu
Filtered2
0.00
810.1
0.11
250.03
880.59
970.042
70.080
40.3383
0.0672
clefi
p-dcu
Filtered
0.00
810.07
50.07
50.03
620.59
240.024
40.046
70.3213
0.0484
clefi
p-ugbm25m
edstandard
0.00
090.
0.0.
0.02
570.
0.
0.0.0002
clefi
p-uginfdocfreqBM25
EnglishTerms
0.
0.0.
0.0.
0.0.
0.0.
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Continued
onnextpage
Evaluation Summary - Part Two 9
Tab
le7:Measuresforbundle
M,task
Main,continued
run
idP
P5
P10
P100
RR5
R10
R100
MAP
clefi
p-uginfdocfreqBM25F
renchTerms
0.
0.0.
0.0.
0.0.
0.0.
clefi
p-uginfdocfreqBM25G
erman
Terms
0.
0.0.
0.0.
0.0.
0.0.
clefi
p-uginfdocfreqCosE
nglishTerms
0.00
970.07
50.07
50.03
120.49
320.020
70.031
20.2188
0.0323
clefi
p-unigeRUN1
0.00
790.2
0.13
750.04
370.55
730.111
60.128
60.3687
0.0998
clefi
p-unigeRUN2
0.00
860.15
0.11
250.03
750.61
090.094
0.124
90.3588
0.0934
clefi
p-unigeRUN3
0.00
790.2
0.13
750.04
370.55
730.111
60.128
60.3687
0.0998
clefi
p-unigeRUN4
0.00
760.1
0.08
750.03
250.55
330.061
80.120
50.3242
0.0563
clefi
p-unigeRUN5
0.00
580.12
50.1
0.02
630.44
710.044
70.061
70.2003
0.0462
cwibm25
0.00
650.1
0.07
50.01
750.52
980.089
60.102
90.1578
0.0471
cwiboolean
0.01
860.02
50.01
250.00
620.32
010.003
70.003
70.0354
0.0036
cwicategorybm25
0.00
670.02
50.03
750.01
750.46
590.003
70.048
50.1414
0.0164
cwicatego
ry0.00
580.02
50.03
750.02
130.50
290.003
10.015
90.163
0.0164
hcu
geBiTeM
0.00
860.02
50.05
0.03
620.56
680.007
40.031
70.3494
0.0348
humb1
0.00
990.22
50.16
250.04
250.47
840.128
20.156
60.3658
0.149
TUD
800n
oTitle
0.00
950.12
50.1
0.04
880.62
60.023
20.033
70.372
0.0574
UniN
Estrat8
0.00
880.02
50.05
0.03
120.61
080.003
70.038
50.2862
0.0261
UTASIC
Sab
s-des-ratf-G
T0.01
010.1
0.12
50.04
250.69
650.034
60.076
50.3171
0.071
UTASIC
Sab
s-des-ratf
0.01
0.1
0.08
750.04
630.66
840.024
90.041
90.4046
0.0677
UTASIC
Sabs-tit-cla-ratf-G
T-ipcr
0.00
930.12
50.1
0.03
630.70
080.048
40.072
80.2257
0.0539
UTASIC
Sab
s-tit-cla-ratf-ipcr
0.00
930.17
50.11
250.03
50.67
560.058
90.090
60.2402
0.0583
UTASIC
Sall-ratf-G
T-ipcr
0.01
060.2
0.13
750.04
630.72
680.072
80.093
40.367
0.0851
UTASIC
Sall-ratf-ipcr
0.01
050.17
50.13
750.04
0.71
720.069
10.092
90.3449
0.0717
UTASIC
Sall-tf-idf-GT-ipcr
0.00
980.2
0.11
250.04
750.68
350.079
30.086
60.2662
0.0954
UTASIC
Sall-tf-idf-ipcr
0.00
990.17
50.1
0.04
50.68
970.075
60.083
0.3103
0.0792
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Evaluation Summary - Part Two 10
Tab
le8:
Measuresforbundle
XL,task
Main
run
idP
P5
P10
P100
RR5
R10
R100
MAP
clefi
p-dcu
baseline
0.011
20.08
330.14
170.06
750.65
990.021
80.066
40.4171
0.084
clefi
p-dcu
Filtered2
0.011
80.11
670.15
830.05
830.62
50.039
80.087
60.3797
0.0935
clefi
p-dcu
Filtered
0.011
80.1
0.12
50.05
670.62
010.027
60.062
20.3684
0.0759
clefi
p-uginfdocfreqCosE
nglishTerms
0.013
10.13
330.11
670.04
830.54
520.030
10.053
50.2673
0.0614
clefi
p-unigeRUN1
0.010
20.26
670.17
50.05
50.58
570.109
90.127
70.3619
0.1156
clefi
p-unigeRUN2
0.010
30.21
670.15
0.04
750.60
130.095
0.122
0.3381
0.1077
clefi
p-unigeRUN3
0.010
20.26
670.17
50.05
580.58
570.109
90.127
70.3652
0.1157
clefi
p-unigeRUN4
0.010
70.11
670.16
670.05
330.61
70.052
70.131
0.3557
0.0837
clefi
p-unigeRUN5
0.007
70.11
670.10
830.03
420.46
880.037
20.060
10.2076
0.0481
cwibm25
0.009
80.11
670.08
330.04
750.60
630.076
40.088
40.2642
0.0736
cwiboolean
0.052
80.03
330.04
170.03
170.36
590.006
60.018
20.1451
0.0258
cwicatego
ry0.013
10.08
330.07
50.03
920.53
960.016
40.033
20.2223
0.0407
hcu
geBiTeM
0.026
70.13
330.12
50.05
50.57
880.034
90.065
60.3712
0.0748
humb1
0.031
60.28
330.23
330.06
170.50
890.124
90.172
60.4129
0.1664
TUD
800
noT
itle
0.012
90.15
0.11
670.06
420.64
860.028
80.042
90.3951
0.0685
UniN
Estrat8
0.008
80.08
330.07
50.03
250.53
760.018
20.044
60.2393
0.0337
UTASIC
Sabs-des-ratf-GT
0.012
40.13
330.15
0.06
670.71
420.037
60.077
60.3852
0.098
UTASIC
Sab
s-des-ratf
0.012
40.18
330.12
50.07
080.70
30.047
0.061
30.4508
0.1034
UTASIC
Sabs-tit-cla-ratf-G
T-ipcr
0.012
40.2
0.16
670.05
920.76
420.059
90.093
80.32
0.0956
UTASIC
Sab
s-tit-cla-ratf-ipcr
0.012
10.18
330.14
170.05
670.72
920.057
70.093
60.3179
0.0874
UTASIC
Sall-ratf-G
T-ipcr
0.013
60.26
670.19
170.07
170.78
30.075
20.107
40.4331
0.1245
UTASIC
Sall-ratf-ipcr
0.013
10.16
670.17
50.06
0.76
160.061
50.107
30.379
0.099
UTASIC
Sall-tf-idf-GT-ipcr
0.012
60.23
330.15
830.07
080.73
780.073
50.090
60.36
0.122
UTASIC
Sall-tf-idf-ipcr
0.012
40.2
0.14
170.06
580.73
30.073
0.091
40.3716
0.105
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Evaluation Summary - Part Two 11
4.2 Language Tasks
Table 9: Measures for bundle S, task DE
run id P R MAPhcuge BiTeM 0.0073 0.4622 0.0242
humb 1 0.0073 0.3859 0.0373TUD 800noTitle 0.0085 0.5991 0.0516
run id P R MAP
Table 10: Measures for bundle M, task DE
run id P R MAPhcuge BiTeM 0.0073 0.4937 0.0231
humb 1 0.0094 0.4659 0.0597TUD 800noTitle 0.009 0.6405 0.0596
run id P R MAP
Table 11: Measures for bundle XL, task DE
run id P R MAPhumb 1 0.0315 0.5095 0.0782
TUD 800noTitle 0.0134 0.6489 0.0753
run id P R MAP
Evaluation Summary - Part Two 12
Tab
le12
:Measuresforbundle
S,task
DE
run
idP
P5
P10
P100
RR5
R10
R100
MAP
hcu
geBiTeM
0.00
730.033
30.06
670.02
670.46
220.00
980.065
40.20
480.0242
humb1
0.00
730.066
70.08
330.01
330.38
590.03
130.115
30.20
460.0373
TUD
800n
oTitle
0.00
850.1
0.11
670.03
830.59
910.02
120.043
50.26
710.0516
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Tab
le13
:Measuresforbundle
M,task
DE
run
idP
P5
P10
P100
RR5
R10
R100
MAP
hcu
geBiTeM
0.00
730.02
50.05
0.02
370.49
370.00
740.049
10.27
860.02
31humb1
0.00
940.12
50.12
50.02
750.46
590.03
450.104
90.28
090.05
97TUD
800n
oTitle
0.00
90.15
0.13
750.04
50.64
050.02
690.047
30.32
410.05
96
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Table
14:Measuresforbundle
XL,task
DE
run
idP
P5
P10
P100
RR5
R10
R100
MAP
humb1
0.03
150.166
70.13
330.04
080.50
950.05
210.108
30.31
030.0782
TUD
800n
oTitle
0.01
340.166
70.15
830.06
170.64
890.03
130.064
50.36
340.0753
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Evaluation Summary - Part Two 13
Table 15: Measures for bundle S, task EN
run id P R MAPclefip-run ClaimsBOW 0.0217 0.1971 0.0174
hcuge BiTeM 0.0069 0.4396 0.0272humb 1 0.0079 0.4025 0.1414
TUD 800noTitle 0.0087 0.5427 0.0407UAIC MethodAnew 0.0002 0.0128 0.0026
run id P R MAP
Table 16: Measures for bundle M, task EN
run id P R MAPhcuge BiTeM 0.0078 0.5062 0.0292
humb 1 0.0099 0.4784 0.1137TUD 800noTitle 0.009 0.5945 0.0429
run id P R MAP
Table 17: Measures for bundle XL, task EN
run id P R MAPhumb 1 0.0319 0.5208 0.1319
TUD 800noTitle 0.0127 0.6244 0.0595
run id P R MAP
Evaluation Summary - Part Two 14
Tab
le18
:Measuresforbundle
S,task
EN
run
idP
P5
P10
P100
RR5
R10
R100
MAP
clefi
p-runClaim
sBOW
0.02
170.
0.03
330.02
170.19
710.
0.03
130.197
10.0174
hcu
geBiTeM
0.00
690.
0.01
670.02
830.43
960.
0.00
420.246
50.0272
humb1
0.00
790.2
0.11
670.02
830.40
250.14
260.14
680.271
30.1414
TUD
800n
oTitle
0.00
870.06
670.08
330.03
50.54
270.01
70.08
660.219
20.0407
UAIC
MethodAnew
0.00
020.03
330.01
670.00
170.01
280.01
280.01
280.012
80.0026
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Table
19:Measuresforbundle
M,task
EN
run
idP
P5
P10
P100
RR5
R10
R100
MAP
hcu
geBiTeM
0.00
780.
0.01
250.03
380.50
620.
0.003
10.33
560.02
92humb1
0.00
990.15
0.08
750.03
120.47
840.10
70.110
10.34
680.11
37TUD
800n
oTitle
0.00
90.07
50.07
50.03
880.59
450.01
640.068
60.31
510.04
29
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Tab
le20
:Measuresforbundle
XL,task
EN
run
idP
P5
P10
P100
RR5
R10
R100
MAP
humb1
0.03
190.216
70.14
170.05
50.52
080.10
150.113
0.40
340.1319
TUD
800n
oTitle
0.01
270.1
0.10
830.05
920.62
440.02
730.075
50.35
680.0595
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Evaluation Summary - Part Two 15
Table 21: Measures for bundle S, task FR
run id P R MAPhcuge BiTeM 0.0056 0.4018 0.0425
humb 1 0.0064 0.365 0.067TUD 800noTitle 0.0083 0.5488 0.0296
run id P R MAP
Table 22: Measures for bundle M, task FR
run id P R MAPhcuge BiTeM 0.0072 0.4704 0.038
humb 1 0.0087 0.4503 0.0551TUD 800noTitle 0.0082 0.5844 0.0264
run id P R MAP
Table 23: Measures for bundle XL, task FR
run id P R MAPhumb 1 0.0311 0.499 0.0786
TUD 800noTitle 0.0134 0.6209 0.0458
run id P R MAP
Evaluation Summary - Part Two 16
Table
24:Measuresforbundle
S,task
FR
run
idP
P5
P10
P100
RR5
R10
R100
MAP
hcu
geBiTeM
0.00
560.033
30.03
330.02
50.40
180.04
170.045
80.27
780.0425
humb1
0.00
640.133
30.08
330.01
330.36
50.07
370.115
30.20
460.067
TUD
800n
oTitle
0.00
830.066
70.06
670.03
330.54
880.02
270.059
30.21
810.0296
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Tab
le25
:Measuresforbundle
M,task
FR
run
idP
P5
P10
P100
RR5
R10
R100
MAP
hcu
geBiTeM
0.00
720.02
50.02
50.02
870.47
040.03
120.034
40.35
170.038
humb1
0.00
870.1
0.06
250.01
880.45
030.05
530.086
50.21
720.05
51TUD
800n
oTitle
0.00
820.05
0.05
0.03
250.58
440.01
70.044
50.26
160.02
64
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Table
26:Measuresforbundle
XL,task
FR
run
idP
P5
P10
P100
RR5
R10
R100
MAP
humb1
0.03
110.15
0.12
50.03
250.49
90.06
570.114
20.25
890.07
86TUD
800n
oTitle
0.01
340.08
330.07
50.04
830.62
090.02
150.046
0.30
20.04
58
run
idP
P5
P10
P100
RR5
R10
R100
MAP
Evaluation Summary - Part Two 17
References
[1] Florina Piroi, Giovanna Roda, and Veronika Zenz. CLEF-IP 2009 Evaluation Summary. July2009.
Evaluation Summary - Part Two 18
List of Tables
1 List of active participants and runs submitted . . . . . . . . . . . . . . . . . . . . 11 List of active participants and runs submitted . . . . . . . . . . . . . . . . . . . . 22 Topics manually assessed by patent experts . . . . . . . . . . . . . . . . . . . . . 23 Bundle S, task Main . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Bundle M, task Main . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Bundle XL, task Main . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Bundle S , task Main . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Bundle M , task Main . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 Bundle XL , task Main . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Bundle S, task DE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1110 Bundle M, task DE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1111 Bundle XL, task DE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1112 Bundle S , task DE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1213 Bundle M , task DE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1214 Bundle XL , task DE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1215 Bundle S, task EN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1316 Bundle M, task EN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1317 Bundle XL, task EN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1318 Bundle S , task EN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1419 Bundle M , task EN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1420 Bundle XL , task EN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1421 Bundle S, task FR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1522 Bundle M, task FR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1523 Bundle XL, task FR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1524 Bundle S , task FR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1625 Bundle M , task FR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1626 Bundle XL , task FR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
