Introduction to Information Retrieval
CS3245
Information Retrieval
Lecture 9: IR Evaluation 9
CS3245 – Information Retrieval
Last TimeThe VSM Reloaded
… optimized for your pleasure!
Improvements to the computation and selection process
Use of heuristics to avoid unnecessary / time consuming computations1. Index elimination 2. Tiered lists3. Early termination 4. Cluster pruning
Information Retrieval 2
Ch. 7
CS3245 – Information Retrieval
Today: Evaluation How do we know if our results are any good? Benchmarks Precision and Recall; Composite measures Test collection
A/B Testing
Information Retrieval 3
Ch. 8
CS3245 – Information Retrieval
EVALUATING SEARCH ENGINES
Information Retrieval 4
CS3245 – Information Retrieval
Measures for a search engine How fast does it index? Number of documents/hour (Average document size)
How fast does it search? Latency as a function of index size
Expressiveness of query language? Ability to express complex information needs Speed on complex queries
How much does it cost? Fee required to use the search engine
Information Retrieval 5
Sec. 8.6
CS3245 – Information Retrieval
Measures for a search engine All of the preceding criteria are measurable: we can
quantify speed/size we can make expressiveness precise
The key measure: user happiness Speed of response/size of index are factors But blindingly fast, useless answers won’t make a user
happy
Need a way of quantifying user happiness
Information Retrieval 6
Sec. 8.6
CS3245 – Information Retrieval
Measuring user happiness Question: who is the user we are trying to make happy? Depends on the setting
Web engine: Users find what they want and return to the engine next time
Can measure rate of returning users User completes their task – search as a means, not end
eCommerce site: Users find what they want and buy
Is it the end-user, or the eCommerce site, whose happiness we measure? Measure time to purchase, or fraction of searchers who become buyers?
Information Retrieval 7
Sec. 8.6.2
CS3245 – Information Retrieval
Measuring user happiness Enterprise (company/govt/academic): Care about “user productivity”
How much time do my users save when looking for information? Many other criteria having to do with breadth of access, secure
access, etc.
Information Retrieval 8
Sec. 8.6.2
CS3245 – Information Retrieval
Happiness: elusive to measure Most common proxy: relevance of search results But how do you measure relevance?
We’ll examine one method and the issues around it Relevance measurement requires 3 elements:
1. A set document collection2. A set suite of queries3. A usually binary assessment of either Relevant or
Non-relevant for each query and each document Some work on graded relevance, but not the standard
Information Retrieval 9
Sec. 8.1
CS3245 – Information Retrieval
Evaluating an IR system Note: the information need is translated into a
query Relevance is assessed relative to the information
need not the query
E.g., Information need: I'm looking for information on whether drinking red wine is more effective at reducing your risk of heart attacks than white wine.
Query: wine red white heart attack effective
i.e., we evaluate whether the doc addresses the information need, not whether it has these words
Information Retrieval 10
Sec. 8.1
CS3245 – Information Retrieval
Why it’s important:Example Think-Aloud Session
Information Retrieval 11
Slide courtesy Google Inc.
CS3245 – Information Retrieval
Unranked retrieval evaluation:Precision and Recall
Information Retrieval 12
Relevant Non-relevant
Retrieved true positive false positive
Not Retrieved false negative true negative
Sec. 8.3
Precision: fraction of retrieved docs that are relevant = P(relevant | retrieved)
Recall: fraction of relevant docs that are retrieved = P(retrieved | relevant)
Precision P = tp / (tp + fp)Recall R = tp / (tp + fn)
CS3245 – Information Retrieval
Should we use accuracy forevaluation instead?
Information Retrieval 13
Sec. 8.3
Given a query, a Boolean engine classifies each doc as Relevant or Non-Relevant
The accuracy of an engine: the fraction of these classifications that are correct Accuracy = (tp + tn) / (tp + fp + tn + fn)
Accuracy is a commonly used evaluation measure in classification (e.g., HW1)
Quick Question: Why is this not a very useful evaluation measure in IR?
CS3245 – Information Retrieval
Precision/Recall You can get high recall (but low precision) by
retrieving all docs for all queries! Recall is a non-decreasing function of the number
of docs retrieved
In a good system, precision decreases as either the number of docs retrieved or recall increases This is not a theorem, but a result with strong empirical
confirmation
Information Retrieval 14
Sec. 8.3
CS3245 – Information Retrieval
Difficulties in using precision/recall Should average over large document
collection/query ensembles Need human relevance assessments But people are subjective; they aren’t reliable assessors
Assessments have to be binary Can we give graded assessments?
Heavily skewed by collection/queries pairing Results may not translate from one collection to another
Information Retrieval 15
Sec. 8.3
We’ll return to this point later
CS3245 – Information Retrieval
A combined measure: F
Information Retrieval 16
Sec. 8.3
Combined measure that assesses precision / recall tradeoff is F measure (weighted harmonic mean):
𝐹𝐹 = 1𝛼𝛼1𝑃𝑃+(1−𝛼𝛼)
1𝑅𝑅
= 𝛽𝛽2+1 𝑃𝑃𝑃𝑃𝛽𝛽2𝑃𝑃+𝑃𝑃
People usually use balanced F1 measure i.e., with β = 1 or α = 1
2
Harmonic mean is a conservative average
2𝑃𝑃𝑃𝑃𝑃𝑃 + 𝑃𝑃
CS3245 – Information Retrieval
Information Retrieval 17
Sec. 8.3
Combined Measures
0
20
40
60
80
100
0 20 40 60 80 100
Precision (Recall fixed at 70%)
MinimumMaximumArithmeticGeometricHarmonic
Co
mb
ined
Avera
ge
Blanks on slides, you may want to fill in
Chart1
00000
11111
22222
33333
44444
55555
66666
77777
88888
99999
1010101010
1111111111
1212121212
1313131313
1414141414
1515151515
1616161616
1717171717
1818181818
1919191919
2020202020
2121212121
2222222222
2323232323
2424242424
2525252525
2626262626
2727272727
2828282828
2929292929
3030303030
3131313131
3232323232
3333333333
3434343434
3535353535
3636363636
3737373737
3838383838
3939393939
4040404040
4141414141
4242424242
4343434343
4444444444
4545454545
4646464646
4747474747
4848484848
4949494949
5050505050
5151515151
5252525252
5353535353
5454545454
5555555555
5656565656
5757575757
5858585858
5959595959
6060606060
6161616161
6262626262
6363636363
6464646464
6565656565
6666666666
6767676767
6868686868
6969696969
7070707070
7171717171
7272727272
7373737373
7474747474
7575757575
7676767676
7777777777
7878787878
7979797979
8080808080
8181818181
8282828282
8383838383
8484848484
8585858585
8686868686
8787878787
8888888888
8989898989
9090909090
9191919191
9292929292
9393939393
9494949494
9595959595
9696969696
9797979797
9898989898
9999999999
100100100100100
Minimum
Maximum
Arithmetic
Geometric
Harmonic
Precision (Recall fixed at 70%)
Combined Measures
0
70
35
0
0
1
70
35.5
8.3666002653
1.9718309859
2
70
36
11.8321595662
3.8888888889
3
70
36.5
14.4913767462
5.7534246575
4
70
37
16.7332005307
7.5675675676
5
70
37.5
18.7082869339
9.3333333333
6
70
38
20.4939015319
11.0526315789
7
70
38.5
22.1359436212
12.7272727273
8
70
39
23.6643191324
14.358974359
9
70
39.5
25.099800796
15.9493670886
10
70
40
26.4575131106
17.5
11
70
40.5
27.748873851
19.012345679
12
70
41
28.9827534924
20.487804878
13
70
41.5
30.166206258
21.9277108434
14
70
42
31.304951685
23.3333333333
15
70
42.5
32.403703492
24.7058823529
16
70
43
33.4664010614
26.0465116279
17
70
43.5
34.4963766213
27.3563218391
18
70
44
35.4964786986
28.6363636364
19
70
44.5
36.4691650576
29.8876404494
20
70
45
37.4165738677
31.1111111111
21
70
45.5
38.3405790254
32.3076923077
22
70
46
39.2428337407
33.4782608696
23
70
46.5
40.1248052955
34.623655914
24
70
47
40.9878030638
35.7446808511
25
70
47.5
41.8330013267
36.8421052632
26
70
48
42.6614580154
37.9166666667
27
70
48.5
43.4741302386
38.9690721649
28
70
49
44.2718872424
40
29
70
49.5
45.0555213043
41.0101010101
30
70
50
45.8257569496
42
31
70
50.5
46.5832587954
42.9702970297
32
70
51
47.3286382648
43.9215686275
33
70
51.5
48.0624593628
44.854368932
34
70
52
48.7852436706
45.7692307692
35
70
52.5
49.4974746831
46.6666666667
36
70
53
50.199601592
47.5471698113
37
70
53.5
50.8920426
48.4112149533
38
70
54
51.5751878329
49.2592592593
39
70
54.5
52.2494019105
50.0917431193
40
70
55
52.9150262213
50.9090909091
41
70
55.5
53.5723809439
51.7117117117
42
70
56
54.2217668469
52.5
43
70
56.5
54.8634668974
53.2743362832
44
70
57
55.497747702
54.0350877193
45
70
57.5
56.1248608016
54.7826086957
46
70
58
56.7450438364
55.5172413793
47
70
58.5
57.3585215988
56.2393162393
48
70
59
57.9655069848
56.9491525424
49
70
59.5
58.5662018574
57.6470588235
50
70
60
59.160797831
58.3333333333
51
70
60.5
59.7494769852
59.0082644628
52
70
61
60.332412516
59.6721311475
53
70
61.5
60.9097693314
60.325203252
54
70
62
61.4817045958
60.9677419355
55
70
62.5
62.04836823
61.6
56
70
63
62.60990337
62.2222222222
57
70
63.5
63.1664467894
62.8346456693
58
70
64
63.7181292883
63.4375
59
70
64.5
64.2650760522
64.0310077519
60
70
65
64.8074069841
64.6153846154
61
70
65.5
65.3452370108
65.1908396947
62
70
66
65.878676368
65.7575757576
63
70
66.5
66.4078308635
66.3157894737
64
70
67
66.9328021227
66.8656716418
65
70
67.5
67.4536878162
67.4074074074
66
70
68
67.9705818719
67.9411764706
67
70
68.5
68.4835746731
68.4671532847
68
70
69
68.9927532426
68.9855072464
69
70
69.5
69.4982014156
69.4964028777
70
70
70
70
70
70
71
70.5
70.4982269281
70.4964539007
70
72
71
70.9929573972
70.985915493
70
73
71.5
71.4842640026
71.4685314685
70
74
72
71.9722168618
71.9444444444
70
75
72.5
72.4568837309
72.4137931034
70
76
73
72.9383301152
72.8767123288
70
77
73.5
73.4166193719
73.3333333333
70
78
74
73.8918128076
73.7837837838
70
79
74.5
74.3639697703
74.2281879195
70
80
75
74.8331477355
74.6666666667
70
81
75.5
75.2994023881
75.0993377483
70
82
76
75.7627876995
75.5263157895
70
83
76.5
76.2233560006
75.9477124183
70
84
77
76.6811580507
76.3636363636
70
85
77.5
77.1362431027
76.7741935484
70
86
78
77.5886589651
77.1794871795
70
87
78.5
78.0384520605
77.5796178344
70
88
79
78.4856674814
77.9746835443
70
89
79.5
78.9303490427
78.3647798742
70
90
80
79.3725393319
78.75
70
91
80.5
79.8122797569
79.1304347826
70
92
81
80.249610591
79.5061728395
70
93
81.5
80.6845710158
79.8773006135
70
94
82
81.1171991627
80.243902439
70
95
82.5
81.5475321515
80.6060606061
70
96
83
81.9756061277
80.9638554217
70
97
83.5
82.4014562978
81.3173652695
70
98
84
82.8251169634
81.6666666667
70
99
84.5
83.2466215531
82.0118343195
70
100
85
83.6660026534
82.3529411765
Sheet1
PrecisionRecallMinimumMaximumArithmeticGeometricHarmonic
0700703500
14,000,000.0017017035.58.36660026531.9718309859
2702703611.83215956623.8888888889
37037036.514.49137674625.7534246575
4704703716.73320053077.5675675676
57057037.518.70828693399.3333333333
6706703820.493901531911.0526315789
77077038.522.135943621212.7272727273
8708703923.664319132414.358974359
97097039.525.09980079615.9493670886
107010704026.457513110617.5
1170117040.527.74887385119.012345679
127012704128.982753492420.487804878
1370137041.530.16620625821.9277108434
147014704231.30495168523.3333333333
1570157042.532.40370349224.7058823529
167016704333.466401061426.0465116279
1770177043.534.496376621327.3563218391
187018704435.496478698628.6363636364
1970197044.536.469165057629.8876404494
207020704537.416573867731.1111111111
2170217045.538.340579025432.3076923077
227022704639.242833740733.4782608696
2370237046.540.124805295534.623655914
247024704740.987803063835.7446808511
2570257047.541.833001326736.8421052632
267026704842.661458015437.9166666667
2770277048.543.474130238638.9690721649
287028704944.271887242440
2970297049.545.055521304341.0101010101
307030705045.825756949642
3170317050.546.583258795442.9702970297
327032705147.328638264843.9215686275
3370337051.548.062459362844.854368932
347034705248.785243670645.7692307692
3570357052.549.497474683146.6666666667
367036705350.19960159247.5471698113
3770377053.550.892042648.4112149533
387038705451.575187832949.2592592593
3970397054.552.249401910550.0917431193
407040705552.915026221350.9090909091
4170417055.553.572380943951.7117117117
427042705654.221766846952.5
4370437056.554.863466897453.2743362832
447044705755.49774770254.0350877193
4570457057.556.124860801654.7826086957
467046705856.745043836455.5172413793
4770477058.557.358521598856.2393162393
487048705957.965506984856.9491525424
4970497059.558.566201857457.6470588235
507050706059.16079783158.3333333333
5170517060.559.749476985259.0082644628
527052706160.33241251659.6721311475
5370537061.560.909769331460.325203252
547054706261.481704595860.9677419355
5570557062.562.0483682361.6
567056706362.6099033762.2222222222
5770577063.563.166446789462.8346456693
587058706463.718129288363.4375
5970597064.564.265076052264.0310077519
607060706564.807406984164.6153846154
6170617065.565.345237010865.1908396947
627062706665.87867636865.7575757576
6370637066.566.407830863566.3157894737
647064706766.932802122766.8656716418
6570657067.567.453687816267.4074074074
667066706867.970581871967.9411764706
6770677068.568.483574673168.4671532847
687068706968.992753242668.9855072464
6970697069.569.498201415669.4964028777
70707070707070
7170707170.570.498226928170.4964539007
727070727170.992957397270.985915493
7370707371.571.484264002671.4685314685
747070747271.972216861871.9444444444
7570707572.572.456883730972.4137931034
767070767372.938330115272.8767123288
7770707773.573.416619371973.3333333333
787070787473.891812807673.7837837838
7970707974.574.363969770374.2281879195
807070807574.833147735574.6666666667
8170708175.575.299402388175.0993377483
827070827675.762787699575.5263157895
8370708376.576.223356000675.9477124183
847070847776.681158050776.3636363636
8570708577.577.136243102776.7741935484
867070867877.588658965177.1794871795
8770708778.578.038452060577.5796178344
887070887978.485667481477.9746835443
8970708979.578.930349042778.3647798742
907070908079.372539331978.75
9170709180.579.812279756979.1304347826
927070928180.24961059179.5061728395
9370709381.580.684571015879.8773006135
947070948281.117199162780.243902439
9570709582.581.547532151580.6060606061
967070968381.975606127780.9638554217
9770709783.582.401456297881.3173652695
987070988482.825116963481.6666666667
9970709984.583.246621553182.0118343195
10070701008583.666002653482.3529411765
Sheet1
Minimum
Maximum
Arithmetic
Geometric
Harmonic
Precision (Recall fixed at 70%)
Combined Measures
Sheet2
Sheet3
CS3245 – Information Retrieval
Evaluating ranked results Evaluation of ranked results: The relevant documents should be ranked higher than the
non relevant documents By taking various numbers of the top returned documents
(levels of recall), we can produce a precision-recall curve
Information Retrieval 18
Sec. 8.4
CS3245 – Information Retrieval
A precision-recall curve
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0Recall
Pre
cisi
on
Information Retrieval 19
Sec. 8.4
Chart1
0.02500.575
0.0250.025
0.050.025
0.0750.05
0.0750.075
0.10.075
0.10.1
0.10.1
0.10.1
0.1250.1
0.150.125
0.150.15
0.1750.15
0.20.175
0.2250.2
0.250.225
0.250.25
0.250.25
0.2750.25
0.2750.275
0.2750.275
0.30.275
0.30.3
0.30.3
0.3250.3
0.3250.325
0.3250.325
0.3250.325
0.3250.325
0.350.325
0.350.35
0.350.35
0.3750.35
0.3750.375
0.3750.375
0.3750.375
0.40.375
0.4250.4
0.4250.425
0.4250.425
0.4250.425
0.450.425
0.450.45
0.450.45
0.450.45
0.4750.45
0.4750.475
0.4750.475
0.50.475
0.50.5
0.50.5
0.50.5
0.50.5
0.5250.5
0.5250.525
0.5250.525
0.5250.525
0.550.525
0.550.55
0.550.55
0.550.55
0.5750.55
0.5750.575
0.5750.575
0.5750.575
0.5750.575
0.60.575
0.60.6
0.60.6
0.60.6
0.60.6
0.60.6
0.60.6
0.60.6
0.6250.6
0.6250.625
0.6250.625
0.6250.625
0.6250.625
0.6250.625
0.6250.625
0.6250.625
0.650.625
0.650.65
0.650.65
0.6750.65
0.6750.675
0.6750.675
0.6750.675
0.6750.675
0.6750.675
0.6750.675
0.6750.675
0.6750.675
0.6750.675
0.6750.675
0.6750.675
0.70.675
0.70.7
0.70.7
0.70.7
0.70.7
0.70.7
0.70.7
0.70.7
0.70.7
0.70.7
0.70.7
0.70.7
0.7250.7
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.7250.725
0.750.725
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.750.75
0.7750.75
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.7750.775
0.80.775
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.80.8
0.8250.8
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.8250.825
0.850.825
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.850.85
0.8750.85
0.8750.875
0.90.875
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.90.9
0.9250.9
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.9250.925
0.950.925
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.950.95
0.9750.95
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
0.9750.975
10.975
11
11
11
11
11
11
11
11
11
11
11
11
11
1
Precison
interp prec
F1
Recall
Precision
1
1
0.3709677419
0.5
1
0.6666666667
0.75
0.75
0.75
0.6
0.75
0.6666666667
0.6666666667
0.5714285714
0.6666666667
0.5
0.625
0.4444444444
0.625
0.5
0.625
0.5454545455
0.625
0.5
0.625
0.5384615385
0.625
0.5714285714
0.625
0.6
0.625
0.625
0.625
0.5882352941
0.625
0.5555555556
0.5882352941
0.5789473684
0.5789473684
0.55
0.5789473684
0.5238095238
0.55
0.5454545455
0.5454545455
0.5217391304
0.5454545455
0.5
0.5217391304
0.52
0.52
0.5
0.52
0.4814814815
0.5
0.4642857143
0.4814814815
0.4482758621
0.4666666667
0.4666666667
0.4666666667
0.4516129032
0.4666666667
0.4375
0.4545454545
0.4545454545
0.4545454545
0.4411764706
0.4545454545
0.4285714286
0.4473684211
0.4166666667
0.4473684211
0.4324324324
0.4473684211
0.4473684211
0.4473684211
0.4358974359
0.4473684211
0.425
0.4358974359
0.4146341463
0.4285714286
0.4285714286
0.4285714286
0.4186046512
0.4285714286
0.4090909091
0.4186046512
0.4
0.4130434783
0.4130434783
0.4130434783
0.4042553191
0.4130434783
0.3958333333
0.4081632653
0.4081632653
0.4081632653
0.4
0.4081632653
0.3921568627
0.4
0.3846153846
0.3921568627
0.3773584906
0.3888888889
0.3888888889
0.3888888889
0.3818181818
0.3888888889
0.375
0.3818181818
0.3684210526
0.3793103448
0.3793103448
0.3793103448
0.3728813559
0.3793103448
0.3666666667
0.3728813559
0.3606557377
0.3709677419
0.3709677419
0.3709677419
0.3650793651
0.3709677419
0.359375
0.3650793651
0.3538461538
0.359375
0.3484848485
0.3582089552
0.3582089552
0.3582089552
0.3529411765
0.3582089552
0.347826087
0.3529411765
0.3428571429
0.347826087
0.338028169
0.3428571429
0.3333333333
0.338028169
0.3287671233
0.3333333333
0.3243243243
0.3333333333
0.3333333333
0.3333333333
0.3289473684
0.3333333333
0.3246753247
0.3289473684
0.3205128205
0.3246753247
0.3164556962
0.3205128205
0.3125
0.3164556962
0.3086419753
0.3139534884
0.3048780488
0.3139534884
0.313253012
0.3139534884
0.3095238095
0.3139534884
0.3058823529
0.3139534884
0.3139534884
0.3139534884
0.3103448276
0.3139534884
0.3068181818
0.3103448276
0.3033707865
0.3068181818
0.3
0.3033707865
0.2967032967
0.3
0.2934782609
0.2967032967
0.2903225806
0.2934782609
0.2872340426
0.2903225806
0.2842105263
0.2872340426
0.28125
0.2857142857
0.2783505155
0.2857142857
0.2857142857
0.2857142857
0.2828282828
0.2857142857
0.28
0.2828282828
0.2772277228
0.28
0.2745098039
0.2772277228
0.2718446602
0.2745098039
0.2692307692
0.2718446602
0.2666666667
0.2692307692
0.2641509434
0.2666666667
0.261682243
0.2641509434
0.2592592593
0.2636363636
0.2568807339
0.2636363636
0.2636363636
0.2636363636
0.2612612613
0.2636363636
0.2589285714
0.2612612613
0.2566371681
0.2589285714
0.2543859649
0.2566371681
0.252173913
0.2543859649
0.25
0.252173913
0.2478632479
0.25
0.2457627119
0.2478632479
0.243697479
0.2457627119
0.2416666667
0.243697479
0.2396694215
0.2416666667
0.237704918
0.2396694215
0.2357723577
0.237704918
0.2338709677
0.2357723577
0.232
0.234375
0.2301587302
0.234375
0.2283464567
0.234375
0.234375
0.234375
0.2325581395
0.234375
0.2307692308
0.2325581395
0.2290076336
0.2307692308
0.2272727273
0.2290076336
0.2255639098
0.2272727273
0.223880597
0.2255639098
0.2222222222
0.223880597
0.2205882353
0.2222222222
0.2189781022
0.2205882353
0.2173913043
0.2189781022
0.2158273381
0.2173913043
0.2142857143
0.2158273381
0.2127659574
0.2142857143
0.2112676056
0.2127659574
0.2097902098
0.2112676056
0.2083333333
0.2097902098
0.2068965517
0.2083333333
0.2054794521
0.2068965517
0.2040816327
0.2054794521
0.2027027027
0.2040816327
0.2013422819
0.2027027027
0.2
0.2013422819
0.1986754967
0.2
0.1973684211
0.1986754967
0.1960784314
0.1973684211
0.1948051948
0.1960784314
0.1935483871
0.1948051948
0.1923076923
0.1935483871
0.1910828025
0.1923076923
0.1898734177
0.1910828025
0.1886792453
0.1898734177
0.1875
0.1886792453
0.1863354037
0.1875
0.1851851852
0.1863354037
0.1840490798
0.1851851852
0.1829268293
0.1840490798
0.1818181818
0.1829268293
0.1807228916
0.1818181818
0.1796407186
0.1807228916
0.1785714286
0.1802325581
0.1775147929
0.1802325581
0.1764705882
0.1802325581
0.1754385965
0.1802325581
0.1802325581
0.1802325581
0.1791907514
0.1802325581
0.1781609195
0.1791907514
0.1771428571
0.1781609195
0.1761363636
0.1771428571
0.1751412429
0.1761363636
0.1741573034
0.1751412429
0.1731843575
0.1741573034
0.1722222222
0.1731843575
0.1712707182
0.1722222222
0.1703296703
0.1712707182
0.1693989071
0.1703296703
0.1684782609
0.1693989071
0.1675675676
0.1684782609
0.1666666667
0.1675675676
0.1657754011
0.1666666667
0.164893617
0.1657754011
0.164021164
0.164893617
0.1631578947
0.164021164
0.1623036649
0.1631578947
0.1614583333
0.1623036649
0.1606217617
0.1614583333
0.1597938144
0.1606217617
0.158974359
0.1597938144
0.1581632653
0.158974359
0.1573604061
0.1581632653
0.1565656566
0.1573604061
0.1557788945
0.1565656566
0.155
0.1557788945
0.1542288557
0.155
0.1534653465
0.1542288557
0.1527093596
0.1534653465
0.1519607843
0.1527093596
0.1512195122
0.1519607843
0.1504854369
0.1512195122
0.1497584541
0.1504854369
0.1490384615
0.1497584541
0.1483253589
0.1490384615
0.1476190476
0.1483253589
0.1469194313
0.1476190476
0.1462264151
0.1469194313
0.1455399061
0.1462264151
0.1448598131
0.1455399061
0.1441860465
0.1448598131
0.1435185185
0.1441860465
0.1428571429
0.1435185185
0.1422018349
0.1428571429
0.1415525114
0.1422018349
0.1409090909
0.1415525114
0.1402714932
0.1409090909
0.1396396396
0.1402714932
0.1390134529
0.1396396396
0.1383928571
0.1390134529
0.1377777778
0.1383928571
0.1371681416
0.1377777778
0.1365638767
0.1371681416
0.1359649123
0.1365638767
0.135371179
0.1359649123
0.1347826087
0.135371179
0.1341991342
0.1347826087
0.1336206897
0.1341991342
0.1330472103
0.1338912134
0.1324786325
0.1338912134
0.1319148936
0.1338912134
0.1313559322
0.1338912134
0.1308016878
0.1338912134
0.1302521008
0.1338912134
0.1338912134
0.1338912134
0.1333333333
0.1338912134
0.132780083
0.1333333333
0.132231405
0.132780083
0.1316872428
0.132231405
0.131147541
0.1316872428
0.1306122449
0.131147541
0.1300813008
0.1306122449
0.1295546559
0.1300813008
0.1290322581
0.1295546559
0.1285140562
0.1290322581
0.128
0.1285140562
0.1274900398
0.128
0.126984127
0.1274900398
0.1264822134
0.126984127
0.125984252
0.1264822134
0.1254901961
0.125984252
0.125
0.1254901961
0.1245136187
0.125
0.1240310078
0.1245136187
0.1235521236
0.1240310078
0.1230769231
0.1235521236
0.122605364
0.1230769231
0.1221374046
0.122605364
0.1216730038
0.1221374046
0.1212121212
0.1216730038
0.120754717
0.1212121212
0.1203007519
0.120754717
0.1198501873
0.1203007519
0.1194029851
0.1198501873
0.1189591078
0.1194029851
0.1185185185
0.1189591078
0.1180811808
0.118705036
0.1176470588
0.118705036
0.1172161172
0.118705036
0.1167883212
0.118705036
0.1163636364
0.118705036
0.115942029
0.118705036
0.1155234657
0.118705036
0.118705036
0.118705036
0.1182795699
0.118705036
0.1178571429
0.1182795699
0.1174377224
0.1178571429
0.1170212766
0.1174377224
0.1166077739
0.1170212766
0.1161971831
0.1166077739
0.1157894737
0.1161971831
0.1153846154
0.1157894737
0.1149825784
0.1153846154
0.1145833333
0.1149825784
0.1141868512
0.1145833333
0.1137931034
0.1141868512
0.1134020619
0.1137931034
0.1130136986
0.1134020619
0.1126279863
0.1130136986
0.112244898
0.1126279863
0.1118644068
0.112244898
0.1114864865
0.1118644068
0.1111111111
0.1114864865
0.110738255
0.1111111111
0.110367893
0.110738255
0.11
0.110367893
0.1096345515
0.11
0.1092715232
0.1096345515
0.1089108911
0.1092715232
0.1085526316
0.1089108911
0.1081967213
0.1085526316
0.1078431373
0.1081967213
0.1074918567
0.1078431373
0.1071428571
0.1074918567
0.1067961165
0.1071428571
0.1064516129
0.1067961165
0.1061093248
0.1064516129
0.1057692308
0.1061093248
0.1054313099
0.1057692308
0.1050955414
0.1055900621
0.1047619048
0.1055900621
0.1044303797
0.1055900621
0.1041009464
0.1055900621
0.1037735849
0.1055900621
0.1034482759
0.1055900621
0.103125
0.1055900621
0.1028037383
0.1055900621
0.1055900621
0.1055900621
0.1052631579
0.1055900621
0.1049382716
0.1052631579
0.1046153846
0.1049382716
0.1042944785
0.1046153846
0.1039755352
0.1042944785
0.1036585366
0.1039755352
0.103343465
0.1036585366
0.103030303
0.103343465
0.1027190332
0.1031518625
0.1024096386
0.1031518625
0.1021021021
0.1031518625
0.1017964072
0.1031518625
0.1014925373
0.1031518625
0.1011904762
0.1031518625
0.1008902077
0.1031518625
0.100591716
0.1031518625
0.1002949853
0.1031518625
0.1
0.1031518625
0.0997067449
0.1031518625
0.0994152047
0.1031518625
0.0991253644
0.1031518625
0.0988372093
0.1031518625
0.0985507246
0.1031518625
0.098265896
0.1031518625
0.1008645533
0.1031518625
0.1005747126
0.1031518625
0.1031518625
0.1031518625
0.1028571429
0.1031518625
0.1025641026
0.1028571429
0.1022727273
0.1025641026
0.1019830028
0.1022727273
0.1016949153
0.1019830028
0.1014084507
0.1016949153
0.1011235955
0.1014084507
0.1008403361
0.101369863
0.1005586592
0.101369863
0.1002785515
0.101369863
0.1
0.101369863
0.0997229917
0.101369863
0.0994475138
0.101369863
0.0991735537
0.101369863
0.0989010989
0.101369863
0.101369863
0.101369863
0.1010928962
0.101369863
0.1008174387
0.1010928962
0.1005434783
0.1008174387
0.1002710027
0.1005434783
0.1
0.1002710027
0.0997304582
0.1
0.0994623656
0.0997304582
0.0991957105
0.0994623656
0.0989304813
0.0991957105
0.0986666667
0.0989304813
0.0984042553
0.0986666667
0.0981432361
0.0984042553
0.0978835979
0.0981432361
0.0976253298
0.0978835979
0.0973684211
0.0976253298
0.0971128609
0.0973684211
0.0968586387
0.0971128609
0.0966057441
0.0968586387
0.0963541667
0.0966057441
0.0961038961
0.0963541667
0.0958549223
0.0961038961
0.0956072351
0.0958549223
0.0953608247
0.0956072351
0.0951156812
0.0953608247
0.0948717949
0.0951156812
0.094629156
0.0948717949
0.0943877551
0.094629156
0.0941475827
0.0943877551
0.0939086294
0.0941475827
0.0936708861
0.0939086294
0.0934343434
0.0936708861
0.0931989924
0.0934343434
0.0929648241
0.0931989924
0.0927318296
0.0929648241
0.0925
0.0927318296
0.0922693267
0.0925
0.092039801
0.0922693267
0.0918114144
0.092039801
0.0915841584
0.0918114144
0.0913580247
0.0915841584
0.0911330049
0.0913580247
0.0909090909
0.0911330049
0.0906862745
0.0909090909
0.0904645477
0.0906862745
0.0902439024
0.0904645477
0.0900243309
0.0902439024
0.0898058252
0.0900243309
0.0895883777
0.0898058252
0.0893719807
0.0896226415
0.0891566265
0.0896226415
0.0889423077
0.0896226415
0.0887290168
0.0896226415
0.0885167464
0.0896226415
0.0883054893
0.0896226415
0.0880952381
0.0896226415
0.0878859857
0.0896226415
0.0876777251
0.0896226415
0.0874704492
0.0896226415
0.0896226415
0.0896226415
0.0894117647
0.0896226415
0.0892018779
0.0894117647
0.0889929742
0.0892018779
0.0887850467
0.0889929742
0.0885780886
0.0887850467
0.088372093
0.0885780886
0.0881670534
0.088372093
0.087962963
0.0881670534
0.0877598152
0.087962963
0.0875576037
0.0877598152
0.0873563218
0.0875576037
0.0871559633
0.0873563218
0.0869565217
0.0871559633
0.0867579909
0.0869565217
0.0865603645
0.0867579909
0.0863636364
0.0865603645
0.0861678005
0.0863636364
0.0859728507
0.0861678005
0.085778781
0.0859728507
0.0855855856
0.085778781
0.0853932584
0.0855855856
0.0852017937
0.0853932584
0.0850111857
0.0852017937
0.0848214286
0.0850111857
0.0846325167
0.0848214286
0.0844444444
0.0846325167
0.0842572062
0.0845986985
0.0840707965
0.0845986985
0.0838852097
0.0845986985
0.0837004405
0.0845986985
0.0835164835
0.0845986985
0.0833333333
0.0845986985
0.0831509847
0.0845986985
0.0829694323
0.0845986985
0.082788671
0.0845986985
0.0826086957
0.0845986985
0.0845986985
0.0845986985
0.0844155844
0.0845986985
0.0842332613
0.0844155844
0.0840517241
0.0842332613
0.0838709677
0.0840517241
0.0836909871
0.0838709677
0.0835117773
0.0836909871
0.0833333333
0.0835117773
0.0831556503
0.0833333333
0.0829787234
0.0831556503
0.0828025478
0.0829787234
0.0826271186
0.0828025478
0.0824524313
0.0826271186
0.082278481
0.0824524313
0.0821052632
0.082278481
0.0819327731
0.0821355236
0.0817610063
0.0821355236
0.0815899582
0.0821355236
0.0814196242
0.0821355236
0.08125
0.0821355236
0.0810810811
0.0821355236
0.0809128631
0.0821355236
0.0807453416
0.0821355236
0.0805785124
0.0821355236
0.0804123711
0.0821355236
0.0802469136
0.0821355236
0.0821355236
0.0821355236
0.0819672131
0.0821355236
0.081799591
0.0819672131
0.0816326531
0.081799591
0.0814663951
0.0816326531
0.081300813
0.0814663951
0.0811359026
0.081300813
0.0809716599
0.0811359026
0.0808080808
0.0809716599
0.0806451613
0.0808080808
0.0804828974
0.0806451613
0.0803212851
0.0804828974
0.0801603206
0.0803212851
0.08
0.0801603206
0.08
Sheet1
RelevantPrecisonRecallF1Fb=0.33Fb=3interp precSpecificity1-SpecRecInterp prec
001.0001.00000.01.00
111.001.00000.02500.04880.20410.02771.0000100.10.67
201.000.50000.02500.04760.17240.02760.75000.9979959920.0020040080.20.63
312.000.66670.05000.09300.29850.05510.75000.99799196790.00200803210.30.55
413.000.75000.07500.13640.39470.08240.75000.99798792760.00201207240.40.45
503.000.60000.07500.13330.35290.08220.66670.99597585510.00402414490.50.41
614.000.66670.10000.17390.42550.10930.66670.99596774190.00403225810.60.36
704.000.57140.10000.17020.38830.10900.62500.99395161290.00604838710.70.29
804.000.50000.10000.16670.35710.10870.62500.99193548390.00806451610.80.13
904.000.44440.10000.16330.33060.10840.62500.98991935480.01008064520.90.10
1015.000.50000.12500.20000.38460.13510.62500.98989898990.01010101011.00.08
1116.000.54550.15000.23530.43170.16170.62500.98987854250.0101214575
1206.000.50000.15000.23080.40540.16130.62500.9878542510.012145749
1317.000.53850.17500.26420.44590.18770.62500.98782961460.0121703854
1418.000.57140.20000.29630.48190.21390.62500.9878048780.012195122
1519.000.60000.22500.32730.51430.24000.62500.98778004070.0122199593
16110.000.62500.25000.35710.54350.26600.62500.9877551020.012244898
17010.000.58820.25000.35090.51810.26530.58820.98571428570.0142857143
18010.000.55560.25000.34480.49500.26460.57890.98367346940.0163265306
19111.000.57890.27500.37290.52130.29020.57890.98364008180.0163599182
20011.000.55000.27500.36670.50000.28950.55000.9815950920.018404908
21011.000.52380.27500.36070.48030.28870.54550.97955010220.0204498978
22112.000.54550.30000.38710.50420.31410.54550.97950819670.0204918033
23012.000.52170.30000.38100.48580.31330.52170.97745901640.0225409836
24012.000.50000.30000.37500.46880.31250.52000.97540983610.0245901639
25113.000.52000.32500.40000.49060.33770.52000.97535934290.0246406571
26013.000.50000.32500.39390.47450.33680.50000.97330595480.0266940452
27013.000.48150.32500.38810.45940.33590.48150.97125256670.0287474333
28013.000.46430.32500.38240.44520.33510.46670.96919917860.0308008214
29013.000.44830.32500.37680.43190.33420.46670.96714579060.0328542094
30114.000.46670.35000.40000.45160.35900.46670.96707818930.0329218107
31014.000.45160.35000.39440.43890.35810.45450.96502057610.0349794239
32014.000.43750.35000.38890.42680.35710.45450.9629629630.037037037
33115.000.45450.37500.41100.44510.38170.45450.96288659790.0371134021
34015.000.44120.37500.40540.43350.38070.44740.96082474230.0391752577
35015.000.42860.37500.40000.42250.37970.44740.95876288660.0412371134
36015.000.41670.37500.39470.41210.37880.44740.95670103090.0432989691
37116.000.43240.40000.41560.42900.40300.44740.95661157020.0433884298
38117.000.44740.42500.43590.44500.42710.44740.95652173910.0434782609
39017.000.43590.42500.43040.43480.42610.43590.95445134580.0455486542
40017.000.42500.42500.42500.42500.42500.42860.95238095240.0476190476
41017.000.41460.42500.41980.41560.42390.42860.9503105590.049689441
42118.000.42860.45000.43900.43060.44780.42860.95020746890.0497925311
43018.000.41860.45000.43370.42150.44670.41860.94813278010.0518672199
44018.000.40910.45000.42860.41280.44550.41300.94605809130.0539419087
45018.000.40000.45000.42350.40450.44440.41300.94398340250.0560165975
46119.000.41300.47500.44190.41850.46800.41300.94386694390.0561330561
47019.000.40430.47500.43680.41040.46680.40820.94178794180.0582120582
48019.000.39580.47500.43180.40250.46570.40820.93970893970.0602910603
49120.000.40820.50000.44940.41580.48900.40820.93958333330.0604166667
50020.000.40000.50000.44440.40820.48780.40000.93750.0625
51020.000.39220.50000.43960.40080.48660.39220.93541666670.0645833333
52020.000.38460.50000.43480.39370.48540.38890.93333333330.0666666667
53020.000.37740.50000.43010.38680.48430.38890.931250.06875
54121.000.38890.52500.44680.39920.50720.38890.93110647180.0688935282
55021.000.38180.52500.44210.39250.50600.38180.92901878910.0709812109
56021.000.37500.52500.43750.38600.50480.37930.92693110650.0730688935
57021.000.36840.52500.43300.37970.50360.37930.92484342380.0751565762
58122.000.37930.55000.44900.39150.52630.37930.92468619250.0753138075
59022.000.37290.55000.44440.38530.52510.37290.92259414230.0774058577
60022.000.36670.55000.44000.37930.52380.37100.92050209210.0794979079
61022.000.36070.55000.43560.37350.52260.37100.91841004180.0815899582
62123.000.37100.57500.45100.38460.54500.37100.91823899370.0817610063
63023.000.36510.57500.44660.37890.54370.36510.91614255770.0838574423
64023.000.35940.57500.44230.37340.54250.35940.91404612160.0859538784
65023.000.35380.57500.43810.36800.54120.35820.91194968550.0880503145
66023.000.34850.57500.43400.36280.53990.35820.90985324950.0901467505
67124.000.35820.60000.44860.37330.56210.35820.90966386550.0903361345
68024.000.35290.60000.44440.36810.56070.35290.90756302520.0924369748
69024.000.34780.60000.44040.36310.55940.34780.90546218490.0945378151
70024.000.34290.60000.43640.35820.55810.34290.90336134450.0966386555
71024.000.33800.60000.43240.35350.55680.33800.90126050420.0987394958
72024.000.33330.60000.42860.34880.55560.33330.89915966390.1008403361
73024.000.32880.60000.42480.34430.55430.33330.89705882350.1029411765Example 11pt precision (SabIR/Cornell 8A1) from TREC 8 (1999)
74024.000.32430.60000.42110.33990.55300.33330.89495798320.1050420168Recall LevelAve. Precision
75125.000.33330.62500.43480.34970.57470.33330.89473684210.105263157900.736
76025.000.32890.62500.43100.34530.57340.32890.89263157890.10736842110.10.5107
77025.000.32470.62500.42740.34110.57210.32470.89052631580.10947368420.20.4059
78025.000.32050.62500.42370.33690.57080.32050.88842105260.11157894740.30.3424
79025.000.31650.62500.42020.33290.56950.31650.88631578950.11368421050.40.2931
80025.000.31250.62500.41670.32890.56820.31400.88421052630.11578947370.50.2457
81025.000.30860.62500.41320.32510.56690.31400.88210526320.11789473680.60.1873
82025.000.30490.62500.40980.32130.56560.31400.880.120.70.1391
83126.000.31330.65000.42280.33040.58690.31400.87974683540.12025316460.80.0881
84026.000.30950.65000.41940.32660.58560.31400.87763713080.12236286920.90.0545
85026.000.30590.65000.41600.32300.58430.31400.87552742620.124472573810.0197
86127.000.31400.67500.42860.33170.60540.31400.87526427060.1247357294Average precision:0.2553
87027.000.31030.67500.42520.32810.60400.31030.87315010570.1268498943
88027.000.30680.67500.42190.32450.60270.30680.87103594080.1289640592
89027.000.30340.67500.41860.32100.60130.30340.86892177590.1310782241
90027.000.30000.67500.41540.31760.60000.30000.8668076110.133192389
91027.000.29670.67500.41220.31430.59870.29670.86469344610.1353065539
92027.000.29350.67500.40910.31110.59730.29350.86257928120.1374207188
93027.000.29030.67500.40600.30790.59600.29030.86046511630.1395348837
94027.000.28720.67500.40300.30470.59470.28720.85835095140.1416490486
95027.000.28420.67500.40000.30170.59340.28570.85623678650.1437632135
96027.000.28130.67500.39710.29870.59210.28570.85412262160.1458773784
97027.000.27840.67500.39420.29570.59080.28570.85200845670.1479915433
98128.000.28570.70000.40580.30370.61140.28570.85169491530.1483050847
99028.000.28280.70000.40290.30080.61000.28280.84957627120.1504237288
100028.000.28000.70000.40000.29790.60870.28000.84745762710.1525423729
101028.000.27720.70000.39720.29500.60740.27720.84533898310.1546610169
102028.000.27450.70000.39440.29230.60610.27450.8432203390.156779661
103028.000.27180.70000.39160.28960.60480.27180.84110169490.1588983051
104028.000.26920.70000.38890.28690.60340.26920.83898305080.1610169492
105028.000.26670.70000.38620.28430.60220.26670.83686440680.1631355932
106028.000.26420.70000.38360.28170.60090.26420.83474576270.1652542373
107028.000.26170.70000.38100.27920.59960.26360.83262711860.1673728814
108028.000.25930.70000.37840.27670.59830.26360.83050847460.1694915254
109028.000.25690.70000.37580.27420.59700.26360.82838983050.1716101695
110129.000.26360.72500.38670.28160.61700.26360.82802547770.1719745223
111029.000.26130.72500.38410.27910.61570.26130.82590233550.1740976645
112029.000.25890.72500.38160.27670.61440.25890.82377919320.1762208068
113029.000.25660.72500.37910.27440.61310.25660.8216560510.178343949
114029.000.25440.72500.37660.27200.61180.25440.81953290870.1804670913
115029.000.25220.72500.37420.26980.61050.25220.81740976650.1825902335
116029.000.25000.72500.37180.26750.60920.25000.81528662420.1847133758
117029.000.24790.72500.36940.26530.60800.24790.8131634820.186836518
118029.000.24580.72500.36710.26320.60670.24580.81104033970.1889596603
119029.000.24370.72500.36480.26100.60540.24370.80891719750.1910828025
120029.000.24170.72500.36250.25890.60420.24170.80679405520.1932059448
121029.000.23970.72500.36020.25690.60290.23970.8046709130.195329087
122029.000.23770.72500.35800.25480.60170.23770.80254777070.1974522293
123029.000.23580.72500.35580.25280.60040.23580.80042462850.1995753715
124029.000.23390.72500.35370.25090.59920.23440.79830148620.2016985138
125029.000.23200.72500.35150.24890.59790.23440.79617834390.2038216561
126029.000.23020.72500.34940.24700.59670.23440.79405520170.2059447983
127029.000.22830.72500.34730.24510.59550.23440.79193205940.2080679406
128130.000.23440.75000.35710.25170.61480.23440.79148936170.2085106383
129030.000.23260.75000.35500.24980.61350.23260.78936170210.2106382979
130030.000.23080.75000.35290.24790.61220.23080.78723404260.2127659574
131030.000.22900.75000.35090.24610.61100.22900.7851063830.214893617
132030.000.22730.75000.34880.24430.60980.22730.78297872340.2170212766
133030.000.22560.75000.34680.24250.60850.22560.78085106380.2191489362
134030.000.22390.75000.34480.24080.60730.22390.77872340430.2212765957
135030.000.22220.75000.34290.23900.60610.22220.77659574470.2234042553
136030.000.22060.75000.34090.23730.60480.22060.77446808510.2255319149
137030.000.21900.75000.33900.23570.60360.21900.77234042550.2276595745
138030.000.21740.75000.33710.23400.60240.21740.7702127660.229787234
139030.000.21580.75000.33520.23240.60120.21580.76808510640.2319148936
140030.000.21430.75000.33330.23080.60000.21430.76595744680.2340425532
141030.000.21280.75000.33150.22920.59880.21280.76382978720.2361702128
142030.000.21130.75000.32970.22760.59760.21130.76170212770.2382978723
143030.000.20980.75000.32790.22610.59640.20980.75957446810.2404255319
144030.000.20830.75000.32610.22460.59520.20830.75744680850.2425531915
145030.000.20690.75000.32430.22300.59410.20690.75531914890.2446808511
146030.000.20550.75000.32260.22160.59290.20550.75319148940.2468085106
147030.000.20410.75000.32090.22010.59170.20410.75106382980.2489361702
148030.000.20270.75000.31910.21870.59060.20270.74893617020.2510638298
149030.000.20130.75000.31750.21720.58940.20130.74680851060.2531914894
150030.000.20000.75000.31580.21580.58820.20000.74468085110.2553191489
151030.000.19870.75000.31410.21440.58710.19870.74255319150.2574468085
152030.000.19740.75000.31250.21310.58590.19740.74042553190.2595744681
153030.000.19610.75000.31090.21170.58480.19610.73829787230.2617021277
154030.000.19480.75000.30930.21040.58370.19480.73617021280.2638297872
155030.000.19350.75000.30770.20910.58250.19350.73404255320.2659574468
156030.000.19230.75000.30610.20780.58140.19230.73191489360.2680851064
157030.000.19110.75000.30460.20650.58030.19110.7297872340.270212766
158030.000.18990.75000.30300.20520.57920.18990.72765957450.2723404255
159030.000.18870.75000.30150.20390.57800.18870.72553191490.2744680851
160030.000.18750.75000.30000.20270.57690.18750.72340425530.2765957447
161030.000.18630.75000.29850.20150.57580.18630.72127659570.2787234043
162030.000.18520.75000.29700.20030.57470.18520.71914893620.2808510638
163030.000.18400.75000.29560.19910.57360.18400.71702127660.2829787234
164030.000.18290.75000.29410.19790.57250.18290.7148936170.285106383
165030.000.18180.75000.29270.19670.57140.18180.71276595740.2872340426
166030.000.18070.75000.29130.19560.57030.18070.71063829790.2893617021
167030.000.17960.75000.28990.19440.56930.18020.70851063830.2914893617
168030.000.17860.75000.28850.19330.56820.18020.70638297870.2936170213
169030.000.17750.75000.28710.19220.56710.18020.70425531910.2957446809
170030.000.17650.75000.28570.19110.56600.18020.70212765960.2978723404
171030.000.17540.75000.28440.19000.56500.18020.70.3
172131.000.18020.77500.29250.19520.58270.18020.69936034120.3006396588
173031.000.17920.77500.29110.19410.58160.17920.6972281450.302771855
174031.000.17820.77500.28970.19300.58050.17820.69509594880.3049040512
175031.000.17710.77500.28840.19200.57940.17710.69296375270.3070362473
176031.000.17610.77500.28700.19090.57840.17610.69083155650.3091684435
177031.000.17510.77500.28570.18980.57730.17510.68869936030.3113006397
178031.000.17420.77500.28440.18880.57620.17420.68656716420.3134328358
179031.000.17320.77500.28310.18780.57510.17320.6844349680.315565032
180031.000.17220.77500.28180.18670.57410.17220.68230277190.3176972281
181031.000.17130.77500.28050.18570.57300.17130.68017057570.3198294243
182031.000.17030.77500.27930.18470.57200.17030.67803837950.3219616205
183031.000.16940.77500.27800.18380.57090.16940.67590618340.3240938166
184031.000.16850.77500.27680.18280.56990.16850.67377398720.3262260128
185031.000.16760.77500.27560.18180.56880.16760.6716417910.328358209
186031.000.16670.77500.27430.18090.56780.16670.66950959490.3304904051
187031.000.16580.77500.27310.17990.56670.16580.66737739870.3326226013
188031.000.16490.77500.27190.17900.56570.16490.66524520260.3347547974
189031.000.16400.77500.27070.17810.56470.16400.66311300640.3368869936
190031.000.16320.77500.26960.17710.56360.16320.66098081020.3390191898
191031.000.16230.77500.26840.17620.56260.16230.65884861410.3411513859
192031.000.16150.77500.26720.17530.56160.16150.65671641790.3432835821
193031.000.16060.77500.26610.17450.56060.16060.65458422170.3454157783
194031.000.15980.77500.26500.17360.55960.15980.65245202560.3475479744
195031.000.15900.77500.26380.17270.55860.15900.65031982940.3496801706
196031.000.15820.77500.26270.17180.55760.15820.64818763330.3518123667
197031.000.15740.77500.26160.17100.55660.15740.64605543710.3539445629
198031.000.15660.77500.26050.17010.55560.15660.64392324090.3560767591
199031.000.15580.77500.25940.16930.55460.15580.64179104480.3582089552
200031.000.15500.77500.25830.16850.55360.15500.63965884860.3603411514
201031.000.15420.77500.25730.16770.55260.15420.63752665250.3624733475
202031.000.15350.77500.25620.16680.55160.15350.63539445630.3646055437
203031.000.15270.77500.25510.16600.55060.15270.63326226010.3667377399
204031.000.15200.77500.25410.16520.54960.15200.6311300640.368869936
205031.000.15120.77500.25310.16450.54870.15120.62899786780.3710021322
206031.000.15050.77500.25200.16370.54770.15050.62686567160.3731343284
207031.000.14980.77500.25100.16290.54670.14980.62473347550.3752665245
208031.000.14900.77500.25000.16210.54580.14900.62260127930.3773987207
209031.000.14830.77500.24900.16140.54480.14830.62046908320.3795309168
210031.000.14760.77500.24800.16060.54390.14760.6183368870.381663113
211031.000.14690.77500.24700.15990.54290.14690.61620469080.3837953092
212031.000.14620.77500.24600.15910.54200.14620.61407249470.3859275053
213031.000.14550.77500.24510.15840.54100.14550.61194029850.3880597015
214031.000.14490.77500.24410.15770.54010.14490.60980810230.3901918977
215031.000.14420.77500.24310.15700.53910.14420.60767590620.3923240938
216031.000.14350.77500.24220.15630.53820.14350.605543710.39445629
217031.000.14290.77500.24120.15550.53730.14290.60341151390.3965884861
218031.000.14220.77500.24030.15480.53630.14220.60127931770.3987206823
219031.000.14160.77500.23940.15420.53540.14160.59914712150.4008528785
220031.000.14090.77500.23850.15350.53450.14090.59701492540.4029850746
221031.000.14030.77500.23750.15280.53360.14030.59488272920.4051172708
222031.000.13960.77500.23660.15210.53260.13960.5927505330.407249467
223031.000.13900.77500.23570.15140.53170.13900.59061833690.4093816631
224031.000.13840.77500.23480.15080.53080.13840.58848614070.4115138593
225031.000.13780.77500.23400.15010.52990.13780.58635394460.4136460554
226031.000.13720.77500.23310.14950.52900.13720.58422174840.4157782516
227031.000.13660.77500.23220.14880.52810.13660.58208955220.4179104478
228031.000.13600.77500.23130.14820.52720.13600.57995735610.4200426439
229031.000.13540.77500.23050.14750.52630.13540.57782515990.4221748401
230031.000.13480.77500.22960.14690.52540.13480.57569296380.4243070362
231031.000.13420.77500.22880.14630.52450.13420.57356076760.4264392324
232031.000.13360.77500.22790.14570.52360.13390.57142857140.4285714286
233031.000.13300.77500.22710.14510.52280.13390.56929637530.4307036247
234031.000.13250.77500.22630.14450.52190.13390.56716417910.4328358209
235031.000.13190.77500.22550.14390.52100.13390.56503198290.4349680171
236031.000.13140.77500.22460.14330.52010.13390.56289978680.4371002132
237031.000.13080.77500.22380.14270.51930.13390.56076759060.4392324094
238031.000.13030.77500.22300.14210.51840.13390.55863539450.4413646055
239132.000.13390.80000.22940.14610.53420.13390.55769230770.4423076923
240032.000.13330.80000.22860.14550.53330.13330.55555555560.4444444444
241032.000.13280.80000.22780.14490.53240.13280.55341880340.4465811966
242032.000.13220.80000.22700.14430.53160.13220.55128205130.4487179487
243032.000.13170.80000.22610.14370.53070.13170.54914529910.4508547009
244032.000.13110.80000.22540.14310.52980.13110.5470085470.452991453
245032.000.13060.80000.22460.14250.52890.13060.54487179490.4551282051
246032.000.13010.80000.22380.14200.52810.13010.54273504270.4572649573
247032.000.12960.80000.22300.14140.52720.12960.54059829060.4594017094
248032.000.12900.80000.22220.14080.52630.12900.53846153850.4615384615
249032.000.12850.80000.22150.14030.52550.12850.53632478630.4636752137
250032.000.12800.80000.22070.13970.52460.12800.53418803420.4658119658
251032.000.12750.80000.21990.13920.52370.12750.53205128210.4679487179
252032.000.12700.80000.21920.13860.52290.12700.52991452990.4700854701
253032.000.12650.80000.21840.13810.52200.12650.52777777780.4722222222
254032.000.12600.80000.21770.13760.52120.12600.52564102560.4743589744
255032.000.12550.80000.21690.13700.52030.12550.52350427350.4764957265
256032.000.12500.80000.21620.13650.51950.12500.52136752140.4786324786
257032.000.12450.80000.21550.13600.51860.12450.51923076920.4807692308
258032.000.12400.80000.21480.13550.51780.12400.51709401710.4829059829
259032.000.12360.80000.21400.13500.51700.12360.5149572650.485042735
260032.000.12310.80000.21330.13450.51610.12310.51282051280.4871794872
261032.000.12260.80000.21260.13390.51530.12260.51068376070.4893162393
262032.000.12210.80000.21190.13340.51450.12210.50854700850.4914529915
263032.000.12170.80000.21120.13290.51360.12170.50641025640.4935897436
264032.000.12120.80000.21050.13250.51280.12120.50427350430.4957264957
265032.000.12080.80000.20980.13200.51200.12080.50213675210.4978632479
266032.000.12030.80000.20920.13150.51120.12030.50.5
267032.000.11990.80000.20850.13100.51040.11990.49786324790.5021367521
268032.000.11940.80000.20780.13050.50960.11940.49572649570.5042735043
269032.000.11900.80000.20710.13000.50870.11900.49358974360.5064102564
270032.000.11850.80000.20650.12960.50790.11870.49145299150.5085470085
271032.000.11810.80000.20580.12910.50710.11870.48931623930.5106837607
272032.000.11760.80000.20510.12860.50630.11870.48717948720.5128205128
273032.000.11720.80000.20450.12820.50550.11870.4850427350.514957265
274032.000.11680.80000.20380.12770.50470.11870.48290598290.5170940171
275032.000.11640.80000.20320.12720.50390.11870.48076923080.5192307692
276032.000.11590.80000.20250.12680.50310.11870.47863247860.5213675214
277032.000.11550.80000.20190.12630.50240.11870.47649572650.5235042735
278133.000.11870.82500.20750.12980.51720.11870.47537473230.5246252677
279033.000.11830.82500.20690.12940.51640.11830.47323340470.5267665953
280033.000.11790.82500.20630.12890.51560.11790.47109207710.5289079229
281033.000.11740.82500.20560.12850.51480.11740.46895074950.5310492505
282033.000.11700.82500.20500.12800.51400.11700.46680942180.5331905782
283033.000.11660.82500.20430.12760.51320.11660.46466809420.5353319058
284033.000.11620.82500.20370.12710.51240.11620.46252676660.5374732334
285033.000.11580.82500.20310.12670.51160.11580.4603854390.539614561
286033.000.11540.82500.20250.12620.51080.11540.45824411130.5417558887
287033.000.11500.82500.20180.12580.51000.11500.45610278370.5438972163
288033.000.11460.82500.20120.12540.50930.11460.45396145610.5460385439
289033.000.11420.82500.20060.12500.50850.11420.45182012850.5481798715
290033.000.11380.82500.20000.12450.50770.11380.44967880090.5503211991
291033.000.11340.82500.19940.12410.50690.11340.44753747320.5524625268
292033.000.11300.82500.19880.12370.50610.11300.44539614560.5546038544
293033.000.11260.82500.19820.12330.50540.11260.4432548180.556745182
294033.000.11220.82500.19760.12290.50460.11220.44111349040.5588865096
295033.000.11190.82500.19700.12240.50380.11190.43897216270.5610278373
296033.000.11150.82500.19640.12200.50300.11150.43683083510.5631691649
297033.000.11110.82500.19580.12160.50230.11110.43468950750.5653104925
298033.000.11070.82500.19530.12120.50150.11070.43254817990.5674518201
299033.000.11040.82500.19470.12080.50080.11040.43040685220.5695931478
300033.000.11000.82500.19410.12040.50000.11000.42826552460.5717344754
301033.000.10960.82500.19350.12000.49920.10960.4261241970.573875803
302033.000.10930.82500.19300.11970.49850.10930.42398286940.5760171306
303033.000.10890.82500.19240.11930.49770.10890.42184154180.5781584582
304033.000.10860.82500.19190.11890.49700.10860.41970021410.5802997859
305033.000.10820.82500.19130.11850.49620.10820.41755888650.5824411135
306033.000.10780.82500.19080.11810.49550.10780.41541755890.5845824411
307033.000.10750.82500.19020.11770.49480.10750.41327623130.5867237687
308033.000.10710.82500.18970.11740.49400.10710.41113490360.5888650964
309033.000.10680.82500.18910.11700.49330.10680.4089935760.591006424
310033.000.10650.82500.18860.11660.49250.10650.40685224840.5931477516
311033.000.10610.82500.18800.11620.49180.10610.40471092080.5952890792
312033.000.10580.82500.18750.11590.49110.10580.40256959310.5974304069
313033.000.10540.82500.18700.11550.49030.10560.40042826550.5995717345
314033.000.10510.82500.18640.11510.48960.10560.39828693790.6017130621
315033.000.10480.82500.18590.11480.48890.10560.39614561030.6038543897
316033.000.10440.82500.18540.11440.48820.10560.39400428270.6059957173
317033.000.10410.82500.18490.11410.48740.10560.3918629550.608137045
318033.000.10380.82500.18440.11370.48670.10560.38972162740.6102783726
319033.000.10340.82500.18380.11340.48600.10560.38758029980.6124197002
320033.000.10310.82500.18330.11300.48530.10560.38543897220.6145610278
321033.000.10280.82500.18280.11270.48460.10560.38329764450.6167023555
322134.000.10560.85000.18780.11570.49850.10560.38197424890.6180257511
323034.000.10530.85000.18730.11540.49780.10530.37982832620.6201716738
324034.000.10490.85000.18680.11500.49710.10490.37768240340.6223175966
325034.000.10460.85000.18630.11470.49640.10460.37553648070.6244635193
326034.000.10430.85000.18580.11430.49560.10430.37339055790.6266094421
327034.000.10400.85000.18530.11400.49490.10400.37124463520.6287553648
328034.000.10370.85000.18480.11360.49420.10370.36909871240.6309012876
329034.000.10330.85000.18430.11330.49350.10330.36695278970.6330472103
330034.000.10300.85000.18380.11300.49280.10320.3648068670.635193133
331034.000.10270.85000.18330.11260.49200.10320.36266094420.6373390558
332034.000.10240.85000.18280.11230.49130.10320.36051502150.6394849785
333034.000.10210.85000.18230.11200.49060.10320.35836909870.6416309013
334034.000.10180.85000.18180.11160.48990.10320.3562231760.643776824
335034.000.10150.85000.18130.11130.48920.10320.35407725320.6459227468
336034.000.10120.85000.18090.11100.48850.10320.35193133050.6480686695
337034.000.10090.85000.18040.11060.48780.10320.34978540770.6502145923
338034.000.10060.85000.17990.11030.48710.10320.3476394850.652360515
339034.000.10030.85000.17940.11000.48640.10320.34549356220.6545064378
340034.000.10000.85000.17890.10970.48570.10320.34334763950.6566523605
341034.000.09970.85000.17850.10940.48500.10320.34120171670.6587982833
342034.000.09940.85000.17800.10900.48430.10320.3390557940.660944206
343034.000.09910.85000.17750.10870.48360.10320.33690987120.6630901288
344034.000.09880.85000.17710.10840.48300.10320.33476394850.6652360515
345034.000.09860.85000.17660.10810.48230.10320.33261802580.6673819742
346034.000.09830.85000.17620.10780.48160.10320.3304721030.669527897
347135.000.10090.87500.18090.11070.49500.10320.32903225810.6709677419
348035.000.10060.87500.18040.11030.49440.10320.32688172040.6731182796
349136.000.10320.90000.18510.11320.50780.10320.32543103450.6745689655
350036.000.10290.90000.18460.11290.50700.10290.32327586210.6767241379
351036.000.10260.90000.18410.11250.50630.10260.32112068970.6788793103
352036.000.10230.90000.18370.11220.50560.10230.31896551720.6810344828
353036.000.10200.90000.18320.11190.50490.10200.31681034480.6831896552
354036.000.10170.90000.18270.11160.50420.10170.31465517240.6853448276
355036.000.10140.90000.18230.11130.50350.10140.31250.6875
356036.000.10110.90000.18180.11100.50280.10140.31034482760.6896551724
357036.000.10080.90000.18140.11070.50210.10140.30818965520.6918103448
358036.000.10060.90000.18090.11040.50140.10140.30603448280.6939655172
359036.000.10030.90000.18050.11010.50070.10140.30387931030.6961206897
360036.000.10000.90000.18000.10980.50000.10140.30172413790.6982758621
361036.000.09970.90000.17960.10950.49930.10140.29956896550.7004310345
362036.000.09940.90000.17910.10920.49860.10140.29741379310.7025862069
363036.000.09920.90000.17870.10890.49790.10140.29525862070.7047413793
364036.000.09890.90000.17820.10860.49720.10140.29310344830.7068965517
365137.000.10140.92500.18270.11130.51030.10140.29157667390.7084233261
366037.000.10110.92500.18230.11100.50960.10110.28941684670.7105831533
367037.000.10080.92500.18180.11070.50890.10080.28725701940.7127429806
368037.000.10050.92500.18140.11040.50820.10050.28509719220.7149028078
369037.000.10030.92500.18090.11010.50750.10030.2829373650.717062635
370037.000.10000.92500.18050.10980.50680.10000.28077753780.7192224622
371037.000.09970.92500.18000.10950.50620.09970.27861771060.7213822894
372037.000.09950.92500.17960.10920.50550.09950.27645788340.7235421166
373037.000.09920.92500.17920.10890.50480.09920.27429805620.7257019438
374037.000.09890.92500.17870.10860.50410.09890.27213822890.7278617711
375037.000.09870.92500.17830.10830.50340.09870.26997840170.7300215983
376037.000.09840.92500.17790.10810.50270.09840.26781857450.7321814255
377037.000.09810.92500.17750.10780.50200.09810.26565874730.7343412527
378037.000.09790.92500.17700.10750.50140.09790.26349892010.7365010799
379037.000.09760.92500.17660.10720.50070.09760.26133909290.7386609071
380037.000.09740.92500.17620.10690.50000.09740.25917926570.7408207343
381037.000.09710.92500.17580.10670.49930.09710.25701943840.7429805616
382037.000.09690.92500.17540.10640.49870.09690.25485961120.7451403888
383037.000.09660.92500.17490.10610.49800.09660.2526997840.747300216
384037.000.09640.92500.17450.10580.49730.09640.25053995680.7494600432
385037.000.09610.92500.17410.10560.49660.09610.24838012960.7516198704
386037.000.09590.92500.17370.10530.49600.09590.24622030240.7537796976
387037.000.09560.92500.17330.10500.49530.09560.24406047520.7559395248
388037.000.09540.92500.17290.10480.49470.09540.24190064790.7580993521
389037.000.09510.92500.17250.10450.49400.09510.23974082070.7602591793
390037.000.09490.92500.17210.10420.49330.09490.23758099350.7624190065
391037.000.09460.92500.17170.10400.49270.09460.23542116630.7645788337
392037.000.09440.92500.17130.10370.49200.09440.23326133910.7667386609
393037.000.09410.92500.17090.10340.49140.09410.23110151190.7688984881
394037.000.09390.92500.17050.10320.49070.09390.22894168470.7710583153
395037.000.09370.92500.17010.10290.49010.09370.22678185750.7732181425
396037.000.09340.92500.16970.10270.48940.09340.22462203020.7753779698
397037.000.09320.92500.16930.10240.48880.09320.2224622030.777537797
398037.000.09300.92500.16890.10220.48810.09300.22030237580.7796976242
399037.000.09270.92500.16860.10190.48750.09270.21814254860.7818574514
400037.000.09250.92500.16820.10160.48680.09250.21598272140.7840172786
401037.000.09230.92500.16780.10140.48620.09230.21382289420.7861771058
402037.000.09200.92500.16740.10110.48560.09200.2116630670.788336933
403037.000.09180.92500.16700.10090.48490.09180.20950323970.7904967603
404037.000.09160.92500.16670.10070.48430.09160.20734341250.7926565875
405037.000.09140.92500.16630.10040.48370.09140.20518358530.7948164147
406037.000.09110.92500.16590.10020.48300.09110.20302375810.7969762419
407037.000.09090.92500.16550.09990.48240.09090.20086393090.7991360691
408037.000.09070.92500.16520.09970.48180.09070.19870410370.8012958963
409037.000.09050.92500.16480.09940.48110.09050.19654427650.8034557235
410037.000.09020.92500.16440.09920.48050.09020.19438444920.8056155508
411037.000.09000.92500.16410.09900.47990.09000.1922246220.807775378
412037.000.08980.92500.16370.09870.47930.08980.19006479480.8099352052
413037.000.08960.92500.16340.09850.47870.08960.18790496760.8120950324
414037.000.08940.92500.16300.09820.47800.08960.18574514040.8142548596
415037.000.08920.92500.16260.09800.47740.08960.18358531320.8164146868
416037.000.08890.92500.16230.09780.47680.08960.1814254860.818574514
417037.000.08870.92500.16190.09750.47620.08960.17926565870.8207343413
418037.000.08850.92500.16160.09730.47560.08960.17710583150.8228941685
419037.000.08830.92500.16120.09710.47500.08960.17494600430.8250539957
420037.000.08810.92500.16090.09690.47440.08960.17278617710.8272138229
421037.000.08790.92500.16050.09660.47380.08960.17062634990.8293736501
422037.000.08770.92500.16020.09640.47310.08960.16846652270.8315334773
423037.000.08750.92500.15980.09620.47250.08960.16630669550.8336933045
424138.000.08960.95000.16380.09850.48470.08960.16450216450.8354978355
425038.000.08940.95000.16340.09830.48410.08940.16233766230.8376623377
426038.000.08920.95000.16310.09810.48350.08920.16017316020.8398268398
427038.000.08900.95000.16270.09790.48280.08900.1580086580.841991342
428038.000.08880.95000.16240.09760.48220.08880.15584415580.8441558442
429038.000.08860.95000.16200.09740.48160.08860.15367965370.8463203463
430038.000.08840.95000.16170.09720.48100.08840.15151515150.8484848485
431038.000.08820.95000.16140.09700.48040.08820.14935064940.8506493506
432038.000.08800.95000.16100.09670.47980.08800.14718614720.8528138528
433038.000.08780.95000.16070.09650.47920.08780.1450216450.854978355
434038.000.08760.95000.16030.09630.47860.08760.14285714290.8571428571
435038.000.08740.95000.16000.09610.47800.08740.14069264070.8593073593
436038.000.08720.95000.15970.09590.47740.08720.13852813850.8614718615
437038.000.08700.95000.15930.09560.47680.08700.13636363640.8636363636
438038.000.08680.95000.15900.09540.47620.08680.13419913420.8658008658
439038.000.08660.95000.15870.09520.47560.08660.1320346320.867965368
440038.000.08640.95000.15830.09500.47500.08640.12987012990.8701298701
441038.000.08620.95000.15800.09480.47440.08620.12770562770.8722943723
442038.000.08600.95000.15770.09460.47380.08600.12554112550.8744588745
443038.000.08580.95000.15730.09440.47320.08580.12337662340.8766233766
444038.000.08560.95000.15700.09420.47260.08560.12121212120.8787878788
445038.000.08540.95000.15670.09390.47200.08540.1190476190.880952381
446038.000.08520.95000.15640.09370.47150.08520.11688311690.8831168831
447038.000.08500.95000.15610.09350.47090.08500.11471861470.8852813853
448038.000.08480.95000.15570.09330.47030.08480.11255411260.8874458874
449038.000.08460.95000.15540.09310.46970.08460.11038961040.8896103896
450038.000.08440.95000.15510.09290.46910.08460.10822510820.8917748918
451038.000.08430.95000.15480.09270.46860.08460.10606060610.8939393939
452038.000.08410.95000.15450.09250.46800.08460.10389610390.8961038961
453038.000.08390.95000.15420.09230.46740.08460.10173160170.8982683983
454038.000.08370.95000.15380.09210.46680.08460.09956709960.9004329004
455038.000.08350.95000.15350.09190.46630.08460.09740259740.9025974026
456038.000.08330.95000.15320.09170.46570.08460.09523809520.9047619048
457038.000.08320.95000.15290.09150.46510.08460.09307359310.9069264069
458038.000.08300.95000.15260.09130.46450.08460.09090909090.9090909091
459038.000.08280.95000.15230.09110.46400.08460.08874458870.9112554113
460038.000.08260.95000.15200.09090.46340.08460.08658008660.9134199134
461139.000.08460.97500.15570.09310.47500.08460.08459869850.9154013015
462039.000.08440.97500.15540.09290.47450.08440.08242950110.9175704989
463039.000.08420.97500.15510.09270.47390.08420.08026030370.9197396963
464039.000.08410.97500.15480.09250.47330.08410.07809110630.9219088937
465039.000.08390.97500.15450.09230.47270.08390.07592190890.9240780911
466039.000.08370.97500.15420.09210.47220.08370.07375271150.9262472885
467039.000.08350.97500.15380.09190.47160.08350.07158351410.9284164859
468039.000.08330.97500.15350.09170.47100.08330.06941431670.9305856833
469039.000.08320.97500.15320.09150.47040.08320.06724511930.9327548807
470039.000.08300.97500.15290.09130.46990.08300.06507592190.9349240781
471039.000.08280.97500.15260.09110.46930.08280.06290672450.9370932755
472039.000.08260.97500.15230.09100.46880.08260.06073752710.9392624729
473039.000.08250.97500.15200.09080.46820.08250.05856832970.9414316703
474039.000.08230.97500.15180.09060.46760.08230.05639913230.9436008677
475039.000.08210.97500.15150.09040.46710.08210.05422993490.9457700651
476039.000.08190.97500.15120.09020.46650.08210.05206073750.9479392625
477039.000.08180.97500.15090.09000.46590.08210.04989154010.9501084599
478039.000.08160.97500.15060.08980.46540.08210.04772234270.9522776573
479039.000.08140.97500.15030.08960.46480.08210.04555314530.9544468547
480039.000.08130.97500.15000.08940.46430.08210.04338394790.9566160521
481039.000.08110.97500.14970.08930.46370.08210.04121475050.9587852495
482039.000.08090.97500.14940.08910.46320.08210.03904555310.9609544469
483039.000.08070.97500.14910.08890.46260.08210.03687635570.9631236443
484039.000.08060.97500.14890.08870.46210.08210.03470715840.9652928416
485039.000.08040.97500.14860.08850.46150.08210.0325379610.967462039
486039.000.08020.97500.14830.08840.46100.08210.03036876360.9696312364
487140.000.08211.00000.15180.09040.47230.08210.02826086960.9717391304
488040.000.08201.00000.15150.09030.47170.08200.02608695650.9739130435
489040.000.08181.00000.15120.09010.47110.08180.02391304350.9760869565
490040.000.08161.00000.15090.08990.47060.08160.02173913040.9782608696
491040.000.08151.00000.15070.08970.47000.08150.01956521740.9804347826
492040.000.08131.00000.15040.08950.46950.08130.01739130430.9826086957
493040.000.08111.00000.15010.08930.46890.08110.01521739130.9847826087
494040.000.08101.00000.14980.08920.46840.08100.01304347830.9869565217
495040.000.08081.00000.14950.08900.46780.08080.01086956520.9891304348
496040.000.08061.00000.14930.08880.46730.08060.00869565220.9913043478
497040.000.08051.00000.14900.08860.46670.08050.00652173910.9934782609
498040.000.08031.00000.14870.08850.46620.08030.00434782610.9956521739
499040.000.08021.00000.14840.08830.46570.08020.0021739130.997826087
500040.000.08001.00000.14810.08810.46510.080001
Total relevant:400.45100.54350.6170
Sheet1
Precison
interp prec
F1
Recall
Precision
Sheet2
ROC curve
1 − specificity
sensitivity
Sheet3
Ave. Precision
Recall
Precision
CS3245 – Information Retrieval
Interpolated precision Idea: If locally precision increases with increasing
recall, then you should get to count that… So you take the max of precisions to the right of the
value
Information Retrieval 20
Sec. 8.4
CS3245 – Information Retrieval
Evaluation
Information Retrieval 21
Sec. 8.4
Graphs are good, but often we want a summary measure! Precision at fixed retrieval level
Prevision-at-k: Precision of top k results Perhaps appropriate for most of web search: all people want are
good matches on the first one or two result pages But: averages badly and has an arbitrary parameters of k
11-point interpolated average precisionThe standard measure in the early TREC competitions: you take the precision at 11 levels of recall varying from 0 to 1 by tenths of the documents, using interpolation (the value for 0 is always interpolated!), and average them Evaluates performance at all recall levels
CS3245 – Information Retrieval
Yet more evaluation measures… Mean average precision (MAP) Average of the precision value obtained for the top k
documents, each time a relevant doc is retrieved Avoids interpolation, use of fixed recall levels MAP for query collection is arithmetic ave.
Macro-averaging: each query counts equally
R-precision If have known (though perhaps incomplete) set of relevant
documents of size Rel, then calculate precision of top Reldocs returned
Perfect system could score 1.0.
Information Retrieval 22
Sec. 8.4
CS3245 – Information Retrieval
Variance For a test collection, it is usual that a system does
poorly on some information needs (e.g., MAP = 0.1) and excellent on others (e.g., MAP = 0.7)
Indeed, it is usually the case that the variance in performance of the same system across queries is much greater than the variance of different systems on the same query.
That is, there are easy information needs and hard ones!
Information Retrieval 23
Sec. 8.4
CS3245 – Information Retrieval
CREATING TEST COLLECTIONS FOR EVALUATION
Information Retrieval 24
CS3245 – Information Retrieval
Test Collections
Information Retrieval 25
Sec. 8.5
Scientific papers
Medical
Medical
Scientific papers
News
News
CS3245 – Information Retrieval
From document collections to test collectionsStill need the other 2 things
1.Test queries Must be relevant to docs available Best designed by domain experts Random query terms generally not a good idea
2.Relevance assessments Human judges, time-consuming Are human panels perfect?
Information Retrieval 26
Sec. 8.5
CS3245 – Information Retrieval
Kappa measure for inter-judge (dis)agreement
Information Retrieval 27
Sec. 8.5
Kappa measure Agreement measure among judges Designed for categorical judgments Corrects for chance agreement
Kappa (K) = [P(A)-P(E)][1-P(E)] P(A) – proportion of time judges agree P(E) – what agreement would be by chance
Gives 0 for chance agreement, 1 for total agreement
CS3245 – Information Retrieval
Information Retrieval 28
Sec. 8.5
Kappa Measure: Example
# of docs matching judgment type
Judge 1 Judge 2
300 Relevant Relevant70 Non-relevant Non-relevant20 Relevant Non-relevant10 Non-relevant Relevant
What is P(A)?How about P(E)?
CS3245 – Information Retrieval
Kappa Measure: Example
Information Retrieval 29
Sec. 8.5
P(a) = 370 / 400 = 0.925
Judge 1: P(relevant) = 320/400 = 0.8, P(non-relevant) = 1-0.8 = 0.2Judge 2: P(relevant) = 310/400 = 0.775, P(non-relevant) = 1-0.775= 0.225
P(E) = 0.8*0.775 + 0.2*0.225 = 0.62+0.045 = 0.665Kappa = K = (0.925 - 0.665) / (1-0.665) = 0.776
Kappa > 0.8 Good agreement 0.67 < Kappa < 0.8 Tentative conclusions
Depend on purpose of study For >2 judges: average pairwise kappas (or ANOVA)
CS3245 – Information Retrieval
TREC TREC's Ad Hoc task from first 8 TRECs was the standard IR task
50 detailed information needs a year Human evaluation of pooled results returned More recently other related things: Web, Hard, QA, interactive track
A query from TREC 5 (1996)
225
What is the main function of the Federal Emergency Management Age