14
Precision, Average Precision, Average Precision and Mean Precision and Mean Average Precision Average Precision For the Mathematically For the Mathematically Challenged Challenged

Calculating precision

Embed Size (px)

DESCRIPTION

 

Citation preview

Page 1: Calculating precision

Precision, Average Precision Precision, Average Precision and Mean Average Precisionand Mean Average Precision

For the Mathematically For the Mathematically ChallengedChallenged

Page 2: Calculating precision

How to calculate precisionHow to calculate precision

In this search, the total In this search, the total number of hits is 5.number of hits is 5.

Whoops, the History of the Whoops, the History of the Stooges is not relevant.Stooges is not relevant.The total number of The total number of relevant hits is 4relevant hits is 4

The math:The math:

4/5=0.8 or 80%4/5=0.8 or 80%

Relevant hitsTotal hits

Page 3: Calculating precision

How to calculate Average PrecisionHow to calculate Average PrecisionStep 1Step 1

Take each entry Take each entry separately.separately.

Page 4: Calculating precision

How to calculate Average PrecisionHow to calculate Average PrecisionStep 1Step 1

So for item #1So for item #1 Pretend the search Pretend the search

stopped at item one:stopped at item one: Hits= 1Hits= 1 Relevant hits =1Relevant hits =1 Math 1/1=1Math 1/1=1

Do the same calculation Do the same calculation for each itemfor each item

Page 5: Calculating precision

How to calculate Average PrecisionHow to calculate Average PrecisionKeep doing the same calculationKeep doing the same calculation

Next, Next, pretend the pretend the search stopped at item search stopped at item two:two: Hits= 2Hits= 2 Relevant hits =2Relevant hits =2 Math 2/2=1Math 2/2=1

Page 6: Calculating precision

How to calculate Average PrecisionHow to calculate Average PrecisionThings change at 3 hitsThings change at 3 hits

Remember, the third item is not relevant, so we skip this one.

Page 7: Calculating precision

How to calculate Average PrecisionHow to calculate Average Precision 4 hits-- different yet again 4 hits-- different yet again

Pretend the search Pretend the search stopped at item 4:stopped at item 4:

Hits= 4Hits= 4 Relevant hits =3Relevant hits =3 Math 3/4=0.75Math 3/4=0.75

Page 8: Calculating precision

How to calculate Average PrecisionHow to calculate Average PrecisionAnd for all 5 hitsAnd for all 5 hits

Pretend the search Pretend the search stopped at item one:stopped at item one:

Hits= 5Hits= 5 Relevant hits =4Relevant hits =4 Math 4/5=0.8Math 4/5=0.8

Page 9: Calculating precision

Average Precision Step 1Average Precision Step 1

The Math: SummaryThe Math: Summary

Pretend the search Pretend the search stopped at #1stopped at #1

1 relevant hit/ 1 1 relevant hit/ 1 total hits = 1total hits = 1

Pretend the search Pretend the search stopped at #2stopped at #2

2 relevant hits/2 2 relevant hits/2 total hits = 1total hits = 1

Pretend the search Pretend the search stopped at #3stopped at #3

Not relevant –we Not relevant –we skip this oneskip this one

Pretend the search Pretend the search stopped at #4stopped at #4

3 relevant hits/4 3 relevant hits/4 total hits=0.75total hits=0.75

Pretend the search Pretend the search stopped at #5stopped at #5

4 relevant hits/5 4 relevant hits/5 total hits=0.8total hits=0.8

Page 10: Calculating precision

Average Precision: Step 2Average Precision: Step 2

The Math (from step 2)The Math (from step 2)

Pretend the search Pretend the search stopped at #1stopped at #1

1 relevant hit/ 1 1 relevant hit/ 1 total hits = 1total hits = 1

Pretend the search Pretend the search stopped at #2stopped at #2

2 relevant hits/2 2 relevant hits/2 total hits = 1total hits = 1

Pretend the search Pretend the search stopped at #3stopped at #3

skippedskipped

Pretend the search Pretend the search stopped at #4stopped at #4

3 relevant hits/4 3 relevant hits/4 total hits=0.75total hits=0.75

Pretend the search Pretend the search stopped at #5stopped at #5

4 relevant hits/5 4 relevant hits/5 total hits=0.8total hits=0.8

Now add up all the Now add up all the precision calculations precision calculations from the last stepfrom the last step

1+1+0.75+0.81+1+0.75+0.8

The total here is 3.55

Page 11: Calculating precision

Average Precision: Step 3Average Precision: Step 3

Now to take the Now to take the average, take the average, take the total of all the total of all the precisions 3.55 and precisions 3.55 and divide by the number divide by the number of relevant hits (4)of relevant hits (4)

3.55/4=0.888 or 3.55/4=0.888 or 88.8%88.8%

Page 12: Calculating precision

Looks harder than it is.Looks harder than it is.

Page 13: Calculating precision

Precision vs Average PrecisionPrecision vs Average Precision

Precision ResultsPrecision Results

80%80%

Average Precision Average Precision ResultsResults

88.8%88.8%

Now let’s compare the two:

Page 14: Calculating precision

Mean Average PrecisionMean Average Precision

Take the Average Precision of multiple Take the Average Precision of multiple queriesqueriesUse statistical sampling to determine how Use statistical sampling to determine how

many queries you need.many queries you need.Add up all the Average Precisions for each Add up all the Average Precisions for each

query and divide by the number of queries.query and divide by the number of queries.

Add all average precision calculations togetherTotal number of queries

Mean Average Precision

=