Classifying Tags Using Open Content ResourcesSimon Overell, Borkur Sigurbjornsson & Roelof van Zwol

WSDM ‘09

Motivation Classify tags in Flickr as broad categories

such as what, where, when and who Easier indexing and navigation WordNet is usually used for

classification but has limited coverage

Example

The ClassTag System

Classifying Wikipedia Articles Using only metadata (i.e. Categories

and Templates) – high scalability Supervised Classifier

Articles as objects WordNet noun semantic categories as

classification classes Categories and Templates as features

Support Vector Machine (SVM) as classifier

Categories and Templates

Categories and Templates

Supervised Classification Ground Truth

All Wikipedia articles that match WordNet nouns

Data Sparsity WordNet categories under represented

(10 out of 25) Articles have very few features

Reducing Data Sparsity Using category and

template network transclusion

… but noise is added

System Optimization Number of arcs traversed in

Category network Template network

Choice of weighting function Term Frequency (tf) Term Frequency – Inverse Document

Frequency (tf-idf) Term Frequency – Inverse Layer (tf-il)

Example

Fine Tuning Partitioned the ground truth into training

and test sets Criteria

At least 80% precision Maximum possible recall

Resulted optimal values Category arcs: 3, Template arcs: 3, TF-IL Precision: 87% F1-Measure:0.696

SVM Threshold SVM outputs confidence with which an

article is correctly classified as a member of a category

Training experiment with 250 Wikipedia articles (1 assessor)

SVM Threshold

SVM Threshold

Summary Optimised for Recall (ClassTag)

39% of Articles classified 664,770 Wikipedia articles

Optimised for Precision (ClassTag+) 21% of Articles classified 338,061 Wikipedia articles

Comparison with DBpedia• Experimental Setup

– 300 pooled articles– 3 Assessors– Blind Assessments– 50 articles overlap

• Partial Agreement:– 86%

• Total Agreement:– 78%

Results

Classification of Flickr Tags Tag Anchor Text

String matching Anchor Text Wikipedia Article

Number of times an anchor refers to a Wikipedia article

Wikipedia Article Category Output of SVM decision

Ambiguity Tag Anchor Text

Some ambiguity because often tags are lower case with no white spaces

Anchor Text Wikipedia Article 13.4% of Anchor text -> Wikipedia Article mappings

ambiguous 4% of Anchor text -> Category mappings ambiguous Example

George Bush -> George W. Bush, George Bush Senior George Bush -> Person

Wikipedia Article Category 5.7% of classified articles result in multiple classification

Example

Evaluation WordNet classification extended

vocabulary coverage by 115% Taking tag frequency into account

ClassTag classified 69.2% of Flickr tags 22% more than WordNet baseline

Tag distribution

Multilanguage Classification 80% of tags in English, 7% in German

and 6% in Dutch Maybe a portion of the unclassified tags

fall into this category Possible alternate language classification

Run ClassTag using alternate Wikipedia language and a corresponding lexicon

Translate the English classification using Wikipedia’s interlanguage links

Contributions Classifying open content resources

using their structural patterns Presenting ClassTag - a system for

classifying tags ClassTag extends the WordNet lexicon

using the structural patterns of Wikipedia

Conclusion Tuneable system for classifying

Wikipedia pages ClassTag: Nearly 40% of articles classified

with a precision of 72% ClassTag+: 21% of articles classified with

a precision of 86% (equal to assessor agreement)

Nearly 70% of Flickr tags matched to WordNet categories