Training-less Ontology-based Text Categorization

Training-less Ontology-based Text

Categorization.Maciej Janik

December 14th, 2007PhD Prospectus presentation

Major professor:Dr. Krzysztof J. Kochut

CommitteeDr. John A. MillerDr. Khaled RasheedDr. Amit P. Sheth

Computer Science DepartmentUniversity of Georgia

2

Outline• Document categorization …• Classic approach to categorization• Graph categorization and similarity metrics• Ontology-based approach to categorization• Algorithm sketch• Algorithm details and assumptions• Example and preliminary results• Planned work and expected results• References


3

Document categorization

Document classification/categorization is a problem in information science. The task is to assign an electronic document to one or more categories, based on its contents.[Wikipedia]


4

Document categorization by people• People categorize document by

understanding its content, using their knowledge and understanding what the category is.

• Categorization is based on:– Document content– Knowledge– Category– Perceived interest

features, graphontologycategory definitioncategorization context


5

Automatic text categorization• Automatic text classification can be

defined as task of assigning category labels to new documents based on the knowledge gained in a classification system at the training stage.– require training with pre-classified documents

• Proposed solution– use already defined knowledge for document

categorization and skip the training stage


6

Classic categorization• Methods are based on word/phrase statistics, information

gain and other probability or similarity measures.• Examples [Sebastiani]

– Naïve Bayes, SVM, Decision Tree, k-NN• Categorization based on information (frequencies,

probabilities) learned from the training documents.• Vocabulary extension/unification possible by use of

synonyms, homonyms, word groups (eg. from WordNet)

• Document representation for categorization– Set or vector of features - most popular and simple: bag of

words– Does not include information about document structure, relative

position of phrases, etc.


7

Graph representation of text• Graph representation preserves (selected)

structural information from document– Relative words positions to find close co-occurring

phrases.– Paragraph, formatting (eg. emphasize), part of

document.• Sample representations

– Words form a directed graph, chained in order as they appear in each sentence.

– Words form a weighted graph, where edge connects words within certain distance and weight determines closeness.

– Connected terms based on NLP processing or co-occurrence.


8

Graph representations - examples

[Gamon]

[Schenker]


9

Graph-based categorization• Categorization based on similarity metrics [Schenker]

– Isomorphism– Maximum common subgraph/ minimum common supergraph– Graph edit distance– Statistical methods

• Diameter, degree distribution, betwenness– Comparison of node neighbors– Distance preservation measure

• Methods– k-NN – most straightforward– similarity to centroids – graph mean and graph median– term distance to category


10

Ontology• “An explicit specification of a

conceptualization.” [Tom Gruber]

• Ontology is a data model that represents a set of concepts within a domain and the relationships between those concepts. It is used to reason about the objects within that domain. [Wikipedia]


11

Ontology - example


12

Use of ontologies in classification• Term unification• Hierarchy of concepts• Entity recognition and disambiguation• Strengthening co-occurrence of related

entities• Nearest neighbors


13

Ontology-based classification• Ontology IS the knowledge base and

THE CLASSIFIER – no need for training set.– Rich instance base defines known universe.– Schema with taxonomy describe categorization

structure.• Classification is based on recognized entities

in text and semantic relationships between them.

• Categories assigned are based on entities types and taxonomy embedded in schema.


14

OntoCategorization – bases• Probability

– Traditionally, document is classified based on probabilities that given feature (word, phrase) belongs to a certain category.

– Here: the more features belong to a category, the more probable that document belongs to the category.

• Similarity– Category is defined as ontology fragment (entities,

classes, structures, etc.)– Similarity of document graph to given ontology fragment

describes closeness to selected category• Connectivity (components)

– Knowledge is based on associations.– Entities in one category should form a connected

component, as they belong to the same subject.


15

Classes and categories• Classes do not have to be categories• Classes

– Form taxonomy / partonomy– Strict, formal requirements– Membership based on features

• Categories– Can include other categories, intersect with them, etc. –

more set-like approach– Category can be a complex structure of classes,

relationships and instances– Topic of interest that can span multiple, normally

unrelated classes in schema


16

Who? What? Where? When? Why?• WWW – What (who)? Where? When?

– These text dimensions are orthogonal (in most text).– Fairly easy to find place and date/time.– What / who – description of article’s topic .

• Ontology classification– Focus on text core – find ‘what’ and ‘who’ by matching

entities.– Recognize relationships between entities to construct an

initial document graph.– Graph overlay from ontology on core entities reveals

semantics from background knowledge of analyzed text.• Why? Hmm …


17

OntoCategorization system


18

Algorithm sketch• Convert text to thematic graph

– From words to entities (spotting).– Extract relationships and form triples (NLP).– Overlay background knowledge.– Remove unwanted entities (time/place).

• Categorize graph using ontology– Select thematic component to categorization

(disambiguation and topic set)– Find best category coverage for selected

thematic graph.


19

Algorithm sketch – more details• Match phrases in text with entities in ontology and assign

initial weight.• Graph overlay – add relationships from ontology between

matched entities.• Mark / remove entities related to dates and places.• Add extracted relationships (NLP) between recognized

entities.• Propagate entity weight in graph in similar way as in hubs-

authorities algorithm [Kleinberg].• Find thematic graph(s) for further analysis – connected

component.• Calculate most important entities based on weight and

graph centrality.• Find categories in schema that cover largest part of

thematic component, are lowest in hierarchy and include most important entities.


20

Experiments• Wikipedia ontology

– Includes around 2,000,000 entries• Multiple entity names (variations for matching)

– Has rich instance base (articles)– Internal href, templates and “infobox” relations

carry semantic connections among entries– Has large schema with categories – over 310,00

categories• They DO NOT form a taxonomy, just a graph (even

include cycles)


21

Experiments (2)• Wikipedia 2 RDF

– Created initially by dbpedia.org[Auer, Lehmann]

– Creation of RDF – some modifications• Focus on href, infoboxes and templates

– Special relationships for entities in infoboxes and templates

• Only English version of Wikipedia• Entity name variations for matching

– Name, short name (no brackets), redirect, disambiguation, alternate names


22

Algorithm details (1)• Entity name matching

– Entities and relationships are the content of document – they define topic(s).

– Ontology defines known entities, literals or phrases assigned to them and classifications.

– Analyzed text must contain some of these entities to be categorizable – otherwise it is outside of the ontology scope.

– Matching assigns spotted phrases to known literals, and later to entities.

• Possible use of stop words and/or stemming.


23

Ford Motor Co. is in the process of selling

Jaguar and Land Rover, according to Ford

CEO Alan Mulally.

Example of entity matching

Ford Motor Company

Jaguar (animal)

Business process Process (science)

Alan_MulallyChief Executive Officer

Process (computing)

Jaguar Cars Ltd.Land_Rover Ford Motor Company

Sales


24

Algorithm details (2)• Semantic graph construction

– Add relationships between recognized entities from ontology, as ontology defines meaningful (semantic) connections between them.

– Add relationships extracted from NLP analysis of annotated text.

– Connected entites enable to perform graph analysis, connectivity, finding paths, etc.

• Date and place elimination– Dates and places are orthogonal to topic.– Path connecting entities through place or date is very

little meaningful for document topic.


25

Example – parse tree and triplesFord Motor Co. is in the process of selling Jaguar and Land Rover,

according to Ford CEO Alan Mulally.


26

Example – NLP + ontology knowledgeFord Motor Co. is in the process of selling Jaguar and Land Rover,

according to Ford CEO Alan Mulally.

Ford Motor Company

Jaguar Cars

Land RoverAlan Mulally

Chief Executive OfficerJaguar (animal)

sells

sells

parent_company

parent_company

has_CEO

CEO_of

is_a

named_after


27

Algorithm details (3)• Weight propagation

– Each entity has its initial weight assigned by strength of phrase matching.

– Like in the web, entities are interconnected influence each other.

– We are looking for ‘authority’ entities – assumption is they are most representative for topic.


28

Algorithm details (4)• Thematic subgraph in matched graph

– Assumption is that entities associated with the same or related topics are interconnected in ontology – same as in real life.

– Graph component = topic-related entites.– Each document (or document fragment) should

treat about one or two main topics – leave only most important (weight) and largest component(s).


29

Thematic graph examples

Ford Motor Company

Jaguar Cars

Land RoverAlan Mulally

Chief Executive OfficerJaguar (animal)

Sales

Business

Buyer Newspaper

Announcement

News


30

Algorithm details (5)• Most important and central entities

– Topic tends to center around few entites that are either most important (weight) or are most central in graph.

– Also classification of whole subgraph should be a subset of possible classification of these entities.


31

Algorithm details (6)

• Categorization– Category is defined as set and/or hierarchy of

classes defined in ontology schema.– Each entity has a hierarchy of assigned

categories.– Best ontology class for graph should:

• Cover maximum number of entities in the graph.• Be on relatively lowest level in hierarchy.• Be close in hierarchy to classified entity.• Include most important entities (the more, the better)


32

Entities and categories

Ford Motor Company

Jaguar Cars

Land Rover

Alan Mulally

Chief Executive Officer

Jaguar (animal)

Ford

Car Manufacturers

JaguarFord people

Ford executives

Living people

Felines

Panthera

PantherinaeOff-road wehicles


33

Longer exampleFord, utility ready to work on plug-in car Automaker, Southern California Edison to unveil alliance in

response to demand for energy-efficient vehicles. DETROIT (Reuters) -- Ford Motor Co. and power utility Southern California Edison will announce an

unusual alliance Monday aimed at clearing the way for a new generation of rechargeable electric cars, the companies said.

Ford (Charts , Fortune 500) Chief Executive Alan Mulally and Edison International (Charts , Fortune 500) Chief Executive John Bryson are scheduled to meet with reporters at Edison's headquarters in Rosemead, Calif., the companies said. [...]

Led by Toyota Motor Corp's (Charts) Prius, the current generation of hybrid vehicles uses batteries to power the vehicle at low speeds and in to provide assistance during stop-and-go traffic and hard acceleration, delivering higher fuel economy.

General Motors Corp. (Charts , Fortune 500) has already begun work this year to develop its own plug-in hybrid car, designed to use little or no gasoline over short distances. The company showed off a concept version of the Chevrolet Volt in January at the Detroit Auto show and has awarded contracts to two battery makers to research advanced batteries for a possible production version.


34

Longer exampleFord, utility ready to work on plug-in car Automaker, Southern California Edison to

unveil alliance in response to demand for energy-efficient vehicles. DETROIT (Reuters) -- Ford Motor Co. and power utility Southern California Edison

will announce an unusual alliance Monday aimed at clearing the way for a new generation of rechargeable electric cars, the companies said.

Ford (Charts , Fortune 500) Chief Executive Alan Mulally and Edison International (Charts , Fortune 500) Chief Executive John Bryson are scheduled to meet with reporters at Edison's headquarters in Rosemead, Calif., the companies said. [...]

Led by Toyota Motor Corp's (Charts) Prius, the current generation of hybrid vehicles uses batteries to power the vehicle at low speeds and in to provide assistance during stop-and-go traffic and hard acceleration, delivering higher fuel economy.

General Motors Corp. (Charts , Fortune 500) has already begun work this year to develop its own plug-in hybrid car, designed to use little or no gasoline over short distances. The company showed off a concept version of the Chevrolet Volt in January at the Detroit Auto show and has awarded contracts to two battery makers to research advanced batteries for a possible production version.


35


36

Longer example graph properties• Initial number of vertexes : 205• Initial number of edges : 361• Largest component : 95• Component for analysis : 35• Central and most important entities:

– Hybrid_vehicle * Centrality 208, * weight 1.516873

– Automobile * Centrality 213, weight 1.249790,

– Internal_combustion_engine* Centrality 233, weight 1.069511

– Ford_Motor_CompanyCentrality 237, * weight 1.451533,

– Southern_California_EdisonCentrality 351, * weight 1.308824


37

Longer example categories• Category:Automobiles

– CAT instances <13>, (avg. height 2.384615)weight [0.874697]

• Category:Alternative_propulsion– CAT instances <4>, (avg. height 1.250000)

weight [0.873287]• Category:Car_manufacturers

– instances <3> (avg. height 1.000000) weight [0.781271]

• Category:Vehicles– CAT instances <13>, (avg. height 2.923077)

weight [0.647903]• Category:Transportation

– CAT instances <11>, (avg. Height 3.090909) weight [0.629714]


38

Wikipedia categories• Wikipedia categories DO NOT form a taxonomy

– It is just a directed graph, that contains cycles.– Not possible to use subsumption for categories.– Thesaurus-like structure. [Voss]

• Categories may be very deep and detailed, or very broad– Hard to pinpoint the cut-off point good for

categorization.– There is no simple mapping between news categories

and categories in Wikipedia.


39

Overall performance of initial tests• Tests against classic BOW statistic

classifier [McCallum].• Source articles and categories taken from

CNN – total of 7158 documents in 14 categories.– Divided into 50% training / 50% testing split

• Mapping between Wikipedia and CNN categories done manually by crawling generated Wikipedia schema (still not really precise)


40

Text corpora – CNN news


41

CNN and Wikipedia• CNN categories

– Classified by people– Describe mostly article interest, not necessarily

its content• Frequently described reader’s interest rather than

true subject.– Hard to match to Wikipedia categories

• Wikipedia categories– Content-based– Very detailed and deep


42

Categorization results - BOW


43

Categorization results – BOW on Wikipedia


44

Categorization results - Wikipedia


45

Summary of work• Ontology storage and querying

– Brahms RDF/S storage– Sparqler – query language extension with path queries

• For use in Glycomics project• Prototype of ontology-based categorization

– Partial implementation – not all modules included yet– Use of general-purpose ontology – RDF graph created

from English Wikipedia– Initial tests confirm proof of concept– Published as technical report, submitted to WWW 2008


46

Remaining research• Goal

– Create comprehensive model for ontology-based categorization.

• Create semantic context definition• Modify and/or create graph similarity

measures that exploit context information


47

Current work in progress• Goal

– Create a system, where user can categorize text document with given ontology using specified semantic context.

• NLP module for relationship extraction• Definition of query context

– Extension of SPARQL with context queries


48

Proposed work• Include NLP analysis in creating relationships between

entities– Will help to link entities that do not have connection in

ontology or strengthen this connection.• Explore categorization to a user-defined context (collection

of instances, classes, structures, path expressions).• Extend definition of category to include context.• Experiment with other well-developed ontologies to

categorize more specialized documents– Eg. PubMed

• (optional) Study the applicability of the method for ontology-based document summarization.


49

Published papers• Maciej Janik, Krys Kochut. "BRAHMS: A WorkBench RDF Store And High

Performance Memory System for Semantic Association Discovery", Fourth International Semantic Web Conference, ISWC 2005, Galway, Ireland, 6-10 November 2005

• Krys Kochut, Maciej Janik. "SPARQLeR: Extended Sparql for Semantic Association Discovery", Fourth European Semantic Web Conference, ESWC 2007, Innsbruck, Austria, 3-7 June 2007

• Matthew Perry, Maciej Janik, Cartic Ramakrishnan, Conrad Ibanez, Budak Arpinar, Amit Sheth. "Peer-to-Peer Discovery of Semantic Associations", Second International Workshop on Peer-to-Peer Knowledge Management, San Diego, CA, July 17, 2005

• Maciej Janik, Krys Kochut. "Wikipedia in action: Ontological Knowledge in Text Categorization", UGA Technical Report No. UGA-CS-TR-07-001, November 2007 – submitted to WWW 2008

• S. Nimmagadda, A. Basu, M. Evenson, J. Han, M. Janik, R. Narra, K. Nimmagadda, A. Sharma, K.J. Kochut, J.A. Miller and W. S. York, "GlycoVault: A Bioinformatics Infrastructure for Glycan Pathway Visualization, Analysis and Modeling," Proceedings of the 5th International Conference on Information Technology: New Generations (ITNG'08), Las Vegas, Nevada (April 2008) [to appear]


50

References• Auer, S. and Lehmann, J., What have Innsbruck and Leipzig in common?

Extracting Semantics from Wiki Content. in European Semantic Web Conference (ESWC'07), (Innsbruck, Austria, 2007), Springer, 503-517.

• Gamon, M., Graph-Based Text Representation for Novelty Detection. in Workshop on TextGraphs at HLT-NAACL 2006, (New York, NY, US, 2006).

• Gruber, T. A Translation Approach to Portable Ontology Specifications. Knowledge Acquisition, 5 (2). 199-220, 1993.

• Kleinberg, J.M., Authoritative Sources in a Hyperlinked Environment. in ACM-SIAM Symposium on Discrete Algorithms, (1998).

• McCallum, A.K. Bow: A toolkit for statistical language modeling, text retrieval, classification and clustering. http://www.cs.cmu.edu/~mccallum/bow, 1996.

• Nagarajan, M., Sheth, A.P., Aguilera, M., Keeton, K., Merchant, A. and Uysal, M. Altering Document Term Vectors for Classification - Ontologies as Expectations of Cooccurrence LSDIS Technical Report, November, 2006.

• Schenker, A., Bunke, H., Last, M. and Kandel, A. Graph-Theoretic Techniques for Web Content Mining. World Scientific, London, 2005.

• Sebastiani, F. Machine learning in automated text categorization. ACM Computing Surveys (CSUR), 34 (1). 1 - 47.

• Voss, J. Collaborative thesaurus tagging the Wikipedia way. ArXiv Computer Science e-prints, cs/0604036.

Documents

Training-less Ontology-based Text Categorization