A Genetic Algorithm Using Semantic Relations for Word ...kalita/work/StudentResearch/Haus... · A Genetic Algorithm Using Semantic Relations for Word Sense Disambiguation Michael

A Genetic Algorithm Using Semantic

Relations for Word Sense Disambiguation

Michael Hausman

A master’s project report submitted to the Graduate Facility of the

University of Colorado at Colorado Springs

in partial fulfillment of the requirements for the degree of

Masters of Engineering in Software Engineering

Department of Computer Science

2011

This master’s project report for

Masters of Engineering in Software Engineering

degree by

Michael Hausman

has been approved for the

Department of Computer Science

By

_________________________ Jugal Kalita, Chair

_________________________ Edward Chow

_________________________ Al Brouillette

__________ Date

Hausman iii

A Genetic Algorithm Using Semantic Relations for Word Sense Disambiguation

Abstract

Word Sense Disambiguation is a formal way of saying, “Which dictionary definition is correct in

context?” Humans are adept at extracting the context of a sentence and applying it to every word. If

the sentence is “I like to swim next to the river bank,” then the word “bank” means “sloping land” and

does not mean “a financial institution.” Humans know this because the words “river” and “swim” have

very little to do with finance. Humans have years of knowledge and experience to quickly contextualize

the meaning of every word. A machine, however, has a much harder time finding the correct meaning.

It takes thousands of computations for even the simplest algorithms, which are not very accurate. Even

so, many applications such as language translators are still available and sold today. Language

translation relies heavily on word sense disambiguation. For this reason many translated sentences do

not make much sense. Solving word sense disambiguation would help with many applications such as

language translation.

This project explores solutions to the word sense disambiguation dilemma. There are a variety

of tools such as WordNet and SemCor referenced in this project. WordNet is a lexical database that

investigates several relations for comparing words and glosses of a word. SemCor is a collection of text

tagged with the proper part of speech and definitions. This researcher uses the semantic relations from

WordNet and examples from SemCor to measure which definitions are most likely to be correct in the

context of the communication.

A genetic algorithm employs these measurements to find the optimal set of definitions across

several sentences. Then, the researcher compares the algorithm to other word sense disambiguation

algorithms.

Hausman iv


Table of Contents

Abstract ............................................................................................................................................ iii

Table of Figures ............................................................................................................................... vii

Table of Tables ................................................................................................................................ viii

Table of Equations ............................................................................................................................ ix

Table of Examples .............................................................................................................................. x

Chapter 1: Introduction .................................................................................................................... 11

Chapter 2: Background ..................................................................................................................... 15

2.1 Path‐Based Approaches .................................................................................................................... 15

2.2 Information‐Based Methods ............................................................................................................. 16

2.3 Gloss Based Methods ........................................................................................................................ 17

2.4 Vector Based Methods ...................................................................................................................... 18

2.5 Using Multiple Approaches ............................................................................................................... 18

2.6 A Genetic Algorithm Approach ......................................................................................................... 19

2.7 Main Ideas behind the Approach in this Project............................................................................... 21

Chapter 3: Tools/Resources .............................................................................................................. 22

3.1 WordNet ........................................................................................................................................... 22

3.2 WordNet Interface ............................................................................................................................ 22

3.3 Part of Speech Tagger ....................................................................................................................... 23

3.4 SemCor .............................................................................................................................................. 24

3.5 SemEval ............................................................................................................................................. 24

3.6 OntoNotes ......................................................................................................................................... 25

Chapter 4: Introduction to Semantic Relations .................................................................................. 27

4.1 Frequency.......................................................................................................................................... 27

4.2 Hypernym .......................................................................................................................................... 29

4.3 Coordinate Sisters ............................................................................................................................. 30

4.4 Domain .............................................................................................................................................. 31

4.5 Synonym ............................................................................................................................................ 32

4.6 Antonym ............................................................................................................................................ 33

Chapter 5: A Word Sense Disambiguation Genetic Algorithm ............................................................ 35

Hausman v


5.1 Evolution of the Cost Function .......................................................................................................... 36

5.1.1 Notes on Sampling ..................................................................................................................... 37

5.1.2 Semantic Relation Investigation ................................................................................................ 39

5.1.2.1 Frequency Behavior ............................................................................................................. 41

5.1.2.2 Hypernym Behavior ............................................................................................................. 43

5.1.2.3 Coordinate Sister Behavior .................................................................................................. 45

5.1.2.4 Domain Behavior ................................................................................................................. 46

5.1.2.5 Synonym Behavior ............................................................................................................... 48

5.1.2.6 Antonym Behavior ............................................................................................................... 49

5.1.3 The Optimal Cost Function ......................................................................................................... 51

5.1.4 Cost Function Method 1: Simple Addition .................................................................................. 51

5.1.5 Cost Function Method 2: Regression .......................................................................................... 52

5.1.6 Cost Function Method 3: Proportional Placement in a Range ................................................... 55

5.1.7 Cost Function Method 4: Add Sense Distribution ....................................................................... 58

5.1.8 Cost Function Method 5: Add Semantic Relation Distribution ................................................... 60

5.2 Mating ............................................................................................................................................... 62

5.2.1 Mating Method 1: Mate Top Third ............................................................................................ 63

5.2.2 Mating Method 2: Mate Middle Third ....................................................................................... 63

5.3 Mutation ........................................................................................................................................... 64

5.3.1 Mutation Function 1: Random Mutation ................................................................................... 65

5.3.2 Mutation Function 2: Semantic Relation Score Mutation .......................................................... 65

5.3.3 Mutation Function 3: Sense Distribution.................................................................................... 66

5.3.4 Mutation Function 4: Semantic Relation Distribution ................................................................ 67

5.4 Main Genetic Algorithm Function ..................................................................................................... 67

5.5 Notes about Speed ............................................................................................................................ 68

Chapter 6: Results and Analysis ........................................................................................................ 70

6.1 Measuring the Results ...................................................................................................................... 70

6.2 Comparison to Michael Billot ............................................................................................................ 71

6.3 Comparison to Zhang ........................................................................................................................ 72

6.4 SemEval ............................................................................................................................................. 72

6.4.1 SemEval 2 ................................................................................................................................... 72

6.4.2 SemEval 3 ................................................................................................................................... 73

Hausman vi


6.4.3 SemEval 2007 ............................................................................................................................. 74

Chapter 7: Conclusions and Future Research ..................................................................................... 76

7.1 Algorithm Weaknesses ..................................................................................................................... 76

7.2 Future Possibilities ............................................................................................................................ 77

Appendix ......................................................................................................................................... 79

Appendix A: SemCor Files ....................................................................................................................... 79

Appendix B: Proportional Placement Statistics ...................................................................................... 92

Appendix C: Sense Distribution Statistics ............................................................................................... 94

Appendix D: Semantic Relation Distribution Statistics ........................................................................... 94

Appendix E: SemCor Results ................................................................................................................... 98

References ..................................................................................................................................... 104

Hausman vii


Table of Figures

Figure 1: An example graph indicating the various sample areas .............................................................. 41

Figure 2: The graph of the frequency semantic relation using a noun‐noun part of speech combination 42

Figure 3: The graph of the frequency semantic relation using an adjective‐adjective part of speech combination ................................................................................................................................................ 42

Figure 4: The graph of the hypernym semantic relation using a noun‐noun part of speech combination 44

Figure 5: The graph of the hypernym semantic relation using a verb‐verb part of speech combination .. 44

Figure 6: The graph of the coordinate sister semantic relation using a noun‐noun part of speech combination ................................................................................................................................................ 45

Figure 7: The graph of the coordinate sister semantic relation using a verb‐verb part of speech combination ................................................................................................................................................ 46

Figure 8: The graph of the domain semantic relation using a noun‐noun part of speech combination .... 47

Figure 9: The graph of the domain semantic relation using an adverb‐adverb part of speech combination .................................................................................................................................................................... 47

Figure 10: The graph of the synonym semantic relation using a verb‐verb part of speech combination . 48

Figure 11: The graph of the synonym semantic relation using an adjective‐adjective part of speech combination ................................................................................................................................................ 49

Figure 12: The graph of the antonym semantic relation using a noun‐noun part of speech combination 50

Figure 13: The graph of the antonym semantic relation using an adjective‐adjective part of speech combination ................................................................................................................................................ 50

Figure 14: The graph indicating the shape of the ideal cost function ........................................................ 51

Figure 15: The graph indicating the shape of Cost Function Method 1 ..................................................... 52

Figure 16: A graph with an example regression equation .......................................................................... 53

Figure 17: A graph indicating an example of the regression of multiple regressions for frequency ......... 54


Figure 19: A graph showing an example of the weaknesses of multiple hypernym regressions ............... 56

Figure 20: A graph showing an example using Cost Function Method 3 ................................................... 57



Hausman viii


Table of Tables

Table 1: The results from SemEval 2 ........................................................................................................... 73

Table 2: The results from SemEval 3 ........................................................................................................... 74

Table 3: The coarse results from SemEval 2007 ......................................................................................... 75

Table 4: The SemCor file letter indicating type of resource the original text came from. ......................... 79

Table 5: The various SemCor files ............................................................................................................... 79

Table 6: The values for Cost Function Method 3 used in this project ........................................................ 92



Table 9: The results of one run of every SemCor file using all the parts of speech.................................... 99

Hausman ix


Table of Equations

Equation 1 and Example 1: Three Path‐Based Methods ............................................................................ 16

Equation 2 and Example 2: Three Information‐Based Methods ................................................................ 17

Equation 3: Cost function for Zhang’s solution ........................................................................................... 20

Equation 4: Semantic Relation Equation for Frequency ............................................................................. 28

Equation 5 and Example 4: Semantic Relation Equation and Example for Hypernyms ............................. 30

Equation 6 and Example 5: Semantic Relation Equation and Example for Coordinate Sisters .................. 31

Equation 7 and Example 6: Semantic Relation Equation and Example for Domain ................................... 31

Equation 8 and Example 7: Semantic Relation Equation and Example for Synonyms ............................... 33

Equation 9 and Example 8: Semantic Relation Equation and Example for Antonyms ............................... 34

Equation 10: The Weighted Probability Equation ...................................................................................... 38

Equation 11: The equation for Cost Function Method 2 ............................................................................ 54


Equation 13: The equation for sense distribution error ............................................................................. 59


Equation 15: The equation for semantic relation distribution error .......................................................... 61


Equation 17: The equation for comparing the current solution to the optimal solution ........................... 66

Hausman x


Table of Examples

Equation 1 and Example 1: Three Path‐Based Methods ............................................................................ 16

Equation 2 and Example 2: Three Information‐Based Methods ................................................................ 17

Example 3: Example using the Frequency Semantic Relation Equation ..................................................... 29

Equation 5 and Example 4: Semantic Relation Equation and Example for Hypernyms ............................. 30

Equation 6 and Example 5: Semantic Relation Equation and Example for Coordinate Sisters .................. 31

Equation 7 and Example 6: Semantic Relation Equation and Example for Domain ................................... 31

Equation 8 and Example 7: Semantic Relation Equation and Example for Synonyms ............................... 33

Equation 9 and Example 8: Semantic Relation Equation and Example for Antonyms ............................... 34

Example 9: An example using the Weighted Probability Equation ............................................................ 38

Hausman 11


Chapter 1: Introduction

This project explores word sense disambiguation. A word sense is a definition or meaning of a

word. Disambiguation serves to “remove all ambiguity.” Therefore, this research explores the technical

way of saying, “Which dictionary definition is correct in context?” For example, let us suppose that the

application is a GPS in a car that uses verbal locations. If the user says, “I want to go to the bank,” where

is the location? The noun “bank” has several meanings. Some of these possibilities are below. This

could be a financial institution to deposit a check. This could also be the river bank next to the house.

From this sentence alone, the GPS cannot tell the difference. It could just assume the most commonly

used meaning and point to the financial institution. However, if the next sentence is, “I want to go for

a swim,” then the assumption would be wrong. To understand which location is the destination, the

GPS would need to know the correct sense of the location. To understand which sense is correct, the

GPS needs to understand the context.

Possible senses of the noun “bank” (Princeton University, 2010) 1. a financial institution that accepts deposits and channels the money into lending activities 2. sloping land (especially the slope beside a body of water) 3. a supply or stock held in reserve for future use (especially in emergencies)) 4. a building in which the business of banking transacted 5. an arrangement of similar objects in a row or in tiers 6. a container (usually with a slot in the top) for keeping money at home 7. a long ridge or pile 8. the funds held by a gambling house or the dealer in some gambling games 9. a slope in the turn of a road or track; the outside is higher than the inside in order to reduce

the effects of centrifugal force 10. a flight maneuver; aircraft tips laterally about its longitudinal axis (especially in turning)

Word sense disambiguation is not as simple as it sounds. In order for a machine to understand

which sense is correct, it must correlate several things. Some of these are: A) how many senses are

possible for this word? B) How do those senses correlate to other words? and C) What is important in a

Hausman 12


sentence? In the preceding example, the machine would need to understand that swimming involves

water, that water implies sense 2, and the location of the desired body of water. If the GPS were to

accept any sentence, then the knowledge base required to connect every word would be immense. It

requires several years for humans to make sense and then making meaning of the simplest sentences.

To achieve this with a machine is significantly more complex.

The fictional GPS is not the only application of word sense disambiguation. If word sense

disambiguation were solved, then many applications would be possible. This could include computers

that actually talk and interact with humans as seen in science fiction movies. This also includes language

translators such as Google Translate (Google, 2010).

At present, many translated sentences do not make much sense or are hard to understand

because words are missing and the context is often incorrect. Just knowing the correct sense would

help convey the correct context after the translation. Other applications that benefit from word sense

disambiguation include text classification, automatic summaries, or anywhere text or language is

analyzed. This project solely focuses on the problem of word sense disambiguation.

The first step to word sense disambiguation is to understand how words relate within a specific

context. Humans understand the context of a word by looking at the surrounding words. Humans also

compare words in several different ways. For example, a human knows that a lake, a river, and the

ocean are all related because they are all bodies of water. Swimming is a water sport, so water must be

involved. Many people visit rivers that are near hills and mountains in order to enjoy the scenery.

Therefore, if the word “bank” were in the middle of the statement, then the correct sense of the word

bank must be 2. It probably is not a financial institution. Perhaps a machine could try the same

technique. The first step is to examine the surrounding words and compare them with each other. A

semantic relation allows the machine to compare two words in a specific way. For example, a hypernym

of a word is a more generic way of saying that word. A more generic way of saying “lake”, “river” and

Hausman 13


“ocean” is “body of water.” This means that “lake”, “river”, and “ocean” all share similarity through the

hypernym semantic relation. “Hillside” and the second sense of “bank” are also similar via a hypernym

semantic relation. Semantic relations provide a way for a machine to compare words with each other.

Exploring Semantic relations is the primary lens for this project. This exploration compares

several words. Earlier, the hypernym semantic relation showed some similarity between several words.

The words “lake”, “river”, and “ocean” were similar because their hypernym was “body of water.”

However, it takes more hypernyms to compare “hillside” and “bank.” This implies that the first set is

more related to each other than the second set. It also means that a portion of the project is examining

how to properly measure each relation to account for this varying similarity. Not all words have

hypernyms. To account for this, there are several semantic relations in this project: frequency,

hypernym, coordinate sister, domain, synonym, and antonym. Each of these semantic relations has

different ways of comparing two words.

As the title suggests, semantic relations are only part of the project. A genetic algorithm uses

these semantic relations to provide the senses for a given text. A genetic algorithm uses Darwin’s

theory of evolution to evolve a solution over time for optimization problems. The advantage of a

genetic algorithm is that it does not need to compare every possible solution. This is important because

there are billions of possible sense combinations for a paragraph of text. The disadvantage of this

method is that the solution may not be the best possible combination, but it should be a “good” solution

if the genetic algorithm is performed correctly.

Chapter One of this report has introduced the concept of word sense disambiguation. Chapter

Two describes several techniques which other researches use in word sense disambiguation. Chapter

Three describes all of the tools and resources that this project uses to provide all of the sense and

semantic relation information related to this study. Chapter Four explains what the semantic relations

are and how the project measures the similarity between two words with each relation. Chapter Five

Hausman 14


explains the genetic algorithm in this project. The major portion of this chapter explains how the

genetic algorithm transforms these measures to make word sense disambiguation an optimization

problem. Chapter Six compares the results to several researchers and competitions. Chapter Seven

concludes by suggesting possible areas for research in the future.

Hausman 15


Chapter 2: Background

There are various ways to attempt word sense disambiguation. There are also various ways to

classify each attempt. One classification is supervised versus unsupervised. A supervised algorithm

depends on some training data to learn or compare information. The algorithm then uses that

knowledge to more accurately tag other text. An unsupervised system does not require any training

data. Typically the supervised approaches are more accurate if the training data are available.

However, this classification is extremely generic. Zesch and Gurevych classify four categories of

approaches in their paper: path‐based, information content based, gloss based, and vector based (Zesch

& Gurevich, 2010). These classifications give a very high‐level indication on how early researchers

approached the problem. Many of the newer researchers often combine several of these approaches to

achieve better results.

2.1 Path‐Based Approaches

Path‐based approaches measure the length of the path between two words. The shorter the

path, the more related the two words are. This method relies on a resource that supplies the paths.

This is usually a graph‐like structure, like WordNet (Princeton University, 2010). Many early researchers

focus on a single relation, like hypernyms, and optimize for that relation. What hypernyms are is not as

important at this point as the fact that hypernyms create a tree. Once the relation is in a tree structure,

then someone can measure the paths in the tree. Rada et al. begin by measuring the hypernym edges

between the words (Rada, Mili, Bicknell, & Blettner, 1989). Then, Leacock and Chodorow normalize the

length by accounting for the tree depth (Leacock & Chodorow, 1998). Wu and Palmer also use this idea

of depth in their equation (Wu & Palmer, 1994). The difference is that Wu and Palmer use the depth to

Hausman 16


the “lowest common subsumer,” which is where the hypernyms of two words intersect. This creates

three different measurements for the “hypernym tree” alone. An example and the two equations are

below. The equations are the main part of both Rada’s and Leacock’s methods. The majority of their

papers describe equations performance for various text examples. However the path is measured, a

path‐based method takes the measurement and optimizes for the shortest path between all words.

Equation 1 and Example 1: Three Path‐Based Methods

2.2 Information‐Based Methods

Information‐based methods take into account how much information the two words share. The

more information the two words share, the more similar the two words are. For example, start with the

hypernym tree structure from before. Just as before, the only important fact is that the relation creates

a tree structure. Since it is a tree structure, someone can measure the number of nodes that both

words share. The more nodes two words share, the more related the two words are. Resnik adapted

this idea and defines the “lowest common subsumer” as the point where the two words intersect in the

hypernym tree (Resnik, 1995). All the words from the top of the tree to the lowest common subsumer

are common subsumers. Resnik adds the probability of all the subsumers appearing in a corpus as the

Depth

w

w

w

w

w

w

w

w

W1

W2

LCS Depth

Length

𝑅𝑎𝑑𝑎(𝑤", 𝑤%) = 𝐿𝑒𝑛𝑔𝑡ℎ 𝑅𝑎𝑑𝑎(𝑤", 𝑤%) = 3

𝐿𝐶(𝑤", 𝑤%) = −log 𝐿𝑒𝑛𝑔𝑡ℎ2 ∗ 𝐷𝑒𝑝𝑡ℎ

𝐿𝐶(𝑤", 𝑤%) = 0.426

𝑊𝑃(𝑤", 𝑤%) =2 × 𝐿𝐶𝑆 𝐷𝑒𝑝𝑡ℎ

𝐿𝑒𝑛𝑔𝑡ℎ + 2 × 𝐿𝐶𝑆 𝐷𝑒𝑝𝑡ℎ

𝑊𝑃(𝑤", 𝑤%) = 0.727 Length: length of the path between the words Depth: the longest depth to the words LCS Depth: the depth to the lowest common

subsumer

Hausman 17


measure of similarity. Jiang and Conrath modify Resnik’s idea to include the distance between the word

and the lowest common subsumer (Jiang & Conrath, 1997). Lin begins with Jiang’s hypothesis and uses

the universal measure from information theory instead (Lin, 1998). All three of these measures use the

same idea, which is adding the probability of a word appearing in a corpus. The difference is in the

equations. An algorithm depending on the amount of shared information indicates an information‐

based approach.

Equation 2 and Example 2: Three Information‐Based Methods

2.3 Gloss Based Methods

Gloss‐based methods rely on the definition of the word. Dictionaries describe definitions with

different words. A dictionary will describe two similar words with other similar words. Keep track of all

the similar words between two senses, and the overlap becomes a measurement. Lesk is famous for

using this concept to tell a “pine cone” from an “ice cream cone” (Lesk, 1986). The words “pine” and

“cone” both have definitions that contain the words “tree” and “fruit.” This is a completely different

overlap than the senses of “ice cream” and “cone.” Simply pick the senses that provide the most

overlap with each other. There are, however, drawbacks to this method. If the glosses are not

descriptive enough, or if there are many false positives, this method will fail. Banerjee and Pederson

attempt to account for both problems with their Adapted Lesk Algorithm (Patwardhan, Banerjee, &

w

w

wLCS

wC

wB

wA

wD

w

W1

W2

𝐼𝐶(𝑤∗) = −log7(𝑤1) + 𝑃(𝑤2) + ⋯+ 𝑃(𝑤∗)9P 𝑅𝑒𝑠(𝑤",𝑤%) = 𝐼𝐶(𝑤456)

𝐽𝐶(𝑤",𝑤%) = 𝐼𝐶(𝑤") + 𝐼𝐶(𝑤%) − 2 ∗ 𝐼𝐶(𝑤456) 𝐿𝑖𝑛(𝑤",𝑤%) =

𝐼𝐶(𝑤456)𝐼𝐶(𝑤") + 𝐼𝐶(𝑤%)

IC(w*): the probabilities of all the subsumers leading up to the given word

P(w*): The probability of the given word appearing in the selected corpus

Hausman 18


Pedersen, An Adapted Lesk Algorithm for Word Sense Disambiguation Using WordNet, 2002). First they

ignore any word that they deem a non‐content word, such as “of” or “the.” Then they expand the

number of words to compare by looking up the glosses of words found through several semantic

relations. These semantic relations are: synonym, hypernym, hyponym, holonym, meronym, troponym,

and attribute. Gloss‐based methods depend on the overlap between the glosses of two words.

2.4 Vector Based Methods

Vector‐based methods take each individual measurement and represent it as a vector. The

cosine of the angle between the vectors is an indication of how related the two concepts are. If the

angle is really large, then the two measurements are not very related. If the angle is very small, then the

two measurements are related. The best example given by Zesch (Zesch & Gurevich, 2010) is from

Patwardahan et al. (Patwardhan, Using WordNet‐Based Context Vectors to Estimate the Semantic

Relatedness of Concepts, 2006). They start with two words and find all of the glosses that match using

the Adapted Lesk Algorithm (Patwardhan, Banerjee, & Pedersen, An Adapted Lesk Algorithm for Word

Sense Disambiguation Using WordNet, 2002). Then they say that each word is a dimension. Each vector

is a gloss of the one of the relations from either word. The angle of the vector is the number of words

that overlap in the glosses of the Adapted Lesk Algorithm. The strength of this concept is that every

relation turns into a vector. Since every relation is a vector, it is easy to compare two words. However,

turning everything into a vector is a problematic task.

2.5 Using Multiple Approaches

Most of the preceding examples from early researchers have the same thought, especially in the

development of path‐based and information‐based methods. For each word, compare the word with

the surrounding words within one equation. The sense with the highest score is selected. Many of the

later researchers start to attempt using several approaches. The example cited here precedes the

Hausman 19


Adapted Lesk Algorithm (Patwardhan, Banerjee, & Pedersen, An Adapted Lesk Algorithm for Word Sense

Disambiguation Using WordNet, 2002). This method starts with Lesk and expands on it by using several

semantic relations. One of these semantic relations, hypernyms, is the main relation in the path and

information examples. Patwardahan et al. even tried to combine Adapted Lesk with the earlier

information‐based methods (Patwardhan, Banerjee, & Pedersen, Using Measures of Semantic

Relatedness for Word Sense Disambiguation, 2003). They found that the equation from Jiang and

Conrath worked best in their situation (Jiang & Conrath, 1997). Then they moved on to the vector‐

based method described above, which starts with Adapted Lesk. All of these examples yield better

results with the combination of approaches.

Basile et al. realize that every part of speech has different relationships with other parts of

speech (Basile, Degemmis, Gentile, Lops, & Semeraro, 2007). A changing a noun has a different effect

on another noun than on a verb for a given relation. Also, one approach may work better on a specific

part of speech. With this in mind, Basile et al. use a different approach depending on the part of speech.

Nouns use a modified Leacock measure as a starting point (Leacock & Chodorow, 1998). Their

algorithm uses an extra Gaussian factor for the distance between the words in the disambiguation text

and a factor for the frequency in WordNet. Verbs use a similar approach, except with a different

Gaussian factor. Adjectives and Adverbs use the Adapted Lesk Algorithm (Patwardhan, Banerjee, &

Pedersen, An Adapted Lesk Algorithm for Word Sense Disambiguation Using WordNet, 2002). The

underlying concept of this application is that using a different approach for every part of speech can

improve the overall score.

2.6 A Genetic Algorithm Approach

This paper employs a genetic algorithm, and therefore it follows to investigate other proposed

genetic algorithm approaches as well. The genetic algorithm approach for word sense disambiguation

Hausman 20


using the Zhang et al. formula is the conceptual base for this paper [15]. Zhang and fellow researchers

focus only upon nouns, but their approach can easily apply to other parts of speech.

This paper also focuses on the most important part of a genetic algorithm: the cost function.

The cost function determines how the genetic algorithm compares solutions. If a genetic algorithm

cannot compare solutions correctly, then the rest of the algorithm is irrelevant. Zhang explains the core

of the algorithm in two sentences. These two sentences prove that the approach is extremely simple

beyond the cost function. The cost function begins with the hypernym tree equation from Wu and

Palmer (Wu & Palmer, 1994). This equation is an input to an equation that accounts for the domain of a

word. The domain is the word or collection to which a word belongs. After this, the results are

weighted based on the frequency of the word. The frequency comes from statistics found in WordNet

(Princeton University, 2010). The significance of this cost function is that it relies on several semantic

relations at the same time.

Equation 3: Cost function for Zhang’s solution

𝐶𝑜𝑠𝑡(𝑤",𝑤%)= ;𝑆𝑒𝑛𝑠𝑒𝐶𝑛𝑡(𝑤")𝑇𝑜𝑡𝑎𝑙𝐶𝑛𝑡(𝑤")

+ 𝑆𝑒𝑛𝑠𝑒𝐶𝑛𝑡(𝑤%)𝑇𝑜𝑡𝑎𝑙𝐶𝑛𝑡(𝑤%)<

∗ =1 +𝑊𝑃(𝑤",𝑤%)

2 , 𝐷𝑜𝑚(𝑤") = 𝐷𝑜𝑚(𝑤%)𝑊𝑃(𝑤",𝑤%)

2 , 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

SenseCnt(w): The number of times this was referenced in WordNet corpus TotalCnt(w): The total number of references for this word in WordNet corpus SenseTotal(w): The total number of senses for this word

Sense(w): The sense number of the word currently in use Dom(w*): the domain of the word given

Hausman 21


2.7 Main Ideas behind the Approach in this Project

There are a variety of ways to solve word sense disambiguation. To combine every possible

approach is not the best option. However, this author has chosen to investigate examples from various

sources. Zhang’s idea of using several semantic relations in a genetic algorithm is very appealing

(Zhang, Zhou, & Martin, 2008). However, it may make more sense to use more semantic relations. It

also may help to examine each relation independently before combining them into a large cost function

equation. Many of those semantic relations should probably begin with equations that other

researchers have developed, such as the hypernym equation from Wu and Palmer (Wu & Palmer,

1994). It is possible that some of these relations correlate better than others, so adjusting for this may

make a more accurate solution. Basile’s idea of separating the parts of speech also seems cogent

(Basile, Degemmis, Gentile, Lops, & Semeraro, 2007). For example, there are only hypernyms of nouns

and verbs. It is for this reason that Zhang only looks at nouns in their solution. The melding of the

preceding concepts provides the starting point of this project.

Hausman 22


Chapter 3: Tools/Resources

This project uses a variety of tools and resources that are referenced throughout the project.

The tools and resources include: WordNet, a C# interface to WordNet, a part of speech tagger from

Tokyo University, SemCor, and SemEval. Some of these tools, WordNet for example, provide all the

definitions and relations. Some resources, like SemCor, provide examples of correctly translated text.

The sections below explain what each tool/resource is and how this project uses them.

3.1 WordNet

WordNet is a publicly available lexical database developed by Princeton University (Miller). It

only defines nouns, verbs, adjectives, and adverbs. There are 206941 words across 117659 SynSets,

which are groups of synonyms, in WordNet 3.0. This means that there are 117659 unique definitions

available. Each of these SynSets relates to other SynSets in various ways, hence the “net” in WordNet.

This project uses many of these relations and definitions to disambiguate text. These relations include

antonyms, hypernyms, synonyms, meronyms, holonyms, domain, coordinate sisters, polysemy count,

and several others. All of these relations about words are why WordNet is a lexical database. WordNet

is so rich in information and so well executed that it is one of the most common tools for word sense

disambiguation.

3.2 WordNet Interface

There are a few options available in order to use WordNet. One option is to use an API for the

online version of WordNet (Princeton University, 2010). This requires constant internet access and all

the aforementioned resources to use the available version of WordNet. A second option is to use the

Hausman 23


binary that comes with the install of WordNet. The program would have to run the executable and

parse the output with regular expressions. This is possible, but it is also very labor intensive. A third

option is to parse the data files that the WordNet executable does. There is documentation provided

with the WordNet installation explaining how those files operate. The WordNet website also provides

links to APIs that already parse the files in several different programming languages. Simply pick the

best API for the preferred language.

This project is implemented in C#. Matt Gerber of Michigan State University writes the best

WordNet API for C# (Gerber, 2010). What sets this API apart is the option to either store the WordNet

database in memory or on disk. The in‐memory option is very fast, but uses ~200 MB of memory.

However, there are drawbacks to this API. There is no morphological logic to extract the base word, or

lemma, from a given word. To do this, the morph.c source file from the WordNet installation was

translated into C#. There is also no access to the polysemy file that WordNet uses. This file lists the

number of times every sense was referenced in the corpus used to create WordNet. Other than those

two flaws, the API works well.

3.3 Part of Speech Tagger

Some words occur as several parts of speech. For example, the word “blue” can be a noun,

verb, or adjective. Each of these parts of speech has several senses, or definitions. The only way to tell

the difference is to look at the context around the word. Knowing the part of speech would dramatically

decrease the number of senses that are necessary, and it simplifies the problem. In addition, many

researchers and competitions purposely focus on a single part of speech. Therefore, having an accurate

part of speech tagger is very useful in word sense disambiguation. The author has selected a part of

speech tagger from The University of Tokyo (Tsuruoka & Tsujii, 2005). This part of speech tagger is fast

(2400 tokens/sec) and very accurate (97.10% on WSJ corpus).

Hausman 24


3.4 SemCor

Princeton University developed Semcor, which originates from the Brown Corpus (Princeton

University, 2011). The Brown Corpus is one of the original tagged collections of text for creating a

dictionary. Many dictionaries today use larger and broader corpora. The SemCor files contain over

20,000 tagged words across 352 files. The appendix contains a short table indicating what kinds of text

sources SemCor tags. 186 of these files contain tagged nouns, verbs, adjectives, and adverbs. The

remaining 166 files only tag verbs. Every tag contains the part of speech, the lemma, and the correct

WordNet sense. This makes SemCor extremely useful for researchers using WordNet. SemCor provides

a professionally tagged resource to compare the accuracies of word sense disambiguation algorithms.

3.5 SemEval

SemEval is a language processing competition that has occurred approximately every three

years since 1998. Each competition requires several “tasks” in which participants can compete. These

tasks range from translating text to word sense disambiguation in many languages. This project focuses

on the “all words” task. The goal of the all words task is to assign the correct sense to every given word

in the supplied text. Investigating the work done in these competitions is an optimal way to compare

the results of the author’s project with other techniques.

Each SemEval competition has a slightly different focus and goal. The first SemEval competition

took place in 1998 and set the stage for the competition (ACL‐SIGLEX, 2011). The original goal was to

evaluate which senses and algorithms work best. This original competition used Hector as the reference

dictionary. The second SemEval competition took place in 2001. This competition separated the

competition into tasks, including the all‐words task, and introduced several languages as separate tasks.

This is the first SemEval competition using WordNet, which was version 1.7 at the time. The third

SemEval competition took place in 2004. There was the addition of several tasks on top of the tasks

Hausman 25


available in SemEval 2. The reference dictionary was WordNet 1.7.1. The fourth SemEval competition

took place in 2007. The sponsors took a critical look at the previous two competitions and realized that

WordNet is too detailed. Many of the senses for a word are so subtle that even humans have a hard

time choosing the correct sense. This is why the best algorithms in the second and third competitions

had a worse accuracy than during the first competition. Therefore, the sponsors took the WordNet 2.1

definitions and grouped similar senses together. When evaluating any of the tasks, any sense in the

correct group was correct as a “coarse” result. The fifth SemEval competition took place in 2010. This

time the goal is to use the same algorithm across several languages. Of course, this means the reference

dictionary was different for every language. To account for this, a common representation of all the

various WordNet resources was created. This “WordNet Lexical Markup Framework” was provided for

in the trial data for the all words task. The tasks and goals of the SemEval competition have definitely

evolved and changed over 12 years.

3.6 OntoNotes

The SemEval 2010 competition also saw a different English reference dictionary for some of the

tasks. Many of the tasks still use WordNet, but some started to use OntoNotes. OntoNotes is a

collaborative effort between Raytheon BBN Technologies, the University of Colorado, the University of

Pennsylvania, and the University of Southern California's Information Sciences Institute (Raytheon BBN

Technologies, the University of Colorado, the University of Pennsylvania, and the University of Southern

California's Information Sciences Institute , 2011). The idea is to make a resource as available as

WordNet, but more accurate and useful for language processing. The past decade has shown several

weaknesses and lessons from WordNet, so the goal of OntoNotes is to overcome them. OntoNotes

originally started with the WordNet glosses and groups them together like in SemEval 2007. Many of

the semantic relations are still in development as a “PropBank” component and should be a much richer

resource according to the advertisements. Currently, it is possible to translate between WordNet and

Hausman 26


OntoNotes for research purposes (Zhong, Ng, & Chan, 2008) (Patwardhan, Banerjee, & Pedersen,

UMND1: Unsupervised Word Sense Disambiguation Using Contextual Semantic Relatedness, 2007).

Keep in mind that OntoNotes is slowly evolving to a different resource and may be just as useful in the

future. This author’s project does not use OntoNotes, but moving towards OntoNotes may make the

approach in the project much more successful at a later time.

Hausman 27


Chapter 4: Introduction to Semantic Relations

A semantic relation uses some aspect of a word to compare two words. There are several

possible semantic relations since there are several aspects to a word. Many early researchers focus on a

single semantic relation. The all words tasks in the SemEval competitions use the “most common” sense

as their baseline (ACL‐SIGLEX, 2011). Wu and Palmer look at hypernyms (Wu & Palmer, 1994). Many

later researchers use more than one semantic relation since each relation has its strengths and

weaknesses. For example, Zhang uses frequency, hypernyms, and domain to disambiguate nouns

(Zhang, Zhou, & Martin, 2008). Many researchers found that they achieved better results because they

used several different relations.

In this project, all the semantic relation information comes from WordNet. The algorithm then

calculates a number using an equation specific to that relation. For example, the equation comparing

frequency would be different than an equation comparing synonyms. They may be similar, but the

details of the equation are slightly different. Also, the equation could be different for each part of

speech. A noun will have different similarities to a second noun than it would to a verb. In this project,

there are six semantic relations: frequency, hyponyms, coordinate sisters, domain, antonyms, and

synonyms.

4.1 Frequency

Frequency is how often a definition appears in context compared to other words. Most

definitions in dictionaries are in order. The most “frequent” or “common” definition is the first sense.

The least frequent definition is the last sense. The question is how to take advantage of this

information. As mentioned before, the baseline for many word sense disambiguation projects, including

Hausman 28


SenseEval, is to compare the performance to an answer using the first sense for all words (ACL‐SIGLEX,

2011). Many methods use the most common sense when other relations don’t give a clear answer.

Frequency is a very common and useful comparison.

Conceptually, there are several possible ways to calculate the Frequency for two words. The

developers of WordNet kept track of how many times a sense was used in the corpus used to develop

WordNet. As expected, the sense with the most occurrences is the first definition. The number of

occurrences for each sense is available to the users of WordNet. Some researchers, including this

project, use the number of occurrences of a sense and divide it by the total number of occurrences of

that word (Zhang, Zhou, & Martin, 2008). However, not all words appear in the WordNet corpus. To

account for these exceptions, a linear distribution is applied. The equations are below.

Equation 4: Semantic Relation Equation for Frequency

Notice that frequency is easily applied to a single sense of a word. If the word is “frequency”, as

below, the value for sense 1 is 0.48. However, a semantic relation compares two words, not just one.

To account for this, one adds the calculated frequency of the two words. If the first word is the first

sense of frequency and the second word is the second sense of frequency, then the calculated

frequency is 0.48+0.44=0.92. In this fashion, the genetic algorithm uses frequency in the same manner

as any of the other semantic relations.

𝐹𝑟𝑒𝑞(𝑤) =

⎩⎪⎨⎪⎧

𝑆𝑒𝑛𝑠𝑒𝐶𝑛𝑡(𝑤)𝑇𝑜𝑡𝑎𝑙𝐶𝑛𝑡(𝑤) , 𝑇𝑜𝑡𝑎𝑙𝐶𝑛𝑡(𝑤) > 0

𝑆𝑒𝑛𝑠𝑒𝑇𝑜𝑡𝑎𝑙(𝑤) − 𝑆𝑒𝑛𝑠𝑒(𝑤) − 1𝑆𝑒𝑛𝑠𝑒𝑇𝑜𝑡𝑎𝑙(𝑤) , 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

SenseCnt(w): The number of times this was referenced in WordNet corpus TotalCnt(w): The total number of references for this word in WordNet corpus

SenseTotal(w): The total number of senses for this word

Sense(w): The sense number of the word currently in use

Hausman 29


Example 3: Example using the Frequency Semantic Relation Equation

4.2 Hypernym

A hypernym is a more generic way of saying a word. For example, a more generic way of saying

man would be human. A more generic way of saying human would be organism. Note that some words

have a different hypernym depending on definition. A “bank” next to the river has a different hypernym

than a financial “bank.” Finding the hypernym of every sense creates a tree‐like structure of hypernym

relations. Many researchers take advantage of this tree in many different ways. This project uses Wu

and Palmer (Wu & Palmer, 1994).

Wu and Palmer account for several aspects of the hypernym tree. When comparing two words,

the point where the hypernyms of both words match is the most significant ordinate or lowest common

subsumer. The depth to this point indicates how specific the two words are. The distance between the

two words in the tree indicates how similar the two words are. Wu and Palmer accounts for both so

that the similarity of two words is just as important as how specific the two words are.

1. (12) frequency, frequence, oftenness ‐‐ (the number of occurrences within a given time period)

a. 12/25 = 0.48 2. (11) frequency, relative frequency ‐‐ (the ratio of the number of

observations in a statistical category to the total number of observations)

a. 11/25 = 0.44 3. (2) frequency, absolute frequency ‐‐ (the number of observations in a

given statistical category) a. 2/25 = 0.08

Hausman 30


Equation 5 and Example 4: Semantic Relation Equation and Example for Hypernyms

In the example above, there are three base words: boy, girl, and animal. The distance between

the words boy and girl is 4. The distance to the most significant ordinate of these two words, person, is

5. This makes the hypernym score between these two words 0.714. Note that comparing animal and

boy is 0.666, which is lower than the comparison of boy and girl. Animal and boy may be just as similar,

but they are more generic. Wu Palmer accounts for both the similarity of two words and how similar

they are.

4.3 Coordinate Sisters

Two words that have the same hypernym are coordinate sisters. Looking at these separately

could provide different and potentially useful insight than when only using Wu and Palmer strategies.

This gives extra emphasis on closely related terms whether they are generic terms or not.

The equation for coordinate sisters is simple. Either the two words are coordinate sisters, or

they are not. Therefore, the equation returns 1 when they are coordinate sisters and 0 when they are

not. In the example tree below, male and female are coordinate sisters. Boy and animal are not

coordinate sisters.

entity

3 hypernyms…

organism

person

male female

boy girl

MSO

D

animal

B A

𝐻𝑦𝑝(𝑤", 𝑤%) =2 × 𝐷

A + 𝐵 + 2 × 𝐷

𝐻𝑦𝑝(𝑏𝑜𝑦,𝑔𝑖𝑟𝑙) = 2 × 52 + 2 + 2 × 5

𝐻𝑦𝑝(𝑏𝑜𝑦,𝑔𝑖𝑟𝑙) ≈ 0.714

MSO: most specific common

hypernym

D: length from the root to MSO

A: length from w1 to MSO

B: length from w2 to MSO

Hausman 31


Equation 6 and Example 5: Semantic Relation Equation and Example for Coordinate Sisters

4.4 Domain

Domain is the word or collection a word belongs to. For example, several programming and

computer terms belong to the domain computer science. Several terms describing plants and gardening

belong to the domain botany. This can make a big difference when looking at senses. The idea is that

some text describing something in a domain will use several words from a domain in the descriptions. If

the domain of the text is around computer science, then a sense describing a computer disk drive is the

most likely to be the correct sense for disk. Very domain specific text can take advantage of this

information, which is why SemEval 2010 has a domain specific task. Domain can make a big difference

in some situations.

Equation 7 and Example 6: Semantic Relation Equation and Example for Domain

entity

3 hypernyms…

organism

person

male female

boy girl

animal

𝐶𝑜𝑜𝑟(𝑤",𝑤%) = G1,𝐶𝑜𝑜𝑟(𝑤") ∋ 𝑤%0,𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 w1 and w2: the two words being compared Coor(w*): all of the coordinate sisters of the

word Coor(w1) ∋ w1: if w2 is any of the coordinate

sisters of w1

Computer Science

Buffer, drive, cache, program, software

Sports

Skate, backpack, ball, foul, snorkel

𝐷𝑜𝑚(𝑤",𝑤%)= G1, 𝐷𝑜𝑚(𝑤") = 𝐷𝑜𝑚(𝑤%)

0, 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 w1 and w2: the two words being

compared

Dom(w*): the domain of the word given

Hausman 32


The calculation for domain is very simple. It is either part of the domain, or it is not. There are

no sub‐domains or generic domains available in WordNet. Therefore, the equation for domain below

returns a 1 if the words are in the same domain, otherwise it returns a 0. For example, comparing

computer and cache returns 1. Comparing computer and baseball will return 0.

4.5 Synonym A word is a synonym of second word if one can interchange the words without changing the

meaning of the sentence. A different way of defining synonyms is to say that two words are synonyms if

they have the same meaning. WordNet takes advantage of this fact by grouping all synonyms into a

SynSet (Synonym Set). Any word in the SynSet returns the same exact meaning. Even though two

words have the same meaning, people will interchange synonyms because one synonym “sounds”

better. Sometimes people simply want a little variety in their sentences and pull out their thesaurus.

Whatever the reason is, comparing the synonyms of senses could provide insight into which sense is

correct.

Every word is part of a SynSet, which is a synonym that has the exact same meaning. Just like a

thesaurus, WordNet also keeps track of SynSets that have similar meanings. If two words are Synonyms

of each other, exact or similar, then the equation below returns 1. Otherwise, the equation below

returns 0.

Hausman 33


Equation 8 and Example 7: Semantic Relation Equation and Example for Synonyms

4.6 Antonym

An antonym has the opposite meaning. Anything that is an opposite is an antonym. Some

examples are: black vs. white, angel vs. demon, good vs. evil, vague vs. clear, antonym vs. synonym, et‐

cetera. This would work well for comparison papers. One item did this while the other paper did the

opposite. Many researchers who look at synonyms often look at antonyms because these same

comparison papers also describe any similarities.

As with many of the other semantic relations, two words are either antonyms of each other or

are not. Therefore, the equation below returns the value 1 if two words are antonyms. Otherwise, the

equation returns 0.

1. computer, computing machine, computing device, data processor, electronic computer, information processing system

2. sport, athletics 3. frolic, lark, rollick, skylark,

disport, sport, cavort, gambol, frisk, romp, run around, lark about

𝑆𝑦𝑛(𝑤",𝑤%) = G1, 𝑆𝑦𝑛(𝑤") ∋ 𝑤%0, 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 w1 and w2: the two words being

compared Syn(w*): all of the synonyms of the

word Syn(w1) ∋ w2: if w2 is any of the

synonyms of w1

Hausman 34


Equation 9 and Example 8: Semantic Relation Equation and Example for Antonyms

1. good, goodness a. evil, evilness

2. pure a. defiled, maculate

3. vague a. clear, distinct

𝐴𝑛𝑡(𝑤",𝑤%) = G1, 𝐴𝑛𝑡(𝑤") ∋ 𝑤%0, 𝑂𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 w1 and w2: the two words being

compared Ant(w*): all of the antonyms of the

word Ant(w1) ∋ w2: if w2 is any of the

antonyms of w1

Hausman 35


Chapter 5: A Word Sense Disambiguation Genetic Algorithm

Word Sense Disambiguation for all words in a file creates a large solution space. For example,

suppose that there exists a file with five paragraphs of information. Each of these paragraphs could

have 100 words, which means the file has 500 words. Each word can have many definitions, so assume

there are five definitions per word. This makes 5100 = 7.888 * 1069 possible word sense combinations for

a paragraph or 5500 = 3.055*10349 word sense combinations for the entire file. This solution space is far

too large to check every word sense combination. If there is a way to measure the accuracy of a word

sense combination, genetic algorithms provide a way to check a subset of these solutions in order to

obtain a good solution. This solution will not be “the solution,” but the solution should be close.

The genetic algorithm is based on Darwin’s theory of evolution to solve optimization problems.

The general idea is to start with a set of solutions, using that set of solutions to make better solutions,

and only keeping the best solutions. In genetic algorithms, chromosomes define all of the information

necessary to define a solution. The individual pieces of the chromosome are genes that define a certain

aspect of the solution. In this case, the chromosome defines the word senses for the file. Each gene

represents the chosen word sense for a single word. The generic process of creating “better”

chromosomes, or solutions, with genetic algorithms is below.

1. Start with a set of solutions (1st Generation) 2. Take original “parent” solutions and combine them with each other to create a new set of

“child” solutions (Mating) 3. Introduce some random changes in case solutions are “stuck” or are all the same (Mutation) 4. Somehow measure the solutions (Cost Function) to evaluate the best solution 5. Repeat starting with Step 2

Hausman 36


Genetic Algorithm implementations can vary in many respects. The preceding steps describe

the most common case at a very high level. Most genetic algorithms have three main parts. They are:

mating, mutation, and cost function. The preceding steps above do not describe how to implement

mating, mutation, and cost function. The implementation of these portions has different strengths and

weaknesses depending upon the problem. It also does not describe how to select chromosomes for

each step. One should expect the implementation to change for every problem, and most authors will

have a cost function, a mating phase, and a mutation phase as part of their genetic algorithm.

5.1 Evolution of the Cost Function

The cost function is possibly the most important part of a genetic algorithm. It is the portion of

the algorithm that determines which solution is the “correct” or “better” solution. How to determine

the best answer depends upon the problem and on the approach. If the cost function’s approach

incorrectly compares solutions, then the genetic algorithm will lead to an increasingly incorrect answer.

For example, if the problem is to find the highest point on a mountain, the cost function needs to

correctly determine which point is higher in elevation. Assume that the cost function compares two

areas according to the slope of the hillside under the assumption that the slope on the top of the

mountain is zero. In this case, the genetic algorithm will not be able to correctly compare two

mountains or plateaus on the map. In addition, it does not matter how well the rest of the genetic

algorithm functions because the results will most likely be incorrect. If the genetic algorithm is to find a

“good” answer, then the cost function must be able to determine which answers are correct in most

cases.

For this project, the cost function must be able to compare the definitions of a word and

determine which definition is most likely to be correct. All of the preceding semantic relation equations

are different measurements that can be part of the cost function. Each of these measurements has its

strengths and weaknesses. A large part of this author’s project involves examining these measurements

Hausman 37


and determining how to use them most accurately. The sections below begin by examining each

semantic relation. The author then incorporates them into a cost function and analyzes the results. This

leads to several variations of the cost function using the knowledge and results of the last cost function.

The last cost function is the one the genetic algorithm uses.

5.1.1 Notes on Sampling

Earlier in this report, the author explored SemCor and WordNet. The SemCor files list the

correct WordNet senses for the given text. Since this project needs to compare incorrect senses with

correct senses, there is a need for many other possible sense combinations. Throughout this project, a

solution is a specific selection of sense combinations. The number of senses that match the senses in

the SemCor file indicate how “good” the solution is. With that in mind, the SemCor selection is the best

solution, and the worst solution matches none of the SemCor senses. The author has noted a variety of

solutions and several levels of correctness.

The problem surfaces when examining the number of possibilities. Each SemCor file contains

approximately 2000 words. Of these, 1100 to 1300 words have a WordNet definition. These definitions

may be a noun, verb, adjective, or adverb. Assuming that the part of speech is already identified for

each word, each of these defined words has the following average number of definitions: noun 1.24,

verb 2.17, adjective 1.4, and adverb 1.25. If every defined word were a noun, the number would be

1.241200 = 1.2765*10112 possible combinations per SemCor file. If there were an equal distribution for

each part of speech, this would be 1.5151200 = 3.1272*10216 possible combinations per SemCor file. This

is an inordinate number of combinations. To further clarify this, imagine that if a solution were

calculated every nanosecond; it would require 9.9164*10199 years to investigate each combination. It

follows that a subset of solutions must represent all of the solutions. The selected subset areas are:

1. Near the correct solution 2. Randomly picked solutions 3. Near the most frequent senses

Hausman 38


4. Random solutions weighted towards the most frequent senses 5. Min/Max areas for each semantic relation via a Mutation method

The author has selected the five primary areas to represent all of the possible solutions. Each

area is a starting point. Every solution begins in one area and randomly changes a percentage of the

word senses toward another area. The first area is near the correct solution. These answers give insight

regarding how to reach the correct solution. The second area contains randomly selected senses to

provide variety. The third area is located near the most frequent senses since the most frequent sense

is a common baseline. The fourth area contains “weighted” randomly selected solutions. This provides

answers between the randomly selected area and the most frequent area. The weighted random

equation and an example are formulated below. The last area describes the boundaries of each

individual semantic relation. A function that eventually became one of the mutation functions defines

these boundaries by building a solution to match the given value for a semantic relation. (See the

mutation section for further explanation.) Using 3000 solutions and using these five areas gives variety

and focus to address most of the cost function behavior.

𝑊𝑒𝑖𝑔ℎ𝑡𝑒𝑑𝑃𝑟𝑜𝑏(𝑤, 𝑠) = 𝑇𝑜𝑡𝑎𝑙𝑆𝑒𝑛𝑠𝑒(𝑤) − 𝑠 + 1∑ 𝑠FGH1I6JKLJ(M)

𝑊𝑒𝑖𝑔ℎ𝑡𝑒𝑑𝑃𝑟𝑜𝑏(𝑤, 1) = 4 − 1 + 11 + 2 + 3 + 4

𝑊𝑒𝑖𝑔ℎ𝑡𝑒𝑑𝑃𝑟𝑜𝑏(𝑤, 1) = 0.4

w and s: the word and sense TotalSense(w): the number of senses for this word

Equation 10: The Weighted Probability Equation

If the word has 4 senses, the probability of returning the 1st sense is:

Example 9: An example using the Weighted Probability Equation

Hausman 39


5.1.2 Semantic Relation Investigation

Studying semantic relations behavior is the first step to making the cost function. Each semantic

relation is a small part of the cost function. The behavior of the semantic relations in this section has a

major influence on the cost function variations in the following sections. Therefore, the coverage and

representation for each semantic relation is crucial to the process. The variety and boundaries of the

semantic relation indicate where and how each semantic relation influences the solution. This includes

the parts of speech which the semantic relation represents. To illustrate, hypernyms are available solely

for nouns and verbs in WordNet, and therefore the relation is useless when comparing adjectives and

adverbs. In addition, the hypernym trees for nouns do not overlap with hypernym trees for verbs. For

this reason the results for every part of speech combination is unique within each semantic relation.

The cost function can then account for each semantic relation and part of speech combination

separately.

A second thing to note is that each semantic relation compares two words in order to return a

value. There are several possible word pair combinations in the given text. The challenge is to

determine which combinations are important. Various researchers try different techniques. Zhang finds

every word combination in a paragraph (Zhang, Zhou, & Martin, 2008). The idea is that a single

sentence does not contain enough information to properly determine the correct word sense with

semantic relations. A paragraph would have more information because it has several related sentences.

Zhang’s solution contains one drawback. Very large paragraphs have an extremely large number of

combination pairs. In addition, the SemEval competitions do not indicate where the paragraph

boundaries are located. The competitors in many recent SemEval competitions use a sliding window

technique for this reason (Patwardhan, Banerjee, & Pedersen, UMND1: Unsupervised Word Sense

Disambiguation Using Contextual Semantic Relatedness, 2007) (Bosch, Hoste, Daelemans, & Den, 2004)

(Mihalcea & Csomai, 2005). The first step is to remove all the words that don’t have definitions in

Hausman 40


WordNet. Then use all the combinations “n” words before and after the target word. The number of

words before and after varies from three words to five words in many papers. This project checks for

words ten words before and after the target word. This is a compromise between a paragraph and the

sliding window technique. Each word has a window with 45 possible combinations while a paragraph

can easily have thousands of possible combinations. Note that each combination uses the semantic

relation equations below to return a value. The overall semantic relation behavior is the average of all

the individual values.

One of the most transparent ways to view behavior is by means of a graph. With graphs of the

semantic relation, it is easier to see the estimated solution space of the relation, the boundaries of the

relation results, the variety of solutions that give a specific result from a semantic relation equation, and

the coverage. Therefore, a scatter plot with 3000 different solutions is available for every semantic

relation. These graphs have the semantic relation as the independent variable. The accuracy of a

solution is the independent variable. In several graphs, each of the individual sample areas is visible.

The graph below uses frequency and indicates the location of five sample areas. The sections below do

not show the individual areas. However, all these areas are very important for finding the boundaries of

each semantic relation.

Hausman 41


Figure 1: An example graph indicating the various sample areas

5.1.2.1 Frequency Behavior

The graphs for two part of speech combinations for frequency are included below. All part of

speech combinations have approximately the same shape. The lowest frequencies have very low

accuracy, but more than zero. This implies that some words only have one definition or use the least

frequent definition. The highest frequencies have ~75% accuracy. This solution would be the typical

“baseline” solution recognized by many authors. The area between them contains a distinct point

where a small range of frequencies contain “good” solutions. However, this small range of frequencies

also contains the widest range of accuracies. This means that it will take more than frequency to sort

out the subset of solutions in this range. The five sample areas are clear and appear to cover several

boundaries of the semantic relation. All in all, frequency looks like a promising semantic relation as an

individual measurement.

1) Correct Area

5) Min Mutation Area

4) Weighted Random Area

2) Random Area

3) Most Frequent Area

5) Max Mutation Area

Average Solution Value

Hausman 42


Figure 2: The graph of the frequency semantic relation using a noun‐noun part of speech combination

Figure 3: The graph of the frequency semantic relation using an adjective‐adjective part of speech combination

Distinct Maximum

Large Range at Correct Solution

Distinct Maximum

Large Range at Correct Solution

Hausman 43


5.1.2.2 Hypernym Behavior

There are only two part of speech combinations that work for hypernyms: comparing two nouns

or comparing two verbs. In both cases the sample areas are not as distinct. The noun‐noun

combinations appear to have more solutions on the left side of the correct solution. This may explain

why many researchers who use hypernyms can achieve greater accuracy by maximizing hypernyms

scores. The bulk of the solutions have a lower hypernym score and other researchers don’t access these

exceptions very often. In either case, the noun‐noun hypernym graph has a distinct point around the

correct solution near the middle of the range. The verb‐verb hypernym graph does not display the same

distinct point that the noun‐noun hypernym displays. It appears to be inversely proportional to

accuracy. This implies that actions are not as related as the subjects. Both graphs indicate that there is

a relation between hypernyms and accuracy. Both graphs also indicate that the score around the

correct solution has the widest range of accuracies. Of the two graphs, the noun‐noun version looks

more useful.

Hausman 44


Figure 4: The graph of the hypernym semantic relation using a noun‐noun part of speech combination

Figure 5: The graph of the hypernym semantic relation using a verb‐verb part of speech combination

Distinct Maximum

Very Large and Dense Range at Correct Solution

Distinct Maximum

Very Large and Very Dense Range at Correct Solution

Hausman 45


5.1.2.3 Coordinate Sister Behavior

Both coordinate sisters and hypernyms rely on the same hypernym tree, so both of them also

have the same part of speech limitations. Coordinate sisters can only compare two nouns or two verbs.

In this case, the graphs are somewhat similar. The noun‐noun combination has a distinct point around

the correct solution in the middle of the coordinate sister range. The verb‐verb version has the distinct

point near the beginning of the range. The verb‐verb graph indicates the correct solution is not as close

to zero as the hypernym graphs. Both graphs show that the area around the solution has a majority of

the solutions and the widest range of accuracies. Overall, the coordinate sister results are slightly

different than the hypernym results, which may prove useful later on.

Figure 6: The graph of the coordinate sister semantic relation using a noun‐noun part of speech combination

Distinct Maximum Very Large and Very

Dense Range at Correct Solution

Hausman 46


Figure 7: The graph of the coordinate sister semantic relation using a verb‐verb part of speech combination

5.1.2.4 Domain Behavior

Few words have a domain in WordNet. This fact is evident in the graphs for domain because the

possible values show up as columns. Perhaps a different SemCor file that is more domain specific may

have a larger variety of results, but br‐g23 only has a few. A majority of the given solutions have an

average domain value below 0.001. In addition, the correct solution is not unique among the given

solutions. This makes it very hard to see a correlation between domain and accuracy if one indeed

exists. Overall, domain does not look promising at this point. At least it does not look promising for this

SemCor example.

Distinct Maximum

Very Large Range at Correct Solution

Hausman 47


Figure 8: The graph of the domain semantic relation using a noun‐noun part of speech combination

Figure 9: The graph of the domain semantic relation using an adverb‐adverb part of speech combination

Not Very Distinct Maximum




Hausman 48


5.1.2.5 Synonym Behavior

Synonyms show similar issues that domain did, except on a lesser scale. There simply are not

enough synonyms in the br‐g23 SemCor file to provide a variety of results. In the verb‐verb example

below, a large majority of the answers have approximately the same two values. The larger group

happens to contain the correct answer. There is a distinct maximum point, but this is a 50% chance in

the first place. As for the adjective‐adjective example, it does have more than two solutions. However,

it shows very little correlation between synonyms and accuracy. It is unlikely that synonyms will be

useful in the cost function later on.

Figure 10: The graph of the synonym semantic relation using a verb‐verb part of speech combination

Distinct Maximum

Very Small Variation

Hausman 49


Figure 11: The graph of the synonym semantic relation using an adjective‐adjective part of speech combination

5.1.2.6 Antonym Behavior

If synonyms do not look very promising, then antonyms will probably have the same results.

The two are extremely similar in nature, so each of them should have similar strengths and weaknesses.

Antonyms do not have sufficient variety in the results to be reliable. None of the solutions have an

antonym average above 0.005! There is no distinct point near the correct solution. The correlation

between antonyms and accuracy is very hard to determine. Overall, antonyms will perform as badly as

synonyms and will probably not be useful in the cost function.


Very Small Correlation

Hausman 50


Figure 12: The graph of the antonym semantic relation using a noun‐noun part of speech combination

Figure 13: The graph of the antonym semantic relation using an adjective‐adjective part of speech combination

Very Small Variation




Hausman 51


5.1.3 The Optimal Cost Function

At this point there are several semantic relation equations and graphs. The author’s plan is to

use those equations in such a way so that the best solutions have the highest costs while the lower

accuracy solutions have lower costs. The goal of the genetic algorithm is to aim for the highest cost

solution in every generation. Ideally, the highest cost possible will be the correct solution and nothing

but the correct solution. A different way of describing it is, “The accuracy of the solutions is

proportional to the cost of the solution with little or no variance near the maximum solution.” The

scatter plot of the cost functions should resemble the one below.

Figure 14: The graph indicating the shape of the ideal cost function

5.1.4 Cost Function Method 1: Simple Addition

The easiest way to create a cost function is to add the average values of each semantic relation.

This provides a good starting point. It is not, and was never intended to be, the final cost function. It

serves to determine how much the cost function needs to change in order to make an optimal cost

Max Cost is Correct Solution

Very Little Variation in Accuracy near

Max Cost

Hausman 52


function. This provides a baseline to determine whether the future solutions actually improve beyond

the simplest solution. The resulting graph is below.

Figure 15: The graph indicating the shape of Cost Function Method 1

The result is not ideal. The highest cost solution is not the correct solution, but variance at this

highest point is small. The result takes the same shape as the frequency semantic relation. This is not

surprising considering the fact that frequency has the highest average values by a significant amount.

This also means that the answer this cost function converges to is the most frequent sense. Considering

this is the most common baseline, this answer is not an acceptable one.

5.1.5 Cost Function Method 2: Regression

Thorough examination of the semantic relations will reveal three main points. The maximum

score in all the semantic relations is not the correct answer. For example, the maximum frequency

score gives an accuracy of ~75%. If each semantic relation had the maximum point as the correct

solution, then the addition of the semantic relations should be the correct solution. Some semantic

Maximum Cost, ≈75% Accuracy

Correct Answer Not the Maximum Cost

Hausman 53


relations appear to relate to accuracy better than others. Antonyms, for example, do not appear to

have an effect on accuracy. Those semantic relations which do not relate as well have a higher chance

of influencing the cost function to an incorrect solution. The semantic relations vary a little between

different SemCor files. A good cost function would need to account for all of these points in order to

succeed.

One possibility is polynomial regression. Take the semantic relation values and accuracies and

fit a 6th order polynomial to the result. Then use that equation to transform the average semantic

relation value. This would make the maximum value at or close to the correct answer. To account for

the multiple SemCor files, simply make a regression of the regressions. Then this average regression

would account for as many of the SemCor files as possible. Multiply by the R2 value of this average

regression to adjust for semantic relation variance. A step by step example is below.

1. Take several samples for one SemCor file 2. Find the regression for one semantic relation.

Figure 16: A graph with an example regression equation

Hausman 54


3. Find the regression of regressions of that semantic relation from multiple SemCor files.

Figure 17: A graph indicating an example of the regression of multiple regressions for frequency

4. Use the R2 value as a multiplier for the semantic relation 5. Calculate the total weighted cost with the following equation. It adds the transformed semantic

relation value for every word combination.

An examination of the resulting cost function is better, according to the following graph. The

correct solution has a higher cost than the first method, but it still is not the highest. There appear to be

𝑊𝑒𝑖𝑔ℎ𝑡𝑒𝑑𝐶𝑜𝑠𝑡(𝑠)

= J 𝐴𝑣𝑔K J J 𝑆𝑒𝑚𝑅𝑒𝑙𝐸𝑞𝑛(𝑆𝑒𝑚𝑅𝑒𝑙,𝑊𝑜𝑟𝑑",𝑊𝑜𝑟𝑑%)PGQRLPGQRM

N6JSTJI

S: The current solution Word1 and Word2: Every word in the solution SemRel: Every semantic relation SemRelEqn: Use the words in the correct semantic relation equation Avg: Find the average value then transform with the regression equation

Equation 11: The equation for Cost Function Method 2

Hausman 55


several answers that have a higher cost. Also, there is a large variance near the maximum with as low as

65% accuracy. This method remains insufficient.


5.1.6 Cost Function Method 3: Proportional Placement in a Range

The regression for some semantic relations, such as frequency, work fairly well. However,

regression has its weaknesses. Any extraneous points tend to have a large influence on the regression.

The maximum value of the curve does not match the correct answer. In most cases it only varies a

slightly as in Figure 18.

The results between the SemCor files make the difference here. Hypernyms, for example, do

not always have the same range of values for every SemCor file as shown in the graph below. The graph

illustrates some of the files that have the best R2 values for the hypernym noun‐noun regression. The

correct hypernym value for one SemCor file can be the worst answer for another file. In this case, 0.26

is the best answer for br‐g16, and it is the worst answer for br‐e30. This skews the cost function despite

Correct Answer Near Maximum Cost

Maximum Cost, Several Accuracies

Hausman 56


the fact that the original regressions fit better than most of the semantic relations. As a result of this

skewing, one might determine that something other than regression works better for hypernyms.

Figure 19: A graph showing an example of the weaknesses of multiple hypernym regressions

The graph above illustrates that regression between files fails. It also implies a trend about

hypernyms. The ranges change, but the correct answer is always at or near the middle of that range.

For frequency, the correct answer is closer to the maximum value of the range. In the preceding

samples, one of the areas is the boundaries of a semantic relation. The function that locates these

boundaries is already available, so it is possible for the genetic algorithm to find the range. Since

dynamically finding the range is not an issue, an alternate possibility is to determine what proportion in

that range is typically correct. Then an equation based on these proportions will provide a better

transformation than would regression. The steps and example below illustrate how to find these

equations.

1. Take several samples for one SemCor file 2. Record the min, max, and ratio of the correct answer for each Semantic Relation

Correct Solution Varies Same Cost, Opposite Effect Between Files

Hausman 57


3. Find the average and standard deviation of the correct solution ratio for each semantic relation and part of speech combination from multiple SemCor files

4. Use four lines to transform the results. The lines connect five crucial points: the min, the max, the average of the correct solution, and the two points twice the standard deviation away from the average. Note that the genetic algorithm will find the range dynamically. A graphical example is below.

Figure 20: A graph showing an example using Cost Function Method 3

5. Calculate the total cost with the following equation. It adds the transformed semantic relation value for every word combination.

𝑊𝑒𝑖𝑔ℎ𝑡𝑒𝑑𝐶𝑜𝑠𝑡(𝑠)

= J 𝐴𝑣𝑔K J J 𝑆𝑒𝑚𝑅𝑒𝑙𝐸𝑞𝑛(𝑆𝑒𝑚𝑅𝑒𝑙,𝑊𝑜𝑟𝑑",𝑊𝑜𝑟𝑑%)PGQRLPGQRM

N6JSTJI

S: The current solution Word1 and Word2: Every word in the solution SemRel: Every semantic relation SemRelEqn: Use the words in the correct semantic relation equation Avg: Find the average value then transform with the proportional equation

Equation 12: The equation for Cost Function Method 3

(Avg, stdDev2)

(Avg – 2*stdDev, 0.9*stdDev2)

(Min, 0) (Max, 0)

(Avg + 2*stdDev, 0.9*stdDev2)

Hausman 58


Both regression and this technique find the most likely values that contain the correct solution.

The difference is that this technique takes into account the changing range between the SemCor files.

As such, this method has fewer solutions that have a higher score than the correct solution, as shown

below. Just like regression, however, there is a large variance in accuracies around the highest cost.

More exploration is necessary to eliminate these false positives.


5.1.7 Cost Function Method 4: Add Sense Distribution

The method above is very promising. Its main weakness is that the highest cost can be a variety

of answers, ranging anywhere from a very inaccurate answer to the correct answer. All of these

answers provide average semantic relation scores that are in the optimum ranges. However, the lower

accuracy answers have a different sense distribution than the correct answer. They contain a different

Correct Answer Near the Maximum Cost

Maximum Cost, Several Accuracies

Hausman 59


number of senses that use the first sense and a different number of senses that use the second sense.

The steps below describe how to find and apply these distributions.

1. Take several SemCor files and find the average percentage of each sense 2. Find the error using the following equation. It adds the error for each sense.

3. Multiply the proportional range weighted cost function with this error value

The premise behind this additional measurement is to force high cost solutions to have optimal

semantic relation scores and the proper sense distribution. As a result, many of the answers that did

not have the proper sense distribution have lower scores. This separates a large part of the variation at

the maximum cost function score, as shown below. The correct solution is also close to the maximum

score. Overall, this is much closer to the optimal cost function shape than many of the previous

techniques. This looks promising when one assumes that the subset of examples matches the real

𝐸𝑟𝑟𝑜𝑟(𝑠) = 1 − J 𝐴𝑏𝑠(𝑇𝑜𝑡𝑎𝑙𝐶𝑛𝑡(𝑠𝑒𝑛𝑠𝑒) − 𝑆𝑒𝑛𝑠𝑒𝐶𝑛𝑡(𝑠𝑒𝑛𝑠𝑒))𝑇𝑜𝑡𝑎𝑙𝐶𝑛𝑡(𝑠𝑒𝑛𝑠𝑒)6JKLJ

S: The current solution Sense: The sense number SenseCnt: Find the current number of the given sense TotalCnt: The expected number of the given sense Abs: Absolute value

Equation 13: The equation for sense distribution error

𝑊𝑒𝑖𝑔ℎ𝑡𝑒𝑑𝐶𝑜𝑠𝑡(𝑠) = 𝐸𝑟𝑟𝑜𝑟(𝑆𝑒𝑛𝑠𝑒)

∗ J 𝐴𝑣𝑔K J J 𝑆𝑒𝑚𝑅𝑒𝑙𝐸𝑞𝑛(𝑆𝑒𝑚𝑅𝑒𝑙,𝑊𝑜𝑟𝑑",𝑊𝑜𝑟𝑑%)PGQRLPGQRM

N6JSTJI

S: The current solution Word1 and Word2: Every word in the solution

SemRel: Every semantic relation SemRelEqn: Use the words in the correct semantic relation equation Avg: Find the average value then transform with the proportional

equation Equation 14: The equation for Cost Function Method 4

Hausman 60


solution space. At this point in the project, the author began to investigate the rest of the genetic

algorithm.


5.1.8 Cost Function Method 5: Add Semantic Relation Distribution

Despite what the sample solutions show, the genetic algorithm finds solutions that have better

scores than the correct solution. However, these solutions do not exhibit the same accuracy as the

correct solution. These are select solutions that are not part of the sample space and are “optimal” as

far as the cost function is concerned. The last modification accounts for the sense distribution, which

helps remove unwanted solutions such as these. Applying the same concept on the semantic relation

level may remediate these new solutions. The steps for applying this concept are below.

1. Take several SemCor files and find the average semantic relation value for each sense

Correct Answer Near the Maximum Cost Maximum Cost,

Less Accuracy

Hausman 61


2. Find the error using the following equation. It adds the error for each sense.

3. Multiply the semantic relation by this error when calculating the cost function

The concept of applying the distribution on the semantic relation level helps, but not

significantly. Some experiments indicate that this improves the accuracy by only 1% to 2%, a small

percentage compared with the error. This implies that the semantic relation distribution does not

matter as much or that the rest of the cost function already accounts for most of these solutions. At the

very least, this last modification lacks what is necessary to account for the incorrect solutions which the

genetic algorithm finds.

𝐸𝑟𝑟𝑜𝑟(𝑆𝑒𝑚𝑅𝑒𝑙) = 1 − J 𝐴𝑏𝑠(𝑆𝑒𝑚𝑂𝑝𝑡(𝑠𝑒𝑛𝑠𝑒) − 𝑆𝑒𝑚𝑉𝑎𝑙(𝑠𝑒𝑛𝑠𝑒))

𝑆𝑒𝑚𝑂𝑝𝑡(𝑠𝑒𝑛𝑠𝑒)6JKLJ

SemRel: The current semantic relation Sense: The sense number SemVal: The average semantic relation value for this sense SemOpt: The optimal semantic relation value for this sense Abs: Absolute value

Equation 15: The equation for semantic relation distribution error

𝑊𝑒𝑖𝑔ℎ𝑡𝑒𝑑𝐶𝑜𝑠𝑡(𝑠) = 𝐸𝑟𝑟𝑜𝑟(𝑆𝑒𝑛𝑠𝑒) ∗ J O𝐸𝑟𝑟𝑜𝑟(𝑆𝑒𝑚𝑅𝑒𝑙)6JSTJI

∗ 𝐴𝑣𝑔K J J 𝑆𝑒𝑚𝑅𝑒𝑙𝐸𝑞𝑛(𝑆𝑒𝑚𝑅𝑒𝑙,𝑊𝑜𝑟𝑑",𝑊𝑜𝑟𝑑%)PGQRLPGQRM

NP

S: The current solution Word1 and Word2: Every word in the solution SemRel: Every semantic relation SemRelEqn: Use the words in the correct semantic relation equation Avg: Find the average value then transform with the proportional

equation Equation 16: The equation for Cost Function Method 5

Hausman 62


5.2 Mating

The concept underlying mating is to combine two “parent” solutions somehow and create a

“child” solution. This “child” solution should be similar to the solutions before and can potentially be

better than the parents. Any weaker solutions will eventually “die” and will not mate in the next

solution, as in Darwin’s theory of survival of the fittest. In genetic algorithms terms, this is “elitism.”

Over several generations, the good portions of all the ancestors should collect together into one very

high scoring child.

There are hundreds of ways two parents can mate. In this project, the author has chosen a

dominant gene approach that has worked well in the past (Hausman, A Dominant Gene Genetic

Algorithm for a Transposition Cipher in Cryptography, 2009) (Hausman, A Dominant Gene Genetic

Algorithm for a Substitution Cipher in Cryptography, 2009). The concept is to take the cost function and

apply it on the gene level, or to the individual portions of the solution. This provides a way to determine

which genes are strongest or more “dominant.” These strong genes should have a better chance of

matching the correct solution. Then any children will inherit the dominant genes from both parents.

This makes the child stronger than either parent since the child inherits only the optimal parts of the

solution. Over several generations, this mating technique approaches a strong solution faster than

many other approaches.

In this project, a solution is the current set of senses for each word. Each gene represents the

sense for a single word. To apply the cost function at the gene level, one must keep track of the

semantic relation scores for each word. This means that each word has a contribution to the total cost

function. The words that contribute the most are dominant genes. Since the maximum cost should

represent the best solution, those dominant genes are most likely correct.

There are two ways to mate solutions in this project. When two parents mate, they randomly

choose one of the two methods. Both methods focus on dominant genes. The first method begins with

Hausman 63


the best genes and then combines the mid‐range genes. The second method starts with the mid‐range

genes and then moves on to the best genes.

5.2.1 Mating Method 1: Mate Top Third

As the title suggests, this mating method sorts the genes by their cost, divides this group into

three, and starts with the highest scoring genes. The top third originate from the first parent. Following

this, the child inherits the top two thirds of the second parent. Any remaining genes are derived from

the first parent. The theory is that the child will inherit the best genes while any other genes are

secondary. This formula focuses on the best possible gene distribution. The detailed steps and example

are below.

1. Find Dominant Genes Parent 1 Words: a1 a2 a3 a4 a5 a6 … Gene Cost: 11 12 3 24 15 4 … Select the upper 1/3 Dominant Words: a4, a5, and so forth

Parent 2 Words: b1 b2 b3 b4 b5 b6 … Gene Cost: 25 15 5 10 8 14 … Select the upper 2/3 Dom. Genes: b1, b2, b4, b6, and so forth

2. Place Dominant Genes Based on First Parent Child: * * * a4 a5 * …

3. Fill in Blanks from Second Parent Child: b1 b2 * a4 a5 b6 …

4. Fill in any Remaining Blanks from First Parent Child: b1 b2 a3 a4 a5 b6 …

5.2.2 Mating Method 2: Mate Middle Third

This mating method also divides the genes into three. This time, however, the first parent

focuses on the middle third. Then the child inherits the top two thirds of the second parent and fills any

remaining genes from the first parent. Sometimes the top genes are very dominant as far as cost is

Hausman 64


concerned, but they are incorrect. Since the genes are so strong, they influence the rest of the solution

incorrectly. By focusing on the middle genes, there is a chance that the middle genes may match those

from the second parent and grow stronger. Over time, any high scoring, incorrect genes will grow

weaker and hopefully change to the correct sense. The detailed steps and example are below.

1. Find Dominant Genes Parent 1 Words: a1 a2 a3 a4 a5 a6 … Gene Cost: 11 12 3 24 15 4 … Select the middle 1/3 Dominant Words: a1, a2, and so forth

Parent 2 Words: b1 b2 b3 b4 b5 b6 … Gene Cost: 25 15 5 10 8 14 … Select the upper 2/3 Dom. Genes: b1, b2, b4, b6, and so forth

2. Place Dominant Genes Based on First Parent Child: a1 a2 * * * * …

3. Fill in Blanks from Second Parent Child: a1 a2 * b4 * b6 …

4. Fill in any Remaining Blanks from First Parent Child: a1 a2 a3 b4 a5 b6 …

5.3 Mutation

The concept underlying mutations is to give a genetic algorithm a second chance. In many

solutions, there are local maximum solutions that mating can find and get stuck at the local maximum.

For example, if the challenge is to find the highest point on a map, each hill and mountain is a local

maximum solution. The mating function can “get stuck” at the second highest mountain. Had the

mating function turned the other direction at a valley, it would have found the higher mountain. The

mutation function randomly changes some of the solutions so mating may find the other mountain over

time.

In the dominant gene approach, mutations have two objectives. The first objective is to provide

alternate avenues and second chances by providing alternate dominant genes. The second objective is

Hausman 65


to focus on ways to improve the lower cost or recessive genes. The mating function examines only a

part of the solution and does not improve genes that are not already strong. Eventually the mating

function will use all the available “strong” genes and not be able to move toward a stronger solution.

The mutations can modify a recessive gene to make it stronger. This new gene may be strong enough to

become a new dominant gene, a factor which leads to better solution overall. The dominant gene

approach needs to focus on the dominant and recessive genes in order to be successful.

There are four main mutation functions in the sections below. When a solution mutates, it

randomly chooses one of the four mutation functions. In some implementations, the mutated solutions

replace the original solution. In this case, the mutations return a mutated clone to prevent original

solutions from becoming worse. This makes it harder to find alternate avenues, but the solution never

becomes weaker.

5.3.1 Mutation Function 1: Random Mutation

One of the most popular mutations in many genetic algorithms is a random mutation. It is

quick, easy, and finds solutions that are not possible through other means. In this case, it is the main

way to find alternate avenues and paths not naturally found by improving recessive genes. Most of the

time the results are a weaker gene, but this mutation helps in the long run. The steps for using this

mutation are below.

1. Randomly pick a percentage between 0% and 20% 2. Randomly pick that percentage of words from the solution 3. For each of those words, randomly pick one of the available senses

5.3.2 Mutation Function 2: Semantic Relation Score Mutation

The semantic relation score mutation is perhaps the most useful mutation of all the mutations.

This started out as a function to find the boundaries of a semantic relation in the samples section. A

slight modification allows this mutation to modify a solution so it has or is near the given semantic

Hausman 66


relation score. This makes this very useful for moving a solution towards the optimal semantic relation

values, thus creating stronger solutions. It also provides a way for the genetic algorithm to establish the

range of values necessary for the cost function. The steps to the semantic relation score mutation are

below.

1. Start with the given semantic relation average 2. Find the percentage this average is off from the optimal semantic relation score using the

equation below.

3. Randomly pick the comparison percentage of words 4. If the comparison was negative, the semantic relation cost needs to be increased

a. If the semantic relation is frequency, then randomly pick a sense lower than the current sense. Otherwise use step b.

b. Look at each sense starting at the first sense. Stop and select that sense when the sense increases the semantic relation cost.

5. If the comparison was positive, the semantic relation cost needs to be decreased a. If the semantic relation is frequency, then randomly pick a sense higher than the current

sense. Otherwise use step b. b. Look at each sense starting at the first sense. Stop and select that sense when the sense

decreases the semantic relation cost.

5.3.3 Mutation Function 3: Sense Distribution

The cost function has the following main parts: a semantic relation portion, a sense distribution

portion, and a semantic relation distribution portion. The mutation above covers the semantic relation

portion, so it follows that the second mutation focuses upon the sense distribution. On a higher level,

this mutation finds the current distribution and shuffles the lowest cost genes around to match the

optimal sense distribution. This process achieves two goals. It gives some recessive genes a chance to

become dominant genes and makes the correct distribution to maximize the cost function. This makes

𝐶𝑜𝑚𝑝𝑎𝑟𝑖𝑠𝑜𝑛(𝑠) = 𝐴𝑣𝑔(𝑠) − 𝑂𝑝𝑡𝑖𝑚𝑎𝑙(𝑠)𝑂𝑝𝑡𝑖𝑚𝑎𝑙(𝑠)

S: The current semantic relation Avg: the average value of the semantic relation for this solution Optimal: the optimal value for this semantic relation

Equation 17: The equation for comparing the current solution to the optimal solution

Hausman 67


the sense distribution mutation very useful if it were run several times during the genetic algorithm

process. The steps to the sense distribution mutation are below.

1. Find the number of words extra or missing for each sense compared to the optimal sense distributions

2. Find the genes that have the lowest gene cost 3. Look at each sense distribution

a. If the current sense distribution has extra words, move the lowest cost genes to a sense distribution needing words

5.3.4 Mutation Function 4: Semantic Relation Distribution

The last area of the cost function, not covered by a mutation function, is the semantic relation

distribution. The concept is very similar to the sense distribution mutation. Find the current distribution

for the given semantic relation and shuffle the recessive genes to match the optimal distribution. This

maximizes the cost function by incorporating the proper semantic relation distributions. It also provides

recessive genes a chance to become dominant genes, just as in the sense distribution mutation. The

steps to the semantic relation distribution are below.

1. Compare each average semantic relation value for each sense to the optimal semantic relation value for that sense. This should result in the number of words that need to change for each sense.

2. Find the genes that have the lowest gene cost 3. Look at each sense distribution

a. If the current sense distribution has extra words, move the lowest cost genes to a sense distribution needing words

5.4 Main Genetic Algorithm Function

Earlier the author provided the generic main steps to create a genetic algorithm. The preceding

sections explain how the three main parts (cost function, mating, and mutation) function. However,

none of these sections explain the size of a generation, where the range of values for the cost function

originates, and how chromosomes for the mutation and mating are selected. Details such as this are the

responsibility of the main genetic algorithm function. This function is the center of the genetic

algorithm. The step by step process of this algorithm is below.

Hausman 68


1. Read in the given text, initialize all the WordNet semantic relation information, and assign the parts of speech with a part of speech tagger

2. Initialize the generation pool. Note that this pool does not keep track of duplicate solutions. 3. Initially find the range of possible scores for each semantic relation using the semantic relation

score mutation. Each semantic relation has an upper and lower boundary. There will be one process for each boundary. This process is below. The cost function relies on these ranges for comparing solutions.

a. Start with a solution using the first sense for all words (or the given solution if started from step 5)

b. Mutate that solution towards the given boundary (maximum or minimum possible value)

c. Repeat step b until there is a repeat solution or until 25 mutations are performed d. Report the current semantic relation score as the given boundary.

4. Add all of the solutions from step 3 into the generation pool. On top of that use the weighted random function from the sampling section to create 25 new solutions.

5. If the generation number is divisible by 10, repeat step 3 using the top solution. Sometimes the new starting point finds a slightly wider semantic relation range.

6. Mate the top 5 solutions as parent 1 with randomly chosen solutions from the generation pool as parent 2. After this mate two randomly chosen solutions from the generation pool 20 times. Add all children to the generation pool.

7. Mutate the top 5 solutions. Then mutate 20 randomly chosen solutions from the generation pool. Add all new solutions to the generation pool.

8. Reduce the current generation pool by only keeping the top 25 solutions. 9. Repeat steps 5‐8 for 25 generations.

5.5 Notes about Speed

Many of the sections above describe complex algorithms containing a large number of

combinations. There are 25 generations with each generation creating 50 new solutions. The cost

function for each solution uses several semantic relations. Each semantic relation is runs on every word

pair combination in a window with 45 possible combinations. The number of windows is the total

number of words in a solution. Every time the cost function ranges change, all of the solutions must

rerun the cost function. The semantic relation score mutation in particular manipulates genes one at a

time, which changes the score of the entire solution. There are a very large number of calculations for

this genetic algorithm.

To deal with the number of calculations, there are several architectural choices and

modifications to the code to reduce the impact of the changes. All of the information is stored in RAM

Hausman 69


to reduce the time it takes to look at the disk, which makes the program run with ~700MB of memory.

The cost function tracks the information in various data structures and minimizes the number of

recalculations mutations and semantic relation range changes cause. All mating and mutation

operations run in parallel to take advantage of multicore processors. However, all of these

improvements are limited. The number of calculations alone causes this program to run for a few

minutes for each SemCor file. Many of the SemEval competitions have more words, so the process will

take longer. This is not a real time solution for large groups of text.

Hausman 70


Chapter 6: Results and Analysis

This project has compared the results within three main areas. The first area focused upon a

comparison against a colleague working on the same problem. The colleague, Michael Billot, has

implemented a page rank algorithm to solve word sense disambiguation. The second area is the

“Genetic Word Sense Disambiguation” study by Zhang (Zhang, Zhou, & Martin, 2008). This paper was

the starting point for this study, and it has significantly influenced the conceptual framework of the

author’s approach. The author has included several changes which are based on reports from other

studies, but their influence has not been as significant. The third area is the SemEval competitions.

Many researchers who did and did not participate in the competitions use these results as a comparison

against other algorithms. All three comparisons should provide significant insight to how well this

algorithm functions.

6.1 Measuring the Results

When the first SemEval competition occurred in 1998, the sponsors sought to compare the

results from all the competitors. They devised three categories for the competition: coverage, recall,

and precision. Coverage is the ratio of how many words the competitor answered to the total number

of words overall. This reflects how much of the solution a competitor answered. Recall is the ratio of

the total number of senses correct within the total number of words provided. This indicates how well a

solution was answered. Precision is the total number of senses correct within the total number of

words answered. This indicates how well an algorithm answers when it does give a sense. SemEval

2007 introduced two different ways to measure a result because some senses in WordNet are too subtle

for even humans to disambiguate. The first method uses the exact sense as a fine word assessment, as

Hausman 71


was done in the previous competitions. The second method combines subtly different senses into one

possible answer as a coarse words assessment. The sponsors evaluated the SemEval 2007 competition

by using the same three numbers separately for both the fine and coarse word assessments.

6.2 Comparison to Michael Billot

Michael Billot is the author’s colleague at the University of Colorado at Colorado Springs. His

solution uses the page rank algorithm to disambiguate verbs. In his study, he used several walkthroughs

for Zelda as the input. He then evaluated the results by hand, a method which presents obvious

difficulties. Unless the verb is clearly a sense, which is rare, it is hard to prove one sense is correct. Even

during the SemEval competitions, the sponsors only have the competitors attempt to find the sense of a

word when over 90% of a group of professional linguists agree. Even if the project uses the same text,

there is no clear indication as to whether the author has chosen the same senses as did Billot, or

whether the sense is correct. With that in mind, comparing against Billot may be a problematic

challenge.

Billot’s page rank algorithm has a 46.4% accuracy according to his study (Billot, 2010).

Considering that he supplies a sense for every verb, this 46.4% figure is the recall and the precision. The

value for the first sense baseline is 77.4%. Since evaluation by hand is problematic at best, this author

did not attempt to disambiguate the walkthrough. However, this author does have the results for

SemCor. For a first sense baseline, the 77.4% baseline is higher than the average baseline, but it is still

within acceptable range. The average coverage for verbs is 99.26%. This is very high, which makes the

recall and precision are very close. The average recall is 49.24% and the average precision is 49.61%.

These percentages suggest that this author’s project evaluates the results more accurately, but the

results are still too close to be certain. The most effective way to tell for sure is for Billot to provide

results for the SemCor files as well.

Hausman 72


6.3 Comparison to Zhang

Zhang’s study, “Genetic Word Sense Disambiguation”, represents the most similar approach to

this project. However, Zhang only evaluates nouns and has not participated in any of the SemEval

competitions. Zhang does, however, provide references to several SemCor files. Overall, Zhang’s

coverage is 100%, so his recall and precision results are the same. This project purposely ignores

pronouns, resulting in average coverage of 89.25%. Within all the SemCor files, Zhang has a

recall/precision of 71.96%. This project has a precision of 62.13% for nouns. Zhang reports a 70%

accuracy on 51 files. This project reports a 70% precision on nine files. This author’s conclusion is that

that this project is not as successful as was Zhang’s. However, one should keep in mind that this

author’s project solved for all words, a significantly more complex study overall.

6.4 SemEval

The SemEval competitions occur every three years. The competitions are devoted to language

processing and typically incorporate an “all words” task. This task requires performing word sense

disambiguation on all the words they have tagged. They then use this information to provide recall,

precision, and coverage for each competing algorithm. Many researchers compare themselves to the

competitors in the SemEval competitions.

6.4.1 SemEval 2

In the SemEval‐2 competition in 2001, there were 22 systems participating for the English all

words task (ACL‐SIGLEX, 2011). The baseline for using the first sense was 57.0%. The best system had

100% coverage and a 69.8% precision. This author’s project achieved 95.12% coverage and a 52.29%

precision. The comparison shows a 4.71% precision below the baseline and a 17.51% precision below

the best system. The winner was an outlier in this competition. The second place system had a

Hausman 73


precision 5.3% below the first place winner. If this project were competing, it would be in sixth place.

This performance lies somewhere between the middle ranks and the top ranks.

Table 1: The results from SemEval 2 Rank System Coverage (%) Recall Precision 1 SMUaw 100 0.69 0.69 2 CNTS‐Antwerp 100 0.636 0.636 3 Sinequa‐LIA ‐ HMM 100 0.618 0.618 4 UNED ‐ AW‐U2 98.908 0.569 0.575 5 UNED ‐ AW‐U 98.908 0.55 0.556 ‐ Michael Hausman 95.12 0.4973 0.5229 6 UCLA ‐ gchao2 95.552 0.454 0.475 7 UCLA ‐ gchao3 95.552 0.453 0.474 8 CL Research ‐ DIMAP 108.5 0.451 0.416 9 CL Research ‐ DIMAP (R) 100 0.451 0.451 10 UCLA ‐ gchao 89.729 0.449 0.5 11 Universiti Sains Malaysia 2 99.96 0.36 0.36 12 IRST 47.756 0.357 0.748 13 Universiti Sains Malaysia 1 97.897 0.338 0.345 14 Universiti Sains Malaysia 3 99.96 0.336 0.336 15 BCU ‐ ehu‐dlist‐all 50.789 0.291 0.572 16 Sheffield 45.37 0.2 0.44 17 Sussex ‐ sel‐ospd 29.883 0.169 0.566 18 Sussex ‐ sel‐ospd‐ana 31.055 0.169 0.545 19 Sussex ‐ sel 23.332 0.14 0.598 20 IIT 2 11.646 0.038 0.328 21 IIT 3 11.646 0.034 0.294 22 IIT 1 11.646 0.033 0.287

6.4.2 SemEval 3

In the SemEval‐3 competition of 2004, 26 systems participated in the all words task (Snyder &

Palmer, 2004). The baseline of using the first sense was 60.9% or 62.4% depending on the treatment of

compound words. This project used compound words; 62.4% was the baseline. The best system had

100% coverage and a 65.1% precision. This project had 96.93% coverage and a 53.79% precision, 8.61%

below the baseline and 11.31% below the best system. These results are disappointing considering that

the average SemCor precision is close to the top performing system. It should be noted, however, that

Hausman 74


this this author’s work would rank 15th in the competition. This is in the middle ranks of the

competition.

Table 2: The results from SemEval 3 Rank System Coverage (%) Recall Precision 1 GAMBL‐AW‐S 100 0.651 0.651 2 SenseLearner‐S 98.62 0.642 0.651 3 Koc University‐S 98.61 0.639 0.648 4 R2D2: English‐all‐words 100 0.626 0.626 5 Meaning‐allwords‐S 99.68 0.623 0.625 6 Meaning‐simple‐S 99.84 0.61 0.611 7 LCCaw 98.70 0.606 0.614 8 upv‐shmm‐eaw‐S 98.21 0.605 0.616 9 UJAEN‐S 97.84 0.588 0.601 10 IRST‐DDD‐00‐U 99.83 0.582 0.583 11 University of Sussex‐Prob5 97.09 0.568 0.585 12 University of Sussex‐Prob4 95.65 0.55 0.575 13 University of Sussex‐Prob3 95.46 0.547 0.573 14 DFA‐Unsup‐AW‐U 98.03 0.546 0.557 ‐ Michael Hausman 96.93 0.5213 0.5379 15 KUNLP‐Eng‐All‐U 97.25 0.496 0.51 16 IRST‐DDD‐LSI‐U 75.04 0.496 0.661 17 upv‐unige‐CIAOSENSO‐eaw‐U 82.62 0.48 0.581 18 merl.system3 97.64 0.456 0.467 19 upv‐unige‐CIAOSENSO2‐eaw‐U 74.18 0.451 0.608 20 merl.system1 97.39 0.447 0.459 21 IRST‐DDD‐09‐U 60.49 0.441 0.729 22 autoPS‐U 88.37 0.433 0.49 23 clr04‐aw 85.18 0.431 0.506 24 autoPSNVs‐U 62.88 0.354 0.563 25 merl.system2 73.33 0.352 0.48 26 DLSI‐UA‐all‐Nosu 80.17 0.275 0.343

6.4.3 SemEval 2007

In the SemEval 2007 competition, there were 14 systems participating in the competition (15 if

the task organizers system is included) (Navigli, 2007). In this case, all the results are for the coarse

words evaluation. The baseline using the first sense has a 78.89% precision. The best participating

system has 100% coverage and an 82.5% precision (the organizer’s system has 100% coverage and an

83.21% precision). This project has 100% coverage and a 74.51% precision. This is 4.38% below the

Hausman 75


baseline and 8.34% below the organizer’s system. This definitely does not perform as well as the top

system. If this project participated in the competition, it would have ranked in 7th place. This is the

center rank.

Table 3: The coarse results from SemEval 2007 Rank System Coverage (%) Recall (%) Precision (%) 1 NUS‐PT 100 82.5 82.5 2 NUS‐ML 100 81.58 81.58 3 LCC‐WSD 100 81.45 81.45 4 GPLSI 100 79.55 79.55 5 BLMFS 100 78.89 78.89 6 UPV‐WSD 100 78.63 78.63 ‐ Michael Hausman 100 74.51 74.51 7 TKB‐UO 100 70.21 70.21 8 PU‐BCD 90.1 62.8 69.72 9 RACAI‐SYNWSD 100 65.71 65.71 10 SUSSX‐FR 72.8 52.23 71.73 11 USYD 95.3 56.02 58.79 12 UOFL 92.7 48.74 52.59 13 SUSSX‐C‐WD 72.8 39.71 54.54 14 SUSSX‐CR 72.8 39.53 54.3

Hausman 76


Chapter 7: Conclusions and Future Research

This project has attempted to solve for word sense disambiguation. Solving word sense

disambiguation would allow several applications, like language translation, to work much more

accurately. To accomplish this, the author investigated several semantic relations and transformed

them into an optimization problem. This way a genetic algorithm could solve the problem. The genetic

algorithm used a dominant gene technique to converge on an answer. The author then compared the

results to several other researchers, including the competitors in the SemEval competitions. If this

algorithm had competed in the SemEval competitions, it would have rank in the middle. This means

that it is not the optimal algorithm, but it still works well enough to be of interest since it is not in the

lower ranks.

7.1 Algorithm Weaknesses

One of the reasons why the results are not 100% has to do with the available semantic relations.

An in‐depth investigation of some of the words reveals that there is simply not enough information

provided in context. Some words have a correct sense of something other than the most common

sense, yet frequency is the only semantic relation that applies to the word. Frequency directs the result

to the incorrect sense. False positives and misleading information do have an effect, but a large

proportion of the words have insufficient information in order to provide acceptable results. In most of

these cases, the information supplied by WordNet will not help. This includes several other semantic

relations that were not part of this project. The author cursorily explored Lesk, but upon reflection it

proved unhelpful. It is significant that many other algorithms appear to have similar problems. One of

the top competitors in the third SemEval competition, Sense Learner, openly admits that they do have

Hausman 77


any information on 15% of the words (Mihalcea & Csomai, 2005). These words simply chose the most

frequent definition as a default. Regardless, something needs to account for these words.

This algorithm does appear to work fairly well with the available WordNet information. Many

other researchers assume that the highest scores work while this algorithm attempts to adjust for the

weaknesses within each semantic relation. However, there still are possible improvements around

semantic relations in the cost function. One of the more significant problems is how large the 90%

window is for some semantic relations. If there were a way to dynamically adjust the maximum point

and reduce the deviation, the answers would improve. At this writing, both the distributions and the

relations rely on averages. The distributions require more investigation and adjustments. Perhaps the

distributions would function more effectively if they accounted for the various parts of speech

individually. One can only surmise in this regard, but it seems logical that additional percentage points

make a significant difference in the SemEval competitions.

7.2 Future Possibilities

There were two tool changes mentioned earlier that may foreshadow future research. The first

was importing the “WordNet Lexical Markup Framework” from the SemEval 2010 competition. This

would allow the author’s project to compare the SemEval 2010 results and to look at results in different

languages. Since one possible application of word sense disambiguation is machine text translation, the

performance in other languages would be very helpful to know. The second change is continuing

investigation of OntoNotes. The semantic and lexical relations will be different for OntoNotes. If the

plans for exploring OntoNotes take place, the research results will be more useful than those relations in

WordNet. At the very least, the algorithm will be able to compete in SemEval in the future if the

competition replaces WordNet. These two changes would help in the future and may provide insight for

other changes to this algorithm.

Hausman 78


The top‐rated algorithms in the SemEval competition typically account for a wider range of

information. This algorithm uses semantic relation information from WordNet. Their algorithms literally

look at everything they could think of. This includes information such as: A) “Is this sense typically the

end of a sentence?” or B) “Is this word followed by a noun?” and C) “Does this word end in ‘ing’ all the

time?” This information feeds into various types of algorithms many of which are extremely

complicated. Outstanding competitors have used multiple information sources. One option that may

help this algorithm is the use of more than WordNet semantic relations.

A final option is exploration of punctuation. For example, examine the statement, “Let’s go eat,

grandma!” If no comma separated eat and Grandma, the sentence would take on a most horrific

meaning. Sentence sense changes with punctuation. This author suggests that exploration of the field

of punctuation research in computer systems language development studies may grow into a field that

significantly changes the focus of the discipline and provides data to scaffold word sense disambiguation

on a most significant scale.

Hausman 79


Appendix

The following sections contain several of the detailed results developed throughout this project.

This includes a short table of the types of SemCor files, the proportion statistics for Cost Method 3, the

distributions for Cost Method 4, the distributions for Cost Method 5, and results for every SemCor file.

Appendix A: SemCor Files

Each SemCor file has a similar name: br‐*##. The “br” is short for the brown corpus. The “*”

represents a letter that indicates what genre of text is tagged. The “##” represents a number that

separates different files in a genre.

Table 4: The SemCor file letter indicating type of resource the original text came from. Informative Prose (374 samples) Imaginative Prose (126 samples) Letter Genre Letter Genre A Press: Reportage K General Fiction B Press: Editorial L Mystery and Detective Fiction C Press: Reviews M Science Fiction D Religion N Adventure and Western Fiction E Skill and Hobbies P Romance and Love Story F Popular Love R Humor G Belles Letters, Biography, Memoirs, etc. H Miscellaneous J Learned

Table 5: The various SemCor files File Author Source A01

Atlanta Constitution Political Reportage A02 Dallas Morning News Political Reportage A03 Chicago Tribune Political Reportage A04 Christian Science Monitor Political Reportage A05 Providence Journal Political Reportage A06 Newark Evening News Political Reportage A07 New York Times Political Reportage A08 Times‐Picayune, New Orleans Political Reportage A09 Philadelphia Inquirer Political Reportage A10 Oregonian, Portland Political Reportage A11 Sun, Baltimore Sports Reportage A12 Dallas Morning News Sports Reportage

http://icame.uib.no/brown/brownwww.html#A12












Hausman 80


File Author Source A13 Rocky Mountain News Sports Reportage A14 New York Times Sports Reportage. A15 St. Louis Post‐Dispatch Sports Reportage A16 Chicago Tribune Society Reportage A17 Rocky Mountain News Society Reportage A18 Philadelphia Inquirer Society Reportage A19 Sun, Baltimore Spot News A20 Chicago Tribune Spot News A21 Detroit News Spot News A22 Atlanta Constitution Spot News A23 Oregonian, Portland Spot News A24 Providence Journal Spot News A25 San Francisco Chronicle Spot News A26 Dallas Morning News Financial Reportage A27 Los Angeles Times Financial Reportage A28 Wall Street Journal Financial Reportage A29 Dallas Morning News Cultural Reportage. A30 Los Angeles Times Cultural A31 Miami Herald Cultural Reportage A32 San Francisco Chronicle Cultural Reportage A33 Washington Post Cultural Reportage A34 New York Times News of the Week in Review A35 James J. Maguire A Family Affair A36 William Gomberg Unions and the Anti‐Trust Laws A37 Time National Affairs A38 Sports Illustrated A Duel Golfers Will Never Forget A39 Newsweek Sports A40 Time People. Art & Education A41 Robert Wallace This Is The Way It Came About A42 Newsweek National Affairs A43A U. S. News & World Report Better Times for Turnpikes A43B U. S. News & World Report A Plan to Free U. S. Gold Supply A44A John Tebbel Books Go Co‐operative A44B Gilbert Chapman Reading and the Free Society B01

Atlanta Constitution Editorials B02 Christian Science Monitor Editorials B03 Detroit News Editorials B04 Miami Herald Editorials B05 Newark Evening News Editorials B06 St. Louis Post‐Dispatch Editorials B07 New York Times Editorials B08 Atlanta Constitution Columns B09 Christian Science Monitor Columns

http://icame.uib.no/brown/brownwww.html#B09











































Hausman 81


File Author Source B10 Sun. Baltimore Columns B11 Los Angeles Times Columns B12 Newark Evening News Columns B13 Times‐Picayune, New Orleans Columns B14 Atlanta Constitution Columns B15 Providence Journal Letters to the Editor B16 Chicago Tribune Voice of the People B17 Newark Evening News What Readers Have to Say B18 New York Times Letters to the Times B19 Philadelphia Inquirer The Voice of the People B20 Nation Editorials B21A Gerald W. Johnson The Cult of the Motor Car B21B James Deakin How Much Fallout Can We Take B22 Commonweal Week by Week B23A William F. Buckley, Jr. We Shall Return B23B James Burnham Tangle in Katanga B24 Time Reviews B25A Alexander Werth Walkout in Moscow B25B Peter Solsich, Jr. The Armed Superpatriots B26 National Review To the Editor B27 Saturday Review Letters to the Editor C01

Chicago Daily Tribune Reviews C02 Christian Science Monitor Reviews C03 New York Times Reviews C04 Providence Journal Reviews C05 Christian Science Monitor Reviews C06 Wall Street Journals Reviews C07 New York Times Reviews C08 Providence Journal Reviews C09 New York Times Reviews C10 Providence Journal Reviews C11 New York Times Reviews C12 Christian Science Monitor Reviews C13 Wall Street Journal Reviews C14 New York Times Reviews C15 Life Reviews C16 Saturday Review Reviews C17 Time Reviews D01

William Pollard Physicist and Christian D02 Schubert Ogden Christ Without Myth D03 Edward E. Kelly Christian Unity in England D04 Jaroslav Pelikan The Shape of Death D05 Perry Miller Theodore Parker: Apostasy With in Liberalism

http://icame.uib.no/brown/brownwww.html#D05






















http://icame.uib.no/brown/brownwww.html#C01





















Hausman 82


File Author Source D06 A Howard Kelly Out of Doubt into Faith D06B Shirley Schuyler Not as the World Giveth D06C Nathanael Olson Are You in Orbit? D07 Peter Eldersveld Faith Amid Fear D08 Schuyler Cammann The Magic Square of Three D09 Eugene E. Golay Organizing the Local Church D10 Huston Smith Interfaith Communication: The Contemporary Scene D11 Paul Ramsey War & the Christian Conscience D12 Kenneth Underwood and Widen

Jacobson Probing the Ethics of Realtors

D13A Donald H. Andrews The New Science & the New Faith D13B George Bo Longstreet The Seeming Impossible D14 Kenneth S. Latourette Christianity in a Revolutionary Age D15 Ernest Becker Zen: A Rational critique D16A Anonymous What the Holy Catholic Bible Teach D16B Harold Brenneman Notice You May Do As You Please D17A Anonymous Guideposts: 15th Anniversary Issue D17B J. I. Rivero The Night Our Paper Died E01A

Ben Welder Henri de Courcy: Jr. Mr. Canada E0lB Joe Welder The Mark of the Champion E02A Dorothy Schroeder Plant a Carpet of Bloom E02B Anonymous Avocado is Something Special E03 D. F. Martin Will Aircraft or Missiles Win Wars? E04A Harris Goldsmith The Schnabel Pro Arte Trout E04B Robert C. Marsh The True Sound of a Solid Second E04C R.D.D. Review of Adam, Giselle E05A Paul Nigro The Younger Generation E05B Patricia Barney Use of Common Sense Makes Dogs Acceptable E05C Anonymous The Malady Lingers On E06 Joseph E. Choate The American Boating Scene E07 Paul Larson and Gordon Odegard How to Design Your Interlocking Frame E08 Don Francisco Formulas and Math Every Hot Rodder Should Know E09A Don McMahan The Week at Ben White Raceway E09B Edith Shaw The Picture at Del Mar E10 Larry Koller The New Guns of 61 E11 Idwal Jones Santa Cruz Run E12 Julia Newman Travel and Camera USA E13 Robert Deardorff Step by Step through Istanbul E14 Ann Carnahan Nick Manero's Cook‐out Barbecue Book E15A Anonymous Pottery from Old Molds E15B Anonymous Knitting Knacks E16 Hal Kelly Build Hotei E17A Anonymous This is the Vacation Cottage You Can Build

http://icame.uib.no/brown/brownwww.html#E17










































Hausman 83


File Author Source E17B Patrick K. Snook Care and Basic Use of the Drill Press E18A Lura W. Watkins The Bridge Over the Merrimac E18B Boyd B. Stutler Veteran Philippi Bridge E19 Booth Hemingway and Stuart H.

Brown How to Own a Pool and Like It.

E20 Anonymous What You Should Know About Air Conditioning E21 Richard McCosh Recreation Site Selection E22A Roy Harris Roy Harris Salutes Serge Prokofieff E22B Helen Havener A 30 Years War E23 Norman Kent The Watercolor Art of Roy M. Mason E24 Bonnie Prudden The Dancer & the Gymnast E25 Walter Ho Buchsbaum Advances in Medical Electronics E26 Bern Dibner Oersted & the Discovery of Electromagnetism E27A Mike Bay What Can Additives Do for Ruminants? E27B James S. Boyd Which Feed Bunk for You E28 John R. Sargent Where to Aim Your Planning E29 Edward A. Walton On Education for the Interior Designer E30 Anonymous The Attack on Employee Services E31A Jim Dee Expanding Horizons E31B George Laycock The Challenge E32 E. J. Tangerman Which Way Up. Technical or Management? E33A Robert Gray Fifty Houses, One Tank E33B Chet Cunningham Truck Talk E34 Anonymous The New Look in Signs E35 Anonymous The Industrial Revolution in Housing E36 Ethel Norling Renting a Car in Europe F01

Rosemary Blackmon How Much Do You Tell When You Talk?

F02 Glenn Infield America's Secret Poison Gas Tragedy F03 Nathan Rapport I've Been Here Before F04 Ruth F. Rosevear North Country School Cares for the Whole Child F05 Richard S. Allen When Fogg Flew the Mail F06A Alice Ho Austin Let's Discuss Retirement F06B Harold P. Winchester What It Means to be Creative F07A Marvin Sentnor and Stephen Hult How to Have a Successful Honeymoon F07B Ho Walter Yoder Attitudes Toward Nudity F08 Philip Reaves Who Rules the Marriage Bed? F09A David Martinson Fantastic Life & Death of the Golden Prostitute. F09B Isel D. Rugget When It Comes to Carpets F10 Jack Kaplan Therapy by Witchcraft F11 Lillian Pompian Tooth‐Straightening Today F12 Marian Neater New Methods of Parapsychology. F13 Orlin J. Scoville Part‐time Farming

http://icame.uib.no/brown/brownwww.html#F13









































Hausman 84


File Author Source F14 Harold Rosenberg The Trial and Eichmann F15 John A. O'Brien Let's Take Birth Control Out of Politics F16 James Boylan Mutiny F17 John Harnsberger and Robert P.

Wilkins Transportation on the Northern Plains

F18 Bell I. Willy Home Letters of Johnny Reb & Billy Yank Fl9 Tristram P. Coffin Folklore in the American Twentieth Century F20 Kenneth Allsop The Bootleggers and Their Era F21A Joseph Bernstein Giant Waves F21B L. Don Leet Introduction F21C L. Don Leet The Restless Earth and Its Interiors F22 Booton Herndon From Custer to Korea, the 7th Cavalry F23 Barry Goldwater A Foreign Policy for America F24 Peter J. White Report on Laos F25 David Boroff Jewish Teen‐Age Culture F26 Amy Lathrop Pioneer Remedies from Western Kansas F27 Creighton Churchill A Notebook for the Wines of France F28 Frank O. Gatell Doctor Palfrey Frees His Slaves F29 Douglass Cater The Kennedy Look in the Arts F30 Frederic A Birmingham The Ivy League Today F31 Edward Do Radin Lizzie Borden: The Untold Story F32 Florence M. Read The Story of Spelman College F33 James Be Conant Slurs and Suburbs F34 Frederic R. Senti and W. Dayton

Maclay Age‐old uses of Seeds and Some New Ones

F35 Ramon F. Adams The Old‐time Cowhand F36 Robert Easton and Mackenzie Brown Lord of Beasts F37 Samuel M. Cavert On the Road to Christian Unity F38 Robert Smith Baseball in America F39 Clark E. Vincent Unmarried mothers F40 William Greenleaf Monopoly on Wheels F41 George W. Oakes Turn Right at the Fountain F42 James Baldwin Nobody Knows My Name F43 Frank Getlein and Harold C. Gardiner Movies, Morals, and Art F44 Gibson Winter The Suburban Captivity of the Churches F45 Paul C. Phillips The Fur Trade F46 Russell Baker An American in Washington F47 Clara L. Simerville Home Visits Abroad F48 Paul Ramsey Christian Ethics & the Sit‐In G01 Edward P. Lawton Northern Liberals & Southern Bourbons G02 Arthur S. Miller Toward a Concept of National Responsibility G03 Peter Wyden The Chances of Accidental War G04 Eugene Burdick The Invisible Aborigine

http://icame.uib.no/brown/brownwww.html#G04









































Hausman 85


File Author Source G05 Terence O'Donnell Evenings at the Bridge G06A Ruth Berges William Steinberg, Pittsburgh's Dynamic Conductor G06B Henry W. Koller German Youth Looks to the Future G07 Richard B. Morris Seven Who Set Our Destiny G08 Frank Murphy New Southern Fiction: Urban or Agrarian? G09 Selma J. Cohen Avant‐Garde Choreography G10 Clarence Streit How the Civil War Kept You Sovereign G11 Frank Oppenheimer Science and Fear G12 Tom F. Driver Beckett by the Madeleine G13 Charles Glicksberg Sex in Contemporary Literature G14 Helen H. Santmeyer There Were Fences G15 Howard Nemerov Themes and Methods: Early Storie of Thomas Mann G16 John F. Hayward Mimesis & Symbol in the Arts G17 Randall Stewart A Little History, a Little Honesty G18 Charles W. Stork Verner von Heidenstam Gl9 R. F. Shaw The Private Eye G20 Dan McLachlan, Jr. Communication Networks & Monitoring G21 Brainard Cheney Christianity & the Tragic Vision G22 Kenneth Reiner Coping with Runaway Technology G23 William C. Smith Why Fear Ideas G24 Sanchia Thayer Personality & Moral Leadership G25 Stanley Parry The Restoration of Tradition G26 Selma Fraiberg Two Modern Incest Heroes G27 Matthew Josephson Jean Hélion. The Return from Abstract Art G28 Arlin Turner William Faulkner, Southern Novelist G29 Anonymous References for the Good Society G30 Norwood R. Hanson Copernican & Keplerian Astronomy G31 Irving Fineman Woman of Valor: Life of Henrietta Szold 1860‐1945 G32 Finis Farr Frank Lloyd Wright G33 Virgilia Peterson A Matter of Life and Death G34 Harry Golden Carl Sandburg G35 Dwight D. Eisenhower Peace With Justice G36 DeWitt Copp & Marshall Peck Betrayal at the UN G37 Gordon L. Hall Golden Boats from Burma G38 Bertrand A. Goldgar The Curse of Party G39 Edward Jablonski Harold Arlen Happy with the Blues G40 Gene Fowler Skyline: A Reporter's Reminiscences of the 1920s. G41 Lillian R. Parka and Frances S.

Leighton My Thirty Years Backstairs at the White House

G42 Harold D. Lasswell Epilogue G43 Robert E. Lane The Liberties of Wit G44 Newton Stallknecht Ideas and Literature G45 W. A. Swanberg Citizen Hearst: A Biography of W. R. Hearst











































Hausman 86


File Author Source G46 Henry R. Winkler George Macaulay Trevelyan G47 Carry Davis The World Is My Country G48 Francis F. McKinney Education in Violence G49 Paul van K. Thomson Francis Thompson, a Critical Biography G50 Curtis C. Davis The King's Chevalier G51 Ilka Chase The Carthaginian Rose G52 Robert L. Duncan Reluctant General G53 Bertram Lippincott Indians, Privateers, and High Society G54 Mabel W. Wheaton & LeGette Blythe Thomas Wolfe & His Family G55 Ralph E. Flanders Senator from Vermont,. 112 G56 Keith F. McKean The Moral Measure of Literature G57 Robin M. Williams, Jr. Values & Modern Education in the United States G58 North Callahan Daniel Morgan G59 Esther R. Clifford A Knight of Great Renown G60 Gertrude Berg & Cherney Berg Molly and Me G61 Donald A. White Litus Saxonicum G62 C. H. Cramer Newton D. Baker G63 George Steiner The Death of Tragedy G64 Mark Eccles Shakespeare in Warwickshire G65 Timothy P. Donovan Henry Adams & Brooks Adams G66 Van Wyck Brooks From the Shadow of the Mountain G67 Mark Schorer Sinclair Lewis: An American Life G68 Harris F. Fletcher The Intellectual Development of John Milton G69 Mark R. Hillegas Dystopian Science Fiction G70 Joseph W. Krutch If You Don't Mind My Saying So G71 Joseph Frank André Malraux: The Image of Man G72 J W. Fulbright For a Concert of Free G73 Carolyn See The Jazz Musician as Patchen's Hero G74 John McCormick The Confessions of Jean Jacques Krim G75 George Garrett A Wreath for Garibaldi H01

U. S Dep't of Commerce Handbook of Federal Aids to Communities

H02 U. S. Dep't of State An Act for International Development H03 U. S. 87th Congress House Document No. 487 H04 R. I. Legislative Council State Automobiles & Travel Allowances H05 R. I. Leglelative Council Taxing of Movable Tangible Property H06 R. I. Development Council Annual Report, 1960 H07 R. I. Legislative Council linlform Fiscal Year for Municipalities H08 John A. Notte, Jr. R. I. Governor's Proclamations H09 U. S. 87th Congress Public Laws 295, 300, 347 H10 U. S. Dep't of Defense Medicine in National Defense H11 U. S. Dep't of Commerce 1961 Reaearch Highlights, Nat'1 Bureau of Standards H12 U. S. 87th Congress Legislation on Foreign Rels

http://icame.uib.no/brown/brownwww.html#H12










































Hausman 87


File Author Source H13 U. S. 87th Congreas Congressional Record: Extension of Remarks. May 2,

1961 H14 U. S. Dep't of Health, Education &

Welfare Grants‐in‐Aid and Other Financial Assistance Programs

H15 U. S. Office of Civil and Defence Mobilization

The Family Fallout Shelter

H16 U. S. Reports Cases AdJudged in the Supreme Court, October Tenm 1960

H17 U. S. Reports Cases AdJudged in the Supreme Court, October Tenm 1959‐60

H18 Dean Rusk The Department of State Hl9 Peace Corps Fact Book H20 U. S. Dep't of Agriculture Development Program for the National Forests H21 Dwight D. Eisenhower Public Papers, 1960‐61 H22 U. S. Dep't of State U. S. Treatiea and Other International Agreements H23 U. S. Federal Communications

Commiasion Pederal Communications Commission Reports

H24 U. S. Tresaury Dep't Your Federal Income Tax H25 Guggenheim Foundation Report of the Secretary Gen'1 H26 Anonymous A Brief Background of Brown & Sharpe H27A Robert Leeson Leesona Corporation President's Report H27B Leesona Corporation More Efficient Production for Expanding Textile

Markets H28 Carleton College Carleton College Bulletin H29 Sprague Electric Company Sprague Log H30 Carnegie Foundation Annual Report of Year Ending June 30, 1961 J01

Cornell H. Mayer Radio. Emission of the Moon and Planets

J02 R. C. Binder et al. 1961 Heat Transfer & Fluid Mechanics Institute J03 Harry H. Hull Normal Forces & Their Thermodynamic Significance J04 James A. Ibers et al. Proton Magnetic Resonance Study J05 John R. Van Wazer, ed. Phosphorus and Its Compounds J06 Francis J. Johnston & John E. Willard Exchange Reaction Between C12 and CC14 J07 J. F. Vedder Micrometeorites J08 LeRoy Fothergill Biological Warfare J09 M. Yokayama et al Chemical & Serological Characteristics J10 B. J. D. Meeuse The Story of Pollination J11 Clifford H Pope The Ciant Snakes J12 Richard F McLaughlin et al. A Study of the Subgross Pulmonary Anatomy J13 S. Idell Pyle et al. Onsets, Completions & Spans J14 Jacob Robbins et al. The Thyroid‐Stimulating Hormone J15 J. W. C. Hagstrom et. al. Debilitating Muscular Weakness Jl6 A. N. Nagaraj & L. M. Black Localization of Wound‐Tumor Virus Antigen J17 E. Gellhorn Prolegomena to a Theory of the Emotions

http://icame.uib.no/brown/brownwww.html#J17




































Hausman 88


File Author Source J18 Kenneth Hoffman & Ray Kunze Linear Algebra Jl9 Frederick Mosteller et al. Probability with Statistical Applications J20 R. P. Jerrard Inscribed Squares in Plane Curves J21 C. R. Wylie, Jr. Line Involutions in S3 J22 Max F. Millikan & Donald L.

Blackmer, editors The Emerging Nations

J23 Joyce O. Hertzler American Social Institutions J24 Howard J. Parad Preventive Casework: Problems & Implications J25 Sister Claire M. Sawyer Some Aspects of Fertility of a Tri‐Racial Isolate J26 Frank Lorimer Demographic Information on Tropical Africa J27 Dale L. Womble Functional Marriage Course for the Already Married J28 William H. Ittelson & Samuel B.

Kutash, editors Perceptual Changes in Psychopathology

J29 Jesse W. Grimes & Wesley Allinsmith Compulsivity, Anxiety & School Achievement J30 Raymond J. Corsini Roleplaying in Business & Industry J31 Harold Searles Schizophrenic Communication J32 Hugh Kelly & Ted Ziehe Glossary Lookup Made Easy J33 Ralph Bc Long The Sentence & Tts Parts J34 H.A. Cleason Review of African Language Studies J35 A. T. Kroeber Semantic Contribution of Lexicostatistics J36 D. F. Fleming The Cold War & Its Origins J37 Douglas Ashford Elections' in Morocco: Progress or Confusion J38 Committee for Economic

Development Distressed Areas in a Growing Economy

J39 William O'Connor Stocks, Wheat & Pharaohs J40 James J. O'Leary The outlook for Interest Rates in 1961 J41 Allan J. Braff & Roger F. Miller Wage‐Price Policies Under Public Pressure J42 Morton A. Kaplan ~ Nicholas

Katzenbach The Political Foundation of Internationa1 Law

J43 Wallace Mendelson Justices Black & Frankfurter J44 J. Mitchell Reese, Jr, Reorganization Transfers J45 Albert Schreiber et al. Defense Procurement & Small Business J46 Irving Perluss Agricultural Labor Disputes in California 1960 J47 William S. Ragan Teaching America's Children. J48 Paul Cooke Desegregated Education in the Middle‐South Region J49 Robert J. Havighurst Social‐Class Influences on American Education J50 James C. Bonbright Principles of Public Utility Rates J51 Irving L. Horowitz Philosophy, Science & the Sociology of Knowledge J52 Brand Blanshard The Emotive Theory J53 William S. Haymond Is Distance an Original Factor in Vision? J54 Chester G. Starr The Origins of Greek Civilization 1100‐650 B. C J55 Jim B. Pearson The Maxwell Land Grant J56 Edwin L. Bigelow & Nancy H. Otis Manchester, Vermont, A Pleasant Land J57 J. H. Hexter Thomas More: on the Margins of Modernity









































Hausman 89


File Author Source J58 John M, Ray Rhode Island's Reactions to John Brown's Raid J59 Clement Greenberg Collage J60 Robert A. Futterman The Future of Our Cities J61 Allyn Cox Completing & Restoring the Capitol Frescos J62 Jimmy Ernst A Letter to Artists of the Soviet Union J63 John H. Schaar Escape from Authority, Perspectives of Erich Fromm J64 Katherine G. McDonald Figures of Rebellion J65 Samuel Hynes The Pattern of Hardy's Poetry J66 Kenneth Rexroth Disengagament: The Art of the Beat Generation J67 William Whallon The Diction of Beowulf J68 Charles R. Forker The Language of Hands in Great Expectations J69 I. B. M. Corporation IBM 7070, Autocoder Reference Manual J70 Ross E. McKinney & Howard Edde Aerated Lagoon for Suburban Sewage Disposal J71 Thomas D. McGrath Submarine Defense J72 Mellon Institute Annual Report; 1960, Independent Research J73 Nat'l Research Council Directory of Continuing Numerical Data Projects J74 Harlan W. Nelson Food Preservation by Ionizing Radiation J75 W. K. Asbeck Forces in Coatings Removal Knife Cutting Method J76 Joel Frados, editor Survey of Foamed Plastics J77 William D. Appel, editor 1961 Technical Manual of American Ass'n of Textile

Chemists & Colorists J78 Paul J. Dolon & Wilfrid F. Niklas Gain & Resolution of Fiber Optic Intensifier J79 Rutherford Aris The O'ptim.A1 Design of Chemical Reactors J80 C. J. Savant Jr. & R. C. Howard Principles of Inertial Navigation K01

Christopher Davis First Family K02 Clayton C. Barbeau The Ikon K03 Tristram Coffin Not to the Swift K04 W. E. B. Du Bois Worlds of Color K05 David Stacton The Judges of the Secret Court K06 Louis Zara Dark Rider K07 Francis Pollini Night K08 Guy Endore Voltaire! Voltaire! K09 Howard Fast April Morning K10 Gladys H. Barr The Master of & Geneva K11 Robert Penn Warren Wilderness K12 Gerald Green The Heartless Light Kl3 William Maxwell The Chateau K14 Irving Stone The Agony & the Ecstasy K15 Ann Hebson The Lattimer Legend K16 Stephen Longstreet Eagles Where I Walk Kl7 Leon Uris Mila 8 K18 John Dos Passos Midcentury K19 Robert J Duncan The Voice of Strangers

http://icame.uib.no/brown/brownwww.html#K19










































Hausman 90


File Author Source K20 Guy Bolton The Olympians K21 Bruce Palmer My Brother's Keeper K22 John Cheever The Brigadier & the Golf Widow K23 Prieda Arkin The Tight of the Sea K24 W. H. Gass The Pedersen Kid K25 Arthur Miller The Prophecy K26 Jane G. Rushing Against the Moon K27 E. Lucas Myers The Vindication of Dr. Nestor K28 Sallie Bingham Moving Day K29 Marvin Schiller The Sheep's in the Meadow. L01

Winfred Van Atta Shock Treatment L02 A. A. Fair Bachelors Get Lonely L03 Amber Dean Encounter With Evil L04 David Alexander Bloodstain L05 Brett Halliday The Careless Corpse L06 Thomas B. Dewey Hunter at Large L07 Genevieve Golden Deadlier Than the Male L08 Dell Shannon The Ace of Spades L09 Mignon G. Eberhart The Cup, the Blade or the Swords L10 Harry Bleaker Impact L11 Hampton Stone The Man Who Looked Death in the Eye L12 Whit Masterson Evil Come, Evil Go L13 Dolores Hitchens Footsteps in the Night Ll4 Frances & Richard Lockridge Murder Has Its Points L15 Doris M. Disney Mrs. Meeker's Money L16 Alex Gordon The Cipher L17 Brent James Night of the Kill L18 George H. Coxe Error of Judgment L19 Brad Williams Make a Killing L20 Ed Lacy Death by the Numbers L21 Helen McCloy The Black Disk L22 S. L. M. Barlow Monologue of Murder L23 J. W. Rose Try My Sample Murders L24 Fredric Brown The Murders M01

Robert Heinlein Stranger in a Strange Land M02 Philip J. Farmer The Lovers M03 James Blish The Star Dwellers M04 Jim Harmon The Planet with No Nightmare M05 Anne McCaffrey The Ship Who Sang M06 Cordwainer Smith A Planet Named Shayol N0l

Wayne D. Overholser The Killer Marshall N02 Clifford Irving The Valley N03 Cliff Farrell The Trail of the Tattered Star

http://icame.uib.no/brown/brownwww.html#N03











http://icame.uib.no/brown/brownwww.html#L01
























http://icame.uib.no/brown/brownwww.html#M01








Hausman 91


File Author Source N04 James D. Horan The Shadow Catcher N05 Richard Ferber Bitter Valley N06 Thomas Anderson Here Comes Pete Now N07 Todhunter Ballard The Night Riders N08 Mary Savage Just for Tonight N09 Jim Thompson The Transgressors N10 Joseph Chadwick No Land Is Free N11 Gene Caesar Rifle for Rent N12 Edwin Booth Outlaw Town N13 Martha F. McKeown Mountains Ahead N14 Peter Field Rattlesnake Ridge N15 Donald J. Plantz Sweeney Squadron N16 Ralph J. Salisbury On the Old Sante Fe Trail to Siberia N17 Richard S. Prather The Bawdy Beautiful N18 Peter Bains With Women Education Pays off Nl9 David Jackson The English Gardens N20 T. C. McClary The Flooded Dearest N21 C. T. Sommers The Beautiful Mankillers of Eromonga N22 Gordon Johnson A Matter of Curiosity. N23 Wheeler Hall Always Shoot to Kill N24 T. K. Brown III The Fifteenth Station N25 Wesley Newton Aid & Comfort to the Enemy N26 Paul Brock Toughest Lawman in the Old West N27 James Hines & James Morris Just Any Girl N28 Ralph Grimshaw Mrs. Hacksaw, New Orleans Society Killer N29 Harlan Ellison Riding the Dark Train Out P01

Octavia Waldo A Cup of the Sun P02 Ann Ritner Seize a Nettle P03 Clark McMeekin The Fairbrothers P04 B. J. Chute The Moon & the Thorn P05 Allan R. Bosworth The Crows of Edwina Hill P06 Richard Tiernan Land of the Silver Dollar P07 Vina Delmar The Big Family P08 R. Leslie Course With Gall & Honey P09 Jesse Hill Ford Mountains of Gilead P10 Jay Williams The Forger P11 Bessie Breuer Take Care of My Roses P12 Morley Callaghan A Passion in Rome P13 Frank B. Hanes The Fleet Rabble P14 Livingston Biddle, Jr. Sam Bentley's Island P15 Loretta Burrough The Open Door P16 Margery F. Brown A Secret Between Friends P17 Al Hine The Huntress

http://icame.uib.no/brown/brownwww.html#P17











































Hausman 92


File Author Source P18 Anonymous No Room in My Heart to For Give Pl9 Anonymous This Cancer Victim May Ruin My Life P20 Spencer Norris Dirty Dog Inn P21 Elizabeth Spencer The White Azalea P22 Anonymous A Husband Stealer from Way Back P23 Barbara Robinson Something Very Much in Common P24 Samuel Elkin The Ball Player P25 William Butler The Pool at Ryusenji P26 Ervin D. Krause The Snake P27 Lee McGiffin Measure of a Man P28 Carol Hoover The Shorts on the Bedroom Floor P29 Robert Carson My Hero R01

Anita Loos No Mother to Guide Her R02 Jean Mercier Whatever You Do, Don't Panic R03 Patrick Dennis Little Me R04 Edward Streeter The Chairman of the Bored R05 Evan Esar Humorous English R06 James Thurber The Future, If Any, of Comedy R07 John H. Wildman Take It Off R08A Leo Lemon Catch Up With R08B Leo Lemon Something to Talk About R09 S. J. Perelman The Rising Gorge

Appendix B: Proportional Placement Statistics

The statistics below come from 40 randomly picked SemCor files. See Cost Function Method 3

in the Cost Function section for more details about these numbers. The POS‐POS column represents the

parts of speech of the word pair combination. (NN = Noun, JJ = Adjective, VB = Verb, RB = Adverb)

Table 6: The values for Cost Function Method 3 used in this project Semantic Relation POS‐POS Min Max Average Standard Deviation Frequency NN‐JJ 0.792254615 0.918299831 0.864021328 0.026870632 Frequency NN‐RB 0.808804114 0.919316349 0.862272345 0.027024865 Frequency NN‐NN 0.775888726 0.910315112 0.849278482 0.028957857 Frequency NN‐VB 0.775888726 0.910315112 0.849278482 0.028957857 Frequency NN‐VB 0.762541491 0.897634401 0.826850626 0.029391924 Frequency VB‐JJ 0.777054072 0.921952008 0.848030644 0.032030693 Hypernym VB‐VB 0 0.173931701 0.049567971 0.033316047 Frequency JJ‐RB 0.804506863 0.962140412 0.87887566 0.033752072

http://icame.uib.no/brown/brownwww.html#R09






















Hausman 93


Semantic Relation POS‐POS Min Max Average Standard Deviation Frequency VB‐RB 0.757327974 0.900365277 0.839290654 0.035019609 Frequency VB‐VB 0.760132026 0.896886676 0.827761357 0.036453501 Frequency JJ‐JJ 0.766452317 0.949310958 0.879606997 0.040958319 Coordinate Sister VB‐VB 0.031847134 0.246575342 0.110664492 0.042858687 Frequency RB‐RB 0.661223098 0.959398173 0.867375297 0.062517667 Hypernym NN‐NN 0.387156873 0.736300313 0.517819878 0.06645126 Coordinate Sister NN‐VB 0.03125 0.378378378 0.185552494 0.083965331 Domain NN‐VB 0 0.5 0.029727096 0.105596541 Synonym VB‐VB 0 0.5 0.183663818 0.107686637 Coordinate Sister NN‐NN 0.08 0.738095238 0.422957976 0.11296247 Synonym VB‐RB 0 1 0.01754386 0.131286224 Domain NN‐JJ 0 1 0.047660819 0.156454335 Domain JJ‐JJ 0 0.75 0.05380117 0.159678574 Coordinate Sister NN‐RB 0 1 0.046783626 0.170294968 Coordinate Sister VB‐RB 0 1 0.062238931 0.183549623 Antonym NN‐VB 0 1 0.035087719 0.184001552 Antonym VB‐JJ 0 1 0.035087719 0.184001552 Antonym VB‐RB 0 1 0.035087719 0.184001552 Antonym JJ‐RB 0 1 0.035087719 0.184001552 Domain NN‐RB 0 1 0.035087719 0.184001552 Domain NN‐NN 0 0.826086957 0.148698419 0.185235944 Coordinate Sister VB‐JJ 0 1 0.125798471 0.211082617 Domain RB‐RB 0 1 0.104720134 0.252556051 Synonym RB‐RB 0 1 0.109210526 0.264628929 Coordinate Sister NN‐JJ 0 1 0.244054581 0.272936939 Domain VB‐VB 0 1 0.105263158 0.277470436 Synonym NN‐VB 0 1 0.228777335 0.285901603 Synonym VB‐JJ 0 1 0.122807018 0.295394004 Synonym JJ‐JJ 0 1 0.545846949 0.311513247 Synonym JJ‐RB 0 1 0.131578947 0.331481467 Synonym NN‐NN 0 1 0.582170994 0.334616491 Antonym NN‐JJ 0 1 0.140350877 0.3473507 Antonym RB‐RB 0 1 0.152046784 0.353595707 Synonym NN‐JJ 0 1 0.216374269 0.384855752 Antonym VB‐VB 0 1 0.207602339 0.390282426 Antonym JJ‐JJ 0 1 0.469444444 0.423756446 Antonym NN‐NN 0 1 0.368788424 0.439226787

Hausman 94


Appendix C: Sense Distribution Statistics

The statistics below come from the “Brown 1” SemCor files. See Cost Function Method 4 in the

Cost Function section for more details about these numbers. Note that this table assumes the most

common sense is sense 0.

Table 7: The values for Cost Function Method 4 used in this project Sense Value Sense Value Sense Value 0 0.745002967 17 0.000179044 34 0 1 0.143447519 18 0.000144862 35 0 2 0.051918954 19 0.000114338 36 0 3 0.024731353 20 0.000141419 37 0 4 0.013769042 21 7.17999E‐05 38 0 5 0.006769587 22 4.29754E‐05 39 0 6 0.004881351 23 7.21787E‐05 40 0 7 0.002858328 24 1.13685E‐05 41 0 8 0.001849772 25 0 42 0 9 0.000993421 26 3.30413E‐05 43 0 10 0.000760418 27 1.15856E‐05 44 0 11 0.000593476 28 1.10957E‐05 45 0 12 0.000572185 29 1.18544E‐05 46 0 13 0.000436927 30 2.06378E‐05 47 0 14 0.000215302 31 0 48 0 15 0.000170978 32 0 49 0 16 0.000152483 33 9.73795E‐06

Appendix D: Semantic Relation Distribution Statistics

The statistics below come from 40 randomly picked SemCor files. See Cost Function Method 5

in the Cost Function section for more details about these numbers. Note that this table assumes the

most common sense is sense 0.

Table 8: The values for Cost Function Method 5 used in this project Type Sense Value Type Sense Value Frequency 0 0.781961482 Synonym 0 0.708127392 Frequency 1 0.126728106 Synonym 1 0.14722019 Frequency 2 0.043575442 Synonym 2 0.058309409

Hausman 95


Type Sense Value Type Sense Value Frequency 3 0.020242591 Synonym 3 0.035265127 Frequency 4 0.011330363 Synonym 4 0.02488885 Frequency 5 0.005382083 Synonym 5 0.008951525 Frequency 6 0.003820112 Synonym 6 0.006850447 Frequency 7 0.002217267 Synonym 7 0.003343709 Frequency 8 0.001386978 Synonym 8 0.002257055 Frequency 9 0.000719614 Synonym 9 0.000917186 Frequency 10 0.000550252 Synonym 10 0.000415363 Frequency 11 0.000428899 Synonym 11 0.001272953 Frequency 12 0.000401828 Synonym 12 0 Frequency 13 0.000303701 Synonym 13 0.000649926 Frequency 14 0.000142087 Synonym 14 0.000374687 Frequency 15 0.000122769 Synonym 15 0 Frequency 16 9.93788E‐05 Synonym 16 6.48281E‐05 Frequency 17 0.000126437 Synonym 17 0.000302171 Frequency 18 9.51612E‐05 Synonym 18 0.000120573 Frequency 19 7.45493E‐05 Synonym 19 0 Frequency 20 9.84838E‐05 Synonym 20 0.000268828 Frequency 21 4.59405E‐05 Synonym 21 5.53748E‐05 Frequency 22 2.64911E‐05 Synonym 22 0 Frequency 23 4.80187E‐05 Synonym 23 7.21131E‐05 Frequency 24 6.51078E‐06 Synonym 24 0 Frequency 25 0 Synonym 25 0 Frequency 26 2.37718E‐05 Synonym 26 0 Frequency 27 6.93931E‐06 Synonym 27 0 Frequency 28 6.74259E‐06 Synonym 28 0 Frequency 29 7.47323E‐06 Synonym 29 0 Frequency 30 1.47131E‐05 Synonym 30 0.000272292 Frequency 31 0 Synonym 31 0 Frequency 32 0 Synonym 32 0 Frequency 33 5.81201E‐06 Synonym 33 0 Frequency 34 0 Synonym 34 0 Frequency 35 0 Synonym 35 0 Frequency 36 0 Synonym 36 0 Frequency 37 0 Synonym 37 0 Frequency 38 0 Synonym 38 0 Frequency 39 0 Synonym 39 0 Frequency 40 0 Synonym 40 0 Frequency 41 0 Synonym 41 0

Hausman 96


Type Sense Value Type Sense Value Frequency 42 0 Synonym 42 0 Frequency 43 0 Synonym 43 0 Frequency 44 0 Synonym 44 0 Frequency 45 0 Synonym 45 0 Frequency 46 0 Synonym 46 0 Frequency 47 0 Synonym 47 0 Frequency 48 0 Synonym 48 0 Frequency 49 0 Synonym 49 0 Hypernym 0 0.780778412 Antonym 0 NaN Hypernym 1 0.127113255 Antonym 1 NaN Hypernym 2 0.045911925 Antonym 2 NaN Hypernym 3 0.021512378 Antonym 3 NaN Hypernym 4 0.010228877 Antonym 4 NaN Hypernym 5 0.006096897 Antonym 5 NaN Hypernym 6 0.003875595 Antonym 6 NaN Hypernym 7 0.001693082 Antonym 7 NaN Hypernym 8 0.001142567 Antonym 8 NaN Hypernym 9 0.000634825 Antonym 9 NaN Hypernym 10 0.000521345 Antonym 10 NaN Hypernym 11 0.000139175 Antonym 11 NaN Hypernym 12 0.000156363 Antonym 12 NaN Hypernym 13 0.000138005 Antonym 13 NaN Hypernym 14 0 Antonym 14 NaN Hypernym 15 2.79886E‐05 Antonym 15 NaN Hypernym 16 9.4065E‐06 Antonym 16 NaN Hypernym 17 1.02999E‐05 Antonym 17 NaN Hypernym 18 0 Antonym 18 NaN Hypernym 19 0 Antonym 19 NaN Hypernym 20 0 Antonym 20 NaN Hypernym 21 9.6045E‐06 Antonym 21 NaN Hypernym 22 0 Antonym 22 NaN Hypernym 23 0 Antonym 23 NaN Hypernym 24 0 Antonym 24 NaN Hypernym 25 0 Antonym 25 NaN Hypernym 26 0 Antonym 26 NaN Hypernym 27 0 Antonym 27 NaN Hypernym 28 0 Antonym 28 NaN Hypernym 29 0 Antonym 29 NaN Hypernym 30 0 Antonym 30 NaN

Hausman 97


Type Sense Value Type Sense Value Hypernym 31 0 Antonym 31 NaN Hypernym 32 0 Antonym 32 NaN Hypernym 33 0 Antonym 33 NaN Hypernym 34 0 Antonym 34 NaN Hypernym 35 0 Antonym 35 NaN Hypernym 36 0 Antonym 36 NaN Hypernym 37 0 Antonym 37 NaN Hypernym 38 0 Antonym 38 NaN Hypernym 39 0 Antonym 39 NaN Hypernym 40 0 Antonym 40 NaN Hypernym 41 0 Antonym 41 NaN Hypernym 42 0 Antonym 42 NaN Hypernym 43 0 Antonym 43 NaN Hypernym 44 0 Antonym 44 NaN Hypernym 45 0 Antonym 45 NaN Hypernym 46 0 Antonym 46 NaN Hypernym 47 0 Antonym 47 NaN Hypernym 48 0 Antonym 48 NaN Hypernym 49 0 Antonym 49 NaN Coordinate Sister 0 0.731642312 Domain 0 NaN Coordinate Sister 1 0.140189319 Domain 1 NaN Coordinate Sister 2 0.059664121 Domain 2 NaN Coordinate Sister 3 0.026321333 Domain 3 NaN Coordinate Sister 4 0.014048575 Domain 4 NaN Coordinate Sister 5 0.008563011 Domain 5 NaN Coordinate Sister 6 0.007495793 Domain 6 NaN Coordinate Sister 7 0.002041513 Domain 7 NaN Coordinate Sister 8 0.002195311 Domain 8 NaN Coordinate Sister 9 0.00124681 Domain 9 NaN Coordinate Sister 10 0.000714507 Domain 10 NaN Coordinate Sister 11 0.000830106 Domain 11 NaN Coordinate Sister 12 0.001219296 Domain 12 NaN Coordinate Sister 13 0.000939302 Domain 13 NaN Coordinate Sister 14 0.000651397 Domain 14 NaN Coordinate Sister 15 0.000227097 Domain 15 NaN Coordinate Sister 16 0.000161478 Domain 16 NaN Coordinate Sister 17 0.000455964 Domain 17 NaN Coordinate Sister 18 0.000287357 Domain 18 NaN Coordinate Sister 19 0.000232809 Domain 19 NaN

Hausman 98


Type Sense Value Type Sense Value Coordinate Sister 20 0.000430405 Domain 20 NaN Coordinate Sister 21 0 Domain 21 NaN Coordinate Sister 22 0 Domain 22 NaN Coordinate Sister 23 0.000109042 Domain 23 NaN Coordinate Sister 24 0 Domain 24 NaN Coordinate Sister 25 0 Domain 25 NaN Coordinate Sister 26 0 Domain 26 NaN Coordinate Sister 27 0 Domain 27 NaN Coordinate Sister 28 0.000117354 Domain 28 NaN Coordinate Sister 29 4.43262E‐05 Domain 29 NaN Coordinate Sister 30 0.000171463 Domain 30 NaN Coordinate Sister 31 0 Domain 31 NaN Coordinate Sister 32 0 Domain 32 NaN Coordinate Sister 33 0 Domain 33 NaN Coordinate Sister 34 0 Domain 34 NaN Coordinate Sister 35 0 Domain 35 NaN Coordinate Sister 36 0 Domain 36 NaN Coordinate Sister 37 0 Domain 37 NaN Coordinate Sister 38 0 Domain 38 NaN Coordinate Sister 39 0 Domain 39 NaN Coordinate Sister 40 0 Domain 40 NaN Coordinate Sister 41 0 Domain 41 NaN Coordinate Sister 42 0 Domain 42 NaN Coordinate Sister 43 0 Domain 43 NaN Coordinate Sister 44 0 Domain 44 NaN Coordinate Sister 45 0 Domain 45 NaN Coordinate Sister 46 0 Domain 46 NaN Coordinate Sister 47 0 Domain 47 NaN Coordinate Sister 48 0 Domain 48 NaN Coordinate Sister 49 0 Domain 49 NaN

Appendix E: SemCor Results

Below are the results for every SemCor file that tags nouns, verbs, adjectives, and adverbs. The

authors ignore all files only tagging verbs. See the results section for more details.

Hausman 99


Table 9: The results of one run of every SemCor file using all the parts of speech File Coverage Fine Recall Fine Precision Coarse Recall Coarse Precision Average 0.939458723 0.581803497 0.619021038 0.741617232 0.789075105 br‐a01 0.905273438 0.548828125 0.606256742 0.715820313 0.790722762 br‐a02 0.863770977 0.548864758 0.635428571 0.695952616 0.805714286 br‐a11 0.82650143 0.500476644 0.605536332 0.636796949 0.770472895 br‐a12 0.855847688 0.539437897 0.63029661 0.666364461 0.778601695 br‐a13 0.813155386 0.490943756 0.603751465 0.617731173 0.759671747 br‐a14 0.842799189 0.519269777 0.61612515 0.674442191 0.800240674 br‐a15 0.80563654 0.481049563 0.597104946 0.621963071 0.772014475 br‐b13 0.944649446 0.600553506 0.635742188 0.757380074 0.801757813 br‐b20 0.922244094 0.601377953 0.65208111 0.766732283 0.831376734 br‐c01 0.889622642 0.53490566 0.601272534 0.691509434 0.777306469 br‐c02 0.944139194 0.56959707 0.603297769 0.723443223 0.766246363 br‐c04 0.879882813 0.541992188 0.615982242 0.698242188 0.793562708 br‐d01 0.982758621 0.559837728 0.569659443 0.74137931 0.754385965 br‐d02 0.956435644 0.597029703 0.624223602 0.771287129 0.806418219 br‐d03 0.957913998 0.603842635 0.630372493 0.764867338 0.798471824 br‐d04 0.974180735 0.544190665 0.55861366 0.72591857 0.745158002 br‐e01 0.947725073 0.559535334 0.590398366 0.715392062 0.75485189 br‐e02 0.979648474 0.598519889 0.61095373 0.753931545 0.769593957 br‐e04 0.937725632 0.572202166 0.610202117 0.732851986 0.781520693 br‐e21 0.989266547 0.590339893 0.596745027 0.77549195 0.783905967 br‐e22 0.953929539 0.59168925 0.620265152 0.786238532 0.791320406 br‐e23 0.947569113 0.565300286 0.596579477 0.790856031 0.80734856 br‐e24 0.993577982 0.610091743 0.614035088 0.766634523 0.790258449 br‐e25 0.975297347 0.604757548 0.620075047 0.720038351 0.781477627 br‐e26 0.925343811 0.542239686 0.585987261 0.765409384 0.79313632 br‐e27 0.991517436 0.706880302 0.712927757 0.758325404 0.779843444 br‐e28 0.980786825 0.614821592 0.626865672 0.775047259 0.791505792 br‐e29 0.979571984 0.620622568 0.633565045 0.809297913 0.818618042 br‐e30 0.972122302 0.613309353 0.630897317 0.740186916 0.788059701 br‐e31 0.959821429 0.6 0.625116279 0.781683626 0.818005808 br‐f03 0.970106075 0.612343298 0.631212724 0.814629259 0.855789474 br‐f08 0.967654987 0.621743037 0.642525534 0.820781697 0.830279653 br‐f10 0.921380633 0.559923298 0.607700312 0.756397134 0.770594369 br‐f13 0.987985213 0.629390018 0.637043966 0.794545455 0.815298507 br‐f14 0.950654582 0.586102719 0.616525424 0.808173478 0.823979592 br‐f15 0.976325758 0.581439394 0.595538312 0.780943026 0.787128713 br‐f16 0.903970452 0.566020314 0.626149132 0.756141947 0.78028169

Hausman 100


File Coverage Fine Recall Fine Precision Coarse Recall Coarse Precision br‐f17 0.868937049 0.544891641 0.627078385 0.810859729 0.816773017 br‐f18 0.914201183 0.581854043 0.636461704 0.782073814 0.80835604 br‐f19 0.965041398 0.599816007 0.621544328 0.767419962 0.778414518 br‐f20 0.875121951 0.548292683 0.626532887 0.760144274 0.799810247 br‐f21 0.973256925 0.595988539 0.612365064 0.81736795 0.852474323 br‐f22 0.873441994 0.535953979 0.613611416 0.777484609 0.7829938 br‐f23 0.966730038 0.622623574 0.644051131 0.760338346 0.815524194 br‐f24 0.940186916 0.590654206 0.628230616 0.826778243 0.832350463 br‐f25 0.951871658 0.622994652 0.654494382 0.773743017 0.802898551 br‐f33 0.986486486 0.634169884 0.642857143 0.811270125 0.819331527 br‐f43 0.972407231 0.571836346 0.588062622 0.743830787 0.757177033 br‐f44 0.991877256 0.63267148 0.637852593 0.74829932 0.77 br‐g01 0.979206049 0.602079395 0.614864865 0.756849315 0.767361111 br‐g11 0.988614801 0.613851992 0.620921305 0.810360777 0.817927171 br‐g12 0.964717742 0.548387097 0.568443051 0.768376068 0.780381944 br‐g14 0.97790586 0.598463016 0.611984283 0.754681648 0.789422135 br‐g15 0.939252336 0.571962617 0.608955224 0.78639745 0.793991416 br‐g16 0.985808893 0.555345317 0.563339731 0.791338583 0.80239521 br‐g17 0.9453125 0.565429688 0.598140496 0.759962929 0.769953052 br‐g18 0.951661631 0.542799597 0.57037037 0.735576923 0.802728227 br‐g19 0.951866405 0.614931238 0.646026832 0.653039832 0.811197917 br‐g20 0.993269231 0.623076923 0.627299129 0.763653484 0.788910506 br‐g21 0.955555556 0.571428571 0.598006645 0.682555781 0.787134503 br‐g22 0.981768459 0.610756609 0.622098422 0.730808598 0.771891892 br‐g23 0.967615309 0.617271835 0.637931034 0.795620438 0.824196597 br‐g28 0.915151515 0.55959596 0.611479029 0.83262891 0.85355286 br‐g31 0.865927419 0.535282258 0.618160652 0.716666667 0.787385554 br‐g39 0.88334995 0.513459621 0.581264108 0.72338403 0.772588832 br‐g43 0.981354269 0.565260059 0.576 0.706126687 0.770101925 br‐g44 0.967615309 0.555446516 0.574036511 0.707556428 0.767021277 br‐h01 0.95559667 0.635522664 0.665053243 0.719836401 0.776185226 br‐h09 0.990224829 0.579667644 0.585389931 0.696886447 0.773373984 br‐h11 0.964749536 0.641001855 0.664423077 0.734513274 0.780258519 br‐h12 0.982010582 0.582010582 0.592672414 0.721584984 0.76974416 br‐h13 0.924599434 0.531573987 0.574923547 0.747288503 0.77765237 br‐h14 0.968430826 0.616527391 0.63662512 0.714697406 0.791489362 br‐h15 0.978947368 0.612280702 0.625448029 0.744918699 0.78987069 br‐h16 0.881709742 0.550695825 0.624577227 0.731549815 0.809183673 br‐h17 0.946666667 0.606153846 0.640303359 0.709876543 0.764119601

Hausman 101


File Coverage Fine Recall Fine Precision Coarse Recall Coarse Precision br‐h18 0.983350676 0.566077003 0.575661376 0.715820313 0.778958555 br‐h21 0.982342007 0.601301115 0.612109745 0.676441838 0.785471056 br‐h24 0.967213115 0.677254098 0.700211864 0.757236228 0.811 br‐j01 0.951903808 0.659318637 0.692631579 0.680705191 0.793378995 br‐j02 0.988560534 0.651096282 0.658630665 0.690140845 0.773393461 br‐j03 0.981576254 0.612077789 0.623566215 0.735439289 0.801937567 br‐j04 0.974545455 0.643636364 0.660447761 0.745694022 0.80349345 br‐j05 0.980817348 0.643869892 0.656462585 0.72037037 0.77029703 br‐j06 0.992141454 0.635559921 0.640594059 0.719626168 0.769145394 br‐j07 0.969062784 0.596906278 0.615962441 0.734032412 0.776209677 br‐j08 0.992760181 0.657918552 0.6627165 0.680467091 0.758579882 br‐j09 0.967486819 0.665202109 0.687556767 0.778474399 0.791710946 br‐j10 0.985875706 0.603578154 0.612225406 0.706766917 0.775943396 br‐j11 0.950405771 0.622182146 0.654648956 0.766144814 0.803076923 br‐j12 0.958818263 0.681289167 0.710550887 0.697368421 0.770693512 br‐j13 0.99296394 0.616534741 0.620903454 0.746421268 0.776595745 br‐j14 0.932330827 0.636278195 0.682459677 0.741573034 0.784017279 br‐j15 0.993305439 0.689539749 0.694187026 0.693501455 0.78831312 br‐j16 0.963687151 0.672253259 0.697584541 0.710659898 0.787401575 br‐j17 0.990161002 0.676207513 0.682926829 0.745711403 0.810307018 br‐j18 0.982373678 0.588719154 0.599282297 0.703858186 0.75 br‐j19 0.971817298 0.594752187 0.612 0.760692464 0.795527157 br‐j20 0.98630137 0.593607306 0.601851852 0.764981273 0.797851563 br‐j22 0.990749306 0.602220167 0.607843137 0.737588652 0.799121844 br‐j23 0.984615385 0.599145299 0.608506944 0.75123885 0.803817603 br‐j29 0.984587489 0.616500453 0.626151013 0.720772947 0.797860963 br‐j30 0.992102665 0.597235933 0.60199005 0.783619818 0.819238901 br‐j31 0.982741117 0.624365482 0.635330579 0.757904246 0.794507576 br‐j32 0.992031873 0.642430279 0.647590361 0.756911344 0.798792757 br‐j33 0.958105647 0.571948998 0.596958175 0.786825252 0.806754221 br‐j34 0.98280543 0.637104072 0.64825046 0.720039293 0.778131635 br‐j35 0.970509383 0.587131367 0.604972376 0.823751178 0.830798479 br‐j37 0.955992509 0.575842697 0.602350637 0.794144556 0.809701493 br‐j38 0.986336465 0.643894108 0.652813853 0.758992806 0.780758557 br‐j41 0.989266547 0.576923077 0.58318264 0.776785714 0.809302326 br‐j42 0.981532779 0.638042475 0.650047037 0.778077269 0.804085422 br‐j52 0.990435707 0.597236982 0.603004292 0.792051756 0.801683817 br‐j53 0.986220472 0.607283465 0.615768463 0.763343404 0.802966102 br‐j54 0.987025023 0.609823911 0.617840376 0.768939394 0.787584869

Hausman 102


File Coverage Fine Recall Fine Precision Coarse Recall Coarse Precision br‐j55 0.916346154 0.597115385 0.651626443 0.694367498 0.768130746 br‐j56 0.805031447 0.525157233 0.65234375 0.666666667 0.767220903 br‐j57 0.967984934 0.585687382 0.605058366 0.711045365 0.777777778 br‐j58 0.867139959 0.531440162 0.612865497 0.690731707 0.789297659 br‐j59 0.946775844 0.551688843 0.582702703 0.756446991 0.777232581 br‐j60 0.965328467 0.626824818 0.649338374 0.667305849 0.763995609 br‐j70 0.975486052 0.680473373 0.697573657 0.771863118 0.798426745 br‐k01 0.910185185 0.576851852 0.633774161 0.745794393 0.793240557 br‐k02 0.936311787 0.580798479 0.620304569 0.792335116 0.832397004 br‐k03 0.916926272 0.539979232 0.588901472 0.787644788 0.798434442 br‐k04 0.922473013 0.560353288 0.607446809 0.796028881 0.802547771 br‐k05 0.927402863 0.570552147 0.615214994 0.731854839 0.75862069 br‐k06 0.901098901 0.571428571 0.634146341 0.766570605 0.78388998 br‐k07 0.941371681 0.574115044 0.60987074 0.775780511 0.786948177 br‐k08 0.937434828 0.572471324 0.610678532 0.741210938 0.784090909 br‐k09 0.960954447 0.5835141 0.607223476 0.729103726 0.766137566 br‐k10 0.902977906 0.555235351 0.614893617 0.777996071 0.817337461 br‐k11 0.943089431 0.584349593 0.619612069 0.783653846 0.788964182 br‐k12 0.904059041 0.557195572 0.616326531 0.754497354 0.789590255 br‐k13 0.929012346 0.573045267 0.61683278 0.790337284 0.805013928 br‐k14 0.918945313 0.53515625 0.582359192 0.793915604 0.820486815 br‐k15 0.861192571 0.518084066 0.601589103 0.714141414 0.780353201 br‐k16 0.933706816 0.614379085 0.658 0.672379032 0.776484284 br‐k17 0.85798237 0.538687561 0.627853881 0.657028913 0.743792325 br‐k18 0.892354125 0.527162978 0.590755355 0.782139352 0.797 br‐k19 0.917077986 0.556762093 0.607104413 0.760549558 0.786004057 br‐k20 0.928064843 0.576494428 0.621179039 0.77028348 0.777887463 br‐k21 0.935185185 0.575 0.614851485 0.796846011 0.825961538 br‐k22 0.935617861 0.568016615 0.607103219 0.741798942 0.755387931 br‐k23 0.945662536 0.559580553 0.591733871 0.708765316 0.76656473 br‐k24 0.897027601 0.52866242 0.589349112 0.770659239 0.7957814 br‐k25 0.983281087 0.60815047 0.618490967 0.776315789 0.793010753 br‐k26 0.91084855 0.566058002 0.621462264 0.686878728 0.77903044 br‐k27 0.954011742 0.62035225 0.65025641 0.762051282 0.804983749 br‐k28 0.9048583 0.557692308 0.616331096 0.780437045 0.793650794 br‐k29 0.961145194 0.583844581 0.607446809 0.787174721 0.801324503 br‐l08 0.87270974 0.531340405 0.608839779 0.793032787 0.819915254 br‐l09 0.859035005 0.509933775 0.593612335 0.803263826 0.815837937 br‐l10 0.865306122 0.52244898 0.603773585 0.801579467 0.807960199

Hausman 103


File Coverage Fine Recall Fine Precision Coarse Recall Coarse Precision br‐l11 0.945863126 0.576098059 0.609071274 0.769543147 0.783057851 br‐l12 0.879728419 0.537342386 0.610804851 0.770916335 0.777108434 br‐l13 0.912641316 0.535457348 0.586711712 0.734972678 0.767110266 br‐l14 0.905365854 0.545365854 0.60237069 0.812669683 0.826887661 br‐l15 0.869 0.508 0.584579977 0.788203753 0.812154696 br‐l16 0.891069676 0.504416094 0.566079295 0.806148591 0.817316017 br‐l17 0.873891626 0.534975369 0.612175874 0.779964222 0.788426763 br‐l18 0.914036997 0.528835691 0.578571429 0.807017544 0.822201317 br‐m01 0.902538071 0.566497462 0.627671541 0.670202507 0.767955801 br‐m02 0.920282543 0.596367306 0.648026316 0.660359508 0.768722467 br‐n05 0.938477581 0.549530761 0.585555556 0.655102041 0.757075472 br‐n09 0.903664921 0.561256545 0.621089224 0.70709147 0.774774775 br‐n10 0.869047619 0.537698413 0.618721461 0.708292683 0.782327586 br‐n11 0.923149016 0.555763824 0.602030457 0.671 0.772151899 br‐n12 0.835834897 0.489681051 0.585858586 0.648675172 0.727973568 br‐n14 0.895699909 0.507776761 0.566905005 0.670935961 0.767756483 br‐n15 0.907246377 0.543961353 0.599574015 0.693144723 0.758333333 br‐n16 0.892792793 0.567567568 0.635721493 0.710994764 0.786790267 br‐n17 0.919753086 0.553497942 0.601789709 0.68452381 0.787671233 br‐n20 0.971652004 0.596285435 0.613682093 0.714151828 0.773604061 br‐p01 0.956211813 0.585539715 0.612353568 0.636022514 0.760942761 br‐p07 0.878235858 0.534995206 0.609170306 0.666971638 0.744637385 br‐p09 0.951025057 0.563781321 0.592814371 0.68115942 0.750798722 br‐p10 0.921686747 0.524096386 0.568627451 0.730630631 0.818365288 br‐p12 0.925213675 0.536324786 0.579676674 0.709876543 0.771812081 br‐p24 0.82792527 0.50245821 0.606888361 0.765395894 0.787726358 br‐r04 0.919491525 0.558262712 0.607142857 0.705656759 0.80349345 br‐r05 0.958801498 0.602996255 0.62890625 0.742596811 0.780838323 br‐r06 0.922998987 0.558257345 0.604829857 0.710843373 0.77124183 br‐r07 0.934588702 0.585728444 0.626723224 0.68482906 0.740184758 br‐r08 0.903381643 0.580676329 0.642780749 0.657817109 0.794536817 br‐r09 0.956521739 0.646107179 0.675475687 0.702330508 0.763824885

Hausman 104


References

ACL‐SIGLEX. (2011, March). Retrieved from SENS EVAL: http://www.senseval.org

Basile, P., Degemmis, M., Gentile, A. L., Lops, P., & Semeraro, G. (2007). The JIGSAW Algorithm for Word Sense Disambiguation and Semantic Indexing of Documents. Proceedings of the 10th Congress of the Italian Association for Artificial Intelligence on AI*IA 2007: Artificial Intelligence and Human‐Oriented Computing, (pp. 314‐325). Rome, Italy.

Billot, M. (2010). Combing Lexical Resources for Text Analysis of Game Walkthroughs. Summer 2010 National Science Foundation REU. Colorado Springs.

Bosch, B. D., Hoste, V., Daelemans, W., & Den, W. V. (2004). GAMBL, Genetic Algorithm Optimization of Memory‐Based WSD. Proceedings of ACL/SIGLEX Senseval‐3, (pp. 108‐112).

Gerber, M. (2010, October). Retrieved from NLP Software: http://links.cse.msu.edu:8000/members/matt_gerber/index.php/NLP_software#WordNet_API

Google. (2010). Retrieved from Google Translate: http://translate.google.com/?hl=en&tab=wT#

Hausman, M. (2009). A Dominant Gene Genetic Algorithm for a Substitution Cipher in Cryptography. CS591 at the University of Colorado. Colorado Springs.

Hausman, M. (2009). A Dominant Gene Genetic Algorithm for a Transposition Cipher in Cryptography. CS‐571 at the University of Colorado. Colorado Springs.

Jiang , J. J., & Conrath, D. W. (1997). Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy . In International Conference Research on Computational Linguistics , (pp. 19‐33).

Leacock, C., & Chodorow, M. (1998). Combining Local Context and WordNet Similarity for Word Sense Identification. In C. Fellbaum (Ed.), WordNet: An Electronic Lexical Database (pp. 305‐332). MIT Press.

Lesk, M. (1986). Automatic Sense Disambiguation using Machine Readable Dictionaries: How to Tell a Pine Cone from an Ice Cream Cone. Proceedings of the 5th Annual International Conference on Systems Documentation, (pp. 24‐26). Toronto, Ontario, Canada.

Lin, D. (1998). An Information‐Theoretic Definition of Similarity. In Proceedings of the 15th International Conference on Machine Learning, (pp. 296‐304).

Mihalcea, R., & Csomai, A. (2005). SenseLearner: Word Sense Disambiguation for All Words in Unrestricted Text. Proceedings of the ACL 2005 on Interactive Poster and Demonstration Sessions, (pp. 53‐56). Stroudsburg, PA.

Miller, G. A. (n.d.). WordNet: A Lexical Database for English. Communications of the ACM, 38(11), 39‐41.

Hausman 105


Navigli, R. (2007). SemEval‐2007 Task 07: Coarse‐Grained English All‐Words Task. SemEval 2007.

Patwardhan, S. (2006). Using WordNet‐Based Context Vectors to Estimate the Semantic Relatedness of Concepts. Proceedings of the EACL, (pp. 1‐8).

Patwardhan, S., Banerjee, S., & Pedersen, T. (2002). An Adapted Lesk Algorithm for Word Sense Disambiguation Using WordNet. Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing, (pp. 136‐145). London, UK.

Patwardhan, S., Banerjee, S., & Pedersen, T. (2003). Using Measures of Semantic Relatedness for Word Sense Disambiguation. Proceedings of the Fourth International Conference on Intelligent Text Processing and Computational Linguistics, (pp. 241‐257). Mexico City, Mexico.

Patwardhan, S., Banerjee, S., & Pedersen, T. (2007). UMND1: Unsupervised Word Sense Disambiguation using Contextual Semantic Relatedness. Proceedings of the 4th International Workshop on Semantic Evaluations, (pp. 390‐393). Prague, Czech Republic.

Patwardhan, S., Banerjee, S., & Pedersen, T. (2007). UMND1: Unsupervised Word Sense Disambiguation Using Contextual Semantic Relatedness. Proceedings of the 4th International Workshop on Semantic Evaluations, (pp. 390‐393). Prague, Czech Republic.

Princeton University. (2010). About WordNet. Retrieved from About WordNet: http://wordnet.princeton.edu

Princeton University. (2011, March). Retrieved from The SemCor Corpus: http://multisemcor.fbk.eu/semcor.php

Rada, R., Mili, H., Bicknell, E., & Blettner, M. (1989, Jan/Feb). Development and Application of a Metric on Semantic Nets. Systems, Man and Cybernetics, IEEE Transactions, 19(1), 17‐30.

Raytheon BBN Technologies, the University of Colorado, the University of Pennsylvania, and the University of Southern California's Information Sciences Institute . (2011, March). Retrieved from OntoNotes: http://www.bbn.com/ontonotes/

Resnik, P. (1995). Using Information Content to Evaluate Semantic Similarity in a Taxonomy. Proceedings of the 14th International Joint Conference on Artificial Intelligence, (pp. 448‐453).

Snyder, B., & Palmer, M. (2004). The English all‐words task. Proceedings of the 3rd International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, (pp. 41‐43). Barcelona, Spain.

Tsuruoka, Y., & Tsujii, J. (2005). Bidirectional Inference with the Easiest‐First Strategy for Tagging Sequence Data. HLT/EMNLP, (pp. 467‐474).

Wu, Z., & Palmer, M. S. (1994). Verb Semantics and Lexical Selection. Proceedings of the 32th Annual Meeting on Association for Computational Linguistics, (pp. 133‐138). Las Cruces, New Mexico.

Hausman 106


Zesch, T., & Gurevich, I. (2010). Wisdom of Crowds Versus Wisdom of Linguists: Measuring the Semantic Relatedness of Words. Natural Language Engineering, 16(01), 1351‐3249.

Zhang, C., Zhou, Y., & Martin, T. (2008). Genetic Word Sense Disambiguation Algorithm. Intelligent Information Technology Applications, (pp. 123‐127).

Zhong, Z., Ng, H. T., & Chan, Y. S. (2008). Word sense disambiguation using OntoNotes: an empirical study. Proceedings of the Conference on Empirical Methods in Natural Language Processing, (pp. 1002‐‐1010). Honolulu, Hawaii.