What computers can and cannot do for lexicography or Us precision, them recall

1

What computers can and cannot do for lexicography

or

Us precision, them recall

Adam Kilgarriff

Lexicography Masterclass Ltd

and

University of Brighton, UK

27-29 Aug 2003 Adam Kilgarriff: Us precision them recall 2

Outline Precision and recall History of corpus lexicography Natural Language Processing Cyborgs


Find me all the fat cats

a request for information


High recall

Lots of responses Maybe not all good


High precision

Fewer hits Higher confidence



Recall Precision

Computers good bad

People bad good



True in many areas– web searching, google– finding an image to illustrate a talk

Nowhere more so than

lexicography


Lexicography: finding facts about words

collocations grammatical patterns idioms synonyms antonyms meanings translations




Four ages of corpus lexicography


Age 1:Precomputer

Oxford English Dictionary:• 5 million index cards


Age 2: KWIC Concordances From 1980 Computerised COBUILD project was innovator asian-kwic.html the coloured-pens method


Age 2: limitations

as corpora get bigger:too much data

• 50 lines for a word: :read all • 500 lines: could read all, takes a long time,

slow • 5000 lines: no


Age 3: Collocation statistics

Problem:too much data - how to summarise?

Solution:list of words occurring in neighbourhood of headword, with frequencies

Sorted by salience


Collocation listingFor right collocates of save (>5 hits)

word fr(x+y) fr(y) word fr(x+y) fr(y)

forests 6 170 life 36 4875

$1.2 6 180 dollars 8 1668

lives 37 1697 costs 7 1719

enormous 6 301 thousands 6 1481

annually 7 447 face 9 2590

jobs 20 2001 estimated 6 2387

money 64 6776 your 7 3141


Collocation statistics Which words?

– next word – last word – window, +1 to +5; window, -5 to -1

How sorted? most common collocates --but for most

nouns it's the most salient collocates --how to

measure salience?


Mutual Information

Church and Hanks 1989 How much more often does a word pair

occur, than one might expect by chance “Chance” of x and y occurring together:

p(x) * p(y) Probabilities approximated by

frequencies

p(x) =(approx) f(x)/N


Mutual Information

X fr eat fr X fr eat X

MI* rank

it 1000 400,000

404 1/

1M

3

meat 1000 6000 136 23/

1M

2

sushi 1000 100 5 50/

1M

1

* numbers are log-proportional to MI


Problem mathematical salience = lexicographic

salience? no! higher-frequency items are

lexicographically more salient Solution multiply MI by raw frequency


Mutual Information

X fr eat fr X fr eat X

MI rank

it 1000 400,000

404 1/

1M

3

meat 1000 6000 136 23/

1M

2

sushi 1000 100 5 50/

1M

1

MI x fr

new rank

400/M

3

3128/M

1

2500/M

2


Collocation listingFor right collocates of save (>5 hits)

word fr(x+y) fr(y) word fr(x+y) fr(y)

forests 6 170 life 36 4875

$1.2 6 180 dollars 8 1668

lives 37 1697 costs 7 1719

enormous 6 301 thousands 6 1481

annually 7 447 face 9 2590

jobs 20 2001 estimated 6 2387

money 64 6776 your 7 3141


Age-3 collocation statistics: limitations

Lists contain junk unsorted for type --MI lists mix adverbs,

subjects, objects, prepositions

What we really want: noise-free lists one list for each grammatical relation


Age 4: The word sketch Large well-balanced corpus Parse to find

– subjects, objects, heads, modifiers etc

One list for each grammatical relation Statistics to sort each list, as before


Can we do it? high-accuracy parsing is hard lots of NLP work, many parsing frameworks

exist if any parser can handle large corpus, it's

probably good enough--- sorting, statistics, make us error-tolerant


Can we do it? high-accuracy parsing is hard lots of NLP work, many parsing frameworks

exist if any parser can handle large corpus, it's

probably good enough--- sorting, statistics, make us error-tolerant

Poor man’s parsing:– object (of active verb) = last noun in any sequence

of nouns, adjectives, determiners, numbers and adverbs following the verb


The word sketch coffee_n.html


Macmillan Dictionary of English for Advanced Leaners, 2002: editor: Rundell. Work done 1999.

Word sketches produced for 6000 most common nouns, verbs, adjectives of English

using British National Corpus (100 M words, already POS-tagged)

lemmatized using John Carroll's lemmatizer parsed using regular expressions over POS-tags HTML files with hyperlinked corpus examples lexicographers used them extensively, used instead

of going direct to corpus positive feedback




Natural Language Processing

The academic discipline which provides the tools– Also known as Computational Linguistics,

Human Language Technology (HLT), Language Engineering

Good at evaluation of its tools Good news for lexicography:

– identify the best tools, apply them to our corpora


An Anglophone Apology Technology, tools, resources most often

available for English This talk centres on English Other languages often present new

problems– Finding word delimiters for Chinese is hard– Finding bunsetsu for Japanese is hard

Fewer resources available, less work done Recommendation:

– find the local experts for your language


Recap: Lexicography: finding facts about words

collocations grammatical patterns idioms synonyms antonyms meanings translations


Recap: Lexicography: finding facts about words

collocations - sketches grammatical patterns - sketches idioms synonyms antonyms meanings translations


Idioms Extreme case of collocation/multi word

expressions Sequence of workshops on

collocations, MWE Technical terms (of great interest to

technologists, technical): TERMIGHT


Antonyms Essential semantic relation


Antonyms Essential semantic relation

but Justeson and Katz 1995: distributional

evidence for typical antonym pairs– rich men and poor men– the big ones and the small ones– black and white issues

Perhaps antonyms are ‘really’ distributional


Thesauruses Also near-synonyms

– are there any true synonyms? Distributional: which words share same

distributions– if corpus contains

object(drink, wine), object(drink, beer)

– 1 pt similarity between wine and beer– gather all points; find nearest neighbours

Sparck Jones, Lin, Grefenstette


Nearest neighbours


Translation Parallel corpora

– Texts and their translations or Comparable corpora

– Matched for source and target (genre and subject matter), not translations

Which L1 words occur in equivalent L1 settings to L2 words in L2 settings?– They are candidate translation pairs

Very hard problem Lots of high quality research


The WASPbench with David Tugwell, supported by UK EPSRC, grant

M54971 A lexicographer's workbench runtime creation of word sketches integration with Word Sense Disambiguation technology output is "disambiguating dictionary" - analysis of

word's meaning into senses, plus computer program for disambiguating contextualised instances of the word

First release now available. http://wasps.itri.brighton.ac.uk/ Sketches at http://www.itri.brighton.ac.uk

/~Adam.Kilgarriff/wordsketches.html

http://wasps.itri.brighton.ac.uk/





http://www.itri.brighton.ac.uk/~Adam.Kilgarriff/wordsketches.html








The Sketch Engine Input:

– any corpus, any language Lemmatised, part-of-speech tagged

– specification of grammatical relations Word sketches integrated with Corpus query system

– Supports complex searching, sorting etc First release early 2004




Cyborgs Robots: will they take over? Rod Brooks’s answer:

– Wrong question: greatest advances are in what the human+computer ensemble can do


Cyborgs A creature that is partly human and

partly machine – Macmillan English Dictionary






Cyborgs and the Information Society

The dictionary-making agent is part human (for precision), part computer (for recall).


Treat your computer with respect. You and it can do

great things together.


Lexicographers of the future?

Documents

What computers can and cannot do for lexicography or Us precision, them recall