Counting Words: Introductionzipfr.r-forge.r-project.org/.../ESSLLI/01_introduction.slides.pdf ·...

Introduction

Baroni & Evert

Roadmap

Lexical statistics:the basics

Zipf’s law

Typical frequencypatterns

Zipf’s law

Consequences

Applications

Productivity inmorphology

Productivitybeyond morphology

Lexical richness

Conclusion andoutlook

Counting Words:Introduction

Marco Baroni & Stefan Evert

Malaga, 7 August 2006

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Roadmap

I Introduction and motivation

I LNRE modeling: soft

I LNRE modeling: hard

I Playtime!

I The bad news and outlook

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Roadmap

I Playtime!

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Roadmap

I Playtime!

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Roadmap

I Playtime!

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Roadmap

I Playtime!

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Outline

Roadmap

Lexical statistics: the basics

Zipf’s law

Applications

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Lexical statisticsZipf 1949/1961, Baayen 2001, Evert 2005

I Statistical study of distribution of types (words andother units) in texts

I Different from other categorical data because of extremerichness of types

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Basic terminology

I N: sample/corpus size, number of tokens in the sample

I V : vocabulary size, number of distinct types in thesample

I Vm: type count of spectrum element m, number oftypes in the sample with token frequency m

I V1: hapax legomena count, number of types that occuronly once in the sample (for hapaxes, Count(types) =Count(tokens))

I A sample: a b b c a a b a

I N: 8; V : 3; V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Basic terminology

I N: 8; V : 3; V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Basic terminology

I N: 8; V : 3; V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Basic terminology

I N: 8; V : 3; V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Basic terminology

I N: 8; V : 3; V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Basic terminology

I N: 8; V : 3; V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Rank/frequency profile

I The sample: a b b c a a b a d

I Frequency list ordered by decreasing frequency

a 4b 3c 1d 1

I Replace type labels with ranks to obtain rank/frequencyprofile:

1 42 33 14 1

I Allows expression of frequency in function of rank of type

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

a 4b 3c 1d 1

1 42 33 14 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

a 4b 3c 1d 1

1 42 33 14 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

a 4b 3c 1d 1

1 42 33 14 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Rank/frequency profile of Brown corpus

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Frequency spectrum

I Frequency classes: 1 (c, d), 3 (b), 4 (a)

I Frequency spectrum:

1 23 14 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Frequency spectrum

1 23 14 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Frequency spectrum

1 23 14 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Rank/frequency profiles and frequency spectra

I From rank/frequency profile to spectrum: countoccurrences of each f in profile to obtain Vf values ofcorresponding spectrum elements

I From spectrum to rank/frequency profile: given highest f(i.e., m) in a spectrum, the ranks 1 to Vf in thecorresponding rank/frequency profile will have frequencyf , the ranks Vf + 1 to Vf + Vg (where g is the secondhighest frequency in the spectrum) will have frequency g ,etc.

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Frequency spectrum of Brown corpus

1 2 3 4 5 6 7 8 9 11 13 15

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Vocabulary growth curve

I The sample: a b b c a a b a

I N: 1, V : 1, V1: 1

I N: 3, V : 2, V1: 1

I N: 5, V : 3, V1: 1

I N: 8, V : 3, V1: 1

I (Most VGCs on our slides smoothed with binomialinterpolation)

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

I N: 1, V : 1, V1: 1

I N: 3, V : 2, V1: 1

I N: 5, V : 3, V1: 1

I N: 8, V : 3, V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

I N: 1, V : 1, V1: 1

I N: 3, V : 2, V1: 1

I N: 5, V : 3, V1: 1

I N: 8, V : 3, V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

I N: 1, V : 1, V1: 1

I N: 3, V : 2, V1: 1

I N: 5, V : 3, V1: 1

I N: 8, V : 3, V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

I N: 1, V : 1, V1: 1

I N: 3, V : 2, V1: 1

I N: 5, V : 3, V1: 1

I N: 8, V : 3, V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

I N: 1, V : 1, V1: 1

I N: 3, V : 2, V1: 1

I N: 5, V : 3, V1: 1

I N: 8, V : 3, V1: 1

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Vocabulary growth curve of Brown corpusWith V1 growth in red

0e+00 2e+05 4e+05 6e+05 8e+05 1e+06

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Outline

Roadmap

Zipf’s law

Applications

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Typical frequency patternsTop and bottom ranks in the Brown corpus

top frequencies bottom frequenciesrank fq word rank range fq randomly selected examples1 62642 the 7967-8522 10 recordings undergone privileges2 35971 of 8523-9236 9 Leonard indulge creativity3 27831 and 9237-10042 8 unnatural Lolotte authenticity4 25608 to 10043-11185 7 diffraction Augusta postpone5 21883 a 11186-12510 6 uniformly throttle agglutinin6 19474 in 12511-14369 5 Bud Councilman immoral7 10292 that 14370-16938 4 verification gleamed groin8 10026 is 16939-21076 3 Princes nonspecifically Arger9 9887 was 21077-28701 2 blitz pertinence arson10 8811 for 28702-53076 1 Salaries Evensen parentheses

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Typical frequency patternsBNC

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Typical frequency patternsOther corpora

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Typical frequency patternsBrown bigrams and trigrams

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Typical frequency patternsThe Italian prefix ri- in the la Repubblica corpus

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s law

I Language after language, corpus after corpus, linguistictype after linguistic type. . .

I same “few giants, many dwarves” pattern is encountered

I Similarity of plots suggests that relation between rankand frequency could be captured by a law

I Nature of relation becomes clearer if we plot log f infunction of log r

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s law

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s law

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s law

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s law

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s law

I Straight line in double-logarithmic space corresponds topower law for original variables

I This leads to Zipf’s (1949, 1965) famous law:

f (w) =C

I With a = 1 and C = 60, 000, Zipf’s law predicts thatmost frequent word has frequency 60,000; second mostfrequent word has frequency 30,000; third word hasfrequency 20,000. . .

I and long tail of 80,000 words with frequency between 1.5and 0.5

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s law

f (w) =C

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s law

f (w) =C

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s law

f (w) =C

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s lawLogarithmic version

I Zipf’s power law:

f (w) =C

I If we take logarithm of both sides, we obtain:

log f (w) = log C − a log r(w)

I I.e., Zipf’s law predicts that rank/frequency profiles arestraight lines in double logarithmic space, which, we saw,is a reasonable approximation

I Best fit a and C can be found with least squares methodI Provides intuitive interpretation of a and C :

I a is slope determining how fast log frequency decreaseswith log rank

I log C is intercept, i.e., predicted log frequency of wordwith rank 1 (log rank 0), i.e., most frequent word

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

f (w) =C

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

f (w) =C

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

f (w) =C

I Best fit a and C can be found with least squares method

I Provides intuitive interpretation of a and C :I a is slope determining how fast log frequency decreases

with log rankI log C is intercept, i.e., predicted log frequency of word

with rank 1 (log rank 0), i.e., most frequent word

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

f (w) =C

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf’s lawFitting the Brown rank/frequency profile

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Fit of Zipf’s law

I At right edge (low frequencies):

I “Bell-bottom” pattern expected as we are fittingcontinuous model to discrete frequencies

I More worryingly, in large corpora frequency drops morerapidly than predicted by Zipf’s law

I At left edge (high frequencies):I Highest frequencies lower than predicted → Mandelbrot’s

correction

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Fit of Zipf’s law

I At right edge (low frequencies):I “Bell-bottom” pattern expected as we are fitting

continuous model to discrete frequencies

I More worryingly, in large corpora frequency drops morerapidly than predicted by Zipf’s law

correction

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Fit of Zipf’s law

continuous model to discrete frequenciesI More worryingly, in large corpora frequency drops more

rapidly than predicted by Zipf’s law

correction

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Fit of Zipf’s law

I At left edge (high frequencies):

I Highest frequencies lower than predicted → Mandelbrot’scorrection

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Fit of Zipf’s law

I At left edge (high frequencies):I Highest frequencies lower than predicted

→ Mandelbrot’scorrection

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Fit of Zipf’s law

correction

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Zipf-Mandelbrot’s lawMandelbrot 1953

I Mandelbrot’s extra parameter:

f (w) =C

(r(w) + b)a

I Zipf’s law is special case with b = 0I Assuming a = 1, C = 60, 000, b = 1:

I For word with rank 1, Zipf’s law predicts frequency of60,000; Mandelbrot’s variation predicts frequency of30,000

I For word with rank 1,000, Zipf’s law predicts frequencyof 60; Mandelbrot’s variation predicts frequency of 59.94

I No longer a straight line in double logarithmic space;finding best fit harder than least squares

I Zipf-Mandelbrot’s law is basis of LNRE statistical modelswe will introduce

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

f (w) =C

(r(w) + b)a

I Zipf’s law is special case with b = 0

I Assuming a = 1, C = 60, 000, b = 1:I For word with rank 1, Zipf’s law predicts frequency of

60,000; Mandelbrot’s variation predicts frequency of30,000

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

f (w) =C

(r(w) + b)a

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

f (w) =C

(r(w) + b)a

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

f (w) =C

(r(w) + b)a

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Mandelbrot’s adjustmentFitting the Brown rank/frequency profile

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

More fits

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

A few mildly interesting things aboutZipf(-Mandelbrot)’s law

I a is often close to 1 for word frequency distributions(hence simplified version: f = C/r , and -1 slope inlog-log space)

I Zipf’s law also provides good fit to frequency spectra

I Monkey languages display Zipf’s law (intuition: few shortwords have very high chances to be generated; long tailof highly unlikely long words)

I Zipf’s law is everywhere (Li 2002)

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Consequences

I Data sparseness

I Standard statistics, normal approximation notappropriate for lexical type distributions

I V is not stable, will grow with sample size, we needspecial methods to estimate V and related quantities atarbitrary sizes (including V of whole type population)

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Consequences

I Data sparseness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Consequences

I Data sparseness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Consequences

I Data sparseness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

V , sample size and the Zipfian distribution

I Significant tail of hapax legomena indicates that chancesof encountering new type if we keep sampling are high

I Zipfian distribution implies vocabulary curve that is stillgrowing at largest sample size

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Pronouns in Italian (la Repubblica)Rank/frequency profile

●●●●●●●●●●

●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●

●●●

●●

0 20 40 60 80

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Pronouns in ItalianFrequency spectrum

●● ● ● ●●●●●● ●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●●●●●●●●●●●●●●●●●●●●●●●

1 100 10000

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Pronouns in ItalianVocabulary growth curve

0e+00 1e+06 2e+06 3e+06 4e+06

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Pronouns in ItalianVocabulary growth curve (zooming in)

0 2000 4000 6000 8000 10000

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

ri- in Italian (la Repubblica)Rank/frequency profile

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

ri- in ItalianFrequency spectrum

1 2 3 4 5 6 7 8 9 11 13 15

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

ri- in ItalianVocabulary growth curve

0 200000 600000 1000000

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Outline

Roadmap

Zipf’s law

Applications

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Applications

I Productivity (in morphology and elsewhere)

I Lexical richness (in stylometry, languageacquisition/pathology and elsewhere)

I Extrapolation of type counts and type frequencydistribution for practical NLP purposes (e.g., estimatingproportion of OOV words, typos, etc.)

I ... (e.g., Good-Turing smoothing, prior distribution forBayesian language modeling)

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Productivity

I In many linguistic problems, rate of growth of VGC isinteresting issue in itself

I Baayen (1989 and later) makes link between linguisticnotion of productivity and vocabulary growth rate

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Productivity in morphology: the classic definitionSchultink (1961), translated by Booij

Productivity as morphological phenomenon is the possibilitywhich language users have to form an in principle uncountablenumber of new words unintentionally, by means of amorphological process which is the basis of the form-meaningcorrespondence of some words they know.

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

V as a measure of productivity

I Comparable for same N only!

I Good first approximation, but it is measuringattestedness, not potential:

I (According to rough BNC counts) de- verbs have V of141, un- verbs have V of 119, contra our intuition

I We want productivity index of pronouns to be 0, not 72!

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

I Comparable for same N only!I Good first approximation, but it is measuring

attestedness, not potential:

I (According to rough BNC counts) de- verbs have V of141, un- verbs have V of 119, contra our intuition

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

attestedness, not potential:I (According to rough BNC counts) de- verbs have V of

141, un- verbs have V of 119, contra our intuition

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

attestedness, not potential:I (According to rough BNC counts) de- verbs have V of

141, un- verbs have V of 119, contra our intuitionI We want productivity index of pronouns to be 0, not 72!

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Baayen’s P

I Operationalize productivity of a process as probabilitythat the next token created by the process that wesample is a new word

I This is same as probability that next token in sample ishapax legomenon

I Thus, we can estimate probability of sampling a newword as relative frequency of hapax legomena in oursample:

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Baayen’s P

I Probability to sample token representing type we willnever encounter again (token labeled “hapax”) at firststage of sampling (when we are at the beginning ofN-token-sample) is given by the proportion of hapaxes inthe whole N-token-sample divided by the total number oftokens in the sample

I Thus, this must also be probability that last tokensampled represents new type

I P as productivity measure matches intuition thatproductivity should measure potential of process togenerate new forms

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

P as vocabulary growth rate

I P measures the potentiality of growth of V in a veryliteral way, i.e., it is the growth rate of V , the rate atwhich vocabulary size increases

I P is (approximation to) the derivative of V at N, i.e.,the slope of the tangent to the vocabulary growth curveat N (Baayen 2001, pp. 49-50)

I Again, “rate of growth” of vocabulary generated by wordformation process seems good match for intuition aboutproductivity of word formation process

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

ri- in Italian la Repubblica corpus

0 200000 600000 1000000 1400000

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Pronouns in Italian la Repubblica corpus

0 2000 4000 6000 8000 10000

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Baayen’s P and intuition

class V V1 N Pit. ri- 1098 346 1,399,898 0.00025it. pronouns 72 0 4,313,123 0

en. un- 119 25 7,618 .00328en. de- 141 16 86,130 .000185

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

P and sample size

I We saw that as N increases, V also increases (forat-least-mildly-productive processes)

I Thus, V cannot be compared at different Ns

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

P and sample size

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

V and NEnglish re- and mis-

0 10000 20000 30000 40000 50000

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

P and sample size

I However, growth rate is also systematically decreasing asN becomes larger

I At the beginning, any word will be a hapax legomenon;as sample increases, hapaxes will be increasingly lowerproportion of sample

I A specific instance of the more general problem of“variable constants” (Tweedie and Baayen 1998) inlexical statistics (cf. type/token ratio)

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

P and sample size

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

P and sample size

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

P and sample size

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Growth rate of re- at different sample sizes

0 50000 100000 150000 200000

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

P as a function of N (re-)

0 50000 100000 150000 200000

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

V and P at arbitrary Ns

I In order to compare V and P of processes (and predicthow process will develop in larger samples). . .

I we need to be able to estimate V and V1 at arbitrary Ns

I Once we compare P at same N, we might as wellcompare V1 directly (since P = V1/N and N will beconstant across compared processes)

I Most intuitive: VGC plot comparison

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Productivity beyond morphology

I Measuring generative potential of process/category notlimited to morphology

I Applications in lexicology, collocation and idiom studies,morphosyntax, syntax, language technology

I E.g., measure growth of nouns, adjectives, loanwords,relative productivity of two constructions, growth ofUNKNOWN lemmas as dataset increases. . .

I An example: measuring productivity of NP and PPexpansions in German TIGER treebank

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

TIGER expansions

I Types are non-terminal rewrite rules for NP and PP, e.g:I NP → ART ADJA NNI PP → APPR ART NN

I Frequency of occurrence of expansions collected fromabout 900,000 tokens (50,000 sentences) of Germannewspaper text from Frankfurter Rundschau

I http://www.ims.uni-stuttgart.de/projekte/TIGER

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

NP spectrum

1 2 3 4 5 6 7 8 9 11 13 15

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

PP spectrum

1 2 3 4 5 6 7 8 9 11 13 15

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Growth curves of NP and PP

0e+00 2e+04 4e+04 6e+04 8e+04 1e+05

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

I How many words did Shakespeare know? Are the laterHarry Potters more lexically diverse than the early ones?

I Are advanced learners distinguishable from nativespeakers in terms of vocabulary richness? How manywords do 5-year-old children know?

I Can changes in V detect the onset of Alzheimer’sdisease? (Garrard et al. 2005)

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

The Dickens’ datasets

I Dickens corpus: collection of 14 works by Dickens, about2.8 million tokens

I Oliver Twist: early work (1837-1839), about 160k tokens

I Great Expectations: later work (1860-1861), consideredone of Dickens’ masterpieces, about 190k tokens

I Our Mutual Friend: last completed novel (1864-1865),about 330k tokens

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Dickens’ V

0 500000 1000000 1500000 2000000 2500000

dickensomfgeot

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

The novels compared

0 50000 100000 150000 200000 250000 300000

omfgeot

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Oliver vs. Great Expectations

0 50000 100000 150000

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Conclusion and outlook

I Productivity, lexical richness, extrapolation of typecounts for language engineering purposes. . .

I all applications require a model of the larger populationof types that our sample comes from

I Two reasons to construct model of type populationdistribution:

I Population distribution interesting by itself, fortheoretical reasons or in NLP applications

I We know how to simulate sampling from population; thusonce we have population model we can obtain estimatesof type-related quantities (e.g., V and V1) at arbitrary Ns

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Modeling the populationProductivity

I Distribution of types of category of interest necessary toestimate V and V1 at arbitrary Ns, in order to compareVGCs and P of different processes

I However, type population distribution of word formationprocess (or other category) might be of interest by itself,as model of a part of the mental lexicon of speaker

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Modeling the populationProductivity

I Distribution of types of category of interest necessary toestimate V and V1 at arbitrary Ns, in order to compareVGCs and P of different processes

I However, type population distribution of word formationprocess (or other category) might be of interest by itself,as model of a part of the mental lexicon of speaker

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Modeling the populationLexical richness

I Lexical richness = V of whole population (how manywords did Shakespeare know? Was the lexical repertoireof young Dickens smaller than that of old Dickens? Howmany words do 5-year-old children know?)

I Accurate estimate of population V would solve “variableconstant” problem

I Sampling from population, in particular to computeVGC, also of interest

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Modeling the populationSome NLP applications

I Estimate number (and growth rate) of typos,UNKNOWNs (or other target tokens) in larger samples

→ estimate V and V1 at arbitrary Ns

I Estimate proportion of OOV words under assumptionthat lexicon contains top n most frequent types (seezipfR tutorial) → requires estimation of V andfrequency spectrum at arbitrary Ns (to find out for howmany tokens do the top n types account for)

I Good-Turing estimation, Bayesian priors → require fulltype population model

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

I Estimate number (and growth rate) of typos,UNKNOWNs (or other target tokens) in larger samples→ estimate V and V1 at arbitrary Ns

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

I Estimate proportion of OOV words under assumptionthat lexicon contains top n most frequent types (seezipfR tutorial)

→ requires estimation of V andfrequency spectrum at arbitrary Ns (to find out for howmany tokens do the top n types account for)

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

I Good-Turing estimation, Bayesian priors

→ require fulltype population model

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Outlook

I We need model of type population distribution

I We will use Zipf(-Mandelbrot)’s law as starting point tomodel how population looks like

TO BE CONTINUED

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Outlook

TO BE CONTINUED

Introduction

Baroni & Evert

Roadmap

Zipf’s law

Consequences

Applications

Lexical richness

Outlook

TO BE CONTINUED

Counting Words: Introductionzipfr.r-forge.r-project.org/.../ESSLLI/01_introduction.slides.pdf ·...

Documents

Proceedings of the ESSLLI Workshop on Distributional ...events.illc.uva.nl/ESSLLI2008/Materials/BaroniEvertLenci/ESSLLI... · ESSLLI Workshop on Distributional Lexical Semantics Bridging

Cities as Six-by-Six-Mile Squares: Zipf’s Law?users.econ.umn.edu/~holmes/papers/holmes_lee_nber_revision2.pdf · Cities as Six-by-Six-Mile Squares: Zipf’s Law? Thomas J. Holmes∗

On the Implications of Zipf’s Law in Passwordswangdingg.weebly.com/.../0/3/6/20366987/esorics16_final.pdf · 2018-09-09 · On the Implications of Zipf’s Law in Passwords 113

Zipf’s monkeys

The evolution of Zipf’s coeﬃcient and the kernel lexicon ... · The evolution of Zipf’s coeﬃcient and the kernel lexicon in six European languages. 3 This avoids possible

ESSLLI 2013 Schedule WEEK 1 SLOT 1 WEEK 1 … · ESSLLI 2013 Schedule WEEK 1 ‐ SLOT 1 ... ESSLLI 2013 Schedule WEEK 2 ... Sandra Kübler and Markus Dickinson Extrinsic Parse Improvement

An Outlier in Zipf’s World? A Case Study of China’s City

Computational Semantics coli.uni-sb.de/cl/projects/milca/esslli

Visual Analytics for Linguistics - Day 3 ESSLLI

An Introduction to Description Logics - esslli2018esslli2018.folli.info/wp-content/uploads/ESSLLI-Part1.pdfIvan Varzinczak (CRIL) An Introduction to Description Logics (Part 1) ESSLLI

plotting with matlab - ETH Z · 2013-09-30 Modeling and Simulating Social Systems with MATLAB 33 Exercise 4 – optional Zipf’s law: Zipf’s law says that the rank, x, of

CALCULATING ZIPF’S LAW - Weeblymvanegmond.weebly.com/uploads/2/2/1/5/22153706/... · CALCULATING ZIPF’S LAW ... the graphs of the alpha and beta distribution and the ... In the

Zipf’s Law for Cities: A Cross Country Investigation · Zipf’s Law for Cities: A Cross Country Investigation ... C16, R12 1 Correspondence ... existence of this fact comes in

Handout - Computational Lexical Semantics at ESSLLI 09

Power laws, Pareto distributions and Zipf’s la · 2011-11-08 · Power laws, Pareto distributions and Zipf’s law M.E.J. NEWMAN* Department of Physics and Center for the Study

Zipf’s law of abbreviation as a language universal · Bentz, Christian & Ferrer-i-Cancho, Ramon (2016). Zipf’s law of abbreviation as a language universal. In Bentz, Christian,

Power laws, Pareto distributions and Zipf’s lawtuvalu.santafe.edu/~aaronc/courses/5352/readings/... · Power laws, Pareto distributions and Zipf’s law M.E.J. NEWMAN* Department

Power laws, Pareto distributions and Zipf’s la · PDF filePower laws, Pareto distributions and Zipf’s law ... laws in a variety of systems and describe some of the ... from a power-law

ESSLLI 2013: Computational Morphology - …hana/teaching/2013-esslli/2013...Phonology/Phonetics in 5 slides What is morphology Words, Morphemes, etc. Morphological processes Typology

Zipf’s Law of Abbreviation and the Principle of Least E ort: …jasmeenkanwal.tk/wp-content/uploads/2017/05/Kanwal_et_al... · 2020. 4. 22. · Zipf’s Law of Abbreviation and