Distributional Semanticsattach.matita.net/caterinamauri/sitovecchio/692485562_1.pdfDistributional...

Distributional Semantics

Magnus Sahlgren

Pavia, 11 September 2012

Course web page:

http://www.gavagai.se/distributional_semantics.php

This course:

Theoretical foundations (the distributional hypothesis)

Basic concepts (co-occurrence matrix, distributional vectors)

Distributional semantic models (LSA, HAL, Random Indexing)

Applications (lexical semantics, cognitive modelling, data

mining)

Research questions (semantic relations, compositionality)

This course:

mining)

This course:

mining)

This course:

mining)

This course:

mining)

This course:

mining)

What are the underlying assumptions and theories?

What are the main models, and how do they di�er?

How to build and use a model

How to evaluate a model

What can we use the models for?

What are the main research questions at the moment?

Theoretical basisThe Distributional Hypothesis

Representational frameworkVector Space

The Distributional Hypothesis

Extracting semantics from the distributions of terms

Firth:�you shall know a word by the company it keeps�

Wittgenstein:�the meaning of a word is its use in language�

?:�words with similar distributions have similar meanings�

=The Distributional Hypothesis

Usually motivated by Zellig Harris' distributional methodology:

Linguistics can only deal with what is internal to language

Linguistic explanans is distributional facts

Harris' distributional methodology is the discovery procedure bywhich we can quantify distributional similarities between linguisticentities

But why would such distributional similarities indicate semantic

similarity? (And what does �meaning� mean anyway?)

Harris' distributional methodology is the discovery procedure bywhich we can quantify distributional similarities between linguisticentities

But why would such distributional similarities indicate semantic

similarity? (And what does �meaning� mean anyway?)

If linguistics is to deal with meaning, it can only do so throughdistributional analysis

Di�erential, not referential

Structuralist legacy � Bloom�eld, Saussure

Structuralism:

Language as a system

Signs (e.g. words) are identi�ed by their functional di�erenceswithin this system

Two kinds: syntagmatic and paradigmatic

Structuralism:

Syntagmatic relations:

Words that co-occur

Combinatorial relations (phrases, sentences, etc.)

Words that co-occur

Paradigmatic relations:

Words that do not co-occur

Substitutional relations (co-occur with same other words)

Example:

I drink co�eeyou sip teathey gulp cocoa

I drink co�ee

you sip tea

they gulp cocoa

I drink co�eeyou sip teathey gulp cocoa

Synonyms (e.g. �good� / �great�)

Antonyms (e.g. �good� / �bad�)

Hypernyms (e.g. �dog� / �animal�)

Hyponyms (e.g. �dog� / �poodle�)

Meronymy (e.g. ��nger� / �hand�)

�drink� and �co�ee� have a syntagmatic relation if they co-occur

�co�ee� and �tea� have a paradigmatic relation if they co-occurwith the same other words (e.g. �drink�)

�drink� and �co�ee� have a syntagmatic relation if they co-occur

�co�ee� and �tea� have a paradigmatic relation if they co-occurwith the same other words (e.g. �drink�)

To summarize:

Harris: If linguistics is to deal with meaning, it can only do sothrough distributional analysis

Structuralism: Meaning = relations between words

Saussure: Two types of relations: syntagmatic and paradigmatic

To summarize:

=Extracting syntagmatic or paradigmatic relations between wordsfrom co-occurrences or second-order co-occurrences

The Distributional Hypothesis makes sense from a linguisticperspective

Does it also make sense from a cognitive perspective?

The Distributional Hypothesis makes sense from a linguisticperspective

Does it also make sense from a cognitive perspective?

We (i.e. humans) obviously can learn new words based oncontextual cues

He filled the wapimuk with the substance, passed it

around and we all drunk some

We found a little, hairy wapimuk sleeping behind the

We encounter new words everyday...

but rarely ask for de�nitions

The absence of de�nitions in normal discourse is a sign ofcommunicative prosperity

Abstract terms (Vigliocco et al. 2009)justice, tax, etc.

Concrete terms for which we have no direct experience of thereferentsaardvark, cyclotron, etc.

Abstract terms (Vigliocco et al. 2009)justice, tax, etc.

Concrete terms for which we have no direct experience of thereferentsaardvark, cyclotron, etc.

Congenitally blind individuals show normal competence of colorterms and visual perception verbs (Landau & Gleitman 1985)

The cognitive representation of a word is some abstraction orgeneralization derived from the contexts in which the word hasbeen encountered (Miller & Charles 1991)

A contextual representation may include extra-linguistic contexts

De facto, context is equated with linguistic context

Practical reason � it is easy to collect linguistic contexts(from corpora) and to process them

Theoretical reason � it is possible to investigate the role oflinguistic distributions in shaping word meaning

Weak Distributional Hypothesis

A quantitative method for semantic analysis and lexical resourceinduction

Meaning (whatever this might be) is re�ected in distribution

Strong Distributional Hypothesis

A cognitive hypothesis about the form and origin of semanticrepresentations

Word distributions in context have a speci�c causal role in theformation of the semantic representation for that word

The distributional properties of words in linguistic contexts explainshuman semantic behavior (e.g. judgment of semantic similarity)

Vector space

1 2 3 4

a = (4, 3)

b = (5, 6)

Vector space

A2,2 =a

[(4 3)(5 6)

Vector space

A2,2 =a

[(4 3)(5 6)

Vector space

A2,2 =a

[(4 3)(5 6)

Vector space

Am,n =

a1,1 a1,2 a1,3 . . . a1,na2,1 a2,2 a2,3 . . . a2,na3,1 a3,2 a3,3 . . . a3,n...

......

. . ....

am,1 am,2 am,3 . . . am,n

Vector space

High-dimensional vector spaces (i.e. dimensionalities on the order ofthousands)

Mathematically straightforward

Warning: high-dimensional spaces can be counterintuitive!

Vector space

Imagine two nested cubes:

How big must the inner cube be in order to cover 1% of the area ofthe outer cube?

Answer: 10% of the side length (0.1× 0.1 = 0.01)

Vector space

For 3-dimensional cubes, the inner cube must have ≈ 21% of theside length of the outer cube (0.21× 0.21× 0.21 ≈ 0.01)

For n-dimensional cubes, the inner cube must have 0.011n of the

side length

Vector space

For 3-dimensional cubes, the inner cube must have ≈ 21% of theside length of the outer cube (0.21× 0.21× 0.21 ≈ 0.01)

For n-dimensional cubes, the inner cube must have 0.011n of the

side length

Vector space

For n = 1 000 that is 99.5%

That is, if the outer 1 000-dimensional cube has sides 2 units long,and the inner 1 000-dimensional cube has sides 1.99 units long, theouter cube will still contain 100 times more volume

Vector space

For n = 1 000 that is 99.5%

That is, if the outer 1 000-dimensional cube has sides 2 units long,and the inner 1 000-dimensional cube has sides 1.99 units long, theouter cube will still contain 100 times more volume

Distributional semantics + vector space

=Word Space Models

Distributional semantics + vector space

=Word Space Models

Word Space

�Vector similarity is the only information present in Word Space:semantically related words are close, unrelated words are distant.�

(Hinrich Schütze: Word space, 1993)

Word Space

(Schütze: Dimensions of Meaning, 1992)

Word Space (A Brief History)

Manually de�ned features

Psychology: Osgood (1950s)

Human ratings over contrastive adjective-pairs

small � large bald � furry docile � dangerous

mouse 2 6 1rat 2 6 4

Connectionism, AI: Waltz & Pollack (1980s), Gallant (1990s),Gärdenfors (2000s)

human machine politics ...

astronomer +2 -1 -1 ...

Connectionism, AI: Waltz & Pollack (1980s), Gallant (1990s),Gärdenfors (2000s)

human machine politics ...

astronomer +2 -1 -1 ...

Pros: Expert knowledge

Cons: Bias; how many features are enough?

Word Space

Automatic methods: count co-occurrences (c.f. the distributionalhypothesis)

Co-occurrence matrix

Distributional vector

Word Space

The Co-occurrence Matrix

c1 c2 c3 . . . cn

Am,n =

......

. . ....

am,1 am,2 am,3 . . . am,n

w are the m (target) word types

c are the n contexts in the data

and the cells are word-context co-occurrence frequencies

c1 c2 c3 . . . cn

Am,n =

......

. . ....

am,1 am,2 am,3 . . . am,n

c1 c2 c3 . . . cn

Am,n =

......

. . ....

am,1 am,2 am,3 . . . am,n

c1 c2 c3 . . . cn

Am,n =

......

. . ....

am,1 am,2 am,3 . . . am,n

c1 c2 c3 . . . cn

Am,n =

0 1 0 . . . 11 0 2 . . . 00 2 1 . . . 3...

......

. . ....

0 0 1 . . . 0

Distributional Vectors

c1 c2 c3 . . . cn

Am,n =

0 1 0 . . . 11 0 2 . . . 00 2 1 . . . 3...

......

. . ....

0 0 1 . . . 0

Rows are n-dimensional distributional vectors

Words that occur in similar contexts get similar distributionalvectors

So how should we de�ne �context�?

c1 c2 c3 . . . cn

Am,n =

0 1 0 . . . 11 0 2 . . . 00 2 1 . . . 3...

......

. . ....

0 0 1 . . . 0

c1 c2 c3 . . . cn

Am,n =

0 1 0 . . . 11 0 2 . . . 00 2 1 . . . 3...

......

. . ....

0 0 1 . . . 0

Words-by-region Matrices

r1 r2 r3 . . . rn

Am,n =

co�ee...wm

0 1 0 . . . 11 0 1 . . . 00 1 1 . . . 1...

......

. . ....

0 0 1 . . . 0

A region is a text unit (document, paragraph, sentence, etc.)r2 = �I drink co�ee�

Syntagmatic similarity: words that co-occur in the same region

r1 r2 r3 . . . rn

Am,n =

co�ee...wm

0 1 0 . . . 11 0 1 . . . 00 1 1 . . . 1...

......

. . ....

0 0 1 . . . 0

r1 r2 r3 . . . rn

Am,n =

co�ee...wm

0 1 0 . . . 11 0 1 . . . 00 1 1 . . . 1...

......

. . ....

0 0 1 . . . 0

Words-by-words Matrices

w1 drink w3 . . . wm

Am,n =

co�ee

tea...wm

0 1 0 . . . 11 0 2 . . . 00 2 0 . . . 3...

......

. . ....

0 0 1 . . . 0

Paradigmatic similarity: words that co-occur with the same other

Syntagmatic similarity by looking at the individual dimensions

Am,n =

co�ee

tea...wm

0 1 0 . . . 11 0 2 . . . 00 2 0 . . . 3...

......

. . ....

0 0 1 . . . 0

Am,n =

co�ee

tea...wm

0 1 0 . . . 11 0 2 . . . 00 2 0 . . . 3...

......

. . ....

0 0 1 . . . 0

Context window

The region within which co-occurrences are collected

Parameters:

Direction

Weighting

Context window

Parameters:

Direction

Weighting

Context window

Parameters:

Direction

Weighting

Context window

Parameters:

Direction

Weighting

Context window

Parameters:

Direction

Weighting

Context window

[I drink very strong] coffee [at the cafe down the street ]1 1 1 1 1 1 1 1 1 1

An entire sentence as a �at context window

Context window

[I drink very strong] coffee [at the cafe down the street ]1 1 1 1 1 1 1 1 1 1

An entire sentence as a �at context window

Context window

I drink [very strong] coffee [at the ] cafe down the street

1 1 1 1

A 2+2-sized �at context window

Context window

[I drink very strong] coffee [at the cafe down ] the street

1 2 3 4 4 3 2 1

A 4+4-sized distance weighted context window

Context window

I drink very strong coffee [at the cafe down the street]1 0.9 0.8 0.7 0.6 0.5

A 0+6-sized distance weighted context window

Context window

I [drink very strong] coffee [at the cafe] down the street

1 0 3 0 0 1

A 3+3-sized distance weighted context window that ignores stopwords

Distributional vectors

Count how many times each target word occurs in a certain context

Collect (a function of) these frequency counts in vectors

[12,0,234,92,1,0,87,525,0,0,1,2,0,8129,1,0,51,0,235...]

And then what?

[12,0,234,92,1,0,87,525,0,0,1,2,0,8129,1,0,51,0,235...]

And then what?

[12,0,234,92,1,0,87,525,0,0,1,2,0,8129,1,0,51,0,235...]

And then what?

[12,0,234,92,1,0,87,525,0,0,1,2,0,8129,1,0,51,0,235...]

And then what?

The Meaning of Life

Vector similarity

co�ee

Vector similarity

co�ee

Vector similarity

co�ee

Vector similarity

co�ee

Vector similarity

co�ee

Vector similarity

Minkowski distance:(∑n

i=1|xi − yi |N

City-Block (or Manhattan) distance: N = 1

Euclidean distance: N = 2

Chebyshev distance: N →∞

Scalar product: x · y = x1y1 + x2y2 + ...+ xnyn

Cosine: x ·y|x ||y | =

i=1 xiyi√∑n

i=1 x2i

√∑n

i=1 y2i

Vector similarity

i=1|xi − yi |N

i=1 xiyi√∑n

i=1 x2i

√∑n

i=1 y2i

Vector similarity

i=1|xi − yi |N

i=1 xiyi√∑n

i=1 x2i

√∑n

i=1 y2i

Vector similarity

i=1|xi − yi |N

i=1 xiyi√∑n

i=1 x2i

√∑n

i=1 y2i

And then what?

Compute similarity between such vectors

And then what?

Compute similarity between such vectors

Nearest Neighbors

Nearest neighbor search: extracting the k nearest neighbors to atarget word

1 compute the cosine similarity between the context vector ofthe target word and the context vectors of all other words inthe word space

2 sort the resulting similarities

3 return only the top-k words

Nearest Neighbors

dog 1.000000cat 0.774165horse 0.668386fox 0.648134pet 0.626650

rabbit 0.615840pig 0.570504

animal 0.566003mongrel 0.560158sheep 0.551973pigeon 0.547442deer 0.534663rat 0.531442bird 0.527370

red 1.000000yellow 0.824409white 0.789056brown 0.723576grey 0.720103blue 0.700047pink 0.672573black 0.671302shiny 0.661379purple 0.633858striped 0.619801dark 0.610804gleam 0.603501palea 0.595221

Building a word space: step-by-stepThe �linguistic� steps

I drank very strong �Arabica� coffees, at the cafe.

Pre-process a corpus (optional, but recommended)(e.g. tokenization, downcasing, stemming/lemmatization...)

Linguistic mark-up(PoS tagging, syntactic parsing, named entity recognition...)

Select (the target words and) the linguistic contexts(e.g. text region, words...)

I drank very strong �Arabica� coffees, at the cafe.

I drank very strong Arabica coffees at the cafe

i drank very strong arabica coffees at the cafe

i drink very strong arabica coffee at the cafe

i/PN drink/VB very/AV strong/AJ arabica/NN coffee/NN at/PR the/DT

cafe/NN

(S (NP i/PN) (VP drink/VB (NP (AP very/AV strong/AJ) arabica/NN

coffee/NN) (PP at/PR (NP the/DT cafe/NN))))

Building a word space: step-by-stepThe �mathematical� steps

Count the target-context co-occurrences

Weight the contexts (optional, but recommended)(e.g. raw co-occurrence frequency, entropy, association

measures...)

Reduce the matrix dimensions (optional)(dimension reduction method: context selection, variance,

SVD, RI...)

(dimensionality)

Compute vector similarities (do nearest neighbor search)between distributional vectors(e.g. cosine, euclidean distance, etc.)

measures...)

SVD, RI...)

(dimensionality)

measures...)

SVD, RI...)

(dimensionality)

measures...)

SVD, RI...)

(dimensionality)

measures...)

SVD, RI...)

(dimensionality)

measures...)

SVD, RI...)

(dimensionality)

measures...)

SVD, RI...)

(dimensionality)

measures...)

SVD, RI...)

(dimensionality)

Building a word space

Tools:

GSDM Guile Sparse Distributed Memory (Guile)

S-Space https://github.com/fozziethebeat/S-Space (Java)

SemanticVectors http://code.google.com/p/semanticvectors/(Java)

GenSim http://radimrehurek.com/gensim/ (Python)

Word2word http://www.indiana.edu/ semantic/word2word/(Java)

Tools:

Use GSDM to:

Build a words-by-documents model

Build a words-by-words model

Extract similar words

Use GSDM to:

Distributional Semanticsattach.matita.net/caterinamauri/sitovecchio/692485562_1.pdfDistributional...

Documents

ANLP Horacio Rodríguez - cs.upc.eduageno/anlp/semanticParsing.pdf · •Compositional Semantics ... Introduction to Formal Semantics for ... –Distributional Semantics •These

Semantics at Scale: A Distributional Approach

The semantics of poetry: a distributional reading

Distributional Semanticsattach.matita.net/caterinamauri/sitovecchio/455463743_41.pdfRecap Distributional Semantics and The Distributional Hypothesis Syntagmatic and paradigmatic similarities

INF5820 Distributional Semantics: Extracting Meaning from ... › studier › emner › matnat › ifi › INF... · INF5820 Distributional Semantics: Extracting Meaning from Data

Towards a Formal Distributional Semantics: Simulating Logical

Words & their Meaning: Distributional Semantics

Distributional semantics (II) - UMass Amherstbrenocon/anlp2018/lectures/10-distsem.pdf · Distributional semantics (II) CS 690N, Spring 2018 ... vector representations of the terms

Doctoral Thesis Distributional semantic phrases vs ...eprints-phd.biblio.unitn.it/1060/1/EMVecchi_Thesis.pdf · Compositional Distributional Semantics ... in distributional semantics

Compositional Operators in Distributional Semantics · PDF file · 2015-10-05Compositional Operators in Distributional Semantics ... distributional semantics; compositionality; vector

Entity Typing Using Distributional Semantics and DBpedia

Natural Language Processing - University of Cambridge · Natural Language Processing Lecture 8: Distributional semantics Introduction to the distributional hypothesis From last time:

Distributional Semantic Models for Affective Text …potam/talks/google_semsim_v7.pdf · · 2014-08-15Compositional semantics and affect ... (distributional hypothesis of meaning)

Composition in distributional semantics

Compositional Operators in Distributional Semantics - · PDF file · 2017-08-28Semantics and Distributional Semantics I provide an introduction to compositional and distributional

Distributional Semantics - Home page | Department of ...disi.unitn.it/~bernardi/Courses/CL/Slides/ds_semantics.pdfBackground Distributional Semantics: sense The main questions have

Distributional Semantics - The unsupervised modeling of

The Challenge of Composition in Distributional and Formal Semantics · PDF file · 2017-11-29•Learning Semantically and Additively Compositional Distributional ... ªFormal Semantics:

DISTRIBUTIONAL HYPOTHESIS: WORDS FOR ‘HUMAN …

Compositional Distributional Semantics with Compact Closed