Idiom Token Classification using Sentential Distributed Semantics (Giancarlo D. Salton)

Idiom Token Classification using SententialDistributed Semantics

Giancarlo D. Salton Robert J. Ross John D. Kelleher

Applied Intelligence Research CentreSchool of Computing

Dublin, 28 September 2016

Idiom Token Classification using Sentential Distributed Semantics NLP Dublin Meetup

Outline

Idioms

Distributed Representations

“Per-expression” classification

“General” classification

Conclusions

Future Work on Idiom Token Classification

Idiom Classification on Machine Translation Pipeline

2/45


Outline

Idioms




Conclusions



Idioms 3/45


Idioms

I Idioms are multiword expressions (MWE)

I Their meaning is non-compositional

I No linguistic agreement upon the set of characteristics definingidioms

Idioms 4/45


Idiomatic and Literal Usages

I Literally...

I Actually...

I How to distinguish between a literal and idiomatic usage?

Idiom token classificationIdioms 5/45


Previous Work

I Previous work used “per-expression” models

– different set of features for each expression

I in general, these features are not reusable

– i.e., a model is trained for each particular expression

I In our opinion the state-of-the-art is Peng et al. (2014)

– Also “per-expression” classification

– Topic models

– Up to 5 paragraphs of context!

Idioms 6/45


Per-expression classifiers

I Per-expression classifiers

– Expensive

I Idioms samples are rare

– Time-consuming

I Feature engineering

Idioms 7/45


General Classifiers?

I Can we find a common set of features?

I Can we train a general classifier?

hold+horses

vs.

break+ice

vs.

spill+beans

Idioms 8/45


Outline

Idioms




Conclusions



Distributed Representations 9/45


Distributed Representations of Words

I Word2vec (Mikolov et al., 2013)



Skip-thought Vectors (or Sent2Vec)(Kiros et al., 2015)

I Encoder/Decoder Framework





Distributed representations = features!















– Encoder learns to encode information about the context of aninput sentence




Distributed Representations vs Idioms

I Distributed representations cluster words (word2vec) orsentences (sent2vec) with similar semantics

– Empirical results have shown that

I Idiomatic vs. literal usages

– Idioms should alse be in a different part of space than literalexpressions (at least when considering the same expression)



Outline

Idioms




Conclusions



“Per-expression” classification 17/45


“Per-expression” settings

I Following baseline evaluation (Peng et al., 2014)

I 4 expressions from VNC-Tokens dataset:

– blow+whistle, lose+head, make+scene and take+heart

I Balanced training sets

I Imbalanced test sets



“Per-expression” classifiers

I K-Nearest Neighbours

– 2, 3, 5 and 10 neighbours

I Support Vector Machines

– Linear SVM: linear kernel and grid search for best parameters

– Grid SVM: grid search for best kernel/parameters

– SGD SVM: linear kernel trained with Stochastic Gradient Descent



blow+whistle results

Models Precision Recall F1-ScorePeng et al. (2014)FDA-Topics 0.62 0.60 0.61FDA-Topics+A 0.47 0.44 0.45FDA-Text 0.65 0.43 0.52FDA-Text+A 0.45 0.49 0.47SVMs-Topics 0.07 0.40 0.12SVMs-Topics+A 0.21 0.54 0.30SVMs-Text 0.17 0.90 0.29SVMs-Text+A 0.24 0.87 0.38Distributed RepresentationsKNN-2 0.61 0.41 0.49KNN-3 0.84 0.32 0.46KNN-5 0.79 0.28 0.41KNN-10 0.83 0.30 0.44Linear SVM 0.77 0.50 0.60Grid SVM 0.80 0.51 0.62SGD SVM 0.70 0.40 0.51



lose+head results




make+scene results




take+heart results




“Per-expression” evaluation

I No single model performed best for all expressions

I SVM consistently outperformed K-NNs

I Peng et al. (2014) features may capture a different set ofdimensions

I Combination with baseline model may result in stronger classifier



Outline

Idioms




Conclusions



“General” classification 25/45


“General classifier” settings

I Simulation of expected behaviour on real data

I 27 expressions of “balanced” part of VNC-Tokens dataset

I Imbalanced training set

I Imbalanced test set



“General classifier” classifiers

I SVMs only

– Linear SVM: linear kernel and grid search for best parameters

– Grid SVM: grid search for best kernel/parameters

– SGD SVM: linear kernel trained with Stochastic Gradient Descent



“General classifier” results

Linear SVM Grid SVM SGD SVMExpressions Pr. Rec. F1 Pr. Rec. F1 Pr. Rec. F1blow+whistle 0.84 0.67 0.75 0.84 0.68 0.75 0.67 0.59 0.63lose+head 0.78 0.66 0.72 0.75 0.64 0.69 0.75 0.67 0.71make+scene 0.92 0.84 0.88 0.92 0.81 0.86 0.78 0.81 0.79take+heart 0.94 0.79 0.86 0.94 0.80 0.86 0.86 0.80 0.83Total 0.84 0.80 0.83 0.84 0.80 0.83 0.79 0.79 0.78



“General classifier” evaluation

I Expected behaviour on “real world”

– Consider imbalances of real data

I 2 classifiers had high performance

– Same general precision, recall and F1

– Deviations occurred across individual expressions

I Performance is still not consistent over all classifiers and acrossexpressions



PCA Analysis of Distributed Representations on

“General” classifier



Outline

Idioms




Conclusions



Conclusions 31/45


Conclusions

I Our approach needs less resources to achieve roughly the sameperformance

I SVM generally perform better than KNNs

I “General classifier” is feasible

I “Per-expression” does achieve better results in some cases

Conclusions 32/45


Outline

Idioms




Conclusions



Future Work on Idiom Token Classification 33/45



I Apply to other languages than English

I Apply to other datasets

– e.g., the IDX Corpus

I What are the main sources of error for the “general classifier”?

– Better understanding of representations is needed

Future Work on Idiom Token Classification 34/45


Outline

Idioms




Conclusions



Idiom Classification on Machine Translation Pipeline 35/45


Idiom Token Classification on Machine TranslationPipeline

(Salton et al., 2014b)































References

Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun,Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances inNeural Information Processing Systems 28, pages 3276–3284.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.Distributed representations of words and phrases and their compositionality. InAdvances in Neural Information Processing Systems 26, pages 3111–3119.

Jing Peng, Anna Feldman, and Ekaterina Vylomova. 2014. Classifying idiomatic andliteral expressions using topic models and intensity of emotions. In Proceedings of the2014 Conference on Empirical Methods in Natural Language Processing (EMNLP),pages 2019–2027, October.

Giancarlo D. Salton, Robert J. Ross, and John D. Kelleher. 2014a. An EmpiricalStudy of the Impact of Idioms on Phrase Based Statistical Machine Translation ofEnglish to Brazilian-Portuguese. In Third Workshop on Hybrid Approaches toTranslation (HyTra), pages 36–41.

Giancarlo D. Salton, Robert J. Ross, and John D. Kelleher. 2014b. Evaluation of asubstitution method for idiom transformation in statistical machine translation. In The10th Workshop on Multiword Expressions (MWE 2014), pages 38–42.

44/45


Thank you!

Giancarlo D. Salton would like to thank CAPES (“Coordenao deAperfeioamento de Pessoal de Nvel Superior”) for his Science Without

Borders scholarship, proc n. 9050-13-2

45/45

Technology

Idiom Token Classification using Sentential Distributed Semantics (Giancarlo D. Salton)