Unsupervised Part-of-Speech Tagging with Bilingual Graph-Based Projections

Unsupervised Part-of-Speech Tagging

with Bilingual Graph-Based Projections

June 21ACL 2011

Slav PetrovGoogle Research

Dipanjan DasCarnegie Mellon University

Part-of-Speech Tagging

Portland has a thriving music scene .NOUN VERB DET ADJ NOUN NOUN .

Supervised POS Tagging

ArabicBasque

BulgarianCatalanChinese

CzechDanishEnglishFrench

GermanGreek

HungarianItalian

JapaneseKorean

PortugueseRussianSloveneSpanishSwedishTurkish

84 86 88 90 92 94 96 98 100

93.797.8

93.499.1

POS Accuracy

Supervised setting: average accuracy is 96.2% 3

TnT (Brants, 2000)

Resource-Poor Languages

Several major languages with no or little annotated data

OriyaIndonesian-Malay

Azerbaijani

See http://www.ethnologue.org/ethno_docs/distribution.asp?by=size

Haitian

However, lots of parallel and

unannotated data!Basic NLP tools like POS

tagging essential for development of

language technologies

PunjabiVietnamese

Polish32 million37 million20 million

Native speakers

7.7 million

109 million69 million40 million

(Nearly) Universal Part-of-Speech Tags

VERB DETNOUN CONJPRON NUMADJ PRTADV .ADP X

Example Penn Treebank tag maps:

NN NOUNNNP NOUNNNPS NOUNNNS NOUN

PRP PRONPRP$ PRONWP PRONWP$ PRON

np NOUNnc NOUN

Example Spanish Treebank tag maps:p0 PRONpd PRONpe PRONpi PRONpn PRON

pp PRONpr PRONpt PRONpx PRON

See Petrov, Das and McDonald (2011)

Portland has a thriving music scene .NOUN VERB DET ADJ NOUN NOUN .

Portland hat eine prächtig gedeihende Musikszene .NOUN VERB DET ADJ ADJ NOUN .

প�োর্ট�ল্যোন্ড শহর এর সঙ্গীত �রিরবে�শ প�শ উন্নত | NOUN NOUN ADP NOUN NOUN ADJ ADJ .

Supervised Universal POS Tagging

ArabicBasque

BulgarianCatalanChinese

CzechDanishEnglishFrench

GermanGreek

HungarianItalian

JapaneseKorean

PortugueseRussianSloveneSpanishSwedishTurkish

84 86 88 90 92 94 96 98 100

93.797.8

93.499.1

POS Accuracy

Less variance in accuracy than fine tags 8

TnT (Brants, 2000)

State of the Art in Unsupervised POS Tagging

Portland hat eine prächti

g gedeihende Musikszene .

? ? ? ? ? ? ?

Hidden Markov Model (HMM) estimated with the Expectation-Maximization algorithm

: observation sequence: state sequence

10Merialdo (1994)

? ? ? ? ? ? ?

one of the 12 coarse

11Merialdo (1994)

Portland hat

transition multinomials

12Merialdo (1994)

Portland hat

emission multinomials

13Merialdo (1994)

? ? ? ? ? ? ?

Danish

Dutch German

Greek Italian Portuguese

Spanish

Swedish

Average

68.7 57.0 75.9 65.8 63.7 62.9 71.5 68.4 66.7EM-HMM

Poor average result

14Johnson (2007)

Unsupervised Part-of-Speech TaggingHidden Markov Model (HMM)

with locally normalized log-linear models

: observation sequence

Portland hat

? ? emission multinomials

: state sequence

Berg-Kirkpatrick et al. (2010)

Portland hat

suffixhyphen

capital lettersnumbers

...: state sequence

Portland hat

suffixhyphen

capital lettersnumbers

Estimated using gradient-based methods

: state sequence

Portland hat

Danish

Dutch German

Spanish

Swedish

Average

68.7 57.0 75.9 65.8 63.7 62.9 71.5 68.4 66.769.1

65.1 81.3 71.8 68.1 78.4 80.2 70.1 73.0EM-HMM

Feature-HMM

Estimated using gradient-based methods

Improvements across all languages

NOUN VERB

PRONDETADJNUM

ADJADV ADJ NOUN .

Unsupervised POS Tagging with Dictionaries

Hidden Markov Model (HMM) with locally normalized log-linear modelsState space constrained by possible gold

NOUN VERB

PRONDETADJNUM

ADJADV ADJ NOUN .

Unsupervised POS Tagging with Dictionaries

Hidden Markov Model (HMM) with locally normalized log-linear modelsState space constrained by possible gold

Danish

Dutch German

Spanish

Swedish

Average

68.7 57.0 75.9 65.8 63.7 62.9 71.5 68.4 66.769.1 65.1 81.3 71.8 68.1 78.4 80.2 70.1 73.0

94.7 93.5 96.6 96.4 94.0 95.8 85.5 93.7

EM-HMMFeature-HMM

w/ gold dictionary

Average result close to supervised accuracy! 20

For most languages, access to high-quality

tag dictionaries is not realistic.

Ideas: 1) Use supervision in resource-rich

languages2) Use parallel data

3) Construct projected tag lexicons

Morphologically rich languages only have base

forms in dictionaries

Bilingual Projection

Portland has a thriving music scene .

NOUN VERB DET ADJ NOUN NOUN .

automatic labels from supervised tagger, 97% accuracy

Portland hat eine prächtig gedeihende Musikszene .

Automatic unsupervised alignments from translation data(available for more than 50 languages)

Idea 1: direct projectionunaligned word

NOUN(most frequent tag)

24Yarowsky and Ngai

(2001)

Idea 1: direct projection

Portland hat eine prächtig gedeihende Musikszene .NOUN VERB DET NOUN ADJ NOUN .

+more projected tagged sentences

supervised training

tagger

(Brants, 2000)

Yarowsky and Ngai (2001)

Danish

Dutch German

Spanish

Swedish

Average

68.7 57.0 75.9 65.8 63.7 62.9 71.5 68.4 66.769.1 65.1 81.3 71.8 68.1 78.4 80.2 70.1 73.0

77.0 83.2 79.3 79.7 82.6 80.1 74.7 78.8

EM-HMM

Directprojection

Feature-HMM

Danish

Dutch German

Spanish

Swedish

Average

68.7 57.0 75.9 65.8 63.7 62.9 71.5 68.4 66.769.1 65.1 81.3 71.8 68.1 78.4 80.2 70.1 73.0

77.0 83.2 79.3 79.7 82.6 80.1 74.7 78.8

EM-HMM

Directprojection

Feature-HMM

consistent improvements over unsupervised models

Bilingual ProjectionIdea 2: lexicon projection

NOUNPortland

VERBhas

ADJthriving

NOUNmusic

NOUNscene

Portland

prächtig

gedeihende

Musikszene

NOUNPortland Portlan

ADJthrivinggedeihe

ndeprächtigVERB

NOUNsceneMusiksze

NOUNmusic

ignore unaligned word

ADJthrivinggedeihe

VERBhas

NOUNsceneMusiksze

NOUNmusic

Bag of alignments

ADJthrivinggedeihe

VERBhas

NOUNsceneMusiksze

NOUNmusic

ADJNOUN

PRONVERB

32ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

DET ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJthrivinggedeihe

VERBhas

NOUNscene

NOUNmusic

DETa NUMon

PRONone

Musikszene

33ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

DET ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJthrivinggedeihe

VERBhas

NOUNscene

NOUNmusic

DETa NUMon

PRONone

Musikszene

VERBthriving

34ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

DETADJ

NUMPRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

Portland

gedeihende

Musikszene

After scanning all the parallel data:

= probability of a tag given a word

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

Feature HMM constrained with projected dictionary

Improvements over simple projection for majority of the languages

Danish

Dutch German

Spanish

Swedish

Average

68.7 57.0 75.9 65.8 63.7 62.9 71.5 68.4 66.769.1 65.1 81.3 71.8 68.1 78.4 80.2 70.1 73.0

73.6 77.0 83.2 79.3 79.7 82.6 80.1 74.7 78.879.0

78.8 82.4 76.3 84.8 87.0 82.8 79.4 81.3

EM-HMM

DirectprojectionProjectedDictionar

Feature-HMM

Can coverage be improved?Idea:Projected lexicon

expansion and refinement using

a lot of unlabeled data

No information about unaligned words

Brief Overview:Graph-Based Learning

with Labeled and Unlabeled Data

labeled datapoints unlabeled datapoints

supervised label distributions

distributions to be found

Zhu, Ghahramani and Lafferty (2003) 39

= symmetric weight matrix

Label Propagation

set of distributions over unlabeled vertices

Label Propagation

unlabeled vertices

Label Propagation

brings the distributions of similarvertices closer

Label Propagation

brings the distributions of uncertain neighborhoods

close to the uniform distribution

Size of the label set

Label Propagation

Iterative updates for optimization

Label Propagation

How can label propagation help? For a language:

1) Build graph over a 2M trigram types as vertices• compute similarity matrix using co-occurrence

statistics

2) Label distribution at each vertex tag distribution over the trigram’s middle word

Subramanya, Petrov and Pereira (2010)

Idea 3: Graph-Based Projections

ist gut bei

ist lebhafter bei

ist wichtig bei

ist fein bei

gutem Essen zugetan

fuers Essen drauf

1000 Essen proschlechtes Essen

zum Essen niederlassen

zu realisieren ,zu essen ,

zu stecken , zu erreichen ,

Example Graph in German

ist gut bei

ist lebhafter bei

ist wichtig bei

ist fein bei

gutem Essen zugetan

fuers Essen drauf

zu stecken , zu erreichen ,

Example Graph in German

How can label propagation help? For a target language:

1) Build graph over a 2M trigram types as vertices• compute similarity matrix using co-occurrence

statistics

2) Label distribution at each vertex tag distribution over the trigram’s middle word3) Plug in auto-tagged words from a source language

4) Links between source and target language units are word alignments

ist gut bei

ist lebhafter bei

ist wichtig bei

ist fein bei

gutem Essen zugetan

fuers Essen drauf

zu stecken , zu erreichen ,eat

eateating

NOUNVERB

VERBVERB

goodADJ

nicelyADV

fineADJ

importantADJBilingual Graph

How can label propagation help?

For a target language:

3) Plug in auto-tagged words from a source language

5) Run first stage of label propagation• Source language target language

ist gut bei

ist lebhafter bei

ist wichtig bei

ist fein bei

gutem Essen zugetan

fuers Essen drauf

eateating

NOUNVERB

VERBVERB

goodADJ

nicelyADV

fineADJ

importantADJ

First Stage of Label Propagation

ist gut bei

ist lebhafter bei

ist wichtig bei

ist fein bei

gutem Essen zugetan

fuers Essen drauf

eateating

NOUNVERB

VERBVERB

goodADJ

nicelyADV

fineADJ

importantADJ

First Stage of Label Propagation

VERBNOUN

ADJADV

How can label propagation help?

For a target language:

3) Plug in auto-tagged words from a source language

5) Run first stage of label propagation• Source language target language

6) Run second stage of label propagation • within target language vertices• graph objective function with squared

penalties

ist gut bei

ist lebhafter bei

ist wichtig bei

ist fein bei

gutem Essen zugetan

fuers Essen drauf

eateating

NOUNVERB

VERBVERB

goodADJ

nicelyADV

fineADJ

importantADJ

Second Stage of Label Propagation

VERBNOUN

ADJADV

ADJist gut bei

ist lebhafter bei

ist wichtig bei

ist fein bei

gutem Essen zugetan

fuers Essen drauf

eateating

NOUNVERB

VERBVERB

goodADJ

nicelyADV

fineADJ

importantADJ

VERBNOUN

ADJADV

ADJist gut bei

ist lebhafter bei

ist wichtig bei

ist fein bei

gutem Essen zugetan

fuers Essen drauf

eateating

NOUNVERB

VERBVERB

goodADJ

nicelyADV

fineADJ

importantADJ

VERBNOUN

ADJADV

ADJist gut bei

ist lebhafter bei

ist wichtig bei

ist fein bei

gutem Essen zugetan

fuers Essen drauf

eateating

NOUNVERB

VERBVERB

goodADJ

nicelyADV

fineADJ

importantADJ

VERBNOUN

ADJADV

Continues till convergence...

fein lebhafter realisieren

Portland

gedeihende

Musikszene

End result?

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

ADVNUM

PRONVERB

ADJNOUN

PRONVERB

fein lebhafter realisieren

Portland

gedeihende

Musikszene

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

PRONVERB

ADJNOUN

ADVNUM

PRONVERB

ADJNOUN

PRONVERB

A larger set of tag distributions better and larger dictionary

Feature HMM constrained with graph-based dictionary

Danish

Dutch German

Spanish

Swedish

Average

68.7 57.0 75.9 65.8 63.7 62.9 71.5 68.4 66.769.1 65.1 81.3 71.8 68.1 78.4 80.2 70.1 73.0

73.6 77.0 83.2 79.3 79.7 82.6 80.1 74.7 78.879.0 78.8 82.4 76.3 84.8 87.0 82.8 79.4 81.383.2

79.5 82.8 82.5 86.8 87.9 84.2 80.5 83.4

EM-HMM

yGraph-BasedProjections

Feature-HMM

Feature HMM constrained with graph-based dictionary

Danish

Dutch German

Spanish

Swedish

Average

68.7 57.0 75.9 65.8 63.7 62.9 71.5 68.4 66.769.1 65.1 81.3 71.8 68.1 78.4 80.2 70.1 73.0

73.6 77.0 83.2 79.3 79.7 82.6 80.1 74.7 78.879.0 78.8 82.4 76.3 84.8 87.0 82.8 79.4 81.383.2

79.5 82.8 82.5 86.8 87.9 84.2 80.5 83.4

EM-HMM

Feature-HMM

yGraph-BasedProjections

93.1 94.7 93.5 96.6 96.4 94.0 95.8 85.5 93.7w/ gold dictionary

96.9 94.9 98.2 97.8 95.8 97.2 96.8 94.8 96.6supervised62

Lexicon Expansion

Danish

German

Italian

20000400006000080000

100000120000140000

Projected Dictionary Graph-Based Projections

thousandsof words

Concluding Notes• Soft expansion of lexicon using parallel

data and supervision in a resource rich language–Graph-based learning helps in almost all cases

• Reasonably accurate POS taggers without direct supervision• Traditional evaluation of unsupervised POS

taggers done using greedy metrics that use labeled data–Our presented models avoid these evaluation methods

• Practically no hyperparameter tuning–except a threshold parameter for dictionary construction

Future Directions• Scaling up the number of nodes in the

graph from 2M to billions may help create larger lexicons

• Including penalties in the graph objective that induce sparse tag distributions at each graph vertex

• Inclusion of multiple languages in the graph may further improve results–Label propagation in one huge multilingual graph

Projected Data

http://code.google.com/p/pos-projection/

Available at:

NOUNADJADJ

প�োর্ট�ল্যোন্ড শহর এর সঙ্গীত �রিরবে�শ প�শ উন্নত |

Portland tiene una escena musical vibrante .

波特兰有一个生机勃勃的音乐场景

Portland a une scène musicale florissante .

ADJNOUNNOUN NOUN

.NOUN VERB DET ADJ

.NOUN VERB DET ADJ NOUN NOUNNOUN VERB DET NOUN

Questions?

Unsupervised Part-of-Speech Tagging with Bilingual Graph-Based Projections

Documents

Multilingual NLP via Cross-Lingual Word Embeddings...Bilingual lexicon learning, multi-modal representations Extrinsic (downstream) tasks: Cross-lingual SRL, POS tagging, NER Cross-lingual

Mobile Tagging

Image Tagging

Tagging above Aplin Weir - Infofish Australiainfofishaustralia.com.au/wp-content/uploads/2016/11/Ross-River...Ross River Freshwater Tagging page 2 Ross River Freshwater Tagging Tagging

Multilingual Part-of-Speech Tagging: Two Unsupervised ... · Multilingual Part-of-Speech Tagging: Two Unsupervised Approaches Tahira Naseem TAHIRA@CSAIL.MIT.EDU Benjamin Snyder BSNYDER@CSAIL.MIT.EDU

February 2007CSA3050: Tagging II1 CSA2050: Natural Language Processing Tagging 2 Rule-Based Tagging Stochastic Tagging Hidden Markov Models (HMMs) N-Grams

Unsupervised Feature Learning Based on Deep Models for ... · Unsupervised Feature Learning Based on Deep Models for Environmental Audio Tagging Yong Xu, Qiang Huang, Wenwu Wang,

Lecture 11 - csd.uwo.ca · Lecture 11 Unsupervised Learning EM. Today • New Topic: Unsupervised Learning • supervised vs. unsupervised learning • unsupervised learning • nonparametric

Jacob Eisenstein · [13]Tahira Naseem, Benjamin Snyder, Jacob Eisenstein and Regina Barzilay. Multilingual part-of-speech tagging: Two unsupervised approaches. Journal of Artiﬁcial

Social Tagging, OSocial Tagging, Onlinenline

Unsupervised Approaches to Sequence Tagging, Morphology ...rezab/papers/unsupnlp_slides.pdf · Grenager & Manning (2006) • From dependency parses, a generative model predicts for

UNSUPERVISED JOINT PART-OF-SPEECH TAGGING ...burcucan/NecvaBolucu...UNSUPERVISED JOINT PART-OF-SPEECH TAGGING AND STEMMING FOR AGGLUTINATIVE LANGUAGES SONDAN EKLEMEL˙I D ILLERDE G˙

ELECTRONIC TAGGING MOBILE ELECTRONIC TAGGING (PSEM) DEVICES … · 2018. 11. 30. · ST-lite Electronic Tagging (PSE), Mobile Electronic tagging (PSEM) and Victim Protection (DEPAR)

Unsupervised Part-of-Speech Tagging: An Introduction Alex Brooks

Store Tagging

unsupervised adaptation - arXiv · tribution mismatch between the training and the test data is more often the rule than an exception. Think about part of speech tagging across di

Environmental tagging

NLTK Tagging

LENOVO ASSET TAGGING · LENOVO® ASSET TAGGING Lenovo’s Asset Tagging Service forms the foundation for a strong asset management program. Executed in production, Asset Tagging ensures

Unsupervised Approaches to Sequence Tagging, …stanford.edu/~rezab/papers/unsupnlp_slides.pdf · Arabic, Turkish, German, and Finnish ... Statistical verb lexicon ... frames, or