Semi-supervised Learning for Neural Machine Translation

Yong Cheng

joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu

1

Machine Translation

2

Automated translation using computer software

Machine Translation

3

Rule-based Machine Translation 1970s

Example-based Machine Translation 1984

Statical Machine Translation (SMT) 1993

Neural Machine Translation NMT 2014

Trends: learning to translate from DATA

Machine Translation

4

Parallel Corpora

Monolingual Corpora

Parallel corpora are usually limited in

quantity quality coverage& &

Monolingual Corpora Used in SMT and NMT

N-gram language model in SMT Koehn et al., [2007]

Monolingual corpora as decipherment Ravi and Knight [2011]

Integrate a neural language model into NMT. Gulccehre et al. [2015]

Additional pseudo parallel corpus. Sennrich et al. [2016]

5

Supervised Training

Parallel Corpus

Objective

6

Unsupervised Training

Monolingual Corpus

7

cc Our Approach — Autoencoders

8

bushi yu shalong juxing le huitan x

cc Our Approach — Autoencoders

9

bushi yu shalong juxing le huitan xP(y | x;

!θ )

cc Our Approach — Autoencoders

10

bushi yu shalong juxing le huitan x

Bush held a talk with sharon y

P(y | x;!θ )

latent

cc Our Approach — Autoencoders

11

bushi yu shalong juxing le huitan x

Bush held a talk with sharon y

P(y | x;!θ )

P(x | y;!θ )

latent

cc Our Approach — Autoencoders

12

bushi yu shalong juxing le huitan x

Bush held a talk with sharon

′xbushi yu shalong juxing le huitan

y

P(y | x;!θ )

P(x | y;!θ )

latent

cc Our Approach — Autoencoders

13

source autoencoder target autoencoder

Unsupervised Training (Autoencoders)

Monolingual Corpus

14

target autoencoder

Semi-supervised Training

15

Training Objective

Translation Results

Compared with Moses (SMT) and RNNSearch (NMT)

16

Translation Results

Compared with Moses (SMT) and RNNSearch (NMT)

17

Translation Results

Compared with Moses (SMT) and RNNSearch (NMT)

18

Translation Results

Compared with Moses (SMT) and RNNSearch (NMT)

19

Translation Results

Compared with Moses (SMT) and RNNSearch (NMT)

20

Translation Results

Compared with Sennrich et al. [2015a]

21

Example Translation of Monolingual Corpus

22

ConclusionMonolingual corpora is an important resource for neural machine translation.

We have proposed a semi-supervised approach to training bidirectional neural machine translation models for exploiting monolingual corpora.

As our method is sensitive to the OOVs present in monolingual corpora, we plan to integrate Jean et al. (2015)’s technique on using very large vocabulary into our approach.

23

Thank You !

24

Effect of Sample Size

ZH-EN EN-ZH

25

Effect of OOV ratio

ZH-EN EN-ZH

26