UnsupervisedLearning withText - Sciencesconf.org · 2019. 12. 10. · Phrase-Based Statistical Machine Translation (PBSMT) •Two main components •Phrase-table (needs parallel data)

Unsupervised Learning with Text

Ludovic Denoyer

Context

• Deep Learning (on text) has shown impressive results in the very last years

• But it usually needs very large datasets with rich supervision • Difficult for rare languages• Difficult for specific application domains• Difficult for particular tasks

• Open Question: What can we solve with few/no supervision ?

2

Outline

• Unsupervised Machine Translation

• Unsupervised Text rewriting

• Unsupervised Question Answering

• Building Interpretable text classifiers

Unsupervised Machine Translation

• Machine Translation works well for high-resource language pairs

• Performance drops drastically when parallel data is scarce• Vietnamese, Ukrainian, Tamil, Urdu, etc.

• The creation of parallel data is difficult, and costly• However, monolingual data is much easier to find

• Can we translate languages without any parallel data?

4Slide courtesy of Guillaume Lample

yes, we can…

• Unsupervised Word TranslationWord translation without parallel data (ICLR 2018)Guillaume Lample*, Alexis Conneau*, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégouhttps://github.com/facebookresearch/MUSE

• Unsupervised Neural or Phrase-Based Translation

5

Unsupervised Machine Translation Using Monolingual Corpora Only (ICLR 2018)Guillaume Lample, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato

Phrase-based & Neural Unsupervised Machine Translation (EMNLP 2018 – Best paper award)Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, Marc'Aurelio Ranzato

Slide courtesy of Guillaume Lample

https://github.com/facebookresearch/MUSE

Weakly-supervised word translation• Exploiting similarities among languages for machine translation (Mikolov et al., 2013)

• Start from two pre-trained monolingual spaces (word2vec)

• Totally unsupervised• Widely used• Strong systems for monolingual embeddings• Semantically and syntactically relevant• Not task-specific, useful across domains


Weakly-supervised word translation• Exploiting similarities among languages for machine translation (Mikolov et al., 2013)

• Start from two pre-trained monolingual spaces (word2vec)

• Project the source space onto the target space using a small dictionary

• Feed-forward network does not improve over linear mapping (Mikolov et al., 2013)

• Orthogonal projection works best Xing et al. (2015), Smith et al. (2017)

+ =


Weakly-supervised word translation

8

• Linear projection – Mikolov et al. (2013)

• Orthogonal projection – Xing et al. (2015), Smith et al. (2017) – Procrustes

• Given a source word s, define the translation as:

(nearest neighbor accordingto the cosine distance)

Can we find the mapping W in an unsupervised way?Slide courtesy of Guillaume Lample

Adversarial training

9

• If WX and Y are perfectly aligned, these spaces should be undistinguishable• Train a discriminator D to discriminate elements from WX and Y• Train W to prevent the discriminator from making accurate predictions

Discriminator training

Mapping training


• (A) Train monolingual word embeddings

• (B) Align them using adversarial training

• Refinement step• (C) Select high-confidence translation pairs• (D) Apply Procrustes on the generated dictionary

• Generate translations

10

Unsupervised word translation – Summary


Results on word translation – Refinement

11

Word translation retrieval – P@1 – Adversarial + Refinement1.5k source queries, 200k target keys (vocabulary of 200k words for all languages)

77.4 77.3 74.9 76.1

47

58.2

40.6

30.2

69.8 71.3 70.4

61.9

29.1

41.5

18.522.3

79.1 78.1 78.1 78.2

37.3

54.3

30.9

21.9

10

20

30

40

50

60

70

80

90

en-es es-en en-fr fr-en en-ru ru-en en-zh zh-en

Procrustes (Supervised) Adversarial (Unsupervised) Refinement (Unsupervised)


Unsupervised sentence translation

12

• Can we apply the same procedure to sentences?

• Number of points grows exponentially with sentence length

• No similar embedding structures across languages

• Direct application does not work (even in a supervised setting)


Phrase-Based Statistical Machine Translation (PBSMT)

• Two main components• Phrase-table (needs parallel data)• Language model

13

Source phrase Target phrase Score

il regarde

he is watching 0.6

he looks 0.3

he looks at 0.1

dans

in 0.7

into 0.2

within 0.1

la glace

the ice cream 0.5

ice 0.3

the mirror 0.2

: [il regarde] [dans] [la glace]: he looks in the mirror

[he is watching] [in] [the ice cream][he looks at] [within] [ice][he looks] [in] [the mirror]

Phrase scores0.6 0.7 0.50.1 0.1 0.30.3 0.7 0.2

LM scores

maximizes combined phrase-table and LM scores

InputReference

Hypotheses

Moses (Koehn et al., 2007)

-7-12

-3


Unsupervised word phrase translation• Given independent monolingual

corpora, we can generate:• Cross-lingual phrase tables

dictionaries• Cross-lingual word phrase

embeddings

• The alignment procedure is the same for words and phrases

14Slide courtesy of Guillaume Lample

13.3

14.6

15.6

Lample et al. (2018) Yang et al. (2018) PBSMT 0

Results with unsupervised phrase-table

15

[Raumsonde] Cassini findet Ozean auf [Saturnmond] EnceladusCassini space probe finds ocean on Saturn 's moon Enceladus[NASA spacecraft] Cassini finds a [Saturn moon] Enceladus ocean

InputReferencePBSMT 0

Source phrase Target phrase Score

Raumsonde

NASA spacecraft 0.32

spacecraft 0.26

moon probe 0.12

Voyager 2 0.12

Kepler telescope 0.09

Saturnmond

Saturn moon 0.45

Enceladus 0.21

liquid water 0.11

Titan 0.10

Earth-like planet 0.05

BLEU on German-English - newstest 2016


Back-translation

PBSMT 0PBSMT 1

Translate Training

German ≈English German≈English

Improving neural machine translation models with monolingual data (Sennrich et al., 2016)Slide courtesy of Guillaume Lample

Iterative Back-translation

PBSMT 0PBSMT 1

PBSMT 2

Translate Training

German ≈English

PBSMT 1

English ≈German

German≈English

≈German English


Results with iterative back-translation

18

Raumsonde Cassini findet Ozean auf Saturnmond EnceladusCassini space probe finds ocean on Saturn 's moon EnceladusNASA spacecraft Cassini finds a Saturn moon Enceladus oceanNASA spacecraft Cassini finds ocean on Saturn moon Enceladus

InputReferencePBSMT 0PBSMT n


13.3

14.615.6

22.9

Lample et al. (2018) Yang et al. (2018) PBSMT 0 PBSMT n


Three principles of unsupervised MT

19

• Initialization• Language modeling• Iterative back-translation

• Can we apply these principles to neural architectures?


Three principles for unsupervised MT - NMT

20

• Online back-translation

• Language modeling

• InitializationInitialized shared encoders, with shared cross-lingual BPE embeddings

de<latexit sha1_base64="qyR5ChEikzSYVVxTM8hw2O8lVZE=">AAAB6XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxiv2ANpTNZtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5AKro3rfjultfWNza3ydmVnd2//oHp41NZJphi2WCIS1Q2oRsEltgw3ArupQhoHAjvB+HZW7zyh0jyRj2aSoh/ToeQRZ9RY6yHEQbXm1t25yCp4BdSgUHNQ/eqHCctilIYJqnXPc1Pj51QZzgROK/1MY0rZmA6xZ1HSGLWfzzedkjPrhCRKlH3SkLn7eyKnsdaTOLCdMTUjvVybmf/VepmJrv2cyzQzKNnioygTxCRkdjYJuUJmxMQCZYrbXQkbUUWZseFUbAje8smr0L6oe5bvL2uNmyKOMpzAKZyDB1fQgDtoQgsYRPAMr/DmjJ0X5935WLSWnGLmGP7I+fwBhlaNVw==</latexit><latexit sha1_base64="qyR5ChEikzSYVVxTM8hw2O8lVZE=">AAAB6XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxiv2ANpTNZtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5AKro3rfjultfWNza3ydmVnd2//oHp41NZJphi2WCIS1Q2oRsEltgw3ArupQhoHAjvB+HZW7zyh0jyRj2aSoh/ToeQRZ9RY6yHEQbXm1t25yCp4BdSgUHNQ/eqHCctilIYJqnXPc1Pj51QZzgROK/1MY0rZmA6xZ1HSGLWfzzedkjPrhCRKlH3SkLn7eyKnsdaTOLCdMTUjvVybmf/VepmJrv2cyzQzKNnioygTxCRkdjYJuUJmxMQCZYrbXQkbUUWZseFUbAje8smr0L6oe5bvL2uNmyKOMpzAKZyDB1fQgDtoQgsYRPAMr/DmjJ0X5935WLSWnGLmGP7I+fwBhlaNVw==</latexit><latexit sha1_base64="qyR5ChEikzSYVVxTM8hw2O8lVZE=">AAAB6XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxiv2ANpTNZtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5AKro3rfjultfWNza3ydmVnd2//oHp41NZJphi2WCIS1Q2oRsEltgw3ArupQhoHAjvB+HZW7zyh0jyRj2aSoh/ToeQRZ9RY6yHEQbXm1t25yCp4BdSgUHNQ/eqHCctilIYJqnXPc1Pj51QZzgROK/1MY0rZmA6xZ1HSGLWfzzedkjPrhCRKlH3SkLn7eyKnsdaTOLCdMTUjvVybmf/VepmJrv2cyzQzKNnioygTxCRkdjYJuUJmxMQCZYrbXQkbUUWZseFUbAje8smr0L6oe5bvL2uNmyKOMpzAKZyDB1fQgDtoQgsYRPAMr/DmjJ0X5935WLSWnGLmGP7I+fwBhlaNVw==</latexit><latexit sha1_base64="qyR5ChEikzSYVVxTM8hw2O8lVZE=">AAAB6XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxiv2ANpTNZtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5AKro3rfjultfWNza3ydmVnd2//oHp41NZJphi2WCIS1Q2oRsEltgw3ArupQhoHAjvB+HZW7zyh0jyRj2aSoh/ToeQRZ9RY6yHEQbXm1t25yCp4BdSgUHNQ/eqHCctilIYJqnXPc1Pj51QZzgROK/1MY0rZmA6xZ1HSGLWfzzedkjPrhCRKlH3SkLn7eyKnsdaTOLCdMTUjvVybmf/VepmJrv2cyzQzKNnioygTxCRkdjYJuUJmxMQCZYrbXQkbUUWZseFUbAje8smr0L6oe5bvL2uNmyKOMpzAKZyDB1fQgDtoQgsYRPAMr/DmjJ0X5935WLSWnGLmGP7I+fwBhlaNVw==</latexit>

German encoder

English decoder

en<latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit><latexit sha1_base64="+HDIXYpZQvOdiTn4OonYWizCtHc=">AAAB5XicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZwT6kHUomvdOG5jEkGaEM/RVuXCjiX3LnvzF9LLT1QODjnITce5JMcOvC8DsobW3v7O6V9ysH1cOj49pJtW11bhi2mBbadBNqUXCFLcedwG5mkMpEYCeZ3M3zzjMay7V6dNMMY0lHiqecUeetp772YYFqNqjVw0a4ENmEaAV1WKk5qH31h5rlEpVjglrbi8LMxQU1jjOBs0o/t5hRNqEj7HlUVKKNi8XAM3LhnSFJtfFHObJwf78oqLR2KhN/U1I3tuvZ3Pwv6+UuvYkLrrLcoWLLj9JcEKfJfHsy5AaZE1MPlBnuZyVsTA1lzndU8SVE6ytvQvuqEXl+CKEMZ3AOlxDBNdzCPTShBQwkvMAbvAcmeA0+lnWVglVvp/BHwecPDwCPXA==</latexit><latexit sha1_base64="+HDIXYpZQvOdiTn4OonYWizCtHc=">AAAB5XicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZwT6kHUomvdOG5jEkGaEM/RVuXCjiX3LnvzF9LLT1QODjnITce5JMcOvC8DsobW3v7O6V9ysH1cOj49pJtW11bhi2mBbadBNqUXCFLcedwG5mkMpEYCeZ3M3zzjMay7V6dNMMY0lHiqecUeetp772YYFqNqjVw0a4ENmEaAV1WKk5qH31h5rlEpVjglrbi8LMxQU1jjOBs0o/t5hRNqEj7HlUVKKNi8XAM3LhnSFJtfFHObJwf78oqLR2KhN/U1I3tuvZ3Pwv6+UuvYkLrrLcoWLLj9JcEKfJfHsy5AaZE1MPlBnuZyVsTA1lzndU8SVE6ytvQvuqEXl+CKEMZ3AOlxDBNdzCPTShBQwkvMAbvAcmeA0+lnWVglVvp/BHwecPDwCPXA==</latexit><latexit sha1_base64="jIRXnrwANexB1z+QScNEg85H/F4=">AAAB8HicbZBNTwIxEIZn8QvxC/XopZGYeCK7XvRI9OIREwENbEi3zEJDt920XROy4Vd48aAxXv053vw3FtiDgm/S5Mn7zqQzE6WCG+v7315pbX1jc6u8XdnZ3ds/qB4etY3KNMMWU0Lph4gaFFxiy3Ir8CHVSJNIYCca38zyzhNqw5W8t5MUw4QOJY85o9ZZjz3lwhzltF+t+XV/LrIKQQE1KNTsV796A8WyBKVlghrTDfzUhjnVljOB00ovM5hSNqZD7DqUNEET5vOBp+TMOQMSK+2etGTu/u7IaWLMJIlcZULtyCxnM/O/rJvZ+CrMuUwzi5ItPoozQawis+3JgGtkVkwcUKa5m5WwEdWUWXejijtCsLzyKrQv6oHjO7/WuC7OUYYTOIVzCOASGnALTWgBgwSe4RXePO29eO/ex6K05BU9x/BH3ucPSf+Qsw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit>

English encoder

German decoder de

<latexit sha1_base64="qyR5ChEikzSYVVxTM8hw2O8lVZE=">AAAB6XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxiv2ANpTNZtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5AKro3rfjultfWNza3ydmVnd2//oHp41NZJphi2WCIS1Q2oRsEltgw3ArupQhoHAjvB+HZW7zyh0jyRj2aSoh/ToeQRZ9RY6yHEQbXm1t25yCp4BdSgUHNQ/eqHCctilIYJqnXPc1Pj51QZzgROK/1MY0rZmA6xZ1HSGLWfzzedkjPrhCRKlH3SkLn7eyKnsdaTOLCdMTUjvVybmf/VepmJrv2cyzQzKNnioygTxCRkdjYJuUJmxMQCZYrbXQkbUUWZseFUbAje8smr0L6oe5bvL2uNmyKOMpzAKZyDB1fQgDtoQgsYRPAMr/DmjJ0X5935WLSWnGLmGP7I+fwBhlaNVw==</latexit><latexit sha1_base64="qyR5ChEikzSYVVxTM8hw2O8lVZE=">AAAB6XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxiv2ANpTNZtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5AKro3rfjultfWNza3ydmVnd2//oHp41NZJphi2WCIS1Q2oRsEltgw3ArupQhoHAjvB+HZW7zyh0jyRj2aSoh/ToeQRZ9RY6yHEQbXm1t25yCp4BdSgUHNQ/eqHCctilIYJqnXPc1Pj51QZzgROK/1MY0rZmA6xZ1HSGLWfzzedkjPrhCRKlH3SkLn7eyKnsdaTOLCdMTUjvVybmf/VepmJrv2cyzQzKNnioygTxCRkdjYJuUJmxMQCZYrbXQkbUUWZseFUbAje8smr0L6oe5bvL2uNmyKOMpzAKZyDB1fQgDtoQgsYRPAMr/DmjJ0X5935WLSWnGLmGP7I+fwBhlaNVw==</latexit><latexit sha1_base64="qyR5ChEikzSYVVxTM8hw2O8lVZE=">AAAB6XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxiv2ANpTNZtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5AKro3rfjultfWNza3ydmVnd2//oHp41NZJphi2WCIS1Q2oRsEltgw3ArupQhoHAjvB+HZW7zyh0jyRj2aSoh/ToeQRZ9RY6yHEQbXm1t25yCp4BdSgUHNQ/eqHCctilIYJqnXPc1Pj51QZzgROK/1MY0rZmA6xZ1HSGLWfzzedkjPrhCRKlH3SkLn7eyKnsdaTOLCdMTUjvVybmf/VepmJrv2cyzQzKNnioygTxCRkdjYJuUJmxMQCZYrbXQkbUUWZseFUbAje8smr0L6oe5bvL2uNmyKOMpzAKZyDB1fQgDtoQgsYRPAMr/DmjJ0X5935WLSWnGLmGP7I+fwBhlaNVw==</latexit><latexit sha1_base64="qyR5ChEikzSYVVxTM8hw2O8lVZE=">AAAB6XicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxiv2ANpTNZtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5AKro3rfjultfWNza3ydmVnd2//oHp41NZJphi2WCIS1Q2oRsEltgw3ArupQhoHAjvB+HZW7zyh0jyRj2aSoh/ToeQRZ9RY6yHEQbXm1t25yCp4BdSgUHNQ/eqHCctilIYJqnXPc1Pj51QZzgROK/1MY0rZmA6xZ1HSGLWfzzedkjPrhCRKlH3SkLn7eyKnsdaTOLCdMTUjvVybmf/VepmJrv2cyzQzKNnioygTxCRkdjYJuUJmxMQCZYrbXQkbUUWZseFUbAje8smr0L6oe5bvL2uNmyKOMpzAKZyDB1fQgDtoQgsYRPAMr/DmjJ0X5935WLSWnGLmGP7I+fwBhlaNVw==</latexit>

Cnoise

English encoder

English decoderz

<latexit sha1_base64="ZxpkeWP7OiuYrtNqeUSJ8s3zMA0=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1QQ3+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A6UOM/g==</latexit><latexit sha1_base64="ZxpkeWP7OiuYrtNqeUSJ8s3zMA0=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1QQ3+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A6UOM/g==</latexit><latexit sha1_base64="ZxpkeWP7OiuYrtNqeUSJ8s3zMA0=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1QQ3+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A6UOM/g==</latexit><latexit sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit><latexit sha1_base64="R6oAxW108ykbU5Q7rRtmojdj+UA=">AAAB3XicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHHZgn1AW0omvdPGZjJDckeoQ3+BGxeK+Lfc+W9MHwttPRD4OCch954wVdKS7397ha3tnd294n7poHx4dFw5KbdskhmBTZGoxHRCblFJjU2SpLCTGuRxqLAdTu7mefsJjZWJfqBpiv2Yj7SMpODkrMbzoFL1a/5CbBOCFVRhpfqg8tUbJiKLUZNQ3Npu4KfUz7khKRTOSr3MYsrFhI+w61DzGG0/Xww6YxfOGbIoMe5oYgv394ucx9ZO49DdjDmN7Xo2N//LuhlFN/1c6jQj1GL5UZQpRgmbb82G0qAgNXXAhZFuVibG3HBBrpuSKyFYX3kTWle1wHHDhyKcwTlcQgDXcAv3UIcmCEB4gTd49x69V+9jWVfBW/V2Cn/kff4A1IeLqw==</latexit><latexit sha1_base64="R6oAxW108ykbU5Q7rRtmojdj+UA=">AAAB3XicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHHZgn1AW0omvdPGZjJDckeoQ3+BGxeK+Lfc+W9MHwttPRD4OCch954wVdKS7397ha3tnd294n7poHx4dFw5KbdskhmBTZGoxHRCblFJjU2SpLCTGuRxqLAdTu7mefsJjZWJfqBpiv2Yj7SMpODkrMbzoFL1a/5CbBOCFVRhpfqg8tUbJiKLUZNQ3Npu4KfUz7khKRTOSr3MYsrFhI+w61DzGG0/Xww6YxfOGbIoMe5oYgv394ucx9ZO49DdjDmN7Xo2N//LuhlFN/1c6jQj1GL5UZQpRgmbb82G0qAgNXXAhZFuVibG3HBBrpuSKyFYX3kTWle1wHHDhyKcwTlcQgDXcAv3UIcmCEB4gTd49x69V+9jWVfBW/V2Cn/kff4A1IeLqw==</latexit><latexit sha1_base64="5aRlc8nh+NErCXzOl7sRXDhdtsI=">AAAB6HicbZA9TwJBEIbn8AvxC7W02UhMrMidjZREG0tI5COBC9lb5mBlb++yu2eCF36BjYXG2PqT7Pw3LnCFgm+yyZN3ZrIzb5AIro3rfjuFjc2t7Z3ibmlv/+DwqHx80tZxqhi2WCxi1Q2oRsEltgw3AruJQhoFAjvB5HZe7zyi0jyW92aaoB/RkeQhZ9RYq/k0KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSENT/jMkkNSrb8KEwFMTGZX02GXCEzYmqBMsXtroSNqaLM2GxKNgRv9eR1aF9VPctNt1K/yeMowhmcwyV4cA11uIMGtIABwjO8wpvz4Lw4787HsrXg5DOn8EfO5w/oA4z6</latexit><latexit sha1_base64="ZxpkeWP7OiuYrtNqeUSJ8s3zMA0=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1QQ3+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A6UOM/g==</latexit><latexit sha1_base64="ZxpkeWP7OiuYrtNqeUSJ8s3zMA0=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1QQ3+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A6UOM/g==</latexit><latexit sha1_base64="ZxpkeWP7OiuYrtNqeUSJ8s3zMA0=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1QQ3+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A6UOM/g==</latexit><latexit sha1_base64="ZxpkeWP7OiuYrtNqeUSJ8s3zMA0=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1QQ3+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A6UOM/g==</latexit><latexit sha1_base64="ZxpkeWP7OiuYrtNqeUSJ8s3zMA0=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1QQ3+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A6UOM/g==</latexit><latexit sha1_base64="ZxpkeWP7OiuYrtNqeUSJ8s3zMA0=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1QQ3+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A6UOM/g==</latexit>

en<latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit><latexit sha1_base64="mxM6op2xff7JBykLijMwADucJX8=">AAAB3nicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZxT6gHUomvdOGZjJDckcoQ/+BGxeK+LPc+W9MHwttPRD4OCch954oU9KS7397pa3tnd298n7loHp4dFw7qbZtmhuBLZGq1HQjblFJjS2SpLCbGeRJpLATTe7meecZjZWpfqJphmHCR1rGUnBy1iPqQa3uN/yF2CYEK6jDSs1B7as/TEWeoCahuLW9wM8oLLghKRTOKv3cYsbFhI+w51DzBG1YLCadsQvnDFmcGnc0sYX7+0XBE2unSeRuJpzGdj2bm/9lvZzim7CQOssJtVh+FOeKUcrma7OhNChITR1wYaSblYkxN1yQK6fiSgjWV96E9lUjcPzgQxnO4BwuIYBruIV7aEILBMTwAm/w7k28V+9jWVfJW/V2Cn/kff4AfLCMDQ==</latexit><latexit sha1_base64="mxM6op2xff7JBykLijMwADucJX8=">AAAB3nicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZxT6gHUomvdOGZjJDckcoQ/+BGxeK+LPc+W9MHwttPRD4OCch954oU9KS7397pa3tnd298n7loHp4dFw7qbZtmhuBLZGq1HQjblFJjS2SpLCbGeRJpLATTe7meecZjZWpfqJphmHCR1rGUnBy1iPqQa3uN/yF2CYEK6jDSs1B7as/TEWeoCahuLW9wM8oLLghKRTOKv3cYsbFhI+w51DzBG1YLCadsQvnDFmcGnc0sYX7+0XBE2unSeRuJpzGdj2bm/9lvZzim7CQOssJtVh+FOeKUcrma7OhNChITR1wYaSblYkxN1yQK6fiSgjWV96E9lUjcPzgQxnO4BwuIYBruIV7aEILBMTwAm/w7k28V+9jWVfJW/V2Cn/kff4AfLCMDQ==</latexit><latexit sha1_base64="AlTRGiaoenPSMUjdmeGiXTNzb3M=">AAAB6XicbZBNTwIxEIan+IX4hXr00khMPJFdL3okevGIRpAENqRbZqGh2920XROy4R948aAxXv1H3vw3FtiDgm/S5Mk7M+nMG6ZSGOt536S0tr6xuVXeruzs7u0fVA+P2ibJNMcWT2SiOyEzKIXClhVWYifVyOJQ4mM4vpnVH59QG5GoBztJMYjZUIlIcGaddY+qX615dW8uugp+ATUo1OxXv3qDhGcxKsslM6bre6kNcqat4BKnlV5mMGV8zIbYdahYjCbI55tO6ZlzBjRKtHvK0rn7eyJnsTGTOHSdMbMjs1ybmf/VupmNroJcqDSzqPjioyiT1CZ0djYdCI3cyokDxrVwu1I+Yppx68KpuBD85ZNXoX1R9x3febXGdRFHGU7gFM7Bh0towC00oQUcIniGV3gjY/JC3snHorVEiplj+CPy+QOUP41d</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit>

en<latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit><latexit sha1_base64="A1AdlTh8S7FprVaPiESSPgh1AGg=">AAAB8HicbZDLSgNBEEVr4ivGV9Slm8EguAozIugy6MZlBPOQZAg9nUrSpB9Dd48QhnyFGxeKuPVz3Pk3dpJZaOKFhsO9VXRVxQlnxgbBt1dYW9/Y3Cpul3Z29/YPyodHTaNSTbFBFVe6HRODnElsWGY5thONRMQcW/H4dpa3nlAbpuSDnSQYCTKUbMAosc567CoXZiinvXIlqAZz+asQ5lCBXPVe+avbVzQVKC3lxJhOGCQ2yoi2jHKclrqpwYTQMRlix6EkAk2UzQee+mfO6fsDpd2T1p+7vzsyIoyZiNhVCmJHZjmbmf9lndQOrqOMySS1KOnio0HKfav82fZ+n2mklk8cEKqZm9WnI6IJte5GJXeEcHnlVWheVEPH95eV2k1+jiKcwCmcQwhXUIM7qEMDKAh4hld487T34r17H4vSgpf3HMMfeZ8/Sz+Qtw==</latexit> en

<latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="ck8pdC+ekZH4nUmSP+ZG7r8lEyk=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odOn4MoA7ncAFXEMIN3MEDdKALAhJ4hXdv4r15H6uuat66tDP4I+/zBzjGijg=</latexit><latexit sha1_base64="mxM6op2xff7JBykLijMwADucJX8=">AAAB3nicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZxT6gHUomvdOGZjJDckcoQ/+BGxeK+LPc+W9MHwttPRD4OCch954oU9KS7397pa3tnd298n7loHp4dFw7qbZtmhuBLZGq1HQjblFJjS2SpLCbGeRJpLATTe7meecZjZWpfqJphmHCR1rGUnBy1iPqQa3uN/yF2CYEK6jDSs1B7as/TEWeoCahuLW9wM8oLLghKRTOKv3cYsbFhI+w51DzBG1YLCadsQvnDFmcGnc0sYX7+0XBE2unSeRuJpzGdj2bm/9lvZzim7CQOssJtVh+FOeKUcrma7OhNChITR1wYaSblYkxN1yQK6fiSgjWV96E9lUjcPzgQxnO4BwuIYBruIV7aEILBMTwAm/w7k28V+9jWVfJW/V2Cn/kff4AfLCMDQ==</latexit><latexit sha1_base64="mxM6op2xff7JBykLijMwADucJX8=">AAAB3nicbZBLSwMxFIXv1FetVatbN8EiuCozbnQpuHFZxT6gHUomvdOGZjJDckcoQ/+BGxeK+LPc+W9MHwttPRD4OCch954oU9KS7397pa3tnd298n7loHp4dFw7qbZtmhuBLZGq1HQjblFJjS2SpLCbGeRJpLATTe7meecZjZWpfqJphmHCR1rGUnBy1iPqQa3uN/yF2CYEK6jDSs1B7as/TEWeoCahuLW9wM8oLLghKRTOKv3cYsbFhI+w51DzBG1YLCadsQvnDFmcGnc0sYX7+0XBE2unSeRuJpzGdj2bm/9lvZzim7CQOssJtVh+FOeKUcrma7OhNChITR1wYaSblYkxN1yQK6fiSgjWV96E9lUjcPzgQxnO4BwuIYBruIV7aEILBMTwAm/w7k28V+9jWVfJW/V2Cn/kff4AfLCMDQ==</latexit><latexit sha1_base64="AlTRGiaoenPSMUjdmeGiXTNzb3M=">AAAB6XicbZBNTwIxEIan+IX4hXr00khMPJFdL3okevGIRpAENqRbZqGh2920XROy4R948aAxXv1H3vw3FtiDgm/S5Mk7M+nMG6ZSGOt536S0tr6xuVXeruzs7u0fVA+P2ibJNMcWT2SiOyEzKIXClhVWYifVyOJQ4mM4vpnVH59QG5GoBztJMYjZUIlIcGaddY+qX615dW8uugp+ATUo1OxXv3qDhGcxKsslM6bre6kNcqat4BKnlV5mMGV8zIbYdahYjCbI55tO6ZlzBjRKtHvK0rn7eyJnsTGTOHSdMbMjs1ybmf/VupmNroJcqDSzqPjioyiT1CZ0djYdCI3cyokDxrVwu1I+Yppx68KpuBD85ZNXoX1R9x3febXGdRFHGU7gFM7Bh0towC00oQUcIniGV3gjY/JC3snHorVEiplj+CPy+QOUP41d</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit><latexit sha1_base64="YKt1tCVvCDtX0HFPqxJC8Mf5UmE=">AAAB6XicbZBNS8NAEIYn9avWr6hHL4tF8FQSEfRY9OKxiv2ANpTNdtIu3WzC7kYoof/AiwdFvPqPvPlv3LY5aOsLCw/vzLAzb5gKro3nfTultfWNza3ydmVnd2//wD08aukkUwybLBGJ6oRUo+ASm4YbgZ1UIY1Dge1wfDurt59QaZ7IRzNJMYjpUPKIM2qs9YCy71a9mjcXWQW/gCoUavTdr94gYVmM0jBBte76XmqCnCrDmcBppZdpTCkb0yF2LUoaow7y+aZTcmadAYkSZZ80ZO7+nshprPUkDm1nTM1IL9dm5n+1bmai6yDnMs0MSrb4KMoEMQmZnU0GXCEzYmKBMsXtroSNqKLM2HAqNgR/+eRVaF3UfMv3l9X6TRFHGU7gFM7Bhyuowx00oAkMIniGV3hzxs6L8+58LFpLTjFzDH/kfP4AlX+NYQ==</latexit>


Results with unsupervised NMT

21

Raumsonde Cassini findet Ozean auf Saturnmond EnceladusCassini space probe finds ocean on Saturn 's moon EnceladusNASA spacecraft Cassini finds a Saturn moon Enceladus oceanNASA spacecraft Cassini finds ocean on Saturn moon EnceladusCassini spacecraft takes ocean on Saturnmond Enceladus

InputReferencePBSMT 0PBSMT nNMT


13.3

14.615.6

21

22.9

Lample et al.(2018)

Yang et al.(2018)

PBSMT 0 NMT PBSMT n


• Train unsupervised NMT and PBSMT models independently• Fine-tune NMT model on both NMT and PBSMT back-translations

Hybrid model

Behörden sind dabei , den Zaun schnell zu reparieren .Authorities are quickly repairing the fence .Authorities , who are on the fence to quickly repair .Local authorities are unsure how to repair the fence .Authorities are also getting the fence repaired quickly .

SRCREFPBSMT nNMTHybrid


13.314.6

15.6

21

22.9

25.2

Lample et al.(2018)

Yang et al.(2018)

PBSMT 0 NMT PBSMT n Hybrid


Results on low-resource language pairs

23

22.9 23

23.9

21

21.5

22

22.5

23

23.5

24

24.5

25

Gu et al. (2018) * PBSMT n Hybrid

BLEU on Romanian-English

8.9

12.3

Supervised PBSMT ** PBSMT n

BLEU on Urdu-English

Trained on 800,000 parallel sentences (Opus corpora)**Trained on 6,000 parallel sentences, seed dictionary,multi-NMT system with 6 languages

*


Unsupervised Question Answering by Cloze Translation

Patrick Lewis, Ludovic Denoyer, Sebastian Riedel

Extractive Question Answering (EQA)

Question q: What Denver player caused two fumbles for the Panthers?

Answer a:

P Rajpurkar et al. 201631

Von Miller

Context c: The Broncos took an early lead in Super Bowl 50 and never trailed. [...] Denver linebacker Von Miller was named Super Bowl MVP, recording five solo tackles, 2½ sacks and two forced fumbles.

Slide courtesy of Patrick Lewis

Extractive Question Answering (EQA) (2)

32

Unsupervised EQA

Data Generator

Documentof

Interest

Documentof

Interest

Documentof

Interest

Documentof

Interest

Documentof

Interest

(c, q, a)

..

.

(c, q, a)

(c, q, a)

Off-the-ShelfQA

Model

Step 1: Train Synthetic EQA data generator

Step 2: Generate synthetic EQA data

Step 3: Train off-the-shelf EQA Model


Generating EQA data without Supervision

ContextsSampleContext

...The London Sevens is a rugby tournament held at Twickenham Stadium in London...

Sample

Answer

Which venue hosts the London Sevens?

SampleQuestion

33

NER / Noun Phrases

...The London Sevens is a rugby tournament held at Twickenham Stadium in London...


Experiments• Evaluate EQA performance without explicit supervision• Explore impact of design decisions of data generator

• Context Generator: Paragraphs from English Wikipedia• Cloze Question boundary: Sentence or sub-clause with “S” label• 5M questions mined from common crawl, 5M clozes mined from Wikipedia• Question Answering: BiDAF + Self-attention [1] and fine-tuning BERT [2]

[1] J Devlin et al. 2019[2] C Clark and M Gardner 2018

34Slide courtesy of Patrick Lewis

UNMT Translation Examples

G Lample and A Conneau 2019

WALA would be sold to the Des Moines-based ORG for $86 million

Meredith Corp

Who would buy the WALA Des Moines-based for $86 million?

Cloze Question Answer Translated Question

The NUMERIC on Orchard Street remained open until 2009 second How much longer did Orchard

Street remain open until 2009?

he speaks LANGUAGE, English, and German Spanish What are we , English , and German?

Form a larger Mid-Ulster District Council in TEMPORAL

August When is a larger Mid-Ulster District Council?

Form a larger Mid-Ulster District Council in TEMPORAL August When will a larger Mid-Ulster

District Council be formed?35


Results in context

• Best results with:• Named entity answers• Unsupervised NMT questions• Wh* heuristic• Sub-clause boundaries• Bert-Large QA


Conclusion

• Outperform simple supervised models without explicit supervision• Much scope for future work:• Questions without Answers• “Multi-hop” Questions• Other Question Answering tasks

• 4M UQA training datapoints, and codegithub.com/facebookresearch/UnsupervisedQA

37Slide courtesy of Patrick Lewis

https://github.com/facebookresearch/UnsupervisedQA

Context

• Deep Learning Classifier are usually not interpretable

• For Text categorization, some recent efforts focus on attention models able to extract ‘rationales’ from text

• It usually requires human-annotated datasets

• Objective: Learn interpretable classifier without additional supervision

EDUCE: Classification through concept detection

Principles• Step 1: For each of the C concepts, extract the excerpt that is the

most relevant • Step 2: Classify each excerpt as ’present’ or not in the document• Step 3: Classify the document based on the presence of concepts

Ensuring concept consistency

• Without additional concept, there is no reason that EDUCE extracts meaningful concepts• We consider that concepts will be ‘easy to understand’ if extracted

excerpts are homogeneous

Results

Conclusion

• Learning without supervision (or with weak supervision) can provide very good results in many different problems

• It is mainly relied on the use of additional learning models as regularizers (i.e inductive bias)

• The natural extension of fully unsupervised methods is few-shot / semi-supervised learning that can greatly improve the quality of the models.

Documents

UnsupervisedLearning withText - Sciencesconf.org · 2019. 12. 10. · Phrase-Based Statistical Machine Translation (PBSMT) •Two main components •Phrase-table (needs parallel data)