Upload
jennifer-white
View
216
Download
3
Tags:
Embed Size (px)
Citation preview
META-NET is partially supported by the DG INFSO of the European Commission through the META-NET is partially supported by the DG INFSO of the European Commission through the NoE project T4ME of the Seventh Framework Programme. Grant Agreement No.: 249119NoE project T4ME of the Seventh Framework Programme. Grant Agreement No.: 249119
Dimitris Mavroeidis
Institute for Language and Speech Processing
Greece
Divide and Translate: Using Document Classification to improve
Statistical Machine Translation
DateDivide and TranslateTitle of
Presentation 2
Language Models
Counts n-grams
Models a coprus
Compare with documents outside the corpus
Calculate perplexity to assess similarity
2
DateDivide and TranslateTitle of
Presentation 3
EUROVOC
A multi-lingual thesaurus (eurovoc.europa.eu)
Created by the European Commission
Used primarily for indexing purposes
21 top-level domains:
04 POLITICS
08 INTERNATIONAL RELATIONS
…
GEOGRAPHY
INTERNATIONAL ORGANIZATIONS
Date 3
DateDivide and TranslateTitle of
Presentation 4
Classification using Language Models
Existing corpora with domain labels
Train one LM per domain
Calculate perplexity of incoming document against all LMs
The lowest perplexity is an indication that the text belongs to the corresponding domain
Let’s see if that’s true…
DateDivide and TranslateTitle of
Presentation 5
Context in Machine Translation: Research
Corpus: JRC-Acquis (legal documents of the EU)
Labeled with one or more EUROVOC terms
Each term belongs to one of the 21 domains
Each document can be classified to one of these domains
Date 5
DateDivide and TranslateTitle of
Presentation 6
Corpus
Date 6
Language ISO code
No of texts
Text body Signatures AnnexesTotal No words (text +
signatures + annexes):
Total No words
Total No characters
Average No words
Total No words
Total No words
bg 11384 16140819 1045226711417.85 2170075 14114612 32425506
cs 21438 22843279 1489729811065.55 7225300 16763733 46832312
da 23624 31459627 2134681351331.68 2629786 16855213 50944626
de 23541 32059892 2327486751361.87 2542149 16327611 50929652
el 23184 36453749 2395835431572.37 2973574 16459680 55887003
en 23545 34588383 2106920591469.03 3198766 17750761 55537910
es 23573 38926161 2380167561651.3 3490204 19716243 62132608
et 23541 24621625 1927007041045.9 1336051 14995748 40953424
fi 23284 24883012 2121789641068.67 2677798 12547171 40107981
fr 23627 39100499 2347582901654.91 3021013 19978920 62100432
hu 22801 28602380 2138046141254.44 2529488 15056496 46188364
it 23472 35764670 2306770131523.72 3120797 18331535 57217002
lt 23379 26937773 1994382581152.22 2436585 15018484 44392842
lv 22906 27592514 1964520511204.6 1673124 15437969 44703607
mt 10545 20926909 1289067481984.53 1336042 15620611 37883562
nl 23564 35265161 2319635391496.57 3039580 18467115 56771856
pl 23478 29713003 2144640261265.57 2513141 17027393 49253537
pt 23505 37221668 2274994181583.56 3034308 19350227 59606203
ro 6573 9186947 605373011397.68 514296 11185842 20887085
ro 19211 30832212 1826312771604.92 --- --- 30832212
sk 21943 26792637 1799204341221.01 3227852 16190546 46211035
sl 20642 27702305 1786517671342.04 3103193 16837717 47643215
sv 20243 29433037 1990044011453.99 2575771 14965384 46974192
Total 463792 636216050 42889623481387.23 60368893 358999011 1055583954
DateDivide and TranslateTitle of
Presentation 7
Partitioning JRC-Acquis
For each document
Create 21 counters For each EUROVOC term in the document
Find top-level domains
Add 1 to the corresponding counter Find the domain with the most counts
Find the domain with the most counts
Classify document
Date 7
DateDivide and TranslateTitle of
Presentation 8
Partitioning JRC-Acquis
Date 8
JRC-Acquis Partitioning
04 POLITICS
08 INTL. RELATIONS
76 INTL. ORGANIZATIONS
72 GEOGRAPHY
.
.
.
Sub-corporaCorpus
DateDivide and TranslateTitle of
Presentation 9
Classes info
Date 9
Domain Description Lines Words04 POLITICS 25.197 722.63508 INTERNATIONAL RELATIONS 69.057 1.832.10010 EUROPEAN COMMUNITIES 164.223 4.310.70212 LAW 31.885 917.51016 ECONOMICS 35.207 1.114.05220 TRADE 100.306 2.485.81424 FINANCE 86.440 2.015.94328 SOCIAL QUESTIONS 47.343 1.387.419
32 EDUCATION AND COMMUNICATIONS 22.313 701.76136 SCIENCE 23 20340 BUSINESS AND COMPETITION 51.701 1.388.13344 EMPLOYMENT AND WORKING CONDITIONS 17.644 473.97948 TRANSPORT 26.696 733.19152 ENVIRONMENT 39.448 1.141.31356 AGRICULTURE, FORESTRY AND FISHERIES 88.165 2.235.48060 AGRI-FOODSTUFFS 65.331 1.449.96964 PRODUCTION, TECHNOLOGY AND RESEARCH 50.641 1.313.69566 ENERGY 7.604 226.48668 INDUSTRY 32.874 1.115.37972 GEOGRAPHY 88.119 2.348.37376 INTERNATIONAL ORGANISATIONS 9.666 271.810
DateDivide and TranslateTitle of
Presentation 10
Experimentation
Trained Domain-specific and Generic Language Models
SRILM
N-gram order: 3
Smoothing: Knesser-Ney
Trained Domain and Generic Translation Models
MOSES
Maximum sentence length: 120
Tuning (MERT) on each domain
Evaluate whether the selected domain corresponds to the best BLEU score for the respective translation
Date 10
DateDivide and TranslateTitle of
Presentation 11
Creating individual domains
11
DateDivide and TranslateTitle of
Presentation 12
Creating generic domain
12
+ +04
trainingset
08 training
set
72 training
set
76 training
set. . .
~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~
GENERICTM
GENERICLM
GENERIC test+dev set
GENERIC CORPUS
GENERIC training set
DateDivide and TranslateTitle of
Presentation 13
Results
13
Domain 04-Politics
DOMAIN PERPLEXITYBLEU
SCORE04-Politics 64.552 0.5416
10-European Com. 80.999 0.341708-Int. Relations 99.208 0.3383
28-Social Q. 102.714 0.349152-Environment 107.159 0.3502
12-Law 107.175 0.358020-Trade 109.092 0.3206
32-Education 117.673 0.360764-Pr.Tech.&Res. 117.897 0.3282
24-Finance 120.497 0.323356-Agriculture 122.930 0.3120
40-Business 123.195 0.321344-Employment 123.824 0.3423
16-Economics 126.787 0.339572-Geography 127.058 0.3186
48-Transport 129.126 0.326976-Intl. Organ. 140.126 0.3374
60-Agri-Foodstuffs 149.198 0.298568-Industry 152.372 0.3216
ALL 0.3574
Domain 08-International Relations
DOMAIN PERPLEXITYBLEU
SCORE08-Int. Relations 47.884 0.5777
10-European Com. 77.843 0.368316-Economics 112.017 0.3706
28-Social Q. 114.596 0.364524-Finance 115.043 0.3425
56-Agriculture 116.436 0.339272-Geography 116.737 0.3435
20-Trade 117.706 0.334404-Politics 118.231 0.3801
32-Education 119.815 0.378952-Environment 123.425 0.3687
12-Law 124.239 0.366140-Business 127.654 0.3411
64-Pr.Tech.&Res. 127.786 0.340160-Agri-Foodstuffs 129.150 0.3171
44-Employment 132.615 0.350148-Transport 135.504 0.3349
76-Intl. Organ. 139.757 0.357868-Industry 147.744 0.3384
ALL 0.3876
DateDivide and TranslateTitle of
Presentation 14
Results (2)
14
Domain 10-European Communities
DOMAIN PERPLEXITYBLEU
SCORE10-European Com. 47.636 0.5766
24-Finance 104.841 0.349308-Int. Relations 107.798 0.3560
28-Social Q. 118.812 0.365104-Politics 125.323 0.3813
44-Employment 126.737 0.362832-Education 126.994 0.3868
12-Law 127.538 0.371652-Environment 128.789 0.3663
40-Business 129.283 0.346616-Economics 129.351 0.369272-Geography 132.749 0.3433
64-Pr.Tech.&Res. 132.963 0.341520-Trade 135.006 0.3228
48-Transport 140.200 0.341656-Agriculture 141.078 0.328376-Intl. Organ. 147.871 0.3629
60-Agri-Foodstuffs 162.844 0.307468-Industry 165.735 0.3386
ALL 0.3906
Domain 12-Law
DOMAIN PERPLEXITYBLEU
SCORE12-Law 48.386 0.5654
10-European Com. 72.444 0.371528-Social Q. 95.177 0.3792
04-Politics 104.735 0.387908-Int. Relations 107.970 0.355944-Employment 108.837 0.3791
32-Education 110.842 0.387552-Environment 110.846 0.3686
24-Finance 116.578 0.335020-Trade 119.054 0.3280
40-Business 121.122 0.341772-Geography 121.904 0.3394
64-Pr.Tech.&Res. 123.383 0.343348-Transport 127.228 0.3431
16-Economics 127.581 0.360256-Agriculture 131.967 0.329876-Intl. Organ. 140.408 0.3616
60-Agri-Foodstuffs 153.112 0.307768-Industry 153.198 0.3417
ALL 0.3772
DateDivide and TranslateTitle of
Presentation 15
Results (3)
15
Domain 16-Economics
DOMAIN PERPLEXITYBLEU
SCORE16-Economics 75.695 0.5961
10-European Com. 129.686 0.438872-Geography 135.265 0.4461
40-Business 144.845 0.430908-Int. Relations 148.029 0.4339
24-Finance 156.598 0.424932-Education 157.289 0.4505
28-Social Q. 158.574 0.438352-Environment 162.246 0.433344-Employment 165.868 0.4383
12-Law 170.435 0.442104-Politics 173.194 0.4403
48-Transport 173.647 0.408456-Agriculture 183.242 0.407876-Intl. Organ. 187.693 0.4303
68-Industry 192.026 0.434964-Pr.Tech.&Res. 198.253 0.3772
20-Trade 214.334 0.397860-Agri-Foodstuffs 222.229 0.3898
ALL 0.4436
Domain 20-Trade
DOMAIN PERPLEXITYBLEU
SCORE20-Trade 20.321 0.6993
56-Agriculture 45.361 0.485960-Agri-Foodstuffs 45.952 0.4812
72-Geography 47.882 0.480310-European Com. 62.000 0.4446
08-Int. Relations 63.673 0.448228-Social Q. 64.489 0.465568-Industry 65.617 0.4709
32-Education 69.479 0.491564-Pr.Tech.&Res. 69.571 0.434952-Environment 70.296 0.4713
24-Finance 72.645 0.437748-Transport 73.595 0.4518
40-Business 74.429 0.447712-Law 74.671 0.4597
04-Politics 76.775 0.485476-Intl. Organ. 77.886 0.456516-Economics 80.977 0.4516
44-Employment 81.858 0.4387ALL 0.5039
DateDivide and TranslateTitle of
Presentation 16
Results (4)
16
Domain 24-Finance
DOMAIN PERPLEXITYBLEU
SCORE24-Finance 36.668 0.6569
10-European Com. 74.678 0.442108-Int. Relations 118.026 0.4142
16-Economics 142.699 0.437440-Business 147.090 0.4054
72-Geography 148.029 0.411232-Education 149.387 0.4372
44-Employment 163.200 0.414264-Pr.Tech.&Res. 164.124 0.4019
20-Trade 165.852 0.400112-Law 166.816 0.4201
04-Politics 168.382 0.436656-Agriculture 170.829 0.398976-Intl. Organ. 172.822 0.4213
28-Social Q. 172.843 0.411252-Environment 173.045 0.4210
48-Transport 174.151 0.396360-Agri-Foodstuffs 178.271 0.3885
68-Industry 192.723 0.4046ALL 0.4616
Domain 28-Docial Questions
DOMAIN PERPLEXITYBLEU
SCORE28-Social Q. 45.506 0.5800
10-European Com. 83.198 0.361556-Agriculture 88.334 0.3658
52-Environment 94.853 0.380212-Law 101.534 0.3766
20-Trade 102.380 0.334864-Pr.Tech.&Res. 105.292 0.3493
32-Education 107.702 0.376304-Politics 108.877 0.3731
72-Geography 110.176 0.340808-Int. Relations 110.400 0.343944-Employment 110.673 0.3570
48-Transport 110.821 0.349616-Economics 118.792 0.3607
60-Agri-Foodstuffs 119.393 0.323840-Business 121.580 0.333124-Finance 123.992 0.3329
76-Intl. Organ. 126.679 0.357868-Industry 133.215 0.3429
ALL 0.3829
DateDivide and TranslateTitle of
Presentation 17
Results (5)
17
Domain 32-Education
DOMAIN PERPLEXITYBLEU
SCORE32-Education 60.401 0.5981
10-European Com. 90.841 0.391328-Social Q. 110.099 0.3982
08-Int. Relations 115.880 0.376712-Law 116.422 0.4022
40-Business 118.794 0.372720-Trade 122.841 0.3516
24-Finance 123.094 0.368852-Environment 124.125 0.3901
16-Economics 125.142 0.395804-Politics 126.635 0.3944
48-Transport 126.920 0.360944-Employment 128.339 0.3856
64-Pr.Tech.&Res. 128.441 0.356072-Geography 129.539 0.366056-Agriculture 130.643 0.360776-Intl. Organ. 140.728 0.3821
68-Industry 142.599 0.380260-Agri-Foodstuffs 159.374 0.3396
ALL 0.4078
Domain 36-Science
DOMAIN PERPLEXITYBLEU
SCORE52-Environment 33.351 0.5087
28-Social Q. 37.418 0.498256-Agriculture 39.408 0.4730
10-European Com. 46.291 0.452120-Trade 50.337 0.4257
64-Pr.Tech.&Res. 53.361 0.461232-Education 56.537 0.488348-Transport 57.882 0.4533
44-Employment 59.183 0.459404-Politics 63.028 0.4702
12-Law 63.038 0.462560-Agri-Foodstuffs 65.544 0.4162
08-Int. Relations 66.678 0.428468-Industry 67.845 0.4405
16-Economics 68.114 0.448772-Geography 70.139 0.4210
24-Finance 70.578 0.424440-Business 72.025 0.4204
76-Intl. Organ. 75.036 0.4439ALL 0.4748
DateDivide and TranslateTitle of
Presentation 18
Results (6)
18
Domain 40-Business
DOMAIN PERPLEXITYBLEU
SCORE40-Business 40.779 0.651368-Industry 85.671 0.4733
72-Geography 91.311 0.453932-Education 93.351 0.4779
10-European Com. 94.110 0.438948-Transport 96.484 0.4385
24-Finance 99.133 0.417308-Int. Relations 100.393 0.4303
20-Trade 106.633 0.417756-Agriculture 113.221 0.4397
28-Social Q. 115.252 0.443716-Economics 121.243 0.4446
52-Environment 127.520 0.443312-Law 130.359 0.4406
60-Agri-Foodstuffs 132.788 0.405644-Employment 134.300 0.4316
04-Politics 138.132 0.450276-Intl. Organ. 140.920 0.4453
64-Pr.Tech.&Res. 144.033 0.4191ALL 0.4702
Domain 44-Employment
DOMAIN PERPLEXITYBLEU
SCORE44-Employment 54.777 0.5926
10-European Com. 76.071 0.380928-Social Q. 105.745 0.3703
12-Law 106.553 0.396208-Int. Relations 122.386 0.3555
16-Economics 122.830 0.374824-Finance 123.379 0.3435
32-Education 125.096 0.384204-Politics 126.009 0.3769
52-Environment 128.833 0.358340-Business 130.535 0.3448
20-Trade 132.361 0.325164-Pr.Tech.&Res. 133.646 0.3439
48-Transport 135.727 0.346972-Geography 137.461 0.337156-Agriculture 149.723 0.323576-Intl. Organ. 154.331 0.3605
68-Industry 167.210 0.345560-Agri-Foodstuffs 167.948 0.3027
ALL 0.3828
DateDivide and TranslateTitle of
Presentation 19
Results (7)
19
Domain 48-Transport
DOMAIN PERPLEXITYBLEU
SCORE48-Transport 40.096 0.6298
10-European Com. 86.438 0.401652-Environment 98.284 0.4192
72-Geography 101.568 0.395964-Pr.Tech.&Res. 101.950 0.3867
56-Agriculture 104.398 0.388420-Trade 105.452 0.3877
32-Education 106.483 0.427340-Business 106.639 0.393228-Social Q. 108.948 0.408268-Industry 109.650 0.4048
08-Int. Relations 114.109 0.391812-Law 116.492 0.4119
04-Politics 119.369 0.412944-Employment 121.457 0.3934
24-Finance 123.002 0.375460-Agri-Foodstuffs 124.095 0.3588
16-Economics 126.691 0.402876-Intl. Organ. 134.055 0.3971
ALL 0.4146
Domain 52-Environment
DOMAIN PERPLEXITYBLEU
SCORE52-Environment 40.555 0.5984
56-Agriculture 77.655 0.392628-Social Q. 79.210 0.4077
10-European Com. 80.605 0.381764-Pr.Tech.&Res. 89.542 0.3792
20-Trade 96.919 0.363248-Transport 98.608 0.3801
04-Politics 99.114 0.410912-Law 101.457 0.3959
32-Education 105.314 0.400808-Int. Relations 106.848 0.367444-Employment 110.023 0.3775
72-Geography 112.322 0.359360-Agri-Foodstuffs 112.424 0.3503
24-Finance 113.887 0.357940-Business 114.297 0.3581
16-Economics 116.147 0.384976-Intl. Organ. 119.161 0.3795
68-Industry 121.089 0.3738ALL 0.4042
DateDivide and TranslateTitle of
Presentation 20
Results (8)
20
Domain 56-Agriculture
DOMAIN PERPLEXITYBLEU
SCORE56-Agriculture 21.311 0.6965
20-Trade 44.698 0.463872-Geography 48.244 0.4590
60-Agri-Foodstuffs 49.560 0.462528-Social Q. 50.623 0.4868
52-Environment 52.310 0.490410-European Com. 59.105 0.4466
08-Int. Relations 64.748 0.435832-Education 65.439 0.4849
64-Pr.Tech.&Res. 66.492 0.438948-Transport 66.513 0.4456
68-Industry 67.578 0.459840-Business 69.321 0.4388
16-Economics 72.959 0.452224-Finance 73.614 0.4380
12-Law 73.641 0.459844-Employment 76.696 0.4419
04-Politics 77.201 0.467576-Intl. Organ. 80.833 0.4484
ALL 0.5064
Domain 60-Agri-Foodstuffs
DOMAIN PERPLEXITYBLEU
SCORE60-Agri-Foodstuffs 12.468 0.7513
20-Trade 29.111 0.553256-Agriculture 30.073 0.574672-Geography 45.996 0.5234
28-Social Q. 51.405 0.524324-Finance 51.536 0.5133
10-European Com. 52.493 0.511364-Pr.Tech.&Res. 54.639 0.502752-Environment 56.174 0.538708-Int. Relations 56.408 0.4983
32-Education 58.535 0.544016-Economics 58.863 0.5084
68-Industry 59.566 0.507848-Transport 61.018 0.4891
40-Business 62.123 0.501012-Law 63.708 0.4981
44-Employment 63.930 0.495876-Intl. Organ. 64.322 0.5015
04-Politics 69.611 0.5371ALL 0.5743
DateDivide and TranslateTitle of
Presentation 21
Results (9)
21
Domain 64-Production Tech. & Research
DOMAIN PERPLEXITYBLEU
SCORE64-Pr.Tech.&Res. 26.989 0.6789
76-Intl. Organ. 89.607 0.454620-Trade 104.523 0.4013
10-European Com. 114.279 0.419328-Social Q. 126.368 0.4230
52-Environment 129.476 0.441832-Education 142.329 0.436648-Transport 148.674 0.4065
68-Industry 148.929 0.406156-Agriculture 161.790 0.3989
12-Law 165.351 0.417444-Employment 166.413 0.417608-Int. Relations 168.470 0.4022
24-Finance 170.674 0.386804-Politics 178.878 0.4266
40-Business 180.687 0.396460-Agri-Foodstuffs 180.718 0.3729
16-Economics 196.437 0.409072-Geography 199.332 0.3891
ALL 0.4533
Domain 68-Energy
DOMAIN PERPLEXITYBLEU
SCORE10-European Com. 102.909 0.3678
52-Environment 103.278 0.391528-Social Q. 120.341 0.3704
08-Int. Relations 121.407 0.359740-Business 124.397 0.361124-Finance 128.022 0.3494
32-Education 128.202 0.389148-Transport 128.228 0.3560
64-Pr.Tech.&Res. 133.942 0.349304-Politics 135.144 0.3786
20-Trade 139.109 0.335416-Economics 140.030 0.3678
12-Law 140.222 0.362856-Agriculture 140.689 0.3396
44-Employment 143.205 0.358572-Geography 143.519 0.348176-Intl. Organ. 147.300 0.3774
68-Industry 156.370 0.360060-Agri-Foodstuffs 173.901 0.3178
ALL 0.3837
DateDivide and TranslateTitle of
Presentation 22
Results (10)
22
Domain 68-Industry
DOMAIN PERPLEXITYBLEU
SCORE68-Industry 33.509 0.6569
72-Geography 58.345 0.440940-Business 72.790 0.4251
56-Agriculture 76.111 0.414620-Trade 82.786 0.4018
32-Education 101.920 0.433548-Transport 101.944 0.3987
60-Agri-Foodstuffs 110.246 0.365928-Social Q. 124.250 0.3936
10-European Com. 124.879 0.371408-Intl. Relations 132.218 0.381364-Pr.Tech.&Res. 150.997 0.347152-Environment 152.684 0.3849
76-Intl. Organ. 153.576 0.388224-Finance 153.804 0.3494
16-Economics 156.407 0.389912-Law 157.606 0.3774
04-Politics 164.156 0.390944-Employment 178.374 0.3605
ALL 0.4343
Domain 72-Geography
DOMAIN PERPLEXITYBLEU
SCORE72-Geography 34.878 0.6754
40-Business 84.797 0.442356-Agriculture 87.246 0.4358
68-Industry 90.483 0.472510-European Com. 93.257 0.4326
20-Trade 95.614 0.425508-Intl. Relations 109.182 0.4285
16-Economics 110.311 0.448432-Education 110.528 0.4652
28-Social Q. 111.068 0.438448-Transport 115.785 0.4186
60-Agri-Foodstuffs 117.086 0.403152-Environment 121.141 0.4360
24-Finance 121.150 0.408512-Law 129.067 0.4290
76-Intl. Organ. 134.859 0.438544-Employment 137.654 0.4172
04-Politics 142.644 0.439064-Pr.Tech.&Res. 148.283 0.3877
ALL 0.4793
DateDivide and TranslateTitle of
Presentation 23
Results (11)
23
Domain 76-International Organisations
DOMAIN PERPLEXITYBLEU
SCORE76-Intl. Organ. 52.157 0.6045
64-Pr.Tech.&Res. 76.672 0.396310-European Com. 115.134 0.3852
20-Trade 125.731 0.352228-Social Q. 127.764 0.3801
52-Environment 140.542 0.385832-Education 141.639 0.3904
8-Intl. Relations 155.090 0.374568-Industry 162.352 0.3628
12-Law 163.230 0.375740-Business 163.792 0.3585
48-Transport 167.686 0.36554-Politics 170.762 0.3785
72-Geography 171.254 0.363224-Finance 173.396 0.3423
16-Economics 178.379 0.369144-Employment 178.388 0.3656
56-Agriculture 180.153 0.346060-Agri-Foodstuffs 203.256 0.3229
ALL 0.4019
DateDivide and TranslateTitle of
Presentation 24
Future work
Better classification algorithms (for new documents)
Application on other corpora (e.g. Europarl, SE-Times)
More language pairs
Add the domain-specific corpus multiple times
Different approaches on partitioning JRC-Acquis
Date 24