Embed Size (px)
Citation preview
Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4199–4207November 16 - 20, 2020. c©2020 Association for Computational Linguistics
4199
Assessing Human-Parity in Machine Translation on the Segment LevelYvette Graham
ADAPTTrinity College [email protected]
Christian FedermannMicrosoft Research
Maria EskevichCLARIN ERIC
Barry HaddowSchool of Informatics
University of [email protected]
Abstract
Recent machine translation shared tasks haveshown top-performing systems to tie or insome cases even outperform human transla-tion. Such conclusions about system and hu-man performance are, however, based on es-timates aggregated from scores collected overlarge test sets of translations and so leavesome remaining questions unanswered. For in-stance, simply because a system significantlyoutperforms the human translator on averagemay not necessarily mean that it has done sofor every translation in the test set. Further-more, are there remaining source segmentspresent in evaluation test sets that cause sig-nificant challenges for top-performing systemsand can such challenging segments go unno-ticed due to the opacity of current human eval-uation procedures? To provide insight intothese issues we carefully inspect the outputs oftop-performing systems in the recent WMT19news translation shared task for all languagepairs in which a system either tied or outper-formed human translation. Our analysis pro-vides a new method of identifying the remain-ing segments for which either machine or hu-man perform poorly. For example, in our closeinspection of WMT19 English to German andGerman to English we discover the segmentsthat disjointly proved a challenge for humanand machine. For English to Russian, therewere no segments included in our sample oftranslations that caused a significant challengefor the human translator, while we again iden-tify the set of segments that caused issues forthe top-performing system.
1 Introduction
Recent results of machine translation evaluationshared tasks indicate that state-of-the-art is nowachieving and possibly even surpassing human per-formance, with the most recent annual Conferenceon Machine translation (WMT) news task provid-
ing extensive human evaluation of systems, con-cluding that several systems performed on averageas well as human for English to Russian, Englishto German and German to English translation anda top system even surpassed human performancefor the last two language pairs.
Since 2017 the official results of the WMT newstasks have been based on the human evaluationmethodology known as Direct Assessment (DA)(Graham et al., 2016), due to its many advantagesover older technologies. DA, for example, includesquality control mechanisms that allow data col-lected anonymously from crowd-sourced workersto be filtered according to reliability.1 AlthoughWMT news task results are admittedly based onsubstantially more valid methodology than thoseusually found in general in system comparisonsusing automatic metrics such as BLEU, results inWMT human evaluations still leave some ques-tions unanswered. For example, DA scores arebased on average ratings attributed to translationssampled from large test sets, and although suchmethodology does allow application of statisticalsignificance testing to identify potentially mean-ingful differences in system performance, they donot provide any insight into the reasons behind asignificantly higher score or the degree to whichsystems perform better when translating individualsegments. Furthermore, DA score distributions pro-duced in the human evaluation of the news task arebased on individual DA scores that alone cannotbe relied upon to reflect the quality of individualsegments (Graham et al., 2015).
Past work, has however provided a means of run-ning a DA human evaluation in such a way thatDA scores accurately reflect the performance ofa system on a given individual segment (Graham
1DA is also used in other task evaluations such as VideoCaptioning and Multilingual Surface Realisation (Awad et al.,2019; Graham et al., 2018; Mille et al., 2019).
4200
et al., 2015). This method comes with the trade-offof requiring substantially more repeat assessmentsper segment than the test set level evaluation gen-erally run, for example, to evaluate all primarysubmissions in the WMT news task. In this workwe demonstrate how this method has the potentialto be employed as a secondary method of evalu-ation in WMT tasks for a smaller subset of sys-tems to provide segment-level insight into why thetop-performing systems outperform one another orindeed to investigate the degree to which humanand machine performance differs for individual seg-ments.
2 Related Work
Over the past number of years, machine translationhas been biting at the heels of human translationfor a small number of language pairs. Beginningwith the first claims that machines have surpassedhuman quality of translation for Chinese to En-glish news text, conclusions received with someskepticism and even controversy (Hassan et al.,2018), as claims of human performance resulted inre-evaluations that scrutinized the methodology ap-plied, highlighting the influence of reverse-createdtest data and lack of wider document context inevaluations (Laubli et al., 2018; Toral et al., 2018).Despite re-evaluations taking somewhat more careto eliminate such sources of inaccuracies, they ad-ditionally included some potential issues of theirown, such as employing somewhat outdated humanevaluation methodologies, non-standard methodsof statistical significance testing and lack of plan-ning evaluations in terms of statistical power. Gra-ham et al. (2019, 2020), on the other hand re-runthe evaluation, identify and fix remaining causes oferror, and subsequently confirm that, on the over-all level of the test set, with increased scrutiny onevaluation procedures, conclusions of human paritywere still overly ambitious at that time.
It was not long before results were shown to havereached human performance however, accordingto more scrutinous human evaluation procedures,as one year later at WMT 2019, MT system per-formance for some language pairs reached humanperformance and even surpassed it for two languagepairs (Barrault et al., 2019).
Although the admittedly rigorous human eval-uation employed in WMT evaluations providesvalid conclusions about systems significantly out-performing human translation, it nonetheless em-
ploys the somewhat opaque average Direct Assess-ment scores computed over large test sets of seg-ments that subsequently leave some important ques-tions unanswered in terms of human parity. Forexample, even if a system performs better on av-erage than a given human translator, this does notnecessarily mean that the system translates everysentence better than the human translator. Whena tie occurs between human and machine transla-tion, it would be useful to know how performancecompares between the two on individual segments.The current WMT human evaluation methodologydoes not allow for this, however.
In this paper, we carry out fine-grained segment-level comparison of system and human translationsusing human evaluation and provide a comparisonon the segment-level of the top-performing MTsystems from WMT-19 news task and the humantranslator for all language pairs in which a systemwas shown to either tie (English to Russian) orsurpass human performance (English to German;German to English). Human evaluation is required,as opposed to segment-level BLEU, for example,because metrics such as BLEU are not sufficientlyaccurate to identify fine-grained segment-level dif-ferences in quality, as can be seen from low corre-lations with human assessment (Ma et al., 2019).We make all code and data collected in this workpublicly available to aid future research.2
3 Segment-level Direct Assessment
Segment-level Direct Assessment requires runninghuman evaluation with sampling of translationscarefully structured to ensure that repeat assess-ment of the same set of translations occurs a min-imum of 15 times for both the translations pro-duced by the systems of interest (Graham et al.,2015). For example, this can be carried out for areduced number of translations and for a reducednumber of systems than the entire test set, sincecollecting 15 repeat assessments makes exhaustivesegment-level evaluation for every participatingsystem likely to be overly costly. It is reasonableto focus the segment-level evaluation on a sam-ple of approximately 500 translations selected atrandom for the two top-performing systems or in-deed, as we do now, the top-performing system andthe human translator. An important considerationhowever, is that regardless of which systems may
2http://www.scss.tcd.ie/˜ygraham/emnlpfindings20datacode
4201
be selected for fine-grained segment-level analy-sis, segment-level evaluation should be run for pre-cisely the same set of segments for all systems ofinterest so that a comparison of the performance ofsystems on the same segments will ultimately bepossible.
The desired number of source language seg-ments should therefore be sampled at random fromthe test set before pooling target side translationsfor the systems of interest, shuffling and arrangingthem within human intelligence tasks (HITs). Weconstruct HITs of 100 translated segments, subse-quently evaluated by humans blind to which systemhas produced each translation, or as in our case,blind to whether a human or machine produced thetranslated segment. The configuration of DA weemploy is a source-based segment-level evaluationin which human assessors are (i) shown the sourcelanguage input segment; (ii) the translated text (ei-ther human or machine-produced); and (iii) askedto rate the adequacy of the translation on a 0–100rating scale. Source-based DA has the advantageof freeing up reference translations so that they canbe included in the evaluation as if they had beenproduced by a system. Source-based DA comeswith the trade-off, however of requiring bilingualhuman assessors.
4 Experiments
In order to investigate the degree to which humanand machine perform differently on individual testset segments, we run segment-level DA on transla-tions of the same random sample of 540 segmentsby the top-performing system and the human trans-lator. We do this for each language pair in whichthere was a tie with human performance WMT-19(English to Russian) or where machine translationperformance had surpassed human translation qual-ity (German to English; English to German).3
In order to access the bilingual speakers requiredfor the source-based DA configuration we runall source-based DA HITs on an in-house crowd-sourcing platform. In total 108,829 assessmentswere collected via the in-house platform. After re-moving quality controls, we ended up with 87,211assessments for which we are confident of workerreliability, and employ all those assessments in ourfinal analysis.
3For German to English translation, although HUMAN andthe top-performing system, FACEBOOK-FAIR, are ranked inthe same cluster, the system significantly outperforms humantranslation in head-to-head significance test results.
English to Russian
0.00
0.02
0.04
0.06
0 25 50 75 100adequacy
dens
ity
systemFacebook-FAIRHuman
Figure 1: Density plot of sample of 540 accuratesegment-level DA scores for English to Russian newstranslation for top-performing system, FACEBOOK-FAIR, in WMT-19 versus the human translator wherein the official results the system tied with human perfor-mance; Human denotes evaluation of segments trans-lated by the creator of the standard WMT referencetranslations
English to German
0.000
0.025
0.050
0.075
40 60 80 100adequacy
dens
ity
systemFacebook-FAIRHuman
Figure 2: Density plot of sample of 540 accuratesegment-level DA scores for English to German newstranslation for the top-performing system, FACEBOOK-FAIR, in WMT-19 versus the human translator wherein the official results the system beat human perfor-mance; Human denotes evaluation of segments trans-lated by the creator of the standard WMT referencetranslations
4202
German to English
0.000
0.025
0.050
0.075
0.100
25 50 75 100adequacy
dens
ity
systemFacebook-FAIRHuman
Figure 3: Density plot of sample of 540 accuratesegment-level DA scores for German to English trans-lation new translation for the top-performing sys-tem, FACEBOOK-FAIR, in WMT-19 versus the humantranslator where in the official results the system beathuman performance; Human denotes evaluation of seg-ments translated by the creator of the standard WMTreference translations
Figures 1, 2 and 3 include density plotsfor human translation and the top-performingFACEBOOK-FAIR system (Ng et al., 2019) for thesame 540 translated segments from WMT-19 forthe three language pairs we investigate.
For German to English and English to Germantranslation in Figures 2 and 3 a similar patternemerges in terms of comparison of human andmachine-translated segments, as for both a slightlylarger proportion of FACEBOOK-FAIR translationsare scored high compared to the human translator– as can be seen from the higher red peak close tothe extreme right of both plots indicating that themachine produces a marginally higher number oftranslations with higher levels of adequacy. ForEnglish to Russian translation, however, a differentpattern occurs, as shown in Figure 1, as it appearsthat there are locations lower down on the adequacyscale in which the FACEBOOK-FAIR system per-forms worse than the human translator in threenoticeable locations within its score distribution.However, these differences between language pairsare somewhat unsurprising considering that humanand system were tied for English to Russian butsystem beat human in terms of statistical signifi-cance for both English to German and German to
English to Russian
+++++
+
++
++
+
+
+
++++ ++
+
+
+
+
+
+++
++
+
+++
+
++
+
+
++
+++
++
+
+ ++
+
+
+++
+
+
++
++
++
++
+ +
+
+
+
+
+
+
++
+
++
+
+
+
+
++
+ ++
+ +++
+ +++
+++
++
+
++
+
+
+
+
+
+
+++++
++ +
+
+
+
+
+
+
+
+
++
+
+
+ ++
++
+
+
+
+++
+
+
++
++++
++
++
+
+
+
++
+
+ +
+
+++
+
++
+ ++++
+ ++
++
+ +
+
++
++
++
+
++
+
++
+ + ++ ++
+
++
+
++++
+
+ ++
++
+
+++
++
++
+
++ ++
+
++
+
+++++
+
++
+
++
+ +
++
+
+++
++++
++
+
+
+
++
+
+
++
+ +++
++
+
++
+
++
++
+
+
+
+ ++
+
++
++ +
+
+++
+++
+
++ +
++
++
++
+
++
+ +
+ +
+
+
++
+
+
+
+
++
++
+
+
++
+ +
+
++
+
+
+
+++ +
+
+++
+
+ +
+
+
++
++
+
++++
+
+
+
+
+
+
++
++
++
+
+
++
++
++ +++
++
+
++
++
+
++
++ + +
+
++
++
+
+
+++
+++
+
+++
+
+ +
+
+
+
++++ +
+++ +
+
+
+
+
+
+
+
+ +
+
+
+
+
+ +
+
+
+
+
+
++
++
+
+ +++
++
+
+ ++
+
++
++
+
+
+++
+
+
++
+
+ +++ +
+
++
+
+++
++
+ +
+
+ ++
+
+
+
+
+
+
+
+++
+
++
+++ +
+
+
+
+
+
++
+
+
+
+++
+
++
++ +
+
+
+
+
+
+
++
+
++
++
+
++
+
++
+
++
+ +
+
++
0 20 40 60 80 100
020
4060
8010
0
Facebook-FAIR (src)
Hum
an A
(src
)
Figure 4: Scatter plot of accurate segment-level DAscores for top-performing system, FACEBOOK-FAIRin WMT-19 versus the human translator where in the of-ficial results the system tied with human performance;Human A denotes evaluation of segments translated bythe creator of the standard WMT reference translations;src denotes a source-based configuration of Direct As-sessment was employed to collect scores; segment-level scores for human and machine are the average ofa minimum of 15 human assessment scores
English.
4.1 Human V FACEBOOK-FAIR: English toRussian
As revealed in the WMT-19 human evaluation re-sults, a single system achieved a statistical tie withhuman assessment for English to Russian newstranslation. Differences in average overall scorescomputed on large test sets still leave some ques-tions unanswered however, particularly in terms ofwhich specific source inputs the machine or evenhuman translator might still find challenging. Fur-thermore it does not provide any insight into differ-ences in performance for specific source languageinput segments.
Since we desire the ability to examine differ-ences in translations of individual source segmentsfor machine and human we examine scatter plotsof accurate segment scores for translations of thesame input source segment by the human translatorand the top-performing machine, shown in Figure4 for English to Russian for WMT-19 data.
For English to Russian translation, the scatterplot of adequacy scores for human translator ver-
4203
sus machine shown in Figure 4, in which each “+”signifies the translation of the same source lan-guage input test segment, reveals distinct levels ofperformance for human versus machine for indi-vidual segments. Figure 4 reveals that as expectedthe vast majority of translations score high for bothhuman and machine translations, depicted by thelocation of the main bulk of translations within theupper right quadrant, as both human and machinetranslations in this quadrant received an averagescore above 50%. A perhaps more interesting in-sight revealed by Figure 4 is the lack of translationsappearing in the bottom right quadrant and this in-dicates that when the system does well on an inputsource segment so does the human. The reversecannot be said however of the system, as the upperleft quadrant in Figure 4 for English to Russian con-tains albeit a relatively small number of segments(12 or 2.4%) for which Facebook-FAIR translatespoorly while corresponding adequacy scores forthe human translator remain above 50%.
To gain more insight into what might take placein the case that either the machine or human per-forms poorly for the input segments scored be-low the 50% threshold for English to Russiantranslation see Table 1, where we include the fulltranslation examples for the two lowest scoredFACEBOOK-FAIR translations. In example (a) inTable 1 the system is scored lower because it trans-lates an unknown person on Capitol Hill incorrectly.While the human translator correctly expresses thefact that the person is from Capitol Hill, the sys-tem instead implies that the unknown person is onCapitol hill, i.e. as if that person were physicallystanding on a hill. All the other differences be-tween the human and machine in terms of selectionof words in the Russian translation are not criticaland read well in terms of the fluent Russian.
In example (b) in Figure 1 there is firstly a mis-take in the system translation as it translates de-tained into Russian as delayed instead of the cor-rect translation that is produced by the human trans-lator. Secondly, in this same example, the systemtranslates migrant children using a Russian termthat only refers to children who are migrants them-selves, while the human translator uses an arguablybetter term that includes both children who are mi-grants and the children of migrants. Finally, inexample (b) the system translation appears to losethe intensity of the causality implication that thesentence originally has in English, while the human
English to German
+
+
++
+
+
+
+ +++
+
+
+++
+
+++
+++++
+
++
+
+
++
+
+
+
+
+
+++
+ +
++
++ ++
++ + +
+++
+
++
+
+++
+
+
+ +
+++++
++++ +++ +
++
+
+
+
+++
+ ++
+
++
+
++
+
++
+
+
+
++
+++
++++
+
+
+
+
++ +
+
+
+
++
+
++
+
++
+ +++
+
++
+
+ +++
++
++
+
+++
+++
+++
+
+++
+
++
+
+
++
+
+ +
+
+
+
+ + +++ +
+
+ +
+
++
++
+ +++
+
++
+
+
+
+
+
++
+
++
+
+
++
+
+
+
+
++
+
++
+
++++
+
+
+
+ +++ +
++ ++
+
+ +
++
++
++
++
+
++
++++ ++
+++ + +++
+
++ + +++
++
+
+
+ ++
+
+
++
+++
+
+
+++
+
++
+++
+
++
+
+
+++
++ +++
++
+
+ ++++
+ ++
++ +
+
+++++
+
+++
+
+ ++
++
+
+++ +
+ ++
+
++
++
+
++
+
++
++
++
+++
++
+++
++
+
+
++++
+
+
++
+
+
+
+++
+
+
+ +++
+
++
+
+
+
+
++
+
+
+
+
+
+
+ +
+
+++
+
+++
+
+
++
++
+++
++
+
+
+
++ + +
+
++
+
+
+
+
+
+
++
+
+++
+ ++
+
+
+
+
+
+
++
++
+
+
+++ ++
+
+
+ +++
++
+
++
+
+
+
+ +
+
++
+++
+++
+
+
++ +
+ +
+
+
+
+
+ ++
+
+
+
+
++
+
+
++++
++
+
++
+
+
+
+
+++
++
+++
++
+
++
++
+
++++ ++
+
+
++
++
+
+++
0 20 40 60 80 100
020
4060
8010
0
Facebook-FAIR (src)
Hum
an A
(src
)
Figure 5: Scatter-plot of segment-level DA scores fortop-performing system, FACEBOOK-FAIR in WMT-19 versus human translator; Human A (src) denotesevaluation of segments translated by the creator of thestandard WMT reference translations in a source-basedconfiguration of DA; segment-level scores for humanand machine are the average of a minimum of 15 hu-man assessment scores
translation keeps this using the active form of theverb. Remaining English to Russian translationsfor which the system score falls below 50% areincluded in Appendix A.
As mentioned previously, for this English to Rus-sian our analysis found no translations for whichthe human translator performed very poorly whilethe system succeeded.
4.2 Human V Super-humanFACEBOOK-FAIR: English to German
In the official WMT-19 human evaluation resultsof the English to German news task, again thesame single system, FACEBOOK-FAIR, stood outas quite remarkably outperforming the human trans-lator according to human assessment scores com-puted over the entire test set (Barrault et al., 2019).In order to further investigate this super-human per-formance, after collecting accurate segment-levelscores for translations of the same 540 source lan-guage input segments for both FACEBOOK-FAIRand the human translator, we plot correspondingadequacy scores in Figure 5.
In contrast to English to Russian (Figure 4), andperhaps not surprisingly since the system signifi-cantly outperforms the human translator as opposed
4204
DA (%)
(a) Source : The information appeared online Thursday, posted by an unknown person on Capitol Hill during aSenate panel’s hearing on the sexual misconduct allegations against Supreme Court nominee BrettKavanaugh.
Facebook-FAIR: 3.3
Human: 78.9
(b) Source : The number of detained migrant children has spiked even though monthly border crossings haveremained relatively unchanged, in part because harsh rhetoric and policies introduced by the Trumpadministration have made it harder to place children with sponsors.
Facebook-FAIR: 2.1
Human: 64.8
Table 1: English to Russian example translations from WMT-19 news task for which the top-performing systemperformed poorly; DA denotes average direct assessment scores for translations computed on a minimum of 15human assessments; DA scores below the 50% threshold highlighted in orange; DA scores above the 50% thresholdhighlighted in blue
to merely tying with it, the English to German sys-tem shows fewer machine translations receiving alow adequacy score combined with a high humanscore, as only two translations appear in the top-leftquadrant of Figure 5. This highlights the fact thateven though on average the system performs in-credibly well, by on average outperforming humantranslation, there remains the possibility that thiscan take place in combination with a albeit smallnumber of poor translations.
To gain more insight into what might take placein the case that either the machine or human per-forms poorly for the input segments scored belowthe 50% threshold see Table 2. Two out of the fivetranslations that scored below 50% by either humanor machine were translated worst by machine as op-posed to the human translator as can be seen by thelower DA scores (a) and (b) in Table 2. Firstly, inexample (a) in Table 2 the system translation devi-ates from the syntactic structure of the source inputsentence. It additionally ignores and in addition totranslating scene as Unfallort (lit: location of theaccident). In contrast, the human translator insteadproduces Ort des Geschehens which is arguably abetter way to express scene.
In example (b) in Table 2, the source word troughis mistranslated as Trog by the system, which is amore common translation of the word trough but isin this context an incorrect lexical choice given that
the source input sentence originates in the weatherreport domain, for which Tief is the appropriatetranslation, which the human translator correctlytranslates.
Despite the system performing poorly on twosegments for which the human translates correctly,perhaps more surprising is that there are threesource input segments for which the machine trans-lates well but the human translator does not. Inexample (c) in Table 2, the human translates broad-cast networks somewhat too literally as Rund-funknetze instead of Rundfunksender. In addition,the human translator incorrectly changes the tense.Finally in example (c) in Table 2 full Senate is againtranslated too literally into vollem Senat.
In example (d) in Table 2, the human translatorchooses the incorrect present tense for the mainverb, kundigt ... an as opposed to the future tense.Lastly, in example (e) in Table 2, the human trans-lator converts two-foot into 60 cm which is onlyapproximately correct, the source word brim istranslated into Rand which is arguably correct butis nonetheless an unusual lexical choice comparedto the system translation, Krempe. Again, tense inthe latter part of the source input sentence is notpreserved well in the human translation.
4205
DA (%)
(a) Source : The driver of the car stopped and paramedics attended, but the man died at the scene.
Facebook-FAIR: Der Fahrer des Autos hielt an, Sanitater kummerten sich um ihn, doch der Mann starb noch am Unfallort. 40.7Human : Der Fahrer des Autos hielt an und Sanitater kamen, aber der Mann starb am Ort des Geschehens. 91.3
(b) Source : The approaching trough will bring some locally heavy rain to parts of the Southern California coastline.
Facebook-FAIR: Der herannahende Trog wird Teilen der sudkalifornischen Kuste lokal heftigen Regen bringen. 40.9Human : Das sich nahernde Tief wird einige ortlich starke Regenfalle fur Teile der sudkalifornischen Kuste mit sich
bringen.70.1
(c) Source : The cable and broadcast networks were all covering live hours later, when the Judiciary Committee was to voteto advance Kavanaugh’s nomination to the full Senate for a vote.
Facebook-FAIR: Die Kabel- und Rundfunksender berichteten alle live Stunden spater, als der Justizausschuss abstimmen sollte,um Kavanaughs Nominierung dem gesamten Senat zur Abstimmung vorzulegen.
74.5
Human : Die Kabel- und Rundfunknetze haben spater live ubertragen, als der Justizausschuss abstimmen sollte, um dieErnennung von Kavanaugh zum vollen Senat zur Abstimmung voranzutreiben.
46.5
(d) Source : Foreign buyers are set to be charged a higher stamp duty rate when they buy property in the UK - with the extracash used to help the homeless, Theresa May will announce today.
Facebook-FAIR: Auslandischen Kaufern soll beim Kauf von Immobilien in Großbritannien eine hohere Stempelsteuer in Rech-nung gestellt werden – mit dem zusatzlichen Geld, das fur Obdachlose verwendet wird, wird Theresa May heutebekannt geben.
90.9
Human : Auslandischen Kaufern wird beim Kauf von Immobilien in Großbritannien ein hoherer Stempelsteuersatz inRechnung gestellt - das zusatzliche Geld wird fur Obdachlose verwendet werden, kundigt Theresa May heutean.
49.5
(e) Source : The out-sized hats come hot on the heels of ’La Bomba’, the straw hat with a two-foot wide brim that’s beenseen on everyone from Rihanna to Emily Ratajkowski.
Facebook-FAIR: Die uberdimensionalen Hute sind auf den Fersen von “La Bomba”, dem Strohhut mit zwei Fuß breiter Krempe,den man von Rihanna bis Emily Ratajkowski gesehen hat.
87.3
Human : Die uberdimensionalen Hute haben sich an die Fersen von ”La Bomba” geklebt, dem Strohhut mit einem 60 cmbreiten Rand, der bei jedem von Rihanna bis Emily Ratajkowski zu sehen ist.
48.3
Table 2: English to German translations from WMT-19 news task for which either the top-performing system orhuman translator perform poorly; DA denotes average direct assessment scores for translations computed on aminimum of 15 human assessments; DA scores below the 50% threshold highlighted in orange; DA scores abovethe 50% threshold highlighted in blue
German to English
+
+
+
++
++
+
+
++
+
++
++
++
++
++
+
+
+++
+++++
++
++
++ +
+
+ ++
+
++ ++ ++++
+
++
+ +
+
++++++
+
+
++
+++
++
++
+
+
++
+
+
+
+++ ++
+
+
++
++
++ ++
+++
++ +
+
++
+
+
++
++
++
+
++
+
+
+
++ +
+
+
+
+
++
+
++
+
+
+
+++++
+
+++
+ +
++++
++
+
+++ +
+++ +++
+
+
++
++ +
+
++
++
++
+++
+++
++++
+
+
++++
+
++++
+
+
+ ++
++
+
+
++
+
++
+
+
+
++++
+
+
+
+
+++
+
+
+
++
+
+
+
+
+
++
+
++ +
+ +
++
+
+
++
++
++
++ + ++++
+++
+
+++
+
++
+ +
+ + +
+
+ ++++
++
++
+
++
+
+
+
++ +
+
+++++
+
+
+
+ + ++
+
+
+
++
+ +
+
+
+
+
+
+ +
+
+
+
++ +++
+
+
+
++
++
++
+
++
+
++
+
++
+
++++
+
+
++++
+
++
+
+++
+ +
+
+++ +
+ +
+
+
++
+
++
+
++
+
+
++
+++
+++ +
+
+
+
+++++
+
+
+
+
++
+
+++
++
+
++ +
++
+
+
+
+++
+
+++
+
++
++
++ +
+
+
+
+++
++
+++ +
+ + +
++
+++
+
+
+
+++ ++
++
+
++
++++
++
+
+
+
++
++
+
+
+
+ ++++
+
+
++
+++++
++
+
+
+++
++
+
+
+ ++
++
++
++
++
++
+
+
++ ++
+
+
+ ++
+++
+++ +
+
++
++++
+
+
++
++
+
0 20 40 60 80 100
020
4060
8010
0
Facebook-FAIR (src)
Hum
an A
(src
)
Figure 6: Scatter plot of adequacy scores of transla-tions of the same source language input segment pro-duced by (i) human and (ii) top-performing machinetranslation system from WMT-19, FACEBOOK-FAIRfor German to English where machine significantly out-performed human translation
4.3 Human V Super-humanFACEBOOK-FAIR: German to English
For German to English translation, the scatter-plotof translation scores for our 540 source segmentsample shown in Figure 6 reveals the bulk of trans-lations located to a more extreme degree in theupper right corner of the plot compared to the othertwo language pairs. Like both English to Russianand English to German, there are segments for t hislanguage pair for which the top-performing system,FACEBOOK-FAIR performs poorly on comparedto the human translator, as seven source segments(1.4%) appear in the upper-left quadrant, where thesystem received an adequacy score lower than 50%while the human translation received a score higherthan 50%. Like English to German, however, forGerman to English translation, the reverse is alsotrue, there are translations that catch out the humantranslator, for which he/she received a low score,while for the same source input, the machine re-ceives a high score. Such translations, there aresix in total (1.2%), are located in the bottom-rightquadrant of Figure 6.
Table 3 shows the most extreme examples in
4206
DA (%)
(a) Source : Im Ziel warf er sein Paddel vor Freude weg und reckte beide Arme siegessicher in die Hohe - wohlwissend,dass es mindestens fur eine Medaille reichen wurde.
Facebook-FAIR: At the finish, he threw away his paddle for joy and raised both arms in victory - knowing that it would be enoughfor at least one medal.
23.4
Human : He threw his paddle with joy at the finishing line and, confident of victory, threw both arms in the air - safe in theknowledge that his efforts would secure him a medal.
67.5
(b) Source : Zur Vorsicht wurde auch noch der OAMTC-Notarzthubschrauber gerufen.Facebook-FAIR: The OAMTC emergency medical helicopter was also called out as a precaution. 42.7Human : As a precautionary measure, an emergency air ambulance helicopter was also called into action. 84.5
(c) Source : Hintergrund ist Musks uberraschende Ankundigung vom August, Tesla von der Borse nehmen zu wollen.Facebook-FAIR: The background is Musk’s surprise announcement in August that he would take Tesla off the stock market. 96.0Human : The background is Musk’s surprise announcement in August to take Tesla off the stock exchange. 7.3
(d) Source : Zum 100-Jahr-Jubilaum der Republik, das in diesem Gedenkjahr seit mittlerweile fast zehn Monaten gefeiertwird, sind zahlreiche neue Bucher erschienen, die diese Frage meist im Ruckblick auf die vergangenen hundertJahre beantworten.
Facebook-FAIR: On the occasion of the 100th anniversary of the Republic, which has been celebrated in this commemorativeyear for almost ten months now, numerous new books have been published, most of which answer this questionin retrospect of the past hundred years.
97.7
Human : At the 100-year anniversary of the republic that has been celebrated in this commemorative year for almost tenmonths, many new books appeared that answer this question mainly looking back over the past hundred years.
49.1
Table 3: German to English translations from WMT-19 news task for which either the top-performing system orhuman translator perform poorly; DA denotes average direct assessment scores for translations computed on aminimum of 15 human assessments; DA scores below the 50% threshold highlighted in orange; DA scores abovethe 50% threshold highlighted in blue
terms of contrast in adequacy scores for humanversus machine translation for German to Englishfor the top-performing system FACEBOOK-FAIR.Two of the examples (a) and (b) show segments forwhich the system performs worse than the humantranslator and on close inspection we can see whythis could be. For example, the machine translatesthe source segment in example (a) in Table 3 tooliterally and omits the phrase “in the air”. Althoughthe human translator scores higher at 67.5% theyare still docked some marks probably because thehuman translator has also slightly mistranslated theGerman verb wegwerfen – to throw away, omit-ting away from his/her translation. Example (b)in Table 3 the machine translation system is hin-dered by the presence of an unknown acronymcontaining the German umlaut that remains as suchincorrectly present in the English translation, re-ceiving a score of 42.7%. The human translator,achieving a score of 84.5%, handles this better byomitting the acronym from the translation, but stillthere is possibly some meaning missing from itstranslation.
Table 3 additionally includes some examples, (c)and (d), in which it was the human translator whowas caught off guard by a particular source segmentand substantially scored lower than the machine forits translation. For instance, example (c) in Table3 the system correctly translates the German termBorse as stock market while the human translator
chooses stock exchange which has likely caused alow human assessment score, as in general com-panies are added and removed from stock marketsas opposed to stock exchanges. In example (d)in Table 3 it is likewise the human translator whotranslates the German term erschienen as appearedinstead of the more appropriate published producedby FACEBOOK-FAIR. In addition the human mis-places the translation of meist – most – which refersback to the books in the preceding phrase and at-taches it to the translation of Ruckblick – lookingback or retrospect – while the machine correctlytranslates meist. Remaining German to Englishtranslations for which either the system or humanscore falls below 50% are included in AppendicesB and C.
5 Conclusions
The question we ask in this work is highly rele-vant – what are the differences between humantranslations and the top MT translations on the seg-ment level when “human parity” is reached. Forthe English-Russian system our analysis makes itclear that there are a number of segments where thehuman did better than the machine, but on close in-spection of these sentences, there appears to be nogeneralizable difference that clearly characterizesthese kinds of sentences.
For English to/from German, the situation be-tween human and machine is more finely balanced,
4207
and segment-level analysis has shown only a smallnumber of random errors on each side, revealingonly minor differences are present even on the seg-ment level when we compare human and machinetranslations.
Acknowledgments
This study was supported by the ADAPT Centre forDigital Content Technology (www.adaptcentre.ie)at Trinity College Dublin funded under the SFIResearch Centres Programme (Grant 13/RC/2106)co-funded under the European Regional Develop-ment Fund, and has received funding from the Eu-ropean Union’s Horizon 2020 research and innova-tion programme under grant agreement No. 825299(Gourmet). We would also like to thank the anony-mous reviewers for their feedback.
ReferencesGeorge Awad, A. Butt, K. Curtis, Y. Lee, J. Fis-
cus, A. Godil, A. Delgado, J. Zhang, E. Godard,L. Diduch, A. F. Smeaton, Y. Graham, and W. Kraaij.2019. Trecvid 2019: An evaluation campaign tobenchmark video activity detection, video caption-ing and matching, and video search & retrieval. InProceedings of TRECVID, volume 2019.
Loıc Barrault, Ondrej Bojar, Marta R. Costa-jussa,Christian Federmann, Mark Fishel, Yvette Gra-ham, Barry Haddow, Matthias Huck, Philipp Koehn,Shervin Malmasi, Christof Monz, Mathias Muller,Santanu Pal, Matt Post, and Marcos Zampieri. 2019.Findings of the 2019 conference on machine transla-tion (WMT19). In Proceedings of the Fourth Con-ference on Machine Translation (Volume 2: SharedTask Papers, Day 1), pages 1–61, Florence, Italy. As-sociation for Computational Linguistics.
Yvette Graham, George Awad, and Alan Smeaton.2018. Evaluation of automatic video captioning us-ing direct assessment. PLOS ONE, 13(9):1–20.
Yvette Graham, Timothy Baldwin, Alistair Moffat, andJustin Zobel. 2016. Can machine translation sys-tems be evaluated by the crowd alone. Natural Lan-guage Engineering, FirstView:1–28.
Yvette Graham, Barry Haddow, and Philipp Koehn.2019. Translationese in machine translation evalu-ation. CoRR, abs/1906.09833.
Yvette Graham, Barry Haddow, and Philipp Koehn.2020. Statistical power and translationese in ma-chine translation evaluation. In Proceedings of the2020 Conference on Empirical Methods in NaturalLanguage Processing, Virtual. Association for Com-putational Linguistics.
Yvette Graham, Nitika Mathur, and Timothy Baldwin.2015. Accurate evaluation of segment-level ma-chine translation metrics. In Proceedings of the2015 Conference of the North American Chapter ofthe Association for Computational Linguistics Hu-man Language Technologies, Denver, Colorado.
Hany Hassan, Anthony Aue, Chang Chen, VishalChowdhary, Jonathan Clark, Christian Feder-mann, Xuedong Huang, Marcin Junczys-Dowmunt,William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu,Renqian Luo, Arul Menezes, Tao Qin, Frank Seide,Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, YingceXia, Dongdong Zhang, Zhirui Zhang, and MingZhou. 2018. Achieving human parity on auto-matic chinese to english news translation. CoRR,abs/1803.05567.
Samuel Laubli, Rico Sennrich, and Martin Volk. 2018.Has Neural Machine Translation Achieved HumanParity? A Case for Document-level Evaluation. InEMNLP 2018, Brussels, Belgium. Association forComputational Linguistics.
Qingsong Ma, Johnny Wei, OndATMej Bojar, andYvette Graham. 2019. Results of the wmt19 met-rics shared task: Segment-level and strong mt sys-tems pose big challenges. In Proceedings of theFourth Conference on Machine Translation (Volume2: Shared Task Papers, Day 1), pages 62–90, Flo-rence, Italy. Association for Computational Linguis-tics.
Simon Mille, Anja Belz, Bernd Bohnet, Yvette Gra-ham, and Leo Wanner. 2019. The second mul-tilingual surface realisation shared task (SR’19):Overview and evaluation results. In Proceedings ofthe 2nd Workshop on Multilingual Surface Realisa-tion (MSR 2019), pages 1–17, Hong Kong, China.Association for Computational Linguistics.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott,Michael Auli, and Sergey Edunov. 2019. FacebookFAIR’s WMT19 News Translation Task submission.In Proceedings of the Fourth Conference on Ma-chine Translation (Volume 2: Shared Task Papers,Day 1), pages 314–319, Florence, Italy. Associationfor Computational Linguistics.
Antonio Toral, Sheila Castilho, Ke Hu, and AndyWay. 2018. Attaining the unattainable? reassessingclaims of human parity in neural machine translation.CoRR, abs/1808.10432.
