Proz Virtual Conference Post-editing MT overview

Copyright © 2011, Asia Online Pte Ltd Copyright © 2011, Asia Online Pte Ltd

Kirti Vashee [email protected]

Understanding Post-Editing

Copyright © 2011, Asia Online Pte Ltd

• Growth in Word Volume for Traditional Localization Projects

• Faster Turnaround Time Requirements

• Changing Translation Price-Value Expectations

• Increasing Acceptance of MT by Enterprise Buyers

• New Rapidly Growing Types of Content – Patents & Scientific Content – Customer Support & Care Content – Customer Conversations – User Generated Content


User Generated Content

Support / Knowledge Base

Communications

Enterprise Information

User Documentation

User Interface

Products

Corporate Corporate Brochures

Product Brochures

Software Products

Manuals / Online Help

HR / Training / Reports

2,000

10,000

50,000

200,000

500,000

10,000,000

20,000,000+

50,000,000+

Email / IM

Call Center / Help Desk

Blogs / Reviews

Example Words Human

Machine

Existing Markets $31.4B

New Markets


• Traditional Localization Projects – Documentation and Localization

• Focused on improving translation productivity • Same quality deliverable but faster and cheaper

• New MT Enabled Projects – Patents & Scientific Content

• Huge Volume – Hundreds of millions of words

– Customer Support & Care Content • Very high value but short-lived • Technical Support & Knowledge base

– Customer Conversations • Editing work only, focused on corrections.


Linguistic Translation Quality

Target Quality (TEP Level)

Raw MT Output Quality

The effort and linguistic work done to raise RAW MT to target quality levels is PEMT

Common Misperceptions • The target quality level is always the same as TEP or other HT standards

• The raw MT output quality is consistent from system to system

• The corrective effort is always the same from language to language

• There is little or no “a priori” control on the MT output quality

• MT error patterns are consistent from segment to segment


Target Quality

Uneven Raw MT Output Quality

Pre-Analysis of Source Material Linguistic Profiling and Identification of Key Patterns Terminology Standards Development Translation Quality Source Cleanup

Error Pattern Identification Error Pattern Correction Unknown Word Handling Development of Linguistic Rules Expansion of vocabulary Development of TL Style & Expression Data Corrective Feedback Process Development Raw Corrections Amplification

• MT engines (especially SMT-based ones) get better with feedback

• MT is not exactly the T of the TEP process

• MT engines require upfront investments and analysis for best results

• MT engines differ from language to language (FIGS easier than CJK)

• MT error patterns can vary from segment to segment


How do you pay post-editors fairly if each engine is different?

Tools Needed:

• Effective Quality metrics – Automated – Human

• Confidence scores – Scores on a 0-100 scale – Can be mapped to fuzzy TM match

equivalents

• Post Edit Quality Analysis – After editing is complete or even

while editing is in progress, effort can be easily measured


MT System Quality

Characteristics – Productivity Implications

Free Online Engines Can be useful in some languages but often lower productivity than using TM alone and impossible to adapt to specific needs 1,000 to 3,000 Words/ Day per human editor Average segment quality = 40% - 50% TM Fuzzy Match

Human TEP Process Typically produce 2,500 Words / Day per translator

Low Quality - Moses Less than 5% of these systems can outperform free online MT and best case productivity may be in the 3,000 Words/Day range Average segment quality = 50% - 60% TM Fuzzy Match

Average Expert System

These systems can provide 5,000 to 7,000 Words/Day per editor Average segment quality = 60% - 75% TM Fuzzy Match

Superior Expert These systems can provide 9,000 to 12,000 Words/Day per editor

Average segment quality = 70% - 85% TM Fuzzy Match

Exceptional MT These systems can provide 12,000+ Words/Day per editor Average segment quality = 80% - 90% TM Fuzzy Match


Data Cleaning

Data Preparation

Data Collections

Training

Diagnostics and Fine Tuning

Customer Translation Data and Linguistic Assets

Translate

Quality Assurance

Language Pair Foundation Data

Domain Foundation Data


Identify Language Pair

Identify Top Level Domain

Upload Your Data

Receive Tuning and Test Set

File

Process Data

Select Best 3000 Segments Train Engine

1

2

3 4

5

7 6

Ready to Translate

Client Asia Online Your Data Bilingual Translation Memories In domain historical translations in source and target language.

Bilingual Dictionaries and Glossaries In domain and client specific glossaries and dictionaries.

Source Language Non-Translatable Terms Source language terms such as product names and place names that should not be translated.

Target Language Monolingual Data Monolingual target language text and URLs of in-domain websites.

Quality Improvement

Plan

8

Extra Data (If Available)

Source Material To Be Translated Source material can be analyzed and processed to further improve quality.

Style Guides Rules can be added to match client style guide requirements.


LP Source Human Reference Customized Foundation

JA-EN なお, 以下の座標系の定義は以下の通り。

Definitions pertaining to the coordinate systems are given below.

Furthermore, the definition of the coordinate systems are as follows.

Furthermore, the following coordinate system as defined.

JA-EN

せん断試験の管理特性を規定し判断基準は明確か

Are the control characteristics of shearing test defined to specify criteria for judgement clearly?

Are the control characteristics of shear test defined to specify criteria for judgement clearly?

Shear test criterion for defining characteristics of the clear?

JA-EN

ベントチューブスポット溶接の強度は確認しているか

Is the strength of spot-welds on vent tubes checked?

Is the strength of spot-welds on vent tubes checked?

It is the intensity of the welding spot vent tubes?

EN-DE An alternate host can start the meeting and act as the host.

Alternative Gastgeber können das Meeting starten und als Gastgeber handeln.

Alternative Gastgeber können das Meeting starten und als Gastgeber handeln.

Stellvertretendes Gastgeber beginnen können und so zu tun, als die Tagung des Aufnahmelandes.

EN-DE You can publish a recorded training session that was created with WebEx Recorder.

Sie können eine aufgezeichnete Schulungssitzung veröffentlichen, die mit dem WebEx-Rekorder aufgezeichnet wurde.

Sie können eine aufgezeichnete schulungssitzung veröffentlichen, die mit dem WebEx-Rekorder erstellt wurde.

Sie können eine namentliche Fortbildungsveranstaltung veröffentlichen, mit WebEx Fahrtenschreiber.

EN-DE Once customer approves your request, the customer can select an application to share.

Wenn der Kunde Ihre Anforderung genehmigt, kann er eine Applikation zum Teilen auswählen.

Wenn der Kunde Ihre Anforderung genehmigt, kann der Kunde eine Applikation zum Teilen auswählen.

Wenn Verbraucher stimmt ihrem Antrag, der Kunde auswählen können, einen Antrag zu teilen.

EN-ES Remove the steel ball from the main oil gallery before cleaning.

Retire la bola de acero de la canalización de aceite principal antes de limpiar.

Retire la bola de acero de la canalización de aceite principal antes de la limpieza.

Eliminar la bola de acero de la limpieza galería antes de petróleo.

EN-ES Continuously with the ignition on and the propulsion system active.

Continuamente con el encendido conectado y el sistema de propulsión activo.

Continuamente con el encendido en posición on y el sistema de propulsión activo.

Continuamente con la ignición en activo y el sistema de propulsión.

EN-ES The average response time goal is assigned a specific time goal.

El objetivo del tiempo de respuesta medio se asigna a un objetivo de tiempo específico.

El objetivo del tiempo de respuesta medio se asigna a un objetivo de tiempo específico.

La meta media del tiempo de respuesta se asigna una meta del momento específico.

Customization teaches an engine how to translate using YOUR style and vocabulary


A method of distilling a polymerizable vinyl compound selected from the group consisting of acrolein, methacrolein, acrylic acid, methacrylec acid,

hydroxyethyl acrylate, hydroxyethyl methacrylate, hydroxypropyl acrylate, hydroxypropyl methacrylate, glycidyl acrylate and glycidyl methacrylate, the

method comprising distilling the polymerizable vinyl compound in the presence of a polymerization inhibitor using a distillation tower having

perforated trays without downcomers and wherein the temperature of the inner wall of the tower is maintained at a temperature sufficient to prevent the condensation of the vapor being distilled, whereby the polymerizable

vinyl compound is distilled without the formation of polymer.

Actual sample of Japanese to English MT output • Requires a significant terminology database effort • Special handling for long sentences • Monolingual target language analysis • Linguistic parsing



• Training of post-editors – New Skills – MT Post Editing Is Different to HT Proof Editing

• Different error patterns and different ways to resolve issues • Some LSPs are creating e-learning courses for post editors

• 3 Kinds of Post Editors – Professional Bilingual MT Post Editors:

• Often with domain expertise, these editors have been trained to understand issues with MT and not only correct the error in the sentence, but also create learning material

– Early Career Post Editors: • Editing work only, focused on corrections

– Monolingual Post Editors • Experts in the domain, but may not be fluently bilingual • With a mature engine, this approach will often deliver the best, most

natural sounding results


Metrics That Really Count

• Productivity – Words translated per day per

human resource

• Margin – improvement in the profit margin is

critical to greater use and adoption • Consistency – Writing style and terminology

MT + Human delivers higher quality than a human only approach

Raw MT often has a greater number of errors than first pass human translation but: 1. MT errors are easy to see and easy

to fix (i.e. simple grammar/ word order).

2. MT provides more accurate and consistent terminology

3. Human errors may be fewer, but harder to see and harder to fix.

MT with more total errors is often faster to edit and fix than first pass human translations with fewer number of errors.

Productivity is the Best Quality Measure

Other “Useful” Quality Indicators

Automated Metrics (Good indicators, but not absolute) • BLEU (Bilingual Evaluation Understudy)

• F-Measure (F1 Score or F-Score)

Manual Quality Metrics (Most not designed for MT, more for HT) • Edit Distance (Does not take into account complexity of edit)

• SAE-J2450 (Industry specific)

Time Margin


Standard TEP Excellent Moses

Average Expert

Excellent Expert

Translated Words / Day

2,500 3,000 6,000 9,000

Hourly Rate $45 $45 $45 $45

Word Rate 15 cents 12 cents 10 cents 7.5 cents

Daily Cost at Hourly Rate $360 $360 $360 $360

Daily Cost at Word Rate $375 $360 $600 $675

500,000 Word Project

Hourly Cost $72,000.00 $ 60,000.00

$30,000.00

$20,000.00

Word Rate Cost $75,000.00 $ 60,000.00

$50,000.00

$37,500.00

Man Days 200.00 166.67 83.33 55.56


Translator 1

Translator 2

Translator 3

Translator 4 MT + Post Editing

Human Only

12,000 10,000 8,000 6,000 4,000 2,000 0 Words Per Day

• Productivity improvement results differ by translator. The above data is derived by studying 4 different translators productivity used only human and then with the addition of MT + human post editing by professionals

• Weaker translators often tend to benefit more from technology

• Customization is key to minimizing translator frustration

• Rapid measurement and assessment of quality is key to profitability


Incremental Improvement Training


4. Manage Manage translation projects while generating corrective data for quality improvement.

2. Measure Measure the quality of the engine for rating and future improvement comparisons

3. Improve Provide corrective feedback removing potential for translation errors.

1. Customize Create a new custom engine using foundation data and your own language assets


S

– Original Source: • The original sentences that are to be translated.

– Human Reference • The gold standard of what a high quality human translation would look like.

– Translation Candidate • This is the translated output from the machine translation system that you are comparing.

S

R

C

Machine Translate Compare and Score

Multiple machine translation candidates can be scored at one time to compare against each other. E.g. Asia Online, Google, Systran

Note: C

3 Measurement Tools • Human Quality Assessment • Automated Quality Metrics • Sentence Evaluation

Original Source

Translation Candidate

Human Reference

R C


• The test set being measured: Different test sets will give very different scores. Very small test sets can give misleading results.

• How many human reference translations were used: If there is more than one human reference translation, the resulting BLEU score will be higher.

• The complexity of the language pair: Spanish is a simpler language in terms of grammar and structure than Finnish or Chinese.

• The complexity of the domain: A patent has more complex text and structure than a children’s story book. It is not practical to use two different test sets and conclude that one translation engine is better than the other.

• The capitalization of the segments being measured: When comparing metrics, the most common form of measurement is Case Insensitive.

• The size of the test set: Use 1,000 or more BLIND segments to get good assessments

• The measurement software: There are many measurement tools for translation quality. Each may vary slightly with respect to how a score is calculated

It is clear from the above list of variations that a BLEU score number by itself has no real meaning.


Excellent (4)

Read the MT output first. Then read the source text (ST). Your understanding is not improved by the reading of the ST because the MT output is satisfactory and would not need to be modified (grammatically correct/proper terminology is used/maybe not stylistically perfect but fulfills the main objective, i.e. transferring accurately all information.)

Good (3)

Read the MT output first. Then read the source text. Your understanding is not improved by the reading of the ST even though the MT output contains minor grammatical mistakes .You would not need to refer to the ST to correct these mistakes.

Medium (2) Read the MT output first. Then read the source text. Your understanding is improved by the reading of the ST, due to significant errors in the MT output . You would have to re-read the ST a few times to correct these errors in the MT output.

Poor (1)

Read the MT output first. Then read the source text. Your understanding only derives from the reading of the ST, as you could not understand the MT output. It contained serious errors. You could only produce a translation by dismissing most of the MT output and/or re-translating from scratch.

Evaluation Criteria of MT output


Human evaluators can develop custom error taxonomy to help identify key error pattern problems .


Before Machine Translation

Pre-Translation Corrections (PTC) - A list of terms that adjust the source text fixing common issues and making it more suitable for translation.

Non-Translatable Terms (NTT) - A list of monolingual terms that are used to ensure key terms are not translated.

Runtime Glossary (GLO) - A list of bilingual terms that are used to ensure terminology is translated a specific way.

After Machine Translation Target text is processed and modified.

Post Translation Adjustment (PTA) - A list of terms in the target language that modify the translated output. This is very useful for normalization of target terms.

Each of the above runtime customizations can be applied in 2 forms: Default: Applied to all jobs. Job Specific: A different set of customizations can be applied for different clients.

Source text is processed and modified.


Original Source Corrected Source

PrecisionTMWorkstations Precision™ Workstations

ChinaSingaporeSydney China, Singapore, Sydney

Hyper-VTM Hyper-V™

6TBExternal 6TB External

w/ with

TO Q1 TO QUESTION 1

— <wall/>:<wall/>

(\d)'|"(?=[ ](HD|disp|SAS|SATA)) ${1}-inch

• Support for case sensitive and case insensitive matches • Support for regular expressions


Term

New York Times

PCs Limited

Asia Online Pte Ltd

Fortune 500

John Jacob

Microsoft Office

Cisco Local Director

Man Yee Wai


Original Source Specified Translation

Portugal-Portuguese Portugais (Portugal)

Independent Software Vendor (ISV) éditeurs de logiciels indépendants (ISV)

South Holland Province La Province Hollande-Méridionale

Proof of Concept (POC) engagement mission de validation technique

HBA adaptateur de bus hôte

Fine print Clauses complémentaires

Standup HBA adapter pour adaptateur de bus hôte

HBA standup adapter pour adaptateur de bus hôte


Original Target Adjusted Target

double port 2 port

double-port 2 port

deux port 2 port

deux-port 2 port

I5 i5

e/s E/S

cloud computing Cloud Computing

ompm OMPM


• Pre-Translation Corrections • Non-Translatable Terms • Runtime Glossary • Post-Translation Adjustments

These features enable: • Normalization of terms • Control of preferred terminology • Mapping of complex rules as

specified in the style guide

Runtime Improvements

Bilingual Translation Memories Additional in domain historical translations in source and target language that were not included in earlier training.

Bilingual Dictionaries and Glossaries Additional in domain and client specific glossaries and dictionaries that were not included in earlier training. Source Language Non-Translatable Terms Additional source language terms that should not be translated that were not included in earlier training.

Target Language Monolingual Data Additional monolingual target language text and URLs of in-domain websites that were not included in earlier training.

Additional Training Data Each custom engine is a living engine and constantly improves with use. There are many new kinds of data sources that can improve an engine’s translation quality.

Fine tuning to specific formats and style guide requirements can be performed at runtime without retraining the engine.

Posted Edited Machine Translations Post editing of raw MT rapidly improves translation quality.

Data Manufacturing Language Studio™ will analyze edits and other data and manufacture new data to improve quality.


• Training data can often have gaps in coverage and an excess of data in other areas. • Gaps in coverage reduce translation quality. • Gaps can quickly be filled via post editing the machine translated output and submitting

the data back to the system for further learning. • Many gaps can be filled with monolingual data only. • Further gaps can be identified and resolved by analyzing the text that is to be translated

for high frequency terms and unknown words • In some cases incorrect data may be statistically more relevant. Post editing will raise the

relevance of the correct grammar.

Sufficient Data Threshold

Data Shortfall

Post Edited Feedback and Generated Data to Fill Gaps

Example of Training Data D

ata

Vo

lum

e

More initial data provided for training results in greater vocabulary and grammatical coverage above the Sufficient Data Threshold and less post editing feedback required.

Gaps in Topic Coverage


The quick brown fox over jumps the lazy dog

The quick brown fox jumps over the lazy dog

Language Studio™ Pro analyzes corrections and generates other examples that include the corrected phrase to fill gaps in grammatical patterns. Each post edited correction is amplified producing many

other corrective patterns, improving future translations.

Buddha jumps over Siemens Wind Power CEO jumps over

Judge jumps over Military surveillance bot jumps over

Robbie Maddison jumps over Man jumps over Cow jumps over

With IE9 in sight, Firefox jumps over Long jumper Brian Thomas jumps over

Rally car jumps over Kobe jumps over

A deer jumps over A woman jogging in a California state park jumps over

An Afghan Army soldier jumps over

the wall to Repower bench in courtroom melee 25 foot walls Tower Bridge on motorbike Grand Canyon Moon 50% market share mark a car to raise money a crazy fan! a speeding Aston Martin a motorcycle a 100-foot cliff to get away from attacker a irrigation canal while conducting a foot patrol A

dd

itio

nal

co

rre

ctiv

e d

ata

gen

era

ted

b

y La

ngu

age

Stu

dio

™ P

ro


Original Source File

Machine Translate

Raw Machine Translations

Human Post Editing

Post Edited Translations

Send Raw MT and Post Edited Translations

back to Asia Online

2

1

2

1

Data Analysis and Manufacturing

Incremental Quality

Improvement Supported File Types


Client Requirements An existing technology client that has large (100K+ docs) English knowledge base and technical support document repository and wishes to make this self-support content multilingual

Train Initial Engine Translate Subset

(~5000 docs) Edit Subset

Improve Engine Translate Documents Translated Output

Unedited documents can be retranslated multiple times as engine improves

Repeat as

Required


Correct

Mistranslation

Syntax/Grammar

Terminology

Spelling

Punctuation

Initial System

Spelling and Terminology

Human Feedback

Targeted Corrections of Bad Learning

Correct

Correct

Correct

Correct

Key

Human Feedback can raise the raw output to previously unseen quality levels


Engine Learning Iteration 1 2 5 4 3 6

Publication Quality Target

Post Editing Effort

Qu

alit

y

Post Editing Effort Reduces Over Time

The post editing and cleanup effort gets easier as the MT engine improves.

Initial efforts should focus on error analysis and correction of a representative sample data set.

Each successive project should get easier and more efficient.

Raw MT Quality

Job

Du

rati

on

Human Resources

Human Translation + Human Post Editing

MT + Human Post Editing

Job Duration and Human Resources MT with the same number of physical human resources

can reduce the time required to complete the job (job duration) vs. human only.

MT + human post editing reduces overall project duration by multiples of human only approach.


6

5

4

3

2

1

Post Editing (Human Translation)

MT Post Editing

Co

st P

er W

ord

Post Editing Cost

MT learns from post editing feedback and quality of translation constantly improves.

Cost of post editing progressively reduces as MT quality increases after each engine learning iteration.


Initial System put into production

All editors and users allowed to suggest changes which goes through vetting process

Changes are collected and added to initial corpus to drive continuous retraining

Trained Internal Experts begin initial error analysis and correction process

Experienced editors also allowed to make changes


Publication Quality Target

Post Editing Effort

Qu

alit

y

Raw MT Quality

Post-editing effort and cost can be managed by improving the quality and performance of the MT engine via corrective linguistic feedback


• Hunnect: Eastern European Language Focus

• First Engine – Customized, without any additional engine feedback

• Domain: IT / Engineering

• Words: 25,000

• Measurements: – Cost – Timeframe – Quality

• Quality of client delivery with machine translation + human approach must be the same or better as a human only TEP approach.


Time 38

Proofing

2 Days

Editing

3 Days

Translation

10 Days

Translation

1 Day

Post Editing

5 Days

Proofing

2 Days

46% Time Saving (7 Days)

With PEMT Approach

100%

20%

80%

90%

70%

40%

30%

10%

Co

st

50%

60%

25,000 Words

Copyright © 2011, Asia Online Pte Ltd 39

50%

Margin

Proofing

Editing

Translation Human Translation

TEP Editing

Proofing

Margin

Machine Translation

MT Post Editing

Proofing

Margin

25%

45%

5%

20%

30%

20%

5%

27% Cost Saving


• LSP: Sajan • End Client Profile:

– Large global multinational corporation in the IT domain. – Has developed its own proprietary MT system that has been developed over

many years.

• Project Goals – Eliminate the need for full TEP translation and limit it to MT + Post-editing

• Language Pair: – English -> Simplified Chinese. – English -> European Spanish.

• Domain: IT • 2nd Iteration of Customized Engine

– Customized initial engine, followed by an incremental improvement based on client feedback.

• Data – Client provided millions of TM phrase pairs for training – 26% were rejected in cleaning process as unsuitable for SMT training.


• Quality – Client performed their own metrics – Asia Online Language Studio™ was 5

BLEU points better than the clients own MT solution.

– Significant quality improvement after providing feedback – 65 BLEU score.

– Chinese scored better than first pass human translation as per end client’s feedback

• Result – Client extremely impressed with result

especially when compared to the output of their own MT engine.

– Client has commissioned Sajan to work with more languages

70% Time Saving

60% Cost Saving

LRC have uploaded slides and video presentation from the conference: Slides: http://bit.ly/r6BPkT Video: http://bit.ly/trsyhg

http://bit.ly/r6BPkT

http://bit.ly/r6BPkT

http://bit.ly/trsyhg

http://bit.ly/trsyhg


Linguistic Steering

Pattern Identification, Corpus Analysis, Linguistic Problem Solver, Quality Assessment,

Linguistic Asset Development and Test & Tuning Set Development

MT-Savvy Translators & Editors

Rapid Error Identification / Correction

Manufacture Corrective Data and Drive Early Development of MT Engines

Less Skilled Editors to Correct Target Language Content

Can be Monolingual, Students, Housewives

Monolingual Data Cleanup

N-gram Resolution and Preparation


Corpus Analysis & Preparation Pattern Identification Linguistic Structural Analysis Linguistic Problem Solving

Linguistic Production Process Management Translation & MT Engine Quality Assessment

Rapid Quality Assessment Effective Use and Development of Automated Measurements Steering Guidance to MT Developers

Rapid Error Detection & Correction Open minded translators Better translator workbenches and tools Skilled monolinguals with subject matter expertise (SME)

Community Management Recruiting different types of editors Quality Management


• Better quality MT systems developed by experts working together with linguists will produce the best ROI

• Low initial investment is not the best way to evaluate an MT strategy as these cheap systems often produce marginal benefits

• Careful metric based evaluation is the best way to evaluate different strategies

• Quality is most likely to be a product of systems developed in collaboration with experts (MT + Language)

• Long-term defensible competitive advantage comes from the best systems

Be Wary of Any Instant and Free Solutions


Any LSP not using MT in 5 years time will be marginalized or be a niche player.

In 5 years time, leading LSPs will be translating more content in 1 year than in the

previous 5 years combined.

There will be more demand for translators than ever before, but roles will evolve and

change.


Kirti Vashee – [email protected] Follow on Twitter: @kvashee Join the Automated Language Translation Group in LinkedIn

www.kv-emptypages.blogspot.com

Understanding Post-Editing

mailto:[email protected]

http://www.kv-emptypages.blogspot.com/

