A Comparison of Statistical Post-Editing on Chinese and Japanese Midori Tatsumi and Yanli Sun Under the supervision of: Sharon O’Brien; Minako O’Hagan;

A Comparison of Statistical Post-Editing on Chinese and Japanese

Midori Tatsumi and Yanli SunUnder the supervision of:

Sharon O’Brien; Minako O’Hagan;

Fred Hollowood; Johann Roturier

Outline

Introduction11

Evaluation on Sentence Level 33

Analysis on modifications made by SPE22

Conclusion44

Introduction

• Rule-Based Machine Translation (RBMT)

– Three Stages: • Analysis: analyze a source text into abstract lexical and structural

representations

• Transfer: convert the source language representations into target language representations

• Generation: generate the target text

Introduction


– Three Stages: • Analysis: analyze a source text into abstract lexical and structural representations



• Statistical Machine Translation (SMT)

– Two Stages:

• Training: automatically learn translation and language knowledge from parallel corpus

• Decoding: translate new sentences using the above learned knowledge

Introduction


– Three Stages:

• Analysis: analyze a source text into abstract lexical and structural representations



• Statistical Machine Translation (SMT)

– Two Stages:

• Training: automatically learn translation and language knowledge from parallel corpus

• Decoding: translate new sentences using the above learned knowledge

• Post-Editing (PE)

– Human post-editing

– Automatic post-editing

– Statistical post-editing (SPE)

Introduction

• Statistical Post-editing (SPE) of Rule-Based Machine Translation (RBMT) Output

• Knight & Chander (1994)

• Simard et al. (2007a, 2007b)

Flowchart of RBMT

Human Post-editor

Final output

Output 2

Flowchart of SPE

RBMT

Source

Final output

Output 1

SPE module

SMT

ReferenceRBMT output

RBMT

Source

Output 1

Human Post-editor

Introduction

– Experimental setting

SMT

RBMT

Human Post-editor

SPE module

Source

Final output

Output 1

Output 2


Moses

Translation Memory: 529,822 (ZH) and

143,742 (JA)

Systran -UD: 8,832 entries (ZH) and 6,363 entries (JA)

Chinese (ZH); Japanese

(JA)

English

Introduction

– Evaluate SPE: Compare Output 2 and output 1

SMT

RBMT

Human Post-editor

SPE module

Source

Final output

Output 1

Output 2


Analysis of the Modifications Made by SPEMethodology

• Pilot project

– Random selection of 100 sentences for each language

• Classify and Evaluate the changes

– Classification(Vilar et al. 2006 )• Alteration, Deletion, Addition of Content/Function words

• Form of Tense/Voice/Imperative/Formality (Politeness)

• Fixed expression

• Reordering

• Punctuation

– Evaluation (Dugast et al. 2007 )• Improvement

• Degradation

• Equivalent

Analysis of the Modifications Made by SPEQuantitative Evaluation

• Modifications distribution in Japanese and Chinese

Improvement Degradation Equivalent

ZH JA ZH JA ZH JA

AlterationContent words 137 45 19 40 28 25

Function words 38 45 6 9 17 30

DeletionContent words 0 9 0 2 0 1


AdditionContent words 4 0 3 2 2 0


Forms

Tense or Voice 6 3 0 0 3 5

Formality 0 1 1 0 0 0

Imperative 0 8 0 0 0 2

Fixed Expression 8 0 0 0 0 1

Word / Phrase Reordering 9 1 3 3 0 1

Punctuation 31 47 4 9 0 4

Total 296 217 48 72 77 85

Analysis of the Modifications Made by SPEQualitative Evaluation

• Similarities

Source MT output SPE output

the actions that you specify for that rule

JA: あなたがその規則のために指定する処理

そのルールに指定する処理

After you configure your … ZH: …在您配置您的 …配置

Deletion of function words

Punctuation


To maintain … JA: 保守するため… 維持するには…

Reverts to … ZH: 恢复对… 恢复到 ...


MPE provides an option … JA: オプションを提供します。オプションがあります .

while the synchronization is in progress…

ZH: … ，当同步进展中时 …同步处理 .

Alteration of function words


• Similarities






After you configure your … ZH: 在您配置您的… 配置…


Punctuation


To maintain … JA: …保守するため維持するには…

Reverts to … ZH: …恢复对恢复到 ...




ZH: … ，当同步进展中时 …同步处理 .


• Similarities






After you configure your … ZH: 在您配置您的配置


Punctuation


To maintain … JA: …保守するため維持するには…

Reverts to … ZH: 恢复对恢复到




ZH: ， … 当同步进展中时 …同步处理 .


• Differences

Alteration of content words

Addition of function words


console commands JA: コンソールは命じます console コマンド

number JA: 番号数

subdomains ZH: subdomains 子域


A black dash indicates that it is disabled. ZH: 黑色破折号表明它禁用。黑色线表明它已禁用。

On the Spim tab… ZH: 在 Spim 选项卡… 在 Spim 选项卡上…


• Differences

Alteration of content words

Addition of function words


console commands JA: コンソールは命じます console コマンド

number JA: 番号数

subdomains ZH: subdomains 子域


A black dash indicates that it is disabled. ZH: 黑色破折号表明它禁用。黑色线表明它已禁用。

On the Spim tab… ZH: 在 Spim 选项卡… 在 Spim 选项卡上…


ReorderingSource MT output SPE output

These threats are then… ZH: 这些威胁然后 … 然后 , 这些威胁…


(Imperative ending) JA: して下さいします


In general ZH: 一般情况下 ,… 通常情况下 ,…

Fixed expression

Imperatives forms

• Differences



These threats are then… ZH: 这些威胁然后… 然后 , 这些威胁…




In general,… ZH: 一般情况下 ,… 通常情况下 ,…

Fixed expression

Imperatives forms

• Differences



These threats are then… ZH: 这些威胁然后… 然后 , 这些威胁…




In general,… ZH: 一般情况下 ,… 通常情况下 ,…

Fixed expression

Imperatives forms

• Differences

Evaluation on Sentence Level

• Methodology– Same 100 segments– Effect of SPE on Fluency, Adequacy and PE time– Four evaluators per language – Random distribution of MT output and SPE output

Criteria Chinese Japanese

Fluency 0.276 0.598

Adequacy 0.288 0.582

Less PE time 0.284 0.624

• Kappa scores (Inter-evaluator agreement level)

– Japanese: moderate to substantial agreement

– Chinese: generally fair agreement

Source_EN Output 1 Output 2 Fluency Adequacy Less-PE time

Turns on or off the special meaning of metacharacters.

オン / オフ回転メタ文字の特別な意味。

有効または無効にメタ文字の特別な意味します . 1 / 2 / E 1 / 2 / E 1 / 2 / E

Evaluation on Sentence Level Results and Analysis

• Improvement by SPE:

– Chinese ─ Fluency and Adequacy: ≈ 40%, PE time: ≈ 50%

– Japanese ─ Fluency, Adequacy, PE time: ≈ 60%

Language Chinese Japanese

Criteria Fluency Adequacy Less PE Time Fluency Adequacy Less PE Time

MT 12.75 15.50 15.00 14.50 8.00 9.75

SPE 37.75 38.00 48.25 59.25 61.50 62.50

Equal 49.50 46.50 36.75 26.05 30.50 27.75

Total 100 100 100 100 100 100

Conclusions

• SPE generates more improvement than degradation

• Three fold for Japanese; Six fold for Chinese

• Linguistic changes vary between ZH and JA

• SPE changes are generally limited to word level

• SPE improves fluency, adequacy, and shortens PE time

Questions?

[email protected]

[email protected]

Documents

A Comparison of Statistical Post-Editing on Chinese and Japanese Midori Tatsumi and Yanli Sun Under the supervision of: Sharon O’Brien; Minako O’Hagan;