3/22/07 LIN 6932 1
LIN6932Topics in Computational Linguistics
Lecture 9: Machine Translation
Hana Filip
3/22/07 LIN 6932 2
Outline for MT Week
• Intro and a little history• Language Similarities and Divergences• Three classic MT Approaches
– Transfer– Interlingua– Direct
• Modern Statistical MT• Evaluation
3/22/07 LIN 6932 3
What is MT?
• Translating a text from one language to another automatically.Fully automatic translation lies at one end of the scale and the work of the human translator armed with pencil and paper at the other. Between them are a number of possibilities for collaboration between man and computer which include word processing, terminology databases, voice recognition and translation memory systems.
3/22/07 LIN 6932 4
Machine Translation• dai yu zi zai chuang shang gan nian bao chai you ting jian
chuang wai zhu shao xiang ye zhe shang, yu sheng xi li, qing han tou mu, bu jue you di xia lei lai.
• Dai-yu alone on bed top think-of-with-gratitude Bao-chai again listen to window outside bamboo tip plantain leaf of on-top rain sound sigh drop clear cold penetrate curtain not feeling again fall down tears come
• As she lay there alone, Dai-yu’s thoughts turned to Bao-chai… Then she listened to the insistent rustle of the rain on the bamboos and plantains outside her window. The coldness penetrated the curtains of her bed. Almost without noticing it she had begun to cry.
3/22/07 LIN 6932 6
Machine Translation• The Story of the Stone also called
The Dream of the Red Chamber (Cao Xueqin 1792)• Issues:
– Word segmentation, word order– Sentence segmentation: 4 English sentences to 1 Chinese– Grammatical differences
• Chinese does not grammatically mark tense on the verb:– additional words and tense marking used in E: as, turned to, had begun– tou -> penetrated
• Zero anaphora: a gap, in a phrase or clause, that has an anaphoric function similar to a pro-form
• No articles
– Stylistic and cultural differences• Bamboo tip plaintain leaf -> bamboos and plantains• Ma ‘curtain’ -> curtains of her bed• Rain sound sigh drop -> insistent rustle of the rain
3/22/07 LIN 6932 7
Not just literatureHansard: Canadian parliamentary proceeedings
Hansard is the traditional name for the printed transcripts of parliamentary debates in the Westminster system of government.
3/22/07 LIN 6932 8
Canadian Hansard and MT
• The bilingual nature of the Canadian federal government requires that two equivalent Canadian Hansards be maintained: one in French and one in English.
• This makes it a natural parallel text, and it is often used to train French-English machine translation programs.
• In addition to being already translated and aligned, the size of the Hansards and the fact new material is always being added makes it an attractive corpus.
• Problem: translations are accurate in meaning, but they are not always literally exact
3/22/07 LIN 6932 9
What is MT not good for?
• Really hard stuff– Literature– Natural spoken speech (meetings, court reporting)
• Really important stuff– Medical translation in hospitals
3/22/07 LIN 6932 10
What is MT good for?
• Tasks for which a rough translation is fine– Web pages, email
• Tasks for which MT can be post-edited– MT as first pass– Computer-aided human translation
• Tasks in sublanguage domains where high-quality MT is possible– FAHQT [faah-quit]
(Fully Automatic High-Quality Translation)
3/22/07 LIN 6932 11
Sublanguage domain
• Weather forecasting– “Cloudy with a chance of showers today and Thursday”
– “Low tonight 4”
• Can be modeled completely enough to use raw MT output
• Word classes and semantic features like
MONTH, PLACE, DIRECTION, TIME POINT
3/22/07 LIN 6932 12
Some MT History• 1946 Warren Weaver (mathematician, and science administrator,
director of the Division of Natural Sciences at the Rockefeller Foundation, 1932-55) and Andrew D. Booth (British crystallographer) first discuss the possibility of MT in New York
• 1947-48 idea of dictionary-based (word-for-word) direct translation
• 1949 Weaver’s “Translation” memorandum popularized the MT idea
3/22/07 LIN 6932 13
Some MT History1949 Weaver’s “Translation” memorandum popularized the MT idea• limitations of any simplistic word-for-word approach• four proposals:
– Approach the problem of multiple meanings by the examination of immediate context
– logical elements in language– cryptographic methods were possibly applicable– linguistic universals:
“Think, by analogy, of individuals living in a series of tall closed towers, all erected over a common foundation. When they try to communicate with one another, they shout back and forth, each from his own closed tower. It is difficult to make the sound penetrate even the nearest towers, and communication proceeds very poorly indeed. But, when an individual goes down his tower, he finds himself in a great open basement, common to all the towers. Here he establishes easy and useful communication with the persons who have also descended from their towers.”
3/22/07 LIN 6932 14
Some MT History
• 1952: the first MT (“mechanical translation”) conference held at MIT, 18 MT researchers
• 1954 First public demo of computer translation at Georgetown University: 49 Russian sentences are translated into English using a 250-word vocabulary and 6 grammar rules.
• 1955-65 a number of labs take up MT
3/22/07 LIN 6932 15
History of MT: Pessimism
• 1959/1960: Bar-Hillel “Report on the state of MT in US and GB”– Argued FAHQT too hard (semantic ambiguity, etc)– Should work on semi-automatic instead of automatic translations– His argument
Little John was looking for his toy box. Finally, he found it. The box was in the pen. John was very happy.
– Only human knowledge let’s us know that ‘playpens’ are bigger than boxes, but ‘writing pens’ are smaller
– His claim: in order for MT to succeed, we would have to encode all of human knowledge
3/22/07 LIN 6932 16
Bar-Hillel’s report 1959/1960
MT research – now a “multimillion dollar affair”, as he pointed out – was, with few exceptions, set on a mistaken and unattainable goal, namely, fully automatic translation of a quality equal to that of a good human translator. This he held to be utterly unrealistic, and in his view resources were being wasted which could be more fruitfully be devoted to the development of less ambitious and more practical computer aids for translators.
3/22/07 LIN 6932 17
History of MT: Pessimism1966The ALPAC (Automatic Language Processing Advisory Committee) report– Headed by John R. Pierce of Bell Labs– Main Conclusion: years of research produced no useful results,
all current MT work had to be post-edited• Supply of human translators exceeds demand• All the Soviet literature is already being translated• Sponsored evaluations which showed that intelligibility and informativeness
was worse than human translations– Results:
• MT research suffered– halt in federal funding for machine translation in the US– Number of research labs declined– Association for Machine Translation and Computational Linguistics
dropped MT from its name
3/22/07 LIN 6932 18
History of MT• 1977 METEO System, developed at the Université de Montréal was installed in
Canada to translate weather forecasts from English to French• 1968 Systran founded by Peter Toma, one of the oldest machine translation
companies, extensive work for the United States Department of Defense and the European Commission, provides the technology for Yahoo!, AltaVista's (Babel Fish) and Google's online translation services, among others
• 1970’s: – European focus in MT; mainly ignored in US
• 1980’s– ideas of using AI techniques in MT (Knowledge-based MT at Carnegie
Mellon University)• 1990’s
– Commercial MT systems– Statistical MT– Speech-to-speech translation
3/22/07 LIN 6932 19
Language Similarities and Divergences
• Some aspects of human language are universal or near-universal, others diverge greatly.
• Typology: the study of systematic cross-linguistic similarities and differences
• What are the dimensions along with human languages vary?
3/22/07 LIN 6932 20
Morphological Variation
• Number of morphemes per word:– Isolating languages
• Cantonese, Vietnamese: each word generally has one morpheme
– Polysynthetic languages• Siberian Yupik (`Eskimo’): single word may have very many
morphemes
• Degree to which morphemes are segmentable:– Agglutinative languages
• Turkish: morphemes have clean boundaries
– Vs. Fusion languages• Russian: single affix conflate more than one grammatical categories
(e.g., a case suffix fuses number, gender and case)
3/22/07 LIN 6932 21
Syntactic Variation
• SVO (Subject-Verb-Object) languages– English, German, French, Mandarin
• SOV Languages– Japanese, Hindi
• VSO languages– Irish, Classical Arabic
• SVO lgs generally prepositions: to Yuriko• VSO lgs generally postpositions: Yuriko ni
3/22/07 LIN 6932 22
Segmentation Variation
• Not every writing system has word boundaries marked by visual cues for – Chinese, Japanese, Thai, Vietnamese
• Some languages tend to have sentences that are quite long, closer to English paragraphs than sentences:– Modern Standard Arabic, Chinese
3/22/07 LIN 6932 23
Lexical Divergences
• Word to phrases:– English “computer science” = French “informatique”
• POS divergences– English ‘she likes/VERB to sing’
German ‘Sie singt gerne’/ADV– English ‘I’m hungry’/ADJ
Italian ‘Ho fame’/NOUN
3/22/07 LIN 6932 24
Lexical Divergences: Specificity
• Grammatical constraints– English has gender on pronouns, Mandarin not.
• So translating “3rd person” from Chinese to English, need to figure out gender of the person!
• Similarly from English “they” to French “ils” (masc. plural) or “elles” (feminine plual)
• Semantic constraints– English ‘brother’
Mandarin ‘gege’ (older) versus ‘didi’ (younger)– English ‘wall’
German ‘Wand’ (inside) vs. ‘Mauer’ (outside) cp. die Berliner Mauer– German ‘Berg’
English ‘hill’ or ‘mountain’
3/22/07 LIN 6932 26
Lexical Divergence: lexical gaps
• Japanese: no word for ‘privacy’
• English (and other languages): no single word for German ‘Schadenfreude’ (the enjoyment of another’s misfortune)
• English: no word for Japanese ‘oyakoko’ (something like ‘filial piety’)
• English ‘blue’ versus Russian ‘sinevoy’ (dark blue) and ‘goluboy’ (light blue)
3/22/07 LIN 6932 27
Lexicalization Patterns divergences
Leonard Talmy (1985)“Lexicalization patterns: Semantic structure in lexical forms”• English
The bottle floated out.– Manner of motion lexicalized in the verb – Direction of motion lexicalized in the ‘satellite’ (here V particle)
• SpanishLa botella salió flotando.Lit: the bottle exited floating– Manner of motion lexicalized in the gerund – Direction of motion lexicalized in the verb
3/22/07 LIN 6932 28
Lexicalization Patterns divergences
• Verb-framed lg: mark direction of motion on verb– Romance, Arabic, Hebrew, Japanese, Tamil, Polynesian,
Mayan, Bantu familiies
• Satellite-framed lg: mark direction of motion on satellite– Crawl out, float off, jump down, walk over to, run after– Rest of Indo-European (e.g., Germanic, Slavic), Hungarian,
Finnish, Chinese
3/22/07 LIN 6932 29
Structural divergences
• German: Wir treffen uns am Mittwoch
• English: We’ll meet on Wednesday
3/22/07 LIN 6932 30
Thematic divergence
• German: Mir fällt der Termin ein
• English: I remember the date
3/22/07 LIN 6932 31
MT on the web
• Babelfish:– http://babelfish.altavista.com/
• Google:– http://www.google.com/search?hl=en&lr=&clie
nt=safari&rls=en&q="1+taza+de+jugo"+%28zumo%29+de+naranja+5+cucharadas+de+azucar+morena&btnG=Search
3/22/07 LIN 6932 34
Direct Translation
• Proceed word-by-word through text• Translating each word• No intermediate structures except morphology• Knowledge is in the form of
– Huge bilingual dictionary– word-to-word translation information
• After word translation, can do simple reordering– Adjective ordering English -> French/Spanish
3/22/07 LIN 6932 36
Problems with direct MT
• German
Complex reordering of words and phrases are necessary
3/22/07 LIN 6932 37
The Transfer Model
• Idea: Starting from a structural analysis, use rules about differences between languages to translate directly from one surface structure to another: syntactic transformations (adjusting word order) and lexical transfer (selecting equivalents).
• Steps:– Analysis: Syntactically parse Source language– Transfer: Rules to turn this parse into parse for Target language– Generation: Generate Target sentence from parse tree and lexical transfer via lookup in the bilingual dictionary
3/22/07 LIN 6932 38
English to French
• Generally– English: Adjective Noun– French: Noun Adjective– Note: not always true
• Route mauvaise ‘bad road, badly-paved road’• Mauvaise route ‘wrong road’• But is a reasonable first approximation
– Rule:
3/22/07 LIN 6932 41
Lexical transfer• Transfer-based systems also need lexical transfer rules• Bilingual dictionary (like for direct MT)• English home (lexical ambiguity)• German
– nach Hause (going home)– Heimat (homeland, home country)– zu Hause (at home)
• Can list “at home <-> zu Hause”• Or do Word Sense Disambiguation
3/22/07 LIN 6932 42
Systran: combining direct and transfer
• Shallow syntactic parsing– Morphological analysis, POS tagging– Chunking of NPs, PPs, phrases– Shallow dependency parsing (subjects, passives, head-modifiers)
• Transfer– Translation of idioms– Word sense disambiguation– Assigning prepositions based on governing verbs
• Synthesis– Apply rich bilingual dictionary– Deal with reordering– Morphological generation
3/22/07 LIN 6932 43
Transfer: some problems
• A distinct set of transfer rules for each pair of languages
• Grammar and lexicon full of language-idiosyncratic generalizations
• Hard to build, hard to maintain
3/22/07 LIN 6932 44
Interlingua
• Intuition: Instead of lg-lg knowledge rules, use the meaning of the sentence to help
• Steps:– translate source sentence into meaning
representation– generate target sentence from meaning
3/22/07 LIN 6932 46
Direct MT: pros and cons (Bonnie Dorr)
• Pros– Fast– Simple– Cheap– No translation rules hidden in lexicon
• Cons– Unreliable– Not powerful– Rule proliferation– Requires lots of context– Major restructuring after lexical substitution
3/22/07 LIN 6932 47
Interlingual MT: pros and cons (B. Dorr)
• Pros– Avoids the proliferation of specific rules– Easier to write rules
• Cons:– Semantics is HARD– Useful information lost (paraphrase)
3/22/07 LIN 6932 48
What makes a good translation
• Translators often talk about two factors we want to maximize:
• Faithfulness or fidelity– How close is the meaning of the translation to the
meaning of the original– (Even better: does the translation cause the reader to
draw the same inferences as the original would have)
• Fluency or naturalness– How natural the translation is, just considering its
fluency in the target language
3/22/07 LIN 6932 49
The impossibility of translation
Hebrew “adonai roi” (= The Lord is my Shepherd)
How do you translate it into a language whose culture has no sheep or shepherds
– Something fluent and understandable, but not faithful:• “The Lord will look after me”
– Something faithful, but not fluent and natural• “The Lord is for me like somebody who looks after animals
with cotton-like hair”
3/22/07 LIN 6932 50
Statistical MT: Faithfulness and Fluency formalized
• Best-translation of a source sentence S into the target sentence T:
– Idea: build probabilistic models of faithfulness and fluency, and then combine these models to choose the most probable (= best) translation€
ˆ T = argmaxT fluency(T)faithfulness(T,S)
3/22/07 LIN 6932 51
The IBM model
• those two factors might look familiar…
• Yup, it’s Bayes rule:€
ˆ T = argmaxT fluency(T)faithfulness(T,S)
€
ˆ T = argmaxT P(T)P(S | T)
3/22/07 LIN 6932 52
Noisy channel model for statistical MT
Idea: Statistical machine translation (MT) typically takes as its basis a noisy channel model in which the target language sentence, by tradition labelled E, is seen as distorted by the channel into the foreign language F.
3/22/07 LIN 6932 53
Noisy channel model for MT
Background: The Shannon-Weaver Model of Communication
3/22/07 LIN 6932 54
Noisy channel model for MT
Idea: Assume that the foreign (source language) input F we must translate into English is a corrupted version of some English (target language) sentence E, and that our task is to discover the hidden (target language) sentence E that generated our observation sentence F.
Hidden Markov Model
3/22/07 LIN 6932 55
Noisy channel model for MT
Given a Spanish sentence to translate (source L sentence), we treat it as the output of an English sentence (target L sentence) having gone through the noisy channel, and search for the best possible ‘source’ English sentence: I.e., the probability of the foreign sentence F given the existence of E: P(F|E)
3/22/07 LIN 6932 56
More formally• Assume we are translating from a foreign language sentence F to an English
sentence E:
F = f1, f2, f3,…, fm
• We want a decoder which is given F and produces the most probable (= best) English sentence
E-hat = e1, e2, e3,…, en
E-hat = argmaxE P(E|F)1
= argmaxE P(F|E)P(E)/P(F)2 Bayes rule
= argmaxE P(F|E)P(E)
Translation Model Language Model1 The conditional probability of an English sentence E, given a foreign sentence F2 We can ignore the denominator P(F) inside the argmax since we are choosing the best English
sentence for a fixed foreign sentence F, and hence P(F) is a constant.
3/22/07 LIN 6932 57
More formally
argmaxE P(E|F) = argmaxE P(F|E)P(E)
• This equation leaves much unresolved concerning how the actual translation is to be performed, systems that presuppose it are derived from the early IBM Models originally designed for speech recognition at IBM and work at the word level.
• Called the IBM model of MT • the translation process involves translating words and then
rearranging them to recover the target language sentence.
3/22/07 LIN 6932 58
Fluency: P(T)
• How to measure that this sentence– That car was almost crash onto me
• is less fluent than this one:– That car almost hit me.
• Answer: language models (N-grams!)– For example P(hit|almost) > P(almost|was)
• But we can use any other more sophisticated model of grammar
• Advantage: this is monolingual knowledge!
3/22/07 LIN 6932 59
Faithfulness: P(S|T)probability that each word in target sentence would generate each word in source sentence.
• French: ça me plait [that me pleases]• English:
–that pleases me - most faithful–I like it - most fluent
• How to quantify faithfulness?• Intuition: degree to which words in one sentence are
plausible translations of words in other sentence
3/22/07 LIN 6932 60
Faithfulness P(S|T)
• Need to know, for every target language word, probability of it mapping to every source language word.
• How do we learn these probabilities?• Parallel texts!
– two texts that are translations of each other
3/22/07 LIN 6932 61
Word Alignment
– All statistical translation models are based on the idea of a word alignment
– French - English word alignment
3/22/07 LIN 6932 62
Word Alignment
• The IBM models require that each French word comes from exactly one English word: one-to-one and one-to-many alignments sanctioned
• Many-to-many and many-to-one alignments disallowed by basic MT models• We can represent the above alignment by giving the index number of the
English word that the French word comes from: A = 2,3,4,5,6,6,6.
3/22/07 LIN 6932 63
Word AlignmentTRAINING ALIGNMENT MODELS• All statistical translation models are trained using
a large parallel corpus. • A parallel corpus, parallel text, or bitext is a
text that is available in two languages. • For example, the proceedings of the Canadian
parliament are kept in both French and English. Each sentence spoken in parliament is translated, producing a volume with running text in both languages.
3/22/07 LIN 6932 64
Word Alignment
• First step: Sentence alignment– Figuring out which source language sentence maps to
which target language sentence
• Second step: Word alignment– Figuring out which source language word maps to
which target language word for each sentence pair (F, E).
3/22/07 LIN 6932 65
Back to Faithfulness and Fluency
• Job of the faithfulness model P(S|T) is to model “bag of words”; e.g., which words align from English to Spanish, when translating from Spanish to English.
• P(S|T) does not have to worry about lg particular facts about Spanish word order: that’s the job of P(T) (language model)
• P(T) can do Bag generation: rearrange the words so that they recover the correct word order of the target sentence (from Kevin Knight, USC/Information Sciences Institute)
-
3/22/07 LIN 6932 66
P(T) and bag generation: problem
• Problem: the ‘bag of words’ statistical MT does not model relations among words
• How about:– loves Mary John
3/22/07 LIN 6932 67
Phrase-Based MT
Recently there has been considerable interest in MT systems based not upon words, but rather syntactic phrasesSuch MT systems perform the translation by assuming that during the training phase the target language (but not the source language) specifies not just the words, but rather the complete parse of the sentence.Eugene Charniak, Kevin Knight and Kenji Yamada (2003)“Syntax-based Language Models for Statistical Machine Translation”