THE SIXTH FRAMEWORK PROGRAMMEThe sixth framework programme covers Communityactivities in the eld of research, technologicaldevelopment and demonstration (RTD)for the period 2002 to 2006D4.2 Online Learning Algorithms forComputer-Assisted TranslationVersion 3.0March 19, 2008D4.2 Online Learning for CATExecutive SummaryVERSION DATE AUTHOR0.1 1/2/2008 Gabriele Reverberi, initial version0.2 20/2/2008 Contributions by Sandor Szedmak0.3 25/2/2008 Contributions by Nicol`o Cesa-Bianchi1.0 5/3/2008 Contributions by Gabriele Reverberi2.0 6/3/2008 Editing by Nicol`o Cesa-Bianchi2.1 13/3/2008 Revised by Nicol`o Cesa-Bianchi based on re-viewers comments2.2 17/3/2008 Revised by Sandor Szedmak and Gabriele Re-verberi based on reviewers comments3.0 19/3/2008 Final editing by Nicol`o Cesa-BianchiVersion 3.0 March 19, 2008 Page 1 of 49D4.2 Online Learning for CATTitle D4.2 Online Learning Algorithms for Computer-Assisted TranslationState DraftConfidentiality PUAuthor(s) Nicol`o Cesa-Bianchi, Gabriele Reverberi, SandorSzedmakConfidentality SMART consortiumParticipantpartnersUniMI, SouthamptonWorkpackage WP4Abstract We formulate CAT as an interactive machine trans-lation problem, and show how online learning algo-rithms can be applied to solve it. We describe theintegration of our adaptive learning framework withPortage, a software package for statistical machinetranslation. The eect of online adaptation, with re-spect to a nonadaptive machine translation baseline,is empirically shown by a set of translation experi-ments. Finally, we propose a new machine transla-tion approach based on margin maximization with-norm regularization.Keywords online learning, linear learning, CAT, interactivemachine translation, -norm regularizationReferencesCommentsReviewer Cyril GoutteVersion 3.0 March 19, 2008 Page 2 of 49Contents1 Overview of this document 52 Adaptive approaches to interactive machine translation 72.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 CAT and online learning . . . . . . . . . . . . . . . . . . . . . . . 82.3 Dynamic re-ranking . . . . . . . . . . . . . . . . . . . . . . . . . 112.4 Online algorithms for dynamic re-ranking . . . . . . . . . . . . . 132.5 Some further issues and considerations . . . . . . . . . . . . . . . 173 Integration with Portage 193.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2 Integrating the online learner with Portage . . . . . . . . . . . . 223.2.1 Measuring performance for CAT . . . . . . . . . . . . . . 233.2.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.3 CAT tool architecture . . . . . . . . . . . . . . . . . . . . . . . . 273.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . 293.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324 A large margin translation system 344.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343D4.2 Online Learning for CAT4.2 The translation model . . . . . . . . . . . . . . . . . . . . . . . . 354.3 Training phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354.4 Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.4.1 Phrase prediction . . . . . . . . . . . . . . . . . . . . . . . 384.4.2 Decoding . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.5 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.5.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.6 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Version 3.0 March 19, 2008 Page 4 of 49Chapter 1Overview of this documentIn Chapter 4 of Deliverable D4.1 we described a theoretical framework for thedesign and the analysis of adaptive machine translation approaches. This frame-work builds on margin-based online algorithms that learn a linear function offeatures extracted from both the source sentence and the candidate translation.This linear function can be used to rank a set of translations of the same sourcesentence built on a given phrasetable, an operation that can be performed by asuitably modied decoder.In this document we describe the application of the abovementioned frame-work to the task of computer-assisted translation (CAT). Our adaptive MTtechniques are based on the statistical approach to machine translation (SMT).Unlike the usual suite of CAT tools, such as translation memories, concordancersand terminology databases, SMT can generate reasonably uent translations ofpreviously unseen sentences by combining the entries of a large table of sourcelanguage and target language phrase pairs. Each pair in the table is given ascore measuring the statistical plausibility of traslating the source phrase withthe target phrase. The SMT system uses these scores, along with a model ofthe target language, to drive the search in the phrase table.Adaptive SMT techniques have the additional property of being able to ex-ploit user feedback information to increase their performance. This makes CATan excellent domain for the validation of adaptive MT systems. Indeed, thepost-editing of the systems suggested translations performed by the user natu-rally provides the feedback that the adaptive module can use to improve futuretranslations.5D4.2 Online Learning for CATAlthough there is a very natural t between the online learning protocol and theinteractive machine translation model underlying CAT applications, there arealso several peculiarities that need to be addressed in a practical implementa-tion. For instance, modern CAT systems are based on sophisticated translationmemories that suggest translations to source sentences that are similar enoughto sentences that have appeared previously. The machine-translation modulemust thus interface with the translation memory in a proper way, so that theformer comes into play when the latter fails. Other peculiarities are more intrin-sic to the machine translation problem; for example, the problem of extractinginformative features from sentences. In this work, we associate features mainlywith phrasetable entries, as these are the basic modules used by the decoder tobuild translations. By doing this we end up managing a large number of fea-tures, with a potential risk of overtting due to the extremely sparse encoding.The document is organized as follows. In Chapter 2 we describe CAT as an in-teractive machine translation problem, and show how online learning algorithmscan be applied to solve it. Then we recall the framework of Deliverable D4.1and describe in detail the specic learning algorithms that we use for CAT. InChapter 3 we describe the integration of our adaptive framework with Portage,a software package for statistical machine translation. We use Portage to builda phrasetable from a sentence-aligned parallel corpus. Moreover, we use thePortage decoder to generate a list of candidate translations that are rankedbased on the linear function learned by our adaptive module. The eect ofonline adaptation, with respect to a nonadaptive machine translation baseline,is empirically shown by a set of translation experiments. Finally, in Chapter 4we propose a new machine translation approach based on margin maximizationwith -norm regularization. Experiments show that this new approach is com-petitive with state-of-the-art statistical MT systems. Ways of integrating thisnew system with online learning for CAT are under investigation.Version 3.0 March 19, 2008 Page 6 of 49Chapter 2Adaptive approaches tointeractive machinetranslation2.1 IntroductionThe availability of sophisticated phrase-based statistical machine translationsystems has promoted the development of interactive machine translation ap-proaches. IMT approaches provide models of human-machine interaction thatare suitable for the implementation of machine translation techniques in tasks ofcomputer-assisted translation (CAT). While the initial focus of IMT was on thedisambiguation of the source language (see, e.g., the MIND system [19]), morerecent approaches center the interaction on the structure of the target text [13].This target-text mediated approach to IMT opens a range of new interestingpossibilities in which SMT systems play a prominent role.An important example of this new approach to IMT are the TransType [14, 22,23] and TransType2 [12] projects. The interaction model adopted there is sim-ple: the SMT system interfaces with the CAT tool and provides completions tothe current sentences that are compatible with the input typed by the transla-tor. Furthermore, the system is able to revise its suggestions in real time, as thetranslator enters new characters. If the system provides a correct suggestion,the translator can accept it with a keystroke, thus saving time in producing the7D4.2 Online Learning for CATtarget text. Otherwise, the translator may ignore the systems suggestion andcontinue to type the intended translation.State-of-the-art SMT systems are based on several modules and data structures:decoder, phrasetable, language model, and their associated tunable parameters.These modules are built/trained on large corpora of sentence-aligned paralleltext. In traditional CAT applications the system undergoes a training phasebefore being used in interactive mode. This means that the SMT module doesnot further adapt during the interaction with the user.The purpose of this chapter is to explore adaptive approaches to interactivemachine translation. In particular, our goal is to take advantage of the feedbackprovided by the CAT user to progressively improve the performance of the SMTsystem.2.2 CAT and online learningIn order to design an adaptive SMT module for interactive machine translation(IMT) we rst have to specify a reference model of human-machine interactionfor CAT. This in turn requires a basic description of the main element of amodern CAT system: the translation memory.A translation memory (see, e.g., [3]) consists of text segments in a source lan-guage and their translations into one or more target languages. These segmentscan be blocks, paragraphs, sentences, or phrases. A translator rst supplies asource text The program then analyzes the text, trying to nd segment pairs inits translation memory where the text in the new source segment matches thetext in the source segment in a previously translated segment pair. The programthen presents any matches it nds as translation proposals to the translator forreview. The translator can accept the proposal, reject it, or make modicationsand use the modied version. In this case, the modied version is recorded andsaved again in the translation memory.The translation memories used in our application (see Section 3.3) are based onpartial and/or fuzzy matches of sentences. That is, the memory returns to theuser a sentence pair whenever the match between the source part of the pairand the current source sentence goes above a certain threshold. In practice, thetranslation memory software might use several proprietary criteria to measurethe degree of matching between two sentences.In this context, the SMT system can be used to generate suggestions for the caseswhen the translation memory does not have a good enough match for the sourceVersion 3.0 March 19, 2008 Page 8 of 49D4.2 Online Learning for CATONLINELEARNERTRANSLATIONREFERENCESENTENCESOURCETRANSLATIONMEMORYTRANSLATIONCANDIDATESMT SYSTEMUSER122a45637Figure 2.1: A high-level description of the adaptive CAT system informationow. A source sentence is sent to the translation memory (1). If a matchis found (2), then the corresponding candidate translation is sent back to theuser. Otherwise (2a), the source sentence is passed on to the online learner. Thelearner interacts with the SMT system (3) and obtains a candidate translation(4) which is sent to the user (5). In response, the user generates a referencetranslation (6) that is sent to both the translation memory and to the learner(7).sentence. The systems suggestion is then displayed to the user, who possiblyperforms post-editing actions in order to obtain the desired output, which wecall the reference target sentence. The pair (source sentence, reference sentence)can be used to incrementally train the SMT system. The goal of this trainingis to improve the translation quality of source sentences that are related to theones previously observed (we cannot hope to translate a sentence whose mostwords occur for the rst time), but not similar enough to give a match in thetranslation memory.We summarize this interaction in Figure 2.1. The dashed line connecting thereference to the translation memory reects the common practice of addingnew entries to a translation memory only after a further revision performedby a dierent translator. This is experienced by our system only as a highernumber of memory faults, which may occur even on sentences that were recentlyVersion 3.0 March 19, 2008 Page 9 of 49D4.2 Online Learning for CATobserved.In machine learning several models of interaction between learner (the adaptivemodule in our application) and environment (in our case, the system formed bythe user, the translation memory, and the sequence of source sentences to betranslated) are considered. The online learning protocol (see, e.g., [9]) ts wellwith our CAT scenario. In this protocol the elements of an unknown sequence(x1, y1), (x2, y2), . . . of object pairs are revealed to the learner one by one. Ateach time t = 1, 2, . . . the learner observes xt and must output a prediction forthe associated element yt. Once this prediction is generated the actual yt isrevealed, which the learner can use to rene the next predictions. This can besummarized with the following loop, for t = 1, 2, . . .1. The environment generates an instance xt2. The learner generates a prediction yt3. The environment reveals the element yt associated with xt4. The learner uses the pair (xt, yt) to rene its predictive rule.In our CAT scenario, instances xt are source sentences and their associatedelements yt (which we also call labels) are the reference translations.If we specialize the above schema to CAT, also including the translation memoryin the loop, we obtain the following.1. The next source sentence xt is read.2. If there is a (possibly partial) match (x

t, y

t) in the translation memory,then y

t is used as tentative translation for xt.3. Otherwise, xt is fed to the learning module that generates a tentativetranslation yt.4. The user observes either y

t or yt and performs post-editing operationsresulting in a reference translation yt.5. The pair (xt, yt) is returned to both the translation memory and to thelearning module.The idea pursued in this chapter is to feed each returned pair (xt, yt) to an onlinelearning algorithm that uses this pair to perform incremental adjustments of theSMT module parameters. The overall goal is to obtain a consistent increase inVersion 3.0 March 19, 2008 Page 10 of 49D4.2 Online Learning for CATthe translation quality as more feedback is generated by the user of the CATsystem. This increase in the translation quality should in turn correspond toa reduction of our main performance criterion: the average time the CAT userneeds to translate a certain number of source sentences.2.3 Dynamic re-rankingA simple way to design online learning algorithms is to consider predictionsthat are linear functions of their inputs. In order to do that, we need to rep-resent sentences as vectors of real numbers. We do that via features fi; thatis, functions that map sentence pairs (x, y) into numbers fi(x, y), where x is asource sentence and y is a target sentence (which we view as a candidate trans-lation for x). These features measure relevant properties of the sentence pair.In our online learning architecture for CAT, we obtain a base set of featuresby constructing a map fi for each line i = 1, 2, . . . of the phrasetable. Morefeatures of this kind are described in Chapter 3. Before describing how thesefeatures are computed, we dene how a linear prediction using the feature valuesis computed.Let f(x, y) = f1(x, y), . . . , fd(x, y)

be the feature vector associated to a pair(x, y) of sentences. A linear online algorithm maintains a corresponding vectorof real weights, w = (w1, . . . , wd), where wi is the weight associated with thefeature fi. The linear prediction is then based on the inner product w

f(x, y)as follows.The online algorithm uses the stream (x1, y1), (x2, y2), . . . of source sentencesand reference translations fed back by the user to incrementally adjust its weightvector. For each source sentence xt we introduce the set gen(xt) of candidatetranslations for xt. The online algorithm relies on this function for computingits translation of xt, which is dened by yt = argmaxygen(xt)w

t1f(xt, y) (2.1)where wt1 is the weight vector learnt by the algorithm after observing the rstt 1 pairs.Following the dynamic decoding approach of [25], we use the SMT decoder toimplement (in an approximate way) the argmax operator. That is, we cou-ple online learner and decoder in such a way that, for the current source sen-tence xt, the score the decoder gets when using the i-th phrasetable entry iswi,t1fi(xt, yt). Thus, if we assume that fi(xt, y) > 0 if the i-th phrasetableVersion 3.0 March 19, 2008 Page 11 of 49D4.2 Online Learning for CATentry is used by the decoder when xt is translated with y, and fi(xt, y) = 0otherwise, then the total score of the pair (xt, y) is exactly w

t1f(xt, y). Sincethe decoder orders candidate translations y approximately in decreasing orderwith respect to their total score, we see that the online algorithm obtains thetranslation yt simply by choosing the 1-best translation of the decoder. Notethat in this setup gen(x) is the set of all possible translations of the source xgiven the decoders phrasetable content.After obtaining yt from the decoder, the online algorithm performs the weightupdate wt1 wt. In principle, this update could use the information providedby the decoders ordering of gen(xt). In practice, we only use the decoders N-best list, denoted by nbest(xt), for a suitable value of N. This is typically amuch smaller set than gen(xt).From the decoders viewpoint, the changes performed by the online algorithm onthe weight vector wt aect the selection of the phrasetable entries so that certainentries tend to be preferred over others. In this respect, choosing phrasetableentries as base features is a natural thing to do, as they are the basic buildingblocks that the decoder uses to cover the source sentence. Clearly, we couldhave chosen many other dierent features (lexical, part-of-speech, positional,etc.) see, e.g., [29, 25, 2] for some examples.It is worth mentioning now that we distinguish two basic modes of operation forthe decoder. In the rst mode, where there is no online adaptivity, the decodingprocess is ruled by the base decoder features, such as the (forward and backward)translation model log-probabilities, the language model log-probabilities, thedistortion penalties, and so on. The decoder computes the score of a translationby combining the values of the base features with a set of coecients thatare tuned on a development set (using heuristics like Minimum Error RateTraining [28]). The role of the base feature coecients is dierent from theone played by the weight vector wt. To see this consider the source sentencext and a candidate translation y formed by the decoder using the phrasetableentries i1, . . . , im. Let g1, . . . , gn be the decoders base feature functions. Thenthe score of (xt, y) is computed as followss(xt, y) =mj=1

nk=1ck gk(ij)

(2.2)where c1, . . . , cn are the base feature coecients. Note that here we are onlyconsidering the part of the score that is contributed to by local features gk.That is, features dened on phrases only.In the second mode of operation, the adaptive mode, the decoding process isVersion 3.0 March 19, 2008 Page 12 of 49D4.2 Online Learning for CATruled by the phrasetable features. In the above example, the score of (xt, y) isnow computed as followssada(xt, y) =mj=1wij fij(xt, y) (2.3)where, we recall it, fi is the feature associated with the i-th prhasetable entry.As further discussed in Chapter 3, our CAT system is built in two phases: rst,we build a phrasetable on a training corpus. Then, we train the base featurecoecients on a development corpus. After these two phases, in order to trans-late further sentences we have the option of running the system in nonadaptivemode (with the chosen set of tuned base feature coecients) or adaptive mode(with dynamically changing phrasetable weights).In order to ensure that when the system is started in adaptive mode, we im-mediately improve on the base performance level reached in nonadaptive mode,we use the base feature scores as values of our phrasetable features. That is, ifthe ij-th phrasetable entry is used for the translation (xt, y), thenfij(xt, y) =nk=1ck gk(ij)and we adjust (2.3) so that the score of (xt, y) in adaptive mode is computed assada(xt, y) =mj=1

1 + wij

fij(xt, y) .This ensures that at the beginning of the adaptive mode, when wi = 0 forall entries i, the score of each pair (xt, y) equals the score (2.2) computed innonadaptive mode.As we explained earlier, the decoder also uses nonlocal features that are eval-uated on partial translations of the source sentence, such as the features basedon the language model and on the distortion penalty. These nonlocal featuresinuence the decoder even in adaptive mode; however, they are not accessed bythe online algorithm which acts on local features only.2.4 Online algorithms for dynamic re-rankingIn this section we introduce the linear online algorithms that are used by ourdynamic re-ranking application. We rst recall some notions and terminologydened in Deliverable D4.1.Version 3.0 March 19, 2008 Page 13 of 49D4.2 Online Learning for CATAs mentioned earlier, our goal is to increase the translation quality with the aimof reducing the time needed to translate. In order to take immediate advantageof the user feedback, our online algorithms learn at the sentence-level, and usesmoothed variants of the bleu metric (see, e.g.,[25, footnote 5]) in order tomeasure the quality of each translated sentence. 1As a remark, note that our learning algorithms are oblivious to the choice ofthe quality metric. In particular, they can work with any metric that, givena pair (y, y

) of candidate translations, returns a nonnegative number (y, y

)measuring the similarity of y and y

. In addition, we also need to be bounded,say < 1, and that (y, y) = 1 for every target language sentence y.We dene the target sentence yt for the source xt byyt = argmaxynbest(xt)bleu(yt, y) .This denition is needed to manage the cases when yt nbest(xt). That is,when the decoder fails to put the reference translation yt among the N besttranslations. In general, the decoder may not even be able to generate yt dueto missing entries in the phrasetable.Our general goal is to learn w such that dierences in bleu are reected bycorresponding dierences in margin values,w

f(xt, yt) w

f(xt, y) bleu(yt, yt) bleu(yt, y) y nbest(xt) .This approach has the obvious advantage of taking into account the informationprovided by bleu metric, which is our proxy for translation quality. In practice,this should prevent any bad or not-so-good translation from appearing in the toppositions of the nal ranked list. To achieve this goal, we apply the cost-sensitiveclassication framework of [11, 32, 33]. For each (xt, yt) and y nbest(xt)dene

t(y) = bleu(yt, yt) bleu(yt, y)t(y) = f(xt, yt) f(xt, y) .Then, for any xt we would like to enforce w

t(y) t(y) for all y nbest(xt).Note that each target sentence y nbest(xt) denes a constraint for w. Inwhat follows, we often identify sentences with constraints.1We are aware that this approach could lead to unsatisfactory results because bleu andother metrics, even when they correlate well with human judgement at the document level,might not function well at a sentence level.Version 3.0 March 19, 2008 Page 14 of 49D4.2 Online Learning for CATWe can perform a sort of bias-variance decomposition of the overall translationquality,bleu(yt, yt) = bleu(yt, yt)+bleu(yt, yt) bleu(yt, yt) .The rst term in the right-hand side is the (negative) bias, measuring thequality of the best translation in nbest(xt), the so-called oracle bleu. Thesecond term is the (negative) variance, measuring how much worse is thesystems translation yt with respect to the oracle bleu. Depending on thechoice of the online algorithm, the evolution of these two terms may change:sometimes the algorithm is able to rank rst yt (improving on the second term),but the overall performance remains the same because bleu(yt, yt) gets worse(the N-best list does not contain good translations).We now introduce the main algorithms that are used for the experiments ofChapter 3. All algorithms we consider are based on the following simple additiveupdate rule wt = wt1 +vt for dierent choices of vt. This update arises fromthe solution to the following optimization problemminw

12 |w wt1|2+ C maxynbest(xt)Ht(w, y)

(2.4)where C > 0 is a free parameter and Ht(w, y) is the hinge loss dened byHt(w, y) =

t(y) w

t(y)

+.The problem (2.4) is equivalent to the minimization ofminw,>0

12 |w wt1|2+ C 2

under the constraint w

t(y)

t(y) . The loss function appears underthe square root because of the quadratic slack variable 2(see Deliverable D4.1for details).In choosing updates, we have to strike a good trade-o between ecacy andeciency. The eciency requirement comes mainly from the need of terminatingthe update before the user comes up with the next sentence to translate.Passive-AggressiveThe simplest update we consider is Passive-Aggressive type II [11]. This updateis only applied when the 1-best translation yt = argmaxynbest(xt)w

t1f(xt, y)Version 3.0 March 19, 2008 Page 15 of 49D4.2 Online Learning for CATviolates the constraint w

t1t( yt)

t( yt). That is, when Ht(wt1, yt) > 0.The choice of vt is given by the closed-form solution of the following simpliedversion of (2.4)minw

12 |w wt1|2+ CHt(w, y)

.This solution is wt = wt1 +vt where vt = t( yt) ( yt) andt( yt) = Ht(wt1, yt)|t( yt)|2+ 1C. (2.5)When C , after the update the constraint is satised with equality. Thiscorresponds to an Euclidean projection of the old weight onto the hyperplane

w Rd: w

t =

t( y). As this update only aects wt1 in the directiont( yt) corresponding to the constraint associate with a single y nbest(xt),we call this a local update.Simultaneous projectionsThis is a simple example of global update. That is, when wt1 is moved ina direction which in general is not orthogonal to the hyperplane associatedwith any specic constraint. The update, proposed in [1], consists in movingwt1 in a direction computed as the average of the directions t(y) (y) foreach violated constraint y nbest(xt), where t(y) is the learning rate of thePA-II update (2.5). Let Vt be the set of violated constraints at step t. Thatis, Vt = y nbest(xt) : Ht(wt1, y) > 0. Then the update is dened bywt = wt1 +vt wherevt = 1[Vt[yVtHt(wt1, y)|t(y)|2+ 1Ct(y) .Gradient ascentThis is a slightly more complex global update performing gradient ascent on theconvex dual of (2.4). Let : nbest(xt) R be a nonnegative function and letvt() =yYt(y)t(y) .Version 3.0 March 19, 2008 Page 16 of 49D4.2 Online Learning for CATParameters: Learning rate base constant a > 0; number K of iterations.Initialization: 0(y) = 0 for all y Yt.For t = 1, . . . , K1. For each y Yt, let

t(y) = t1(y) + atDt(t1)(y)2. t = Euclidean projection of

t on [0, )|Yt|Output: KFigure 2.2: Gradient ascent with Euclidean projection.As shown in Deliverable D4.1, the dual of (2.4) isDt() = 12 |vt()|2 12C

yYt(y)

2+yYt(y)Ht(wt1, y)where, in order to keep the notation tidy, we set Yt = nbest(xt). The vari-ables (y) for y nbest(xt) are the Lagrange multipliers associated with theconstraints w

t(y)

t(y).The partial derivative of Dt with respect to any (y) is readily computed,Dt()(y) = (y)

vt() 1Cy

Yt(y

) + Ht(wt1, y) .In order to enforce the constraint (y) 0 for y Yt, after each gradient stepwe project back to the nonnegative orthant [0, )|Yt|.The nal update is wt = wt1 + v(K) where K is computed via K stepsof gradient ascent (see Figure 2.2). The choice of the learning rate a/t ensuresconvergence to the maximum of Dt in time 1/, where is the desired accuracy.This provided a is chosen not larger than the smallest nonzero eigenvalue of theHessian of Dt see [18] for details.2.5 Some further issues and considerationsThe successful application of online learning techniques to CAT requires thecareful consideration of some peculiarities of the language translation domain.Version 3.0 March 19, 2008 Page 17 of 49D4.2 Online Learning for CATIn this section we list some of them and make some preliminary considerations.Further discussion is found in Chapter 3.When translating a sequence of source sentences we expect some of these sen-tences to be much easier to translate than others. This entails a potentiallylarge variance of the performance in a sequence of translations. In order toreduce this variance we use the averaging technique of [16], in which the weightvector used at time t to translate xt is the average of all past vectors, wt1 = 1tt1s=0ws .Note that this requires decoding twice at each time: once to obtain the systemsprediction, which is the 1-best with respect to wt1, and once to obtain the N-best with respect to wt1, which is then used to make the update wt1 wt.Another aspect that is specic to this CAT context is the fact that the perfor-mance of the online learner is measured on a subset of the original sequence ofsource sentences. This is due to the ltering action of the translation memory,which intercepts all sources that are close enough to sentences seen previously.This implies that the actual subsequence observed by the learning module ismade up of sentences that are typically dissimilar to each other. Thus, in acertain sense, the presence of the translation memory makes the learning taskmuch harder.A nal issue concerns the potential advantage one could obtain by adding tothe phrasetable segments originating from new sentences (translation modeladaptation). This is clearly a good thing to do in order to reap the benets ofthe online approach. However, there are serious technological problems relatedto the need of performing segmentation and alignment during the full translationprocess (rather than only when the phrasetable is rst constructed from thetraining corpus).Version 3.0 March 19, 2008 Page 18 of 49Chapter 3Integration with Portage3.1 IntroductionPortage is a statistical machine translation system that has recently been madeavailable to Canadian universities and research institutions under a distributionpackage named Portage Shared. See also [30].Portage is based on the statistical approach to machine translation. This ap-proach is becoming more popular thanks to the availability of large-size trainingcorpora, which allow SMT to obtain generally better performances than tradi-tional rule-based machine translation. The success of SMT systems, especiallyin applications such as computer-assisted translation, is however hampered bysome shortcomings.1 SMT systems do not easily manage cases when training and testing cor-pora come from dierent domains (domain adaptation is currently an areaof active research).2 There is hardly enough exibility in the training process of most SMT sys-tems in use today: in order to incorporate new knowledge a new trainingphase must be started from scratch.3 Typically, there is not much room for interaction with the user. The hu-man translator receives the translation and there is no straightforwardmechanism to adjust the SMT system so that the same error is not re-peated anymore.19D4.2 Online Learning for CATIn this chapter we describe the integration in Portage of an online learningmodule tailored for CAT applications. As described in Chapter 2 this modulehas been designed to address the abovementioned issues, providing a exibleenvironment in which interaction with the human translator and continuoustraining is possible. In managing the interaction with the pre-existing CATtools, we pay special attention to the problem of exploiting the translationssuggested by the adaptive module only when their quality is good enough.Our learning module has been mainly written in C++ and developed underLinux. Before describing it in more detail, we mention the parts of the Portagearchitecture that are most relevant to our application.Portage operates in four main phases.1 Preprocessing of raw data into tokens, with translation suggestions forsome words or phrases being generated by rules.2 Decoding in order to produce one or more translation hypotheses, error-driven rescoring to choose the best nal hypothesis.3 Optional re-ranking: the decoder can produce a list of translation options(the N-best list). It is possible to make Portage learn how to re-rank thislist, also using additional features.4 Postprocessing to enforce appropriate case and punctuation.Preprocessing involves dierent steps, mainly implemented by Perl scripts. Inorder to create a sentence-aligned parallel corpus it is necessary to split thesource and target corpora into dierent les to allow for ecient processing.Each le is then tokenized (the words and punctuation are separated by spaces)using dierent heuristics for each language. The les are then aligned line byline so that the correspondence between source and target sentences is precise.Finally, les are lower-cased.The phrasetable is learned from the sentence-aligned corpora using a word-levelaligner coupled with some heuristics to gather entries for the phrasetable thatare as correct as possible. The language model is trained using the SRILMtoolkit from SRI (see [31]).Decoding is a central phase in statistical machine translation. Almost all moderndecoders are stack-based. This allows a good time/precision trade-o whensearching the huge space of translations of a source sentence. When the decodingprocess starts there are no hypotheses in the stack. Then, translations coveringjust a single source word are added to the stack. All successive hypotheses areVersion 3.0 March 19, 2008 Page 20 of 49D4.2 Online Learning for CATgenerated by adding a word to the set of source words so far covered by theexisting candidate translations. The rst stack is used to generate a secondstack with two-word hypotheses, and so on until we have a stack of candidatesthat fully cover the source sentence.Since we need to evaluate hypotheses even if some of the words are not translatedyet, we make use of a future score, which is an heuristic formula to estimatethe cost of translating the remaining parts of the sentence.During this search it is important to prune the data structures and to drive thesearch toward the best translation options. There is no guarantee that the besttranslation will be included in the output; however, there is a high probabilitythat it will not be discarded.For each preprocessed source sentence, N-best translations are identied usingthe search algorithm with a loglinear model. The system is open in manyaspects. Indeed, it is possible:1 to dene new features to be used inside of the decoder, or in the re-rankingphase,2 to dene new parameters, which are loaded by the decoder,3 to add language models or phrasetables.Moreover, Portage allows one to dene annotations in the corpora for describ-ing rule-based translation for: numbers, dates, and domain specic named en-tities. These rules generate entries which are inserted into all phrasetables witha specied probability. This makes it possible to overcome obvious problemswhen translating those special entities based only on their statistics. Rule-basedtranslations usually override automatically learned translations.We will call the features dened inside the decoder local features, and the fea-tures for hypothesis re-ranking global features. The local features are evaluatedfor every translation option during the beam search decoding process. A localfeature must yield a value for every segment (phrase) of the sentence.There are constraints on how this value is calculated, otherwise the decoderwould not be able to perform the beam search. The local features cannot refer-ence the whole target sentence, because this is calculated during the decodingphases, when the full translation is not completely formed. Global features havefewer constraints: it is possible to calculate the value over the whole sentence.So, it is possible to move the most complex features outside of the decoder toovercome such problems.Version 3.0 March 19, 2008 Page 21 of 49D4.2 Online Learning for CATWhen a sentence is translated incorrectly, a possible explanation is that thedecoder does not have the segments needed to generate the reference sentence;alternatively, if the segments are present in the phrasetable, they could be scoredso low by the decoder that the associated translation gets a low position in theranking of candidate translations. The rst case occurs less frequently giventhe availability of large training corpora. The second case, instead, calls for abetter assignment of scores to segments.In order to cope with this scoring problem, Portage allows for an optimizationof the decoder weights. Every internal feature used by the decoder is associatedwith a weight. This weight is used along with the feature value to compute thescore associated with a certain segment. Typically, these weights are optimizedoine, after building the phrasetable, and then not updated anymore while thesystem is being used.3.2 Integrating the online learner with PortageAs mentioned in Chapter 2, the goal of adding adaptivity to CAT is to improvethe set of tools a professional translator uses in order to boost his productivity,and the purpose of this case study is to measure the improvement provided byonline learning used jointly with SMT.Dictionaries and translation memories are often used by professional translators.These tools store text segments (that is words, phrases, or entire paragraphs)and analyse the source text presented to the translator in order to nd matchesto previously translated text. The best matches are presented to the translator,who accepts or reject the proposals. Machine translation, in the context ofthis application, is used to generate suggestions for the cases when translationmemory does not have a good match for the source sentence.The process of building and training a learning system for CAT using SMTtechnology has not been deeply investigated in the literature. It is not straight-forward to understand what are the best quality measures for this kind of appli-cation. Simply speaking, we would like to have a system able to help the humantranslator to recall previously used translations and give meaningful suggestionsusing the generalization ability of SMT techniques.To complicate things, the translator, in the setting we are considering, is makinguse of a translation memory which will suggest a previously observed sentencewhen the condence of this suggestion exceeds a given threshold. This factmeans that the SMT system is going to receive the hardest sentences to trans-Version 3.0 March 19, 2008 Page 22 of 49D4.2 Online Learning for CATlate, and should be able to suggest (at least) the parts of the sentences alreadyseen in previous translations.When a sentence, or part of it, is almost unknown to the system, it is better notto emit any suggestion than to suggest a sentence which is probably completelywrong. So, it is important to calculate a condence value for the translationbefore emitting a suggestion. To achieve the assessment of this value we mustrely on the history of the sentence. For example if a sentence was never seen,but most of its bigrams have already been processed many times and they werealways translated in the same way by the human translator, then it is highlylikely that we can achieve a good condence in the translation. On the otherhand, if the system has seen a part of the sentence just once in all our history,then we could easily be wrong because that translation could be a mistake or itcould have been found in a very dierent context when it was used to feed thelearner.To partially overcome this problem, dierent techniques to smooth the statis-tics of the language model are often used. Moreover, smoothing has been alsoproposed for the translation model [15]. In our adaptive architecture, smooth-ing can be viewed as the requirement of maintaining a correct trade-o betweengeneralization capability, which is obtained by a suitable representation of allpast sentences, and the need to adjust the system in order to x the translationof the current sentence. We will sometimes call this trade-o aggressiveness,suggesting that an aggressive learner is stronger in modifying the parameters toquickly adapt to the new data, taking the risk of overtting. This aggressive-ness trade-o is controlled using dierent averaging techniques, as mentioned inChapter 2.3.2.1 Measuring performance for CATIn such a framework it is dicult even to keep track of the performance of thelearner because there is not a clear measure of eectiveness of the translationtools beside the judgement of the human translator. In particular, the keypoints we are interested to measure in a CAT tool are the following.1. Speed. This is a key factor because in many cases there are time con-straints. Moreover, post-editing is almost always needed to guarantee agood quality level before the translation is released. We are thus interestedin giving to the user a translation proposal that minimizes the number ofwords changed or moved is minimized. It is also important to measure thespeed improvement while the translation advances and the online learningVersion 3.0 March 19, 2008 Page 23 of 49D4.2 Online Learning for CATin the MT system receives more input from the translator.2. Quality: A good translation quality means less post-editing work.3. Cost: This is relevant to a translation agency as well, but it is not some-thing we would like to measure because there are too many factors thataect it.Our learning algorithms are very general: they can work with any sentence-level machine-computable measure of translation quality. In the learning phasewe make use of BLEU score, WER (word error rate), and PER (position-independent word error rate). This is just an example of a large set of possiblemeasures. Actually we could even use a mixture of dierent measures in theattempt to nd a better correlation with the human judged quality. Since weare interested in the adaptation of the system to new data, to keep track ofthe relative improvement of the system when the learner receives more inputswe measure performance in terms of BLEU, WER, PER, and we compare it toPortage without adaptation.3.2.2 FeaturesThe Portage decoder makes use of a set of features to drive the beam search.Before the decoding process for a given source sentence starts all the possibleranges of words in the sentence are scanned, and a data structure holds theminimum cost needed to translate that segment. This is important in order tohave the possibility of estimating the cost to translate the rest of the sentencegiven a certain translation hypothesis.The PORTAGE decoder is based on a loglinear model that incorporates thefollowing features1. Phrase translation probabilities stored in one or more phrasetables; theseare to estimate the cost of using a target phrase to translate a source text.2. One or more language models trained using the SRILM toolkit.3. A distortion penalty penalizing a translation when the number of sourcewords that are skipped when generating a new target phrase from thesource is too high.4. A word penalty, to avoid that long translations could be considered goodjust because of the addition of more scores.Version 3.0 March 19, 2008 Page 24 of 49D4.2 Online Learning for CATThe choice of features, as in every learning application, is very important be-cause it is the way the learning module sees the problem. In our case wemean to add a huge number of features, from thousands to millions. Portagelets the user add new features to drive the search according to improved modelsor data. But, even if Portage is open, it is infeasible to add so many featuresusing Portage standard data structures. We decided to make use of a specialC++ class SparseVector, representing a sparse vector of features. This datastructure is specialised to allow ecient storage, management, and computa-tion over sparse vectors, where sparse means that most of the values stored areactually zeroes. Inside the SparseVector class we introduced also methods forreporting and debugging to check the integrity and correctness of the system ina core component. Since we were not allowed to introduce such a huge numberof features inside the Portage decoder, we used a trick: all the features are cal-culated and their contribution is stored in a single Portage feature. So just asingle new feature is added inside the decoder.We also have to keep a dierent set of weights: during the decoding, whenPortage asks for the score of our new feature, we return the sparse vector dotproduct of the feature evaluations with the sparse weight vector. As describedin Chapter 2, the original set of loglinear Portage weights are optimized on thetraining set and then remain unchanged during the online training phase.The phrasetable is controlled by the learning algorithm using a set of featuresthat can be activated for just a single entry in the table or for a group of entries.The original decoder features used by Portage, on the other hand, are commonfor all the entries of the table, as you can see in Figure 3.1, where they crossvertically all the entries. This is an important dierence: since we are dealingwith online adaptation to user decisions and preferences, we judge that loglinearfeatures are not discriminant enough for our purposes.The new features for the ne tuning of the decoder are:1. Target words2. Target phrases3. Single phrasetable entries4. Single phrasetable entries + target words.After introducing this huge set of features, we still want the decoder to behavethe same way as before our intervention if there are no corrections for the currentcandidate sentence. At the same time we want the learning process to performVersion 3.0 March 19, 2008 Page 25 of 49D4.2 Online Learning for CATLOGLINEARWEIGHTSPHRASETABLEWEIGHTSPHRASETABLEFigure 3.1: Relationship between old and new features: the value of both sets offeatures is computed for each phrasetable entry. However, the original Portageloglinear features are combined using weights that are common to all phrasetableentries. The new features, whose values is computed using the loglinear score,are instead combined using a specic weight for each phrasetable entry.a smooth transition towards the new acquired translation examples. This isachieved using the technique described in Section 2.3Since we deal with a large number of features, and the decoder needs to eval-uate the value of the active features thousands of times, it is obvious that theperformance in term of elapsed time could be really disappointing. To mitigatethis problem we cache the value of the dot product of the features with weights,so that it is necessary to calculate them just the rst time, or in the case whenan associated weight is updated by the algorithm.We now move on to describe what happens when a new sentence arrives. Thedata structures involved and their relationships are depicted in Figure 3.2.1. The decoder is invoked using the initial weights. The decoder, during thenormal process of pruning and ranking hypothesis, makes several calls toget the values of the new features.2. When the decoder is nished we get the result: a list of N-best translationVersion 3.0 March 19, 2008 Page 26 of 49D4.2 Online Learning for CATLEARNERONLINEDECODERPHRASETABLE WEIGHTSWEIGHTSLINEARLOGNBEST LISTPHRASETABLEFigure 3.2: The decoder builds the N-best list using the phrasetable by com-bining the loglinear score (based on the original Portage loglinear weights) withthe new score computed using the phrasetable features and their associatedweights. Once the reference sentence is known, the online learner updates thephrasetable weights based on the ranking of the N-best list.options (with N user denable, typically set around 200 1000).3. Optionally, we run a static re-ranker, meaning that we change the relativescores of hypotheses in the N-best list. The importance of this step isrelated to the possibility of applying global features which are forbiddeninside the decoder because of the beam search algorithm requirements.4. We send the best translation option to the translator.5. The translation option is stored with the feature vector calculated usingour features.6. If we receive a correction because the decoder did not provide a perfecttranslation we call an update function from our suite of algorithms.7. If some of the words or segments of the sentence are unknown to thedecoder, we run an on the y alignment and store the new fragments (thisstep is still under development).3.3 CAT tool architectureThe concrete CAT tool we chose for integration in our adaptive system is Tra-dos from SDL International. Trados is a well-known product for translationmemory management. It improves the speed and reliability of the translationVersion 3.0 March 19, 2008 Page 27 of 49D4.2 Online Learning for CATby fuzzy matching previously translated sentences. Thanks to this approach,our SMT tool is integrated seamlessly into an environment that is familiar tomany professional translators.We make use of Trados APIs to interact with external tools. Trados connects toan external web server where our system is running, thus providing centralizedSMT translation facilities for all the translators involved in the project. Theserver sends the received translation requests to our modied version of Portagewhere the translation takes place. All the requests and timestamps are loggedto a le. We use a standard webserver with CGI interface to a script. Thisscript in turn interfaces the SMT system. The sentences are written as httprequests using post methods. In the future, communications will be realizedthrough standard webservices interfaces.It is very important to make the work of the human user as easy and comfort-able as possible, and minimize the dierences from known tools and translationprocedures. It is considered pointless, and even negative, to present a com-pletely wrong translation option. So we plan to add modules to lter out badtranslations based on condence measures of the translation.The workow of the CAT interaction is as follows (see Figure 3.3).1. A sentence is presented to the translator. Trados looks for a (fuzzy) matchin the translation memory and outputs it if the match score between thetwo sentences goes above a certain threshold. Otherwise, the sentence issent by our module inside Trados to the web server.2. The web server sends the sentence to the modied version of Portage,which is waiting with all the necessary data preloaded.3. The SMT system translates the sentence and then uses the features fromthe new data acquired to update the internal weights.4. When translation and learning by the system is nished the results aresent back to the Trados API and presented to the user, allowing a simpleselect-and-click action to accept a candidate translation.5. The user can obviously modify the proposal or reject it and insert manuallythe text.6. A feedback is sent back to the web server to feed the SMT system withcorrections.Version 3.0 March 19, 2008 Page 28 of 49D4.2 Online Learning for CATCLIENT SUB NETWORK SERVER SUB NETWORKPORTAGEONLINE LEARNERWORD PROCESSORTRADOSSMT APIHUMAN TRANSLATORWEB SERVERCGI SCRIPTFIREWALL OR PROXYFigure 3.3: Software modules and network connections for the CAT evaluationtask.3.4 Experimental ResultsWe validated our approach via several experiments using mainly French-Englishand Spanish-English language pairs. The reason for this choice is to be able toeasily compare our results with established baselines, and to prepare for theCAT evaluation task which will be held with professional Spanish translatorsfrom English to Spanish. The training data is from the freely available Europarlcorpus and was obtained and prepared as described in [20].After removing HTML tags, aligning the sentences and tokenizing, the Span-ish corpus resulted in 1,476,106 sentences and 41,408,300 words. The Frenchcorpus had 1,487,459 sentences and 44,688,872 words. Finally, the English had1,461,429 sentences and 39,618,240 words. To limit the possibility of errorsduring the preprocessing phases, particularly alignment errors, we decided toVersion 3.0 March 19, 2008 Page 29 of 49D4.2 Online Learning for CAT-5 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 0 100 200 300 400 500 600 700 800 900 1000BLEU DIFFSENTENCESTarget wordsTarget phrasesphrasetable entrysingle entry + target wordsFigure 3.4: A comparison of our four feature sets: (1) One feature for everytarget word. (2) One feature for every target phrase. (3) One feature for everyphrasetable entry. (4) A combination of 3 and 1. In this picture, and in thefollowing ones, the cumulative bleu dierences exhibit long at areas. This isbecause the feature sets are large, and the weight update caused by a sentencehas no impact on subsequent sentences whenever these contain dierent words.remove all sentences shorter than three words or longer than 25 words. It isimportant to point out that in SMT wrong phrasetable entries are not excep-tional events. It is the statistics that helps (with high probability) to lter outthe incorrect phrasetable entries. However, since a cleaner phrasetable speedsup learning, we took care of providing as clean as possible inputs to the system.We split the corpora into a training set, a development set (10,000 sentences)and a test set (5,000 sentences). The development set was used to perform anoptimization of the loglinear weights of the basic features of the decoder usinga specic module of Portage named Cow. This is only a generic optimizationto maximize the BLEU score over a dataset. In our case it was mainly usedto adapt the decoder to the corpus in use. We did not use it during the onlinelearning phase.The training set corpus was further split in blocks of 300 sentences each. Eachblock was encoded in a format suitable for being submitted to the preprocessingstages of Portage. This splitting allows an ecient and parallel computation ofVersion 3.0 March 19, 2008 Page 30 of 49D4.2 Online Learning for CAT-6-5-4-3-2-1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 100 200 300 400 500 600 700 800 900 1000BLEU DIFFSENTENCESITER1ITER2Figure 3.5: Results of two iterations with Passive-Aggressive type II algorithmusing the target words feature. Cumulative dierences from baseline Portageperformance with no online adaptationthe phrasetable. The outcome of this phase, described earlier in detail, is thetranslation table and the language model.We ran several experiments with the algorithms described in Section 2.4. How-ever, we did not nd that any specic algorithm had a signicant advantageover the others. Therefore, we report experimental results only for the simplestof these algorithms; i.e., Passive-Aggressive type II.In all experiments, we plot the performance measured in cumulative BLEUDIFFpoints. This is the dierence in sentence-level bleu points between the adaptiveonline system and the Portage baseline (no adaptation) summed over a sequenceof translations.In Figure 3.4 we measure the cumulative BLEUDIFF for Passive-Aggressive typeII using our four feature sets. The target words turn out to be worst choice.However, Figure 3.5 shows that even these weak features carry informationuseful for the translation. Indeed, they pick up some signal during a seconditeration over the same dataset.Since our online learner works only when the match score of the translationmemory is low, we implemented a translation memory simulator based onVersion 3.0 March 19, 2008 Page 31 of 49D4.2 Online Learning for CAT 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 0 100 200 300 400 500 600 700 800 900 1000BLEU DIFFSENTENCESTM 00 otherwise.Remark 2. The reason why we expect the 1-norm might be a good choice forthe learning problem arises from knowledge about the distribution of words in anatural language text. This distribution is very irregular and can be describedby the Zipf s law (see, e.g., [24]). This kind of distributions has no expectedvalue, and in strict sense they might not be distributions because their partitionfunctions can diverge when the sample size increases. As a consequence, we canhardly give a valid meaning to the probabilities derived. Weights based on theseprobabilities might thus be able to express relationships relative to a xed sample,yet fail to generalize them to other samples. In the L1 space, instead, we onlyexploit the coexistence of phrases, which remains valid independently of how thesample has been chosen.The phrase prediction process is described in Figure 4.2. Because a predictionis a set of candidate phrases that should appear in the target sentence, thescores given by the corresponding components of the vector W(x) have to betransformed into a form that can be fed to the sentence decoder. To achievethis, a bag-of-candidates structure is built upon the scores. This structure isused to assign a set of phrases to each word of the source sentence using thehighest scores received from the phrase predictor.Source sentenceHigh-score phrasesLow-score phrasesWordsws1 ws2 . . . wsN1 wsN Bagsp1,1 p2,1 . . . pN1,1 pN,1... ... ... ... ...p1,K p2,K . . . pN1,K pN,Kp1,K+1 p2,K+1 . . . pN1,K+1 pN,K+1... ... ... ... ...Version 3.0 March 19, 2008 Page 40 of 49D4.2 Online Learning for CATW(x)

(x)output phrases in a neighbourhoodscores . . . p {1 p {2 p {3 . . .0 . . .3 1 1 1 . . .0 . . .2 1 1 . . .0 . . .2 1 1 . . .0 . . .... ... ... ... ... ... ... ...

column(row)vectorsare indicatorsof candidates WFigure 4.2: The phrase prediction process: high-scoring target phrases are fedto the decoder.Here K is the maximum number of accepted candidates. If ties occur in scores,then the F1 measure is used to break them.4.4.2 DecodingThe task of the decoder is to merge the phrases emitted by the phrase translatorinto a grammatical and uent sentence. Before describing the sentence decoderwe need to explain some basic elements of the procedure.Merging phrases, drift, contradictionsA sentences is constructed by merging candidate phrases. The merging algo-rithm is based on the following notions. Drift: this is the tail of the next phrase that is not matched to the currentsubsentence. This is used to force the decoder to add at least one newword to the subsentence when a new candidate phrase is appended.sub-sentence . . . h h h hnext phrase t t t t....DriftVersion 3.0 March 19, 2008 Page 41 of 49D4.2 Online Learning for CAT Distance: the number of non-matched words between the current sub-sentence and the next phrase.Fitted part Driftsub-sentence . . . a b c d # inequalities = = = =next phrase b q d eContradictionDistance is measured by number of inequalitieslength of the next phrase.Decoding and reordering by dynamic programming with tabu searchThe target sentence is constructed using a dynamic programming technique,see [4, 5, 6, 7]. In order to achieve a grammatically smooth sentence, the conceptof Tabu-search based reordering [17] is added to the dynamic programmingschema.We use B = (B1, . . . , BN) to denote the ordered set of bags provided for eachsource word by the phrase predictor. Here N is the number of words in thesource sentence for which we have a bag of candidate phrases. The order of Breects the order of the words in the source sentence.We consider the following dynamic programming approximation of the wordorder in the target sentence: The state space is a set of labels of the phrases dened on the concate-nation of the bags, in this way some phrases may appear in more thanone bag. In order to distinguish between the states of dierent bags la-belling the same phrases, the concatenation of the bag relative indices ofthe phrases is used to identify them:o =

(p1,1, . . . , p1,n1). .. .B1, . . . , (pN,1, . . . , pN,nN). .. .BN

where the state pi,j as a label denotes phrase j in bag i. The sub-sentence built in the rst t steps, with pi,j being the last phrasemerged into, is denoted by s(pi,j)t.Version 3.0 March 19, 2008 Page 42 of 49D4.2 Online Learning for CAT The distance between a sub-sentence s(pi,j)t and a phrase pk,l in step t isdenoted by d

s(pi,j)t, pk,l

. The total length of a sub-sentence, denoted by

s(pi,j)t

, is measured bysumming up the distances between the included phrases and their prexes,assuming that the empty sub-sentence has 0 length. The tabu list (pi,j)t is a set of phrases used to build s(pi,j)t. The merging operator acting on a sub-sentence and a phrase is denotedby .The dynamic programming algorithm looks for the shortest sentence that con-tains the smallest number of dierent words in the overlapping parts of thesub-sentences and phrases merged.The algorithm works in a forward fashion using the following update rule. Letpr,s be the state at the beginning of step t +1. Choose the state correspondingto the optimum solution to the shortest sub-sentence problem,pk,l = argminpi,j

s(pi,j)t

+ d

s(pi,j)t, pr,s

subject to pr,s / (pi,j)t .Then merge the phrase pr,s into the sub-sentence belonging to the optimumstate pk,l and append it to the corresponding tabu-lists(pr,s)t+1 = s(pk,l)t pr,s(pr,s)t+1 = (pk,l)t pr,s .If t = 0 then the subsentence is initialised to the empty string and the corre-sponding tabu list is the empty set.4.5 ExperimentIn this nal section we describe the results of an experiment in which we testedour translation system on the French and English versions of the Xerox printersmanual provided by XRCE. The direction of the translation was from Frenchto English. The baseline translator, which we used against our system, was theGIZA++ [27] aligner with the Pharaoh [21] decoder. Three random subsetsof sentences were used as test sets. Two of them only included sentences withVersion 3.0 March 19, 2008 Page 43 of 49D4.2 Online Learning for CATsentence ngrams recall Bleulength Method 1 2 3 4 score 30 GIZA+PHARAOH 65.6 41.9 29.6 21.5 36.2TR candidate phrases 88.3 60.5 40.6 28.9 50.1TR drift, no contra 73.9 45.7 30.7 22.0 38.9TR drift, contra 66.1 41.5 29.2 22.0 36.4 30 GIZA+PHARAOH 70.2 47.5 35.6 27.4 42.0TR candidate phrases 90.6 64.1 43.9 30.8 52.9TR drift, no contra 75.6 49.1 33.9 24.8 42.1TR drift, contra 70.7 45.2 31.7 23.5 39.3no limit GIZA+PHARAOH 69.0 43.8 29.8 20.9 36.7TR candidate phrases 89.6 62.2 42.8 31.1 52.2TR drift, no contra 74.5 44.1 28.7 19.8 37.0TR drift, contra 68.9 41.3 27.6 19.7 35.3Figure 4.3: Result produced on the Xerox printer manual dataset. Sentencesoccurring both in test and training sets are deleted from test set.length at most 30, whereas one test set contained sentences with no lengthlimitation. The quality of the translation is measured bleu points. The bleuscore was computed by the algorithm in the Moses package.The explanation of the items in Figure 4.3 is the following.TR candidate phrases. This is the concatenation of the phrases provided bythe phrase predictor. It shows how much information is contained in thecandidate phrases and can be interpreted as a measure of n-gram recallamong the candidates; obviously, the corresponding precision is inaccu-rate.TR drift, no contra. The sentences are decoded under the assumption thatphrases are used to extend the sentence if they perfectly t the prex sub-sentence, it overestimates the sentence length by approximately 40-50%.TR drift, contra. In case of contradiction between the merged phrases andthe prex sub-sentence the word with higher score provided by the phraseprediction is inserted.Version 3.0 March 19, 2008 Page 44 of 49D4.2 Online Learning for CATFrench source English reference TranslationCong : contient leschiers de congura-tion des services denumerisation reseau.Cong contains congu-ration les for the Net-work Scanning Servicessetup.contains the system con-guration les the net-work scanning servicesContexte NDS (le casecheant)NDS Context (if applica-ble)nds tree if available selectUne fois que le compteutilisateur du DocumentCentre a ete cree, vouspouvez installer les ser-vices de numerisation surun poste de travail.Once you have your Doc-ument Centre logon ac-count, you are ready toinstall Network ScanningServices on a worksta-tion.you have your documentcentre user account pass-word is if a documentcentre has been createdyou can install networkscanning services on aworkstationEtape 4 : Tester la con-guration du DocumentCentrePart 4: Test the Docu-ment Centre Congura-tionthe part 4 test the doc-ument centre congura-tion3 Cliquez sur Terminerpour enregistrer lesmodications et quitterlassistant.3 Click Finish to saveyour changes and closethe wizard.3 click nish to save thechanges and exit the wiz-ard andLe menu deroulant Ges-tionnaire dimpressioncomprend les optionssuivantesThe Printer ManagerOptions menu containsthe following optionspull down menu theprinter manager windowmenu contains the fol-lowing optionsTable 4.1: Translation examplesVersion 3.0 March 19, 2008 Page 45 of 49D4.2 Online Learning for CAT4.5.1 ExamplesTable 4.1 consists of some sentences translated from French into English takenfrom the Xerox Guide. The last row, Le menu, shows a side eect of therare words. These words cause very occasional phrases which may repeat somewords, here the menu, which is inserted twice. Note that this guide con-tains several incomplete sentences and words used in the technical jargon, e.g.,Cong.4.6 Future workThe aim of this rst implementation of our translation system was to collectexperimental results about its main building blocks: the 1-norm based phrasepredictor and the decoder. The decoder, built upon the combination of dynamicprogramming and tabu search, provides good results even when the qualityof the inputs is not very accurate. Namely, the phrasetable contains severalirrelevant phrase pairs, but it also includes some good candidates. The weakpoint is the translation of the rare words and phrases, when good candidatesare generally missing.A further task that we need to address concerns the size of the database of co-occurrences. The long consecutive phrases, consisting of three or more words,are important in building smooth translations, but most of them are occasionalco-occurrences of otherwise not related words, causing only noise in the trans-lation procedure. To this end we are developing a phrase generator which canconsider non-consecutive word sequences too.We are going to incorporate a language model into the phrase selection in whichwords are joint into a phrase if they can predict the occurrences of each other.In order to implement this model we plan to use a similar model that has beenemployed in the training phase of the rst version of our translation system.Version 3.0 March 19, 2008 Page 46 of 49Bibliography[1] Y. Amit, S. Shalev-Shwartz, and Y. Singer. Online classication for com-plex problems using simultaneous projections. Advances in Neural Infor-mation Processing Systems, 18, 2006.[2] A. Arun and P. Koehn. Online learning methods for discriminative trainingof phrase based statistical machine translation. In MT Summit XI, 2007.[3] Various authors. Translation memory. Wikipedia, 2008. URL:en.wikipedia.org/wiki/Translation memory.[4] R. Bellman. Dynamic Programming. Princeton University Press, 1957.Dover paperback edition (2003).[5] D.P. Berstsekas. Neuro-Dynamic Programming. Athena Scientic, 1996.[6] D.P. Berstsekas. Dynamic Programming and Optimal Control, volume I.Athena Scientic, 3rd edition, 2005.[7] D.P. Berstsekas. Dynamic Programming and Optimal Control, volume II.Athena Scientic, 3rd edition, 2007.[8] J. Blatz, E. Fitzgerald, G. Foster, S. Gandrabur, C. Goutte, A. Kulesza,A. Sanchis, and N. Ueng. Condence estimation for machine transla-tion. Proceedings of the 20th international conference on ComputationalLinguistics, 2004.[9] N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cam-bridge University Press, 2006.[10] C. Cortes, M. Mohri, and J. Weston. A general regression technique forlearning transductions. Proceedings of the 22nd International Conferenceon Machine Learning, pages 153160, 2005.47D4.2 Online Learning for CAT[11] K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, and Y. Singer. OnlinePassive-Aggressive Algorithms. The Journal of Machine Learning Research,7:551585, 2006.[12] J. Esteban, J. Lorenzo, S. Antonio, and G. Lapalme. TransType2An Inno-vative Computer-Assisted Translation System. The Companion Volume tothe Proc. of the 42nd Annual Meeting of the Association for ComputationalLinguistics, 2004.[13] G. Foster, P. Isabelle, and P. Plamondon. Target-Text Mediated InteractiveMachine Translation. Machine Translation, 12(1):175194, 1997.[14] G. Foster, P. Isabelle, and P. Plamondon. Target-text mediated interactivemachine translation. Machine Translation, 12(12):175194, 1997.[15] G. Foster, R. Kuhn, and H. Johnson. Phrasetable Smoothing for Statis-tical Machine Translation. Conference on Empirical Methods in NaturalLanguage Processing (EMNLP), 2006.[16] Y. Freund and R.E. Schapire. Large Margin Classication Using the Per-ceptron Algorithm. Machine Learning, 37(3):277296, 1999.[17] F. Glover and M. Laguna. Tabu Search. Kluwer, 1997.[18] E. Hazan, A. Agarwal, and S. Kale. Logarithmic regret algorithms foronline convex optimization. Machine Learning, 69(2):169192, 2007.[19] M. Kay. The MIND system. Natural Language Processing, 8, 1973.[20] P. Koehn. Europarl: A parallel corpus for statistical machine translation.MT Summit X, pages 1216, 2005.[21] P Kohn. Pharaoh: a beam search decoder for phrase-based statisticalmachine translation models. In AMTA, 2004.[22] P. Langlais, G. Foster, and G. Lapalme. TransType: a computer-aidedtranslation typing system. In Proceedings of Embedded Machine TranslationSystems Workshop (in conjunction with NAACL/ANLP2000), pages 4651, 2000.[23] P. Langlais and G. Lapalme. Trans Type: Development-Evaluation Cy-cles to Boost Translators Productivity. Machine Translation, 17(2):7798,2002.[24] W. Li. Random texts exhibit Zipfs-law-like word frequency distribution.IEEE Transactions on Information Theory, 38(6):18421845, 1992.Version 3.0 March 19, 2008 Page 48 of 49D4.2 Online Learning for CAT[25] P. Liang, A. Bouchard-Cote, D. Klein, and B. Taskar. An end-to-enddiscriminative approach to machine translation. Proceedings of the 21st In-ternational Conference on Computational Linguistics and the 44th annualmeeting of the ACL, pages 761768, 2006.[26] D.J.C. Mackay. Information Theory, Inference, and Learning Algorithms.Cambridge University Press, 2003.[27] F.J. Och and H. Ney. A systematic comparison of various statistical align-ment models. Computational Linguistics, 29(1):1951, 2003.[28] F.J. Och. Minimum error rate training in statistical machine translation.Proc. of the 41th Annual Meeting of the Association for ComputationalLinguistics (ACL), pages 160167, 2003.[29] F.J. Och, D. Gildea, S. Khudanpur, A. Sarkar, K. Yamada, A. Fraser,S. Kumar, L. Shen, D. Smith, K. Eng, et al. A smorgasbord of features forstatistical machine translation. Proceedings of HLT-NAACL 2004, pages161168, 2004.[30] F. Sadat, H. Johnson, A. Agbago, G. Foster, R. Kuhn, J. Martin, andA. Tikuisis. Portage: A Phrase-based Machine Translation System. Pro-ceedings of the ACL Workshop on Building and Using Parallel Texts, pages129132, 2005.[31] A. Stolcke. SRILM an Extensible Language Modeling Toolkit. SeventhInternational Conference on Spoken Language Processing, 2002.[32] B. Taskar, C. Guestrin, and D. Koller. Max-margin Markov networks.Advances in Neural Information Processing Systems, 16:51, 2004.[33] I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun. Large MarginMethods for Structured and Interdependent Output Variables. The Journalof Machine Learning Research, 6:14531484, 2005.Version 3.0 March 19, 2008 Page 49 of 49