A Basic Language Technology Toolkit for Quechua

Una coleccion de herramientas basicas para el procesamiento automaticodel Quechua

Annette RiosInstitute of Computational Linguistics

University of Zurichrios@cl.uzh.ch

Resumen: Tesis escrita por Annette Rios en la Universidad de Zurich bajo la di-reccion de Prof. Dr. Martin Volk. La tesis fue defendida el 21 de septiembre de2015 en la Universidad de Zurich ante el tribunal formado por Prof. Dr. MartinVolk (Universidad de Zurich, Departamento de Linguıstica Computacional), Prof.Dr. Balthasar Bickel (Universidad de Zurich, Departamento de Linguıstica Compar-ativa) y Dr. Paul Heggarty (Instituto Max Planck para la Antropologıa Evolutiva).La tesis obtuvo la calificacion ‘Summa cum Laude’.Palabras clave: Traduccion automatica, analisis morfologico, programa de con-cordancia, transductor de estados finitos, traduccion automatica hıbrida, treebank,gramatica de dependencias

Abstract: Thesis written by Annette Rios under the supervision of Prof. Dr. Mar-tin Volk at the University of Zurich. The thesis defense was held at the Universityof Zurich on September 21, 2015 and was awarded ‘Summa Cum Laude’. The mem-bers of the committee were Prof. Dr. Martin Volk (University of Zurich, Instituteof Computational Linguistics), Prof. Dr. Balthasar Bickel (University of Zurich,Department of Comparative Linguistics) and Dr. Paul Heggarty (Max Planck In-stitute for Evolutionary Anthropology).Keywords: Machine translation, morphological analysis, concordancer, finite state,hybrid MT, treebank, dependency grammar

1 Introduction

In this thesis, we describe the developmentof several natural language processing toolsand resources for the Andean language CuzcoQuechua as part of the SQUOIA project atthe University of Zurich.

Quechua is a group of closely relatedlanguages, spoken by 8-10 million people inPeru, Bolivia, Ecuador, Southern Colombiaand the North-Western parts of Argentina.Although Quechua is often referred toas a ‘language’ and its local varieties as‘dialects’, Quechua represents a languagefamily, comparable in depth to the Romanceor Slavic languages (Adelaar and Muysken,2004).Mutual intelligibility, especially be-tween speakers of distant dialects, is notalways given. The applications describedin this thesis were developed for CuzcoQuechua, but some applications, such as themorphology tools and the spell checker, can

also be used for closely related varieties.

The main focus of this work lies on theimplementation of a machine translation sys-tem for the language pair Spanish-CuzcoQuechua. Since the target language Quechuais not only a non-mainstream language inthe field of computational linguistics, butalso typologically quite different from thesource language Spanish, several rather un-usual problems became evident, and we hadto find solutions in order to deal with them.Therefore, the first part of this thesis presentsmonolingual tools and resources that are notdirectly related to machine translation, butare nevertheless indispensable.

All resources and tools are freely availablefrom the project’s website.1

Apart from the scientific interest in devel-oping tools and applications for a language

1https://github.com/ariosquoia/squoia

Procesamiento de Lenguaje Natural, Revista nº 56, marzo de 2016, pp 91-94 recibido 29-10-2015 revisado 17-12-2015 aceptado 28-01-2016

ISSN 1135-5948 © 2016 Sociedad Española para el Procesamiento del Lenguaje Natural

that is typologically distant from the main-stream languages in computational linguis-tics, we hope that the various resources pre-sented in this thesis will be useful not onlyfor language learners and linguists, but alsoto Quechua speakers who want to use moderntechnology in their native language.

2 Structure of Thesis

The thesis is structured as follows:In the first chapter, we set the broader

context for the development of NLP tools fora low-resource language and we give a shortoverview on the characteristics and the dis-tribution of the Quechua languages.

Chapter 2 explains the morphologicalstructures in Quechua word formation andhow we deal with them in technological ap-plications, this includes morphological anal-ysis, disambiguation, automatic text normal-ization and spell checking.

Chapter 3 summarizes the treebankingprocess and how existing tools were adaptedfor the syntactic annotation of Quechua textswith dependency trees.

Chapter 4 describes how Bilingwis, an on-line service for searching translations in par-allel, word-aligned texts, was adapted to thelanguage pair Spanish-Quechua.

Chapter 5 describes the implementation ofthe hybrid machine translation system for thelanguage pair Spanish-Quechua, with a spe-cial focus on resolving morphological ambi-guities.

The main part of this work was the de-velopment of the translation system Spanish-Cuzco Quechua. However, the special situa-tion of the target language Quechua as a non-mainstream language in computational lin-guistics and as a low-prestige language in so-ciety resulted in many language specific prob-lems that had to be solved along the way.

For instance, the wide range of differentorthographies used in written Quechua is aproblem for any statistical text processing,such as the training of a language model torank different translation options. Thus, inorder to get a statistical language model, wehad to first find a way to normalize Quechuatexts automatically.

Furthermore, the resulting normalizationpipeline can be adapted for spell checkingwith little effort. For this reason, an entirechapter of this thesis is dedicated to the treat-ment of Quechua morphology: although not

directly related to machine translation, au-tomatic processing of Quechua morphologyprovides the necessary resources for impor-tant parts of the translation system.

Moreover, the project involved the cre-ation of a parallel treebank in all 3 languages.While the Spanish and German parts of thetreebank were finished within the first yearof the project, building the Quechua tree-bank took considerably longer: before the ac-tual annotation process started, the texts hadto be translated into Quechua. Additionally,we had to design an annotation scheme fromscratch and set up pre-processing and anno-tation tools.

A by-product of the resulting parallel cor-pus is the Spanish-Quechua version of Bil-ingwis, a web tool that allows to search fortranslations in context in word-aligned texts.

Generally speaking, the monolingual toolsand resources described in the first part ofthis thesis are necessary to build the multi-lingual applications of the second part.

3 Contributions

The main contributions of this thesis are asfollows:

• We built a hybrid machine translationsystem that can translate Spanish textinto Cuzco Quechua. The core systemis a classical rule-based transfer engine,however, several statistical modules areincluded for tasks that cannot be re-solved reliably with rules.

• We have created an extensive finitestate morphological analyzer for South-ern Quechua that achieves high cover-age. The analyzer consists of a set cas-caded finite state transducers, where thelast transducer is a guesser that can an-alyze words with unknown roots. Fur-thermore, we included the Spanish lem-mas from the FreeLing library2 into theanalyzer in order to recognize the nu-merous Spanish loan words in Quechuatexts (words that consist of a Spanishroot with Quechua suffixes).

• We implemented a text normaliza-tion pipeline that automatically rewritesQuechua texts in different orthographiesor dialects to the official Peruvian stan-dard orthography. Additionally, we cre-

2http://nlp.lsi.upc.edu/freeling/

Annette Rios

92

ated a slightly adapted version thatcan be used as spell checker back-end,in combination with a plug-in for theopen-source productivity suite LibreOf-fice/OpenOffice.3

• We built a Quechua dependency tree-bank of about 2000 annotated sentences,that provided not only training data forsome of the translation modules, butalso served as a source of verification,since it allows to observe the distribu-tion of certain syntactic and morpholog-ical structures. Furthermore, we traineda statistical parser on the treebank andthus have now a complete pipeline tomorphologically analyze, disambiguateand then parse Quechua texts.

4 Conclusions and Outlook

We have created tools and applications for alanguage with very limited resources: Whileprinted Quechua dictionaries and grammarsexist, some of them quite outdated, digitalresources are scarce. Furthermore, the lackof standardization in written Quechua textscombined with the rich morphology hampersany statistical approach. We have imple-mented a pipeline to automatically analyzeand normalize Quechua word forms, whichlays the foundation for any further process-ing (Rios and Castro Mamani, 2014).

Due to the rich morphology, we decided touse morphemes as basic units instead of com-plete word forms in several of our resources:For instance, the dependency treebank isbuilt on morphemes since many of the typical’function words’ in European languages cor-respond to Quechua suffixes, e.g. we considercase markers as equivalent to prepositions inlanguages such as English (e.g. Quechua in-strumental case -wan corresponds to English‘by, with’). In accordance with the StanfordDependency scheme (de Marneffe and Man-ning, 2008) we treat case suffixes as the headof the noun they modify (Rios and Gohring,2012).

Furthermore, the tools for the morpholog-ical analysis and normalization are relevantas well for machine translation: The statisti-cal language model that we use in our hybrid

3The plug-in was implemented by Richard Cas-tro from the Universidad Nacional de San Anto-nio Abad in Cuzco and is available from: https://github.com/hinantin/LibreOfficePlugin.

translation system was trained on normalizedmorphemes instead of word forms, in order tomitigate data sparseness.

Apart from the rich morphology, Quechuahas several characteristics that have rarely(or not at all) been dealt with in machinetranslation. One of the most important issuesconcerns verb forms in subordinated clauses:while Spanish has mostly finite verbs in sub-ordinated clauses that are marked for tense,aspect, modality and person, Quechua oftenhas nominal forms that vary according tothe clause type. Furthermore, Quechua usesswitch-reference as a device of clause linkage,while in Spanish, co-reference of subjects isunmarked and pronominal subjects are usu-ally omitted (‘pro-drop’). This leads to am-biguities when Spanish text is translated intoQuechua.

Another special case are relative clauses:the form of the nominalized verb in theQuechua relative clause depends on whetherthe head noun is the semantic agent of the rel-ative clause. In Spanish, on the other hand,relative clauses can be highly ambiguous, asin certain cases relativization on subjects andobjects is not formally distinguished, but in-stead requires semantic knowledge to under-stand (Rios and Gohring, 2016). Considerthese examples:

(1) non-agentive:

elthe

panbread

queREL

lathe

mujerwoman

comioate

’the bread that the woman ate’

(2) agentive:

lathe

mujerwoman

queREL

comioate

elthe

panbread

’the woman who ate the bread’

Furthermore, Spanish can express possessionin a relative clause with cuyo - ‘whose’, whilethere is no such option in Quechua. Thetranslation system can currently not handlethis case, as it would require a complete re-structuring of the sentence.

Another difficult case are translations thatinvolve the first person plural: Spanish hasonly one form, but Quechua distinguishes be-tween an inclusive (‘we and you’) and an ex-clusive (‘we, but not you’) form. Unless theSpanish source explicitly mentions if the ‘you’is included or not, we cannot know whichform to use in Quechua and thus generate

A Basic Language Technology Toolkit for Quechua

93

both. The user will have to choose whichform is appropriate.

Furthermore, Quechua conveys informa-tion structure in discourse not only throughword order, but also through morphologicalmarkings on in situ elements, while in Span-ish, information structure is mostly expressedthrough non-textual features, such as into-nation and stress. We have experimentedwith machine learning to insert discourse-relevant morphology into the Quechua trans-lation, but the results are not good enoughto be used reliably for machine translation.Apart from a few cases that allow a rule-based insertion, we do not include the suffixesthat mark topic and focus in the Quechuatranslation by default, but the module canbe activated through an optional parameterat runtime.

However, the most challenging issue re-garding different grammatical categories forthe translation system is evidentiality:4 whileSpanish, like every language, has means toexpress the source of knowledge for an utter-ance, evidentiality is not a grammatical cat-egory in this language. Therefore, explicitmention of the data source is optional andusually absent. In Quechua, on the otherhand, evidentiality needs to be expressed forevery statement. Unmarked sentences arepossible in discourse, but they are usuallyunderstood as the speaker having direct evi-dence, unless the context clearly indicates in-direct evidentiality (Faller, 2002). Since ev-identiality encodes a relation of the speaker(or writer) to his proposition, and thus re-quires knowledge about the speaker and hisexperience in the world, this information can-not be inferred from the Spanish source text.Since we cannot automatically infer the cor-rect evidentiality, the translation system hasa switch that allows the user to set eviden-tiality for the translation of a document.

5 Acknowledgements

This research was funded by the SwissNational Science Foundation under grants100015 132219 and 100015 149841.

4Evidentiality is the indication of the source ofknowledge for a given utterance. Cuzco Quechua dis-tinguishes three evidential categories: direct (speakerhas witnessed/experienced what he describes), indi-rect (speaker heard from someone else) and conjec-ture (speaker makes an assumption). In fact, this isa highly simplified description, for a more elaborateanalysis of Quechua evidentiality see (Faller, 2002).

References

Adelaar, W. F. H. and P. Muysken. 2004.The Languages of the Andes. CambridgeLanguage Surveys. Cambridge UniversityPress, Cambridge.

de Marneffe, M.-C. and C. D. Manning. 2008.Stanford Dependencies manual. Technicalreport.

Faller, M. 2002. Semantics and Pragmaticsof Evidentials in Cuzco Quechua. Ph.D.thesis, Stanford University.

Rios, A. and R. Castro Mamani. 2014. Mor-phological Disambiguation and Text Nor-malization for Southern Quechua Vari-eties. In Proceedings of the First Work-shop on Applying NLP Tools to SimilarLanguages, Varieties and Dialects (Var-Dial), pages 39–47, Dublin, Ireland, Au-gust. Association for Computational Lin-guistics.

Rios, A. and A. Gohring. 2012. A tree isa Baum is an arbol is a sach’a: Creat-ing a trilingual treebank. In N. Calzolari,K. Choukri, T. Declerck, M. U. Dogan,B. Maegaard, J. Mariani, J. Odijk, andS. Piperidis, editors, Proceedings of theEight International Conference on Lan-guage Resources and Evaluation (LREC2012), pages 1874–1879, Istanbul, Turkey,May. European Language Resources Asso-ciation (ELRA).

Rios, A. and A. Gohring. 2016. MachineLearning applied to Rule-Based MachineTranslation. In M. Costa-jussa, R. Rapp,P. Lambert, K. Eberle, R. E. Banchs, andB. Babych, editors, Hybrid Approaches toMachine Translation, Theory and Appli-cations of Natural Language Processing.Springer International Publishing.

Annette Rios

94