SINMINCORPUS FOR SINHALA LANGUAGELiterature Review
Upeksha W. D.
Wijayarathna D. G. C. D.
Siriwardena M. P.
Lasandun K. H. L.
Supervisors :
Dr. Chinthana Wimalasuriya
Prof. Gihan Dias
Mr. N. H. N. D. De Silva
Sinmin is a Corpus for Sinhala language which is
➢Continuously updating
➢Dynamic (Scalable)
➢Covers wide range of language (Structured and
unstructured)
OUTLINE
● Literature Review
● Introduction to corpus linguistics and What is a Corpus
● Usages of a corpus
● Existing Corpus Implementations
● Identifying Sinhala Sources and Crawling
● Data Storage and Information Retrieval from Corpus
● Information Visualization
● Extracting Linguistic Feature
● Current Progress
INTRODUCTION TO CORPUS LINGUISTICS
AND WHAT IS A CORPUS
Handford, M. and McCarthy, M. J. (2004) “Invisible to us” - A preliminary corpus based study of spoken
business english, Discourse In the Profession: Perspectives from Corpus Linguistics 167-201
WHAT IS A CORPUS??
“A corpus is a principled collection of authentic texts
stored electronically that can be used to discover
information about language that may not have been
noticed through intuition alone.” - Bennet (2010)
Bennet, G. R. (2010) Using Corpora in the Language Learning Classroom,Michigan ELT.
● There are mainly 8 kinds of corpora.
● They are generalized corpuses, specialized corpuses,
learner corpuses, pedagogic corpuses, historical
corpuses, parallel corpuses, comparable corpuses,
and monitor corpuses.
● The broadest type of corpus is the genarilezed
corpes.
“Sinmin” will
be a generalized corpus.
cover all types of Sinhala Language.
USAGES OF A CORPUS
● Implementing translators, spell checkers and grammar
checkers.
● Identifying lexical and grammatical features of a language.
● Identifying varieties of language of context of usage and
time.
● Retrieving statistical details of a language.
● Providing backend support for tools like OCR, POS Tagger,
etc.
EXISTING CORPUS IMPLEMENTATIONS
● There is a implemented corpus for Sinhala language
which is known as UCSC Text Corpus of
Contemporary Sinhala.
● It consists of about 10 million words, but it covers
very little amount of language and it is not updating.
CORPUS FOR SINHALA LANGUAGE?
COMPOSITION OF THE CORPUS
● Language comprising the corpus cannot be random
but chosen according to specific characteristics.
● It must use authentic texts. The language it contains
is not made up for the sole purpose of creating the
corpus
EXAMPLE - COMPOSITION OF COCA
● The COCA contains more than 385 million words
from 1990–2008 (20 million words each year).
● Texts are evenly divided between 5 genres, spoken
(20%), fiction (20%), popular magazines (20%),
newspapers (20%) and academic journals (20%).
COMPOSITION OF UCSC TEXT CORPUS OF
CONTEMPORARY SINHALA
DATA STORAGE AND INFORMATION
RETRIEVAL FROM CORPUS
Existing corpora uses two main technologies for data
storage
● Relational Databases
● Indexed file Systems
INDEXED FILE SYSTEMS AS STORAGE
● BNC uses this mechanism.
● data is stored as XML like files which follows a
scheme known as the Corpus Data Interchange
Format.
● This supports to store a great deal of detail about the
structure of each text, such as its division into
sections or chapters, paragraphs, verse lines, etc.
RELATIONAL DATABASE AS STORAGE
● COCA, Corpus del Español use relational databases.
DATA MODEL IN COCA
CORPUS DEL ESPAÑOL USES SEPARATE
TABLES FOR BIGRAMS AND TRIGRAMS
RELATIONAL DB VS INDEXED FILE
SYSTEMS
● Indexed file systems use extensive use of indexes
● Relational Database models are relatively fast.
● In Indexed file systems, difficult to add additional
layers of annotation.
No study has been done on
how NoSQL performs in
implementing Corpora.
INFORMATION VISUALIZATION
Most of the popular corpora like BNC, COCA, Corpus
Del Espanol, Google books corpus use similar kind of
Web Interface.
USER INTERFACE
OF COCA
GOOGLE BOOKS NGRAM VIEWER UI
EXTRACTING LINGUISTIC FEATURES
● A main usage of a language corpus is extracting
linguistic features of a language.
● Linguistic features for many languages has been
identified using Corpora.
● Example - A corpus-based linguistics analysis on
written corpus: colligation of “TO” and “FOR.”
CURRENT PROGRESS
IDENTIFIED SINHALA RESOURCES
● Online Newspapers
● News Websites
● School Textbooks
● Sinhala Wikipedia
● Online Mahawansaya
● Subtitles
● Sinhala Fiction
● Sinhala Blogs
● Sinhala Magazines
● Gazette
DIVIDED INTO 5 MAIN GENRES
News Academic Creative
Writing
Spoken Gazette
News Paper Text books Fiction Subtitle Gazette
News Items Religious Blogs
Wikipedia Magazine
mahawansa
Implemented Crawlers for different sources,
adhering to same format.
https://github.com/madurangasiriwardena/corpus.sinhala.crawler
FINISHED CRAWLERS
CRAWLED DATA SAVED TO XML FILES WITH
FOLLOWING META DATA
● Post Name
● Author
● Link
● Published Date
CRAWLER CONTROLLER
Crawler controller monitors and handles the status of
the web crawlers.
Crawler controller address -
http://Sinhala-corpus.projects.uom.lk:8080/CrawlerControllerWeb
We tested performance of several database
systems to determine what should we use
to store data.
WE CONSIDERED FOLLOWING DATA
STORAGE SYSTEMS
We considered performance for inserting
data and for retrieving 12 different
information needs.
Data set and source code -
https://github.com/madurangasiriwardena/performance-test
DATA INSERTION TIME COMPARISON
INFORMATION RETRIEVAL PERFORMANCE
COMPARISON - PART 1
INFORMATION RETRIEVAL PERFORMANCE
COMPARISON - PART 2
Cassandra performed better than others in
most of the scenarios, and its insertion
time increased linearly.
So we chose it for implementing the
corpus.
USER INTERFACE DESIGN AND
IMPLEMENTATION
● Web interface of Sinmin has been designed for users
who would prefer a visualised and summarized view
of statistical data of Sinmin.
● Visual design of the interface has been made in a
way that any user without prior experience of the
interface is able to fulfill his information
requirements with little effort.
http://sinhala-corpus.projects.uom.lk/sinmin-web/
CORPUS API DESIGN AND IMPLEMENTATION
• REST API to expose Corpus services
• Much complex and customizable data retrieval and
filtering
• Interface for third party applications to consume
PUBLICATIONS
● Comparison between performance of various
database systems for implementing a language
corpus - 11th Beyond Databases, Architectures and
Structures conference (Pending)
● Implementing a Corpus for Sinhala Language -
Symposium on Language Technology for South Asia
(Pending)
REMAINING WORK FOR THE NEXT PHASE
• Finish writing crawlers
• Feed data to Cassendra database
• Connecting front end with API calls
Questions?
Thank you!