Clasificacion conjunta de frases clave y sus relacionesen documentos electronicos de salud en espanol

Joint classification of Key-Phrases and Relations inElectronic Health Documents

Salvador Medina1, Jordi Turmo1

1TALP Research Center - Universitat Politecnica de Catalunyasmedina@cs.upc.edu y turmo@cs.upc.edu

Abstract: This paper describes the approach presented by the TALP team for Task3 of TASS-2018 : a convolutional neural network to jointly deal with classification ofkey-phrases and relationships in eHealth documents written in Spanish. The resultsobtained are promising as we ranked in first place in scenarios 2 and 3.Keywords: Relation extraction, joint classification, key-phrase classification, con-volutional nerural networks.

Resumen: Este artıculo describe el metodo presentado por el equipo TALP en laTarea 3 de TASS-2018 : una red neuronal convolucional para tratar conjuntamentela clasificacion de frases clave y sus relaciones en documentos de salud escritos enespanol. La propuesta quedo en primera posicion en los escenarios 2 y 3.Palabras clave: Extraccion de relaciones, clasificacion conjunta, clasificacion defrases clave, redes neuronales convolucionales.

1 Introduction

This article describes the model presented bythe TALP Team for solving B and C sub-tasks of Task 3 in the Taller de AnalisisSemantico en la SEPLN 2018 (TASS-2018)(Martınez-Camara et al., 2018). TASS-2018’sTask 3 consists in recogniting and classifyingkey-phrases as well as identifying the rela-tionships between them in Electronic HealthDocuments (i.e., eHealth documents) writtenin Spanish. Task 3 is divided in sub-tasksA, B and C, which correspond to key-phraseboundary recognition, key-phrase classifica-tion and relation detection, respectively.

In this task, a key-phrase stands for anysub-phrase included in eHealth documentsthat is relevant from the clinical viewpointand can be classified into Concept or Action.The relationships between them are classifiedinto 6 types: 4 of them are between Concepts(is-a, part-of, property-of and same-as) whi-le the rest are between an Action and anot-her key-phrase (subject and target). The pro-posed task is similar to previous competi-tions such as Semeval-2017 Task 10: Scien-ceIE (Gonzalez-Hernandez et al., 2017), butuses a simpler categorization for key-phraseswhile considering a broader range of possiblerelationships.

Participants in the Semeval-2017 Task 10:ScienceIE (Gonzalez-Hernandez et al., 2017)shared task considered a large plethora of su-pervised learning models, ranging from Con-volutional or Recurrent Neural Networks toSupport Vector Machines, Conditional Ran-dom Fields and even rule-based systems, of-ten applying radically different models foreach one of the three sub-tasks. Note thatsome of the teams did not participate in allthree sub-tasks, this was in fact the case forthe winners of sub-tasks BC (MayoNLP (Liuet al., 2017)) and C (MIT (Lee, Dernoncourt,and Szolovits, 2017)).

1.1 Joint classification ofkey-phrases and relationships

In our implementation we tackle both theclassification of key-phrases and the identi-fication of the relationships between them,corresponding to scenarios 2 and 3 of TASS2018’s Task 3, as a single task. The intuitionbehind this decision is that the categories ofkey-phrases are influenced by the relations-hips they hold with other key-phrases. Forinstance, a verb is an Action key-phrase ifand only if it relates to another Action orConcept by either being the subject or tar-get, which means that sometimes phrases are

TASS 2018: Workshop on Semantic Analysis at SEPLN, septiembre 2018, págs. 83-88

ISSN 1613-0073 Copyright © 2018 by the paper's authors. Copying permitted for private and academic purposes.

not key-phrases by themselves but when theyrelate to other phrases.

2 Implementation

The architecture that we propose is repre-sented in Figure 1 and consists of a two-layerConvolutional Neural Network (CNN) whichtakes a vectorial representation of the docu-ments and the position of two key-phrases asinput and applies several convolution filtersfor window sizes from 1 to 4 tokens. The out-puts of these filters are then max-pooled andfed to a fully connected output layer, whichhas two outputs for the given key-phrase pair-wise: the probabilities of either key-phases forbeing Action or Concept, and the probabili-ties of the pairwise for being each possiblekind of relationship, including “other” for norelationship.

At first glance, our architecture is simi-lar to the one proposed by the MIT teamfor the ScienceIE task, which also consists ofa CNN using word-embedding, relative posi-tion and PoS-tags as input features. Howe-ver, it presents some noticeable differences.First of all, our architecture jointly tacklessub-tasks B and C. For this reason, it doesnot take the key-phrase category as an inputand has two additional outputs which holdthe source and destination key-phrases’ clas-ses. Moreover, we optimize all three outputsat the same time and consequently our lossfunction is designed to reflect this.

2.1 Layout of the network andparameter optimization

Artificial Neural Networks (ANN) and mo-re specifically CNNs have proven to be ca-pable of jointly identifying entities and re-lationships in various kinds of textual docu-ments and relation extraction tasks, as it hasbeen demonstrated in recent articles such as(Singh et al., 2013), (Shickel et al., 2017)and (Li et al., 2017). This joint identifica-tion takes advantage of the correlation thatexists between linked entities aiming to pro-vide better results for both named entity re-cognition classification and relation extrac-tion tasks respect to a classical two-step sys-tem.

The loss function used by the parameteroptimization algorithm is computed indepen-dently for the three outputs using soft-maxcross-entropy, as classes are mutually exclu-sive for a single output, and is then combined

by just adding the three losses. By adoptingthese three independent loss functions we cantake profit of the fact that output classes fora single output are mutually exclusive andmake their probabilities add up to one, inde-pendently of the other two outputs.

As for the optimization algorithm, we useTensorFlow’s Adam optimizer with a lear-ning rate of 0,005. The system was trainedin batches of 128 sentences which were pre-viously stripped to up to 50 tokens and pad-ded. We also apply a dropout rate of 0,5 tothe fully-connected output layer for regulari-zation purposes. The parameter optimizationprocess is stopped either when the averageloss in the development corpus remains flatfor 1000 iterations or when 1e5 iterations ha-ve been run.

2.2 Input parameters andencoding

In order to come up with a manageable vec-torial representation of the input sentences,they are previously tokenized using Free-Ling ’s with multi-word and quantity detec-tion as well as Named Entity Classification(NEC) modules disabled, so that multiple to-kens are never joined together. These tokensare then passed through a lookup table con-taining their pre-computed word-embeddingsvectors, which are then joined one-hot en-codings of the relative positions respect tothe target source and destination key-phrasesand their respective Part-of-Speech (PoS) tagdetermined by FreeLing ’s PoS-Tagger modu-le. A more detailed description of the inputproperties is listed below:

Word-Embedding: 300-dimensionvectorial representation of words inword2vec format. We used the pre-trained general-purpose vectors fromSBWCE (Cardellino, 2016), trainedfrom multiple sources.

Distance to source or destinationkey-phrase: One-hot encoding of thedistance respect to the key-phrases. Weconsider two types of distances: absolutedistance in terms of the number of to-kens between each token and key-phraseand number of arcs in the dependencytree between each token and key-phrase,not taking into account the dependencyclass. The latter option was finally se-lected as it yielded better results in the

Salvador Medina y Jordi Turmo

84

Un

Ataque

De

Asma

Se

Produce

Cuando

Los

Síntomas

Empeoran

.

z

Other

Is-a

Part-of

Property-of

Same-as

Subject

target

Action

Concept

Action

Concept

Relation

Source Key-Phrase

Destination Key-Phrase

Convolution Layer Max-Pooling Layer Fully-Connected Layer

(With Dropout)

Word-Embedding So

urc

e K

ey-

Ph

rase

d

ep

en

de

ncy

tre

e

dis

tan

ce

De

stin

atio

n K

ey-

Ph

rase

de

pe

nd

en

cy

tre

e d

ista

nce

Pa

rt-o

f-S

pe

ech

E

mb

ed

din

g

Figura 1: Layout of the proposed Convolutional Neural Network architecture

validation corpus.

Part-of-Speech tag: One-hot encodingof the token’s PoS-tag determined byFreeLing. For simplicity, we only con-template the category and type positionsof the PoS-tag, hence reducing the num-ber of different tags to 33.

2.3 Data augmentation

Relation extraction is a difficult task andusually requires big amounts of trainingexamples in order to be able to correctly ge-neralize the relationship classes. This is spe-cially so for ANN based models, which canbe prone to over-fitting. The training corpusthat was provided for the TASS-2018 is verylimited and classes are considerably unbalan-ced. To give an example, it only includes 30instances of class same-as compared to the911 examples provided for class target.

Because of this, we evaluated several da-ta augmentation alternatives, which addedslight modifications of the original traininginstances to the training set. These modifi-cations included replacing some or all key-phrases by their class name or other key-phrases in the training corpus, or trimmingthe sentences removing some of their tokens.The alternative that worked best in the vali-dation corpus and was used in the final modelwas to trim the context before and after therelationships to 1 and 3 tokens. For instance,

in sentence “Un ataque de asma se producecuando los sıntomas empeoran.”, the targetrelationship between produce and ataque deasma, adds “Un ataque de asma se producecuando” and “Un ataque de asma se produ-ce cuando los sıntomas”, as well as the fullsentence.

3 Results

As it can be seen in Table 1, our model sco-red first in the evaluation scenarios 2 and3, which evaluate sub-tasks BC and C res-pectively. As it was mentioned in Section 1,our system was designed for sub-tasks B andC, so no submission was sent for scenario 1,which also evaluates sub-task A. In terms ofthe individual sub-tasks, our system rakedfirst for sub-task C but was outperformed byrriveraz ’s model in sub-task B.

3.1 Analysis of errors

In this Subsection, we analyze the errors ma-de by our model in Scenarios 2 and 3. Tables2 and 3 show the confusion matrices for sub-tasks B and C in the evaluation of Scenario2. Results for sub-task C in Scenario 3 areanalogous to Scenario 2 and are not shown,as our model does not make use of the addi-tional information given in Scenario 2.

3.1.1 Sub-Task B

Table 2 shows the confusion matrix for sub-task B in Scenario 2. Our model achieves si-

Clasificación Conjunta de Frases Clave y sus Relaciones en Documentos Electrónicos de Salud en Español

85

Scenario plubeda rriveraz upf upc VSP baseline Marcelo TALP

1 0.71 0.744 0.681 0.297 0.566 0.181 N/A*2 0.674 0.648 0.622 0.275 0.577 0.255 0.7223 N/A* N/A* 0.036 0.42 0.107 0.018 0.448

avg 0.461 0.464 0.446 0.331 0.417 0.151 0.39

Tabla 1: Micro-averaged F1 score for evaluation scenarios 1 to 3 and global average. TALPcolumn shows our model’s score. N/A*: Not Available, counted as 0 in the average score.

true\pred. Concept Action recall

Concept 432 7 0.984Action 34 120 0.779

precision 0.927 0.945 Acc = 0,931

Tabla 2: Confusion matrix of our model’s pre-dictions for sub-task B in scenario 2.

milar precision for classes Concept and Ac-tion, but recall for the latter is 0.205 smaller.This is not only due to the fact that clas-ses are unbalanced (439 and 154 instances ofclasses Concept and Action respectively), butalso to other reasons listed below:

The Shared-Task’s description definesActions as a particular kind of Conceptthat modifies another concept. Conse-quently, in some cases, the same phra-se can either be an Action or a Conceptdepending on whether or not the modi-fied Concept is explicitly mentioned. Asan illustration, the noun causa (cause)is labeled as a Concept in sentence “Eltratamiento depende de la causa.” (Thethreatment depends on the cause.). Ho-wever, in sentence “Es una causa comunde sordera.” (It is a common cause ofdeafness.), it is labeled as Action, as it issupposed to modify sordera (deafness).

Errors which were in part due to inco-rrect dependency parsing or PoS-taggingby FreeLing, specially when verbs areidentified as nouns.

For example, the noun oıdo (ear) wasidentified as a verb in sentence “Sueleafectar solo un oıdo.” (It usually affectsjust one ear.) by FreeLing, which leadto confusion. Similarly, in sentence “Es-to causa una acumulacion de sustanciasgrasosas en el bazo, hıgado, pulmones,huesos y, a veces, en el cerebro. (Thiscauses an accumulation of fatty substan-ces in the arm, liver, lungs, bones and,

sometimes, the brain.), causa (causes) isincorrectly labeled as a noun.

Other instances where it is difficult todetermine the label assigned to the en-tity, even for us, as they do not seem tocorrespond to any of the criteria exposedin the description.

For instance, in sentence “Si usted yatiene diabetes, el mejor momento paracontrolar su diabetes es antes de quedarembarazada.” (If you alredy have diabe-tes, the best moment to control your dia-betes is before getting pregnant.), theadverb antes (before) is labeled as Ac-tion and is related to controlar (control,keep) and quedar (get, become) as sub-ject and target respectively.

On the other hand, in sentence “La ex-posicion al arsenico puede causar mu-chos problemas de salud.” (The exposi-tion to arsenic can cause several healthproblems), the noun exposicion (exposi-tion) is labeled as Concept, while we un-derstand it as the Action of being expo-sed to something. This is not coherentto other instances such as “No se conocela causa de la destruccion celular.” (Thecause of cell destruction is not known.),where destruction is labeled as Action -the Action of being destroyed.

3.1.2 Sub-Task C

Table 3 shows the confusion matrix for sub-task C in Scenario 2. Class other is used forall pairs of entities that have no specified re-lationship in the training set, making it themost frequent class in the training set. Themodel seems to prioritize precision over re-call, which vary from class to class. Recall andprecision for same-as, although 0,000, are notsignificant, as just one instance is present inthe test set. The list below describes multiplereasons for the most common errors producedby our model:

Salvador Medina y Jordi Turmo

86

true\pred. oth

er

is-a

part

-of

pro

per

ty-o

f

sam

e-as

sub

ject

targ

et

reca

ll

other 0 0 0 0 0 0 0 0.000is-a 31 58 1 2 0 0 0 0.630

part-of 26 2 5 0 0 0 0 0.152property-of 34 0 3 18 0 0 3 0.310

same-as 0 1 0 0 0 0 0 0.000subject 65 0 0 2 0 42 8 0.359target 84 0 1 7 0 12 91 0.467

precision 0.000 0.951 0.500 0.621 0.000 0.778 0.892 F1 = 0,431

Tabla 3: Confusion matrix, precision and recall of our model’s predictions for sub-task C inscenario 2. F1 is micro-averaged for all classes.

Annotated instances in both trainingand test sets are unbalanced. Relations-hip counts in the training set range from991 for target and 693 for subject to 149and 30 for part-of and same-as respec-tively. What is more, the auxiliary classother amounts to 16478 instances. Moreinstances for the two less common clas-ses seem to be required, as the modelachieves much lower recall and precisionthan the most common ones.

Relationships subject and target are pro-ne to be mutually confused, speciallyfor reflexive or passive verbs, and labe-ling is not always coherent. For exam-ple, in “Algunos sarpullidos se desarro-llan inmediatamente.” (Some skin ras-hes are developed immediately.), sarpu-llidos (skin rashes) is subject of se desa-rrollan (are developed). However, in sen-tence “Existen muchas razones para so-meterse a una cirugıa.” (There are se-veral reasons to have surgery.), razones(reasons) is target of existen (there are).

Multi-label relationships were not consi-dered by our model, as we did not realizeinstances such as Durante cada trimes-tre, el feto crece y se desarrolla. (Duringeach quarter, the fetus grows and deve-lops.), where the relationships betweenfeto (fetus) and crece (grows), and simi-larly between feto and se desarrolla (de-velops), are both target and subject.

Errors due to incorrect parsing by Free-Ling, which were already discussed inSection 3.1.1.

4 Conclusions and future work

In this paper, we have described the modelpresented by the TALP team for Task 3 ofTASS-2018. In addition we have presentedsome reasons for our model to wrongly clas-sify key-phrases and relationships.

The results achieved by our model whencompared to the rest of the challengers pro-ve that a model that jointly classifies entitiesand relations can outperform traditional two-step systems in tasks where some entity clas-ses are defined by the relationships they holdwith others. There is however a big room forimprovement, specially in the relation extrac-tion task, mainly due to the increased com-plexity and the limited amount of examplesavailable in the training set.

Our model was designed to solve the key-phrase classification and relation extractiontasks, leaving the key-phrase recognition asfuture work, as our focus was joint recogni-tion and we did not have enough time to de-sign and optimize a single model that couldtackle all three tasks. We are committed tocontinue this line of investigation and extendthe architecture so that it is also able to de-termine the key-phrases’ boundaries.

Additionally, there are several improve-ments that could be applied to the currentmodel, that we realized after analyzing thecurrently most common errors. To beginwith, our model should allow for multi-labelrelation extraction, as mentioned in Section3.1.2. Second, more syntactical features couldbe added, by for instance providing a com-plete and more appropriate encoding of thePoS-tags or by including not only the depen-dency tree distances but also the types.

Clasificación Conjunta de Frases Clave y sus Relaciones en Documentos Electrónicos de Salud en Español

87

Acknowledgments

This works has been partially fundedby the Spanish Goverment and by theEuropean Union through GRAPHMEDproject (TIN2016-77820-C3-3-R andAEI/FEDER,UE.)

References

Cardellino, C. 2016. Spanish Billion WordsCorpus and Embeddings, March.

Gonzalez-Hernandez, G., A. Sarker,K. O’Connor, and G. Savova. 2017. Cap-turing the patient’s perspective: a reviewof advances in natural language proces-sing of health-related text. Yearbook ofmedical informatics, 26(01):214–227.

Lee, J. Y., F. Dernoncourt, and P. Szolo-vits. 2017. Mit at semeval-2017 task10: Relation extraction with convolutio-nal neural networks. arXiv preprint ar-Xiv:1704.01523.

Li, F., M. Zhang, G. Fu, and D. Ji. 2017. Aneural joint model for entity and relationextraction from biomedical text. BMCbioinformatics, 18(1):198.

Liu, S., F. Shen, V. Chaudhary, and H. Liu.2017. Mayonlp at semeval 2017 task10: Word embedding distance pattern forkeyphrase classification in scientific publi-cations. In Proceedings of the 11th Inter-national Workshop on Semantic Evalua-tion (SemEval-2017), pages 956–960.

Martınez-Camara, E., Y. Almeida-Cruz,M. C. Dıaz-Galiano, S. Estevez-Velarde,M. A. Garcıa-Cumbreras, M. Garcıa-Vega, Y. Gutierrez, A. Montejo-Raez,A. Montoyo, R. Munoz, A. Piad-Morffis,and J. Villena-Roman. 2018. Over-view of TASS 2018: Opinions, healthand emotions. In E. Martınez-Cama-ra, Y. Almeida Cruz, M. C. Dıaz-Galiano, S. Estevez Velarde, M. A.Garcıa-Cumbreras, M. Garcıa-Vega,Y. Gutierrez Vazquez, A. Montejo Raez,A. Montoyo Guijarro, R. Munoz Guillena,A. Piad Morffis, and J. Villena-Roman,editors, Proceedings of TASS 2018:Workshop on Semantic Analysis atSEPLN (TASS 2018), volume 2172 ofCEUR Workshop Proceedings, Sevilla,Spain, September. CEUR-WS.

Shickel, B., P. J. Tighe, A. Bihorac, andP. Rashidi. 2017. Deep ehr: A surveyof recent advances in deep learning tech-niques for electronic health record (ehr)analysis. IEEE Journal of Biomedical andHealth Informatics.

Singh, S., S. Riedel, B. Martin, J. Zheng, andA. McCallum. 2013. Joint inference of en-tities, relations, and coreference. In Pro-ceedings of the 2013 workshop on Auto-mated knowledge base construction, pages1–6. ACM.

Salvador Medina y Jordi Turmo

88