One stage versus two stages deep learningapproaches for the extraction of drug-drug

interactions from texts

Comparando enfoques deep learning en una fase y en dosfases para extraer interacciones farmacologicas de texto

Antonio Miranda-Escalada1, Isabel Segura-Bedmar21Barcelona Supercomputing Center, Barcelona, Spain2Universidad Carlos III de Madrid, Leganes, Spain

antonio.miranda@bsc.es, isegura@inf.uc3m.es

Abstract: Drug-drug interactions (DDI) are a cause of adverse drug reactions.They occur when a drug has an impact on the effect of another drug. There is not acomplete, up to date database where health care professionals can consult the inter-actions of any drug because most of the knowledge on DDI is hidden in unstructuredtext. In last years, deep learning has been succesfully applied to the extraction ofDDI from texts, which requires the detection and later classification of DDI. Mostof the deep learning systems for DDI extraction developed so far have addressedthe detection and classification in one single step. In this study, we compare theperformance of one-stage and two-stage architectures for DDI extraction. Our ar-chitectures are based on a bidirectional recurrent neural network layer composed ofGated Recurrent Units. The two-stage system obtained a 67.45 % micro-average F1score on the test set.Keywords: Relation Extraction, Drug-drug interaction, Recurrent Neural Net-work, Gated Recurrent Unit

Resumen: Las interacciones farmacologicas (DDI) son una de las causas de reac-ciones adversas a medicamentos. Ocurren cuando una medicina interfiere en laaccion de una segunda. En la actualidad, no existe una base de datos completa yactualizada donde los profesionales de la salud puedan consultar las interaccionesde cualquier medicamento porque la mayor parte del conocimiento sobre DDIs estaoculto en texto no estructurado. En los ultimos anos, el aprendizaje profundo se haaplicado con exito a la extraccion de DDIs de los textos, lo que requiere la detecciony posterior clasificacion de DDIs. La mayorıa de los sistemas de aprendizaje pro-fundo para extraccion de DDIs desarrollados hasta ahora han abordado la detecciony clasificacion en un solo paso. En este estudio, comparamos el rendimiento de lasarquitecturas de una y dos etapas para la extraccion de DDI. Nuestras arquitecturasse basan en una capa de red neuronal recurrente bidireccional compuesta de GatedRecurrent Units (GRU). El sistema en dos etapas obtuvo un puntaje F1 promediode 67.45 % en el dataset de evaluacion.Palabras clave: Extraccion de relaciones, interacciones farmacologicas, Redes neu-ronales recurrentes, Gated Recurrent Unit

1 Introduction

One of the causes of adverse drug reactions(ADR) is the wrong combination of differ-ent drugs. That is, when one drug influencesthe effect of another. This is known as adrug-drug interaction (DDI). DDIs may havea positive effect on human health. Never-theless, many DDIs may trigger adverse drugreactions, which can cause health problemsand increase healthcare costs.

Despite the efforts in reporting ADRs,such as the monitoring system maintained bythe World Health Organization, there is nota single up-to-date database where clinicianscan look for all the known DDIs of a drug.Current databases have varying update fre-quencies, being some of them up to 3 years(Segura-Bedmar, 2010). In addition, mostof the information available about DDIs isunstructured, written in natural language in

Procesamiento del Lenguaje Natural, Revista nº 64, marzo de 2020, pp. 69-76 recibido 30-11-2019 revisado 14-01-2020 aceptado 16-01-2020

ISSN 1135-5948. DOI 10.26342/2020-64-8 © 2020 Sociedad Española para el Procesamiento del Lenguaje Natural

scientific articles, books and reports.

In this scenario, a challenge has been iden-tified: an automatic system to extract DDIsfrom biomedical literature is needed to cre-ate complete and up-to-date databases withinformation about DDIs for healthcare pro-fessionals. These databases would contributeto reduce adverse drug reactions.

The DDI corpus (Herrero-Zazo et al.,2013) as well as the shared tasks DDIExtrac-tion 2011 (Segura Bedmar, Martinez, andSanchez Cisneros, 2011) and 2013 (Segura-Bedmar, Martınez, and Herrero-Zazo, 2013)have undoubtedly contributed to the progressof NLP research for the extraction of DDIfrom texts. Since then, the popularity ofthis task has rapidly increased over the pastfew years and a considerable number of pa-pers devoted to the topic is published everyyear. Early systems (Segura-Bedmar, Mar-tinez, and de Pablo-Sanchez, 2011; Thomaset al., 2013; Chowdhury and Lavelli, 2013;Abacha et al., 2015; Kim et al., 2015) werebased on linguistic features combined withclassical machine learning algorithms such asSVM (with F-measures around 67%)(Kim etal., 2015). Recently, deep learning meth-ods have triggered a revolution in NLP,demonstrating tremendous success in numer-ous NLP tasks. The task of DDI extrac-tions has not been oblivious to this rev-olution. Various deep learning architec-tures based on Convolutional Neural Net-work (CNN) and Recurrent Neural Network(RNN) have been explored to this task in thelast five years, achieving state-of-the-art per-formance (Dewi, Dong, and Hu, 2017; Sun etal., 2018).

In most of these architectures, the detec-tion and classification of DDIs are carried outin one only stage. The recent deep learn-ing architectures have achieved state-of-artresults (around 80% of micro-F1), but withlong training times because of their deep ar-chitectures. Our hypotehsis is that perform-ing first the detection and then the classifi-cation could give better results and reducethe training time. The detection step couldsignificantly decrease the number of instancesto classify, and thereby, reducing the trainingtime of the classification task. Our goals ofthis work are two-fold: (1) to study if a two-stage arquictecture could give better resultsthan an one-stage one, and (2) to test if theuse of Gated Recurrent Units (GRUs)(Cho et

al., 2014), instead of using Long Short-TermMemory (LSTM) units, can improve the re-sults of DDI extraction. GRUs show betterperformance on smaller datasets (Chung etal., 2014). Moreover, we also study the ef-fect of different pre-trained word embeddingsmodels (Pyysalo et al., 2013) on the results.

2 State of the Art

This section provides a review of the mostrecent studies based on deep learning aboutDDI extraction. Deep CNN models (Dewi,Dong, and Hu, 2017; Sun et al., 2018) haveshown the state-of-the-art performance ( 85%in F1-score).

In addition to CNN, several systems haveexploited Recurrent Neural Network (RNN)for the task of DDI extraction. While CNNhas proved to be successful in discovering dis-criminative and meaningful phrases in a text,RNN models are able capable of capturingcontextual information and long-term depen-dencies (Lai et al., 2015), which is very im-portant in our case since the clues about aDDI could appear anywhere in a sentence.Most RNN systems for DDI extraction haveused LSTM units, to our knowledge. Thestandard architecture includes, after prepro-cessing, an embedding layer, a bidirectionalLSTM, a pooling layer and a softmax layerthat retrieves a probability for each DDItype. Some systems incorporate a dense layerbefore the softmax layer, as it also happenedin some CNN-based systems.

Zheng et al. (2017) used a RNN andobtained 77.3% F1-score. Input attentionmechanism was used only on the word em-bedding vectors. Also, a part of speech(POS) tag embedding and position embed-dings were concatenated to the word. Unlikethe most systems for DDI extraction based ondeep learning methods, there was not poolinglayer between the bidirectional LSTM andthe softmax layer.

Despite most works employ LSTM units,the work of Yi et al. (2017) proposed a sys-tem utilizing GRU. The system concatenateda word embedding with two position embed-dings. Then, embedding vectors were fedinto a bidirectional GRU layer. The outputof the bi-GRU layer was transformed by anattention-pooling layer, and later by anotherattention layer. Finally, a softmax layer re-turned the DDI class probabilities. An F1score of 72.23% on the DDI corpus was re-

Antonio Miranda-Escalada, Isabel Segura-Bedmar

70

ported.All these systems have in common that the

detection and classification of DDIs are per-formed in one only step. This is the first workthat compares one-stage and two-stages deeplearning architectures for DDI extraction.

3 Approach

This section details the corpus employed to-gether with the two approaches tested. Thefirst solution, named one-stage architecture,detects and classifies drug-drug interactionsemploying one single GRU. On the otherhand, the second approach, named two-stagearchitecture, utilizes two steps: one GRU todetect the DDIs and a second GRU to classifythem. In addition, the common preprocess-ing for both solutions is also described.

3.1 DDI Corpus

The DDI corpus (Herrero-Zazo et al., 2013) isthe standard corpus employed in DDI extrac-tion from texts and as such is used within thiswork. It contains sentences mentioning phar-maceutical substances extracted from textsdescribing DDIs from DrugBank database(Wishart, 2017) and scientific abstracts fromMedline. In total, the DDI corpus contains33,502 DDI instances and it is highly unbal-anced: 85% of instances belong to the neg-ative class -no DDI is present- and 15% ofinstances belong to the positive class -a DDIis described. DDIs are further divided into 4categories:

• mechanism: it is used for pharmacoki-netic mechanisms (e.g. Grepafloxacinmay inhibit the metabolism of theo-bromine).

• effect : it is used for descriptions ofeffects (e.g. In uninfected volunteers,46% developed rash while receiving SUS-TIVA and clarithromycin) or pharmaco-dynamic mechanisms (e.g. Chlorthali-done may potentiate the action of otherantihypertensive drugs).

• advise: it contains DDIs written as ad-vise (e.g. Ocupress should be used withcaution in patients (...)).

• int : this type is used when no extra in-formation is provided (e.g. Interactionof clindamycin and gentamicin in vitro).

The corpus was randomly divided into twodatasets for training and test. In our study,

we assume that the drug mentions are al-ready provided. Thus, we focus on the tasksof detection and classification of DDIs fromtexts.

3.2 Models

Preprocessing is common to both systemsproposed. First, instances were transformedto lowercase. Then, punctuation signs wereremoved. After that, drug blinding was per-formed. That is, the two drugs involved inthe interaction are respectively substitutedby “DRUGA” and “DRUGB”, and otherdrugs mentioned in the sentence are substi-tuted by “DRUGN”.

After these steps, sentences are tokenized(split into separate tokens). As a means ofhaving all sentences of equal length, paddingwas applied after tokenization. A maximumlength of 89 was established, since the longestinstance in the training dataset of the DDIcorpus contains 89 words.

Finally, position flags are added to ev-ery token. They indicate the distance ofeach token to the two drugs involved inthe potential interaction (named “DRUGA”and “DRUGB”). Note that, for the tokens“DRUGA” and “DRUGB”, one of the posi-tion flags is zero, since the distance to their-selves is zero.

After preprocessing, every instance is rep-resented as a matrix composed by n to-kens. This representation is known as theembedding layer, where each token is rep-resented by a vector that concatenates oneword embedding and two position embed-dings. We explored the use of two differ-ent word embedding models. The first onewas a pre-trained word2vec 200-dimensionalmodel, which was trained with biomedicaltexts taken from PubMed and PubMed Cen-tral (PMC). The second pre-trained word em-bedding model was trained using PubMed,PMC and Wikipedia. So, unlike the firstmodel, it contains information about generalknowledge texts. Both embeddings were de-veloped by Pyysalo et al. (2013); the corpusto train the first one contained 5.5B tokensand for the second one 7B tokens. Each po-sition embedding codifies the distance fromthe word to one of the interacting drugs inthe DDI instance. Position embeddings wererandomly initialized to 32-dimensional vec-tors from the uniform distribution U(0,0.01).Then, at this point, every instance, S, is a

One stage versus two stages deep learning approaches for the extraction of drug-drug interactions from texts

71

matrix of vectors, −→gi such as:

S = [−→g1 , ...,−→gN ],

−→gi = −→wi||−→e1i||−→e2i

(1)

3.3 one-stage approach for DDIextraction

First, we describe the first approach for DDIextraction, where the detection and classi-fication tasks are performed in one singlestep. Once an instance has been preprocessedand represesented by the embedding layer de-scribed above, the one-stage system classifiesit as either belonging to the negative classor to one of the four positive DDI types ofthe DDI corpus (effect, mechanism, advise orint). It employs four layers to classify the in-stances: embedding, bidirectional GRU, maxpooling and output layer (figure 1).

After the embedding layer, a bidirectionalrecurrent neural network with GRUs ensues.We decided to use GRU with 512 units perdirection. The output vectors from both di-

rections,−→hf,i and

−→hb,i, are concatenated:

−→hf,i =

−−−→GRU(−→gi ),

←−hb,i =

←−−−GRU(−→gi ),

~hi =−→hf,i||

−→hb,i,

S = [−→h1, ...,

−→hN ]

(2)

Then, the instance tensor, S, is inputinto a max pooling layer, in which only thetimestep of highest magnitude for each fea-ture is kept.

−→qi = max(hii, ..., hiN ),

S = [−→q1 , ...,−→qN ](3)

Finally, the instance vector, S, is inputinto a softmax layer with five output units.It contains 5 neurons employing softmax ac-tivation functions. Every neuron returns theprobability that the instance belongs to oneof the five possible classes, the four DDI typesdefined in the DDI corpus and the non-DDItype. Therefore, a 5-dimensional output vec-tor represents the confidence the system as-signs to each of the five classes. The classwith higher confidence is selected as the pre-dicted class.

Figure 1: one-stage architecture

3.4 two-stage approach for DDIextraction

Unlike the one-stage system, in the two-stagesystem, there is a first step to tag each in-stance as either negative or positive. Then,instances classified as negative are ruled out.The second step deals with the classificationof the positive instances into one of the fourDDI corpus categories: mechanism, advise,effect or int (figure 2). We describe in moredetail each stage below.

The first stage (named detector) also em-ploys four layers to detect the DDI instances:an embedding layer, a bidirectional GRU, amax pooling and a softmax layer with twooutputs neurons: one neuron represents theprobability for the positive class (the instanceis a DDI) and the other neuron stores theprobability for the negative class (the in-stance is not a DDI).

The second stage, named as Classifier,only considers those instances that were clas-sified as positive by the previous step, rulingout the rest of instances. The architectureof this stage is very similar to the previousones, however, we also introduce, after themax pooling layer, a 64-unit fully connectedlayer with ReLU activation function, becauseits integration has shown better performancein combination with complex CNN or RNNarchitectures (Mohamed, Hinton, and Penn,2010; Sainath et al., 2015). The output layercontains 4 neurons employing softmax activa-tion functions, one per each DDI type. Thehigher-probability class is selected as the pre-dicted DDI type.

Antonio Miranda-Escalada, Isabel Segura-Bedmar

72

Figure 2: two-stage architecture

3.5 Networks training details

While training the Classifier, class weightswere used when computing the loss functionfor the classes mechanism, advise and effect.This increased the importance on the losscomputation of the underrepresented classesand partially solved the class imbalance prob-lem of the dataset. Finally, for training thissystem, only positive instances were used.

To further deal with class imbalance, neg-ative instance filtering was applied using therules defined by Kavuluru, Rios, and Tran(2017). In addition, naıve oversampling wasemployed in the training of the one-stage sys-tem and the Detector stage. For the differ-ent layers, weights were always initialized ac-cording to the Glorot Initializer (Glorot andBengio, 2010).

In the recurrent layer, we apply always theso-called naıve dropout (Zaremba, Sutskever,and Vinyals, 2015). The dropout probabilityin the GRU layer and in the fully connectedlayer was chosen to be always 0.5. The deci-sion was heuristic, since intra-class variabilitywas higher than inter-class variability for thedifferent dropout probabilities tested.

Early Stopping was applied with a pa-tience of 5 epochs. In the Detector stage,F1 score of the positive class was monitored.In the Classifier stage and in the one-stagesystem, the monitored quantity was micro-average F1 score of the classes mechanism,advise and effect. Class int was not takeninto account since its number of instances issmall (196 in training test, while there are1687 effect instances, for instance).

The optimizer chosen was Adagrad withinitial learning rate 0.01. The loss selected

was the cross-entropy loss. The mini-batchsize was always 50 and the number of epochswas always 25.

4 Results and Discussion

As baseline, we propose the system de-scribed in (Suarez-Paniagua and Segura-Bedmar, 2018), which exploited a CNN withmax pooling operations, obtaining an F1 of64.56%.

Table 4 shows the results for both ap-proaches: one-stage versus two-stages archi-tectures. As described above, we aim to com-pare two different pre-trained word embed-dings models to initialize our networks. Bothpre-trained models are described in Pyysaloet al. (2013). The first model (from nowon, we call it as biomedical) was trained onlyusing biomedical literature such as PubMedand PMC, while the second one (from now ongeneral) was trained also using general textstaken from Wikipedia.

From the results in Table 4, it is seen thatthe use of GRU is superior to the CNN ar-chitecture, since both compared architectures(one-step and two-step) show greater perfor-mance than the baseline (Suarez-Paniaguaand Segura-Bedmar, 2018), even using gen-eral domain word embeddings.

In the one-stage approach, the biomedicalpre-trained word embedding models providesslightly better results than using the generalpre-trained word embedding model (see Ta-ble 4).

Only the precision of the int class is bet-ter using the general pre-trained word em-bedding model. However, this class is theleast represented in the corpus: there are 189

One stage versus two stages deep learning approaches for the extraction of drug-drug interactions from texts

73

One-S. (biomedical) One-S. (general) Two-S. (biomedical) Two-S. (general)P R F1 P R F1 P R F1 P R F1

Int 0.65 0.33 0.44 0.67 0.31 0.43 0.68 0.38 0.48 0.53 0.34 0.45Advise 0.78 0.75 0.77 0.75 0.71 0.73 0.74 0.79 0.76 0.69 0.80 0.74Effect 0.68 0.69 0.69 0.66 0.68 0.67 0.64 0.70 0.67 0.61 0.70 0.65Mechanism 0.72 0.66 0.69 0.72 0.60 0.66 0.68 0.66 0.67 0.66 0.73 0.69Micro-average 0.71 0.66 0.69 0.70 0.62 0.66 0.67 0.67 0.67 0.64 0.70 0.67Macro-avg. 0.71 0.61 0.65 0.70 0.59 0.62 0.69 0.63 0.65 0.62 0.64 0.63

Table 1: Comparative results for one-stage and two-stages systems for DDI extraction

int instances in the training set and 96 inthe test set, while there are 1687 and 360effect instances. Therefore, results from ofclass are less informative of general classifierbehaviour.

Finally, the McNemar test (McNemar,1947) of homogeneity was performed to com-pare the results of the system using bothword embeddings. This test assesses whetherthe proportion of errors of two classifiersis statistically significant. In this case, itis, since the p-value was smaller than 0.05(0.02). Therefore, in the case of the one-stageapproach, using a word embedding trainedwith texts from the domain of knowledgeof the problem appears to be the optimumchoice.

Table 4 also compares the results obtainedwith the two different word embeddings mod-els for the two-stage approach. The generalpre-trained word embedding model obtainsa micro-average F1 score of 66.73%, slightlysmaller than the metric obtained with thebiomedical pre-trained model.

Both word embedding models result insimilar performances. However, there are afew differences whose discussion ensue. First,the recall of the mechanism class is 7 pointshigher using the domain-combined word em-bedding: there are less mechanism instancesmisclassified as other classes. This effect isnot observed in the other positive classes.Therefore, there may be a difference in theway information is encoded in mechanismsentences and it is affected by the use ofa word embedding that incorporates infor-mation from domains of knowledge differentfrom the biomedical one.

Second, there are 14 points of differencein the precision for the Int class. The rea-son is a 209% increment in the negative in-stances misclassified as Int. However, the to-tal number of Int instances is extremely low:the 209% increment corresponds to a changefrom 11 to 23 misclassified instances.

Finally, the McNemar test of homogeneityhas been performed and according to the re-

sulting p-value, p=0.054, the difference in theproportion of errors is on the edge of beingsignificant using the two word embeddings.

We now compare the performance of theone-stage and two-stage approaches. If weuse the biomedical pre-trained word embed-ding model for both approaches, the micro-average F1 score of the one-stage System is68.54% on the test set, while the two-stageapproach achieves an F1 score of 67.45%.

The one-stage approach is superior for allclasses (except for Int) on the relevant met-ric, F1 score. Nonetheless, the two-stage sys-tem is superior in recall for classes Int, ad-vise and effect. Then, in the one-stage ap-proach, there were more false negatives, whilein the two-stage approach, there were morefalse positives. This phenomenon may be aconsequence of the two-stage approach. Init, the first stage (detector) rules out nega-tive instances. However, its performance isnot perfect, and negative instances may bemisclassified as positive instances, which areentered to the second stage, the classifier.There, these negative instances are tagged asmechanism, advise, effect or Int. And there-fore, the number of false positives increasesfor the positive classes. An example of thisphenomenom is shown in Table 4, sentences2 and 3.

To statistically compare both systems,McNemar test of homogeneity was per-formed. The resulting p-value, p=0.147, doesnot allow to say that both systems are signif-icantly different in their error proportion.

Performances of one-stage and two-stagesystems are comparable. The one-stage ar-chitecture obtains a slightly better perfor-mance. This may indicate that one-stage sys-tems are more suitable for DDI extraction.

Some light experimentation showed thatboth proposed architectures misclassify sen-tences with more than two drugs mentioned.As seen in example 1 from Table 4, two drugsadjacent in the sentence are not consideredto interact, but when one of them is furtheraway, an interaction is wrongly undetected by

Antonio Miranda-Escalada, Isabel Segura-Bedmar

74

Sentence One-S. Two-S. Truth(1) DRUGA DRUGB as well as other DRUGN may affect ... to DRUGN None None None

DRUGN DRUGA as well as other DRUGN may affect ... to DRUGB None effect effect(2) the use of DRUGA before DRUGB to attenuate ... DRUGN has not been studied None effect None(3) ... drug drug interaction studies between DRUGA and DRUGB are inconclusive None effect None(4) DRUGA inhibits the glucuronidation of DRUGB and could possibly potentiate DRUGN None effect mechanism

DRUGA inhibits the glucuronidation of DRUGN and could possibly potentiate DRUGB effect None effectDRUGN inhibits the glucuronidation of DRUGA and could possibly potentiate DRUGB None None None

Table 2: Examples of errors in prediction

the one-stage system. In addition, handlingwith negation and uncertainty are recurrentchallenges in NLP systems. Example 4 fromTable 4 shows how both architectures commitmistakes when dealing with the constructioncould possible potentiate. A greater corpuscould allow the network to reduce those mis-takes and others.

5 Conclusions and FutureDirections

Most previous studies on DDI extraction usu-ally opt for one-stage architecture. This workproposes a comparison of one-stage versustwo-stage architectures. The two-stage ap-proach first detects the positive instances andrules out the negative instances. Then, onlythe positive instances are classified in a sec-ond stage. Both approaches use GRU, a typeof recurrent neural network unit.

Results did not show a significant differ-ence in the error distribution of both sys-tems. However, F1 score is slightly higher forthe one-stage System than for the two-stage(68.54% vs 67.45%).

Since performances are comparable, thissuggests that the use of one-stage architec-tures is more suitable in deep learning basedDDI extractors because of its simpler de-sign. On the other hand, experiments showthat the one-stage architecture requires moretraining time, since more sentences are syn-thetically created in the oversampling phaseand therefore more instances are used duringtraining. The pre-trained word embeddingmodel created from biomedical literature alsoprovides better performance than the generalword embedding model.

As future work, we plan to explore hy-brid architectures which exploit advantagesof both CNN and LSTM models.

Acknowledgments

This work was supported by the ResearchProgram of the Ministry of Economy andCompetitiveness - Government of Spain,(DeepEMR project TIN2017-87548-C2-1-R).

References

Abacha, A. B., M. F. M. Chowdhury,A. Karanasiou, Y. Mrabet, A. Lavelli, andP. Zweigenbaum. 2015. Text mining forpharmacovigilance: Using machine learn-ing for drug name recognition and drug–drug interaction extraction and classifica-tion. Journal of biomedical informatics,58:122–132.

Cho, K., B. Van Merrienboer, C. Gulcehre,D. Bahdanau, F. Bougares, H. Schwenk,and Y. Bengio. 2014. Learning phraserepresentations using rnn encoder-decoderfor statistical machine translation. arXivpreprint arXiv:1406.1078.

Chowdhury, M. F. M. and A. Lavelli. 2013.Exploiting the scope of negations and het-erogeneous features for relation extrac-tion: A case study for drug-drug interac-tion extraction. In Proceedings of the 2013Conference of the North American Chap-ter of the Association for ComputationalLinguistics: Human Language Technolo-gies, pages 765–771.

Chung, J., C. Gulcehre, K. Cho, and Y. Ben-gio. 2014. Empirical evaluation of gatedrecurrent neural networks on sequencemodeling. In NIPS 2014 Workshop onDeep Learning.

Dewi, I. N., S. Dong, and J. Hu. 2017. Drug-drug interaction relation extraction withdeep convolutional neural networks. InIEEE International Conference on Bioin-formatics and Biomedicine (BIBM).

Glorot, X. and Y. Bengio. 2010. Understand-ing the difficulty of training deep feed-forward neural networks. In 13th Inter-national Conference on Artificial Intelli-gence and Statistics (AISTATS).

Herrero-Zazo, M., I. Segura-Bedmar,P. Martınez, and T. Declerck. 2013.The DDI corpus: An annotated corpuswith pharmacological substancesanddrug–drug interactions. Journal ofBiomedical Informatics, 46:914–20.

One stage versus two stages deep learning approaches for the extraction of drug-drug interactions from texts

75

Kavuluru, R., A. Rios, and T. Tran.2017. Extracting Drug-Drug interactionswith Word and Character-Level Recur-rent Neural Networks. In IEEE Interna-tional Conference on Healthcare Informat-ics (ICHI).

Kim, S., H. Liu, L. Yeganova, and W. J.Wilbur. 2015. Extracting drug–drug interactions from literature using arich feature-based linear kernel approach.Journal of biomedical informatics, 55:23–30.

Lai, S., L. Xu, K. Liu, and J. Zhao. 2015. Re-current convolutional neural networks fortext classification. In Twenty-ninth AAAIconference on artificial intelligence.

McNemar, Q. 1947. Note on the samplingerror of the difference between correlatedproportions or percentages. Psychome-trika, 12.

Mohamed, A.-R., G. Hinton, and G. Penn.2010. Understanding the difficulty oftraining deep feedforward neural net-works. In Acoustics, Speech and SignalProcessing (ICASSP), 2012 IEEE Inter-national Conference.

Pyysalo, S., F. Ginter, H. Moen,T. Salakoski, and S. Ananiadou. 2013.Distributional Semantics Resourcesfor Biomedical Text Processing. InProceedings of LBM 2013.

Sainath, T. N., O. Vinyals, A. Senior, andH. Sak. 2015. Convolutional, long short-term memory, fully connected deep neuralnetworks. In Acoustics, Speech and Sig-nal Processing (ICASSP), IEEE Interna-tional Conference.

Segura-Bedmar, I. 2010. Application of In-formation Extraction techniques to phar-macological domain: Extracting drug-druginteractions. Ph.D. thesis, Department ofComputer Science, University Carlos IIIof Madrid.

Segura-Bedmar, I., P. Martinez, andC. de Pablo-Sanchez. 2011. Usinga shallow linguistic kernel for drug–drug interaction extraction. Journal ofbiomedical informatics, 44(5):789–804.

Segura-Bedmar, I., P. Martınez, andM. Herrero-Zazo. 2013. SemEval-2013

task 9 : Extraction of drug-drug interac-tions from biomedical texts (DDIExtrac-tion 2013). In Second Joint Conferenceon Lexical and Computational Seman-tics (*SEM), Volume 2: Proceedings ofthe Seventh International Workshop onSemantic Evaluation (SemEval 2013),pages 341–350, Atlanta, Georgia, USA,June. Association for ComputationalLinguistics.

Segura Bedmar, I., P. Martinez, andD. Sanchez Cisneros. 2011. The 1stddiextraction-2011 challenge task: Ex-traction of drug-drug interactions frombiomedical texts. In Proceedings of the 1stChallenge Task on Drug-Drug InteractionExtraction 2011.

Suarez-Paniagua, V. and I. Segura-Bedmar.2018. Evaluation of pooling operations inconvolutional architectures for drug-druginteraction extraction. BMC bioinformat-ics, 19(8):209.

Sun, X., L. Ma, X. Du, J. Feng, and K. Dong.2018. Deep Convolution Neural Networksfor Drug-Drug Interaction Extraction. InIEEE International Conference on Bioin-formatics and Biomedicine (BIBM).

Thomas, P., M. Neves, T. Rocktaschel, andU. Leser. 2013. Wbi-ddi: drug-drug inter-action extraction using majority voting.In Second Joint Conference on Lexical andComputational Semantics (* SEM), Vol-ume 2: Proceedings of the Seventh Inter-national Workshop on Semantic Evalua-tion (SemEval 2013), pages 628–635.

Wishart, D. e. a. 2017. Drugbank 5.0: amajor update to the drugbank databasefor 2018. Nucleid Acird Research, 8.

Yi, Z., S. Li, J. Yu, and Q. Wu. 2017. Drug-drug Interaction Extraction via RecurrentNeural Network with Multiple AttentionLayers. In International Conference onAdvanced Data Mining and Applications.

Zaremba, W., I. Sutskever, and O. Vinyals.2015. Recurrent Neural Network Regular-ization. arXiv preprint arXiv:1409.2329,1.

Zheng, W., H. Lin, L. Luo, Z. Zhao,Z. Li, Y. Zhang, Z. Yang, and J. Wang.2017. An attention-based effective neuralmodel for drug-drug interactions extrac-tion. BMC Bioinformatics, 18.

Antonio Miranda-Escalada, Isabel Segura-Bedmar

76