Proceedings of the 2019 EMNLP and the 9th IJCNLP (System Demonstrations), pages 181–186Hong Kong, China, November 3 – 7, 2019. c©2019 Association for Computational Linguistics

181

PolyResponse: A Rank-based Approach to Task-Oriented Dialogue withApplication in Restaurant Search and Booking

Matthew Henderson, Ivan Vulić, Iñigo Casanueva, Paweł Budzianowski, Daniela Gerz,

Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, Pei-Hao Su

PolyAI Limited, London, UKpoly-ai.com

Abstract

We present PolyResponse, a conversationalsearch engine that supports task-oriented dia-logue. It is a retrieval-based approach that by-passes the complex multi-component designof traditional task-oriented dialogue systemsand the use of explicit semantics in the formof task-specific ontologies. The PolyResponseengine is trained on hundreds of millions ofexamples extracted from real conversations: itlearns what responses are appropriate in dif-ferent conversational contexts. It then ranks alarge index of text and visual responses accord-ing to their similarity to the given context, andnarrows down the list of relevant entities dur-ing the multi-turn conversation. We introducea restaurant search and booking system pow-ered by the PolyResponse engine, currentlyavailable in 8 different languages.

1 Introduction and Background

Task-oriented dialogue systems are primarily de-signed to search and interact with large databaseswhich contain information pertaining to a certaindialogue domain: the main purpose of such sys-tems is to assist the users in accomplishing a well-defined task such as flight booking (El Asri et al.,2017), tourist information (Henderson et al., 2014),restaurant search (Williams, 2012), or booking ataxi (Budzianowski et al., 2018). These systemsare typically constructed around rigid task-specificontologies (Henderson et al., 2014; Mrkšić et al.,2015) which enumerate the constraints the userscan express using a collection of slots (e.g., PRICERANGE for restaurant search) and their slot values(e.g., CHEAP, EXPENSIVE for the aforementionedslots). Conversations are then modelled as a se-quence of actions that constrain slots to particularvalues. This explicit semantic space is manuallyengineered by the system designer. It serves asthe output of the natural language understandingcomponent as well as the input to the language

generation component both in traditional modularsystems (Young, 2010; Eric et al., 2017) and inmore recent end-to-end task-oriented dialogue sys-tems (Wen et al., 2017; Li et al., 2017; Bordes et al.,2017; Budzianowski et al., 2018, inter alia).

Working with such explicit semantics for task-oriented dialogue systems poses several criticalchallenges on top of the manual time-consumingdomain ontology design. First, it is difficult tocollect domain-specific data labelled with explicitsemantic representations. As a consequence, de-spite recent data collection efforts to enable trainingof task-oriented systems across multiple domains(El Asri et al., 2017; Budzianowski et al., 2018),annotated datasets are still few and far between, aswell as limited in size and the number of domainscovered.1 Second, the current approach constrainsthe types of dialogue the system can support, re-sulting in artificial conversations, and breakdownswhen the user does not understand what the systemcan and cannot support. In other words, traininga task-based dialogue system for voice-controlledsearch in a new domain always implies the com-plex, expensive, and time-consuming process ofcollecting and annotating sufficient amounts of in-domain dialogue data.

In this paper, we present a demo system based onan alternative approach to task-oriented dialogue.Relying on non-generative response retrieval wedescribe the PolyResponse conversational searchengine and its application in the task of restau-rant search and booking. The engine is trained on

1For instance, the recently published MultiWOZ dataset(Budzianowski et al., 2018) contains a total of 115,424 dia-logue turns scattered over 7 target domains. Other standardtask-based datasets are typically single-domain and by severalorders of magnitude smaller: DSTC2 (Henderson et al., 2014)contains 23,354 turns, Frames (El Asri et al., 2017) 19,986turns, and M2M (Shah et al., 2018) spans 14,796 turns. On theother hand, the Reddit corpus which supports our system com-prises 3.7B comments spanning a multitude of topics, dividedinto 256M (Reddit) conversational threads and generating727M context-reply pairs.

https://poly-ai.com/

182

hundreds of millions of real conversations from ageneral domain (i.e., Reddit), using an implicit rep-resentation of semantics that directly optimizes thetask at hand. It learns what responses are appropri-ate in different conversational contexts, and conse-quently ranks a large pool of responses accordingto their relevance to the current user utterance andprevious dialogue history (i.e., dialogue context).

The technical aspects of the underlying conver-sational search engine are explained in detail in ourrecent work (Henderson et al., 2019b), while thedetails concerning the Reddit training data are alsoavailable in another recent publication (Hendersonet al., 2019a). In this demo, we put focus on theactual practical usefulness of the search engine bydemonstrating its potential in the task of restau-rant search, and extending it to deal with multi-modal data. We describe a PolyReponse systemthat assists the users in finding a relevant restaurantaccording to their preference, and then addition-ally helps them to make a booking in the selectedrestaurant. Due to its retrieval-based design, withthe PolyResponse engine there is no need to en-gineer a structured ontology, or to solve the dif-ficult task of general language generation. Thisdesign also bypasses the construction of dedicateddecision-making policy modules. The large rank-ing model already encapsulates a lot of knowledgeabout natural language and conversational flow.

Since retrieved system responses are presentedvisually to the user, the PolyResponse restaurantsearch engine is able to combine text responseswith relevant visual information (e.g., photos fromsocial media associated with the current restau-rant and related to the user utterance), effectivelyyielding a multi-modal response. This setup ofusing voice as input, and responding visually isbecoming more and more prevalent with the riseof smart screens like Echo Show and even mixedreality. Finally, the PolyResponse restaurant searchengine is multilingual: it is currently deployed in8 languages enabling search over restaurants in8 cities around the world. System snapshots infour different languages are presented in Figure 2,while screencast videos that illustrate the dialogueflow with the PolyResponse engine are available at:https://tinyurl.com/y3evkcfz.

2 PolyResponse: Conversational Search

The PolyResponse system is powered by a singlelarge conversational search engine, trained on a

conversationalcontext

conversationalreply

photo captions

photos

Reddit727M (context, reply) pairs

Yelp200K (image, caption) pairs

Figure 1: The PolyResponse ranking model: it encodesconversational contexts, replies, and photos to respec-tive vectors hc, hr, and hp.

large amount of conversational and image data, asshown in Figure 1. In simple words, it is a rankingmodel that learns to score conversational repliesand images in a given conversational context. Thehighest-scoring responses are then retrieved as sys-tem outputs. The system computes two sets ofsimilarity scores: 1) S(r, c) is the score of a candi-date reply r given a conversational context c, and2) S(p, c) is the score of a candidate photo p givena conversational context c. These scores are com-puted as a scaled cosine similarity of a vector thatrepresents the context (hc), and a vector that repre-sents the candidate response: a text reply (hr) ora photo (hp). For instance, S(r, c) is computed asS(r, c) = Ccos(hr, hc), where C is a learned con-stant. The part of the model dealing with text input(i.e., obtaining the encodings hc and hr) followsthe architecture introduced recently by Hendersonet al. (2019b). We provide only a brief recap here;see the original paper for further details.

Text Representation. The model, implementedas a deep neural network, learns to respond bytraining on hundreds of millions context-reply (c, r)pairs. First, similar to Henderson et al. (2017), rawtext from both c and r is converted to unigramsand bigrams. All input text is first lower-casedand tokenised, numbers with 5 or more digits gettheir digits replaced by a wildcard symbol #, whilewords longer than 16 characters are replaced by awildcard token LONGWORD. Sentence boundarytokens are added to each sentence. The vocabularyconsists of the unigrams that occur at least 10 timesin a random 10M subset of the Reddit training set(see Figure 1) plus the 200K most frequent bigramsin the same random subset.

https://tinyurl.com/y3evkcfz

183

During training, we obtain d-dimensional featurerepresentations (d = 320) shared between contextsand replies for each unigram and bigram jointlywith other neural net parameters.2 A state-of-the-art architecture based on transformers (Vaswaniet al., 2017) is then applied to unigram and bigramvectors separately, which are then averaged to formthe final 320-dimensional encoding. That encod-ing is then passed through three fully-connectednon-linear hidden layers of dimensionality 1, 024.The final layer is linear and maps the text into thefinal l-dimensional (l = 512) representation: hcand hr. Other standard and more sophisticatedencoder models can also be used to provide finalencodings hc and hr, but the current architectureshows a good trade-off between speed and efficacywith strong and robust performance in our empiri-cal evaluations on the response retrieval task usingReddit (Al-Rfou et al., 2016), OpenSubtitles (Li-son and Tiedemann, 2016), and AmazonQA (Wanand McAuley, 2016) conversational test data, see(Henderson et al., 2019a) for further details.3

In training the constant C is constrained to liebetween 0 and

√l.4 Following Henderson et al.

(2017), the scoring function in the training ob-jective aims to maximise the similarity score ofcontext-reply pairs that go together, while minimis-ing the score of random pairings: negative exam-ples. Training proceeds via SGD with batches com-prising 500 pairs (1 positive and 499 negatives).

Photo Representation. Photos are representedusing convolutional neural net (CNN) models pre-trained on ImageNet (Deng et al., 2009). Weuse a MobileNet model with a depth multiplierof 1.4, and an input dimension of 224 × 224 pix-els as in (Howard et al., 2017).5 This provides a1, 280× 1.4 = 1, 792-dimensional representationof a photo, which is then passed through a singlehidden layer of dimensionality 1, 024 with ReLUactivation, before being passed to a hidden layer ofdimensionality 512 with no activation to providethe final representation hp.

2The model deals with out-of-vocabulary unigrams andbigrams by assigning a random id from 0 to 50,000 to each;this is then used to look up their embedding.

3The comparisons of performance in the response retrievaltask are also available online at: https://github.com/PolyAI-LDN/conversational-datasets/.

4It is initialised to a random value between 0.5 and 1, andinvariably converges to

√l by the end of training. Empirically,

this helps with learning.5The pretrained model downloaded from TensorFlow Slim.

Data Source 1: Reddit. For training text repre-sentations we use a Reddit dataset similar to Al-Rfou et al. (2016). Our dataset is large and providesnatural conversational structure: all Reddit datafrom January 2015 to December 2018, availableas a public BigQuery dataset, span almost 3.7Bcomments (Henderson et al., 2019a). We prepro-cess the dataset to remove uninformative and longcomments by retaining only sentences containingmore than 8 and less than 128 word tokens. Afterpairing all comments/contexts c with their repliesr, we obtain more than 727M context-reply (c, r)pairs for training, see Figure 1.

Data Source 2: Yelp. Once the text encod-ing sub-networks are trained, a photo encoder islearned on top of a pretrained MobileNet CNN,using data taken from the Yelp Open dataset:6 itcontains around 200K photos and their captions.Training of the multi-modal sub-network then max-imises the similarity of captions encoded with theresponse encoder hr to the photo representation hp.As a result, we can compute the score of a photogiven a context using the cosine similarity of therespective vectors. A photo will be scored highly ifit looks like its caption would be a good responseto the current context.7

Index of Responses. The Yelp dataset is used atinference time to provide text and photo candidatesto display to the user at each step in the conversa-tion. Our restaurant search is currently deployedseparately for each city, and we limit the responsesto a given city. For instance, for our English systemfor Edinburgh we work with 396 restaurants, 4,225photos (these include additional photos obtained us-ing the Google Places API without captions), 6,725responses created from the structured informationabout restaurants that Yelp provides, converted us-ing simple templates to sentences of the form suchas “Restaurant X accepts credit cards.”, 125,830sentences extracted from online reviews.

PolyResponse in a Nutshell. The system jointlytrains two encoding functions (with shared wordembeddings) f(context) and g(reply) which pro-duce encodings hc and hr, so that the similarity

6https://www.yelp.com/dataset7Note that not all of the Yelp dataset has captions, which

is why we need to learn the photo representation. If a photocaption is available, then the response vector representation ofthe caption is averaged with the photo vector representationto compute the score. If a caption is not available at inferencetime, we use only the photo vector representation.

https://github.com/PolyAI-LDN/conversational-datasets/https://github.com/PolyAI-LDN/conversational-datasets/https://www.yelp.com/dataset

184

S(c, r) is high for all (c, r) pairs from the Reddittraining data and low for random pairs. The encod-ing function g() is then frozen, and an encodingfunction t(photo) is learnt which makes the similar-ity between a photo and its associated caption highfor all (photo, caption) pairs from the Yelp dataset,and low for random pairs. t is a CNN pretrainedon ImageNet, with a shallow one-layer DNN ontop. Given a new context/query, we then provideits encoding hc by applying f(), and find plausi-ble text replies and photo responses according tofunctions g() and t(), respectively. These shouldbe responses that look like answers to the query,and photos that look like they would have captionsthat would be answers to the provided query.

At inference, finding relevant candidates given acontext reduces to computing hc for the context c ,and finding nearby hr and hp vectors. The responsevectors can all be pre-computed, and the nearestneighbour search can be further optimised usingstandard libraries such as Faiss (Johnson et al.,2017) or approximate nearest neighbour retrieval(Malkov and Yashunin, 2016), giving an efficientsearch that scales to billions of candidate responses.

The system offers both voice and text input andoutput. Speech-to-text and text-to-speech conver-sion in the PolyResponse system is currently sup-ported by the off-the-shelf Google Cloud tools.8

3 Dialogue Flow

The ranking model lends itself to the one-shot taskof finding the most relevant responses in a givencontext. However, a restaurant-browsing systemneeds to support a dialogue flow where the userfinds a restaurant, and then asks questions aboutit. The dialogue state for each search scenario isrepresented as the set of restaurants that are consid-ered relevant. This starts off as all the restaurantsin the given city, and is assumed to monotonicallydecrease in size as the conversation progresses untilthe user converges to a single restaurant. A restau-rant is only considered valid in the context of a newuser input if it has relevant responses correspondingto it. This flow is summarised here:

S1. Initialise R as the set of all restaurants in thecity. Given the user’s input, rank all the responsesin the response pool pertaining to restaurants in R.

S2. Retrieve the top N responses r1, r2, . . . , rNwith corresponding (sorted) cosine similarity

8https://cloud.google.com/speech-to-text/;https://cloud.google.com/text-to-speech/

scores: s1 ≥ s2 ≥ . . . ≥ sN .S3. Compute probability scores pi ∝ exp(a · si)with

∑Ni=1 pi, where a > 0 is a tunable constant.

S4. Compute a score qe for each restaurant/entitye ∈ R, qe =

∑i:ri∈e pi.

S5. Update R to the smallest set of restaurants withhighest q whose q-values sum up to more than apredefined threshold t.

S6. Display the most relevant responses associatedwith the updated R, and return to S2.

If there are multiple relevant restaurants, oneresponse is shown from each. When only onerestaurant is relevant, the top N responses are allshown, and relevant photos are also displayed. Thesystem does not require dedicated understanding,decision-making, and generation modules, and thisdialogue flow does not rely on explicit task-tailoredsemantics. The set of relevant restaurants is keptinternally while the system narrows it down acrossmultiple dialogue turns. A simple set of predefinedrules is used to provide a templatic spoken systemresponse: e.g., an example rule is “One review ofe said r”, where e refers to the restaurant, and rto a relevant response associated with e. Note thatwhile the demo is currently focused on the restau-rant search task, the described “narrowing down”dialogue flow is generic and applicable to a varietyof applications dealing with similar entity search.

The system can use a set of intent classifiers toallow resetting the dialogue state, or to activate theseparate restaurant booking dialogue flow. Theseclassifiers are briefly discussed in §4.

4 Other Functionality

Multilinguality. The PolyResponse restaurantsearch is currently available in 8 languages andfor 8 cities around the world: English (Edinburgh),German (Berlin), Spanish (Madrid), Mandarin(Taipei), Polish (Warsaw), Russian (Moscow), Ko-rean (Seoul), and Serbian (Belgrade). Selectedsnapshots are shown in Figure 2, while we also pro-vide videos demonstrating the use and behaviourof the systems at: https://tinyurl.com/y3evkcfz. A simple MT-based translate-to-source approach at inference time is currently usedto enable the deployment of the system in otherlanguages: 1) the pool of responses in each lan-guage is translated to English by Google Translatebeforehand, and pre-computed encodings of theirEnglish translations are used as representations of

https://tinyurl.com/y3evkcfzhttps://tinyurl.com/y3evkcfz

185

(a) English system. City: Edinburgh. (b) French system. City: Paris.

(c) German system. City: Berlin. (d) Mandarin system. City: Taipei.

Figure 2: Snapshots of the PolyResponse demo system for restaurant search in four different languages. Restaurantnames are anonymised. Translations of non-English sentences are provided in parentheses; they are not part of thesystem output. The output also comprises relevant photos associated with the current restaurant.

Figure 3: An example showing how the system can re-trieve parts of the menu as responses to the current userutterance (if they are relevant to the utterance).

each foreign language response; 2) a provided userutterance (i.e., context) is translated to English on-the-fly and its encoding hc is then learned. We planto experiment with more sophisticated multilingualmodels in future work.

Voice-Controlled Menu Search. An additionalfunctionality enables the user to get parts of therestaurant menu relevant to the current user utter-ance as responses. This is achieved by performingan additional ranking step of available menu itemsand retrieving the ones that are semantically rele-vant to the user utterance using exactly the samemethodology as with ranking other responses. Anexample of this functionality is shown in Figure 3.

Resetting and Switching to Booking. Therestaurant search system needs to support the dis-crete actions of restarting the conversation (i.e.,resetting the set R), and should enable transfer-ring to the slot-based table booking flow. This isachieved using two binary intent classifiers, thatare run at each step in the dialogue. These classi-fiers make use of the already-computed hc vectorthat represents the user’s latest text. A single-layerneural net is learned on top of the 512-dimensionalencoding, with a ReLU activation and 100 hiddennodes.9 To train the classifiers, sets of 20 rele-vant paraphrases (e.g., “Start again”) are providedas positive examples. Finally, when the systemsuccessfully switches to the booking scenario, itproceeds to the slot filling task: it aims to extractall the relevant booking information from the user(e.g., date, time, number of people to dine). The en-tire flow of the system illustrating both the searchphase and the booking phase is provided as thesupplemental video material.

5 Conclusion and Future Work

This paper has presented a general approach tosearch-based dialogue that does not rely on explicit

9Using the Reddit encoding has shown better generalisa-tion when compared to models learned from scratch. This fol-lows a recent trend where small robust classifiers are learnedon pretrained large models (Devlin et al., 2018).

186

semantic representations such as dialogue acts orslot-value ontologies, and allows for multi-modalresponses. In future work, we will extend the cur-rent demo system to more tasks and languages, andwork with more sophisticated encoders and rank-ing functions. Besides the initial dialogue flowfrom this work (§3), we will also work with morecomplex flows dealing, e.g., with user intent shifts.

ReferencesRami Al-Rfou, Marc Pickett, Javier Snaider, Yun-

Hsuan Sung, Brian Strope, and Ray Kurzweil. 2016.Conversational contextual cues: The case of person-alization and history for response ranking. CoRR,abs/1606.00372.

Antoine Bordes, Y.-Lan Boureau, and Jason Weston.2017. Learning end-to-end goal-oriented dialog. InICLR.

Paweł Budzianowski, Tsung-Hsien Wen, Bo-HsiangTseng, Iñigo Casanueva, Stefan Ultes, Osman Ra-madan, and Milica Gašić. 2018. MultiWOZ - Alarge-scale multi-domain wizard-of-oz dataset fortask-oriented dialogue modelling. In EMNLP, pages5016–5026.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, KaiLi, and Fei-Fei Li. 2009. ImageNet: A large-scalehierarchical image database. In CVPR.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2018. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. CoRR, abs/1810.04805.

Layla El Asri, Hannes Schulz, Shikhar Sharma,Jeremie Zumer, Justin Harris, Emery Fine, RahulMehrotra, and Kaheer Suleman. 2017. Frames: Acorpus for adding memory to goal-oriented dialoguesystems. In SIGDIAL, pages 207–219.

Mihail Eric, Lakshmi Krishnan, Francois Charette, andChristopher D. Manning. 2017. Key-value retrievalnetworks for task-oriented dialogue. In SIGDIAL,pages 37–49.

Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Ku-mar, Balint Miklos, and Ray Kurzweil. 2017. Effi-cient natural language response suggestion for smartreply. CoRR, abs/1705.00652.

Matthew Henderson, Pawel Budzianowski, IñigoCasanueva, Sam Coope, Daniela Gerz, GirishKumar, Nikola Mrkšić, Georgios Spithourakis,Pei-Hao Su, Ivan Vulić, and Tsung-Hsien Wen.2019a. A repository of conversational datasets. InNLP4ConvAI Workshop, pages 1–10.

Matthew Henderson, Blaise Thomson, and SteveYoung. 2014. Word-based dialog state tracking withrecurrent neural networks. In SIGDIAL.

Matthew Henderson, Ivan Vulić, Daniela Gerz, IñigoCasanueva, Paweł Budzianowski, Sam Coope,Georgios Spithourakis, Tsung-Hsien Wen, NikolaMrkšić, and Pei-Hao Su. 2019b. Training neural re-sponse selection for task-oriented dialogue systems.In ACL, pages 5392–5404.

Andrew G. Howard, Menglong Zhu, Bo Chen, DmitryKalenichenko, Weijun Wang, Tobias Weyand,Marco Andreetto, and Hartwig Adam. 2017.MobileNets: Efficient convolutional neural net-works for mobile vision applications. CoRR,abs/1704.04861.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017.Billion-scale similarity search with GPUs. arXivpreprint arXiv:1702.08734.

Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao,and Asli Celikyilmaz. 2017. End-to-end task-completion neural dialogue systems. In IJCNLP,pages 733–743.

Pierre Lison and Jörg Tiedemann. 2016. OpenSub-titles2016: Extracting large parallel corpora frommovie and TV subtitles. In LREC.

Yury A. Malkov and D. A. Yashunin. 2016. Efficientand robust approximate nearest neighbor searchusing Hierarchical Navigable Small World graphs.CoRR, abs/1603.09320.

Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thom-son, Milica Gašić, Pei-Hao Su, David Vandyke,Tsung-Hsien Wen, and Steve Young. 2015. Multi-domain dialog state tracking using recurrent neuralnetworks. In ACL, pages 794–799.

Pararth Shah, Dilek Hakkani-Tür, Bing Liu, andGokhan Tür. 2018. Bootstrapping a neural conversa-tional agent with dialogue self-play, crowdsourcingand on-line reinforcement learning. In NAACL-HLT,pages 41–51.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, LukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In NeurIPS, pages 6000–6010.

Mengting Wan and Julian McAuley. 2016. Modelingambiguity, subjectivity, and diverging viewpointsin opinion question answering systems. In ICDM,pages 489–498.

Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić,Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su,Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialoguesystem. In EACL, pages 438–449.

Jason D. Williams. 2012. A belief tracking challengetask for spoken dialog systems.

Steve Young. 2010. Still talking to machines (cogni-tively speaking). In INTERSPEECH, pages 1–10.

http://arxiv.org/abs/1606.00372http://arxiv.org/abs/1606.00372https://openreview.net/forum?id=S1Bb3D5gghttp://aclweb.org/anthology/D18-1547http://aclweb.org/anthology/D18-1547http://aclweb.org/anthology/D18-1547https://doi.org/10.1109/CVPRW.2009.5206848https://doi.org/10.1109/CVPRW.2009.5206848http://arxiv.org/abs/1810.04805http://arxiv.org/abs/1810.04805http://arxiv.org/abs/1810.04805http://aclweb.org/anthology/W17-5526http://aclweb.org/anthology/W17-5526http://aclweb.org/anthology/W17-5526http://aclweb.org/anthology/W17-5506http://aclweb.org/anthology/W17-5506http://arxiv.org/abs/1705.00652http://arxiv.org/abs/1705.00652http://arxiv.org/abs/1705.00652https://www.aclweb.org/anthology/W19-4101http://www.sigdial.org/workshops/conference15/proceedings/pdf/W14-4340.pdfhttp://www.sigdial.org/workshops/conference15/proceedings/pdf/W14-4340.pdfhttps://www.aclweb.org/anthology/P19-1536https://www.aclweb.org/anthology/P19-1536http://arxiv.org/abs/1704.04861http://arxiv.org/abs/1704.04861http://arxiv.org/abs/1702.08734http://aclweb.org/anthology/I17-1074http://aclweb.org/anthology/I17-1074http://www.lrec-conf.org/proceedings/lrec2016/summaries/947.htmlhttp://www.lrec-conf.org/proceedings/lrec2016/summaries/947.htmlhttp://www.lrec-conf.org/proceedings/lrec2016/summaries/947.htmlhttp://arxiv.org/abs/1603.09320http://arxiv.org/abs/1603.09320http://arxiv.org/abs/1603.09320http://aclweb.org/anthology/P15-2130http://aclweb.org/anthology/P15-2130http://aclweb.org/anthology/P15-2130http://aclweb.org/anthology/N18-3006http://aclweb.org/anthology/N18-3006http://aclweb.org/anthology/N18-3006http://papers.nips.cc/paper/7181-attention-is-all-you-needhttp://papers.nips.cc/paper/7181-attention-is-all-you-needhttps://doi.org/10.1109/ICDM.2016.0060https://doi.org/10.1109/ICDM.2016.0060https://doi.org/10.1109/ICDM.2016.0060http://www.aclweb.org/anthology/E17-1042http://www.aclweb.org/anthology/E17-1042http://www.aclweb.org/anthology/E17-1042https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/naaclhlt2012.pdfhttps://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/naaclhlt2012.pdfhttp://mi.eng.cam.ac.uk/~sjy/papers/youn10a.pdfhttp://mi.eng.cam.ac.uk/~sjy/papers/youn10a.pdf