Decision-support for the Masses by Enabling Conversations with Open Data

Biplav SrivastavaIBM

Abstract

Open data refers to data that is freely available for reuse.Although there has been rapid increase in availability ofopen data to public in the last decade, this has not trans-lated into better decision-support tools for them. Wepropose intelligent conversation generators as a grandchallenge that would automatically create data-drivenconversation interfaces (CIs), also known as chatbots ordialog systems, from open data and deliver personal-ized analytical insights to users based on their contex-tual needs. Such generators will not only help bring Ar-tificial Intelligence (AI)-based solutions for importantsocietal problems to the masses but also advance AI byproviding an integrative testbed for human-centric AIand filling gaps in the state-of-art towards this aim.

IntroductionAs the world has shifted towards an increased digital econ-omy and organizations have adopted the open data prin-ciples to make their data widely available for reuse (Her-man and Beeman 2012; Hughes and Rumsey 2018), thereis an unprecedented opportunity to generate insights to im-prove the conditions of people around the world towardsbasic concerns of living like health, water, energy, traf-fic, community and environment (Srivastava 2015). How-ever, the current situation is that available interfaces ofsearch 1 and visualization for open data 2 are targeted to-wards developers and are rudimentary when compared towhat people want - relevant, prescriptive, data-driven in-sights for taking decisions. Building intelligent interfaceswith open data needs specialization in AI and data manage-ment, and is costly and consequently, slow to develop. Thefirst glimpses can be seen in chatbot released by the state ofKansas (Bloomberg 2017) in the US while the private sectorhas tried to provide a natural question-answering interface(Sorkin 2017) to US data. The research community has alsostarted to build many prototypes in the area (Ellis et al. 2018;Kephart et al. 2018), and we seek to accelerate the phe-nomenon.

Copyright c© 2019, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

1https://toolbox.google.com/datasetsearch2Examples - http://data.gov, http://data.gov.in.

S.No. Dimension Variety1 User 1, multiple2 Modality only conversation, only speech,

multi-modal (with point, map, ...)3 Data source none, static, dynamic4 Personalized no, yes5 Form virtual agent, physical device, robot6 Purpose socialize, goal: information seeker,

goal: action delegate7 Domains general, health, water, traffic, ...

Table 1: Different Types of Conversation Interfaces

For people, natural interfaces to data have long beenknown to be effective. Previous attempts to build natural in-terfaces to data include building question answer systems di-rectly from data(Haug et al. 2017; Pasupat and Liang 2015;Yao and Van Durme 2014; Hao et al. 2017) and providingnatural interfaces to a query language(Androutsopoulos etal. 1995). But they support a one-off interaction with datawhile conversation is inherently iterative.

A conversation interface (CI), also known as chatbot ordialog system3 (McTear et al. 2016), is an automated agent,whether physical like robots or abstract like an avatar, thatcan not only interact with a person, but also take actions ontheir behalf and get things done. A simple taxonomy of in-terfaces we consider is shown in Table 1. One can talk toa chatbot or, if speech is not supported, type an input andget the systems response. They can be embedded along withother interaction modalities to give a rich user experience.The chatbot may converse without a goal in pleasantries andhence not need access to data sources, or be connected to astatic data source like a company directory or a dynamic datasource like weather forecast. The application scenarios be-come more compelling when the chatbot works in a dynamicenvironment, e.g., with sensor data, interacts with groups ofpeople who come and go rather than only an individual at atime, and adapts its behavior to peculiarities of user(s).

There has been a surge in availability of chatbots for peo-ple on mobile phones and physical devices like the Ama-

3Some researchers use the term chatbot exclusively for agentsthat perform chit-chat. Instead, we use the terms conversationagents, chatbots and dialog systems interchangeably to mean task-oriented conversation agents which is the focus of this paper.

arX

iv:1

809.

0672

3v2

[cs

.AI]

11

Jan

2019

zon Alexa and Google Home. Numerous platforms haveemerged to create them quickly for any domain (Accen-ture 2016). Chatbots have been deployed in customer care inmany industries where they are expected to save over $8 bil-lion per annum by 2022 (Juniper 2017). Chatbots can helpusers especially in unfamiliar domains when users do notknow everything about the data, its metadata, kinds of anal-yses possible and the implications of using the insights.

However, the process to build chatbots needs long devel-opment cycle and is costly. The main problem here is dialogmanagement, i.e., creating dialog responses to user’s utter-ances by trying to understand the user’s intent, determiningthe most suitable response, building queries to retrieve dataand deciding the next course of action (responding, seekingmore information or deferring). The chatbot so created alsohas to be tested for functional and non-function characteris-tics, and social behavior.

We envision a simpler and cost-efficient process wherethe user can point to a data-source like water quality andregulations, and gets a chatbot custom-generated so that theuser can ask whether they can safely drink a location’s wa-ter. The system could also explain its recommendation andjustify if new information is provided about water usage onother days or locations. The user can then point to a diseasedata-source and the chatbot automatically updates itself sothat it can now converse and answer questions about dis-eases, in general, but also water-based diseases for a specificlocation, in particular. Changing domains, the user can pointto travel data about events happening in their favorite citiesand get a new chatbot to advise on their upcoming vacation.Conversation agents in these and other domains are increas-ingly available but as noted earlier, they are costly and time-consuming to build from scratch.

We envisage software programs, that we call chatbot gen-erators, for generating conversation interfaces to deliverdata-driven insights and also become personalized over timeand data sources. The chatbot generator would be able toquickly adapt to new domains and data sources based ona user’s needs, generate chatbots that are broadly useful,trustable by being able to transparently explain their deci-sion process and aware of fairness issues, and able to deploychatbots widely in different forms (Table 1).

We now discuss open data, challenges in using them andsome use-cases where insights from them can help people.Next, we review how conversation can help address the chal-lenges and how the proposed chatbot generator can fill thegap for common usage patterns. We will use water as a case-study throughout to motivate how general people may ben-efit, where a prototypical multi-modal chatbot called WaterAdvisor (WA) was recently described (Ellis et al. 2018). Weidentify AI opportunities for learning, reasoning, represen-tation and execution, along with human-centric design andethics, to motivate more conversation applications.

Open Data and Its ChallengesOpen data is an important trend in governance and comput-ing over the last decade (Hughes and Rumsey 2018; W3C2018). According to Open Data Catalogs, there are over 550repositories, each with data items (called resources) ranging

from tens to thousands, spanning most areas of human en-deavor4. The Open Data Barometer (W3C 2018) prepareda report surveying 1,725 datasets from 15 different sectorsacross 115 countries. In the first wave of the trend, whichstarted around 2008, the focus was on acquiring data andmaking it available on scalable public platforms. In the sec-ond wave, the focus has been on driving consumption byproviding richer semantics (Wright et al. 2012).

Many use-cases have been published demonstrating howinsights from open data using AI methods can benefit peo-ple. For example, (Srivastava 2015) gives a tutorial focusingon value of new AI insights in a domain, availability of rel-evant open data and context of people’s interaction with the(new) system. Consider water as an example. People makemany daily decisions concerned with water involving pro-fession (e.g., fishing, irrigation, shipping), recreation (e.g,boating), wild life conservation (e.g., dolphins) or just reg-ular living (e.g., drinking, bathing, washing). If accessibletools were available to public, they would be particularlyuseful to handle public health challenges such as the Flintwater crisis (Pieper et al. 2017). The very few tools availabletoday target water experts such as WaterLive mobile app forAustralia 5, Bath app for UK6, and GangaWatch for India(Sandha et al. 2017) and assume technical understanding ofthe science behind water quality indicators.

Psychologists have long explored the sense-making pro-cess of analysts when looking at data through cognitive taskanalysis of their activities (Pirolli and Card 2004). Theyfound that analysts try to explore schema and other meta-data, look at data, build hypotheses and look for evidence.The current interfaces to access data are intended for devel-opers and data analysts. They consist of search interfaces onrepository sites (like data.gov) or via search engines 7; visu-alization8 and application programming interfaces (APIs).Further, for data analysis, analysts want to understand thecontext of data available, the standards and issues prevailingin a domain of their interest and a forum to discuss inter-disciplinary challenges. In fact, in (John et al. 2017), the au-thors have created a chatbot to help with data analysis steps.

But our focus is on how general public may benefit frominsights generated from open data without long developmentcycles and in context of their use. The proposed chatbot gen-erators will enable a new, complimentary, conversational, in-terface to open data that a user can interact with, possibly aspart of a multi-modal user experience. Thus, a user will beable to talk to an automated chatbot to get insight she wantswhile optionally also seeing additional, relevant, visualiza-tions and documents the agent may be able to retrieve.

Conversation InterfacesThere is a long history of Conversational interfaces (CIs)going back to the 1960s when they first appeared to do ca-sual conversation or answer questions (McTear et al. 2016).

4https://datacatalogs.org/, Accessed 28-Dec-2018.5http://www.water.nsw.gov.au/realtime-data6https://environment.data.gov.uk/bwq/profiles/7https://toolbox.google.com/datasetsearch8See examples at http://data.gov, http://data.gov.in.

Figure 1: The architecture of a data-driven chatbot.

A conversation, or dialog, is made up of a series of turns,where each turn is a series of utterances by one or moreparticipants playing one or more roles. A common type ofchatbot deals with a single user at a time and conducts infor-mal conversation, answers the user’s questions or providesrecommendations in a given domain. It needs to handle un-certainties related to human behavior and natural language,while conducting dialogs to achieve system goals.

Building Data-Consuming ChatbotsThe core problem in building chatbots is that of dialog man-agement, i.e., creating dialog responses to user’s utterances.The system architecture of a typical data-consuming dialogmanager (DM) is shown in Figure 1. Given the user’s utter-ance, it is analyzed to detect their intent and a policy forresponse is selected. This policy may call for querying adatabase, and the result is returned which is used by responsegenerator to create a response using templates. The systemcan dynamically create one or more queries which involvesselecting tables and attributes, filtering values and testing forconditions, and assuming defaults for missing values. It mayalso decide not to answer a request if it is unsure of a query’sresult correctness.

Note that the DM may use one or more domain-specificdata bases (sources) as well as one or more domain-independent sources like language models and word em-beddings. Common chatbots use static domain-dependentdatabases like product catalogs or user manuals. The appli-cation scenarios become more compelling when the chatbotworks in a dynamic environment, e.g., with sensor data, andinteracts with groups of people, who come and go, ratherthan only an individual at a time. In such situations, the agenthas to execute actions to monitor the environment, modeldifferent users engaged in conversation over time and tracktheir intents, learn patterns and represent them, reason aboutbest course of action given goals and system state, and exe-cute conversation or other multi-modal actions.

There are many approaches to tackle DM in literature in-

cluding finite-space, frame-based, inference-based and sta-tistical learning-based (Crook 2018; Clark et al. 2010; In-ouye 2004; Young et al. 2013), of which, finite-space andframe-based are most popular with mainstream developers.Task-oriented DMs have traditionally been built using rules.Further, a DM contains several independent modules whichare optimized separately, relying on huge amount of humanengineering . The recent trend is to train DM from end-to-end, allowing error signal from the end output of DM to beback-propagated to raw input, so that the whole DM can bejointly optimized (Bordes et al. 2017). A recent paper re-views the state-of-art and looks at requirements and designoptions to make them customizable for end users as theirown personal bot (Daniel et al. 2018).

There are also unique considerations when exploring datawith dialog:

• Dynamic source selection: the data in a domain may con-sist of multiple tables, attributes (columns) and rows. Theuser utterance could lead to discovering them and theythen become part of the context of query.

• Query patterns: there are often common patterns and natu-ral order of user queries. Users may adopt them to exploredata and can be used as a shared context.

• Query cost: the order of query execution could be impor-tant for cost reasons.

• Query mapping: mappings from natural language to datamodel may have to be learned and adapted based on dif-ferent source models.

• Conversation length: As conversations become long, therecould be increased risk of the user dropping off leading todiminishing returns.

Usability Opportunities and Issues with ChatbotsAn obvious question to ask when considering chatbots iswhen they are most suitable. The effectiveness of conversa-tion versus other modalities has long been studied (Frohlich

1993). Some scenarios where conversation is quite suitableinclude when users are unfamiliar with the domain, expectnon-human actors due to unique form of agent embodiment(e.g., robots) or prefer them (e.g., due to sensitivity of thesubject matter), and where content changes often and usersseek guidance (Srivastava 2017).

However, such systems can also be fraught with ethicalrisks. An extreme and anecdotal example was the Tay (Neffand Nagy 2017) system in 2016 that was designed to en-gage with people on open topics over Twitter and learn fromfeedback, but ended up getting manipulated by users to ex-hibit unacceptable behavior via its extreme responses. Theauthors in (Henderson et al. 2018) systematically identify anumber of potential ethical issues in dialogue systems builtusing learning methods: showing implicit biases from data,being prone to adversarial examples, vulnerable to violateprivacy, need to maintain safety of people, and concernsabout explainability of response and reproducibility of re-sults. To handle these concerns, more research is needed.One idea here is to augment open data repositories that usu-ally consist of standard information like data and size, usagelicense (context), how data was obtained (provenance), se-mantics of missing values and units, and a responsible per-son. Researchers have proposed to further describe protectedvariables and fairness considerations as datasheets (Gebruet al. 2018). The metadata can be used along with recenttechniques and tools9 to address issues of fairness and trans-parency with chatbots (Henderson et al. 2018) and buildtrust with users.

On Use-Cases With Open Data And DialogsOver the years, a number of common analysis patterns haveemerged across domains which have been shown to be use-ful to people around the world (Srivastava 2015). One pat-tern is Return of Investment. Here, the monetary investmentmade into a domain is compared against suitable metricsof outcome and improvement is sought. As example, pub-lic funds invested into water work may be analyzed to seereduction in cases of heavy metal contamination, health caremay be analyzed to see reduction in number of patients anddeaths, or funds invested for tourist promotion be comparedwith increase in economic activity in a city. Another pat-tern is Comparison of Results. Here, if an improvement isfound in one context like domain, region or time, the useris interested to find whether the improvement holds for an-other similar context. Another pattern is Demand-SupplyMismatch where demand of a service/ resource like emer-gency visit in a city is compared with supply like health-careprofessionals. These could be good starting points for auto-mated chatbot generators while technical experts focus oncomplex domain-dependent decision situations.

Now consider a complex decision scenario. To make adecision on water consumption, one needs to consider theactivity (purpose) for water use; relevant water quality pa-rameters and their applicable regulatory standards for safety;available measurement technology, process, skills and costs;and actual data. These thus offer opportunities to integrate

9AI Fairness Toolkit - https://github.com/IBM/AIF360

Figure 2: A screenshot of Water Advisor. See video of it inaction at https://youtu.be/z4x44sxC3zA.

data from multiple sources and reason in the context of per-son’s interest. There are further complicating factors: theremay be overlapping regulations due to geography and ad-ministrative scope; one may have to account for alternativeways to measure a particular water quality parameter thatevolves over time; and water data can have issues like miss-ing values or different levels of granularity. Therefore, use-cases like water which do not follow common patterns needto be tackled outside of the generator.

AI Methods in Multi-Modal Conversation - Waterand BeyondWe now illustrate how different AI methods came togetherto build a multi-modal chatbot like Water Advisor (WA) (El-lis et al. 2018) and highlight how they would be relevant fora chatbot generator that works on common usage patterns.There are also challenges for generalization which createopportunities for further research. WA is intended to be adata-driven assistant that can guide people globally withoutrequiring any special water expertise. One can trigger it via aconversation to get an overview of water condition at a loca-tion, explore it by filtering and zooming on a map, and seekdetails on demand (Figure 2) by exploring relevant regula-tions, data or other locations.

Learning plays an important role in understanding user’sutterances, selecting reliable data sources and improvingoverall performance over time. Specific to water, it is alsoused to discovering issues in water quality together with reg-ulation data. More generally, learning can be used for alter-native DM approaches like end-to-end policy learning fromdata (Bordes et al. 2017).

Representation is needed to model location, time anduser. For water, it encodes regulation and safe limits andmapping of usage purpose to quality parameters.

Reasoning is crucial to keep conversation focused basedon system usability goals and user needs. One can modelcognitive costs to user based on alternative system responsechoices and seek to optimize short-term and long-term be-havior. For water, reasoning is used to short-list regulations

based on water activity and region of interest, generate ad-vice and track explanations.

Execution is autonomous as the agent can choose to actby (a) asking clarifying questions about user intent or loca-tions, (b) asking user’s preference about advice, (c) seekingmost reliable data source (water) for region and time inter-val of interest from available external data sources, and cor-responding subset of compatible other sources (regulations)(d) invoking reasoning to generate an advice (for water us-age using filtered water data and regulations), (e) visualiz-ing its output and advice, and (f) using one or more suitablemodalities available at any turn of user interaction, i.e., chat,maps and document views.

Human Usability Factors have to be explicitly mod-eled and supported. In WA, the controller module for user-interface automatically keeps the different modalities syn-chronized and is aware of missing data or assumptions it ismaking, so that they can be used in system response. Fur-ther extensions can be to measure and track complexity ofinteraction (Liao et al. 2017) and use sensed signals to pro-actively improve user experience and combine close-endedand open-ended questioning strategies for efficient interac-tion (Zhang et al. 2018).

Ethical Issues can emerge whenever a piece of technologyis used among people at large. We discussed handling themin a domain independent manner earlier. There may also use-cases needing domain-specific considerations due to whichscope of chatbot generator has to be selectively expanded.

ConclusionIn this paper, we proposed the challenges of intelligent con-versation interface generator that, given a set of open datasources of interest, would generate a chatbot that can inter-act autonomously with a common person (non-developer)and provide insights about selected data. Such a technologywould bring Artificial Intelligence (AI)-based solutions forimportant societal problems to the masses along commonpatterns of usage while technical AI experts focus on spe-cialized cases. It will also serve as an integrative testbed forhuman-centric AI and advance AI sub-areas.

ReferencesAccenture. Chatbots in customer service. In At:https://accntu.re/2z9s5fH, 2016.Ion Androutsopoulos, Graeme D. Ritchie, and PeterThanisch. Natural language interfaces to databases - an in-troduction. CoRR, cmp-lg/9503016, 1995.Bloomberg. Chatbot makes open data user-friendly.In https://medium.com/@BloombergCities/chatbot-makes-open-data-user-friendly-c1888c863db4, 2017.Antoine Bordes, Y-Lan Boureau, and Jason Weston. Learn-ing end-to-end goal-oriented dialog. In Proc. ICLR, 2017.Alexander Clark, Chris Fox, and Shalom Lappin. Handbookof computation linguistics and natural language processing.In Wiley, ISBN: 978-1-405-15581-6, 2010.

Paul Crook. Statistical machine learning fordialog management: its history and futurepromise. In AAAI DEEP-DIAL 2018 Work-shop, at https://www.dropbox.com/home/AAAI2018 -DEEPDIALWorkshop/Presentations-Shareable?preview=Invited1-PaulCrook-AAAI DeepDialog Feb2018.pdf, 2018.Florian Daniel, Maristella Matera, Vittorio Zaccaria, andAlessandro Dell’Orto. Toward truly personal chatbots: Onthe development of custom conversational assistants. In Pro-ceedings of the 1st International Workshop on Software En-gineering for Cognitive Services, SE4COG ’18, pages 31–36, New York, NY, USA, 2018. ACM.Jason Ellis, Biplav Srivastava, Rachel Bellamy, and AndyAaron. Water advisor - a data-driven, multi-modal, contex-tual assistant to help with water usage decisions. In Proc.32nd AAAI Conference on Artificial Intelligence (AAAI-18),New Orleans, Lousiana, USA., 2018.David Frohlich. The history and future of di-rect manipulation. In HP Tech Report, http://www.hpl.hp.com/techreports/93/HPL-93-47.pdf, 1993.Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jen-nifer Wortman Vaughan, Hanna M. Wallach, Hal Daume III,and Kate Crawford. CoRR, abs/1803.09010, 2018.Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He,Zhanyi Liu, Hua Wu, and Jun Zhao. An end-to-end modelfor question answering over knowledge base with cross-attention combining global knowledge. In Proceedings ofthe 55th Annual Meeting of the Association for Computa-tional Linguistics (Volume 1: Long Papers), pages 221–231.Association for Computational Linguistics, 2017.Till Haug, Octavian-Eugen Ganea, and Paulina Grnarova.Neural multi-step reasoning for question answering on semi-structured tables. CoRR, abs/1702.06589, 2017.Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier,Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and JoellePineau. Ethical challenges in data-driven dialogue systems.In Proc. of AAAI/ACM Conference on AI Ethics and Society(AIES-18), New Orleans, Lousiana, USA., 2018.I. Herman and H. Beeman. Open data in practice. In W3CTutorial Track, 21st International World Wide Web Confer-ence (WWW2012), Lyon, France, 2012.Adam Hughes and Matt Rumsey. The state of theunion for open data. In Data Foundation Report,At http://www.datafoundation.org/the-state-of-the-union-of-open-data-ed-2/,, 2018.R. Bryce Inouye. Minimizing the length of non-mixed ini-tiative dialogs. In Daniel Midgley Leonoor van der Beek,Dmitriy Genzel, editor, ACL 2004: Student Research Work-shop, pages 7–12, Barcelona, Spain, July 2004. Associationfor Computational Linguistics.Rogers Jeffrey Leo John, Navneet Potti, and Jignesh M. Pa-tel. Ava: From data to insights through conversations. InCIDR, 2017.Juniper. Chatbots: Retail, ecommerce, banking & healthcare2017-2022. In At: http://bit.ly/2sJHejY, 2017.

Jeffrey Kephart, Victor Dibia, Jason Ellis, Biplav Srivastava,Kartik Talamadupula, and Mishal Dholakia. Cognitive assis-tant for visualizing and analyzing exoplanets. In Proc. 32ndAAAI Conference on Artificial Intelligence (AAAI-18), NewOrleans, Lousiana, USA., 2018.Q. Liao, B. Srivastava, and P. Kapanipathi. A Measure forDialog Complexity and its Application in Streamlining Ser-vice Operations. ArXiv e-prints, August 2017.M. McTear, Z. Callejas, and D. Griol. Conversationalinterfaces: Past and present. In The Conversational In-terface. Springer, DOI: https://doi.org/10.1007/978-3-319-32967-3 4, 2016.G. Neff and P. Nagy. Automation, algorithms, and politics—talking to bots: Symbiotic agency and the case of tay. InInternational Journal Of Communication, 10, 17. Retrievedfrom http://ijoc.org/index.php, 2017.Panupong Pasupat and Percy Liang. Compositional seman-tic parsing on semi-structured tables. Proc. ACL, 2015.Kelsey J. Pieper, Min Tang, and Marc A. Edwards. Flintwater crisis caused by interrupted corrosion control: Investi-gating ”ground zero” home. Environmental Science & Tech-nology, 51(4):2007–2014, 2017.P. Pirolli and S. Card. The sensemaking process and leveragepoints for analyst technology as identified through cognitivetask analysis. In Proceedings of International Conferenceon Intelligence Analysis, 2004.Sandeep Singh Sandha, Biplav Srivastava, and SukanyaRandhawa. The gangawatch mobile app to enable usage ofwater data in every day decisions integrating historical andreal-time sensing data. CoRR, abs/1701.08212, 2017.Andrew Ross Sorkin. Steve ballmerserves up a fascinating data trove. Inhttps://www.nytimes.com/2017/04/17/business/dealbook/steve-ballmer-serves-up-a-fascinating-data-trove.html, 2017.Biplav Srivastava. Ai for smart city innovations with opendata. In Tutorial at International Joint Conference onArtificial Intelligence (IJCAI), at Buenos Aires, Argentina.Details: https://sites.google.com/site/aismartcitytutorial/,2015.B. Srivastava. Designing talking alecksthat look smart and make users happy. Inhttps://www.linkedin.com/pulse/designing-talking-alecks-look-smart-make-users-happy-srivastava/, 2017.W3C. Open data barometer, 4th ed. In Athttps://opendatabarometer.org/4thedition/report/, 2018.Glover Wright, Pranesh Prakash, Sunil Abraham, and Nis-hant Shah. Open data in india. In CIS Report at http://cis-india.org/openness/publications/ogd-report, 2012.Xuchen Yao and Benjamin Van Durme. Information extrac-tion over structured data: Question answering with freebase.In Proceedings of the 52nd Annual Meeting of the Associa-tion for Computational Linguistics (Volume 1: Long Papers),pages 956–966. Association for Computational Linguistics,2014.Steve Young, Milica Gasic, Blaise Thomson, and Jason D

Williams. Pomdp-based statistical spoken dialog systems: Areview. Proceedings of the IEEE, 101(5):1160–1179, 2013.Yunfeng Zhang, Vera Liao, and Biplav Srivastava. Towardsan optimal dialog strategy for information retrieval usingboth open-ended and close-ended questions. In Proc. In-telligent User Interfaces (IUI 2018, Tokyo, Japan, March.,2018.