Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
POLITECNICO DI MILANOMaster of Science in Computer Science and Engineering
Dipartimento di Elettronica, Informazione e Bioingegneria
A Conceptual Modeling Approach for
the Rapid Development of Chatbots
for Conversational Data Exploration
Supervisor: Prof. Maristella Matera
Co-supervisors:
Prof. Florian Daniel,
Prof. Vittorio Zaccaria
M.Sc. Thesis by:
Nicola Castaldo - 875019
Academic Year 2018-2019
Abstract
A chatbot is a a conversational software agent able to interact with user
through natural language, using voice or text channels and trying to emulate
human-to-human communication. This Thesis presents a framework for the
design of Chatbots for Data Exploration. With respect to conversational vir-
tual assistants (such as Amazon Alexa or Apple Siri), this class of chatbots
exploits structured input to retrieve data from known data sources. The ap-
proach is based on a conceptual representation of the available data sources,
and on a set of modeling abstractions that allow designers to characterize the
role that key data elements play in the user requests to be handled. Starting
from the resulting specifications, the framework then generates a conversa-
tion for exploring the content exposed by the considered data sources. The
contributes of this Thesis are:
• A characterization of Chatbots for Data Exploration, in terms of re-
quirements and goals that must be satisfied when designing these agents;
• A methodology to design chatbots which poses the attention on some
notable data elements, along with a technique to generate conversa-
tional agents based on annotations made directly on the schema of the
data source;
• An architecture for a framework supporting the design, generation and
execution of these chatbots exploiting state-of-art technologies;
• A prototype of the framework integrating different technologies for
chatbot deployment.
The Thesis also reports on some lessons learned that we derived from
a user study that compared the user performance and satisfaction with a
chatbot generated with our framework to the performance and satisfaction
of the same users when interacting the basic SQL command line.
I
Sommario
Il chatbot e un agente informatico capace di interagire con l’utente attraver-
so il linguaggio naturale, utilizzando canali vocali o testuali, nel tentativo
di emulare la comunicazione tipica dell’uomo. Questa Tesi descrive un fra-
mework per il design di Chatbot per l’Esplorazione di Dati. Rispetto agli as-
sistenti virtuali (come per esempio Alexa di Amazon o Siri di Apple), questa
classe di chatbot sfrutta richieste strutturate per estrarre dati da specifiche
sorgenti dati. L’approccio e basato su un insieme di astrazioni concettuali che
permettono ai progettisti di specificare il ruolo che alcuni elementi della base
di dati assumeranno nelle future richieste dell’utente. Sulla base di queste
specifiche, il framework genera dunque una conversazione per l’esplorazione
di tali dati. I contributi di questa Tesi sono:
• Una caratterizzazione dei Chatbot per l’Esplorazione di Dati, con par-
ticolare attenzione ai requisiti e agli obiettivi da considerare durante la
loro progettazione;
• Una metodologia per progettare tali chatbot che privilegia la tipologia
e il ruolo degli elementi che i dati rappresentano, oltre a una tecnica per
la generazione di agenti conversazionali basata su annotazioni effettuate
direttamente sullo schema dell base di dati;
• L’architettura di un framework in grado di supportare la progettazione,
la generazione e l’esecuzione di questi chatbot usando tecnologie allo
stato dell’arte;
• Un prototipo di questo sistema che integra tecnologie innovative.
La Tesi inoltre discute alcune implicazioni sul design di Chatbot per l’Esplo-
razione dei Dati che derivano da quanto osservato in uno studio con gli utenti.
Lo studio ha confrontato le prestazioni e la soddisfazione di un campione di
utenti che ha interrogato un database di esempio usando un chatbot generato
con il nostro framework e la linea di comando SQL.
III
Ringraziamenti
Ringrazio mia madre Cristina, mio padre Roberto e mia sorella Martina, per
esserci sempre stati e non aver mai smesso di credere in me. Grazie per aver-
mi insegnato valori come l’onesta e la cortesia e per avermi mostrato che con
l’impegno, la tenacia e la passione si possono raggiungere gli obiettivi piu
difficili. Ringrazio anche i miei zii, i miei cugini e le mie nonne, per l’affetto
che mi avete sempre dimostrato.
Un ringraziamento speciale, poi, va a Monica, per aver avuto cura di me
nei momenti felici e anche in quelli piu ardui, scegliendo sempre di viverli e
affrontarli assieme.
Vorrei inoltre ringraziare i miei amici, per i divertimenti e le difficolta che
abbiamo condiviso e per avermi sempre rispettato, apprezzato e voluto bene.
Ringrazio inoltre i professori Florian Daniel e Vittorio Zaccaria per il sup-
porto e i consigli che mi hanno dato durante tutta la stesura della Tesi. Un
grazie sincero va anche ai ragazzi del “TIS Lab”, che hanno accettato di par-
tecipare ai test di valutazione del sistema oggetto di questa Tesi.
Infine, un ringraziamento particolare va alla mia relatrice, la professoressa
Maristella Matera, per il tempo, l’energia e la fiducia che ha scelto di dedicare
a me e a questa Tesi.
V
Contents
Abstract I
Sommario III
Ringraziamenti V
1 Introduction 1
1.1 Scenario and Problem Statement . . . . . . . . . . . . . . . . 2
1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Structure of Thesis . . . . . . . . . . . . . . . . . . . . . . . . 5
2 State of the Art 7
2.1 The (R)evolution of Chatbots . . . . . . . . . . . . . . . . . . 7
2.1.1 The Loebner Prize . . . . . . . . . . . . . . . . . . . . 7
2.1.2 Virtual assistants . . . . . . . . . . . . . . . . . . . . . 8
2.1.3 A chatbot for everything . . . . . . . . . . . . . . . . . 8
2.2 Design Approach . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Development Choices . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.1 Local or remote . . . . . . . . . . . . . . . . . . . . . . 9
2.3.2 The chat interface . . . . . . . . . . . . . . . . . . . . . 10
2.4 Different Chatbot Frameworks . . . . . . . . . . . . . . . . . . 11
2.5 Related Research Approaches . . . . . . . . . . . . . . . . . . 13
3 Chatbots for Data Exploration 15
3.1 Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 Main Requirements . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2.1 Connect to the database to obtain the schema . . . . . 17
VII
3.2.2 Extract and learn database-specific vocabulary and ac-
tions from schema . . . . . . . . . . . . . . . . . . . . . 17
3.2.3 Automatic intent and entities generation and extraction 18
3.2.4 Proper communication and data visualization . . . . . 19
3.2.5 Allow the user to manipulate query results . . . . . . . 19
3.2.6 Support context and conversation history management 20
4 Approach 21
4.1 Design Process . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2 Schema Annotation . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2.1 Conversational Objects . . . . . . . . . . . . . . . . . . 22
4.2.2 Display Attributes . . . . . . . . . . . . . . . . . . . . 24
4.2.3 Conversational Qualifiers . . . . . . . . . . . . . . . . . 25
4.2.4 Conversational Types, Values and Operators . . . . . . 26
4.2.5 Conversational Relationships . . . . . . . . . . . . . . . 28
4.3 Conversation Modeling . . . . . . . . . . . . . . . . . . . . . . 28
4.3.1 Intents . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.3.2 Entities . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4 From the Conversation Model
to the Interaction . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.4.1 Interaction paradigm . . . . . . . . . . . . . . . . . . . 34
4.4.2 Automatic query generation . . . . . . . . . . . . . . . 35
4.4.3 Chatbot responses . . . . . . . . . . . . . . . . . . . . 42
4.4.4 History navigation . . . . . . . . . . . . . . . . . . . . 45
5 System Architecture and Implementation 47
5.1 Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.1.1 RASA . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.1.2 Chatito . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.1.3 Telegram . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.2 Framework Architecture . . . . . . . . . . . . . . . . . . . . . 53
5.3 Chatbot Architecture . . . . . . . . . . . . . . . . . . . . . . . 55
5.4 Implementation Tools . . . . . . . . . . . . . . . . . . . . . . . 58
6 User Study 61
6.1 Comparative Study: General Setting and Research Questions . 61
6.2 Participants and Design . . . . . . . . . . . . . . . . . . . . . 63
6.3 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
VIII
6.4 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.5 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.6 Usability Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.6.1 Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . 68
6.6.2 Effectiveness . . . . . . . . . . . . . . . . . . . . . . . . 68
6.7 Satisfaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6.7.1 Usability . . . . . . . . . . . . . . . . . . . . . . . . . . 70
6.7.2 Workload . . . . . . . . . . . . . . . . . . . . . . . . . 71
6.7.3 Easiness of exploring DB/retrieving data/using com-
mands and quality of presentation . . . . . . . . . . . . 72
6.7.4 User ranking of systems along completeness, easiness
and usefulness . . . . . . . . . . . . . . . . . . . . . . . 73
6.8 Qualitative Data Analysis . . . . . . . . . . . . . . . . . . . . 74
6.9 Lesson Learned . . . . . . . . . . . . . . . . . . . . . . . . . . 75
7 Conclusions 79
7.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
7.3 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
References 83
A Parsed Database Schema 87
B Database Schema Annotation 91
C User Study Questionnaires 103
C.1 Demographic Questionnaire . . . . . . . . . . . . . . . . . . . 103
C.2 System Questionnaire . . . . . . . . . . . . . . . . . . . . . . . 104
C.2.1 Ease of use . . . . . . . . . . . . . . . . . . . . . . . . 104
C.2.2 Cognitive load . . . . . . . . . . . . . . . . . . . . . . . 106
C.2.3 General questions . . . . . . . . . . . . . . . . . . . . . 107
C.3 Final Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . 108
IX
List of Figures
3.1 A Conversation example between a user and the chatbot. . . . 16
4.1 The relational schema of the database. . . . . . . . . . . . . . 23
4.2 The Conversational Objects annotation. . . . . . . . . . . . . 24
4.3 The Display Attributes annotation. . . . . . . . . . . . . . . . 25
4.4 The Conversational Qualifiers annotation, focus on customer. . 27
4.5 The Conversational Relationships annotation. . . . . . . . . . 29
4.6 An example of the entity extraction phase. . . . . . . . . . . . 32
4.7 The typo resolution procedure. . . . . . . . . . . . . . . . . . 42
4.8 The response of the chatbot to display a single result. . . . . . 44
4.9 The response of the chatbot to display multiple results. . . . . 44
4.10 An example of conversation history visualization. . . . . . . . 45
5.1 An overview of the Framework Architecture, based on the
three-tier model. . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.2 The Chatbot Architecture in details. . . . . . . . . . . . . . . 55
6.1 The classicmodels database schema used in the evaluation tests. 62
6.2 Comparison for the two systems of the number of successes . . 71
6.3 NASA-TLX dimensions: means and standard deviation com-
parison. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
XI
List of Tables
4.1 Phrase structure for find intents. . . . . . . . . . . . . . . . . 32
4.2 Phrase structure for filter intents. . . . . . . . . . . . . . . . . 33
4.3 Phrase structure for help intents. . . . . . . . . . . . . . . . . 33
6.1 Paired Sample t-test results related to the time the users spent
on each task. . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6.2 Detail on the completion of each task: Failures, Partial suc-
cess, Success . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
6.3 NASA-TLX dimensions values. . . . . . . . . . . . . . . . . . 72
XIII
List of Algorithms
1 Find Query Builder . . . . . . . . . . . . . . . . . . . . . . . . 36
2 Join Query Builder . . . . . . . . . . . . . . . . . . . . . . . . . 40
Listings
5.1 A json extract of the training data. . . . . . . . . . . . . . . . 49
5.2 An example of input document for Chatito. . . . . . . . . . . 51
5.3 A json extract of the simplified database. . . . . . . . . . . . . 58
5.4 A json extract of the database Schema Annotation. . . . . . . 59
XV
Chapter 1
Introduction
Chatbots are growing fast in number and pervade in a broad range of activ-
ities. Their natural language paradigm simplifies the interaction with appli-
cations to the point that experts consider chatbots one of the most promising
technologies to transform instant messaging systems into software delivery
platforms [15, 18]. The major messaging platforms have thus opened their
APIs to third-party developers, to expose high-level services (e.g., messag-
ing, payments, bot directory) and User Interface (UI) elements (e.g., buttons
and icons), to facilitate and promote the development of innovative services
based on conversational UIs [21].
Despite the huge emphasis on these new applications, it is still not clear
what implications their rapid uptake will have on the design of interactive
systems for data access and exploration. The applications proposed so far
mainly support the retrieval of very specific data from online services. It
is still unclear how this paradigm can be applied for the interaction with
large bodies of information and machine agents. A critical challenge lies
in understanding how to make the development of bots for data exploration
scalable and sustainable, in terms of both software infrastructures and design
models and methodologies [2, 27].
To fill this gap, in this Thesis we propose a methodology to support
the design of chatbot interfaces for accessing extensive collections of data by
means of a conversational interaction; in particular, we propose a data-driven
paradigm that, starting from the annotation of a database schema, leads to
the generation of a conversation that enables the exploration of the database
content.
1.1 Scenario and Problem Statement
Let us consider the situation in which a person enters in a library and asks
some information about a novel at the helping desk. Usually librarians are
able to identify books quite well, but this time the task has some unexpected
difficulties: the client does not know either the title or the author and just
wants something to read. The employee cannot do very much without in-
vestigating the client’s preferences or giving personal advices, because the
searches on the online catalog of the library are possible only using keywords
that are somehow related to a book, in terms of title, author or publishing
house.
In this example the attitude of the client is not to simply find something
in particular, based on some known properties of the object being researched,
but is rather to explore the library resources without a fixed target and not
knowing what will be the result of the inspection. What if the librarian
could fulfill the customer’s needs with a tool able to support this kind of
exploration? How could this device be designed?
The key point of this appliance would be the ease to use: its users should
not need to learn how the system works, but they should understand its
features and possibilities while using it. So, a possible approach is to cre-
ate a tool able to emulate human-to-human interaction, by communicating
through the natural language, using text or vocal channels. This technology
is also known as chatbot and its implementation may fit the needs of our
purpose. The chatbot, also known as conversational agent, should allow the
user to navigate across the catalog, exploiting all the relations between the
different entities in it. For example it could support the input phrase “find
the books written by Dante” and execute the search against the catalog, by
automatically understanding that “Dante” is the keyword for the name of
the author. Then it could display the titles of the books in the forms of
buttons and, upon clicking on “La Divina Commedia - Inferno”, it could
present the details of the volume as well as its basic relations, i.e. author,
publishing house, etc.
Up to now, it seems that our imaginary chatbot would be able to act
as a special search engine, that understands and translates sentences into
commands, but still distant from offering the above mentioned explorative
feature. What if, then, the user could navigate from the “La Divina Comme-
dia - Inferno” to the collections of volumes that include the masterpiece and
2
were borrowed by someone else? Then, after clicking on the details of one of
these, the navigation could proceed towards its publishing house, then to its
publishers, then maybe to one of these authors and then again to the books
given to the library by someone that once borrowed a volume from this last
writer. The chatbot could offer all of these features and more, by presenting
to the user different relations and paths to be taken from the various entities
under observation.
The term exploration can be finally applied to this type of navigation,
since the user can navigate among entities and relations of the library catalog
without really having a destination in mind.
There are many real world application where such a conversational agent
could be used and each one of them might have different modalities, with
respect to the users it is supposed to embrace, the quantity of relations among
the data and the values to be displayed. For example, this feature could be
supported by a data storage for investigative research, enabling detectives to
easily and freely navigate across the data describing past cases. This system
could be also applied to a medical database, allowing doctors to explore
attributes and possible relations of clinical past situations. Moreover, this
kind of chatbot may be used within a business company, in order to explore
the data its database contains, with different levels of granularity, in terms
of searches and relations among the various elements.
1.2 Contributions
In this Thesis we define a data-driven design paradigm that, starting
from the structure of a typical data system, enables the generation of con-
versation paths for its exploration.
In particular, we provide:
• An explanation of the concept of Chatbots for Data Exploration, with
reference to the requirements and goals that must be satisfied when
designing these type of systems;
• An architecture for a framework able to support the creation of these
conversational agents, also exploiting state-of-art technologies;
• A methodology to design chatbots which poses the attention on the
data characteristics, rather than the technologies involved;
3
• A technique to generate conversational agents based on annotations
made directly on the schema of the data source;
• A prototype of this framework integrating different technologies to de-
ploy it;
• Some design implications derived from a comparative study that we
conducted to test the performance of the chatbot generated through
our framework with respect to the basic SQL command line interaction.
From the qualitative and quantitative analysis of the data gathered through
the user study we can state that the resulting applications, as well as the
framework for their development, can be considered as a first valid step
towards the generation of Chatbots for Data Exploration.
1.3 Definitions
We provide below the definitions of the main concepts related to our work.
• Chatbot: a conversational software agent able to interact with user
through natural language, using voice or text channels and trying to
emulate human-to-human communication.
• Chatbot Framework: is a cloud based chatbot’s development platform
that enable on line development and testing of chatbots; usually, it also
provides Natural Language Processing (NLP), Artificial Intelligence
and Machine Learning services with Platform as a Service paradigm
(PaaS).
• Chat Interface: with this term, we refer to any system that supports
text-based input from the users.
• Schema Annotation: abbreviation for Database Schema Annotation,
it refers to the procedure (and its outcome) of characterizing tables,
relations and attributes of a database conceptual model, in order to
produce a mapping between the data source and the conversation in-
teraction basis.
• NLU: it stands for Natural Language Understanding, and it is a branch
of Artificial Intelligence that uses computer software to understand
input made in the form of sentences in text or speech format [26].
4
• Intent: it represents the purpose of the user’s interaction with a chat-
bot; for instance, from the phrase “What time is it?” the intent, if
defined, could be to know the current time.
• Intent-matching: it is the NLU process of understanding the intent
from a given input phrase.
• Entity: it is a word or a set of words that represent a concept in a
given utterance; for instance, in the phrase “I live in Milan” the word
“Milan” may be an entity related to the concept of location.
• Entity-extraction: it is the NLU procedure of identifying entities inside
input phrases provided to the chatbot.
• Context: it represents the current background of a user’s request;[12] in
a chatbot, it is needed to correct handle subsequent inputs which may
refer to previous outcomes; for instance, in the consecutive interactions
[User: “What is the weather like in Milan?”] → [Chatbot: “Sunny”]
and [User: “And in Rome?”]→ [Chatbot: “Cloudy”], the second ques-
tion would not make sense without the previous request, which creates
the context (in this case, related to the weather) of the whole dialog.
1.4 Structure of Thesis
• Chapter 2 is an overview of the history of chatbots, the state of art
and the most popular available frameworks that support the deploy-
ment of such applications. In this Chapter we highlight the differences
among the various technologies, highlighting pros and cons of each one,
considering implementation and design choices.
• Chapter 3 provides a definition of Chatbots for Data Exploration and
defines the main goals and requirements that must be treated to design
them.
• In Chapter 4 we provide a conceptual modeling approach for the de-
sign of these kinds of chatbots. We offer a solution to the requirements
defined in the previous Chapter, splitting the design process in three
parts: the Schema Annotation, the Conversation modeling and the In-
teraction modeling.
5
• Chapter 5 describes the architectural and implementation choices that
we made when developing the prototype of a Chatbot for Data Explo-
ration.
• In Chapter 6 we provide an evaluation of our framework, by means
of a comparative study between our conversational agent and the SQL
command line, regarding the execution of some information retrieval
tasks on the same data source. We discuss the details of our tests and
the collected data, making quantitative and qualitative analysis.
• In Chapter 7 we conclude the Thesis with a summary of the overall
study, considering limitations, future works and possible improvements
of our framework.
• The Appendices report:
(i) a JSON-based specification of the database schema of the example
database used throughout the Thesis (Appendix A),
(ii) the annotated database schema, which is the input of the process
leading to the generation of a conversation for the exploration of
the example database content (Appendix B),
(iii) the questionnaires administered during the user study
(Appendix C).
6
Chapter 2
State of the Art
2.1 The (R)evolution of Chatbots
In this Section we will discuss how chatbots grew from past to present, in
terms of technologies and capabilities, and how nowadays there are more and
more domains in which they are applied.
2.1.1 The Loebner Prize
Nowadays, chatbots technologies are growing faster than ever [9], but their
story started in in the second half of 1900s. The Turing Test, also known as
the Imitation Game, was proposed by Alan Turing in 1950 as a replacement
for the question “Can machines think?” [1, 25]. It consists in a quiz to be
held with a computer able to answer questions, requiring the judge to be
uncapable to distinguish whether the interlocutor is a human or not.
This idea was taken by Hugh Loebner, which in 1991 decided to create
an annual competition for evaluating computer programs ability to emulate
a human, just like the original Turing Test required. The contest survived
all of these years and different chatbots won the first place over time.
The first winning machine was ELIZA, a computer program created in
1966 whose conversation structure was then associated to the ones pursued
by certain psychotherapists, known as Rogerians [32]; the program won three
consequent Loebner prizes (’91, ’92, ’93).
Other examples of more recent winning chatbots are A.L.I.C.E (2000,
2001, 2004) and Mitsuku (2013, 2016, 2017, 2018) [1, 29]. Both programs
were created using a AIML knowledge base, that is a derivative of XML, and
performed pattern matching on each input to elaborate a response.
2.1.2 Virtual assistants
The Loebner prize, as introduced in the Subsection 2.1.1, is assigned by
evaluating ability of the chatbot to mock human behavior. However, in the
last few years, another type of chatbot come up: the Virtual Assistant, a
software agent that can interpret human speech and respond via synthesized
voices. The most popular examples of such systems are Apple’s Siri, Ama-
zon’s Alexa, Microsoft’s Cortana, and Google’s Assistant. According to [17],
“these kinds of chatbots are always connected to the Internet and each in-
teraction is sent back to a central computing system that analyzes the user’s
voice commands and provides the assistant with the proper response.”
They offer generic one-size-fits-all services [11], by enabling their users
to access different functionalities like setting reminders, performing inter-
net searches, communicate with remote home automation systems and other
simple tasks that would require longer interactions to be completed manually.
2.1.3 A chatbot for everything
Besides the systems described in the previous subsections, chatbots start to
be created and used in an increasing number of specific domains, each with
its own requirements. For example, a 19-year-old student designed a chatbot
to help drivers that received parking tickets, in London and New York, to
contest the fine; the results of 2016 show that the system had a success rate
of 64%, in less than two years from its launch, by overturning 160,000 tickets
[23]. In 2017, at the Manipal University, a conversational agent was designed
to answer FAQs related to that educational institute [30]. Another more
recent example, dated 2018, consists in the development of a chatbot for
meal recommendation [13].
2.2 Design Approach
From a high perspective, chatbots can be distinguished in two different mod-
els by the way they generate responses: Retrieval-based and Generative-
based [31, 28].
8
The first approach allow to assemble the answer by selecting from a set of
predefined responses according to the input, the context and some heuristic.
The latter may be as simple as a rule-based expression match, or i could
include Machine Learning classifiers trained to predict the matching score.
Their main advantage is that the answers provided are always grammatically
correct, but since they are picked from a finite set of choices, they may lack
originality.
Chatbots designed following the second model, instead, build new re-
sponses from scratch, treating generation of conversational dialogue as a
statistical machine translation problem. In fact, depending on the current
context of the conversation, they are able to autonomously create the answer
word by word, based on a given vocabulary.
2.3 Development Choices
As we have seen in Section 2.1, there are multiple applications of the chatbot
concept, each one with different purposes. It is important to highlight how,
over the years and starting from the first conversational agent ELIZA, these
systems increasingly exploited technologies belonging to the Web and, more
specifically, those related to the APIs.
A common example is the case of the chatbot for weather forecasting, that
is able to help getting information about present and future climate changes
based on the location of the users. This is possible thanks to the availability
of some external Web based APIs which the system can make request to, in
order to transform their response into an utterance that includes the wanted
data about weather.
Let us discuss now which development choices must be taken when start-
ing to developing a chatbot like the one just described.
2.3.1 Local or remote
To begin with, we can distinguish three development choices related to the
possibility of taking advantage of existing Web resources. Thus, let us pose
our attention on where to place the internal modules of the chatbot, i.e. the
Natural Language Understanding service and the application unit.
A first approach considers relying completely on existing development
frameworks available online. These systems usually offer a user friendly in-
9
terface and tools to develop a fully functioning chatbot easily. Their main
drawback is represented by their rigidity and lack of customization features,
along with their strict dependence on the Web platform they stand upon.
Another option, instead, is to rely on a Web based system to develop the
Natural Language Understanding unit, while deploying the application mod-
ule independently. This gives both the opportunity to completely manage
all the logic behind the conversation, while delegating the problem of intent
matching and entity translation to the remote service. In this case, the latter
acts as an on demand translator of input phrases, properly sent by means of
a dedicated API. It is important to remind that this choice does not prevent
the developer from instructing the Web service on how utterances should be
managed: the necessity to define which intents the chatbot can recognize,
as well as the entities, is still present and bound to the remote architecture
specifications.
The last approach is related to the development of a standalone system,
in which the developer has to create an ad-hoc model which includes both the
utterance parsing and the dialogue management part. This development di-
rection allows not to be dependent to Web services, on the Natural Language
Understanding side, but it could still require a stable Internet connection in
case information retrieval is related to some APIs. The main advantage of
this development approach is the possibility to completely craft a system
able to respond to the developers needs, both in terms of how the chatbot
can interpret utterances and how the conversation must continue once this
procedure is completed.
2.3.2 The chat interface
Another aspect that must be considered, during the development of a chat-
bot, is the communication channel that will support the input of the users
that will interact with the system.
A first solution may include the creation of a custom chat interface in-
tegrated to the system and accessible through a dedicated application; it
could be deployed locally or as a remote service, i.e. accessible by means of
a Web page. In this situation an additional effort is required to the devel-
oper, bearing in mind that the conversation quality may be enhanced by the
introduction of dedicated buttons, for instance.
Instead, a more convenient way is to take advantage of existing chat
10
platforms that already offer a solution to these kind of problems. In fact,
they allow the developer to create an account belonging to their system and
provide an API to control the messages sent and received by this online
profile. The main advantage derived by the usage of these online interfaces
is that they are always available and already provide advanced user friendly
communication features, e.g. buttons, images exchange, document sharing,
etc. Moreover, the developer may use more than one of these systems at
a time just enabling the system to understand and communicate with the
APIs, without changing the logic of the application. The main drawback of
such development choice is the dependence on an always available Internet
connection. Examples of these systems are Telegram1, Slack2, Messenger3,
Twitter4, Kik5, Line6, Viber7, etc.
2.4 Different Chatbot Frameworks
In the previous section we analyzed the choices that must be made when
developing a chatbot, in terms of where the various modules could be allo-
cated, if locally or remotely, and whether it is better to use an existing chat
interface or to build a new one from scratch.
Now let us consider in details the most popular frameworks that allow
to develop a chatbot, considering the services they offer with respect to the
development decisions described so far. Thus, during this analysis we will
concentrate both on NLU and dialogue management features, along with the
ease of set up and interoperability with existing chat interfaces.
Dialogflow8, as first example, is an online framework that require a
manual configuration of intents, entities and contexts management and dec-
laration capabilities. Besides the NLU feature, it integrates a dialogue man-
agement system able to perform customizable API calls and to maintain in
memory some parameters with settable lifespans. Moreover, it integrates well
with the majority of chat interfaces available online, it offers SDKs for multi-
1https://telegram.org/2https://slack.com/3https://www.messenger.com/4https://twitter.com/5https://www.kik.com/6https://line.me7https://www.viber.com/8https://dialogflow.com/
11
ple programming language and integrates a Machine Learning procedure to
increase the chatbot comprehension of the user’s input.
Wit.ai9 is a very similar online tool and provides almost the same func-
tionalities. The definition of custom actions is common to both, along with
their ease of integration with external application programming and chat in-
terfaces. Both the systems present an HTTP API to limit the interaction to
create and use the chatbot; moreover, the two present an internal system for
speech recognition.
PandoraBots10, instead, limits its features to the NLU tool, which is
accessible via its API. It is based on the AIML language, a derivation of
XML, which allow the definition of input pattern and related templates for
the answer. Its definition is not as intuitive as the other framework a analyzed
so far, but it allows the total control of the bot’s knowledge based. In any
case there is a limitation in terms of max number interactions with the free
license.
Another online framework is FlowXO11, which allow the definition of the
chatbot conversation by means of an online graphical editor. While being
highly integrated with various chat interfaces and existing APIs, it allows
the connection with a custom Webhook as well. It also present an internal
NLU system for message parsing.
Watson12 is another online tool that provides the development of chat-
bots. As DialogFlow and Wit.ai, it allows the definition of intents and entities
and provides a dialogue management system, as well as a NLU interface. The
developer can also connect a custom webhook and/or communicate with a
dedicated API. The main disadvantages, besides requiring a continue Inter-
net connection since it can only be hosted in cloud, include the necessity to
define sequences of intents for context management.
Chatscript13, instead, is “a rule based engine, where rules are created by
human through a process called dialog flow scripting”. This chatbot tool can
run locally and offer services like: a pattern matching algorithm aimed at
detecting the phrase meanings, an extensive and extensible ontology of nouns,
verbs, adjectives and adverbs, the ability of remembering user interactions
9https://wit.ai/10https://www.pandorabots.com/11https://flowxo.com/12https://www.ibm.com/watson13https://github.com/ChatScript/ChatScript
12
across conversations and the rules capabilities to dynamically alter engine
and script behavior [12].
Another framework is represented by RASA14, which is “a set of open
source machine learning tools for developers to create contextual AI assis-
tants and chatbots”. It offers two different technologies: RASA Core and
RASA NLU.
The first one is a chatbot framework with Machine Learning-based dia-
logue management, which allows the definition of the conversation structure,
by means of a dedicated markup language. During the design phase, the
developer may describe some wanted conversation paths, in terms of likely
future sequences of intents. Then, after the dialogue engine is trained ac-
cording to this model, the chatbot is able to match the received phrases
against these intents while maintaining the context in a previously created
target sequence. Moreover, the developer can specify custom actions which
will be called when a specific intent is matched, as well as direct utterances
to respond to user messages or custom variables to extract, persist and reuse
during the conversation.
RASA NLU, instead, is a library for Natural Language Understanding
which offers intent classification and entity extraction features. Its main
advantages are that it can run locally and it is highly customizable, while
being lightweight from a programming point of view. In fact, the developer
does not have to learn and apply a new language, like in the case of AIML,
since RASA NLU allows the definition of free-text training phrases; these
examples, then, are used to train an Artificial Intelligence model that will be
used during the conversation phase. Moreover, although it is an open-source
solution, it easily competes with the other commercial products analyzed so
far [5].
2.5 Related Research Approaches
As we said, in the last years we have assisted to the creation and spread
of new frameworks for the development of chatbots, as well as the opening
of the APIs conducted by the major messaging platforms towards third-
party developers. However, what seems to be still missing is a common
methodology that guide the creation of a conversational agent, i.e., a model
14https://rasa.com/
13
that define the design procedure in terms of requirements, objectives and
dialogue characteristics.
Some commercial frameworks developed their own dialogue management
system, exploiting the concepts coming from the NLU field and each defining
a dedicated engine based on popular Knowledge Bases, Machine Learning
technologies or Artificial Intelligence concepts. However, besides offering
state-of-the-art technologies and a user-friendly interface for their integration,
they are limited to the creation of specific chatbots; in other words, when
using these systems, the design process is limited to the offered model and,
above all, has to be conducted differently with respect to the requirements
of each conversational agent.
The lack of a models for a design methodology can be traced back to
the complexities that the development of a chatbot involves. A scientific
investigation conducted in 2018 describes “the lack of research about how to
effectively workout a Return-Of-Investment (ROI) plan” when dealing with
the design of a chatbot [27]. In particular, it introduces a few important
dimensions that need to be considered, like the interaction of the user with the
agent, in terms of information retrieval and presentation, or the integration
aspect with databases or external APIs. Moreover, the conversation quality
assurance, maintainability and dialogue analysis are also fundamental aspects
that need to be considered when designing these conversational agents.
A research approach similar to the one we discuss in this Thesis proposes
a data-driven model for the design of a chatbot dedicated to educational sup-
port [2]. Although the project aimed at the creation of a learning assistant,
the first step for its development started from a definition of a paradigm to
annotate the dataset involved. The introduction of a procedure that acts on
the data system enabled the overall study not to be bound to the content of
the data, rather to be extended to all the information systems built with the
same technology.
This is also the aim of our research, i.e., the definition of a model that,
starting from the characterization of the data, enables its exploration by
means of a conversational support.
14
Chapter 3
Chatbots for Data Exploration
3.1 Concepts
Let us introduce the idea of a Chatbot for Data Exploration with the help of
a conversation example between an user and the chatbot itself, as shown in
Figure 3.1.
At first, the user does not know how the system works, but after a few
messages the capabilities of the chatbot are made clear. It is easy to see how
all the interactions have a common thread, that is the dynamic visualization
of some data. What is remarkable is that knowing the structure of data is
not a precondition for their visualization, since the system helps the user
by showing examples and buttons, which can be considered as hints for the
continuation of the conversation. This mechanism of interplay between user
requests and application responses, represents what can be defined as an
exploration of a set of data, by means of a conversational approach.
These three keywords, indicated in bold above, are the concepts that will
guide the definition of the objectives and the characterization of the field of
action to develop our system.
3.2 Main Requirements
The goal of the system that is the object of this work is to offer a tool
capable of navigating among some data. To achieve our objective, there are
a number of requirements to be fulfilled by the environment for the chatbot
generation and execution.
What do you have to show me?
Hi there explorator!Here is a few examples of what you can ask me:Find orders that contained product Fiat PuntoFind customers that are located in MilanFind payments done by American Souvenirs Co.
Find customer with contact John
Let me check...Et voilà! I found 8 values:Car Models - B. Goode JohnJJ Store - Browns JohnAmerican Souvenirs Co. - Wayne John
Select the one you are interested in or clickthe last button to see more.
* User clicks button *
Here is what I know about this customer:- customerName: Car Models- contactLastName: B. Goode- contactFirstName: John - ...If you want more information, I can tell youProducts ordered Payments madeOrders made
Show me the past selections, please
If you want, you can click on a button to go backin the history of the conversation.
Car Models - B. Goode John Currently you are focusing on:
...selected from:A list of type "customer" ...found with attribute(s) "with contact John".
Chatbot
Chatbot
Chatbot
Chatbot
User
User
User
User
SHOW MORE
Figure 3.1: A Conversation example between a user and the chatbot.
16
3.2.1 Connect to the database to obtain the schema
As a first requirement, the chatbot must be able to connect to with a
database. In this Thesis we focus on relational databases, but in princi-
ple this requirement is to be considered for any data model. However, in the
following, we will make reference to the relational model, which is the one
our prototype has been defined for. In any case, the information retrieval
can take place only when both the right query language and the internal
structure of the data system are known to the chatbot.
While the first problem can be solved with the help of the right connec-
tor, which is capable of connecting to a database and performing queries,
the second issue may be more difficult to solve: the system must be able
to correctly identify the relations among the different tables and create an
abstraction of the schema suitable for the type of navigation at the basis of
the chatbot.
3.2.2 Extract and learn database-specific vocabulary
and actions from schema
Not only the conceptual model of a database is unique in terms of tables,
attributes and relations, but also the information it stores varies depending
on the elements its tables represent and the characteristics more valuable
in the context of its usage. A system able to support this conceptual and
logical heterogeneity is not likely to fit the need of a proper conversation by
simply relying on the structure of the database, since there is no guarantee
that the information it represents becomes relevant when visualized as it is.
For example, if we consider a Cross-Reference Table in a relational database,
we clearly see how its rows become human-readable only when analyzed with
the tables it relates to. One possible approach to tackle this problem could
be the automated limitation of the visualization of some tables based on
their structure, but this might lead to the unintentional removal of possible
valuable information.
A better perspective could be to consider the complexity of the database
as a starting point towards the definition of a suitable abstraction in the con-
text of a conversation. The result of such a procedure would be a data-driven
model that represents the database as a network of connected elements that
could fit the necessities of the final user. Specific vocabulary, for example,
17
might be used to refer to the different entities of the data system, as well as
custom actions to take when focusing on each of them, by exploiting existing
relations and defining new ones, in a user-oriented viewpoint.
This customization is more likely to be carried out manually, by a De-
signer that knows how information is organized in the database and what
could be the interests of visualization and navigation of the final users. This
process can be considered as the design of the conversation and may allow the
definition of multiple conversation models for the same data system, depend-
ing on the most relevant interests of the ultimate consumers, as identified by
the Designer.
3.2.3 Automatic intent and entities generation and ex-
traction
Since the key feature of a chatbot is the ability to interpret natural lan-
guage input, one requirement that our system must meet is the capability to
automatically train a natural language model which permits to understand
phrases related to the context of the data it handles. In the previous Sub-
section we discussed how the schema of the database may be abstracted in
a conversation-oriented perspective, but we did not define how this resulting
model might be used to guarantee the specifications just declared.
A possible approach, following along these design choices, could take ad-
vantage of the concepts of intents and entities belonging to the Natural Lan-
guage Processing (NLP) area. While intents, which can be considered as the
intention of the user, could be mapped to specific actions to be performed
on the database (e.g. filter, select, join, etc.), entities represent the
elements that the chatbot can extract from an input phrase and that could
be associated with the elements structuring the data system, i.e., its tables,
relations and values.
Relying on these assumptions, when the chatbot receives a phrase, it
could perform what is called intent matching : the process of understanding
what the objective of the input utterance is and its consequent translation
into a database query. Entity extraction, instead, could be used to correctly
parametrize the request, in terms of table and attribute references.
It is important to consider the fact that the phrases the chatbot receives
don’t have to be formulated in a metalanguage which only relax the syntax
requirements of the SQL language; rather the system should rather accept
18
and correctly understand utterances belonging to the natural language. In
order to reach this objective, the bot might train its NLP model using some
sets of example phrases for each intent to recognize, labelled in accordance
to the entities each pattern contains.
3.2.4 Proper communication and data visualization
Once the system interprets a user utterance and executes the correspond-
ing command, the problem of presenting the results arises. While the chat
environment can be considered a user-oriented communication channel as it
simplifies the way requests can be done by relying on the natural language
format, it is also true that it may represent an issue from the system per-
spective. In fact, the user experience must be designed on the basis of a
conversation, thus with a number of limits that a graphic tool, for instance,
would not have.
Since the object of the navigation are the data, the first issue to solve
is how to present them in a readable way. Let us consider the Figure 3.1:
after the user asks the bot to find some elements and the system performs
the related search, the results are displayed as a list of buttons, identified
with some words representing the elements they link to. This simple example
already highlights how the chatbot should be able to:
• Respond to the user using natural language, in order to present its
results;
• List the results in a readable way, taking into account their quantity
and short messages to identify them;
• Take advantage of buttons, to simplify selections and the overall navi-
gation.
Besides these requirements, the system may handle cases of typos, wrong
insertions/selections, help requests and so on, which are conditions in which
the communication flow skews from its expected path.
3.2.5 Allow the user to manipulate query results
A possible request by the user may involve the visualization of a high number
of elements and, because of this, the chat environment might represent the
19
bottleneck of their readable visualization. Thus, the user may have the need
not to view them all, but to filter them according to some attribute.
3.2.6 Support context and conversation history man-
agement
Other important aspects that may be part of the system refer to the man-
agement of context memory. This represents the capability to interpret each
user request not just as an independent single command, but as the out-
come of an evolution of connected intentions over time. While from a human
perspective this condition represents the basis of a conversation, in which
every interaction is guided directly or indirectly by the progression of the
past ones, this aspect has to be considered carefully in the computer systems
dimension.
For what concerns the system we aim to define, we should consider the
context management as a key functionality, since navigation is only possible
when the steps that shape the chosen path among data are known to the
chatbot. In other words, the the system must remember previous user’s
requests to enable the refinement of the results by some filtering, or the
exploration of elements related to the past ones displayed.
Besides this critical requirement, the chatbot could also offer the possibil-
ity to go back in conversation, by restoring its context to a precise moment
and allowing the navigation through a different dimension from the previ-
ously chosen one. This is the case of the last interaction represented in
Figure 3.1, where the chatbot displays the history of its conversation with
some buttons linking to the elements they refer to.
20
Chapter 4
Approach
Now that the requirements and goals have been made explicit, it is important
to define what are the steps that have to be taken when designing the chatbot
according to our methodology. Since entities of various types are necessary
and each of them play different roles in the system, we need to take particular
care of their heterogeneity during the design phase and we must consider what
is our purpose at each step. Therefore, our modeling choices are directed
towards our objectives, while being feasible with respect to the architectures
they rely upon.
4.1 Design Process
We can consider the creation of the chatbot for the exploration of a database
as a semi-automatic procedure. In fact, the process of designing the system
includes both automatic and manual phases. The manual phases have to
be performed by a Designer, who knows the database, the goals and the
requirements of the bot to be developed.
Parsing of the database schema. This is the first phase of the chatbot
design process and it is automatically executed by the system. In this step,
the schema of the target database is parsed and interpreted, in order to obtain
a simplified version which highlights its structure, in terms of table attributes
and relations among entities, not considering additional details about data
types, nullable or default values, etc. Moreover, this representation is needed
to support the annotation activity described in the next paragraph.
Schema Annotation. This can be considered as the core part of the whole
design process and must be performed manually by the Designer. In this
phase the simplified database schema is annotated with special tags that will
be used in next step to generate the conversation itself.
NLU model extraction This last phase is done automatically by the
system and consists in the training of the Natural Language Understanding
(NLU) model. This process is carried out in two separate steps:
1) The generation of the training phrases, based on the annotation of the
schema.
2) The training of the NLU model, starting from the utterances just cre-
ated.
It is important to say that this phase is quite delicate even if it is completely
performed by the system. In Section 4.3 we explain in details what are the
key points to generate the NLU model, while in Chapter 5 we describe a
possible solution that supports this procedure.
4.2 Schema Annotation
To translate an input phrase into a specific query on the database, we propose
the definition of a mapping between what can be understood by the chatbot
(intents and entities) and the elements of the database (tables, attributes,
relations) by means of a Schema Annotation. This section is devoted to
illustrate the main ingredients for this mapping procedure and the recipe of
the approach; to make the description easier to follow, we will refer to an
example database shown in Figure 4.1.
4.2.1 Conversational Objects
The first step is to characterize the database tables to express the role that
the elements they represent play in the exploration of data:
1) Primary - tag [P]
2) Secondary - tag [S]
3) Crossable Relationship - tag [X]
22
customers
customerNumberPK
customerNameNN
contactLastNameNN
contactFirstNameNN
phoneNN
addressLine1NN
cityNN
state
countryNNpayments
customerNumberPK, FK
checkNumberPK
paymentDateNN
amountNN
orderdetails
orderNumberPK,FK1
productCodePK,FK2
orders
orderNumberPK
orderDateNN
statusNN
customerNumberFK, NN
products
productCodePK
productNameNN
productScaleNN
productDescriptNN
quantityInStockNN
Figure 4.1: The relational schema of the database.
The distinction between the first two types is that while values belonging
to a Primary table may represent useful information without the need of
a context, Secondary tables contain data strongly related to other entities,
which become clear only when that specific relationship is considered. For
example, tagging as Secondary the table payments means that the access
to its values depends on another table, customers. In other words, with
this characterization the Designer defines that instances of payments can be
retrieved only by passing first through instances of customers, for example
because payments data would not be meaningful otherwise. Labeling ta-
bles as Primary, in contrast, relaxes this dependence constraint and enables
direct queries on tables. Finally, Crossable Relationship characterization is
dedicated to bridge tables, i.e., the ones that represent many-to-many re-
lationships between entities. Their rows may have a meaning only when a
join operation across them is executed; thus no direct or deferred search is
allowed on them.
Some tables might have names not suitable in the context of a conver-
sation. Thus, as reported in Figure 4.2, each Primary and Secondary table
has to be labeled with a keyword, i.e., the name that will represent it during
the conversation, that from now on we will call Conversational Object.
23
For example, in the request “find customer with contact John” in Figure
3.1, the chatbot understands that customer is the placeholder for the table
customers. As represented in Figure 4.2, multiple words can be associated
to a table, acting as aliases for the same element; this will allow the users
to refer to them in multiple ways. As a result, the chatbot will be able to
understand customer, client and both their plural form as keywords for the
same, enabling the user to prefer different utterances, like “find client with
contact John”.
customers
payments
orderdetails
orders
products
[X]
[P] product
[P] order [P] customer
[S] payment
customersclient clients
orders
products
item
items payments
Figure 4.2: The Conversational Objects annotation.
4.2.2 Display Attributes
If we pay attention to Figure 3.1, we can see how the chatbot, after the
first interaction, displays a list of customers. What is remarkable is that the
values shown seems to be representative of the objects they refer to. In other
words, the chatbot does not show all the information related to each element
when displaying them in a list and it does not even limits the visualization
to their primary key, which is unique but maybe not informative; it rather
shows the name of the customer, as well as its contact.
This is possible because the Designer, when dealing with the Schema An-
notation, tagged the table attributes customerName and the pair (contact-
24
LastName, contactFirstName) as Display Attributes. In fact, this pro-
cedure enables the chatbot to refer to the values there contained to display
the results in lists as we described before and it can be applied to Primary
and Secondary tables. In Subsection 4.4.3 the problem of results visualiza-
tion is described in details. In Figure 4.3 there is a graphic representation of
the Display Attributes annotation performed on the example database.
By looking at the Figure, we can appreciate how some of these columns
may be aggregated into a single list, like in the case of (contactLastName,
contactFirstName), as we already considered in the example just described.
Some others, instead, present a word or a short phrase linked to them. This
annotation, visible in correspondence to the table payments in which the
label for amount is Euros and the one for paymentDate is in date, will
allow the chatbot to display lists of payments with this example format:
“[Euros: 12345 - in date: 16-04-2019]”
customers
customerName
contactLastName
contactFirstName
payments
paymentDate
amount
orderdetails
orders
orderNumber
orderDate
products
productName
_
_
Euros
in date
_
_
in date
Figure 4.3: The Display Attributes annotation.
4.2.3 Conversational Qualifiers
Defining the role of tables and their related placeholders enables the system
to translate phrases like “find customers”, which aim to select all the in-
stances of the corresponding table. Other elements are needed to interpret
conditions for filtering subsets of instances, for example queries like “find
customers that are located in Milan”. In order to support this granularity in
25
the results, the Designer can define a set of Conversational Qualifiers for
each data element, as shown in Figure 4.4 where the treated entity refers to
the customers.
This annotation procedure considers what are the attributes that will be
used to filter the results and how the final user will refer to them, i.e., through
which expressions. For example, in Figure 4.4, the attributes city, state
and country are labeled with the expression located in: this gives the chat-
bot the capability to understand input phrases like “find customer located
in Milan”, and process it as a query that selects the customers having city
= Milan, or state = Milan, or country = Milan. These Conversational
Qualifiers may belong to other tables, rather than the one directly addressed
by the user request. This is for example the case of the Conversational Qual-
ifier that bought, that, even if it was defined for the field productName of
table products, can be used also in queries like “find customer that bought
Fiat Punto”. The table customers is indeed reachable through the tables
orders and orderdetails. This is graphically represented in Figure 4.4 by
means of dotted arrows.
In some cases the user may also ask: “find customer Car Models”, without
including in the utterance any Conversational Qualifier, i.e., without speci-
fying what the Car Models value stands for. The system, then, will search
for related instances in the table customers by the attribute customerName;
for this attribute, indeed, there is not any specific expression specified as
Conversational Qualifiers (graphically denoted by “ ”).
4.2.4 Conversational Types, Values and Operators
In order to help the chatbot process and interpret correctly the utterance, for
each Conversational Qualifier it is important to specify its Conversational
Type:
• WORD: any information without a particular structure or syntax;
• NUM: numerical values;
• DATE: datetime/date values;
• Custom ENUM: entities that can assume enumerable values.
The last one may be useful when an attribute can assume only a fixed set
of values. An example, not present in our database, could be the attribute
26
customers
customerName
contactLastName
contactFirstName
city
state
countrypayments
amount
orderdetails
orders
products
productName
[P] customer
with contact : WORD
_ : WORD
located in : WORD
that paid : NUM
that bought : WORD
Figure 4.4: The Conversational Qualifiers annotation, focus on customer.
businessName for the table customers and with enumerable values Public
Administration and Private Company.
It is important to clarify how the Conversational Type format is used by
the Designer during the Schema Annotation phase and it will be recovered by
the framework during the Conversation Modeling phase, as we will describe
in Section 4.3. However, from now on when dealing with example phrases, we
will use the term Conversational Value to refer to the real value, always
remembering that its identity is strictly related to the concept of Conversa-
tional Type, as already described. Thus, according to this notation, in the
phrase “find customer Car Models” the bold words will be identified as the
Conversational Value.
The utterances received may include some more complexity, apart from
the examples analyzed so far. For instance, the user may type a phrase like
“find the customer that paid more than 42000 Euros”, and the system should
identify the correct references for what concerns the Conversational Object
“customer”, Conversational Qualifier “that paid” and Conversational Type
NUM, with Conversational Value “42000 ”. The problem here is that the
chatbot may not consider the expression “more than”, leading to a malformed
search with likely wrong results.
This issue can be solved by introducing the concepts of Conversational
Operators and enabling their usage with respect to the defined Conversa-
tional Types.
27
A possible trace could be the following:
• WORD → “like”, “different from”;
• NUM → “more than”, “less than”;
• DATE → “before”, “after”;
4.2.5 Conversational Relationships
Always referring at the the conversation example in Figure 3.1, after the
user has successfully selected a customer, its data and a few buttons are dis-
played. These last represent the relationships that can be navigated starting
from the instance being examined; we will refer to them as Conversational
Relationships. These relationships have to be specified in the annotated
database schema (see labeled arcs in Figure 4.5) so that user utterances can
also imply navigation in the database through join operations. The arcs
in Figure 4.5 are not bidirectional, since during conversation the user may
refer differently to relationships depending on the direction in which they
are traversed. Note that the relation between product and order needs to
be supported by a Crossable Relationship table, being it a many-to-many
relationship.
This kind of support is needed also in the relation named as products
bought, that links the customer entity to product. Here we can appreciate
how enabling the definition of this kind of connection relaxes the constraints
imposed by the structure of the database. In fact, the Conversational Rela-
tionship just analyzed is oriented to what the final user may be interested in,
rather than on how data is physically organized: customers and products,
in this case, are connected with a direct link, while the underlying relational
model needs multiple subsequent relations to obtain the same functionality.
4.3 Conversation Modeling
In the previous section we analyzed how Schema Annotation opens the design
process. In this section, instead, we will consider how that procedure can be
used to obtain a chatbot able to understand phrases related to the considered
data system, by taking advantage of the concepts already defined in the
previous Section.
28
[X]
[P] product
[P] order [P] customer
[S] payment
contained products
in orders
made by customer
orders made
payments made
made by
products bought
Figure 4.5: The Conversational Relationships annotation.
Designing a Natural Language Understanding model means to define the
intents and the entities that the chatbot will be able to match and extract,
respectively. However, we are in the situation in which the target database
of the system is not fixed, so it is not possible to define an end-to-end con-
nectivity between these NLU concepts and the query they refer to when they
appear in a utterance.
This limit becomes evident if we consider that the chatbot may be in-
terfaced with a very large database, with dozens of complex tables and even
more relations: defining, for example, an intent for each possible query struc-
ture and an entity related to each value of the tables could lead to a NLU
model exponentially bigger than the data system.
In order to avoid this situation, we came up with a solution that auto-
matically generates intents and entities starting from the database Schema
Annotation. However, we also needed a procedure to generate the example
phrases to train the chatbot, each with labelled entities and related to an
intent.
29
4.3.1 Intents
With respect to the tools our chatbot should provide, we decided to define
four types of intents related to the following concepts: find, filter, naviga-
tion and help. A description of each one of them follows.
• Find intents. They are related to the action of searching into the
data source specific elements that will represent the starting point for
the consequent conversational exploration.
• Filter intents. They are associated to methods that allow to further
filter a list of elements already presented in the chat.
• History intents. They allow the user to access the history of the
conversation or undoing operations.
• Help intents. They provide general or specific advice and hints during
the navigation.
In order to enable the correct match of these concepts, the NLU model
must be trained with phrases featuring a structure similar to the one of
possible user utterances. In particular, we decided to take advantage of some
words that were more likely to be found in phrases related to some intents
with respect to the others. This way, we could automatize the whole phrase
generation process.
For example, all the training utterances for the find intents start with
a word that can be either “find”, “are there” or “show me”, followed by
a sequence of other words related to the particular elements they refer to.
We will discuss about this pattern composition in the next Subsection, when
talking about the entities, since the majority of phrases cannot be generated
without them.
The only intent category that does not depend on the target database for
which the chatbot is generated refers to the history intents, whose training
phrases are fixed since no entity is used. We provide here an extract of their
possible definition:
• show history → “show me the history”, “the history, please”, “where
are we?”, etc.
• go back → “go back”, “undo, please”, “back”, “undo”, etc.
30
4.3.2 Entities
They are related to the database that the chatbot is going to interface and,
because of this, they depend on the specific outcome of the Schema Anno-
tation phase, as well as on the real values that compose the data system.
While intents are used to understand what is the general purpose of the
received utterance, entities become arguments of the methods that will be
called. Therefore, they cannot be extracted without a previous successful
intent-matching step.
In this context, we used and applied all the modeling primitives con-
cepts described in Section 4.2, in order to define the type of entities that the
chatbot should be able to support. The only exception refers to the Con-
versational relationships introduced in Subsection 4.2.5, which do not have
a corresponding entity to be identified in the utterance, since their usage is
limited to the interaction via buttons and thus they do not pass through the
NLU unit. We will discuss this aspect in the following section, when treating
the issues related to the chat environment.
Let us now describe in details the structure of the training phrases related
to the intents with dynamic content.
Entities for find intents. The intents of this type must contain the Con-
versational Object for which the search is intended and at least one Conver-
sational Value that will be used to filter the results. By looking at Table 4.1
we clearly see that there is a pattern to define and generate these phrases,
whose structure depends on the types and number of the utterance compo-
nents to include. Even if the table only shows generated utterances related
to the customer, we see how the Conversational Qualifiers are coherent with
the related Type:
• “located in” is followed by a WORD whose Value is “Milan”
• “that paid” is linked to a NUM, with Value “20000 ”
What is remarkable is that the phrases have a meaning because the Con-
versational Values are consistent with the rest of the utterance. In fact, while
Conversational Objects and Qualifiers are defined by the Designer, the values
are extracted automatically by the system. This can be made possible by
defining an ad-hoc procedure that connects to the database and executes a
31
Keyword Entities Example
Find* Obj Val Find - customer - Car Models
Find* Obj Qual Val Are there - customers - located in - Milan
Find* Obj Qual Op Val Search - customers - that paid - more than - 20000
Find* Val* Obj Find - Private Company - customers
Find* → Find, Are there, Search
Val* → Values related to ENUM Conversational Types
Table 4.1: Phrase structure for find intents.
series of short queries to extract example Values for each referenced Con-
versational Type. These values will be used in the training phrases, thus
enabling the definition of utterances very similar to the one the chatbot will
receive.
The entity extraction phase, during the conversation, will be able to per-
form the extraction based on the NLU model trained according to these
training phrases. A simple graphical example of how this procedure works
can be seen in Figure 4.6. We can see how in the received utterance the
word “Euros” is present, but this does not limit at all the capabilities of the
system to correctly understand the overall meaning, thanks to an accurate
definition of the training phrases. In fact, since “Euros” is not present in the
examples, it will not be extracted as an entity, but this will not prevent the
correct mapping of the other utterance components.
Find customer that paid 20,000more than
Find customer that paid more than 20,000 Euros
Euros
Object Qualifier
Operator Value
> NUM
PHRASE RECEIVED:
NLU ENTITY EXTRACTION:
Type
Figure 4.6: An example of the entity extraction phase.
32
Entities for filter intents. In case of these types of intents, all the consid-
erations made in the previous paragraph are valid, except for the constraint
on the presence of the Conversational Objects in the phrase. Here that may
be missing, since the filter action can be applied only to a previously re-
trieved list of elements. Thus, including the keyword for the element is not
necessary because the system will be able to extract it from the Context. In
any case, we will analyze these concepts in Subsection 4.4.2.
Table 4.3 graphically summarizes these ideas, providing examples for each
considered pattern.
Keyword Entities Example
Filter* Obj Val Filter - customer - Australian Souvenirs
Filter* Val Show only - Car Models
Filter* Obj Qual Val Filter - customers - that bought - Harley Davidson
Filter* Qual Val Only show - those - located in - Spain
Filter* Obj Qual Op Val Filter - customer - that paid - more than - 40 Euros
Filter* Qual Op Val Filter the ones - different from - Shopping
Filter* Val* Obj Filter - Public Administration - customers
Filter* Val* Show only - Private Companies
Filter* → Filter, Filter the ones, Show only, Only show
Val* → Values related to ENUM Conversational Types
Table 4.2: Phrase structure for filter intents.
Entities for help intents. These intents are quite simple, with respect to
the others analyzed in this Subsection, but still very important. The related
phrases may include the presence of Conversational Objects or not, enabling
the system to give advice on its functionalities and capabilities in general,
or to concentrate into the properties of a particular element; this can be
graphically seen in Table 4.3.
Keyword Entities Example
Help* I need help
Help* Obj Help on - customers
Help* → Help, I need help, Help on
Table 4.3: Phrase structure for help intents.
33
4.4 From the Conversation Model
to the Interaction
Once the conversation model just described in the previous Section is applied
to a given database, the result is a system that can understand natural
language queries on the data source. Let us analyze how this communication
can take place, by solving the problems that arise when trying to enrich the
message exchanges with the final user.
We have to remember that the chat represents a fast and user-friendly
form of communication, but the same does not hold from the system per-
spective: there are a lot of issues that must be considered and solved, which
we discuss in the following.
4.4.1 Interaction paradigm
In order to define how the system supports data exploration, a very important
problem is related to its capabilities to receive requests from the user. Our
system enables the generation of chatbots providing two types of input, here
described.
• Textual input. It consists in the most basic form of communication
and it requires the users to type manually a phrase into the chat sys-
tem. Then, the utterance is received by the chatbot and the natural
language processing unit extracts the corresponding intent and entities,
as described in Section 4.3. The user should always be able to access
this form of interaction and it is fundamental to start the conversation.
• Button based input. As the name suggests, the chat system also
supports the possibility to display buttons among the messages pre-
sented. These items might be labelled with the element or action they
refer to and they may send a customized message to the system when
clicked. Buttons are intended to facilitate the communication, by giv-
ing the user some pre-defined options to navigate among data or quick
hints that remind the possibility to use some commands; we analyze
these features in details in Subsection 4.4.3.
34
4.4.2 Automatic query generation
Once the user makes a request, in form of textual input or by means of a
callback message of a clicked button, the translation procedure takes place.
Select and join. As introduced in Subsection 4.2.3, the system allows
the definition of Conversational Qualifiers for each Primary Conversational
Object, but it does not constraint them to be related to the real attributes
of the database table that element refers to.
Let us take, for instance, the Object customer and the Qualifier that
bought, both visible in Figure 4.4: while the first is the placeholder for
the table customers, the second one refers to the field productName, which
belongs to the table products. Considering this, once the system successfully
recognize the phrase “Find customers that bought an Harley Davidson”, the
corresponding query must be created and executed. The algorithm governing
this translation is shown in Algorithm 1.
We can see from the algorithm how the query is created and step by step
modified according to the inputs, from the basic SELECT and FROM commands,
to the more complex WHERE statement which needs two different phases. The
first part (lines 1-12) is devoted to the composition of the SELECT-FROM clause.
Then, the following part (lines 13-17) refers to the generation of the JOIN
section, in which the various conditions are composed by means of the AND
operator. Then, the last part (lines 18-24) present phrases connected with
the OR conjunction and is related to the value and the operand received.
Now let us spend few words on the methods that appear in the proce-
dure, in order to clarify their utility; besides the “getters”, whose behavior is
intuitive, the “extractor” functions need some clarifications, provided below:
• line 3, extractObjectRelatedQualifiers(Map, obj): it analyzes the Schema
Annotation Map and retrieves all the Conversational Qualifiers that
where defined for the Conversational Object obj ;
• line 5, extractObjectTable(Map, obj): it retrieves the database Table
linked to the Conversational Object obj, by looking at Map;
• line 6, extractTableColumns(DS, t): it analyzes the simplified database
schema DS, in order to extract the column fields composing the Table
t ;
35
Algorithm 1: Find Query Builder
in : the simplified database schema DS, the Schema Annotation
Map, the Conversational Object to find obj, the extracted
Conversational Qualifier qual, the Conversational Value v, the
Conversational Operator op
out: the final query q
1 begin
2 q ← an empty String
3 OQ← extractObjectRelatedQualifiers(Map, obj)
4 if qual ∈ OQ then
5 t← extractObjectTable(Map, obj)
6 TC ← extractTableColumns(DS, t)
7 q ← q + “SELECT” + getColumnNames(TC)
8 q ← q + “FROM” + getTableName(t)
9 ST ← extractSupportTables(Map, qual)
10 foreach st ∈ ST do
11 q ← q + “,” + getTableName(st)
12 end
13 q ← q + “WHERE”
14 JST ← extractJoinSupportTables(Map, qual)
15 foreach (fromTable, toTable) ∈ JST do
16 q ← q +
extractJoinStatement(DS, fromTable, toTable) + “AND”
17 end
18 QC ← extractQualifierColumns(DS, qual)
19 foreach qc ∈ QC do
20 q ← q + extractV alueStatement(DS, qc, v, op)
21 if ac 6= last element of QC then
22 q ← q + “OR”
23 end
24 end
25 end
26 return q
27 end
36
• line 9, extractSupportTables(Map, qual): it checks and extract from the
Mappings Map the tables that supports the Conversational Qualifier
qual ; if qual belongs to the same table the Conversational Object refers
to, then the method returns an empty set;
• line 14, extractJoinSupportTables(Map, qual): it analyzes the Map by
looking at the Conversational Qualifier qual like the previous function,
but it returns a set of pairs (fromTable, toTable) which represent how
the different Tables are related to each other; it may give no result in
case qual belongs to the Conversational Object and no further Table
has to be queried;
• line 16, extractJoinStatement(DS, fromTable, toTable): this function
generate the SQL condition which links the origin Table fromTable to
the destination one toTable, using the correct foreign keys and refer-
ences;
• line 18, extractQualifierColumns(DS, qual): it scans the simplified data-
base schema DS and extracts the column fields of the Table the Con-
versational Qualifier refers to;
• line 20, extractValueStatement(DS, qc, v, op): this function returns the
SQL statement which links the value v to the column qc by means of
the operand op.
If we recap the previous example message “Find customers that bought an
Harley Davidson”, after the execution of the procedure the resulting query
will be:
SELECT C.customerNumber, C.customerName, C.contactLastName,
C.contactFirstName, ...
FROM customer AS C, orders AS O, orderdetails AS X,
products AS P
WHERE C.customerNumber=O.customerNumber AND
O.orderNumber=X.orderNumber AND X.productCode=P.productCode
AND P.productName LIKE ‘%Harley Davidson%’
As we can see, the input phrase gets translated into a complex query by
the system, that autonomously connects the tables that “stands” between
customers and products, which are orders and orderdetails. This is
37
possible thanks to the fact that the Designer tagged these two tables while
defining the Conversational Qualifier that bought, action represented by the
dotted arrows in Figure 4.4. What is remarkable here is that the chatbot
was able to identify autonomously what were the table attributes needed
to perform the join among the different tables.
Moreover, if we pay attention at the last WHERE condition of the query
above, we can see how the Conversational Value “Harley Davidson” is pre-
ceded by the SQL operator LIKE. This is due to the fact that in the entity-
extraction phase the motorcycle name was associated to the Conversational
Type WORD. Again, the system independently creates the query according
to the mapping described in Section 4.2.
Maintaining the Context Previously we analyzed a possible case of Find
intent matching and execution; that was made possible thanks to the dis-
tinction made by the Designer while describing the Conversation Modeling,
as described in Section 4.3. However, what would have happened in case of
a Filter intent?
Let us imagine that the above query had given more than one result and
now the user wanted to filter them based on some other attributes. Again this
is possible thanks to the same Schema Annotation, but from a system point-
of-view accomplishing this may be difficult without a proper strategy. Let us
explain the solution, thanks to another example: once three customers that
bought an Harley Davidson are displayed, according to the previous search,
now the user types “Filter those that paid more than 40,000 Euros”.
In order to interpret this utterance, the system must perform two subse-
quent actions:
1) record the last interaction with the database, with particular attention
to its results and the query that was executed;
2) modify the last interpreted query, updating the last recorded interac-
tion.
If we apply these two operations to an example and update it by adding
the filter condition, the resulting query will be:
38
SELECT C.customerNumber, C.customerName, C.contactLastName,
C.contactFirstName, ...
FROM customer AS C, orders AS O, orderdetails AS X,
products AS P, payments as PAY
WHERE C.customerNumber=O.customerNumber AND
O.orderNumber=X.orderNumber AND X.productCode=P.procuctCode
AND P.productName LIKE ‘%HarleyDavidson%’ AND
P.customerNumber=PAY.customerNumber AND PAY.amount>40000
We can clearly see how the SQL command has been modified to limit the
results even more, thanks to another JOIN and value-condition. Here the
Conversational Type was NUM and this enabled the extraction and usage
of the operator ‘>’.
The procedure that is invoked here is very similar to Algorithm 1, but in
addition to the already identified Conversational Qualifier, it also includes
the ones present in the Context and related to the last Conversational Object.
Repeating all the for-loops related to the Qualifier leads to the creation of the
correct query, but some more attention must be paid in order not to include
repetitions.
In this last example we have seen how recording the last interaction be-
comes useful, if not fundamental; this is actually the Context of the con-
versation and it does not limit to the latest SQL operation and results, but
it can be extended to a list of subsequent previous actions. This may have a
max length after which previous interactions are removed from the records,
and/or a limit in time before it gets deleted if not updated.
It is important to say that the Context is crucial in case the user ac-
cesses a given Conversational Relationship when focusing on a single element.
Again, an example may help the reader to clarify this concept.
If the user has just retrieved a customer, maybe from the previous ana-
lyzed query, and now wants to see the customer’s orders, the system might
display the button “[orders made]” as a possible relation to cross, according
to the annotation in Figure 4.4.
Once the button is clicked a translation procedure starts, which follows
the rules described in Algorithm 2.
This program creates the final query following the same steps of Algo-
rithm 1, but has four main differences with respect to the former:
39
Algorithm 2: Join Query Builder
in : the simplified database schema DS, the Schema Annotation of
the database Map, the Context C, the Conversational
Relationship r
out: the final query q
1 begin
2 q ← an empty String
3 obj ← extractContextObject(Map,C)
4 OR← extractObjectRelationships(Map, obj)
5 if r ∈ OR then
6 t← extractRelationTable(Map, r)
7 TC ← extractTableColumns(DS, t)
8 q ← q + “SELECT” + getColumnNames(TC)
9 q ← q + “FROM” + getTableName(t)
10 ST ← extractSupportTables(Map, r)
11 foreach st ∈ ST do
12 q ← q + “,” + getTableName(st)
13 end
14 q ← q + “WHERE”
15 JST ← extractJoinSupportTables(Map, r)
16 foreach (fromTable, toTable) ∈ JST do
17 q ← q +
extractJoinStatement(DS, fromTable, toTable) + “AND”
18 end
19 PC ← extractPrimaryColumns(DS, c)
20 foreach pc ∈ PC do
21 v ← extractPrimaryV alue(pc, obj)
22 q ← q + extractV alueStatement(DS, pc, v,=′)
23 if r 6= last element of PC then
24 q ← q + “AND”
25 end
26 end
27 end
28 return q
29 end
40
1) the Conversational Object is not an input, but is dynamically extracted
from the Context with the function extractContextObject(Map, C) - line
3;
2) the target Table is not related to the Conversational Object, but it
is the destination Table of the Conversational Relationship r ; this is
possible thanks to the method extractRelationTable(Map, r) - line 6;
3) the Conversational Value is not given as input, but is extracted for each
primary column related to the Conversational Object; this is done by
the method extractPrimaryValue(pc, obj) - line 21;
4) all the conditions in the WHERE statement are combined by the AND
conjunction and present the only possible operator ‘=’.
If we recall the previous example and assume that the customer in the
Context has customerNumber equal to 123, the system translates the request
into the following query:
SELECT O.orderNumber, O.orderDate, O.status, O.customerNumber
FROM orders AS O, customers AS C
WHERE O.customerNumber=C.customerNumber AND
C.customerNumber=123
Here, as before, the program autonomously understands what are the
JOIN attributes and that customerNumber is the identifier for the table
customers. Moreover, the SELECT is not executed on the customers, but
on the table orders, as the relation specified.
Error handling. Once there is an algorithm to convert phrases into queries,
the problems are not all completely solved. In fact SQL, like all program-
ming languages, is constrained to a strict syntax, which admits no typos
and is bound to a specific punctuaction. When dealing with a chatbot, it is
not possible to require such a precision from the user, since one of the main
strength of this kind of system is the ability to accept and understand natural
language inputs. Moreover, besides the likely probability of typo errors and
the consequent user frustration derived from their system refusals, words can
be abbreviated, nouns may be different when their plural is expressed, verbs
may be irregular in their different conjugations, etc.
41
It may be unfeasible to store, inside the chatbot, all these possibilities for
each saved lemma; then, a possible solution may be to include some algo-
rithms able to associate the given word or set of words to the understandable
correspondence. In the next Chapter, which is related to the Architecture
of the system, we will present a procedure that can be used to tackle this
problem; however, for the moment, let us consider an intuitive representation
of problem and solution by looking at Figure 4.7.
Find csstamer Car Models
Object?
Value
WORD
NLU ENTITY EXTRACTION: COMPARISON:
Objects
customer
order
client
product
TYPO RESOLUTION:
csstamer
csstamer VS
customer
Type
Figure 4.7: The typo resolution procedure.
Here we can see how the word csstamer is correctly extracted as a Con-
versational Object by the NLU unit, but it needs to be checked against the
possible objects that the system supports. If the word is “similar enough”
to an existing one, customer in this case, than the conversion is performed
and the related query can be then executed.
4.4.3 Chatbot responses
We have described the process undertaken to govern the conversation, from
the utterances receipt to their transformation into queries. We have also
introduced the idea that stands behind the Context and why it becomes so
important in our system.
Now, let us describe in details how the chatbot may answer to the requests
it receives, after the above translation phase.
Natural language utterances. A possible approach to make the chatbot
give human-like feedback is to have a set of pre-defined utterances available
for any occasion. This allows the system to communicate with its users in a
richer form, guaranteeing complete explanation to the results found.
42
Of course, these phrases do not need to be static, but they may need to
be preceded by a slot-filling phase, based on their execution Context. For
example, after the completion of a search on a database over the customers,
the system may introduce the values found by a simple but effective phrase:
“Hey, I found 12 elements of type customer in the data system!” The struc-
ture of the utterance may be applied to each type of results (i.e., customers,
products, orders, ...) and the feedback is quick and easily understandable.
The example phrase just described may be extracted by the system from
the string: I found [] elements of type [] in the data system!.
This procedure can be applied easily to a variety of different utterances,
leading to the design a more descriptive chatbot.
Result visualization. In Section 4.4.2 we analyzed how input phrases
can be transformed into queries, let us now consider how the results of these
database interactions can be prompted to the user. We can distinguish two
possible cases: Single result and Multiple results.
When we are in the first condition, the extracted element can be shown as
is in the database, after a proper introduction, i.e., “Here is what I know about
this customer:”. In Figure 4.8 there is a visualization example that shows how
a possible customer can be displayed in the chat. Moreover, we can see how
the bot presents the buttons, as introduced in Subsection 4.4.1, which permit
the navigation across the database along the Conversational Relationships
they stand for; in the Figure they are Products bought, Payments made and
Orders made.
If we are in the situation in which multiple results have to be displayed,
instead, we have to guarantee their readability. This can be achieved by the
application of two ideas:
1) displaying only key values for each result;
2) controlling the layout of the outcomes.
In order to apply the first concept, the system needs to know for each ta-
ble in the database what are its main attributes, i.e., those that summarize
and somehow differentiate the various rows. Thus, the Designer can tag the
desired columns during the Schema Annotation phase, as described in Sub-
section 4.2.2; for example the table customers can have as Display Attributes
only the keys customerName, contactLastName and contactFirstName.
43
Here is what I know about this customer:- customerName: Car Models- contactLastName: B. Goode- contactFirstName: John - phone: 1234567890- addressLine1: Via Roma, 1- city: Milan- state: None- country: ItalyIf you want more information, I can tell you:Products bought Payments madeOrders made
Figure 4.8: The response of the chatbot to display a single result.
Layout control, instead, is intended to limit the size of the results dis-
played at a time, giving the user the possibility to explore further elements
only if requested. In case of many results, in fact, a single response which
shows them all may be counterproductive, since it may represent a huge and
unmanageable quantity of information.
To facilitate the user interaction, moreover, multiple results may be dis-
played as buttons linking to the elements they refer to. In Figure 4.9 we can
see how these ideas can be applied, thanks to a simple use case over elements
of type customer.
Let me check...Et voilà! I found 8 values:
Car Models - B. Goode JohnJJ Store - Browns JohnAmerican Souvenirs Co. - Wayne John
Select the one you are interested in or clickthe last button to see more.
SHOW MORE
Shown results from 1 to 3 of 12
Figure 4.9: The response of the chatbot to display multiple results.
44
Help and Fallback. Exhaustive responses are essential for the correct ex-
ploration of data, but we must consider also the case when the user may lack
some information to exploit the different features of the chatbots. This can
be achieved by providing help messages and buttons which briefly describe
these characteristics. For instance, at the beginning, the user may write Help,
leading to a response which describes the chatbot capabilities.
Another case may be to give support on a specific element; upon re-
ceipt of the message Tell me more about customer, for instance, the system
may present a list of examples that the user can use to access that element,
describing its Conversational Qualifiers, e.g. “find customer located in ...”.
Another help, instead, may give the user some hints to filter multiple results,
as an additional advice to shrink their size.
Moreover, even though in the previous Subsection we have seen how the
chatbot handles typos and writing inaccuracies, somehow that procedure
does not solve the problem. In these situations the system may provide an
exhaustive answer based on which concepts it was able to understand and
what is, instead, the missing information. However, in case no intent is
matched, the system may just say something like “I did not get that :( ”
4.4.4 History navigation
If you want, you can click on a button to go backin the history of the conversation.
Car Models - B. Goode John Currently you are focusing on:
...selected from:A list of type "customer" ...found with attribute(s) "with contact John".
Figure 4.10: An example of conversation history visualization.
In Subsection 4.4.2 we introduced the concept of Context and we under-
lined its fundamental role when dealing with query refinement and execution
of queries. Maintaining a record of previous actions, however, can be ex-
ploited in a different way: allowing the user to navigate backwards and reset
the Context to a past moment. This becomes very useful after the execution
of an unwanted action or when resuming a previously crossed object can be
45
more useful than repeating the requests needed to reach those items.
In order to have a descriptive history navigation, we may use buttons that
link to the objects they refer to, as well as some messages which summarize
the actions that were taken to get to them. An example of application is
described in Figure 4.10, where we can see how, in a 2-step conversation,
and how the system helps the user go back.
Another tool that could be made available is the option to move back-
wards to the previously seen objects. This is corresponds to the undo com-
mand and it could be accessed by an ad-hoc Help intent, related, for instance,
to the utterances “go back” or “undo”.
46
Chapter 5
System Architecture and
Implementation
This Chapter describes our prototype of Chatbot for Data Exploration, in
terms of architecture and implementation decisions. We will also discuss
about some tools that helped us during the development of the system.
5.1 Technologies
This Section is dedicated to the choices that we made, in terms of the adopted
technologies, to create a system able to fulfill the requirements described in
Chapter 3 and consistent with the design decisions illustrated in Chapter 4.
5.1.1 RASA
As we said in Section 2.4, there is a great variety of libraries online that
help developers create a conversational agent, each one with strengths and
disadvantages. In particular, for our system, we decided to use the tools
provided by RASA.
During the first phases of the design of our system, RASA Core was the
best choice for what concerns the dialogue management, as it offers high
customizable options and it is deployable in a local machine. However, after
the first initial tests, we decided not to use this framework and we preferred
to create an ad-hoc system and thus handle all the logic upon which the
chatbot should have based. This was due to the fact that, besides all the
features offered by this open source tool, in order to completely control the
Context of the conversation, we would have used only a small part of the
framework, unnecessarily overloading the dependencies of the system.
Instead, we took advantage and widely applied the second technology
above mentioned, i.e., RASA NLU. As explained in Section 2.4, the main
reasons to prefer this framework include the possibility to run the tool locally
and the high customizability of the training model.
Let us discuss in details, then, all the tools belonging to this framework
that we decided to apply in our system.
Training phrases in a markup language. In order to make the chatbot
understand the received messages, RASA NLU supports the definition of the
so called training data. It is a file, or a collection of files, in a markdown
or json format, in which the developer can define the intents the system
will be able to match and provide a set of example phrases related to each
one. These utterances may also contain the entities, marked according to the
format specifications, which will be used in the entity extraction phase.
The json extract in Listing 5.1 is an example of what the training data
file may contain. There, the target intent is find object by qualifier for
which an example utterance is provided: “are there customers located in
France?”. We can also see how the entities are listed in the corresponding
json field, each related to the represented value.
Define and train the Machine Learning NLU model. The training
data described above is used as input to train the NLU model, i.e., the
structure that will guide the intent matching and entity extraction procedures
during the conversation. RASA NLU gives the possibility to choose these
components in order to find the best solution for each problem, or even to
define an ad-hoc one if some additional property is needed.
These components are executed one after another at each incoming mes-
sage, in a so-called processing pipeline, and they may be needed for entity ex-
traction, for intent classification, pre-processing, and other purposes. RASA
NLU also provides some templates, that are pre-configured lists of compo-
nents, among which two of them are the most used:
• spacy sklearn1, which uses pre-trained word vectors from either GloVe2
1https://rasa.com/docs/nlu/components/#intent-classifier-sklearn2https://nlp.stanford.edu/projects/glove/
48
{
"text": "are there customers located in France?",
"intent ": "find_object_by_qualifier",
"entities ": [
{
"end": 19,
"entity ": "object",
"start ": 10,
"value ": "customers"
},
{
"end": 30,
"entity ": "qualifier",
"start ": 20,
"value ": "located in"
},
{
"end": 37,
"entity ": "word",
"start ": 31,
"value ": "France"
}
]
}
Listing 5.1: A json extract of the training data.
49
or fastText3 and it is specific for each language chosen for the chatbot;
• tensorflow embedding4, that does not use any pre-trained word vec-
tors, but instead fits these specifically for the dataset in input.
Since the objective of our investigation is to create a system able to
interact with a domain-specific dataset, designed for the information stored
in the target database, we preferred to use the tensorflow embedding
pipeline. Its components are:
1) tokenizer whitespace, which creates a token for every whitespace
separated character sequence;
2) ner crf, that implements conditional random fields to do named entity
recognition;
3) ner synonyms, which maps synonymous entity values to the same
value;
4) intent featurizer count vectors, that creates bag-of-words repre-
sentation of intent features, needed for the next component;
5) intent classifier tensorflow embedding, needed to embed user in-
puts and intent labels into the same space.
Dynamic intent matching and entity extraction. As we discussed
in the previous chapter, defining and training an NLU model is needed to
perform intent matching and entity extraction. When the system receives
an input message, this one traverses all the components of the pipeline and
a particular object is returned, which specifies what the NLU unit actually
understood of the received message.
Let us consider a trained NLU model in which the input training data in-
cludes the json excerpt described in Listing 5.1. If the model is well defined,
after receiving the phrase “Find customers located in Italy”, for instance,
it should be able to output an object having find object by qualifier as
intent and a correct mapping of the entities, i.e. object→ customers, qual-
ifier → located in, word → Italy.
3https://fasttext.cc/4https://rasa.com/docs/nlu/components/#intent-classifier-tensorflow-embedding
50
%[ find_object_by_qualifier ](’training ’: ’5’)
~[find] @[object] @[qualifier] @[word]
~[find]
find
search
look for
@[object]
customer
client
@[qualifier]
located in
@[word]
France
Italy
England
Listing 5.2: An example of input document for Chatito.
5.1.2 Chatito
Chatito5 is an open-source tool which helps generating datasets for training
and validating chatbot models using a minimalistic Domain-Specific Lan-
guage. It allows the definition of a document in which the developer can
specify grammar rules for each intent, as well as the entity keywords with
the related possible examples.
Listing 5.2 shows an example of the content of this file, in which the target
intent is find element by qualifier.
Giving this document as input to the Chatito framework, the latter gen-
erates five utterances, as requested in the training field, by following the
requested specifications. For example, it may output the following phrases
(here represented without the entities notation, which instead is present):
• look for customer located in Italy
• find customer located in France
• find customer located in England
• search client located in Italy
• find customer located in France
5https://github.com/rodrigopivi/Chatito
51
We decided to include Chatito in the design process of our chatbot, in
order to generate the training data starting from the Schema Annotation.
Besides the fundamental feature of creating large datasets of domain specific
utterances, it was convenient to use the framework for two reasons:
1) It gives the possibility to get the output in a format compatible with
RASA NLU training data specifications;
2) It can be installed and run locally.
5.1.3 Telegram
For what concerns the chat interface, we decided not to create an ad-hoc
system, but we preferred to take advantage of the Telegram Bot API6. In
particular, we logged in the Telegram messaging platform and we contacted
the BotFather7 account, which guided us throughout the few steps towards
the creation of our chatbot interface, that we named DataExplorerBot.
After receiving the authorization key, we managed to connect with our
Bot through the dedicated API; this way, we could receive and send mes-
sages to the users that were interacting with DataExplorerBot directly over
Telegram.
We decided to use this messaging platform, among the others introduced
in Subsection 2.3.2, because of its ease of use and set up. Moreover, Telegram
enables the developers to create custom buttons, besides the possibility to
completely manage the textual conversation with its user, so we decided to
take advantage also of this functionality.
However, as we will discuss in the next Sections, we chose not to bound
the internal management of the conversation to the Telegram API format
specifications, but we preferred to implement an adapter module in charge
of translating the messages for the messaging platform. This way, we could
achieve a good level of scalability, in terms of integration with other chat
interfaces.
6https://core.telegram.org/bots/api7https://telegram.me/botfather
52
5.2 Framework Architecture
Our architecture is based on the three-tier model, which ensures high scal-
ability and provides modularization of the different technologies based on
their application in the system. The tiers, or layers, include: a Data tier, an
Application tier and a Presentation tier. In order to describe their role
and their interactions within the global the architecture, we will refer to the
Figure 5.1 which presents an overview of the system.
Chatbot Architecture
Conversationmanagers
Resources
Applicationmanagers
Databaseconnectors
Chatinterface
Chatinterface
Chatinterface
NLU managers
Database
Presentation Tier Application Tier Data Tier
Figure 5.1: An overview of the Framework Architecture, based on the three-tier model.
Data tier. This layer includes the database where the information is stored
and retrieved, according to the requests made through the system. It may
contain any kind of data, completely unrelated with the chatbot foundations.
However, the structure and hierarchy of these records will be used by the
Designer to create the Schema Annotation and by the system itself to present
their values, as described in Section 4.2 and Section 4.4, respectively.
Our solution does not rely on the database to store the internal resources
of the system, because of the small size of such elements in our applications,
but the system may take advantage of another persistent data storage for
this usage. This option may help to reduce the computational load on the
application side, but might introduce unwanted latencies to use these infor-
mation during the conversation. A feasible alternative could be to place some
of these files in the data system, while maintaining the most used ones at the
application side, but we will discuss about this in the next Subsection.
53
One final comment: the database server may be placed in the same ma-
chine of the logic environment of the system or it could stay on a remote
computing apparatus. Again, this choice impacts on the scalability, on one
side, and on the performance of the overall architecture on the other. In any
case, this hardware based decision must be made considering the maximum
size of the target data storage as well as the number of final users.
Application tier. This layer represents the core of the system and includes
the modules needed in both the design process and the conversation execu-
tion. As Figure 5.1 shows, it includes five interconnected modules in it. We
will describe their internals in details in the next Subsection. For the moment
let us pose our attention on the arrows that exit from the external module:
they represent the communication channels with the modules of the other
two layers. Inside the system, the modules that handle these connections
are the Database connectors and the Conversation managers, which
are responsible for the interaction with the data and the presentation layer,
respectively. These two components must be designed with the purpose of al-
lowing both remote and local connections, in accordance to the architectural
choices regarding the setting of the three tiers.
Moreover, always referring to the same Figure, we can appreciate the het-
erogeneous shape of the different modules. In the next Subsection, we will
describe in details their properties and roles inside the architecture, high-
lighting their differences and justifying the choice to represent them in a
varied graphical format.
Presentation tier. In our system, we decided to take advantage of exist-
ing external interfaces for what concerns the front-end of our architecture. In
fact, as we already mentioned in Chapter 2, there is a great number of online
messaging platforms which already integrate the possibility to create a chat-
bot in their framework and then control its logical behavior by means of an
API. This is exactly the direction that guided our design choices when dealing
with this part of the architecture and it enabled to have a nice-looking and
easy-to-use interface just by setting up the right connector to communicate
with the external API.
However, creating an ad-hoc chat environment, for example in a Website
or even locally where the application runs, remains possible but it may need
the introduction of another connector inside the system or a definition of a
54
dedicated API.
Moreover, we can see from the same Figure 5.1 that the geometric shapes
representing these chat interfaces have a logo sticked on them, each different.
They stand for Facebook Messenger, Slack and Telegram, and suggest how
the system may support different technologies belonging to the presentation
tier, contributing to the scalability and extensibility of the overall architec-
ture.
5.3 Chatbot Architecture
In this section we discuss about the architectural choices that lead to the
definition of the actual internal structure of the chatbot. In order to make
the description as clear as possible, we will refer to Figure 5.2 which shows
how each module is organized inside.
Chatbot Architecture
Conversation managers
Application managers
Database connectors
NLU managers ResourcesNLU managers
Patterns
NLU model
NLU data - element+attr+attr
- attr+element
## intent
Schema
Annotation
element
Schema
NLU library
Trainer
Writer
Extractor
Phrasegenerator
library
Resolver Broker Parser
Database connector libraryCaller Actions Utils
Messages Buttons
Connectors library
Contextmanager
Connectors
Figure 5.2: The Chatbot Architecture in details.
Conversation managers. This module is responsible for the receipt and
dispatch of the messages sent by the end-user. Here, we can distinguish the
55
Connectors component, in charge of communicating with the external chat
interfaces and the Patterns block. The latter is composed by the Messages
and the Buttons elements, which provide the responses as simple text or as
dynamic buttons.
Application managers. It is the core module of the system, as it contains
all the logical components needed to compute a response upon the receipt
of a message. Here we can find the Caller unit, which retrieves the Context
by interacting with the Context manager and calls the right method of the
Actions module. The relation between the message received and the action
to call is made clear thanks to the Extractor, a component belonging to the
NLU managers, which translates the utterances into an intent and a set of
entities. These two elements, as well as the Context, represent the arguments
of the method that gets finally invoked.
Moreover, the Caller is responsible of concurrency, as it creates a new
thread in charge of handling the new request until the response is sent. This
allows the system to manage more connections at a time, each with its own
Context and session state.
Inside the Actions module, instead, the engine of the chatbot lies: here
the actions define different execution paths, based on: the entities extracted,
the actual conversation Context, the query results and in general all the
parameters defined during the design process. The Utils unit is then needed
to leverage the load and complexity of the Actions module, as it contains
simple helper methods.
NLU managers. This unit contains all the needed components of the
chatbot Natural Language Understanding sections, as well as the libraries
to support these technologies. Since the system supports both the design
and the usage of the conversation, the items in this group play different roles
depending on two execution moments.
The Writer module is needed to translate the Schema Annotation made
by the Designer into the NLU data, both contained in the Resources group,
as can be seen by Figure 5.2; this is done by means of the Phrase generator
library, whose implementation details are described in Section 4.3.
Instead, the Trainer unit is needed to create a model able to support the
intent matching and entity extraction phases of the final conversation and it
uses the features provided by the NLU library. The input of this procedure
56
are the generated phrases contained in the NLU data component, while the
output is the NLU model ; they are both visible in the Resources section.
The Extractor component is used during the conversation phase and it
is needed to analyze the utterances received and execute the procedures of
intent matching and entity-extraction.
Database connectors. As the name suggests, this unit includes the com-
ponents needed to interact with the database system. This justifies the pres-
ence of the Database connector library, which provides all the methods and
drivers to communicate with the DBMS.
The Parser component plays a fundamental, but relatively “short”, role
in the design process of the chatbot, as it is needed only in the first part:
it analyzes the database and extracts a simplified version of its structure,
creating the Schema resource.
The Resolver functions, instead, are invoked by the Actions component
in order to execute the queries on the database starting from the intent
and the entities. However, this module does not interact directly with the
database, but it is needed to extract from the Schema Annotation resource
the information about the tables upon which the final query will be executed.
The task of building the query is left to the third unit, that is the Bro-
ker. This construction phase is made based on the Schema Annotation and
the Schema files, needed to correctly parametrize the resulting command
prototype. The query, then, is directly executed on the database system.
Resources. We already described how the elements inside this group are
used and we discussed the alternatives to place some or all of them in a
persistent storage.
The NLU model and NLU data are files used by the Natural Language
Understanding control part, while the Schema document is a simplified rep-
resentation of the database.
The Schema Annotation resource, instead, represents the output of the
modeling phase and it is the main supporting document of both the genera-
tion and execution phases of the chatbot.
The files are both saved in a json format, for which an extract of their
content can be seen in Listing 5.3 and Listing 5.4, respectively.
57
{
"customers ": {
"column_list ": [
"customerNumber",
"customerName",
"contactLastName", "contactFirstName",
"phone",
"addressLine1", "addressLine2",
"city",
"state", "postalCode", "country",
"salesRepEmployeeNumber",
"creditLimit"
],
"primary_key_list ": [" customerNumber "],
"references ": {
"employees ": {
"foreign_key_list ": [" salesRepEmployeeNumber "],
"reference_key_list ": [" employeeNumber "]
}
}
},
...
}
Listing 5.3: A json extract of the simplified database.
5.4 Implementation Tools
The majority of the scripts to design the logic of the application are in Python
and we used Python 3.7 as Interpreter, taking advantage of some modules
belonging to the Python standard library.
To parse the database we used the sqlparse module, along with mysql-
connector to execute queries on the database which we decided to run locally
in a MySQL v5.7.16 server.
We used the rasa nlu Python module along with the tensorflow and
sklearn crfsuite libraries, for the Natural Language Understanding tools.
The Python unit nltk is used for the computation of the edit distance to
perform typo resolution. For the phrase generation part we used Chatito8
which freely available online and needs Node.js.
To interface the system with the Telegram API we used telepot, a
Python library which helps simplifies the execution of the remote requests to
the messaging platform.
8https://github.com/rodrigopivi/Chatito
58
As we already mentioned, the resources are saved in a json format. The
Appendix B reports an example of the Schema Annotation for the classic-
models9 database. The parsed schema of the same database is reported in
Appendix A.
[
{
"object_name ":" customer",
"aliases ": [" customers", "client", "clients"],
"type": "primary",
"table_name ": "customers",
"display_attributes ":[
{
"keyword ": "",
"columns ": [" customerName "]
}
],
"qualifiers ": [
{
"keyword ": "located in",
"type": "word",
"columns ": ["city", "state", "country "]
},
...
]
"relations ": [
{
"keyword ": "payments made",
"object_name ": "payment",
"by": [
{
"from_table_name ": "customers",
"from_columns ": [" customerNumber "],
"to_table_name ": "payments",
"to_columns ": [" customerNumber "]
}
]
},
...
]
},
...
]
Listing 5.4: A json extract of the database Schema Annotation.
9http://www.mysqltutorial.org/mysql-sample-database.aspx
59
Chapter 6
User Study
6.1 Comparative Study: General Setting and
Research Questions
We carried out an experimental study to understand to which extent the
chatbots generated through our design framework would support users in
accessing and exploring the content of a database. We considered a sample
database referring to a company’s customers (very similar to the example
database used throughout the previous chapters). The database schema is
reported in Figure 6.1.
On top of the customer database we developed a chatbot following the
modeling methodology described in Chapter 4. This means that through
the chatbot some tables were directly (primary tables) or indirectly (sec-
ondary tables) accessible by means of natural language requests. Other tables
(crossing relationships) would not be “visible” but they would be anyway ex-
ploited to provide links to navigate across the tables they put in relationship.
Through the study we then compared the performance and the satisfaction
of database experts when querying the database with two paradigms: the
DataExplorer chatbot, enabling natural language queries, and the SQL com-
mand line supporting SQL queries on top of the same database. Through
the comparison we wanted to identify pros and cons of using the DataEx-
plorer chatbot for exploring and retrieving data. More in general, we aimed
to identify guidelines for the design of chatbots for data exploration.
The choice to involve database experts is due to our intention to assess
whether, despite some intrinsic limitations and simplifications (e.g., making
customers
customerNumberPK
customerNameNN
contactLastNameNN
contactFirstNameNN
phoneNN
addressLine1NN
addressLine2
cityNN
state
postalCode
countryNN
salesRepEmployeeNumberFK
creditLimit
payments
customerNumberPK, FK
checkNumberPK
paymentDateNN
amountNN
orderdetails
orderNumberPK,FK1
productCodePK,FK2
quantityOrderedNN
priceEachNN
orderLineNumberNN
orders
orderNumberPK
orderDateNN
requiredDateNN
shippedDate
statusNN
comments
customerNumberFK, NN
products
productCodePK
productNameNN
productLineFK, NN
productScaleNN
productVendorNN
productDescriptionNN
quantityInStockNN
buyPriceNN
MSRPNN
employees
employeeNumberPK
lastNameNN
firstNameNN
extensionNN
emailNN
officeCodeFK, NN
reportsToFK
jobTitleNNoffices
officeCodePK
cityNN
phoneNN
addressLine1NN
addressLine2
cityNN
state
countryNN
postalCodeNN
territoryNN
productlines
productLinePK, FK
textDescription
htmlDescription
image
Figure 6.1: The classicmodels database schema used in the evaluation tests.
62
some tables not visible) deriving from the conversation design, the chatbot
query paradigm would enable users to explore data with performance and
satisfaction comparable to the ones achieved through the SQL language. We
also wanted to understand to which extent experts in database would find
the conversational paradigm interesting and useful.
It is worth remarking that the goal of our research is not to define a new
query language with the same expressive power as SQL; rather, we aim to
identify abstractions for the generation of conversations for data exploration.
For its nature, the SQL language is much more expressive of the chatbot
language. We anyway selected the SQL command line as a rich baseline
that would allow us to gather indications on how to improve the conversa-
tion design and make the resulting conversation paradigm useful for data
exploration.
We identified some tasks that could be tackled with both the systems.
To accomplish them, the participants would be allowed to access and query
directly all the database tables when using the SQL command line; when
using the chatbot they would have a direct visibility on primary and sec-
ondary tables, while they would exploit crossing relationships tables only in
an indirect way, i.e., in form of links to navigate to the tables they put in
relationship.
The research question driving the study was:
What is the difference between the considered systems in terms of the us-
ability of the paradigm for exploring the database content and retrieving
specific information?
6.2 Participants and Design
We recruited 15 participants (2 females, 13 males) among the students of
the Master Degree in Computer Engineering at Politecnico di Milano. Their
mean age was 24.86 years (SD = 2.28, min = 23, max = 32). As resulting
from the demographic questionnaire that they filled in at the beginning of
the study, participants had an excellent experience in IT ( x = 8.07, SD =
0.61, min = 7, max = 9), a good experience in using chatbots (x = 5.92,
SD = 0.99, min = 5, max = 8), and an excellent experience in using SQL
language (x = 7.42, SD = 0.94, min = 5, max = 9). The Likert scales used
63
to assess such skills ranged from 1 to 10 (1 very low - 10 very high).
The controlled experiment adopted a within-subject design, with the sys-
tem as an independent variable and two within-subject factors, i.e., chatbot
and SQL. Each participant used the two systems in sequence. In order to
minimize learning effects [14], based on the Latin-Square design they used the
system in a different order by considering permutations of the two systems
and of the experimental tasks.
6.3 Tasks
With each system, the participants performed 6 tasks that can be classified
in three types:
• Type 1 consisted in understanding the structure of the database and
the content it stores. It included 2 tasks, purposely defined to verify
to which extent users were able to understand which information could
be accessed.
• Type 2 included 2 tasks to verify if the participants were able to identify
specific data items through selection and filtering.
• Type 3 asked the participants to retrieve specific data items by means
of joins. It included 2 tasks asking the users to navigate the database
through the available relationships.
Some tasks were split in 2 or 3 parts, in order to make clearer the objective
of the user activity. With reference to the database reported in Figure 6.1,
the final task list was:
• Task 1 (Type 1 ):
a) Try to understand what are the tables that
compose the database
b) Try to understand what are the attributes
more relevant for each one of the previous
tables
• Task 2 (Type 1 ): Try to understand how the different tables are related
to each other.
• Task 3 (Type 2 ): Select the customers (the name) that have, as con-
tact, a person whose name is Diego.
64
• Task 4 (Type 2 ):
a) Try to select the customers (the name) that
are placed in Australia.
b) How many of them did you find?
c) Now find (the name) the one whose contact
person is named after Peter.
• Task 5 (Type 3 ):
a) Try to find in which city the sales represen-
tative of the client Euro+ Shopping Channel
has the office
b) Is it the same of the one where this customer
is located?
• Task 6 (Type 3 ): Find the payments (for each, the amount of money)
of the clients that paid more than 110000 Euro.
For each task, users had a maximum time of 4 minutes (240 seconds). In
accordance with the within-subjects design, each participant performed 12
tasks that required executing a total of 170 queries.
6.4 Procedure
The study took place in a quiet university room where the study apparatus
was installed. Two HCI researchers were involved: one acted as an observer,
the other as a facilitator. Two laptops with a 15-inch display provided with
an external mouse were available. The study lasted 1 day. Each study session
involved two participants and followed the same procedure. First, a 5-minute
presentation was given by the facilitator to introduce the participants to the
goal of the study and what they had to do. Then, participants were asked to
sign a consent form for video-audio recordings and photo shoots, and to fill
in the questionnaire for collecting demographic data and their competences
on IT in general and on chatbot and database in particular. The facilitator
introduced the systems to be used demonstrating how to perform queries on
a sample database different from the one the participants should refer to.
Then every single participant was invited to perform the experimental tasks.
The participant read aloud the task text and then started it.
At the end of all the experimental tasks with each system, the participant
filled in an online questionnaire about the system used. Then, another online
65
questionnaire was administered at the end of the session. It asked to rank
the two systems on the basis of their usefulness, completeness, and ease
of use, and to choose which system the participants would like to use in
their activities. The procedure was preliminarily assessed by a pilot study
involving two further participants.
6.5 Data Collection
The ISO 9241-11 [19] standard defines usability as “the extent to which a
product can be used by specified users to achieve specified goals with effective-
ness, efficiency and satisfaction in a specified context of use”. The ISO/IEC
9126-4 [20] recommends that usability metrics should focus on:
• Effectiveness : The accuracy and completeness with which users achieve
specified goals.
• Efficiency : The resources expended in relation to the accuracy and
completeness with which users achieve goals.
• Satisfaction: The comfort and acceptability of use.
Both qualitative and quantitative data were collected to analyze these
metrics on two systems. We considered the set of notes taken by the ob-
server during the task execution, the video recorded during the different
study sessions, the questionnaire answers, the free comments participants
provided during the study. Regarding qualitative analysis, two researchers
transcribed the audio/video recordings, the observer’s notes, and the ques-
tionnaire open questions. A thematic analysis was carried out on these data.
Then, the two researchers independently double-checked the results. The ini-
tial reliability value was 91%, thus the researchers discussed the differences
and reached a full agreement [6]. The researchers built an excel file reporting
for each query performed by each user the following data: user ID (from 1
to 15), type of system (SQL, Chatbot) task ID (T1 - T6), time (in seconds),
task success (success - partial success - failure), and significant comments.
Time has been considered to analyze efficiency, while task success was used
to analyze effectiveness.
Regarding user satisfaction, two online questionnaires were administered
during the study. The first questionnaire, organized in 4 sections, was used
to evaluate each system. The first section included the System Usability
66
Score (SUS) [7], which gives an overview of the user’s subjective usability
evaluation of a given system. It is a closed-ended questionnaire encompassing
10 statements on an ordinal 5-point Likert scale from “strongly disagree” to
“strongly agree”. This questionnaire was chosen for its reliability, brevity
and wide adoption [4].
The second section included the NASA-TLX questionnaire, used as “Raw
TLX” (Hart, 2006). It is a 6-item survey that rates perceived workload in
using a system through 6 subjective dimensions, i.e., mental demand, physical
demand, temporal demand, performance, effort, and frustration, which are
rated within a 100-points range with 5-point steps (lower is better). These
ratings were combined to calculate the overall NASA-TLX workload index
[16].
The third section had four questions about:
1) The easiness of exploring and understand the database content, related
to Type 1 tasks,
2) The easiness of retrieving specific data items, related to Type 2 and
Type 3 tasks,
3) The easiness of the commands to be used, and
4) The quality of the presentation of query results.
Likert scales ranging from 1 to 10 (1 very low - 10 very high) were used
for these questions. This section ended with three open questions on the
advantages and disadvantages of the system and on possible suggestions to
improve the systems.
The second questionnaire was administered at the end of each partici-
pant’s session. It asked to rank the systems based on their utility, complete-
ness and ease of use (from 1 to 2, 1 is the best), and to explain the reasons
for the expressed order. Paired-samples t-test was adopted to analyses SUS,
NASA-TLX results and task time. Wilcoxon signed-rank test was used to
analyze task success rate and system ranking.
6.6 Usability Analysis
In order to answer the research question, namely, if there is any difference
in terms of usability in exploring information, we analyzed the user’s per-
formances following the three usability metrics, i.e., effectiveness, efficiency
67
and satisfaction. In the following, we report details about how we measured,
analyzed and compared these metrics.
6.6.1 Efficiency
System efficiency was analyzed by considering the time participants spent
in executing tasks with the two systems (Chatbot x = 107.48, SD = 79.31;
SQL x = 117.57, SD = 78.93). The paired-samples t-test test does not
highlighted any significant difference among these systems (t(1110) = −.968,
p = .335).
A deeper analysis was carried out focusing on the time the users spent
on each task. Table 6.1 summarizes the user performances for each task and
the results of the paired sample t-test. It is clear that no tasks revealed
significant differences since their p-value is always greater than 0.05.
For Task 2, the table does not report the results of the t-test as all the
users took the maximum time (240 seconds), thus the comparison was not
significant. Task 2 indeed did not require the users to output some detailed
answers; rather it required an extensive exploration of the database aimed to
identify the relationships among the different tables. When using the SQL
command line, 3 out of 15 users were able to identify all the relationships,
11 users partially identified the relationships (x ≥ 2) and only 1 completely
failed. With the chatbot, 9 users partially identified the relationships that
were visible, while the other 6 failed. The task, however, was useful to observe
the strategies adopted by the users to explore the database content through
the available commands (see Section 6.2).
6.6.2 Effectiveness
System effectiveness was analyzed by calculating the success rate. This mea-
sure is typically adopted to evaluate how the systems support the completion
of a task. In our study, each query executed by the participants was coded as
“Success” if the task was completely executed, “Partial” if the participant
was able to come out with partial results, and “Failure” if the participant
task execution was wrong. The success rate can be calculated as the per-
centage of users who were able to successfully complete the tasks. The task
success rate for each user has been calculated as:
68
Chatbot SQL Paired sample t-test
Task x SD x SD df t Sig. (2-tailed)
1a 91.78 70.33 68.28 25.02 13 1.378 .192
1b 178.10 27.20 169.40 21.16 9 .687 .509
2 240 0 240 0
3 107.07 57.70 86.71 50.40 13 .971 .349
4a 94.93 42.11 112.06 46.41 14 -1.704 .111
4b 4.73 1.16 11.60 13.64 14 -1.916 .076
4c 82.12 30.05 62.12 26.67 7 1.639 .145
5a 201.00 20.08 216.60 25.65 4 -.807 .465
5b 30.50 14.84 24.50 28.99 1 .194 .878
6 155.70 68.78 165.40 49.55 9 -.285 .782
Table 6.1: Paired Sample t-test results related to the time the users spent on each
task.
successRate =#successfulTasks + (#partialSuccessfulTasks/2)
#tasks× 100
The global success rate for each system is then calculated as the average
success rate of all the users. According to this formula, the systems success
rates are:
successRatechatbot =65 + (46/2)
150= 58.6%
successRateSQL =80 + (34/2)
150= 64.66%
Success rates resulted quite balanced between Chatbot and the SQL com-
mand line, also the Wilcoxon signed-rank test did not reveal any significant
difference (Z = −1.257, p = .209). Table 6.2 reports the detail on the com-
pletion of each task; Figure 6.2 shows the comparison for the two systems of
the number of successes for each task.
6.7 Satisfaction
Users satisfaction was analyzed by considering different dimensions measured
through SUS and NASA-TLX questionnaires, as well as specific Likert-based
questions. In the following, we report on the results of all the satisfaction
dimensions.
69
Chatbot SQL
Partial Partial
Task Failure Success Success Failure Success Success
1a 1 13 1 0 1 14
1b 5 8 2 2 7 6
2 6 9 0 1 11 3
3 1 4 10 0 0 15
4a 0 0 15 0 0 15
4b 0 0 15 0 0 15
4c 3 0 12 7 1 7
5a 9 3 3 10 2 3
5b 9 0 6 13 0 2
6 5 9 1 3 12 0
Table 6.2: Detail on the completion of each task: Failures, Partial success, Success
6.7.1 Usability
SUS analysis revealed different scores about the perceived usability of the two
systems (Chatbot x = 66.3, SD = 14.4; SQL x = 55.7, SD = 16.7). The
Chatbot result is in line with the average SUS scores (69.5) of one thousand
studies reported in [3], while the SQL score was quite lower. However, t-test
highlighted that the SUS scores difference between the two systems is not
statistically significant (t(14) = 1.994, p = .066).
In addition, according to [24], we split the overall SUS score into two
factors, i.e., System Learnability (considering statements #4 and #10) and
System Usability (all the other statements). The Chatbot Learnability score
was 63.0 (SD = 22.8) while SQL Learnability score was 62.5 (SD = 32.1).
However, t-test highlighted that the Learnability score difference is not sta-
tistically significant (t(14) = .280, p = .783). Regarding SUS Usabil-
ity, the Chatbot score was 65.10 (SD = 16.7) while SQL score was 51.25
(SD = 19.8). Even in this case, t-test highlighted that the difference be-
tween these scores is not statistically significant (t(14) = 1.968, p = .069).
Besides providing an objective indication about the usability and learn-
ability of the tested systems, SUS results can be used as benchmarks in the
comparison of further approaches to explore databases.
70
1a 1b 2 3 4a 4b 4c 5a 5b 6
1
3
5
7
9
11
13
15
ChatbotMySQL
Figure 6.2: Comparison for the two systems of the number of successes (Partial Success
+ Success) for each task.
6.7.2 Workload
NASA-TLX measured the workload caused by each system. The Chatbot
had a lower, and thus better, workload (x = 47.88, SD = 12.45) than the
SQL system (x = 59.00, SD = 11.42), as also underlined by the t-test
(t(14) = −2.738, p = .015). We also analyzed possible differences between
the systems by considering each single NASA-TLX dimension, whose mean,
standard deviation and t-test results are reported in Table 6.3 and graphically
displayed in the histogram in Figure 6.3. Only the Mental Demand dimension
presents a significant difference with the Chatbot that causes a lower mental
effort. However, as can be seen by the Figure, there is a positive trend in
favor of the Chatbot for all the dimensions, for which low values represent
high satisfaction.
71
Chatbot SQL Paired sample t-test
Dimension x SD x SD df t Sig. (2-tailed)
Effort 58.66 25.31 69.33 14.86 14 1.621 .127
Frustration 46.66 22.88 57.33 24.92 14 1.204 .249
Mental Demand 49.33 14.86 60.66 13.87 14 3.238 .006
Physical Demand 30.00 15.58 47.33 24.92 14 1.992 .066
Temporal Demand 58.00 23.36 73.33 16.32 14 1.982 .068
Performance 44.66 15.97 46.00 15.94 14 .235 .818
Table 6.3: NASA-TLX dimensions values. Only the Mental Demand dimension
presents a significant difference with the Chatbot that causes a lower mental effort.
6.7.3 Easiness of exploring DB/retrieving data/using
commands and quality of presentation
Participants were asked to answer ad-hoc questions about the: (i) easiness of
exploring and understanding the database content, related to Type 1 tasks,
(ii) easiness of retrieving specific data items, related to Type 2 and Type 3
tasks, (iii) the easiness of the commands to be used, and (iv) the quality of
the presentation of query results.
Regarding the easiness of exploring and understand the database content,
related to Type 1 tasks, Chatbot obtained an average score of 6.13 (SD = 2.5),
while SQL obtained 5.33 (SD = 2.46). However, the t-test does not reveal a
significant difference (t(14) = .893, p = .387).
About the easiness of retrieving specific data items, related to Type 2 and
Type 3 tasks, Chatbot obtained an average score of 7.26 (SD = 2.3), while
SQL obtained 5.53 (SD = 1.72). This difference is statistically significant in
favor of Chatbot, as shown by the t-test (t(14) = 2.179, p = .047).
Concerning the easiness of the commands to be used, Chatbot obtained
an average score of 7.60 (SD = 1.29), while SQL obtained 6.27 (SD = 1.90).
Even in this case, this difference is statistically significant in favor of Chatbot,
as shown by the t-test (t(14) = 2.467, p = .027).
Lastly, about the quality of the presentation of query results, Chatbot
obtained an average score of 6.93 (SD = 1.79), while SQL obtained 5.53
(SD = 2.23). T-test has shown that this difference is statistically significant
in favor of Chatbot (t(14) = 2.365, p = .033).
72
Effort Frustration Mental
Demand
Physical
Demand
Temporal
Demand
Performance0
20
40
60
80
100
ChatbotMySQL
Figure 6.3: NASA-TLX dimensions: means and standard deviation comparison. Low
values represent high satisfaction.
6.7.4 User ranking of systems along completeness, eas-
iness and usefulness
The questionnaire administered at the end of each participant’s session asked
to rank the two systems based on their utility, completeness and ease of use
(from 1 to 2, 1 is the best). Regarding utility, Chatbot obtained an average
rank of 1.35 (SD = .49) while SQL a rank of 1.64 (SD = .49), but Wilcoxon
test highlights that this difference is not statistically significant (Z = −1.069,
p = .285).
In case of completeness, Chatbot obtained an average rank of 1.85 (SD =
.36) while SQL a rank of 1.14 (SD = .36), which was significantly better, as
demonstrated by the Wilcoxon test (Z = −2.673, p = .008).
Finally, about usefulness, Chatbot obtained an average rank of 1.28 (SD =
.47) while SQL a rank of 1.71 (SD = .47), but no significant difference was
found by the Wilcoxon test (Z = −1.604, p = .109).
73
6.8 Qualitative Data Analysis
The participants appreciated the chatbot because it is simple to use and
allow them to use “natural terms” that are “intuitive” and “easy to remem-
ber” without needing to know a specific language. This was also confirmed
by the quantitative analysis of the usability score and of the Mental Demand
dimension of the NASA-TLX questionnaire. Some of them appreciated the
possibility to navigate through the history of commands - this is a very pos-
itive aspect that recognizes one of the main advantages that the interactive
paradigm of chatbots can have with respect to using formal query languages
[22]. As a further advantage they recognized the possibility to “rapidly find
information without knowing the database structure”.
The most severe problem of the chatbot relates to the difficulty to specify
complex queries, for example the ones based on multiple joins. This is specif-
ically due to the explorative nature of the chatbot, which mainly provides
commands to discover the available data progressively navigating through
the available tables. However, apparently they were not able to identify this
peculiarity, as with the chatbot they tried to formulate SQL-like queries.
One participant said that “it was difficult to translate SQL commands into
messages for the chatbot”. However, the same participant highlighted that
“perhaps this difficulty arises only for database experts”.
In general, the majority of participants felt constrained by the chatbot
commands, as they could not specify the same queries they would execute
with the SQL command line. This is in line with what emerged from the
quantitative analysis of the completeness, as reported in Subsection 6.7.4.
They however pointed out that the chatbot conversational paradigm could
be very useful in situations where “one would need to execute simple queries
(for which knowing in details the structure of the database is not needed),
that must be executed rapidly, frequently, with few variations”.
In the comparison between the two systems, the majority of participants
found that the chatbot is more useful than SQL, as it can be easily used also
by people without any experience with databases. Regarding its utility some
of them observed that it can be useful for people with technical expertise,
who however do not work with databases frequently. In this situation, they
would not be required to struggle with the SQL syntax, as the bot can be
queried with intuitive command, and they would be productive in the short
time. With similar motivations, thy also expressed that the chatbot is easier
74
to use, although some of them also recognized that for people who know SQL
the SQL command line is easy as well.
Overall the majority of participants said that they would use SQL more
than the chatbot. The main motivation behind this choice relates to the
completeness of the SQL language, which gives them a greater awareness of
the available data and of the operations that they can perform (“It is easy
to have an overview of the available tables and of their content”; “I can find
any information I need with one unique query”).
6.9 Lesson Learned
This experimental study allowed us to find out whether the Chatbot could
represent a valid alternative to the basic SQL command line, which is a
typical form of interaction with a database system. In relation to our re-
search question, we did not find any significant difference in the effectiveness
and efficiency of the two systems, while the analysis highlighted significant
differences in favor of the Chatbot system for some satisfaction dimensions.
These results are positive: the lack of significant differences for the two
systems for almost all the considered dimensions shows that the users appre-
ciated the Chatbot even though, as database experts, they found the SQL
command line more complete. Moreover, the analysis highlighted a clear
preference for some aspects of the Chatbot:
• The easiness of retrieving specific data items ;
• The easiness of the commands to be used ;
• The quality of the presentation of query results.
We believe these aspects are the ones that more characterize the chatbot
paradigm; therefore, these results are encouraging in relation to our idea to
promote Chatbot for Data Exploration.
In the following we report some conclusions that we drew by considering
the qualitative analysis of the gathered data and what we observed during
the task execution by users.
More significant help messages. By observing the message exchanges,
we saw that the first user messages were mainly intended to receive help
on the capabilities of the chatbot. As we have discussed in Section 4.3,
75
our framework generates a conversational agent able to support assistance
requests, which correspond to general or more element-specific answers to
the user.
A possible improvement of the application may include the extension of
these kind of help responses, by giving extended explanations related to the
system functionalities and the data involved. Moreover, the Chatbot may
provide a sort of tutorial that could be activated through dedicated message
patterns like “tutorial” or “guide me”; the tutorial might be composed of
a sequence of pre-defined interactions that the application would suggest to
the user, while explaining the information displayed.
Feedback on wrong user requests. Some users got stuck after receiving
multiple times the fallback message “I did not get that :( ”, i.e., the default
message that the Chatbot sends in case no intent is matched. These sit-
uations derive, for instance, from: messages that present too many typos,
requests not supported by the Chatbot (such as complex queries, orderings
or searches based on unmapped attributes), or because users tried to click
buttons shown in past interactions and thus inactive.
The problems related to textual inputs could be solved by extending the
conversation model to support the identification of some common (but not
supported) messages and providing appropriate feedbacks. In case of invalid
button clicks, instead, the Chatbot could just respond with: “You can not
click on previously displayed buttons!”.
Complex queries. In the future, the Chatbot may support complex quer-
ies, i.e., requests that use more than one attribute at a time, or the ability
to cross relations also when considering more than one element. We derived
this conclusion by noticing that the majority of the messages sent by the
users aimed to reach the elements required by the tasks in a single request,
without following a step-by-step exploration.
In any case, we believe that this attitude was due to the participants
expertise in database systems, in the sense that often they tried to formulate
long and articulated requests, which corresponds to the a typical approach
followed when writing SQL commands.
Order by clause. Some users reported the absence of the SQL command
ORDER BY, which enables the ordering of the results based on some values.
76
As a future improvement, this feature could be added to support the related
command.
Enable chit-chat. Finally, we noticed that some users tried to chit-chat
with the Chatbot, by sending messages like “Hello, how are you?” or “I am
Nicola, who are you?”. Even though these requests are not related to any
explorative intentions, they may help the users to feel more comfortable and
even enjoy the conversation with the application.
For example, the Chatbot may save personal information of its users,
along with the Context of the conversation, in order to use it in chit-chat
interactions and emulate a real human-to-human conversation by recalling
the user name or age.
We believe that almost all the features reported above can be considered
valid in general for Chatbots for Data Exploration, not only for the specific
conversations that can be generated through our framework. One of the goals
of our research is indeed to identify guidelines for the design of such class of
applications, which is a research field not yet exhaustively explored.
77
Chapter 7
Conclusions
In this Thesis we proposed a novel conceptual modeling approach for the
definition and development of conversational agents for data exploration.
We began our journey by analyzing the state-of-the-art technologies nowa-
days available, with particular attention to the existing frameworks that offer
their usages; they allow developers to create a complete functioning chatbot,
integrated with messaging platforms and third-party API. However, we dis-
cuss the lack of a common methodology that define a general guideline that,
starting from the structure of the data involved, describes the steps to be
considered towards the definition of the final conversational agent. These
steps include dimensions relatable to the capabilities of the chatbot, such as
interaction paradigm, data visualization, context handling, integration.
On this basis, we aim to provide a data-driven paradigm that enables
the development of a conversational agent considering all the phases of its
design. However, we concentrated our analysis on modeling a specific type
of chatbots, i.e., chatbots for data exploration. We provided a definition
of these systems, outlining requirements and objectives of their creation; in
particular, these specifications include the capability to autonomously extract
the schema from a database endpoint, the support of schema annotation by
a Designer and, most importantly, the generation of a dialogue management
system that can be reached by the most popular chat interfaces to access the
data involved.
Afterwards, we provided a solution that could satisfy the requirements de-
fined, describing accurately each step of the design process. In particular, we
started from accurately explaining the above mentioned schema annotation
procedure; then, we illustrated how the chatbot could use these information
to generate an NLU model to support conversations related to the target
data. After that, we defined an execution paradigm, describing all the issues
that arise when dealing with the actual request handling, their translation
into queries, the results visualization and the context management.
Subsequently we described the prototype of our system, with particular
attention on the architecture components and the adopted technologies to
provide the wanted functionalities. We also posed our attention to a few
available tools that helped us in the various development steps.
Finally, we conducted a comparative study between a conversational
agent generated through our framework and the SQL command line, in terms
of the execution of some information retrieval tasks on the same data system.
We provided a quantitative and qualitative analysis based on the interaction
differences with our application and the SQL command line, which allowed
us to draw some positive conclusions about our chatbot prototype, as well
as many considerations about future improvements of our system.
7.1 Limitations
During the definition of our framework, we operated some structural choices
that influenced the design and the characteristics of the overall system.
First of all, our model introduces an abstraction of the target data system:
the translation between the information taken as is and its representation in
a conversational environment leads to the inevitable reformatting and hiding
of less significant data. While this procedure helps the end user easily retrieve
what can be considered as the “most important” elements in the data system,
it introduces difficulties or even limits for what concerns visualization of
“secondary” properties.
Secondly, as we have understood from the qualitative analysis of the user
study, the chatbot generated with our system was expected to understand
complex commands for data manipulation. However, the model provides
the support of a predefined set of actions for each object to retrieve, to be
executed individually and according to the required format. This means that
the ability of the chatbot to express its capabilities is still limited. Strategies
for better guiding the user conversation are needed.
Another limitation of the framework is the necessity of the Designer par-
ticipation, whose task is to annotate the database schema according to the
knowledge of the data system and the ideas for the future conversational ex-
80
ploration. Although this procedure enables the generation of chatbots with
different characteristics and objectives, it limits the autonomy of the overall
generative phase.
7.2 Future Work
The introduction of chatbots for data exploration and our definition of a
possible methodology for their development lead the way towards future im-
provements on this subject.
First of all, future improvements on NLP and Machine Learning tech-
nologies may lead to a better and more complex understanding of the input
phrases. Moreover, the possibility to introduce and use a dedicated Knowl-
edge Base next to the target database may lead to the definition and exploita-
tion of NLU models able to manage requests based also on their semantics,
rather than their only syntax.
Secondly, we developed our system on the basis of a relational database
(OLTP), but as a future work it could support the integration with other data
models, such as NoSQL data storages, Graph-based databases or even Data
Warehouses (OLAP). For what concerns the latter systems, the introduction
of a chatbot for the exploration of data might represent an additional, com-
plementary tool for the data analysis features these data structures already
provide.
Besides the integration with various data models, a Chatbot for Data Ex-
ploration could be also extended to interact with heterogeneous data sources.
In this case, the Context of the conversation might not be limited to influ-
ence the dialogue state, but it could become an active part for the dynamic
selection of resources by means of dedicated on-the-fly Mashups [10].
Furthermore, among the integration of the chatbot with external inter-
faces, the support of voice-based requests along with the introduction of
dedicated libraries for their transcription could lead to a better and easiest
user experience. Also, the results visualization format could be more interac-
tive or include, besides textual and button-based messages, also figures and
graphic content.
For what concerns the Schema Annotation, the procedure may be con-
ducted graphically, with the help of a dedicated software and it might in-
clude some heuristics or even integrate AI units for the automatic detection
of connections among tables and attributes priority. Otherwise, the complete
81
activity may be conducted with the help of the very same chat interface that
later will be used for the data retrieval: this way, a kind of “Conversational”
Schema Annotation could be accomplished.
Another option, deriving from the previous point, is to substitute the
Designer and assign the role directly to the end user, enabling the dynamic
definition of the exploration limits and characteristics with respect to the
will and preferences of the user. In this situation, the architecture should be
able to keep track, along with the conversation Context for each user, also
of the complete NLU model for data access by each single user. This would
lead to the notion of “Personal” Chatbot for Data Exploration.
Finally, as future work, we would like to take into account an observation
we received from a review for a paper accepted to the International Web En-
gineering Conference 2019 [8]. The suggestion described the idea to give the
Designer the possibility to manage “a holistic Conceptual Model that should
be the blueprint of the physical database”. In this scenario, the possibility
to navigate across a data system, that is the goal of a Chatbot for Data
Exploration, would not be seen only as an additional feature for an existing
database, but it could influence the definition from scratch of the model of
new data sources.
7.3 Publications
The main idea of the methodology proposed in this thesis, i.e., the conceptual
modeling of conversational elements by annotation of the database schema,
has been published in the following article:
“Nicola Castaldo, Florian Daniel, Maristella Matera, and Vittorio Zaccaria.
Conversational Data Exploration. In Web Engineering - 19h International
Conference, ICWE 2019, Daejeon, Korea, June 11-14, 2019, Proceedings.
Springer, 2019.”
The full methodology, the framework and the results of the evaluation study
will be the object of a journal paper that we are already authoring.
82
Bibliography
[1] Sameera A. Abdul-Kader and Dr. John Woods. Survey on chatbot de-
sign techniques in speech conversation systems. International Journal
of Advanced Computer Science and Applications, 6(7), 2015.
[2] Damla Ezgi Akcora, Andrea Belli, Marina Berardi, Stella Casola,
Nicoletta Di Blas, Stefano Falletta, Alessandro Faraotti, Luca Lodi,
Daniela Nossa Diaz, Paolo Paolini, Fabrizio Renzi, and Filippo Van-
nella. Conversational support for education. In Artificial Intelligence in
Education - 19th International Conference, AIED 2018, London, UK,
June 27-30, 2018, Proceedings, Part II, pages 14–19, 2018.
[3] Aaron Bangor, Philip Kortum, and James Miller. Determining what
individual sus scores mean: Adding an adjective rating scale. J. Usability
Studies, 4(3):114–123, May 2009.
[4] Simone Borsci, Stefano Federici, and Marco Lauriola. On the dimension-
ality of the system usability scale: A test of alternative measurement
models. Cognitive processing, 10:193–7, 07 2009.
[5] Daniel Braun, Adrian Hernandez-Mendez, Florian Matthes, and Man-
fred Langen. Evaluating natural language understanding services for
conversational question answering systems. In SIGDIAL Conference,
2017.
[6] Virginia Braun and Victoria Clarke. Using thematic analysis in psychol-
ogy. Qualitative research in psychology, 3:77–101, 01 2006.
[7] John Brooke. Sus: A quick and dirty usability scale. Usability Eval.
Ind., 189, 11 1995.
83
[8] Nicola Castaldo, Florian Daniel, Maristella Matera, and Vittorio Zac-
caria. Conversational data exploration. In Web Engineering - 19h Inter-
national Conference, ICWE 2019, Daejeon, Korea, June 11-14, 2019,
Proceedings. Springer, 2019.
[9] Robert Dale. The return of the chatbots. Natural Language Engineering,
22(5):811–817, 2016.
[10] Florian Daniel, Maristella Matera, Elisa Quintarelli, Letizia Tanca, and
Vittorio Zaccaria. Context-aware access to heterogeneous resources
through on-the-fly mashups. In Advanced Information Systems Engi-
neering - 30th International Conference, CAiSE 2018, Tallinn, Estonia,
June 11-15, 2018, Proceedings, pages 119–134, 2018.
[11] Florian Daniel, Maristella Matera, Vittorio Zaccaria, and Alessandro
Dell’Orto. Toward truly personal chatbots: On the development of
custom conversational assistants. In Proceedings of the 1st International
Workshop on Software Engineering for Cognitive Services, SE4COG ’18,
pages 31–36, New York, NY, USA, 2018. ACM.
[12] Alessandro Dell’Orto. Interactive development of personal chatbots: To-
ward enabling non-programmers. Master’s thesis, Politecnico di Milano,
2018. Supervisor: Prof. Florian Daniel.
[13] Ahmed Fadhil. Can a chatbot determine my diet?: Addressing
challenges of chatbot application for meal recommendation. CoRR,
abs/1802.09100, 2018.
[14] A.M. Graziano and M.L. Raulin. Research Methods: A Process of In-
quiry. Prentice Hall PTR, 2012.
[15] Matt Grech. The current state of chatbots in 2017. GetVoIP.com, April
2017.
[16] Sandra G. Hart and Lowell E. Staveland. Development of nasa-tlx (task
load index): Results of empirical and theoretical research. In Peter A.
Hancock and Najmedin Meshkati, editors, Human Mental Workload,
volume 52 of Advances in Psychology, pages 139 – 183. North-Holland,
1988.
84
[17] Matthew B. Hoy. Alexa, siri, cortana, and more: An introduction
to voice assistants. Medical Reference Services Quarterly, 37(1):81–88,
2018. PMID: 29327988.
[18] Inbenta Technologies Inc. The ultimate guide to chatbots for businesses.
Technical report, www.inbenta.com, 2016.
[19] ISO. ISO 9241-11:1998 Ergonomic requirements for office work with vi-
sual display terminals (VDTs) – Part 11: Guidance on usability. Tech-
nical report, International Organization for Standardization, 1998.
[20] ISO/IEC. ISO/IEC 9126. Software engineering – Product quality.
ISO/IEC, 2001.
[21] Lorenz Cuno Klopfenstein, Saverio Delpriori, Silvia Malatini, and
Alessandro Bogliolo. The rise of bots: A survey of conversational in-
terfaces, patterns, and paradigms. In Proc. of the 2017 Conference on
Designing Interactive Systems, DIS ’17, Edinburgh, United Kingdom,
June 10-14, 2017, pages 555–565, 2017.
[22] Lorenz Cuno Klopfenstein, Saverio Delpriori, Silvia Malatini, and
Alessandro Bogliolo. The rise of bots: A survey of conversational inter-
faces, patterns, and paradigms. In Proceedings of the 2017 Conference
on Designing Interactive Systems, DIS ’17, pages 555–565, New York,
NY, USA, 2017. ACM.
[23] Chatbot lawyer overturns 160000 parking tickets in London and New
York. https://www.theguardian.com/technology/2016/jun/28/chatbot-
ai-lawyer-donotpay-parking-tickets-london-new-york. Accessed: 2019-
03-13.
[24] James R. Lewis and Jeff Sauro. The factor structure of the system us-
ability scale. In Masaaki Kurosu, editor, Human Centered Design, First
International Conference, HCD 2009, Held as Part of HCI International
2009, San Diego, CA, USA, July 19-24, 2009, Proceedings, volume 5619
of Lecture Notes in Computer Science, pages 94–103. Springer, 2009.
[25] Michael L Mauldin. Chatterbots, tinymuds, and the turing test: Enter-
ing the loebner prize competition. In AAAI, volume 94, pages 16–21,
1994.
85
[26] Definition: natural language understanding (NLU).
https://searchenterpriseai.techtarget.com/definition/natural-language-
understanding-nlu. Accessed: 2019-03-31.
[27] Juanan Pereira and Oscar Dıaz. Chatbot dimensions that matter:
Lessons from the trenches. In Web Engineering - 18th International
Conference, ICWE 2018, Caceres, Spain, June 5-8, 2018, Proceedings,
pages 129–135, 2018.
[28] Alberto Mario Pirovano. State of art and best practices in generative
conversational ai. Master’s thesis, Politecnico di Milano, 2018. Super-
visor: Prof. Matteo Matteucci.
[29] Loebner Prize. https://www.aisb.org.uk/events/loebner-prize. Ac-
cessed: 2019-03-12.
[30] B. R. Ranoliya, N. Raghuwanshi, and S. Singh. Chatbot for university
related faqs. In 2017 International Conference on Advances in Com-
puting, Communications and Informatics (ICACCI), pages 1525–1530,
Sep. 2017.
[31] Aniruddha Tammewar, Monik Pamecha, Chirag Jain, Apurva
Nagvenkar, and Krupal Modi. Production ready chatbots: Generate
if not retrieve, 2018.
[32] Joseph Weizenbaum et al. Eliza—a computer program for the study of
natural language communication between man and machine. Commu-
nications of the ACM, 9(1):36–45, 1966.
86
Appendix A
Parsed Database Schema
{
"customers ": {
"column_list ": [
"customerNumber",
"customerName",
"contactLastName",
"contactFirstName",
"phone",
"addressLine1",
"addressLine2",
"city",
"state",
"postalCode",
"country",
"salesRepEmployeeNumber",
"creditLimit"
],
"primary_key_list ": [
"customerNumber"
],
"references ": {
"employees ": {
"foreign_key_list ": [
"salesRepEmployeeNumber"
],
"reference_key_list ": [
"employeeNumber"
]
}
}
},
"employees ": {
"column_list ": [
"employeeNumber",
"lastName",
"firstName",
"extension",
"email",
"officeCode",
"reportsTo",
"jobTitle"
],
"primary_key_list ": [
"employeeNumber"
],
"references ": {
"employees ": {
"foreign_key_list ": [
"reportsTo"
],
"reference_key_list ": [
"employeeNumber"
]
},
"offices ": {
"foreign_key_list ": [
"officeCode"
],
"reference_key_list ": [
"officeCode"
]
}
}
},
"offices ": {
"column_list ": [
"officeCode",
"city",
"phone",
"addressLine1",
"addressLine2",
"state",
"country",
"postalCode",
"territory"
],
"primary_key_list ": [
"officeCode"
]
},
"orderdetails ": {
"column_list ": [
"orderNumber",
"productCode",
"quantityOrdered",
"priceEach",
"orderLineNumber"
],
"primary_key_list ": [
88
"orderNumber",
"productCode"
],
"references ": {
"orders ": {
"foreign_key_list ": [
"orderNumber"
],
"reference_key_list ": [
"orderNumber"
]
},
"products ": {
"foreign_key_list ": [
"productCode"
],
"reference_key_list ": [
"productCode"
]
}
}
},
"orders ": {
"column_list ": [
"orderNumber",
"orderDate",
"requiredDate",
"shippedDate",
"status",
"comments",
"customerNumber"
],
"primary_key_list ": [
"orderNumber"
],
"references ": {
"customers ": {
"foreign_key_list ": [
"customerNumber"
],
"reference_key_list ": [
"customerNumber"
]
}
}
},
"payments ": {
"column_list ": [
"customerNumber",
"checkNumber",
"paymentDate",
"amount"
],
89
"primary_key_list ": [
"customerNumber",
"checkNumber"
],
"references ": {
"customers ": {
"foreign_key_list ": [
"customerNumber"
],
"reference_key_list ": [
"customerNumber"
]
}
}
},
"productlines ": {
"column_list ": [
"productLine",
"textDescription",
"htmlDescription",
"image"
],
"primary_key_list ": [
"productLine"
]
},
"products ": {
"column_list ": [
"productCode",
"productName",
"productLine",
"productScale",
"productVendor",
"productDescription",
"quantityInStock",
"buyPrice",
"MSRP"
],
"primary_key_list ": [
"productCode"
],
"references ": {
"productlines ": {
"foreign_key_list ": [
"productLine"
],
"reference_key_list ": [
"productLine"
]
}
}
}
}
90
Appendix B
Database Schema Annotation
[
{
"object_name ":" customer",
"aliases ": [" customers", "client", "clients"],
"type": "primary",
"table_name ": "customers",
"display_attributes ":[
{
"keyword ": "",
"columns ": [" customerName "]
}
],
"qualifiers ": [
{
"keyword ": "",
"type": "word",
"columns ": [" customerName "]
},
{
"keyword ": "with contact",
"type": "word",
"columns ": [" contactLastName", "contactFirstName "]
},
{
"keyword ": "located in",
"type": "word",
"columns ": ["city", "state", "country "]
},
{
"keyword ": "that paid",
"type": "num",
"columns ": [" amount"],
"by": [
{
"from_table_name ": "customers",
"from_columns ": [" customerNumber "],
"to_table_name ": "payments",
"to_columns ": [" customerNumber "]
}
]
},
{
"keyword ": "that bought",
"type": "word",
"columns ": [" productName "],
"by": [
{
"from_table_name ": "customers",
"from_columns ": [" customerNumber "],
"to_table_name ": "orders",
"to_columns ": [" customerNumber "]
},
{
"from_table_name ": "orders",
"from_columns ": [" orderNumber "],
"to_table_name ": "orderdetails",
"to_columns ": [" orderNumber "]
},
{
"from_table_name ": "orderdetails",
"from_columns ": [" productCode "],
"to_table_name ": "products",
"to_columns ": [" productCode "]
}
]
},
{
"keyword ": "that reported to",
"type": "word",
"columns ": [" lastName", "firstName"],
"by": [
{
"from_table_name ": "customers",
"from_columns ": [" salesRepEmployeeNumber "],
"to_table_name ": "employees",
"to_columns ": [" employeeNumber "]
}
]
}
],
"relations ": [
{
"keyword ": "payments made",
"object_name ": "payment",
"by": [
{
"from_table_name ": "customers",
"from_columns ": [" customerNumber "],
"to_table_name ": "payments",
92
"to_columns ": [" customerNumber "]
}
]
},
{
"keyword ": "related sales representative",
"object_name ": "employee",
"by": [
{
"from_table_name ": "customers",
"from_columns ": [" salesRepEmployeeNumber "],
"to_table_name ": "employees",
"to_columns ": [" employeeNumber "]
}
]
},
{
"keyword ": "orders made",
"object_name ": "order",
"by": [
{
"from_table_name ": "customers",
"from_columns ": [" customerNumber "],
"to_table_name ": "orders",
"to_columns ": [" customerNumber "]
}
]
},
{
"keyword ": "products bought",
"object_name ": "product",
"by": [
{
"from_table_name ": "customers",
"from_columns ": [" customerNumber "],
"to_table_name ": "orders",
"to_columns ": [" customerNumber "]
},
{
"from_table_name ": "orders",
"from_columns ": [" orderNumber "],
"to_table_name ": "orderdetails",
"to_columns ": [" orderNumber "]
},
{
"from_table_name ": "orderdetails",
"from_columns ": [" productCode "],
"to_table_name ": "products",
"to_columns ": [" productCode "]
}
]
}
]
93
},
{
"object_name ":" employee",
"aliases ": [" employees", "worker", "workers"],
"type": "primary",
"table_name ": "employees",
"display_attributes ":[
{
"keyword ": "",
"columns ": [" lastName", "firstName "]
}
],
"qualifiers ": [
{
"keyword ": "",
"type": "word",
"columns ": [" lastName", "firstName "]
},
{
"keyword ": "with first name",
"type": "word",
"columns ": [" firstName "]
},
{
"keyword ": "with last name",
"type": "word",
"columns ": [" lastName "]
},
{
"keyword ": "that work in",
"type": "word",
"columns ": ["city", "country"],
"by": [
{
"from_table_name ": "employees",
"from_columns ": [" officeCode "],
"to_table_name ": "offices",
"to_columns ": [" officeCode "]
}
]
}
],
"relations ": [
{
"keyword ": "works in office",
"object_name ": "office",
"by": [
{
"from_table_name ": "employees",
"from_columns ": [" officeCode "],
"to_table_name ": "offices",
"to_columns ": [" officeCode "]
}
94
]
},
{
"keyword ": "related customers",
"object_name ": "customer",
"by": [
{
"from_table_name ": "employees",
"from_columns ": [" employeeNumber "],
"to_table_name ": "customers",
"to_columns ": [" salesRepEmployeeNumber "]
}
]
},
{
"keyword ": "reports to",
"object_name ": "employee",
"by": [
{
"from_table_name ": "employees",
"from_columns ": [" reportsTo"],
"to_table_name ": "employees",
"to_columns ": [" employeeNumber "]
}
]
}
]
},
{
"object_name ":" office",
"aliases ": [" offices"],
"type": "primary",
"table_name ": "offices",
"display_attributes ":[
{
"keyword ": "address",
"columns ": ["city", "addressLine1 "]
}
],
"qualifiers ": [
{
"keyword ": "with address",
"type": "word",
"columns ": ["city", "addressLine1 "]
},
{
"keyword ": "located in",
"type": "word",
"columns ": ["city", "country "]
},
{
"keyword ": "of",
"type": "word",
95
"columns ": [" lastName", "firstName"],
"by": [
{
"from_table_name ": "offices",
"from_columns ": [" officeCode "],
"to_table_name ": "employees",
"to_columns ": [" officeCode "]
}
]
}
],
"relations ": [
{
"keyword ": "employees working here",
"object_name ": "employee",
"by": [
{
"from_table_name ": "offices",
"from_columns ": [" officeCode "],
"to_table_name ": "employees",
"to_columns ": [" officeCode "]
}
]
}
]
},
{
"object_name ":" order",
"aliases ": [" orders"],
"type": "primary",
"table_name ": "orders",
"display_attributes ":[
{
"keyword ": "ordered",
"columns ": [" orderDate "]
},
{
"keyword ": "shipped",
"columns ": [" shippedDate "]
}
],
"qualifiers ": [
{
"keyword ": "made by",
"type": "word",
"columns ": [" customerName "],
"by": [
{
"from_table_name ": "orders",
"from_columns ": [" customerNumber "],
"to_table_name ": "customers",
"to_columns ": [" customerNumber "]
96
}
]
},
{
"keyword ": "with product",
"type": "word",
"columns ": [" productName "],
"by": [
{
"from_table_name ": "orders",
"from_columns ": [" orderNumber "],
"to_table_name ": "orderdetails",
"to_columns ": [" orderNumber "]
},
{
"from_table_name ": "orderdetails",
"from_columns ": [" productCode "],
"to_table_name ": "products",
"to_columns ": [" productCode "]
}
]
}
],
"relations ": [
{
"keyword ": "made by",
"object_name ": "customer",
"by": [
{
"from_table_name ": "orders",
"from_columns ": [" customerNumber "],
"to_table_name ": "customers",
"to_columns ": [" customerNumber "]
}
]
},
{
"keyword ": "contained products",
"object_name ": "product",
"by": [
{
"from_table_name ": "orders",
"from_columns ": [" orderNumber "],
"to_table_name ": "orderdetails",
"to_columns ": [" orderNumber "]
},
{
"from_table_name ": "orderdetails",
"from_columns ": [" productCode "],
"to_table_name ": "products",
"to_columns ": [" productCode "]
}
]
97
}
]
},
{
"object_name ":" payment",
"aliases ": [" payments"],
"type": "secondary",
"table_name ": "payments",
"display_attributes ":[
{
"keyword ": "quantity",
"columns ": [" amount "]
},
{
"keyword ": "date",
"columns ": [" paymentDate "]
}
],
"qualifiers ": [
],
"relations ": [
{
"keyword ": "made by",
"object_name ": "customer",
"by": [
{
"from_table_name ": "payments",
"from_columns ": [" customerNumber "],
"to_table_name ": "customers",
"to_columns ": [" customerNumber "]
}
]
}
]
},
{
"object_name ": "product line",
"aliases ": [" brand", "brands"],
"type": "primary",
"table_name ": "productlines",
"display_attributes ":[
{
"keyword ": "",
"columns ": [" productLine "]
}
],
"qualifiers ": [
{
"keyword ": "",
"type": "word",
"columns ": [" productLine "]
},
98
{
"keyword ": "of product",
"type": "word",
"columns ": [" productName "],
"by": [
{
"from_table_name ": "productlines",
"from_columns ": [" productLine "],
"to_table_name ": "products",
"to_columns ": [" productLine "]
}
]
}
],
"relations ": [
{
"keyword ": "related products",
"object_name ": "product",
"by": [
{
"from_table_name ": "productlines",
"from_columns ": [" productLine "],
"to_table_name ": "products",
"to_columns ": [" productLine "]
}
]
}
]
},
{
"object_name ": "product",
"aliases ": [" products"],
"type": "primary",
"table_name ": "products",
"display_attributes ":[
{
"keyword ": "",
"columns ": [" productName "]
},
{
"keyword ": "of product line",
"columns ": [" productLine "]
}
],
"qualifiers ": [
{
"keyword ": "",
"type": "word",
"columns ": [" productName "]
},
{
"keyword ": "of product line",
99
"type": "word",
"columns ": [" productLine "],
"by": [
{
"from_table_name ": "products",
"from_columns ": [" productLine "],
"to_table_name ": "productlines",
"to_columns ": [" productLine "]
}
]
},
{
"keyword ": "bought by",
"type": "word",
"columns ": [" customerName "],
"by": [
{
"from_table_name ": "products",
"from_columns ": [" productCode "],
"to_table_name ": "orderdetails",
"to_columns ": [" productCode "]
},
{
"from_table_name ": "orderdetails",
"from_columns ": [" orderNumber "],
"to_table_name ": "orders",
"to_columns ": [" orderNumber "]
},
{
"from_table_name ": "orders",
"from_columns ": [" customerNumber "],
"to_table_name ": "customers",
"to_columns ": [" customerNumber "]
}
]
}
],
"relations ": [
{
"keyword ": "related product line",
"object_name ": "product line",
"by": [
{
"from_table_name ": "products",
"from_columns ": [" productLine "],
"to_table_name ": "productlines",
"to_columns ": [" productLine "]
}
]
},
{
"keyword ": "in orders",
100
"object_name ": "order",
"by": [
{
"from_table_name ": "products",
"from_columns ": [" productCode "],
"to_table_name ": "orderdetails",
"to_columns ": [" productCode "]
},
{
"from_table_name ": "orderdetails",
"from_columns ": [" orderNumber "],
"to_table_name ": "orders",
"to_columns ": [" orderNumber "]
}
]
},
{
"keyword ": "bought by",
"object_name ": "customer",
"by": [
{
"from_table_name ": "products",
"from_columns ": [" productCode "],
"to_table_name ": "orderdetails",
"to_columns ": [" productCode "]
},
{
"from_table_name ": "orderdetails",
"from_columns ": [" orderNumber "],
"to_table_name ": "orders",
"to_columns ": [" orderNumber "]
},
{
"from_table_name ": "orders",
"from_columns ": [" customerNumber "],
"to_table_name ": "customers",
"to_columns ": [" customerNumber "]
}
]
}
]
},
{
"object_name ": "orderdetails",
"type": "crossable",
"table_name ": "orderdetails"
}
]
101
Appendix C
User Study Questionnaires
This Appendix reports the English translation (from Italian) of the question-
naires administered during the User Study.
C.1 Demographic Questionnaire
Insert the “participant code” you received:
Insert your age:
Gender:Man Woman Other
� � �
Education:High. S. Bachelor Master PhD Other
� � � � �
How do you evaluate your expertise with information technology?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
How do you rate your expertise with the use of chatbots?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
How do you rate your expertise with querying databases?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
C.2 System Questionnaire
Insert the “participant code” you received:
System used:Chatbot SQL
� �
C.2.1 Ease of use
I think I would like to use this system frequently.
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
I found the system unnecessarily complex.
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
I thought the system was easy to use.
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
104
I think that I would need the support of a technical person to be able to
use this system.
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
I found the various functions in this system were well integrated.
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
I thought there was too much inconsistency in this system.
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
I would imagine that most people would learn to use this system very quickly.
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
I found the system very cumbersome to use.
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
I felt very confident using the system.
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
I needed to learn a lot of things before I could get going with this system.
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
105
How likely are you to recommend this system to a friend or colleague?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
C.2.2 Cognitive load
How mentally demanding were the tasks?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
How physically demanding were the tasks?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
How hurried or rushed was the pace of the tasks?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
How successful were you in accomplishing what you were asked to do?
Perfect 1 2 3 4 5 6 7 8 9 10 Failure
� � � � � � � � � �
How hard did you have to work to accomplish your level of performance?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
106
How insecure, discouraged, irritated, stressed and annoyed were you?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
C.2.3 General questions
How easy was it to explore the database with this system, for example to
identify the information contained in it or to navigate among the various
tables?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
How easy was it to find the requested information, for example the detailed
information on customers?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
How easy was it to understand or remember the commands to use in or-
der to find the requested information?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
How do you consider the way in which the commands and information found
are presented?
Low 1 2 3 4 5 6 7 8 9 10 High
� � � � � � � � � �
107
Which scenarios of use do you think are the most appropriate for the system
you used?
Do you have any ideas on how to improve the presentation of results (for
example through graphics)?
Main advantages of the system used.
Main disadvantages of the system used.
C.3 Final Questionnaire
Insert the “participant code” you received:
With respect to their UTILITY, order the systems you used
(1 = the one you consider most useful, 2 = the one you think is least useful):
Chatbot: 1 2
� �SQL: 1 2
� �
108
Explain the reasons for your ranking:
With respect to their COMPLETENESS, order the systems you used
(1 = the one you consider most useful, 2 = the one you think is least useful):
Chatbot: 1 2
� �SQL: 1 2
� �
Explain the reasons for your ranking:
With respect to their EASE OF USE, order the systems you used
(1 = the one you consider most useful, 2 = the one you think is least useful):
Chatbot: 1 2
� �SQL: 1 2
� �
Explain the reasons for your ranking:
109