978-1-4673-2376-5/12/$31.00 ©2012 IEEE

2012 International Conference on System Engineering and Technology September 11-12, 2012, Bandung, Indonesia

Interactive Question Answering For Customer Service Representative Using Information State Concept

Ayu Purwarianti#1, Zulfikar Hakim#2 #School of Electrical Engineering and Informatics, Institut Teknologi Bandung

Jalan Ganesha 10 Bandung Indonesia 1ayu@stei.itb.ac.id

2zulfikar.hakim@gmail.com

Abstract— Interactive Question Answering (IQA) is a branch

of Question Answering (QA) with interactivity enhancement, enabling dialogue with users. This paper explain how to build a prototype of IQA system, with its supporting aspect such as understanding user’s topic, knowledge structure, and information recording in dialogue. Tree data structure is used for structuring IQA knowledge base. Information recording througout dialogue is using Information State (IS) concept, adapted from two previous model: Poesio-Traum model and Cooper-Larsson Model. IS model is adjusted based on knowledge structure and a specific needs of Customer Service Representative behaviour. An interactivity testing is conducted to test whether aspects of IQA needs is fulfilled. We have succesfully built the prototype, using IS concept.

Keywords — Interactive Question Answering, Knowledge

Structure, Information State

I. INTRODUCTION This paper gives an alternative for company is Customer

Relationship Management (CRM) aspect, especially interaction between company and their consumer. To solve limitation of human resource in handling CRM aspects, we build a prototype of automated system for that purpose, with IQA approach. This approach enabling dialogue between a QA system and users.

In IQA system, the purpose of dialogue with users is to find the most appropriate answer needed [6]. Important feature to accomplish this purpose is an ability of IQA system to understand dialogue topic [1]. Furthermore, an IQA system shoud using an important component, Information State, functioning as dialogue state loggers. Dialogue state is used to arrange a dialogue scenario based on user’s information need.

Another method beside using IS as state logger is using a Finite State (FS) [7]. IQA system with FS has limitation of its ability to accept user’s input [5]. A main purpose of this paper is to discuss how to build a prototype of IQA system and its supporting aspects, using a concept of IS, so the system can accomodate a general dialogue scenario.

II. THEORETICAL APPROACH An IQA system have a specific component that enabling

dialogue with users: dialogue manager. Another component is not really different with antoher QA system, depends on answer domain space and characteristic of problems to be solved [6].

A. Interactive Question Answering Architecture

Fig. 1 IQA Architecture on HITIQA [6]

As shown at Figure 1, conventional QA component on HITIQA is relatively not difference with another QA system. The dialogue manager component hold an important role to enabling dialogue in interactive setting. This component logs information state and change in a dialogue based on information retrieved from users.

B. Issues on Dialogue Modelling Reference [5] shows there are some issues to modelling

dialogue:

1) Ellipsis. Ellipsis is incomplete sentence and does not have a verbal phrase. Interpretation and resolution of an ellipsis needs a dialogue context modelling to complete a missing information from the last user’s question. For instance, an ellipsis dialogue scenario is shown below:

User : “When Shakespeare was born?” System : “At 1564”

User : “Where?”

2) Anaphoric Reference. Anaphoric reference is a linguistic form which the meaning of the part of sentence can be understood by a reference to an referred entity. Anaphoric reference mostly using a personal pronoun. This issue can be solved by finding a mostly meaningful referent which its absent will form a query with has no meaning.

User : “Where Shakespeare was born?”. System : “At 1564”. User : “With whom he was married?”.

3) Grounding and Clarification. In the dialogue scenario below, system should clarify the meaning of “he”, whether refers to Bill Clinton or Yasser Arafat.

User : “When did Bill Clinton and Yasser Arafat meet at Camp David?”.

System : “At 2000”. User : “How old is he?”.

4) Turn Taking. In a real dialogue, human often know when to speak and when to listen. Sometime, they cut another’s sentence. In a human-computer diague system, this dialogue turn should be well defined. This issue is very important in speech-based dialogue, even though not important in text-based dialogue. This is because in a text-based dialogue system, an overlap between user’s input and system response is not possible.

C. Topic Detection To understanding a topic in a human-computer text-based

dialogue, we need a technique to detect a topic in a sentence or dialogue. This topic recognition is done by grouping keywords on each topic, extracted from user’s input ([1], [2]).

Fig. 2 Visualization of Topic Grouping ([1], [2])

With this methods, system can understand what topic discussed/asked in a dialogue discourse. Instead of developing a error-free transcription method, is will be better to recognize words with obvious related topic. Some words can not be directly grouped as a specific topic. For instance, a word “awareness” is not grouped to a topic too general and can not included at any topic group

Topic history in a dialogue is logged in a thread history. Thread history tracks what topic is currently active (a currently discussed topic between user and IQA system). A

topic can intersect with another topic due same keywords, stop in a dialogue discourse, change, or merge with another topic. With this technique, system can know whether a user start a new topic, change it, or end it, so system can arrange an appropriate scenario.

Fig. 3 Visualization of topic thread history ([1], [2])

D. Information State Model Information State is a representation of information needed

to distinguish it to another dialogue discourse [8]. Information State model is represented with a feature structure. In general, feature structure is shown in Figure 4 [3].

Fig. 4 Feature Structure

ln represent atributes and an represent value(s) of it. For instance, an automated trip planning has a feature structure shown in Figure 5.

Fig. 5 Feature Structure for Automated Trip Planning

An IS may have some attributes with a specified type of value(s). Value(s) from attribute can be a list, stack, queue, or another data structure type.

There are two models for IS in Trindi. They are Cooper Larsson model and Poesio-Traum model [5]

1) Cooper-Larsson Model. A common form of this model is shown in Figure 6.

Fig. 6 Cooper-Larsson Model

Significant characteristic of this model is a distinct private and common attribute, specific to dialogue speakers. Private attribute have private belief, containing set of propositions from dialogue. Agenda contains a stack of action will dispatched to dialogue. Common field contains belief and QUD, or Questions Under Discussion. QUD hold value of current question discussed in dialogue.

2) Poesio-Traum Model. Common form of this model is shown in Figure 7 [3]. Instead of using private and common attributes, this model records Grounded (G) and Ungrounded Discourse Unit (UDU) throughout dialogue. Intention also

recorded. Grounded information consist of Obligation (OBL), Dialogue History (DH), Social Commitment to Proposition (SPC), and Option (OPT).

: : :::: : … , …: : :::: : … , …

Fig. 7 Poesio-Traum Model

III. ANALISYS & DESIGN Generally, the purpose of this paper is to define a

knowledge structure of IQA system and Information State structure based on knowledge structure to solve variety of scenario as defined below.

A. Dialogue Scenario We narrowed our research on the following scenario, based

on [6] and some major modifications: 1. Direct answer. The system gives direct answer when

user’s question is quite detailed so that system no longer needs to provide feedback to the user to enter more specific information.

2. Specification. If user give/ask general information with wide answer space, system needs to provide more specific feedback, asking user to give additional information.

3. Generalization. If user ask a question with wide variety of too specific informations, system needs to broaden answer space.

4. Specification with target of information. This scenario happens when user asks a general information, but with a specific information. In this case, system will do a specification, but will not re-ask the information user have entered.

5. Out of topic. When dialogue talks about a topic, then user input a completely different topic, system will ask about out of topic notification and ask user whether he/she will change the topic or not.

B. Knowledge Representation The IQA system’s knowledge is structured as tree. The

level of one node in tree, give an overview about information specificity: a deeper level of a node means a more specific information owned by a node.

As shown in figure 8, a root (1) of a tree is a dummy node which has no information can be provided to a user. A non-

root nodes (2 and 3) have information consisting of α and β information. The combination of it will be a parameter to define dialogue scneario. A leaf node (3), have a descriptive information (4) will be provided to user, means information user provided is quite specific and ready to be answered.

Fig. 8 Knowledge Structure Visualization

α and β information is variables in knowledge structure, organized such that the combination of it can define dialogue scneario. An illustrated explanation about it is shown in Figure 9. In a knowledge structure, a more specific an information is, then a more deeper level of a node. For instance, n3 (i1+i3) have some more specific information than n1 (only i1), and more general information than n5 (i1 + i3 + i5). More specific information means, n3 have more information than n1 and less informations then n5. α contains all information of its ancestor, meanwhile β contains information just a specific information it has.

Fig. 9 Organization of α and β

To generate feedback to the user, each edge connecting two nodes contains information about specification type. Each specification type has a feedback template to the user. The following table describe specification type and its correlating feedback template.

TABLE 1 SPECIFICATION TYPE AND CORRELATED GENERATED FEEDBACK

EXAMPLES

Specification type Generated feedback examples Product What product do you mean? Product A, B,

or Product C? Time In what time do you mean? Place There are several places/area available:

Jakarta, Bandung, and Sumatera. Which one should we show to you?

Service What service do you mean? SMS, MMS, or Phone Call?

Specification type Generated feedback examples Others Which information do you mean? Package Which package do you mean? Package A,

Package B or C? Furthermore, each leaf contains one correlated topic. A

non-leaf node does not save topic. When system parse knowledge structure for the first time, iteratively every non-leaf nodes is assigned with all of their childs topics. Therefore, a root nodes will contains all available topics.

C. Information State Information state contains some attribute to support

dialogue: 1. Active topic. The system record an active topic in a

question/user’s input by extracting keywords occurance in dialogue using topic grouping technique

2. Active node(s). Active node(s) is a node(s) that have the highest similarity with user’s question, consisting active node(s) by α and by β informations

3. Expectation node. When system give a non-answer feedback to user, system expects users to answer an appropriate answer. Therefore, system has a record about what answer should user provide. If user provide answer other than expectation node(s) then the utterance is decided as out of topic

4. History node. Record past active node(s) 5. Target Node. Contains pre-recorded information in

early utterance of dialogue. Entered information will not reask by system and will be noticed.

A node is said “active” when dialogue controller decide

node is relevant user’s input. Information state model shown in Figure 10 :: , , … : , , … : , , …

Fig. 10 Information State Model

D. Illustrated Correlation of Topic and Knowlege Representation

This section show how combination of α, β, target node, and expected node implies to various dialogue scenario

1) Direct Answer. Direct answer scenario will be chosen when active node by α information is a leaf node.

2) Specification. Specification is chosen when an internal node is active. When internal node is active, system choose some expected nodes to keep dialogue topic is on track. This scenario illustration is shown in Figure 11, when user ask “What is the rate of product B?” . At this point, system will give a feedback: “Whick package do you mean, Package A, B, or C?”

3) Generalization. Generalization scenario is chosen when some nodes active by β information. This scenario illustrated

in Figure 12, for instance, user’s question is “What is rate of Short Messaging Service (SMS)?”. At this utterance, system will answer “First of all, which product do you mean, Product A, B, or C?”. The illustrated example of this scenario is shown in figure 12.

Fig. 11 Specification Illustration

Fig. 12 Generalization Illustration

4) Specification with target of information. This scenario is chosen when user has entered some specific information so system should not reasking it. Figure 13 shows the illustration, when user input “What is rate of SMS in Product B? ”

SMS MMS Internet

Target Topic

Root

Product A

Product B

Package A

Package B

Package C

Active

α Information

Expected Node

Fig. 13 Specification with target of information

5) Out of Topic is considered when user’s input in one utterance is not a member of expected node list. Figure 14 shows the illustration.

Fig. 14 Out of Topic Illustration

IV. EVALUATION Some method of IQA evaluation has been formulated

before [4]. Table 1 shows aspects of IQA interactivity measurements and its related questions for questionnaires.

TABLE 2 INTERACTIVITY MEASUREMENTS OF IQA

Measurement Aspets Question • Helps users in gather data

with less time • Reduce user time

compared by reading the whole document

• Compared by opening official website, is IQA system give an easier way to provide your information need?

• Compared by your previous way in seeking same information, did you find that it is faster to use IQA system?

• Menyediakan saran yang baik untuk menemukan jawaban

• Is it difficult to interact with system?

• Is system not flexible? • Easy to use • Is system’s response useful and

understandable? • Is it easy to find your need of

information by using IQA system?

From measurement parameters in table 1, we give

questionnaires to respondents about our IQA system. Table 2 shows the result.

TABLE 3 QUESTIONNAIRES RESULT

Question Yes No Others

Compared by opening official website, is IQA system give an easier way to provide your information need?

78,13% 15,63% 6,25%

Is it difficult to interact with system? 21,88% 75,00% 3,13%

Is system not flexible? 50,00% 34,38% 15,63%

Is system’s response useful and understandable? 93,75% 6,25% 0,00%

Is it easy to find your need of information by using IQA system?

53,13% 40,63% 6,25%

Compared by your previous way in seeking same information, did you find that it is faster to use IQA system?

71,88% 21,88% 6,25%

V. CONCLUSION Our conclusions are: • Prototype of IQA system to represent a customer service

representative has been built, worked, and tested. • Information state contains informations about active

node, expectation node, and past node, taken from knowlede structure manipulation done by dialogue manager

VI. REFERENCES [1] Bergstrom, T. & Karahalios, K. Conversation Clusters: Grouping

Conversation Topics through Human-Computer Dialog, 2008. [2] Bergstrom, T. & Karahalios, K. Conversation clusters: human-

computer dialog for topic extraction. University of Illnois, United States of America. Proceeding pada CHI, 2008.

[3] Cooper, R., Larsson, S., Matheson, C., Poesio M., Traum D. Coding Instructional Dialogue for Information States, 1999.

[4] Kelly D., Kantor, P. B., Morse, E. L., Scholetz, J. dan Sun, Y. Questionnaires for eliciting evaluation data from users of interactive question answering systems. Natural Language Engineering, 15 , pp 119-141 doi:10.1017/S1351324908004932, 2009.

[5] Quarteroni, S, Advanced Techniques For Personalized, Interactive Question Answering: Ph.D. Thesis. University of York, United Kingdom, 2007

[6] Small, S., Liu, T., Shimizu N., Strzalkowski, T. HITIQA: An Interactive Question Answering System: A Preliminary Report, 2003

[7] Sutton,S .Universal speechtools: the CSLU toolkit. In: Proceedings of the International Conference on Spoken Language Processing, 1998.

[8] Larsson , S. And Traum, D. Information State and dialogue management in the TRINDI dialogue move engine toolkit. 2000.