Embed Size (px)
Citation preview
Keyword(s): Abstract:
Identifying Business Tasks and Commitments from Email and ChatConversationsAnup Kalia, Hamid Reza Motahari Nezhad, Claudio Bartolini, Munindar Singh
HP LaboratoriesHPL-2013-4
Task Extraction; natural-language processing; Case Management;
Case management applications and, more generally, people-centric processes involve the definition,resolution and communication of commitments for tasks over channels such as chat and email. Identifyingand tracking tasks and commitments can help in streamlining the collaborative work in businessenvironments. However, doing so proves challenging due to the syntactical, grammatical, and structuralincompleteness of human conversations over chat and email channels. We present a novel approach toautomatically identify tasks and commitment creation, delegation, completion, and cancellation in emailand chat conversations, based on techniques from natural language processing and machine learningdomains. We discover tasks and related parameters from the text of conversations, identify when acommitment to a task emerges and find the state changes of a commitment based features extracted fromthe text of the conversations. We have developed a prototype and evaluated our approach using real-worldchat and email datasets. Our experiments shows high precision for create class i.e., 90% in emails (Enronemail corpus) and 80% in a real-world chat dataset and also provides promising results for discharge,delegate, and cancel classes.
External Posting Date: February 06, 2013 [Fulltext] Approved for External PublicationInternal Posting Date: February 06, 2013 [Fulltext]
Copyright 2013 Hewlett-Packard Development Company, L.P.
Identifying Business Tasks and Commitmentsfrom Email and Chat Conversations
A. K. KaliaNC State University
Raleigh NC [email protected]
H.R. Motahari-Nezhad,C. Bartolini
Hewlett Packard LaboratoriesPalo Alto, CA, United States
M. P. SinghNC State University
Raleigh NC [email protected]
ABSTRACTCase management applications and, more generally, people-centricprocesses involve the definition, resolution and communication ofcommitments for tasks over channels such as chat and email. Iden-tifying and tracking tasks and commitments can help in stream-lining the collaborative work in business environments. However,doing so proves challenging due to the syntactical, grammatical,and structural incompleteness of human conversations over chatand email channels. We present a novel approach to automaticallyidentify tasks and commitment creation, delegation, completion,and cancellation in email and chat conversations, based on tech-niques from natural language processing and machine learning do-mains. We discover tasks and related parameters from the text ofconversations, identify when a commitment to a task emerges andfind the state changes of a commitment based features extractedfrom the text of the conversations. We have developed a proto-type and evaluated our approach using real-world chat and emaildatasets. Our experiments shows high precision for create class i.e.,90% in emails (Enron email corpus) and 80% in a real-world chatdataset and also provides promising results for discharge, delegate,and cancel classes.
1. INTRODUCTIONMany processes in organizations are people-driven, and are man-
aged in a collaborative and conversation-oriented manner. Conver-sations around people-centric processes involve defining and coor-dinating tasks through commitments among workers over informalchannels such as email and chat. Typical examples of such conver-sations include the handling of insurance claims and IT incidents.For instance, in IT incident management, incident reports are as-signed to help desk workers and in some cases to specialized ITexperts. The handling usually proceeds through team collabora-tion and communication over email and chat, in addition to keep-ing record in specialized systems. Keeping track of all agreed-upontasks and commitments made during a conversation is a dauntingtask, and a major source of inefficiencies [4], as the size and numberof interactions and processes that a worker is involved in increases.
We investigate the problem of identifying tasks (and related pa-
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.Copyright 200X ACM X-XXXXX-XX-X/XX/XX ...$10.00.
rameters) and commitments from multiparty textual conversationsover email and chat. Commitments provide a level of deep model-ing that facilitates appropriate progression of tasks. A commitmentis created when a worker becomes responsible for a task, whetherby volunteering or being directed. Once created, a commitmentmay be completed, delegated, or canceled. Identifying and moni-toring commitments automatically in human conversations is chal-lenging as these conversations are ill-structured, are not necessarilygrammatically correct, and contain domain-specific information.For example, such as IP traces, IT-related conversations may con-tain codes and error logs. In addition, as conversations grows inlength and the number of workers involved increases, it becomesdifficult to determine the status of the commitments made by anyof the workers. Moreover, a single sentence may carry informa-tion about several commitments, or several state changes of a givencommitment.
The few existing works dealing with tasks or commitments innatural language processing (NLP) [6, 7, 8] consider only part ofthe problem, i.e., identifying action verb classes in a source suchas message board, email corpus, or chat logs. They do not supportthe identification of commitments, monitoring their progression orlifecycle. In addition, they report low accuracy results.
We introduce a novel approach for the identification of tasks andcommitments based on the speech act theory, people-centric pro-cesses, and machine learning. We summarize the novel contribu-tions of this paper as follows:
• Define commitments and their lifecycle in the context of people-centric processes and case management applications [5].
• Present an NLP-based algorithm by leveraging typed depen-dency [1] for identifying a task and its related parameters(owners, deadline, actions) from a chat or email message.
• Determine whether the communication around a task sig-nals the creation, delegation, cancellation, or discharge of acommitment, by extracting selected features from conversa-tions and a classification-based approach. Analyze complexsentence structures to discover cases where a sentence con-tains multiple task definitions or several lifecycle changes ofa commitment such as creation and delegation.
• Develop an automated agent that monitors conversations toidentify the tasks and progression in a commitment, and non-intrusively presents them to case workers who determine whetherto accept its suggestions.
We have experimentally validated our approach on real-worlddatasets of email and chat conversations. Our approach yields sig-nificantly better accuracy than existing work for task and commit-
1
ment identification, and performs well for identifying the delega-tion, cancellation, and discharge of a commitment, which have notbeen studied by others.
The paper is structured as follows. Section 2 discusses the keybackground topics on commitments. Section 3 defines tasks andcommitments in the context of conversations around people-centricprocesses. Section 4 describes our approach to identifying commit-ments from email and chat conversations. Section 6 explains ourdataset, experimentation and the evaluation results. Section 7 high-lights the limitations of our approach. Section 8 discusses relatedworks. We conclude and discuss future directions in Section 9.
2. BACKGROUNDSeveral researchers have studied how people converse with each
other to create tasks and commitments and collaborate with eachother. We review speech acts [9], language/action perspective [14],and commitments [11] as key concepts to build the foundation forthe approach presented in this paper. These techniques are foun-dations for providing automated support for conversation-orientedmethods for conducting people-centric processes [4].
2.1 Speech ActsSearle [9, 10] classified illocutionary acts into five classes: com-
missives, directives, representatives, expressives, and declarations.A message is classified as a commissive if the sender of the mes-sage promises to take an action in the future. A message is clas-sified as a directive when the speaker intends the receiver to dosomething. A message is classified as a representative if the sendercommits to the truthfulness of the message. A message is classifiedas a expressive when the sender expresses his or her psychologi-cal state. A message is classified as a declaration when the senderof the message brings about a change in status of the referred ob-ject or objects. Searle theory provides the basic idea of identifyingcommissive and directive actions in people’s conversations, how-ever, no work is reported on how commissive and directive actionsprogress in conversation.
2.2 Language/Action PerspectiveWinograd [14] extends speech acts to understand human cooper-
ative activity as conversations. In this model, a message in a con-versation is identified commissive when it is either requested, of-fered, or counter-offered, let’s say from a party (A) to another party(B). The conversation progresses when B accepts the offer from Aand assert A that the conditions are met. Now, if A declares that heor she is satisfied, the conversation reaches a completion state.
2.3 CommitmentsUnlike Winograd’s approach that captures every request as a
commitment, Singh’s model of commitments [11] capture businessrelationships between any two entities. These entities can be eitheremployees within a company or the companies themselves. Specif-ically, commitments denote business meanings underlying the in-teractions between these business entities. In this model, a commit-ment is a conditional business relationship directed from a debtorto a creditor, which can be formalized as
C(DEBTOR, CREDITOR, antecedent, consequent).
The formula shows that the debtor is committed to bringing aboutthe consequent for the creditor provided the antecedent holds. Whena debtor offers a commitment to a creditor, the commitment is cre-ated and becomes active. When the antecedent is brought about, thecommitment is detached and when the consequent holds, the com-mitment is satisfied. If the antecedent holds and the consequent
times out the commitment is violated. If the antecedent is True, thecommitment is unconditional and the antecedent may be omitted:
C(DEBTOR, CREDITOR,>, consequent).
violated
conditional
discharged
consequent
detached
cancel
nullcreate
expire
������
����������
terminated
cancel
Figure 1: The life cycle of a commitment [13].
Telang and Singh [13] present the commitment life cycle as shownin Figure 1. According to Figure 1, a commitment transits fromone state to another due to the following operations: create, de-tach (antecedent holds), discharge (consequent holds), cancel, anddelegate.
• create(c) forms a commitment. A commitment c gets createdwhen a debtor voluntarily offers to do a task or when he isassigned to do a task.
• detach(c) detaches a commitment. A commitment gets de-tached if a condition or an antecedent present for a commit-ment holds true.
• discharge(c) completes a commitment when a debtor exe-cutes a committed task.
• cancel(c) terminates the commitment c. A commitment canbe canceled only by its debtor.
• delegate(c, z) replaces z as the c’s debtor. The debtor of thecommitment c is replaced by z when the debtor delegates hiscommitment.
3. TASKS AND COMMITMENTS IN TEXTIn this section, we discuss tasks and commitments in connection
with textual conversations. To explain the concepts, we use exam-ples from the Enron email corpus and an chat corpus from an HPIT incident management application. The email corpus containsover 250,000 emails sent or received by over 87,000 people. Itconsists of emails exchanged by Enron employees in the time lead-ing up to the Enron bankruptcy that were revealed as part of theFederal investigation of Enron. The chat corpus contains conversa-tions related to HP IT incident management systems. The corpus iscollected from the logs of chat conversations among workers in acustomer facing incident management system for big IT accounts.We extracted logs from the months of May and June 2012 and an-alyzed them for our work.
3.1 TaskA task is a business activity that is either pre-defined (part of a
best practice process) or created on-the-fly by people in a conver-sation [4, 12]. We represent a task as T and define it as
2
T (SUBJECT, OBJECT, action)
In this definition, a subject is a business entity that performs theaction. An object is a business entity for whom an action is beingperformed by a subject. An action is a business activity performedby a subject. An action can be a disjunction or a conjunction ofsubactions. And finally, a deadline represents the time-out condi-tion for an action to be performed. If the time-out occurs, then weconsider the action to be expired. Consider an example from theEnron corpus where Kim sends and email to John: I will pick it uptonight. Here, the subject is Kim and the object is John. The actionis pick it up and the deadline is tonight. We can formally representthis task as T1 = T(KIM, JOHN, pick it up).
3.2 Create CommitmentWe adopt the Singh’s model of commitments [11], reviewed in
Section 2.3, as an action that is performed by a debtor for a creditor.In our context, we map a task to a commitment as follows:
Definition 1. (Mapping a task with a commitment) If a taskT is a commitment C, then subject ∈ T is debtor ∈ C, object ∈ Tis creditor ∈ C, action ∈ T is consequent ∈ C, and deadline ∈ T isconsequent_timeout ∈ C.
Consider an example from the Enron dataset where Earl Chanleysends the following email to Jo Williams We will expedite materi-als and installation in an attempt to meet the target date. To extractthe structure of a commitment from the sentence, first, we identifya task from it and formally we write it as T1 = T(EARL CHANLEY,JO WILLIAMS, expedite materials ∧ expedite installation). Sec-ond, we map T1 to a commitment C1 = C(EARL CHANLEY, JOWILLIAMS, >, expedite materials ∧ expedite installation) as theemail indicates a commitment from Earl (as subject) to Jo (as ob-ject). By the classification of Searle’s illocutionary acts [10] theabove examples represents an offer from Earl to Jo. Therefore wedefine a commissive creates as
Definition 2. (Commissive create) The creation of a commit-ment C is commissive when the debtor voluntarily offers to performthe consequent for the creditor
Similar to a commissive create, a commitment can be a directive.We define a directive create as:
Definition 3. (Directive create) The creation of a commitmentC is directive when the creditor delegates the consequent to thedebtor
Consider and example of a directive create from the Enron corpuswhere Steven Schleimer sends the the following email to Kim Wat-son Please review and send along to your attorneys as soon as pos-sible. We formally write the email as C(KIM WATSON, STEVENSCHLEIMER, >, review ∧ send). In our machine learning ap-proach, we label the sentence as either ccreate or dcreate basedon whether the sentence indicates a commissive or a directive.
3.3 Discharge CommitmentA commitment is discharged when the debtor executes the con-
sequent thereby making it true. An example from the Enron datasetis one where Kim sends an email to Dorothy with the followingmessage, I will also check with Alliance Travel Agency (formerlyTravel Agency in the Park) to see what they may be able to dofor us. The message indicates the creation of a commitment fromKim to Dorothy and can be represented as C1 = C(KIM WATSON,
DOROTHY MCCOPPIN, >, check with Travel Agency). In fol-lowing, Kim sends another email to Dorothy with the followingmessage, I checked with our Travel Agency and they cannot securecheaper tickets. The task T2 from this email T2 = T(KIM, MCCOP-PIN, checked with Travel Agency) discharges C1 as Kim checkedwith the travel agency.
3.4 Delegate CommitmentA commitment is delegated when its debtor outsources it to a
new debtor. There can be two cases of delegation. In the first case,the new debtor does not know about the old creditor. We define thedelegation as
Definition 4. (Unknown creditor delegation) When a debtordelegates his or her commitment C1 to a debtor’, then debtor ∈ C1is the creditor’ ∈ C2 and both the commitments C1 and C2 can bewritten as
C1 = C(debtor, creditor,>, consequent)
C2 = C(debtor′, debtor,>, consequent)
Consider an example from the Enron dataset where Gregory sendsan email to Glen with the following message, Please take a fewmoments to review the same and let me know your thought. Inthis email, Gregory delegates a commitment to Glen for review-ing something. We formally represent the commitment as C1 =C(GLEN HASS, GREGORY KLATT, >, review statements). In thefollow up, Glen sends another email to Steven with the message,This appears to be to be OK and we should be able to sign onhowever please review the statement and let me know if you seea problem with our support of the PHC statement. In this email,Glen delegates his commitment to Steven thereby creating anothercommitment from Steven to him. This commitment can be for-malized as C2 = C〈STEVEN HARRIS, GLEN HASS, >, reviewstatements〉. As, we can see this commitment has a new debtor(debtor’) as Steven while the creditor’ Glen is the debtor of theprevious commitment.
Now, in the second case of the delegation, the new debtor is com-mitted to the old creditor. We define the delegation as
Definition 5. (Known creditor delegation) When a debtor del-egates his or her commitment C1 to a debtor’, then creditor ∈ C1is the creditor’ ∈ C2 and both the commitments C1 and C2 can bewritten as
C1 = C(debtor, creditor,>, consequent)
C2 = C(debtor′, creditor,>, consequent)
Consider an example from the Enron dataset where Robert sendsan email to Drew with the following message, Please let me knowif your business unit has any problem with this course of action.In this email, Drew creates a commitment with Robert for lettinghim know something. The commitment is formalized as C(DREW,ROBERT, >, inform about business unit having any problem). Infollowing up, Drew sends another email to Kim; Please review thismessage and advise Robert Williams of any relevant information.In this email, Drew delegates his commitment to Kim thereby cre-ating another commitment from Kim to Robert. This commitmentcan be formalized as C(KIM, ROBERT, >, provide information).As we can see, this commitment has a new debtor as Kim while thecreditor is still the same i.e., Robert.
3
3.5 Cancel CommitmentA commitment is canceled when the debtor of the commitment
terminates the commitment. An example from the Enron datasetis Diane sends an email to Kimberley, Can you please provide methe details about the amounts from prior months by this Friday?.In this email, Diane delegates a commitment to Kimberley that canbe represented as C1 = C(KIMBERLEY, DIANE, >, provide theinformation). Later, Kimberley sends an email back to Diane withthe message, I cannot not give you the details by Friday, therebycanceling her existing commitment toward Diane.
4. IDENTIFYING TASKS AND COMMITMENTS
Our process for the identification of commitments from emailsand chat conversations is depicted in Figure 2. We first extract sen-tences from the text of conversations. We then identify the tasksand then candidate commitments or changes in the lifecyle of com-mitments using a combined NLP-based and machine learning ap-proach. Using NLP and a set of heuristic-based rules applied onfeatures extracted from the text of conversations, we identify cer-tain tasks and commitments. However, the application of NLP-based rules is limited to identifying pre-determined classes and pat-terns of tasks and commitments. We augument our approach witha supervised machine learning approach for the identification ofcommitments and their lifecycle. This helps in the identification ofcommitments for which their various expressions and forms in thenatural language may not be captured in patterns and rules.
�������
������� ������
�������������
�������
���������
�������
�������
� ������������
���������
� �����
�������
���������
� ������
��������
���������
� �����������
���������
Figure 2: Process followed to identify commitments.
For identification of a task T from a sentence, we check whetherthe sentence has a subject, an object and an action verb is present.Although it sounds simple, the identification of tasks in a sentenceis nontrivial. For example, changing from active to passive voiceresults in shifting positions of words that indicate subject, object, orthe action. To deal with this issue, we chose to parse the sentencesusing the typed dependency method [1] which outputs the relationsbetween individual words in a sentence, and is largely independentof the exact sentence structure. A relation between any two wordsis defined as
Definition 6. (Relation) A relation is a triplet of the name of therelation, governor, and dependent where the governor and depen-dent are words from a sentence and it can be represented as
relation(governor, dependent)
Marneffe et al. [1] represent such relationships in a hierarchicalmanner with the most generic relation as the root. For example, arelationship can start with an arg (argument) and it can branch intothe subj (subject) and the comp (complement) relationship. Con-sider a sentence, Lorraine, I will be in a meeting during this time.
In the sentence, the triplets are root (root, be), nsubj (be, I), aux (be,will), advmod (be, Lorraine), prep_in (be, meeting), det (meeting,a), prep_during (be, time), det (time, this). Figure 3 shows thetriplets in a graph format. In this sentence we can see that there isan nsubj relation in which I is the dependent and be is the governor.This relation suggests that the subject in the sentence is I and hisaction is be.
����
��
����
����
��� �������� �
������ ��
����� � ����� �
�
���
�����
����
����
���
Figure 3: Typed dependencies derived from a sentence “Lor-raine, I will be in a meeting during this time” chosen from theEnron corpus.
4.1 Identifying Conversations and SentencesWe preprocess the Enron and the HP IT incident management
datasets to make them suitable for parsing and extracting features.Since the email and chat dataset types are differently structured,we follow different steps to preprocess them from both these types.For email, we separate information such as sender, receiver, date,and subject. Then we prepare conversation threads by collecting allthe emails either replied or forwarded with the same subject name.Next, we split each email into its constituent sentences and parseeach of these sentences to extract features. Unlike, emails we donot prepare conversation threads for chat conversations as they arealready listed chronologically.
4.2 Extracting FeaturesWe perform the following steps to extract features from each sen-
tence in emails and chat messages.
• Co-reference resolution relates a name with a personal pro-noun. For example, in a pair of sentences Please add JimCurry to your list. He should be part of the due diligenceteam, the co-reference resolution helps to relate Jim Curry(name) with He (personal pronoun). This is important be-cause several conversations start with you or he or she or theyand it is necessary to resolve these pronouns so that we canfind the debtor for a commitment.
• Named entity resolution (NER), identifies a noun whetherit’s a PERSON or an ORGANIZATION. Upon identifying acommitment we check whether the debtor and the creditor ofthe commitment is a valid debtor by checking if it is a PER-SON or an ORGANIZATION from the resolved name entities.
• Part-of-speech tags extraction We extract Part-Of-Speech(POS) tags to tag a word with a part of speech. The POStags help to identify the type of personal pronoun for a sub-ject and the state of the verb associated with the subject soas to identify the debtor of a commitment and the state ofa commitment, respectively. The present tense of the verb
4
indicates that the commitment is either a commissive or a di-rective whereas the past tense indicates the commitment isdischarged. Examples of POS tags used in our work includeNN (for nouns), NNP (pronouns), PRP (personal pronouns),VB (present tense verbs), VBD (past tense verb), VBZ (3rdperson singular verb), VBN (past participle) and VBP (non-3rd person singular).
• Typed dependencies extraction As discussed in Section 4,a typed dependency relates words in a sentence and gives aclue as to its logical structure.
Before explaining the algorithms for identifying tasks and commit-ments, we briefly mention the features in the feature vector that areused to train classifiers for email and chat datasets. The featuresare based on properties that help identify a sentence as creating,delegating, discharging, or cancel. The features are:
A modal verb signals the creation of a commitment. For ex-ample, the sentence He will handle the issuance of the LC. has amodal verb will that indicates the creation of a commitment. Anaction verb indicates whether a commitment is present in a sen-tence. For example, I can imagine that your family reunions arejust a hoot! is not a commitment because the verb imagine is notan action verb. The present tense signals the creation, delegation,or cancellation of a commitment. In the following example: Sheand I will get together on the results of these meetings. that sug-gests a create commitment, handle and get are in the present tense.And, the past tense signals the discharge of a commitment. In thefollowing example: I have reviewed the list you sent me regard-ing items you would like to see in the Data Room. that suggests adischarge commithas has the action verb reviewed in the past tense.
The debtor of a commitment is the task performer. The cred-itor of a commitment is the one debtor commits to. A deadlineindicates a commitment creation or delegation. For example, in thesentence It will be posted today and the policy will go into effect forFriday’s gas day. indicates today and Friday as the deadlines. Theprior creation of a commitment is a prerequisite for discharge, del-egation, and cancellation if the create commitment already exists.And a delegation signal is identified when a debtor occurring as acreditor indicates that the debtor delegates an existing commitmentto a new debtor. The negative verb indicates the presence of a can-celed commitment. For example, in the sentence I cannot give youthe amount details not give indicates a negative verb.
The type of the personal pronoun in the subject indicates a com-mitment being created, canceled, or discharge (first, second, orthird person) or delegation (second, third). The bigram of a modalverb and a second person PRP indicates a directive creation. Forexample, in the sentence Can you help me with the following out-standing items relating to the Info Memo, the bigram Can you in-dicates directive creation. The bigram of a first person PRP and amodal verb indicates a commissive. For example, in the sentenceWe will expedite materials and installation, the bigram We will in-dicates commissive creation.
The bigram of ‘please’ and an action verb indicates a directive.For example, in the sentence Please review and send along to yourattorney as soon as possible, the bigram please review indicates adirective commitment creation. And, a question mark in a sentenceindicates a directive commitment creation.
4.3 Identifying TasksTo identify a task from a sentence, we first extract features dis-
cussed in Section 4.2. The features are input to Algorithm 1 toidentify subject, object and the action that the subject performs fora candidate task. Our algorithm slightly differs between emails and
chats especially in extracting the subject and the object as in caseof chats it is more difficult to find them than in emails. In case ofemails, it is easier to find them because the metadata provides in-formation such as sender’s and receiver’s names. In case of chatmessages, it is difficult because we only get the chat initiator nameas the metadata and the co-reference resolution does not work wellfor resolving pronouns such as you in multiparty conversations.
����
�����
�
����
���������
��
�
� ����
��� �
�� �
������
����� �����
�� ��
������ ������ ����� �������
������ �����
Figure 4: Steps to identify task from a email sentence “I willalso check with Alliance Travel Agency”.
Once we parse a sentence, we get a typed dependency array asin the example shown in Figure 4. We define the typed dependencyarray as
Definition 7. (Typed dependency array) The typed dependencyarray consists of relations derived from a sentence using the typeddependency method. The typed dependency array is represented interms of relations as, typed dependency array = {relation1, relation2,relation3, . . . }.
In the above definition, a relation can be an nsubj, aux, advmod, andso on. In a typed dependency array, we look for nsubject relationand check if the dependent is a valid subject and the governor is avalid verb. We define subject validity as
Definition 8. (Valid subject) A dependent is considered a validsubject if the POS tag associated with the dependent is NNP orNER resolves the subject as a PERSON or an ORGANIZATION.
We define verb validity as
Definition 9. (Valid verb) A governor is considered valid if thePOS tag associated with the governor is either VB, VBD, VBP,VBZ, or VBN and if the governor is an action verb.
We define an action verb as
Definition 10. (Action verb) The verb that expresses an actionor doing something.
By checking for an action verb, we eliminate words that are verbsbut do not express actions. If both the governor and the dependentare valid, we store the dependent as the subject and the governoras the action for the subject. We extract the action details using theaction verb by finding its dependencies in the array of triplets bylooking for nouns or verbs associated with the action verb. Algo-rithm 1 shows the steps to extract subject, object, and action from
5
a sentence in emails. In this algorithm if the POS tags associatedto the dependent is PSP first person (I, we) then the subject is thesender of the email. In case of chats, the subject is the chat initia-tor. If the POS tag associated with the dependent is PRP secondperson (you), then the receiver is the subject of the task. In caseof chats, it is difficult to resolve you in a multiparty conversations.If the POS tag associated with the dependent is PRP third person,then the approach for both emails and chats is the same. We con-sider the dependent as the subject. If the POS tags associated withthe dependent is personal PRP (he, she, they) then we resolve themwith co-reference resolution.
Algorithm 1: extract task(sentence)
1 typed dependency array← get typed dependencies(sentence);2 foreach nsubj ∈ typed dependency array do3 if get POS(dependent ∈ nsubj) is PRP first person then4 subject← sender of the email;5 object← receiver of the email;6 else7 if get POS(dependent ∈ nsubj) is PRP second person
then8 subject← receiver of the email;9 object← sender of the email;
10 else11 if get POS(dependent ∈ nsubj) is PRP third person
then12 subject← dependent;13 object← extractObject(sentence);14 if valid subject(dependent) ∧ valid verb(governor) then15 action← extract action details(governor);
Figure 4 represents the typed dependency array for the sentenceI will also check with Alliance Travel Agency. We extract the nsubjrelationship i.e., nsubj (check, I). The subject is I and the actionverb is check. Once we extracted the subject and the action, weextract nouns or verbs related to action such as Agency, Alliance.
4.4 Identifying CreateOnce we have extracted a task from a sentence, we check whether
the task indicates the creation of a commitment. To identify suchtasks, we check whether the action in a task has a relationship witha modal verb and the word please. A modal verb can be defined as
Definition 11. (Modal verb) A modal verb is a word that ex-presses possibility, likelihood, or obligation. Words that indicatemodality are will, shall, can, could, would, should, may, might andmust.
The word please indicates a request or a delegation of a task. Ifthe verb has a relation with these words (modal or please), then welabel the task as commitment creation and store the subject as thedebtor, the object as the creditor, and the action as the consequentrespectively of the commitment. Also, as discussed in Section 3.2,commitments can be created in two ways: the commissives or di-rectives. We mark a sentence as a commissive when an action verbis following a modal verb. We mark a sentence as a directive whena second person personal pronoun such as you is following a modalverb or an action verb following the word please. Algorithm 2 sum-marizes the method for identifying commitments from sentences.
In Figure 5, the action verb check is in the present tense (VB)and has a relationship with a modal verb will. Therefore, the taskis considered as creating a commitment. Since here the action verb
������ ����� ������ ����
�������
�
������ ����������
��������� ������
Figure 5: Steps to identify a commitment from a email sentence“I will also check with Alliance Travel Agency”.
follows the modal verb, we consider it a commissive. In Algo-rithm 2, we check if the action is in present tense (VB). Then, wecheck if there is a relation that associates the action with a modalverb or please.
Algorithm 2: identify create commitment(sentence)
1 T1← extract task(sentence);2 commitment←⊥;3 if getPOS(action verb ∈ T1) is VB then4 foreach relation ∈ typed dependency array do5 if (dependent ∈ relation is action) ∧ (governor ∈
relation is (modal verb ∨ please)) then6 commitment←>;7 else8 if (governor ∈ relation is action) ∧ (dependent ∈
relation is (modal verb ∨ please)) then9 commitment←>;
4.5 Identifying DelegateWe discussed in Section 3.4 that there are two kinds of delega-
tion. In one, the old creditor is unknown to the new debtor, whereas,in another, the old creditor is known to the new debtor. ConsiderFigure 6(a) that shows an example where the new debtor Stevendoes not know about the old creditor Gregory. Similarly, considerFigure 6(b) where the new debtor Kim is committed to the old cred-itor Robert.
�������� ������ ��
��������� �����������
�����
������
����������
�����
�����
�����
��������
������ ������
������ �� ��������
������!�� ������ ��
������������ �����������
�����
"�����������
����������
!��#��$
!��
�����
��������
������ ������
�� ��������
"�����
��������������� �����
������������� �����
Figure 6: Steps to identify delegate.
To identify delegation in sentences, we check a commitment C2created after another commitment C1. To find the unknown credi-tor delegation, we check if the debtor in C1 is the creditor in C2 andif the consequents in C1 and C2 are the same. To find the knowncreditor delegation, we check if the creditor in C1 is the creditor inC2 and if the consequents in C1 and C2 are the same. To matchthe consequents in the commitments, we check whether the actionverbs in both commitments are the same or related as synonyms,hypernyms, or hyponyms. If they are the same or related, we check
6
whether nouns in both the commitments are the same or related us-ing the co-reference resolution. Finding delegation in case of emailis easier than chats as we know in advance the debtor and creditorof the commitment based on the sender and receiver information.Unlike emails, in chats, finding delegation is difficult as sometimesit is extremely hard to find the creditor of a commitment.
Algorithm 3: identify delegate commitments(sentence)
1 C2← identify create commitments(sentence);2 delegation←⊥;3 unknown creditor←⊥;4 known creditor←⊥;5 foreach C1 ∈ commitment array do6 if debtor ∈ C1 is creditor’ ∈ C2 then7 unknown creditor←>;8 else9 if debtor ∈ C1 is creditor’ ∈ C2 then
10 known creditor←>;11 if action verb ∈ C1 is action verb ∈ C2 then12 if nouns ∈ C1 is nouns ∈ C2 then13 if unknown creditor ∨ known creditor then14 delegation←>;
4.6 Identifying DischargeIf an identified task is in the past tense, it may signals a dis-
charge commitment, and we need to compare it with existing com-mitments. For greater clarity, let us consider an example of a com-mitment C1 and a task T2.
�������� ���� ���� �� �� ��������
���������� ���� �� ������
������
������������
��������
�������
�������
������
��������
Figure 7: Steps to identify discharge.
To check if T2 discharges C1, we compare the subject and objectin T2 with the debtor and creditor in C1 respectively. If they arethe same, we compare their action verbs. We check whether theaction verb in T2 is in the past tense (VBD). Then we compare theaction verb in T2 by converting it into its base form and trying tomatch with the action verb in C1. If both the verbs are the same orrelated as either synonyms, hypernyms, or hyponyms, we comparethe nouns in both the tasks. If they are the same or related, wemark T2 as discharging C1. Figure 7 clearly shows that debtor andcreditor in both C1 and T2 are the same. The main action verbin T2 is checked and it is in the past tense. Therefore, we convertit into its base form check and compare it with the action verb inC1. Note that the base form of a verb is the simplest form in whichit appears in a dictionary without any ending. Since they are thesame, we compare the nouns in C1 and T2. We see that the nounsare related. Therefore, we mark T2 as discharging C1. Algorithm 4describes these steps.
4.7 Identifying CancelFor identifying a canceled commitment, we compare a task with
the commitments that already exist and check whether there is arelation in the type dependency array where the action verb is as-sociated with a negative word such as not.
Algorithm 4: identify discharge commitments(sentence)
1 T2← extract task(sentence);2 discharge←⊥;3 foreach C1 ∈ commitment array do4 if (subject ∈ T2 is debtor ∈ C1) ∧ (object ∈ T2 is
creditor ∈ C1) then5 if (getPOS(action verb ∈ T2) is VBD) ∧
(getBaseVerb(action verb ∈ T2) is action verb ∈ C1)then
6 if nouns ∈ T2 is nouns ∈ C1 then7 discharge←>;
��������� ��� � �� ��������
������ ��� ���
��� ��
�� ���������
���������
����
����
�����
����� ��
�� �� ���
Figure 8: Steps to identify cancel.
Figure 8 describes an example of a commitment C1 and a taskT2. Once we find that the debtor and the creditor in C1 and T2are the same, we compare their main action verbs and nouns re-spectively. If they are same, we check whether the action verb isa negative verb. Note that a verb considered as negative if eitherit has a relation with a negative word such as not. Algorithm 5describes the above steps.
Algorithm 5: identify cancel commitments(sentence)
1 T2← extract task(sentence);2 cancel←⊥;3 foreach C1 ∈ Commitment Array do4 if (subject ∈ T2 is debtor ∈ C1) ∧ (object ∈ T2 is
creditor ∈ C2) then5 if getBaseVerb(action verb ∈ T2) is action verb ∈ C2
then6 if nouns ∈ T2 is nouns ∈ C1 then7 if negative verb(action verb ∈ T2) then8 cancel←>;
5. EVALUATION AND PROTOTYPETo validate our approach, we use two methods: (1) automatic la-
beling of data (sentences) using Algorithms 2, 3, 4, and 5, and thenusing 10-fold cross validation, and (2) manually labeling a subsetof data used for training and testing our approach. We applied threemachine learning classifiers and evaluated them. We describe thedata, the extracted features from the datasets below.
5.1 DataFor evaluation and validation purpose, from the Enron email cor-
pus, we selected 4161 email sentences that were exchanged be-tween Kimberly Watson, an employee of Enron, and more than 50people which includes her co-workers at Enron, clients, friends,and family members. The emails were collected and prepared bycombining the data from text files provided by CMU [3] as well asfrom a database dump [2] provided by UC Berkeley. We combinethe data from both so as to reduce the missing information. Forthe chat data, we selected 271 conversations from HP IT incident
7
management logs comprising of 7154 sentences.
5.2 Labeling DataOnce the features are extracted for each sentence from the dataset,
we had two annotators to label the sentences. We cross checked theresults for any possible conflict resolution. We resolved conflictsby allowing two annotators to collaborate and discuss their labelsfor sentences. Next, we ran classifiers the three classifiers of NaïveBayesian (NB), Logistic Regression (LR), and Support Vector Ma-chines (SVM) and used 10-fold cross validation to produce results.
We annotated 4161 email and 7154 chat sentences. The twoindependent annotators achieved overall an interrater agreement(kappa score) of 0.83 for both email and chat. Table 3 shows thedistribution of the email and chat sentences as annotated.
6. RESULTSWe annotated 4161 email and 7154 chat sentences. The two
independent annotators achieved overall an interrater agreement(kappa score) of 0.83 for both email and chat. Table 3 shows thedistribution of the email and chat sentences as annotated.
Table 3: Distribution in email sentences.Classes Email Chat
Commissive create 342 532Directive create 162 214Discharge 38 250Cancel 7 16Delegate 12 12None 3540 6130
Table 1 and Table 2 represent results for email and chat datarespectively, using NB, LR, and SVM classifiers and ten-fold crossvalidation. In both the tables, we represent C-create as commissivecreate, D-create as directive create, P as precision, R as recall andF as F-measure.
First, we considered one of the simplest probabilistic text classi-fication approach, Naïve Bayes (NB). The NB classifier approachassumes that attributes in consideration are independent of eachother. Using NB, for emails, our results show high precision forcommissive creation(84%) and directive creation(81%) while lowprecision for delegation(32%), discharge(21%), and cancellation(0%).In case of chats, we obtain slightly lower precision for commissivecreation (73%) than email, however, we obtain higher precision fordirective (85%) and discharge (60%). The precision remains samefor cancel (0%). For delegate, the precision (0%) was low com-pared to precision for email.
Second, we used the Logistic Regression (LR) classifier. Us-ing LR, in emails, we obtain significantly high precision values forcommissive (90%), directive (92%) compared to NB while low pre-cision values for delegate (64%), discharge(27%), and cancel (0%).In case of chats, LR performs better than NB for commissive cre-ation and cancellation with precision of 80% and 22% respectively.However, results for other classes are lower, i.e., 74% for direc-tive creation, 64% for discharge, and 0% for delegate. Overall, theresults for chat using LR are lower compared to emails except dis-charge and cancel.
Third, we used the Support Vector Machine (SVM) classifier.Using SVM, in emails, we obtain significantly higher precision,compared to NB and LR, for commissive creation (87%), directivecreation(94%), discharge(100%), and delegation(86%) while sameprecision for cancellation (0%). For chats, using SVM, we obtain
lower precision than emails for commissive creation (79%), direc-tive creation (73%), and discharge (63%). However, the f-measureand recall value for discharge is higher in chats than emails. Again,this is due to the higher percentage distribution of discharge inemails and chats.
Overall we obtain high precision, recall and f-measure for a com-missive creation and a directive creation for both emails and chatsusing NB, LR, and SVM. The results for both the classes are highbecause both the classes are independent of each other and the dis-tribution of these classes are high in both the datasets. The resultsfor other classes are low because other classes depends on the priorexistence of create commitment classes and it is difficult to findthis specific feature automatically. Compared to discharge and del-egate the results for delegate is higher in case of emails becausewe can easily find out the debtor and the creditor of a commitmentbased on the sender’s and receiver’s information. The results fordischarge in email is low because the percentage of distributionof discharge is extremely low as lot of tasks indicating dischargeare executed, however, are never mentioned in emails. In case ofchat, the precision for discharge is higher because the distributionof discharge is high as people in chat conversations immediatelyreport their progress. However, the overall percentage is low forboth emails and chats because it is difficult to compare consequentby matching verbs and nouns. For emails, we find a high precisionvalue using SVM with low recall and f-measure. We attribute thehigh precision of SVM to some of the sentences in emails that wereidentified accurately as discharge by our algorithm. For cancel, wegot 0% precision in emails and 22% in chats, This is because it isagain difficult to find out the prior existence of a commitment aswell find out the negative words associated with the action verb.
We evaluated our training model on independent test datasets.Our test datasets contains 1326 email and 2299 chat sentences. Foremails we used SVM and for chats we used LR.
Table 4: Evaluation on test datasets using SVM for emails andLR for chats respectively.
Email Chat
Classifier P R F P R F
C-create 0.97 0.84 0.90 0.90 0.89 0.89D-create 0.94 0.78 0.86 0.83 0.51 0.63Discharge 0.00 0.00 0.00 0.66 0.71 0.69Cancel 0.00 0.00 0.00 0.00 0.00 0.00Delegate 1.00 0.33 0.98 0.00 0.00 0.00None 0.96 0.99 0.98 0.96 0.98 0.94
6.1 Initial PrototypeFigure 9 represents the architecture of our interactive tool for
commitment identification and tracking. The tool offers an agentthat can be pluggined into online chat messengers, and identifiesdiscovered commitments in a panel for verification and confirma-tion to conversation participants. When a person sends a chat mes-sage, it parses the sentence using our parser component and extractpotential task details and features. Using our predictor componentand a trained model we identify the class based on extracted fea-tures. The parser and predictor components are Java-based anduse the Stanford parser and Weka libraries respectively. We pre-dict classes using the SVM classifier. Finally, the display compo-nent produces the summary of all tasks and commitments based onchats. Figure 10 shows a screenshot of the prototype tool.
8
Table 1: Results for emails.C-create D-create Discharge Cancel Delegate None
Classifier P R F P R F P R F P R F P R F P R F
NB 0.84 0.94 0.89 0.81 0.93 0.87 0.21 0.41 0.27 0.00 0.00 0.00 0.32 0.39 0.35 0.99 0.95 0.97
LR 0.90 0.95 0.95 0.92 0.93 0.94 0.27 0.05 0.08 0.00 0.00 0.00 0.64 0.34 0.48 0.98 0.98 0.98
SVM 0.87 0.97 0.92 0.94 0.97 0.95 1.00 0.02 0.04 0.00 0.00 0.00 0.86 0.33 0.48 0.98 0.98 0.98
Table 2: Results for chats.C-create D-create Discharge Cancel Delegate None
Classifier P R F P R F P R F P R F P R F P R F
NB 0.73 0.90 0.81 0.85 0.50 0.63 0.60 0.70 0.75 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.97 0.80
LR 0.80 0.85 0.83 0.74 0.51 0.60 0.64 0.70 0.67 0.22 0.13 0.16 0.00 0.00 0.00 0.97 0.98 0.97
SVM 0.79 0.85 0.82 0.73 0.53 0.61 0.63 0.71 0.67 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.97 0.97
Figure 10: A screenshot of our prototype tool for commitment identification and monitoring.
��������
����� �������
������
� �� ����
����������
�����
�������������
������
������ �
� �� ����
������
�������
����������������� �����������������������������
����������� ��
Figure 9: The architecture of our commitment identificationand monitoring tool.
7. LIMITATIONSOur work has several limitations. First, to identify a task struc-
ture we depend on the typed dependency feature which does notwork accurately when the sentence is either too long or too short.Identifying multiple tasks in a sentence is again difficult as the
typed dependency feature does not work well for sentences withconjunctions such as and and but. Second, for judging the simi-larity in consequents of two commitments, we rely on comparingwords. Third, we rely on the type of a personal pronoun such asfirst person, second person, and third person and map the pronounseither to a sender or a receiver of an email. Without the presenceof a sender and a receiver we cannot determine the debtor and thecreditor of a commitment accurately. Fourth, in case of chats wecannot find debtor every time and it is difficult to resolve pronounsuch as you. Fifth, if a sentence has a missing subject and an objectthen we cannot determine if a prior commitment exists.
8. RELATED WORKSThe research in this space are just emerging. While there has not
been previous work on commitment lifecycle identification, thereare few works on using NLP and machine learning techniques foridentifying action verbs and tasks from texts.
Qadir and Riloff [7] classify sentences from message board postsas commissives, directives, expressives, and representatives. Forcreating a training model, they extract lexical, syntactic, and se-mantic features for messages. Using these features, they train theirmodel using the SVM classifier. They obtain 45% precision com-
9
missives class and 97% for directives. In the identification of com-missive and directive commitment creation, we consider personalpronoun, modals, and sentence begins with modal/verb. Basedon these and other features and our novel algorithms, we obtaina higher precision for commissive create and directive creation.
Scerri et al. [8] focus on action items in emails and check whetherthese action items fall under the request, suggest, assign, and de-liver classes. For identifying the classes, they use a rule-based clas-sification model. To evaluate their classification model they com-pare the classes identified automatically with the classes identifiedby a group of 12 people. Their evaluation results show the preci-sion value of 56% and recall value of 60% respectively. In con-trast, first we identify classes for each sentence based on a novelNLP-based algorithm. Next, we apply machine learning to identifyclasses based on the identied features from the text, which allowsus to identify classes in case where it is difficult to capture corre-sponding rules. In addition, we monitor and discover the lifecycleof a commitment, once it is created.
Purver et al. [6] focus on multiparty meetings and identify taskowner, task, deadline, and agreement classes from these meetings.For building a training model in the baseline approach, they labelthe data manually and train it using SVM classifier. Their resultshows a precision of 25%. Apart from the baseline approach theyuse hierarchical annotation techniques where they identify classesusing the NOMOS annotation software and build their model us-ing the Naïve n-gram classifier. The precision values for task de-scription, task owner, deadline, and agreement are 23%, 12%, 19%,and 48% respectively. In contrast, we completely rely on the type-dependency feature to extract task owner, task, and deadline in asentence. Although the type-dependency extracts features some-what accurately, it extracts features incorrectly in many cases. There-fore, we emply a machine learning-based approach to identify thestructure of a task more accurately.
9. CONCLUSIONS AND FUTURE WORKWith the rise in the informal collaboration and web-based com-
munication tools, the problem of organizing work in the collabo-rational setting and in the context of people-centric processes be-comes more vital for the organizations. To the best of our knowl-edge, this is is the first work that presents an approach for the auto-mated identification and monitoring of commitment lifecycle fromthe text of conversations amog people. We introduced an approachfor extracting tasks and commitments from sentences in email andchat conversations. Our NLP-base algorithms leverages typed de-pendency for identifying tasks and its related parameters. Our ma-chine learning approach accurately identifies whether a task indi-cates a commitment and further identify whether a task progressesa commitment or terminates it. Using our machine learning ap-proach we get high precision for commissive and directive creationon both emails and chats. Our approach also provides promisingresults for the delegate, discharge and cancel classes. Finally, weprovide a tool that implements the approach, and can be used forassisting workers in capturing commitments in collaboration tools.
We have several future directions. First, we plan to improve re-sults for delegation, discharge, and cancellation, specifically in chatconversations. Second, we plan to explore and compare other ap-proaches for identifying tasks from sentences. Third, we plan toconsider applying unsupervised machine learning such as HiddenMarkov Models (HMM) to this problem as the supervised machinelearning approach relies on manual labeling of data, which is te-dious to provide. Fourth, another future step is to reconstruct con-versations from both emails and chats and then apply our approach.
10. REFERENCES[1] M. De Marneffe, B. MacCartney, and C. Manning.
Generating typed dependency parses from phrase structureparses. Proceedings of LREC, 6:449–454, 2006.
[2] A. Fiore and J. Heer. UC Berkeley Enron email analysis.2004.
[3] B. Klimt and Y. Yang. The enron corpus: A new dataset foremail classification research. In Machine Learning: ECML2004, volume 3201 of Lecture Notes in Computer Science,pages 217–226. Springer Berlin / Heidelberg, 2004.
[4] H. R. Motahari-Nezhad, C. Bartolini, S. Graupner,S. Singhal, and S. Spence. It support conversation manager:A conversation-centered approach and tool for managingbest practice it processes. In Proceedings of the 2010 14thIEEE International Enterprise Distributed ObjectComputing Conference, EDOC ’10, pages 247–256,Washington, DC, USA, 2010. IEEE Computer Society.
[5] H. R. Motahari-Nezhad, C. Bartolini, S. Graupner, andS. Spence. Adaptive case management in the socialenterprise. In Service-Oriented Computing, volume 7636 ofLecture Notes in Computer Science, pages 550–557.Springer Berlin Heidelberg, 2012.
[6] M. Purver, P. Ehlen, and J. Niekrasz. Detecting action itemsin multi-party meetings: Annotation and initial experiments.In Proceedings of the Third International Conference onMachine Learning for Multimodal Interaction, pages200–211. Springer-Verlag, 2006.
[7] A. Qadir and E. Riloff. Classifying sentences as speech actsin message board posts. In Proceedings of the Conference onEmpirical Methods in Natural Language Processing, pages748–758, Stroudsburg, PA, USA, 2011. Association forComputational Linguistics.
[8] S. Scerri, G. Gossen, B. Davis, and S. Handschuh.Classifying action items for semantic email. In Proceedingsof the 7th Conference on International Language Resourcesand Evaluation (LREC), 2010.
[9] J. R. Searle. Speech Acts. Cambridge University Press,Cambridge, UK, 1969.
[10] J. R. Searle. A taxonomy of illocutionary acts. InK. Gunderson, editor, Language, Mind and Knowledge.University of Minnesota Press, Minneapolis, 1975.
[11] M. P. Singh. An ontology for commitments in multiagentsystems: Toward a unification of normative concepts.Artificial Intelligence and Law, 7(1):97–113, Mar. 1999.
[12] P. R. Telang and M. P. Singh. Enhancing Tropos withcommitments. In A. Borgida, V. K. Chaudhri, P. Giorgini,and E. S. K. Yu, editors, Conceptual Modeling: Foundationsand Applications, volume 5600 of LNCS, pages 417–435.Springer, 2009.
[13] P. R. Telang and M. P. Singh. Specifying and verifyingcross-organizational business models: An agent-orientedapproach. IEEE Transactions on Services Computing,5(3):305–318, July 2012.
[14] T. Winograd. A language/action perspective on the design ofcooperative work. In Proceedings of the 1986 ACMConference on Computer-supported Cooperative Work,CSCW ’86, pages 203–220, New York, NY, USA, 1986.ACM.
10
Identifying Business Tasks and Commitmentsfrom Email and Chat Conversations
A. K. Kalia1,2∗, H.R. Motahari-Nezhad1, C. Bartolini1, M. P. Singh2
1Hewlett Packard Laboratories, Palo Alto, CA, United States{hamid.motahari, claudio.bartolini}@hp.com
2NC State University, Raleigh, NC, United States{akkalia, singh}@ncsu.edu
ABSTRACTCase management applications and, more generally, people-centricprocesses involve the definition, resolution and communication ofcommitments for tasks over channels such as chat and email. Iden-tifying and tracking tasks and commitments can help in stream-lining the collaborative work in business environments. However,doing so proves challenging due to the syntactical, grammatical,and structural incompleteness of human conversations over chatand email channels. We present a novel approach to automaticallyidentify tasks and commitment creation, delegation, completion,and cancellation in email and chat conversations, based on tech-niques from natural language processing and machine learning do-mains. We discover tasks and related parameters from the text ofconversations, identify when a commitment to a task emerges andfind the state changes of a commitment based features extractedfrom the text of the conversations. We have developed a proto-type and evaluated our approach using real-world chat and emaildatasets. Our experiments shows high precision for create class i.e.,90% in emails (Enron email corpus) and 80% in a real-world chatdataset and also provides promising results for discharge, delegate,and cancel classes.
1. INTRODUCTIONMany processes in organizations are people-driven, and are man-
aged in a collaborative and conversation-oriented manner. Conver-sations around people-centric processes involve defining and coor-dinating tasks through commitments among workers over informalchannels such as email and chat. Typical examples of such conver-sations include the handling of insurance claims and IT incidents.For instance, in IT incident management, incident reports are as-signed to help desk workers and in some cases to specialized ITexperts. The handling usually proceeds through team collabora-tion and communication over email and chat, in addition to keep-ing record in specialized systems. Keeping track of all agreed-upon
∗The main work was done while the first author was a summerintern at HP Labs, Palo Alto
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.Copyright 200X ACM X-XXXXX-XX-X/XX/XX ...$10.00.
tasks and commitments made during a conversation is a dauntingtask, and a major source of inefficiencies [4], as the size and numberof interactions and processes that a worker is involved in increases.
We investigate the problem of identifying tasks (and related pa-rameters) and commitments from multiparty textual conversationsover email and chat. Commitments provide a level of deep model-ing that facilitates appropriate progression of tasks. A commitmentis created when a worker becomes responsible for a task, whetherby volunteering or being directed. Once created, a commitmentmay be completed, delegated, or canceled. Identifying and moni-toring commitments automatically in human conversations is chal-lenging as these conversations are ill-structured, are not necessarilygrammatically correct, and contain domain-specific information.For example, such as IP traces, IT-related conversations may con-tain codes and error logs. In addition, as conversations grows inlength and the number of workers involved increases, it becomesdifficult to determine the status of the commitments made by anyof the workers. Moreover, a single sentence may carry informa-tion about several commitments, or several state changes of a givencommitment.
The few existing works dealing with tasks or commitments innatural language processing (NLP) [5, 6, 7] consider only part ofthe problem, i.e., identifying action verb classes in a source suchas message board, email corpus, or chat logs. They do not supportthe identification of commitments, monitoring their progression orlifecycle. In addition, they report low accuracy results.
We introduce a novel approach for the identification of tasks andcommitments based on the speech act theory, people-centric pro-cesses, and machine learning. We summarize the novel contribu-tions of this paper as follows:
• Define commitments and their lifecycle in the context of people-centric processes and case management applications [?].
• Present an NLP-based algorithm by leveraging typed depen-dency [1] for identifying a task and its related parameters(owners, deadline, actions) from a chat or email message.
• Determine whether the communication around a task sig-nals the creation, delegation, cancellation, or discharge of acommitment, by extracting selected features from conversa-tions and a classification-based approach. Analyze complexsentence structures to discover cases where a sentence con-tains multiple task definitions or several lifecycle changes ofa commitment such as creation and delegation.
• Develop an automated agent that monitors conversations toidentify the tasks and progression in a commitment, and non-
1
intrusively presents them to case workers who determine whetherto accept its suggestions.
We have experimentally validated our approach on real-worlddatasets of email and chat conversations. Our approach yields sig-nificantly better accuracy than existing work for task and commit-ment identification, and performs well for identifying the delega-tion, cancellation, and discharge of a commitment, which have notbeen studied by others.
The paper is structured as follows. Section 2 discusses the keybackground topics on commitments. Section 3 defines tasks andcommitments in the context of conversations around people-centricprocesses. Section 4 describes our approach to identifying commit-ments from email and chat conversations. Section 5.3 explains ourdataset, experimentation and the evaluation results. Section 6 high-lights the limitations of our approach. Section 7 discusses relatedworks. We conclude and discuss future directions in Section 8.
2. BACKGROUNDSeveral researchers have studied how people converse with each
other to create tasks and commitments and collaborate with eachother. We review speech acts [8], language/action perspective [13],and commitments [10] as key concepts to build the foundation forthe approach presented in this paper. These techniques are foun-dations for providing automated support for conversation-orientedmethods for conducting people-centric processes [4].
2.1 Speech ActsSearle [8, 9] classified illocutionary acts into five classes: com-
missives, directives, representatives, expressives, and declarations.A message is classified as a commissive if the sender of the mes-sage promises to take an action in the future. A message is clas-sified as a directive when the speaker intends the receiver to dosomething. A message is classified as a representative if the sendercommits to the truthfulness of the message. A message is classifiedas an expressive when the sender expresses his or her psychologicalstate. A message is classified as a declaration when the sender ofthe message brings about a change in the status of the referred ob-ject or objects. Searle theory provides the basic idea of identifyingcommissive and directive actions in people’s conversations, how-ever, no work is reported on how commissive and directive actionsprogress in conversation.
2.2 Language/Action PerspectiveWinograd [13] extends speech acts to understand human cooper-
ative activity as conversations. In this model, a message in a con-versation is identified commissive when it is either requested, of-fered, or counter-offered, let’s say from a party (A) to another party(B). The conversation progresses when B accepts the offer from Aand assert A that the conditions are met. Now, if A declares that heor she is satisfied, the conversation reaches a completion state.
2.3 CommitmentsUnlike Winograd’s approach that captures every request as a
commitment, Singh’s model of commitments [10] capture businessrelationships between any two entities. These entities can be eitheremployees within a company or the companies themselves. Specif-ically, commitments denote business meanings underlying the in-teractions between these business entities. In this model, a commit-ment is a conditional business relationship directed from a debtorto a creditor, which can be formalized as
C(DEBTOR, CREDITOR, antecedent, consequent).
The formula shows that the debtor is committed to bringing aboutthe consequent for the creditor provided the antecedent holds. Whena debtor offers a commitment to a creditor, the commitment is cre-ated and becomes active. When the antecedent is brought about, thecommitment is detached and when the consequent holds, the com-mitment is satisfied. If the antecedent holds and the consequenttimes out the commitment is violated. If the antecedent is True, thecommitment is unconditional and the antecedent may be omitted:
C(DEBTOR, CREDITOR,>, consequent).
violated
conditional
discharged
consequent
detached
cancel
nullcreate
expire
������
����������
terminated
cancel
Figure 1: The life cycle of a commitment [12].
Telang and Singh [12] present the commitment life cycle as shownin Figure 1. According to Figure 1, a commitment transits fromone state to another due to the following operations: create, de-tach (antecedent holds), discharge (consequent holds), cancel, anddelegate.
• create(c) forms a commitment. A commitment c gets createdwhen a debtor voluntarily offers to do a task or when he isassigned to do a task.
• detach(c) detaches a commitment. A commitment gets de-tached if a condition or an antecedent present for a commit-ment holds true.
• discharge(c) completes a commitment when a debtor exe-cutes a committed task.
• cancel(c) terminates the commitment c. A commitment canbe canceled only by its debtor.
• delegate(c, z) replaces z as the c’s debtor. The debtor of thecommitment c is replaced by z when the debtor delegates hiscommitment.
3. TASKS AND COMMITMENTS IN TEXTIn this section, we discuss tasks and commitments in connection
with textual conversations. To explain the concepts, we use exam-ples from the Enron email corpus and a chat corpus from an HPIT incident management application. The email corpus containsover 250,000 emails sent or received by over 87,000 people. Itconsists of emails exchanged by Enron employees in the time lead-ing up to the Enron bankruptcy that were revealed as part of theFederal investigation of Enron. The chat corpus contains conversa-tions related to HP IT incident management systems. The corpus iscollected from the logs of chat conversations among workers in acustomer facing incident management system for big IT accounts.We extracted logs from the months of May and June 2012 and an-alyzed them for our work.
2
3.1 TaskA task is a business activity that is either pre-defined (part of a
best practice process) or created on-the-fly by people in a conver-sation [4, 11]. We represent a task as T and define it as
T (SUBJECT, OBJECT, action)
In this definition, a subject is a business entity that performs theaction. An object is a business entity for whom an action is beingperformed by a subject. An action is a business activity performedby a subject. An action can be a disjunction or a conjunction ofsubactions. And finally, a deadline represents the time-out condi-tion for an action to be performed. If the time-out occurs, then weconsider the action to be expired. Consider an example from theEnron corpus where Kim sends an email to John: I will pick it uptonight. Here, the subject is Kim and the object is John. The actionis pick it up and the deadline is tonight. We can formally representthis task as T1 = T(KIM, JOHN, pick it up).
3.2 Create CommitmentWe adopt the Singh’s model of commitments [10], reviewed in
Section 2.3, as an action that is performed by a debtor for a creditor.In our context, we map a task to a commitment as follows:
Definition 1. (Mapping a task with a commitment) If a taskT is a commitment C, then subject ∈ T is debtor ∈ C, object ∈ Tis creditor ∈ C, action ∈ T is consequent ∈ C, and deadline ∈ T isconsequent_timeout ∈ C.
Consider an example from the Enron dataset where Earl Chanleysends the following email to Jo Williams We will expedite materi-als and installation in an attempt to meet the target date. To extractthe structure of a commitment from the sentence, first, we iden-tify a task from it and formally write it as T1 = T(EARL CHAN-LEY, JO WILLIAMS, expedite materials ∧ expedite installation).Second, we map T1 to a commitment C1 = C(EARL CHANLEY,JO WILLIAMS, >, expedite materials ∧ expedite installation) asthe email indicates a commitment from Earl (as subject) to Jo (asobject). By the classification of Searle’s illocutionary acts [9] theabove examples represents an offer from Earl to Jo. Therefore wedefine a commissive creates as
Definition 2. (Commissive create) The creation of a commit-ment C is commissive when the debtor voluntarily offers to performthe consequent for the creditor
Similar to a commissive create, a commitment can be a directive.We define a directive create as:
Definition 3. (Directive create) The creation of a commitmentC is directive when the creditor delegates the consequent to thedebtor
Consider an example of a directive create from the Enron corpuswhere Steven Schleimer sends the the following email to Kim Wat-son Please review and send along to your attorneys as soon as pos-sible. We formally write the email as C(KIM WATSON, STEVENSCHLEIMER, >, review ∧ send).
In our machine learning approach, we label sentences as eitherccreate or dcreate based on whether the sentence indicates a com-missive or a directive respectively.
3.3 Discharge CommitmentA commitment is discharged when the debtor executes the con-
sequent thereby making it true. An example from the Enron dataset
is one where Kim sends an email to Dorothy with the followingmessage, I will also check with Alliance Travel Agency (formerlyTravel Agency in the Park) to see what they may be able to dofor us. The message indicates the creation of a commitment fromKim to Dorothy and can be represented as C1 = C(KIM WATSON,DOROTHY MCCOPPIN, >, check with Travel Agency). In fol-lowing, Kim sends another email to Dorothy with the followingmessage, I checked with our Travel Agency and they cannot securecheaper tickets. The task T2 from this email T2 = T(KIM, MCCOP-PIN, checked with Travel Agency) discharges C1 as Kim checkedwith the travel agency.
3.4 Delegate CommitmentA commitment is delegated when its debtor outsources it to a
new debtor. There can be two cases of delegation. In the first case,the new debtor does not know about the old creditor. We define thedelegation as
Definition 4. (Unknown creditor delegation) When a debtordelegates his or her commitment C1 to a debtor’, then debtor ∈ C1is the creditor’ ∈ C2 and both the commitments C1 and C2 can bewritten as
C1 = C(debtor, creditor,>, consequent)
C2 = C(debtor′, debtor,>, consequent)
Consider an example from the Enron dataset where Gregory sendsan email to Glen with the following message, Please take a fewmoments to review the same and let me know your thought. Inthis email, Gregory delegates a commitment to Glen for review-ing something. We formally represent the commitment as C1 =C(GLEN HASS, GREGORY KLATT, >, review statements). In thefollow up, Glen sends another email to Steven with the message,This appears to be to be OK and we should be able to sign onhowever please review the statement and let me know if you seea problem with our support of the PHC statement. In this email,Glen delegates his commitment to Steven thereby creating anothercommitment from Steven to him. This commitment can be for-malized as C2 = C〈STEVEN HARRIS, GLEN HASS, >, reviewstatements〉. As, we can see this commitment has a new debtor(debtor’) as Steven while the creditor’ Glen is the debtor of theprevious commitment.
Now, in the second case of the delegation, the new debtor is com-mitted to the old creditor. We define the delegation as
Definition 5. (Known creditor delegation) When a debtor del-egates his or her commitment C1 to a debtor’, then creditor ∈ C1is the creditor’ ∈ C2 and both the commitments C1 and C2 can bewritten as
C1 = C(debtor, creditor,>, consequent)
C2 = C(debtor′, creditor,>, consequent)
Consider an example from the Enron dataset where Robert sendsan email to Drew with the following message, Please let me knowif your business unit has any problem with this course of action.In this email, Drew creates a commitment with Robert for lettinghim know something. The commitment is formalized as C(DREW,ROBERT, >, inform about business unit having any problem). Infollowing up, Drew sends another email to Kim; Please review thismessage and advise Robert Williams of any relevant information.In this email, Drew delegates his commitment to Kim thereby cre-ating another commitment from Kim to Robert. This commitment
3
can be formalized as C(KIM, ROBERT, >, provide information).As we can see, this commitment has a new debtor as Kim while thecreditor is still the same i.e., Robert.
3.5 Cancel CommitmentA commitment is canceled when the debtor of the commitment
terminates the commitment. An example from the Enron datasetis Diane sends an email to Kimberley, Can you please provide methe details about the amounts from prior months by this Friday?.In this email, Diane delegates a commitment to Kimberley that canbe represented as C1 = C(KIMBERLEY, DIANE, >, provide theinformation). Later, Kimberley sends an email back to Diane withthe message, I cannot not give you the details by Friday, therebycanceling her existing commitment toward Diane.
4. IDENTIFYING TASKS AND COMMITMENTS
Our process for the identification of commitments from emailsand chat conversations is depicted in Figure 2. We first extractsentences from the text of conversations. We then identify thetasks and then candidate commitments or changes in the lifecy-cle of commitments using a combined NLP-based and machinelearning approach. Using NLP and a set of heuristic-based rulesapplied on features extracted from the text of conversations, weidentify certain tasks and commitments. However, the applicationof NLP-based rules is limited to identifying pre-determined classesand patterns of tasks and commitments. We augment our approachwith a supervised machine learning approach for the identificationof commitments and their lifecycle. This helps in the identificationof commitments for which their various expressions and forms inthe natural language may not be captured in patterns and rules.
�������
������� ������
�������������
�������
���������
�������
�������
� ������������
���������
� �����
�������
���������
� ������
��������
���������
� �����������
���������
Figure 2: Process followed to identify commitments.
For identification of a task T from a sentence, we check whetherthe sentence has a subject, an object and an action verb. Althoughit sounds simple, the identification of tasks in a sentence is non-trivial. For example, changing from active to passive voice resultsin shifting positions of words that indicate subject, object, or theaction. To deal with this issue, we chose to parse the sentences us-ing the typed dependency method [1] which outputs the relationsbetween individual words in a sentence, and is largely independentof the exact sentence structure. A relation between any two wordsis defined as
Definition 6. (Relation) A relation is a triplet of the name of therelation, governor, and dependent where the governor and depen-dent are words from a sentence and it can be represented as
relation(governor, dependent)
Marneffe et al. [1] represent such relationships in a hierarchicalmanner with the most generic relation as the root. For example, arelationship can start with an arg (argument) and it can branch intothe subj (subject) and the comp (complement) relationship. Con-sider a sentence, Diwalkar reports the team will do hardware in-vestigation on EMEGHP151. In the sentence, the triplets are root(root, reports), nsubj (do, team), nsubj (reports, Diwalkar), aux (do,will), prep_on (do, EMEGHSP151) det (team, the), dobj(do, inves-tigation), and nn(investigation, hardware). (time, this). Figure 3shows these triplets in a graph In the graph, we can see that thereis an nsubj relation in which team is the dependent and do is thegovernor. This relation suggests there is a subject in the sentenceis team and his action is do. Even a sentence can have multiple sub-jects. As we can see, there is another nsubj relationship in whichDiwalkar is the subject and his action is reports.
����
�������
���� ���
����
� ����������
����������
��������
���
����
�����
��
��
����
�������
���
�����
��
Figure 3: Typed dependencies derived from a sentence “Di-walkar reports the team will do hardware investigation onEMEGHP151” chosen from the chat corpus.
4.1 Identifying Conversations and SentencesWe preprocess the Enron and the HP IT incident management
datasets to make them suitable for parsing and extracting features.Since the email and chat dataset types are differently structured,we follow different steps to preprocess them from both these types.For email, we separate information such as sender, receiver, date,and subject. Then we prepare conversation threads by collecting allthe emails either replied or forwarded with the same subject name.Next, we split each email into its constituent sentences and parseeach of these sentences to extract features. Unlike, emails we donot prepare conversation threads for chat conversations as they arealready listed chronologically.
4.2 Extracting FeaturesWe perform the following steps to extract features from each sen-
tence in emails and chat messages.
• Co-reference resolution relates a name with a personal pro-noun. For example, in a pair of sentences Please add JimCurry to your list. He should be part of the due diligenceteam, the co-reference resolution helps to relate Jim Curry(name) with He (personal pronoun). This is important be-cause several conversations start with you or he or she or theyand it is necessary to resolve these pronouns so that we canfind the debtor for a commitment.
• Named entity resolution (NER), identifies a noun whetherit’s a PERSON or an ORGANIZATION. Upon identifying a
4
commitment we check whether the debtor and the creditor ofthe commitment is a valid debtor by checking if it is a PER-SON or an ORGANIZATION from the resolved name entities.
• Part-of-speech tags extraction We extract Part-Of-Speech(POS) tags to tag a word with a part of speech. The POStags help to identify the type of personal pronoun for a sub-ject and the state of the verb associated with the subject soas to identify the debtor of a commitment and the state ofa commitment, respectively. The present tense of the verbindicates that the commitment is either a commissive or a di-rective whereas the past tense indicates the commitment isdischarged. Examples of POS tags used in our work includeNN (for nouns), NNP (pronouns), PRP (personal pronouns),VB (present tense verbs), VBD (past tense verb), VBZ (3rdperson singular verb), VBN (past participle) and VBP (non-3rd person singular).
• Typed dependencies extraction As discussed in Section 4,a typed dependency relates words in a sentence and gives aclue as to its logical structure.
Before explaining the algorithms for identifying tasks and commit-ments, we briefly mention the features in the feature vector that areused to train classifiers for email and chat datasets. The featuresare based on properties that help identify a sentence as creating,delegating, discharging, or cancel. The features are:
A modal verb signals the creation of a commitment. For ex-ample, the sentence He will handle the issuance of the LC. has amodal verb will that indicates the creation of a commitment. Anaction verb indicates whether a commitment is present in a sen-tence. For example, I can imagine that your family reunions arejust a hoot! is not a commitment because the verb imagine is notan action verb. The present tense signals the creation, delegation,or cancellation of a commitment. In the following example: Sheand I will get together on the results of these meetings. that sug-gests a create commitment, handle and get are in the present tense.And, the past tense signals the discharge of a commitment. In thefollowing example: I have reviewed the list you sent me regard-ing items you would like to see in the Data Room. that suggests adischarge commitment has has the action verb reviewed in the pasttense.
The debtor of a commitment is the task performer. The creditorof a commitment is the one the debtor commits to. A deadline in-dicates a commitment creation or delegation. For example, in thesentence It will be posted today and the policy will go into effect forFriday’s gas day. indicates today and Friday as the deadlines. Theprior creation of a commitment is a prerequisite for discharge, del-egation, and cancellation if the create commitment already exists.And a delegation signal is identified when a debtor occurring as acreditor indicates that the debtor delegates an existing commitmentto a new debtor. The negative verb indicates the presence of a can-celed commitment. For example, in the sentence I cannot give youthe amount details not give indicates a negative verb.
The type of the personal pronoun in the subject indicates a com-mitment being created, canceled, or discharge (first, second, orthird person) or delegation (second, third). The bigram of a modalverb and a second person PRP indicates a directive creation. Forexample, in the sentence Can you help me with the following out-standing items relating to the Info Memo, the bigram Can you in-dicates directive creation. The bigram of a first person PRP and amodal verb indicates a commissive. For example, in the sentenceWe will expedite materials and installation, the bigram We will in-dicates commissive creation.
The bigram of ‘please’ and an action verb indicates a directive.For example, in the sentence Please review and send along to yourattorney as soon as possible, the bigram please review indicates adirective commitment creation. And, a question mark in a sentenceindicates a directive commitment creation.
4.3 Identifying TasksTo identify a task from a sentence, we first extract features dis-
cussed in Section 4.2. The features are input to Algorithm 1 toidentify subject, object and the action that the subject performs fora candidate task. Our algorithm slightly differs between emails andchats especially in extracting the subject and the object as in caseof chats it is more difficult to find them than in emails. In case ofemails, it is easier to find them because the metadata provides in-formation such as sender’s and receiver’s names. In case of chatmessages, it is difficult because we only get the chat initiator nameas the metadata and the co-reference resolution does not work wellfor resolving pronouns such as you in multiparty conversations.
����
�����
�
����
���������
��
�
� ����
��� �
�� �
������
����� �����
�� ��
������ ������ ����� �������
������� �����
Figure 4: Steps to identify task from a email sentence “I willalso check with Alliance Travel Agency”.
Once we parse a sentence, we get a typed dependency array asin the example shown in Figure 4. We define the typed dependencyarray as
Definition 7. (Typed dependency array) The typed dependencyarray consists of relations derived from a sentence using the typeddependency method. The typed dependency array is represented interms of relations as, typed dependency array = {relation1, relation2,relation3, . . . }.
In the above definition, a relation can be an nsubj, aux, advmod,and so on. In a typed dependency array, we look for the nsubjectrelation and check if the dependent is a valid subject and the gov-ernor is a valid verb. We define subject validity as
Definition 8. (Valid subject) A dependent is considered a validsubject if the POS tag associated with the dependent is NNP orNER resolves the subject as a PERSON or an ORGANIZATION.
We define verb validity as
Definition 9. (Valid verb) A governor is considered valid if thePOS tag associated with the governor is either VB, VBD, VBP,VBZ, or VBN and if the governor is an action verb.
5
We define an action verb as
Definition 10. (Action verb) The verb that expresses an actionor doing something.
By checking for an action verb, we eliminate words that are verbsbut do not express actions. If both the governor and the dependentare valid, we store the dependent as the subject and the governoras the action for the subject. We extract the action details using theaction verb by finding its dependencies in the array of triplets bylooking for nouns or verbs associated with the action verb. Algo-rithm 1 shows the steps to extract subject, object, and action froma sentence in emails. In this algorithm if the POS tags associatedto the dependent is PRP first person (I, we) then the subject is thesender of the email. In case of chats, the subject is the chat initia-tor. If the POS tag associated with the dependent is PRP secondperson (you), then the receiver is the subject of the task. In caseof chats, it is difficult to resolve you in a multiparty conversations.If the POS tag associated with the dependent is PRP third person,then the approach for both emails and chats is the same. We con-sider the dependent as the subject. If the POS tags associated withthe dependent is personal PRP (he, she, they) then we resolve themwith co-reference resolution.
Algorithm 1: extract task(sentence)
1 typed dependency array← get typed dependencies(sentence);2 foreach nsubj ∈ typed dependency array do3 if get POS(dependent ∈ nsubj) is PRP first person then4 subject← sender of the email;5 object← receiver of the email;6 else7 if get POS(dependent ∈ nsubj) is PRP second person
then8 subject← receiver of the email;9 object← sender of the email;
10 else11 if get POS(dependent ∈ nsubj) is PRP third person
then12 subject← dependent;13 object← extractObject(sentence);14 if valid subject(dependent) ∧ valid verb(governor) then15 action← extract action details(governor);
Figure 4 represents the typed dependency array for the sentenceI will also check with Alliance Travel Agency. We extract the nsubjrelationship i.e., nsubj (check, I). The subject is I and the actionverb is check. Once we extracted the subject and the action, weextract nouns or verbs related to action such as Agency, Alliance.
4.4 Identifying CreateOnce we have extracted a task from a sentence, we check whether
the task indicates the creation of a commitment. To identify suchtasks, we check whether the action in a task has a relationship witha modal verb and the word please. A modal verb can be defined as
Definition 11. (Modal verb) A modal verb is a word that ex-presses possibility, likelihood, or obligation. Words that indicatemodality are will, shall, can, could, would, should, may, might andmust.
The word please indicates a request or a delegation of a task. Ifthe verb has a relation with these words (modal or please), then welabel the task as commitment creation and store the subject as the
debtor, the object as the creditor, and the action as the consequentrespectively of the commitment. Also, as discussed in Section 3.2,commitments can be created in two ways: the commissives or di-rectives. We mark a sentence as a commissive when an action verbis following a modal verb. We mark a sentence as a directive whena second person personal pronoun such as you is following a modalverb or an action verb following the word please. Algorithm 2 sum-marizes the method for identifying commitments from sentences.
������ ����� ������ ����
�������
�
������ ����������
��������� ������
Figure 5: Steps to identify a commitment from a email sentence“I will also check with Alliance Travel Agency”.
In Figure 5, the action verb check is in the present tense (VB)and has a relationship with a modal verb will. Therefore, the taskis considered as creating a commitment. Since here the action verbfollows the modal verb, we consider it a commissive. In Algo-rithm 2, we check if the action is in present tense (VB). Then, wecheck if there is a relation that associates the action with a modalverb or please.
Algorithm 2: identify create commitment(sentence)
1 T1← extract task(sentence);2 commitment←⊥;3 if getPOS(action verb ∈ T1) is VB then4 foreach relation ∈ typed dependency array do5 if (dependent ∈ relation is action) ∧ (governor ∈
relation is (modal verb ∨ please)) then6 commitment←>;7 else8 if (governor ∈ relation is action) ∧ (dependent ∈
relation is (modal verb ∨ please)) then9 commitment←>;
4.5 Identifying DelegateWe discussed in Section 3.4 that there are two kinds of delega-
tion. In one, the old creditor is unknown to the new debtor, whereas,in another, the old creditor is known to the new debtor. ConsiderFigure 6(a) that shows an example where the new debtor Stevendoes not know about the old creditor Gregory. Similarly, considerFigure 6(b) where the new debtor Kim is committed to the old cred-itor Robert.
To identify delegation in sentences, we check a commitment C2created after another commitment C1. To find the unknown credi-tor delegation, we check if the debtor in C1 is the creditor in C2 andif the consequents in C1 and C2 are the same. To find the knowncreditor delegation, we check if the creditor in C1 is the creditor inC2 and if the consequents in C1 and C2 are the same. To matchthe consequents in the commitments, we check whether the actionverbs in both commitments are the same or related as synonyms,hypernyms, or hyponyms. If they are the same or related, we checkwhether nouns in both the commitments are the same or related us-ing the co-reference resolution. Finding delegation in case of emailis easier than chats as we know in advance the debtor and creditorof the commitment based on the sender and receiver information.
6
�������� ������ ��
��������� �����������
�����
������
����������
�����
�����
�����
��������
������ ������
������ �� ��������
������!�� ������ ��
������������ �����������
�����
"�����������
����������
!��#��$
!��
�����
��������
������ ������
�� ��������
"�����
��������������� �����
������������� �����
Figure 6: Steps to identify delegate.
Unlike emails, in chats, finding delegation is difficult as sometimesit is extremely hard to find the creditor of a commitment.
Algorithm 3: identify delegate commitments(sentence)
1 C2← identify create commitments(sentence);2 delegation←⊥;3 unknown creditor←⊥;4 known creditor←⊥;5 foreach C1 ∈ commitment array do6 if debtor ∈ C1 is creditor’ ∈ C2 then7 unknown creditor←>;8 else9 if debtor ∈ C1 is creditor’ ∈ C2 then
10 known creditor←>;11 if action verb ∈ C1 is action verb ∈ C2 then12 if nouns ∈ C1 is nouns ∈ C2 then13 if unknown creditor ∨ known creditor then14 delegation←>;
4.6 Identifying DischargeIf an identified task is in the past tense, it may signals a dis-
charge commitment, and we need to compare it with existing com-mitments. For greater clarity, let us consider an example of a com-mitment C1 and a task T2.
�������� ���� ���� �� �� ��������
���������� ���� �� ������
������
������������
��������
�������
�������
������
��������
Figure 7: Steps to identify discharge.
To check if T2 discharges C1, we compare the subject and theobject in T2 with the debtor and the creditor in C1 respectively.If they are the same, we compare their action verbs. We checkwhether the action verb in T2 is in the past tense (VBD). Then wecompare the action verb in T2 by converting it into its base formand trying to match with the action verb in C1. If both the verbs arethe same or related as either synonyms, hypernyms, or hyponyms,we compare the nouns in both the tasks. If they are the same orrelated, we mark T2 as discharging C1. Figure 7 clearly showsthat the debtor and the creditor in both C1 and T2 are the same.The main action verb in T2 is checked and it is in the past tense.
Therefore, we convert it into its base form check and compare itwith the action verb in C1. Note that the base form of a verb isthe simplest form in which it appears in a dictionary without anyending. Since they are the same, we compare the nouns in C1 andT2. We see that the nouns are related. Therefore, we mark T2 asdischarging C1. Algorithm 4 describes these steps.
Algorithm 4: identify discharge commitments(sentence)
1 T2← extract task(sentence);2 discharge←⊥;3 foreach C1 ∈ commitment array do4 if (subject ∈ T2 is debtor ∈ C1) ∧ (object ∈ T2 is
creditor ∈ C1) then5 if (getPOS(action verb ∈ T2) is VBD) ∧
(getBaseVerb(action verb ∈ T2) is action verb ∈ C1)then
6 if nouns ∈ T2 is nouns ∈ C1 then7 discharge←>;
4.7 Identifying CancelFor identifying a canceled commitment, we compare a task with
the commitments that already exist and check whether there is arelation in the type dependency array where the action verb is as-sociated with a negative word such as not.
��������� ��� � �� ��������
������ ��� ���
��� ��
�� ���������
���������
����
����
�����
����� ��
�� �� ���
Figure 8: Steps to identify cancel.
Figure 8 describes an example of a commitment C1 and a taskT2. Once we find that the debtor and the creditor in C1 and T2 arethe same, we compare their main action verbs and nouns respec-tively. If they are the same, we check whether the action verb isa negative verb. Note that a verb considered as negative it has arelation with a negative word such as not. Algorithm 5 describesthe above steps.
Algorithm 5: identify cancel commitments(sentence)
1 T2← extract task(sentence);2 cancel←⊥;3 foreach C1 ∈ Commitment Array do4 if (subject ∈ T2 is debtor ∈ C1) ∧ (object ∈ T2 is
creditor ∈ C2) then5 if getBaseVerb(action verb ∈ T2) is action verb ∈ C2
then6 if nouns ∈ T2 is nouns ∈ C1 then7 if negative verb(action verb ∈ T2) then8 cancel←>;
5. EVALUATION AND PROTOTYPETo validate our approach, we use two methods: (1) automatic
labeling of data (sentences) using Algorithms 2, 3, 4, and 5, and (2)manually labeling a subset of data and using them for training andtesting our approach. We applied three machine learning classifiersand evaluated them.
7
5.1 DataFor evaluation and validation purpose, from the Enron email cor-
pus, we selected 4161 email sentences that were exchanged be-tween Kimberly Watson, an employee of Enron, and more than 50people which includes her co-workers at Enron, clients, friends,and family members. The emails were collected and prepared bycombining the data from text files provided by CMU [3] as well asfrom a database dump [2] provided by UC Berkeley. We combinethe data from both so as to reduce the missing information. Forthe chat data, we selected 271 conversations from HP IT incidentmanagement logs comprising of 7154 sentences.
5.2 Labeling DataOnce the features are extracted for each sentence from the dataset
as give in Section 4.2, we had two annotators to label the sentences.We cross checked the results for any possible conflict resolution.We resolved conflicts by allowing two annotators to collaborate anddiscuss their labels for sentences. Next, we ran classifiers the threeclassifiers of Naïve Bayesian (NB), Logistic Regression (LR), andSupport Vector Machines (SVM) and used 10-fold cross validationto produce results.
We annotated 4161 email and 7154 chat sentences. The twoindependent annotators achieved overall an interrater agreement(kappa score) of 0.83 for both email and chat. Table 3 shows thedistribution of the email and chat sentences as annotated.
5.3 ResultsTable 1 and Table 2 represent results for email and chat data
respectively, using NB, LR, and SVM classifiers and ten-fold crossvalidation. In both the tables, we represent C-create as commissivecreate, D-create as directive create, P as precision, R as recall andF as F-measure.
Table 3: Distribution in email sentences.Classes Email Chat
Commissive create 342 532Directive create 162 214Discharge 38 250Cancel 7 16Delegate 12 12None 3540 6130
First, we considered one of the simplest probabilistic text classi-fication approach, Naïve Bayes (NB). The NB classifier approachassumes that attributes in consideration are independent of eachother. Using NB, for emails, our results show high precision forcommissive creation (84%) and directive creation (81%) while lowprecision for delegation (32%), discharge (21%), and cancellation(0%). In case of chats, we obtain slightly lower precision for com-missive creation (73%) than email, however, we obtain higher pre-cision for directive (85%) and discharge (60%). The precision re-mains same for cancel (0%). For delegate, the precision (0%) waslow compared to precision for email.
Second, we used the Logistic Regression (LR) classifier. Us-ing LR, in emails, we obtain significantly high precision valuesfor commissive (90%), directive (92%) compared to NB while lowprecision values for delegate (64%), discharge (27%), and cancel(0%). In case of chats, LR performs better than NB for commissivecreation and cancellation with precision of 80% and 22% respec-tively. However, results for other classes are lower, i.e., 74% for di-rective creation, 64% for discharge, and 0% for delegate. Overall,
the results for chat using LR are lower compared to emails exceptdischarge and cancel.
Third, we used the Support Vector Machine (SVM) classifier.Using SVM, in emails, we obtain significantly higher precision,compared to NB and LR, for commissive creation (87%), direc-tive creation (94%), discharge (100%), and delegation (86%) whilesame precision for cancellation (0%). For chats, using SVM, weobtain lower precision than emails for commissive creation (79%),directive creation (73%), and discharge (63%). However, the f-measure and recall value for discharge is higher in chats than emails.Again, this is due to the higher percentage distribution of dischargein emails and chats.
Overall we obtain high precision, recall and f-measure for a com-missive creation and a directive creation for both emails and chatsusing NB, LR, and SVM. The results for both the classes are highbecause both the classes are independent of each other and the dis-tribution of these classes are high in both the datasets. The resultsfor other classes are low because other classes depends on the priorexistence of create commitment classes and it is difficult to findthis specific feature automatically. Compared to discharge and del-egate the results for delegate is higher in case of emails becausewe can easily find out the debtor and the creditor of a commitmentbased on the sender’s and receiver’s information. The results fordischarge in email is low because the percentage of distributionof discharge is extremely low as lot of tasks indicating dischargeare executed, however, are never mentioned in emails. In case ofchat, the precision for discharge is higher because the distributionof discharge is high as people in chat conversations immediatelyreport their progress. However, the overall percentage is low forboth emails and chats because it is difficult to compare consequentby matching verbs and nouns. For emails, we find a high precisionvalue using SVM with low recall and f-measure. We attribute thehigh precision of SVM to some of the sentences in emails that wereidentified accurately as discharge by our algorithm. For cancel, wegot 0% precision in emails and 22% in chats, This is because, as wesaid earlier, it is difficult to find out the prior existence of a commit-ment as well find out the negative words associated with the actionverb.
We evaluated our training model on independent test datasets.Our test datasets contains 1326 email and 2299 chat sentences. Foremails we used SVM and for chats we used LR.
Table 4: Evaluation on test datasets using SVM for emails andLR for chats respectively.
Email Chat
Classifier P R F P R F
C-create 0.97 0.84 0.90 0.90 0.89 0.89D-create 0.94 0.78 0.86 0.83 0.51 0.63Discharge 0.00 0.00 0.00 0.66 0.71 0.69Cancel 0.00 0.00 0.00 0.00 0.00 0.00Delegate 1.00 0.33 0.98 0.00 0.00 0.00None 0.96 0.99 0.98 0.96 0.98 0.94
5.4 Prototype ToolFigure 9 represents the architecture of our interactive tool for
commitments identification and tracking. The tool offers an agentthat can be pluggined into online chat messengers. The agent insideidentifies discovered commitments in a panel for verification andconfirmation to conversation participants. When a person sends achat message, it parses the sentence using our parser component
8
Table 1: Results for emails.C-create D-create Discharge Cancel Delegate None
Classifier P R F P R F P R F P R F P R F P R F
NB 0.84 0.94 0.89 0.81 0.93 0.87 0.21 0.41 0.27 0.00 0.00 0.00 0.32 0.39 0.35 0.99 0.95 0.97
LR 0.90 0.95 0.95 0.92 0.93 0.94 0.27 0.05 0.08 0.00 0.00 0.00 0.64 0.34 0.48 0.98 0.98 0.98
SVM 0.87 0.97 0.92 0.94 0.97 0.95 1.00 0.02 0.04 0.00 0.00 0.00 0.86 0.33 0.48 0.98 0.98 0.98
Table 2: Results for chats.C-create D-create Discharge Cancel Delegate None
Classifier P R F P R F P R F P R F P R F P R F
NB 0.73 0.90 0.81 0.85 0.50 0.63 0.60 0.70 0.75 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.97 0.80
LR 0.80 0.85 0.83 0.74 0.51 0.60 0.64 0.70 0.67 0.22 0.13 0.16 0.00 0.00 0.00 0.97 0.98 0.97
SVM 0.79 0.85 0.82 0.73 0.53 0.61 0.63 0.71 0.67 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.97 0.97
and extract potential task details and features. Using our predictorcomponent and a trained model we identify the classes based onthe extracted features. The parser and predictor components areJava-based. We use the Stanford parser and Weka libraries to parseand train datasets respectively. We predict classes using the SVMclassifier. Finally, the display component produces the summaryof all tasks and commitments based on chats. Figure 10 shows ascreenshot of the prototype tool.
��������
����� �������
������
� �� ����
����������
�����
�������������
������
������ �
� �� ����
������
�������
����������������� �����������������������������
����������� ��
Figure 9: The architecture of our commitment identificationand monitoring tool.
6. LIMITATIONSOur work has several limitations. First, to identify a task struc-
ture we depend on the typed dependency feature which does notwork accurately when the sentence is either too long or too short.Identifying multiple tasks in a sentence is again difficult as thetyped dependency feature does not work well for sentences withconjunctions such as and and but. Second, for judging the simi-larity in consequents of two commitments, we rely on comparingwords. Third, we rely on the type of a personal pronoun such asfirst person, second person, and third person and map the pronounseither to a sender or a receiver of an email. Without the presenceof a sender and a receiver we cannot determine the debtor and thecreditor of a commitment accurately. Fourth, in case of chats wecannot find debtor every time and it is difficult to resolve pronounssuch as you. Fifth, if a sentence has a missing subject and an objectthen we cannot determine if a prior commitment exists.
7. RELATED WORKSThe research in this space are just emerging. While there has not
been previous work on commitment lifecycle identification, thereare few works on using NLP and machine learning techniques foridentifying action verbs and tasks from texts.
Qadir and Riloff [6] classify sentences from message board postsas commissives, directives, expressives, and representatives. Forcreating a training model, they extract lexical, syntactic, and se-mantic features for messages. Using these features, they train theirmodel using the SVM classifier. They obtain 45% precision com-missives class and 97% for directives. In the identification of com-missive and directive commitment creation, we consider personalpronoun, modals, and sentence begins with modal/verb. Basedon these and other features and our novel algorithms, we obtaina higher precision for commissive create and directive creation.
Scerri et al. [7] focus on action items in emails and check whetherthese action items fall under the request, suggest, assign, and de-liver classes. For identifying the classes, they use a rule-based clas-sification model. To evaluate their classification model they com-pare the classes identified automatically with the classes identifiedby a group of 12 people. Their evaluation results show the preci-sion value of 56% and recall value of 60% respectively. In con-trast, first we identify classes for each sentence based on a novelNLP-based algorithm. Next, we apply machine learning to identifyclasses based on the identied features from the text, which allowsus to identify classes in case where it is difficult to capture corre-sponding rules. In addition, we monitor and discover the lifecycleof a commitment, once it is created.
Purver et al. [5] focus on multiparty meetings and identify taskowner, task, deadline, and agreement classes from these meetings.For building a training model in the baseline approach, they labelthe data manually and train it using the SVM classifier. Their resultshows a precision of 25%. Apart from the baseline approach theyuse hierarchical annotation techniques where they identify classesusing the NOMOS annotation software and build their model us-ing the Naïve n-gram classifier. The precision values for task de-scription, task owner, deadline, and agreement are 23%, 12%, 19%,and 48% respectively. In contrast, we completely rely on the type-dependency feature to extract task owner, task, and deadline in asentence. Although the type-dependency extracts features some-what accurately, it extracts features incorrectly in many cases. There-fore, we emply a machine learning-based approach to identify thestructure of a task more accurately.
9
Figure 10: A screenshot of our prototype tool for commitment identification and monitoring.
8. CONCLUSIONS AND FUTURE WORKWith the rise in the informal collaboration and web-based com-
munication tools, the problem of organizing work in the collabo-rational setting and in the context of people-centric processes be-comes more vital for the organizations. To the best of our knowl-edge, this is is the first work that presents an approach for the auto-mated identification and monitoring of commitment lifecycle fromthe text of conversations amog people. We introduced an approachfor extracting tasks and commitments from sentences in email andchat conversations. Our NLP-base algorithms leverages typed de-pendency for identifying tasks and its related parameters. Our ma-chine learning approach accurately identifies whether a task indi-cates a commitment and further identify whether a task progressesa commitment or terminates it. Using our machine learning ap-proach we get high precision for commissive and directive creationon both emails and chats. Our approach also provides promisingresults for the delegate, discharge and cancel classes. Finally, weprovide a tool that implements the approach, and can be used forassisting workers in capturing commitments in collaboration tools.
We have several future directions. First, we plan to improve re-sults for delegation, discharge, and cancellation, specifically in chatconversations. Second, we plan to explore and compare other ap-proaches for identifying tasks from sentences. Third, we plan toconsider applying unsupervised machine learning such as HiddenMarkov Models (HMM) to this problem as the supervised machinelearning approach relies on manual labeling of data, which is te-dious to provide. Fourth, another future step is to reconstruct con-versations from both emails and chats and then apply our approach.
9. ACKNOWLEDGMENTSThis research was initiated by HP Labs, Palo Alto, US when
Anup Kalia was a summer intern there and partially supported bythe Army Research Office under the Science of Security Labletgrant to NC State University.
10. REFERENCES[1] M. De Marneffe, B. MacCartney, and C. Manning.
Generating typed dependency parses from phrase structureparses. Proceedings of LREC, 6:449–454, 2006.
[2] A. Fiore and J. Heer. UC Berkeley Enron email analysis.2004.
[3] B. Klimt and Y. Yang. The enron corpus: A new dataset foremail classification research. In Machine Learning: ECML2004, volume 3201 of Lecture Notes in Computer Science,pages 217–226. Springer Berlin / Heidelberg, 2004.
[4] H. R. Motahari-Nezhad, C. Bartolini, S. Graupner,S. Singhal, and S. Spence. It support conversation manager:A conversation-centered approach and tool for managingbest practice it processes. In Proceedings of the 2010 14thIEEE International Enterprise Distributed ObjectComputing Conference, EDOC ’10, pages 247–256,Washington, DC, USA, 2010. IEEE Computer Society.
[5] M. Purver, P. Ehlen, and J. Niekrasz. Detecting action itemsin multi-party meetings: Annotation and initial experiments.In Proceedings of the Third International Conference onMachine Learning for Multimodal Interaction, pages200–211. Springer-Verlag, 2006.
[6] A. Qadir and E. Riloff. Classifying sentences as speech actsin message board posts. In Proceedings of the Conference onEmpirical Methods in Natural Language Processing, pages748–758, Stroudsburg, PA, USA, 2011. Association forComputational Linguistics.
[7] S. Scerri, G. Gossen, B. Davis, and S. Handschuh.Classifying action items for semantic email. In Proceedingsof the 7th Conference on International Language Resourcesand Evaluation (LREC), 2010.
[8] J. R. Searle. Speech Acts. Cambridge University Press,Cambridge, UK, 1969.
[9] J. R. Searle. A taxonomy of illocutionary acts. InK. Gunderson, editor, Language, Mind and Knowledge.University of Minnesota Press, Minneapolis, 1975.
[10] M. P. Singh. An ontology for commitments in multiagentsystems: Toward a unification of normative concepts.Artificial Intelligence and Law, 7(1):97–113, Mar. 1999.
[11] P. R. Telang and M. P. Singh. Enhancing Tropos withcommitments. In A. Borgida, V. K. Chaudhri, P. Giorgini,and E. S. K. Yu, editors, Conceptual Modeling: Foundationsand Applications, volume 5600 of LNCS, pages 417–435.Springer, 2009.
[12] P. R. Telang and M. P. Singh. Specifying and verifyingcross-organizational business models: An agent-oriented
10
approach. IEEE Transactions on Services Computing,5(3):305–318, July 2012.
[13] T. Winograd. A language/action perspective on the design ofcooperative work. In Proceedings of the 1986 ACMConference on Computer-supported Cooperative Work,CSCW ’86, pages 203–220, New York, NY, USA, 1986.ACM.
11
