Upload
interanalytics
View
620
Download
1
Embed Size (px)
DESCRIPTION
Paper presented at the DART 2011 workshop in Palermo. The paper introduces a new type of call center analytics based on interaction mining. It shows how advanced metrics and KPIs for call center quality management can be implemented through interAnalytics NLP technology.
Citation preview
© 2011 interAnalytics 1
Interaction Mining: the new frontier of Call Center Analytics
Vincenzo Pallotta Rodolfo Delmonte Lammert Vrieling
David Walker
© 2011 interAnalytics 2
Outline
• Call Center Analytics• Automatic Argumentative Analysis for
Interaction Mining• Experiments with Call Center Data• Conclusions
3
CALL CENTER ANALYTICS
© 2011 interAnalytics
© 2011 interAnalytics 4
Call Center Analytics
• Call centers data represent a valuable asset for companies, but it is often underexploited for business purposes because:– it is highly dependent on quality of speech recognition
technology– it is mostly based on text-based content analysis.
• Interaction Mining as a viable alternative:– more robust– tailored for the conversational domain– slanted towards pragmatic and discourse analysis
© 2011 interAnalytics 5
Mainstream Call Center Analytics
Does not unveil real
insights about customer
satisfaction
© 2011 interAnalytics 6
Call Center Analytics: metrics and KPIs
• Agent Performance Statistics: – Average Speed of Answer, Average Hold Time, Call Abandonment Rate,
Attained Service Level, and Average Talk Time. – Quantitative measurements that can be obtained directly through ACD
(Automatic Call Distribution), Switch Output and Network Usage Data.• Peripheral Performance Data:
– Cost Per Call, First-Call Resolution Rate, Customer Satisfaction, Account Retention, Staff Turnover, Actual vs. Budgeted Costs, and Employee Loyalty.
– Quantitative, with the exception of Customer Satisfaction that is usually obtained through Customer Surveys.
• Performance Observation: – Call Quality, Accuracy and Efficiency, Adherence to Script,
Communication Etiquette, and Corporate Image Exemplification. – Qualitative metrics based on analysis of recorded calls and session monitoring
by a supervisor.
© 2011 interAnalytics 7
Four objectives
1. Identify Customer Oriented Behaviors, – which are highly correlated to positive customer ratings
(Rafaeli et al. 2007);
2. Identify Root Cause of Problems – by looking at controversial topics and how agents are able to
deal with them;
3. Identify customers who need particular attention – based on history of problematic interactions;
4. Learn best practices in dealing with customers – by identifying agents able to carry cooperative
conversations.
© 2011 interAnalytics 8
ARGUMENTATIVE ANALYSIS FOR INTERACTION MINING
© 2011 interAnalytics 9
Argumentative Structure of ConversationsDISCUSS(issue) <- PROPOSE(alternative)
1702.95 David: so - so my question is should we go ahead and get na- - nine identical head mounted crown mikes ? {qy} 61a
REJECT(alternative)1708.89 John: not before having one come here and have some people try it out . {s^arp^co} 61b.62a
PROVIDE(justification)1714.09 B: because there's no point in doing that if it's not going to be any better . {s} 61b+
ACCEPT(justification)
1712.69 David: okay . {s^bk} 62b
PROPOSE(alternative)
1716.85 John: so why don't we get one of these with the crown with a different headset ? {qw^cs} 63a
ACCEPT(alternative)1721.56 David: yeah . {s^bk} 63b1726.05 Lucy: yeah . {b} 1727.34 John: yeah . {b}
PROVIDE(justification)
1722.4 John: and - and see if that works . {s^cs} 63a+.64a 1723.53 Mark: and see if it's preferable and if it is then we'll get more . {s^cs^2} 64b1725.47 Mark: comfort . {s}
PROVIDE(justification)1714.09 John: because there's no point in doing that if it's not going to be any better . {s} 61b+
Why was David’s proposal on microphones rejected?
© 2011 interAnalytics 10
Automatic Argumentative Analysis
• Based on the GETARUNS system1.• Clauses in Turns are labelled with Primitive Discourse
Relations: – statement, narration, adverse, result, cause, motivation,
explanation, question, hypothesis, elaboration, permission, inception, circumstance, obligation, evaluation, agreement, contrast, evidence, hypoth, setting, prohibition.
• And then Turns are labelled with Argumentative labels:– ACCEPT, REJECT/DISAGREE, PROPOSE/SUGGEST,
EXPLAIN/JUSTIFY, REQUEST EXPLANATION/JUSTIFICATION.
1 Delmonte R., Bistrot A., Pallotta V.,Deep Linguistic Processing with GETARUNS for spoken dialogueUnderstanding. Proceedings LREC 2010 (P31 Dialogue Corpora).
© 2011 interAnalytics 11
Evaluation
Correct Incorrect Total Found Precision
Accept 662 16 678 98%
Reject 64 18 82 78%
Propose 321 74 395 81%
Request 180 1 181 99%
Explain 580 312 892 65%
Total 1826 421 2247 81.26%
Precision: 81.26% Recall: 97.53%
ICSI corpus of meetings (Janin et al., 2003)
Delmonte R., Bistrot A., Pallotta V.,Deep Linguistic Processing with GETARUNS for spoken dialogueUnderstanding. Proceedings LREC 2010 (P31 Dialogue Corpora).
© 2011 interAnalytics 12
EXPERIMENTS WITH CALL CENTER DATA
© 2011 interAnalytics 13
Rationale: implement the four objectives
1. Identify Customer Oriented Behaviors, 2. Identify Root Cause of Problems 3. Identify customers who need particular
attention 4. Learn best practices in dealing with
customers
© 2011 interAnalytics 14
The Data
• Corpus of 213 manually transcribed conversations of a help desk call center in the banking domain.
• Average of 66 turns per conversation.• Average of 1.6 calls per agent. • Collected for a study aimed at identifying
customer oriented behaviors that could favor satisfactory interaction with customers (Rafaeli et al. 2007).
© 2011 interAnalytics 15
Identify Customer Oriented Behaviors
• Based on the work of Rafaeli et al. 2006.• Customer Oriented Behaviors– anticipating customers requests 22,45%– educating the customer 16,91%– offering emotional support 21,57%– offering explanations / justifications 28,57%– personalization of information 10,50%
© 2011 interAnalytics 16
Significant correlation with argumentative labels
© 2011 interAnalytics 17
Identify Root Cause of Problems
• Cooperativeness score – a measure obtained by
averaging the score obtained by mapping argumentative labels of each turn in the conversation into a [-5 +5] scale.
• Sentiment Analysis module.
Argumentative Categories Cooperativeness
Accept explanation 5
Suggest 4
Propose 3
Provide opinion 2
Provide explanation/justification 1
Request explanation/justification 0
Question -1
Raise issue -2
Provide negative opinion -3
Disagree -4
Reject explanation or justification -5
© 2011 interAnalytics 18
Top 20 Controversial Topics with average cooperativeness scores and sentiment
© 2011 interAnalytics 19
Cooperativeness of speakers on top discussed topics
© 2011 interAnalytics 20
Identify problematic customers
© 2011 interAnalytics 21
Select a specific customer
© 2011 interAnalytics 22
Visualize a selected call
© 2011 interAnalytics 23
CONCLUSIONS
© 2011 interAnalytics 24
Conclusions
• New Generation Call Center Analytics requires Interaction Mining– Call Center Qualitative metrics and KPIs can be only
implemented with a full understanding of the customer interaction dynamics
• Argumentation is pervasive in conversations.– In order to recognize argumentative acts, advanced Natural
Language Understanding is necessary.• Future work:– Scalability: need to process millions of call per day!– Multi-language: call centers all over the world.
© 2011 interAnalytics 25
The Team
www.interanalytics.ch…find us at