6/26/2018

1

How Are You Keeping Up with AI? The Creation of Actionable Insights and

Dashboards Around Chatbot Performance

Beth CorsoManager, Customer Insights and

Measurement, American Family Insurance

How are you

keeping up with AI? The Creation of Actionable Insights and Dashboards Around Chatbot Performance

6/26/2018

2

• American Family Insurance

• American Family Insurance and Clarabridge History

• Overview of the Session

• Chatbot and Virtual Assistant Background

• American Family Insurance Customer Bot, Abby

• Utilizing Clarabridge for Abby Performance and Deep Learning Analytics to Drive Prioritization

• LearningsAG

EN

DA

$8.8 Billion in

Premium

2017 American Family Insurance

$10.5 Billion

Active

Policies

Fortune 500

Ranked 311

(moved 92 spots in 22 yrs.)

6/26/2018

3

American Family Insurance

American Family Insurance + ClarabridgeOUR JOURNEY SINCE SUMMER OF 2015

• Started with 2 call centers

• Focused on speech calls

• Voice to text evaluations

• Quality Assurance for

Care Center

• Root cause analysis

• Chatbot monitoring and

intent building

• Multi-Channel Data

Source Injections

• Elimination of call

dispositioning

• Expansion to surveys,

webchats, social data,

digital feedback

BUILD CLOUDIFY & EXPAND LEARN & OPERATIONALIZE

6/26/2018

4

Conversational

CommerceCustomer Chatbot via Webchat

Artificial Intelligence, ChatbotWHAT IS A CHATBOT AND IS YOUR ORGANIZATION UTILIZING AI TODAY?

Software RobotsIncreasingly being

used to automate

business processes

6/26/2018

5

Meet Abby, Our Customer BotBUSINESS GOAL: Create a conversational virtual assistant our customers will benefit

from in their busy daily lives by answering their insurance questions and provide

support to their every day needs.

Wait time optimization

for calls and representative webchats

Continuous customer service

to allow customer toget support they need

TOOLS

• Customer Empathy

• Dialog Testing

• Usability Testing

• Customer Insights

• Chatbot to answer the

customer’s questions

correctly without a wait

time

• Reduce wait times for chat

• 24/7 customer care via chat

• Provide customers consistent

answers

SOLUTION SUCCESS METRICS

CUSTOMER PAIN POINTS TO SOLVE W/ AN AI CUSTOMER CHATBOT

Answer consistency to our

customers regardless of the channel contacted for support

Who is Abby?

LIFE MOTTO: Where there’s a will, there’s a way.

BRAND PERSONA

LIFE MISSION: Helping other people realize and achieve their vision of success.

DEMOGRAPHICS:

• Age: 38 (old enough to have

enough life experience, but young

enough to be relevant)

• Mom of 3

• Life Coach and Licensed Insurance

Agent – protection expert!

BACKSTORY:

• Humble upbringing in the

Midwest. Raised by

grounded and diligent

small farmers who value

and understand the

importance of hard work.

ABBY…

• Inspirationally guides

• Transparently

navigates

• Humbly coaches

• Proactively plans

and manages

6/26/2018

6

How Does Abby Work?

Customer

Utterance NLP

Utterance to Intent

Intent to Content

Abby’s Response

Abby

Intent: Category of what the customer is asking

Add/Remove Vehicle

Utterance: How the customer asks the question

“I just got a new car and would

like to remove my old one and add my new one.”

American Family

Insurance + ClarabridgePerformance and Deep Learning Analytics

6/26/2018

7

Clarabridge and Abby Performance3 STEP IN CREATING AN AI CHATBOT

1 -

Go

als •Containment: How many

chats start with the bot and end with the bot

•Building Knowledge: How are we increasing our intents and utterances through customer interactions

•Correct Intent Mapping: How are the utterance to intent performing

•Conversational Sentiment: How is the bot sentiment compared to the customer sentiment

2 -

Pe

rfo

rma

nc

e •Increasing containment as we increase our knowledge base of utterances and intents, 22%

•Started with 76 FAQ intents and increased to 180+ which consists of 30,000 utterances

•Correct utterance to intent mapping has increased since launch and is at 80%

•Conversational sentiment with the bot and customer is neutral 3

-D

ee

p L

ea

rnin

g •Verification success vs failure

•Intent mapping to categorization through Clarabridge modeling

•Total webchat conversation vs Abby conversations

•Intent sequencing: Where does speak to rep fall in the conversation and what intent came prior to the transfer

•Webchat profiling: Who is our customer that is utilizing webchat compared to phone calls

Clarabridge and Abby PerformanceWHAT ARE OUR MEASUREMENT GOALS AND TRACKING?

• Total Webchats vs. Chatbot• Webchats vs. Chatbot Categorization

Metric Overview

Abby Contained Chats

• Total Contained and Penetration to Total• Chatbot Intents Comparison to Model Categories

Abby to Rep Transfers

• Total and % of Transferred Chats + % of Initials Trans• Chatbot Intents Comparison to Model Categories

Abby Verification• Total and % of Prompted and Successful• % to Success, Failure, Opt Out, Abandon, and Intent Vol

6/26/2018

8

Clarabridge and ABBY INTENT SEQUENCINGSEQUENCING DASHBOARD, DEEPER DIVE INTO THE BOT CONVERSATION

• Total Webchats vs. Chatbot• Contained Chats and Transfer Chats

• Care Center Drillable Intents to Lowest Level

Metric Overview

Webchat Intent Sequencing

• Intent Volume• Speak to Representative sequence evaluation

• 1-4 Intent Sequence Volume

Intent Category Deep Dive

• By category: Billing and Payments, Claims, Policy, Account Management, etc

6/26/2018

9

Abby Customer BotWHAT HAS AMERICAN FAMILY INSURANCE LEARNED AND WHAT IS NEXT 1-2-3 …

1 -

Lea

rnin

gs

•Monitor Daily: if we hadn’t

monitored Abby daily we

wouldn’t have found key insights

•Greeting: does your customer

read it?

•Content: are you explaining

things correctly?

•Verification: does your customer

follow directions?

•Customers need guidance

•You don’t need a big team to

succeed

2 -

Exp

an

sio

n •2018: research, learning,

development

•Understand what metrics

drive a customer

•What platforms best enables

the business to scale and

succeed?

•Expansion

•Data Access

•Channels

•Voice – Alexa, Google, etc

3 -

Re

co

mm

en

da

tio

ns •Monitor Daily

•Empathy Work – do what the

customer tells you they want

•Establish Goals

•Customers need parameters

to work in

•Test and Learn

•Don’t be afraid to fail

•Have fun

6/26/2018

10

Actionable insights from every

customer interaction, in one platform.

www.clarabridge.com