Using Designed Experiments to Improve Service Quality in a Customer Care Environment

Using Designed Experiments to Improve Service Quality in a Customer Care Environment

Using Designed Experiments to Improve Service Quality in a Customer Care

Environment

Rocky Mountain Quality Conference

Denver, June 2003

Ed PowersVP Corp. Planning and Development

QualityCenter Partners, Inc.

Fort Collins, Colorado 80525


Rocky Mountain Quality Conference ‘03 Ed Powers 970/212-8829 [email protected] 2 of 62

Objective

This presentation helps quality professionals better understand and apply Design of Experiments (DOE) principles in non-manufacturing or customer service environments. It describes Center Partners’ business challenges and how DOE techniques have helped determine the effectiveness and ROI of new software and training solutions.



Agenda• Center Partners—Who We Are, What We Do• Client Expectations and Business Challenges• DOE Overview• Using DOE in a Service Environment• Example 1: Using DOE to Evaluate Performance

Management Software on Quality and Average Handle Time Improvement

• Example 2: Using DOE to Evaluate Monitoring Software and Coaching Effectiveness on Quality Improvement

• New DOE Applications• Summary• Q&A



About Center Partners

• Founded 1997, in Fort Collins, Colorado, as a call center outsourcer specializing in high-touch customer care for complex products

• 8000% growth in first five years to over $80M in 2002 billables

• Purchased in 2001 by the WPP Group• Currently answering over 2 million calls a month on

behalf of clients like Qwest Communications, Xerox, Agilent and Comcast

• Hassle Free Contact Center Services. Done Right. On Time.



Center Partners’ Basic Client Expectations

• Meet contract metrics:– Service Level– Average Handle Time (AHT)– Quality– Sales/Retention Goals– Others



About Service Level…



About Average Handle Time…



Center Partners’ Business Challenges

• Meet or exceed contract metrics• Delight clients• Make money

To meet these challenges, continuous service and

process improvement is not optional!

To meet these challenges, continuous service and

process improvement is not optional!



Design of Experiments Defined

The arrangement in which an experimental program is to be conducted, and the selection of the versions (levels) of one or more factors or factor combinations to be included in the experiment.

Source: ASQ Quality Press,Glossary and Tables for Statistical Quality Control, Second Edition, 1983, 160 pages



General Process Model

ProcessInputs Outputs

Controllable Factors

Uncontrollable Factors

x1 x2 x3

z1 z2 z3

Source: Montgomery, D. C., Design and Analysis of Experiments, Fourth Edition, 1997, 704 pages



Example: Golf

Factor Level

Driver Oversized or regular size

Ball Balata or three-piece

Conveyance Walk and carry clubs or use golf cart

Refreshments Beer or water

Time of day Morning or afternoon




Typical Experimentation

• “One factor at a time”– What if something changes??– Were there any interactions?

• “Best guess”– What factor(s) caused the result?? – How do we know this is the best solution?

DOE tests many variables at once, quickly and efficiently

with more useful results

DOE tests many variables at once, quickly and efficiently

with more useful results




ANOVA—Workhorse of DOE

• ANalysis Of VAriance: Statistical method to separate causes (factors) and effects (response) by accommodating systemic randomness (experimental errors)

• Tests statistical hypotheses and provides confidence levels in conclusions.



How ANOVA Works

Group 1 Group 2

100 101

105 98

98 96

99 99

101 89

110 91

103 93

101 92

90 100

Compare means by analyzing variation within and between groups.

Between

Within



Industrial Example

• Engineer studying effect of varying cotton weight percent in synthetic fiber on tensile strength of cloth material

• Randomized experiment with a single factor at multiple levels

• Hypothesis (H1): tensile strength will be different for different percentages of cotton; “Null Hypothesis” (H0): there is no effect




Example Data

Weight % Cotton

Observed Tensile Strength (lb/in2)

1 2 3 4 5 Avg.

15 7 7 15 11 9 9.8

20 12 17 12 18 18 15.4

25 14 18 18 19 19 17.6

30 19 25 22 19 23 21.6

35 7 10 11 15 11 10.8

Variation Within Treatments

Variation Between Treatments




ANOVA Computations

Source of Variation

Sum of Squares

Degrees of Freedom

Mean Square

F0 P-Value

Cotton Weight %

475.76 4 118.94 14.76 <0.01

Error 161.20 20 8.06

Total 636.96 24

H0 is rejected; cotton weight % DOES affect tensile strength




Experimental Design and Analysis

• Fixed effects• Random effects• Regression• Analysis of Covariance• Randomized Complete

Block• Latin Squares• Graeco-Latin Squares• Balanced Incomplete

Block

• Two-Factor Factorial• 2k, 3k

• Confounding• ½ Fraction; ¼ Fraction• General 2k-p, 3k-p

• Multi-Factor Factorial with Random Factors

• Nested and Split-Plot• Multiple Regression



Using DOE in a Services Environment

• Fewer metrics; results often less tangible • Many more potential variables—many uncontrolled• Environments can be highly dynamic and may

influence testing• Higher PEOPLE content



Process Factors

ProcessInputs Outputs



x1 x2 x3

z1 z2 z3•People•Methods•Materials•Equipment•Environment•Information



Continuum of Observed Performance

Performance is Due to Chance

Alone

Performance is Due People

Factors Alone

Process Individual

What % is the mix in our business?



Detecting Non-Random Events

What is the minimum number of times would you need to flip a coin to determine if it were not “fair”?



Detecting Non-Random Events

What is the minimum number of times would you need to flip a coin to determine if it were not “fair”?Solution: 7. Use the binomial distribution. Assume r = n (you get either all “heads” or all “tails”). To be >99% that the effect is non-random, determine n when y <=0.01:

y = pr(1-p)n-rn!

r!(n-r)!

y n

.5 1

.25 2

.125 3

0.0625 4

0.03125 5

0.015625 6

0.0078125 7

When n=r, y reduces to:

y = pn

With a perfectly balanced coin, there is less than 1% chance of getting 7 heads or 7 tails in a row.

<1%>99%



Identifying LIKELY “High Performers”

Quality

AHT

Other-Save Rate?

y n

0.25 1

.0625 2

.0156 3

y n

0.33 1

.1089 2

.0359 3

.0119 4

p=0.25 p=0.33

y n

.10 1

.01 2

p=0.10

Example: 2 metrics both in upper 10% yields a 99.0% certainty of non-randomness

y n

.20 1

.04 2

.008 3

p=0.20



Good Performance Two Months in a Row(One metric, July and August 2002, Population Average 55 Agents/Mo.)

July and August

QA AHT

Upper 10%

(E=1)

Christy, Charles, Robert Christy, Andrea, Brian, Graham

Upper 20%

(E=2)

Christy, Charles, Robert, Rebecca, Thomas

Christy, Andrea, Brian, Graham, Nancy, Matthew

Upper 25%

(E=3)

Christy, Charles, Robert, Rebecca, Thomas

Christy, Andrea, Brian, Graham, Nancy, Matthew, Victor, Kyle

Upper 33%

(E=6)

Christy, Charles, Robert, Rebecca, Thomas, Trula, Jami

Christy, Andrea, Brian, Graham, Nancy, Matthew, Victor, Kyle, Rigoberto, Jennifer, Stephanie, Robyn

In the Agent population, 13% exhibit non-random behavior for QA, 22% for AHT when

considering top 1/3 of Agents

In the Agent population, 13% exhibit non-random behavior for QA, 22% for AHT when

considering top 1/3 of Agents



Good Performance in the Same Month(2 metrics, July and August 2002, Population Average 55 Agents/Mo.)

QA and AHT July August

Upper 10%

(E=1)

Christy Christy

Upper 20%

(E=2)

Christy Christy

Upper 25%

(E=3)

Christy Christy

Upper 33%

(E=6)

Christy, Stephanie, Robyn Christy, Sarah

Less than 6% of Agents exhibit non-random behavior; we can be at least 99% sure

Christy was a stand-out

Less than 6% of Agents exhibit non-random behavior; we can be at least 99% sure

Christy was a stand-out



Results

Agent Performance is Due to Chance

Alone

Agent Performance is

Due Agent Factors Alone

Process Individual

Quality AHT

At best, Agent factors account for about 50% of observed performance in Center Partners’ business

At best, Agent factors account for about 50% of observed performance in Center Partners’ business



More About People Factors

• People VARY from person to person• People are HABITUAL• People RESPOND DIFFERENTLY when obvious

supervision is present



Side-By-Side vs. Remote QA Scores

Analysis of Variance (jump start data.sta)

Marked effects are significant at p < .05000

SS df MS SS df MS

Effect Effect Effect Error Error Error F p

SCORE 3802.597 1 3802.597 21562.79 98 220.0285 17.28229 .000069

Summary Table of Means (jump start data.sta)

N=100 (No missing data in dep. var. list)

SCORE

R 78.29268

S 90.83051

All Grps 85.69000Min-Max

25%-75%

Median value

Box & Whisker Plot: SCORE

HOW

SC

OR

E

10

30

50

70

90

110

R S

>99.9% confidenceThere is a real difference in quality scores between

remote and side-by-side observation

There is a real difference in quality scores between

remote and side-by-side observation



Example 1: Using DOE to Evaluate Performance Management

Software on Quality and Average Handle Time Improvement



Structure of a Simple DOE Study

“Control” Group

“Test” Group

Production Group



The Brain-EKP® Experiment

• The PDCA Cycle• Plan: Issue, Measures, Causes• Do: Experiment• Check: Results• Act: Learning and Next Steps



The Improvement Cycle

Plan

Do

Check

Act

•Understand the issue•Understand the process•Define the measures•Uncover the root cause(s)•Determine the solution•Establish the goals•Plan the project

•Execute the plan•Implement the solution

•Compare results to goals

•If successful, document, share knowledge and leverage solutions•If unsuccessful, revisit root causes and redo the cycle



Plan: Situation Analysis (June, 2002)

• Issue: Center Partners performance below quality goal

• Process: Call Handling (Center Partners Key Business Process 6.4)

• Measures: QA score goal 80%; current process average 76-78% and in a state of statistical process control



Plan: Causal Analysis

Wrong support structure assumptions

Lack of sr. mgt. reinforcement

QA scores not improving

“Best Practices” issues

Client scoring differences

Agents not motivated

Agents not knowledge-able

Not a priority

Too much time spent on other tasks

Don’t know how

Lack time management skills

Lack of automation

“Special” projects

Lack of support

“Emergencies”

Competing responsibilities

Too few support people

Process defects

Agent life issues

Sporadic events/outages

Low support productivity

Don’t know job priorities

Not a personal priority

Lack of CSM reinforcement

No BFTs

Client politics/ structure

Bias

Bad calibration

Complacency

Not reinforced by Coaches

Don’t know importance

Conflicting metrics/rewards

Don’t agree with forms

Changes not communicated

Training ineffective

Coaches don’t communicateChanges frequently

Not enough QA focusNo Jump Start / Base Camp

Bad data format

No ownership

Can’t find information

Too much reliance on memory

Difficult to navigate

Time/AHT pressure

Changes frequently

Unaware of changes

High point weightingToo ‘lazy’ to look

Skills not habitual

Good habits not formed

Cause and Effect analysis identified Best Practices and Agent Motivation as potential

primary causes

Cause and Effect analysis identified Best Practices and Agent Motivation as potential

primary causes



323 216 188 177 120 104 88 87 277

20.4 13.7 11.9 11.2 7.6 6.6 5.6 5.5 17.5

20.4 34.1 46.0 57.2 64.8 71.4 77.0 82.5 100.0

0

500

1000

1500

0

20

40

60

80

100

Defect

CountPercentCum %

Per

cent

Cou

nt

LVD SOC June 1-24th Pareto

Plan: Pareto Analysis

Data validation showed Best Practices and

Complete/Accurate Notes were 34%, coaching issues

were 41% of the issue

Data validation showed Best Practices and

Complete/Accurate Notes were 34%, coaching issues

were 41% of the issue



Plan: Solution Design

• Solution:– Improve coaching process (a separate project)– Address “Best Practices” issue with The Brain-EKP®

• Goal: – Brain-EKP®: 3% improvement in average QA scores

• Project Plan: – Designed experiment with equally balanced Test and

Control groups– Treatment with/without The Brain-EKP®

– Also study impact on Average Handle Time (AHT)



Example 1 Experimental Model

Call Handling Process

Customer CallsResolved Issues•QA Score•AHT

Controllable Factor


Brain-EKP®

Coaching Client Changes



Plan/Do

Task Duration Completion

Kickoff 1 day 6/27/02

Experimental Design 3 days 7/12/02

Installation and Testing 11 days 7/17/02

Knowledge Model Development 7 days 7/24/02

Training and Roll-out 20 days 7/26/02

Experimental Runs 8 weeks 9/20/02

Progress Checks Weekly Weekly

Conclusions 1 week 9/20/02



Co

ntro

l

Te

st

0.7

0.8

0.9

1.0

Group

QA

Sco

re

Boxplots of QA Score by Group(means are indicated by solid circles)

One-way ANOVA: QA Score versus Group

Analysis of Variance for QA Score

Source DF SS MS F P

Group 1 0.04995 0.04995 7.84 0.006

Error 155 0.98816 0.00638

Total 156 1.03811

Individual 95% CIs For Mean

Based on Pooled StDev

Level N Mean StDev --+---------+---------+---------+----

Control 77 0.83519 0.07752 (--------*--------)

Test 80 0.87088 0.08202 (-------*--------)

--+---------+---------+---------+----

Pooled StDev = 0.07985 0.820 0.840 0.860 0.880

Check: QA Analysis

99.4% certainty of a 3.6% difference

Boxplots and ANOVA show that the Test group

outperformed the Control group





62.5016.3022.0026.5026.7029.1033.7038.0039.7041.3356.10

15.9 4.2 5.6 6.8 6.8 7.4 8.6 9.710.110.514.3

100.0 84.1 79.9 74.3 67.5 60.7 53.3 44.7 35.0 24.9 14.3

400

300

200

100

0

100

80

60

40

20

0

Defect

CountPercentCum %

Per

cen

t

Cou

nt

LVD SOC - Aug 19 - Sept 9 Brain Pilot GroupCheck: Pareto Analysis

Best Practices was reduced to the #8 issue;

Complete/Accurate Notes was now out of the top 10

Best Practices was reduced to the #8 issue;

Complete/Accurate Notes was now out of the top 10



Te

st

Co

ntro

l

15

10

5

Group

AH

T

Boxplots of AHT by Group(means are indicated by solid circles)

One-way ANOVA: AHT versus Group

Analysis of Variance for AHT

Source DF SS MS F P

Group 1 19.17 19.17 3.44 0.066

Error 145 808.46 5.58

Total 146 827.64

Individual 95% CIs For Mean

Based on Pooled StDev

Level N Mean StDev ------+---------+---------+---------+

Control 71 10.469 2.061 (----------*----------)

Test 76 9.746 2.611 (----------*----------)

------+---------+---------+---------+

Pooled StDev = 2.361 9.50 10.00 10.50 11.00

Check: AHT Analysis

93.4% certainty of 43.4 second AHT reduction







Act: Learning

• The Brain-EKP® provided a better user interface for call handling than the one provided by our client, resulting in better QA scores and AHT

• Technology alone is not sufficient—adequate coaching and floor support is needed to ensure tool usage and help modify Agent habits



Example 2: Using DOE to Evaluate Monitoring Software and Coaching Effectiveness on Quality Improvement



Plan: Situation Analysis (January, 2002)

• Issue: Center Partners performance at quality goal but wanted incentive levels; quality monitoring software supplier made claims that their system would improve quality

• Process: Call Handling (Center Partners Key Business Process 6.4)

• Measures: QA at 80% and in a state of statistical process control; goal was to increase 5%



Plan: Pareto Analysis

Pareto of Failure Causes

05

10152025303540

Hab

it

Attitu

de/Oth

er

Learn

ing

Category

Co

un

t

020406080100120

Cu

mu

lati

ve %



Plan: Theory & Assumptions

• Rewards programs have proven unsustainable in the past

• To change a habit, people need to focus on the issue and receive feedback multiple times

• Many more observations and feedback in a shorter time will help change the habits

• Seeing audio and video playback of calls during coaching sessions will improve results

• Using visual aids will help remind Agents to conform to QA expectations



Plan: Solution Design

• Solution:– Run a statistically designed experiment across three sites to

test effects of quality monitoring software, intensive feedback, and visual aids on QA scores

• Goal: – 10% improvement in average QA scores

• Project Plan: – 23 design



Example 2 Experimental Model

Call Handling Process

Customer CallsResolved Issues•QA Score


Uncontrollable FactorsSite

DifferencesClient

Changes

Software FeedbackVisual Aids



Experiment Outline

• Identify 36 individuals to be part of the program. Individuals will be chosen based on average score – 80% +/- 5%.

• 8 combinations (Feedback, Visuals, Software) tested 3 times each in Fort Collins (FTC) and Loveland (LVD), 4 combinations (Feedback, Visuals) tested 3 times each in Idaho Falls (IDF)



Experimental Design (LVD, FTC)REPLICAT NAME FEEDBACK VISUALS SOFTWARE LOCATION CHANGE COACH

1 Low Low Low LVD1 High Low Low LVD1 Low High Low LVD1 High High Low LVD1 Low Low High FTC1 High Low High FTC1 Low High High FTC1 High High High FTC2 Low Low Low FTC2 High Low Low LVD2 Low High Low LVD2 High High Low LVD2 Low Low High FTC2 High Low High FTC2 Low High High FTC2 High High High FTC3 Low Low Low FTC3 High Low Low LVD3 Low High Low LVD3 High High Low LVD3 Low Low High FTC3 High Low High FTC3 Low High High FTC3 High High High FTC

24 Agents total

Examples of treatment:

High – Low – Low = Multiple observations

Low – Low – Low = Control Group

Low – High – Low = Delivery of QA as normal leave visual reminder on their computer about areas they missed

Low – Low – High = include viewing software as part of the “normal” feedback session



Plan/Do

Task Duration Completion

Approval 1 day 1/11/02

Experimental Design and Planning 3 days 1/14/02

Kickoff 1 day 1/15/02

Training, Calibrations, and Roll-out 2 days 1/17/02

Experimental Runs 3 weeks 2/7/02

Analysis 2 days 2/11/02

Presentation of Conclusions 1 day 2/14/02



Check: FTC/LVD ResultsMarginal Means (Unweighted); variable: DIFFERENCE

Design: 2**(3-0) design

NOTE: Std.Errs. for means computed from MS Error=.0062194

Pooled Overall Std.Err.

Feedback Visual Witness Means Std.Dev. Std.Dev. N for Mean

High Low Low .089333 .111142 .111142 3 .045532

High Low High .172967 .061224 .061224 3 .045532

High High Low .066667 .043716 .043716 3 .045532

High High High .043800 .118826 .118826 3 .045532

Low Low Low -.053800 .045048 .045048 3 .045532

Low Low High -.002467 .072750 .072750 3 .045532

Low High Low -.048333 .054243 .054243 3 .045532

Low High High .055333 .075755 .075755 3 .045532

The group exposed to high feedback and monitoring software but not visuals

improved 17%

The group exposed to high feedback and monitoring software but not visuals

improved 17%



Check: FTC/LVD Factor Analysis

Effect Std.Err. t(17) p

Mean/Interc. .040437 .016098 2.51198 .022392

(1)FEEDBACK .105508 .032196 3.27709 .004444

(2)VISUAL .022142 .032196 .68772 .500905

(3)WITNESS .053942 .032196 1.67543 .112144

1 by 2 .053775 .032196 1.67025 .113177

1 by 3 .023558 .032196 .73172 .474305

2 by 3 .013542 .032196 .42060 .679313

11% of the 17% gain can be attributed to feedback

11% of the 17% gain can be attributed to feedback



Check: FTC/LVD Factor AnalysisPareto Chart of Standardized Effects; Variable: DIFFEREN

2**(3-0) design; MS Residual=.0062194

DV: DIFFEREN: =v9-v8

Effect Estimate (Absolute Value)

-.420604

-.687721

.7317222

1.670252

1.675429

-3.27709

p=.05

2by3

(2)VISUAL

1by3

1by2

(3)WITNESS

(1)FEEDBACK

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0



Check: Summary of All Sites

• Out of the 17 agents who received high feedback, 13 showed improvement

• Results show three out of four high feedback agents showed an average of 9.7% improvement in their scores in four days

• Model shows about a 1% gain per high frequency feedback session on average

• Subsequent to the experiment, much of the gains were lost in the first month!



Act: Learning

• High feedback is the primary cause of higher scores; software and visuals contributed a minimum amount—software supplier’s claims need refinement!

• Habits are more difficult to change than originally anticipated



Desire(Want to)

Knowledge(What to do

/Why)

Forming Habits

Skill(How to)

Habit

Source: Covey, S., The Seven Habits of Highly Effective People, 1990, 319 pages

Creating or changing a habit requires work in all three

dimensions

Creating or changing a habit requires work in all three

dimensions



New DOE Applications

• Training tactics and tools• New technologies • Incentive programs• Marketing tactics



Summary

• DOE is an efficient and effective means for understanding cause and effect relationships

• Simple DOE techniques can be used effectively in non-manufacturing or service environments

• PEOPLE factors tend to be significant—especially when changes require a change of habit





Author Biographical Information: Ed Powers

• VP Corporate Planning and Development for Center Partners, Inc.

• 16 years of experience in sales, marketing, quality management, and consulting

• Formerly with Hewlett-Packard, Sorcia• BSEE 1987 Illinois Institute of Technology• HP Quality Maturity System Reviewer, ASQ Certified

Quality Manager, 2003 Baldrige Examiner• Published in AMA Marketing News, Call Center

Solutions magazines