A Scientific Approach to Improving VUI Design

Peter U. Leppik, CEORick Rappe, VP of Business Development

Vocal Laboratories Inc.

Agenda

1. Principles of Gourmet Customer Service

2. Data Collection Toolbox

3. The Project Life-Cycle

4. Conclusion/Q&A

Principles of Gourmet Customer Service

• Nearly every effort to improve service makes at least one of these mistakes:

1. No data is collected from actual callers.2. The data collected does not relate to business

goals.3. There is not enough data collected to be

statistically significant.4. There is no context or understanding of the data

(for example: what is a “good” satisfactions score?).

5. No action is taken based on the data collected.

The Scientific Approach1. Set Goals

o Use relevant business goals, not technical metrics

2. Gather datao Multiple kinds of tests may be needed to get a complete

picture

3. Take action

4. Validateo Repeat the tests from Step 2.

5. Commit: Go back to Step 1.

Data Collection Toolbox

• There is no one way to collect data about a VUI or call center.o A VUI designer who only uses one test method

is like a carpenter who only owns a hammer.

The Toolbox: General Tools• Test Goals

• Statistical Considerationso Margin of Erroro Probability of Finding Problems

• Prototypeso Functional Prototype (aka “Rapid Prototype”)o Wizard-of-Oz Prototype

The Toolbox: Idea Generating Tools

• Heuristicso “Best Practices”o Expert Opinions, “Audits,” “Assessments”

• Agent Feedback

• Call Recording and Monitoring

The Toolbox: Data Generating Tools

• Controlled Testingo Lab Testso Rapid Assessmentso Large Controlled Tests

• Follow-Up Surveys

• End-of-Call Surveyso Not Recommended due to extreme sample bias

The Toolbox: More Data Generating Tools

• Friends & Family & Employee Testso Not Recommended due to unrepresentative sample

• Automated Load Testing

• Traversal Testing

• Call Logs and Stats

The Project Life-Cycle

• Every project goes through certain phases.o Analysiso Designo Codeo Testo Releaseo Operational

• Every phase has different testing requirements

Analysis Phase• Lay out the project scope, goals, and budget• Ask the big questions

o Who are these callers, and how do they live?o Why are they using the phone and not (e.g.) the

web?o What is the competition doing?

• Look at the big picture and big ideas:o Generate ideas from consultants, vendors,

agents, and customerso Develop baselines & benchmarks

Analysis Phase Tools• Controlled Testing

o Benchmarking, competitive analysis, baseline measurementso Need to understand your callers before you start

• Agent Feedback

• Follow-up Surveys

• Call Recordings

• Heuristicso “Best practices,” consultants’ opinions, etc.

Design Phase

• This is when you plan the details of the VUI

• Use a Prototype Test Prototype Test iterative design cycle

• The keys to success are speed and flexibilityo Test several designs, sometimes in the same dayo Don’t blow the entire testing budget on The

Usability Test

Design Phase Tools

• Prototypeso WOZ or Functional Prototype

• Controlled testingo Rapid Assessment or Lab Test o Lets you use an iterative design processo Nothing ends arguments faster than solid

data.

Code Phase

• Implement the design

• Most testing is for debuggingo Unit testing and some functional testing

• Relatively little VUI testing takes place hereo Implementation may require some VUI changes;

these should be tested

Test Phase

• Last stage before rolling out the application to live callers

• This is where the most thorough testing takes place

Test Phase Tools

• Traversal testo Ensures that the implementation matches the

design

• Large controlled testso Designed to catch even small usability glitches

• Automated load testingo Done last because changes in the first two test

phases can affect load handling

The Test Phase can also be the Acceptance Phase

• Business goals are what are important, not the exact designo They are what should be in the contract.o Tests which measure business goals should be

the acceptance criteria.

• Many of these tests can be used in the design phase.

Release Phase

• This is the process of rolling out the application

• By this time, you should be confident that the system workso Confident doesn’t mean cocky

• This is the time for a phased roll-outo Some call it a “pilot test”o This is a deployment strategy, not a testing strategy

Operational Phase

• The work doesn’t end when the system is live!o Application Drifto External changes

• Company• Caller Lifestyle• Competition

• This looks a lot like the analysis phase for the next project….

Operational Phase Tools• Benchmarking (Controlled Testing)

• Competitive analysis (Controlled Testing)

• Follow-up surveys

• Agent feedback

• Monitoring call stats and logs

• Listening to call recordings

Lessons Learned1. Good service does not have to be more

expensive than bad service

2. Customers are smarter than machines

3. Every system can stand some improvement

4. The best VUI designs come from paying attention to the right things at the right time

Q & A