How to Conduct Customer Feedback? Ontario Public Service Management Board Secretariat Training Workshop Rob & Peggy Wong: Add you date and location Rob

How to Conduct How to Conduct Customer Customer Feedback?Feedback?

Ontario Public Service

Management Board Secretariat

Training Workshop

Rob & Peggy Wong:

Add you date and location

Rob & Peggy Wong:

Add you date and location

Curriculum Development

A bit about the course developers…

Ontario Public Service Management Board Secretariat Quality Service Project - stimulate and lead the integration of Quality Service

principles and practices across OPS

Quality Service Management Model – world class, customer service measurement and reward/recognition for outstanding quality service

Georgian College Research Analyst Program Post-graduate program - specializing in applied marketing and social

research

11 months - including three month industry internship

Training and Delivery

A bit about the training and delivery …

CON*NECT (Colleges of Ontario Network for Education and Training) partnered with:

OPS Management Board SecretariatShared Services BureauCoordinate curriculum, train the trainer and delivery

Training to College Trainers College Trainers will deliver two day workshop to:

OPS employees responsible for managing customer service feedback

Potentially to other governments, and Institute for Citizen-Centred Service

Workshop Objectives

You will… Clarify your role in the customer

feedback process Learn the basics of survey research

Planning project scale and scope, selecting method(s), critiquing questionnaires, sampling advice, project management and interpreting the results

Learn about the Common Measurements Tool - CMT Design and conduct your own customer feedback survey Get answers to your questions and concerns about survey

research

Resources

How to Conduct Customer Surveys: Manual Ontario Public Service Restructuring Secretariat, July 2001

How to Conduct Customer Feedback? Training Manual Georgian College, February 2002

Workshop Outline

Day OneWhy survey your customers?

Overview: Research Process

The Context

Your Role Break

Step 1: Plan the customer feedback project

Plan your surveyLunch

Step 2: Construct data collection procedures

Break

Design questionnaire with CMT

Day TwoStep 2: Construct data collection

procedures (continued)

Build our surveyBreak

Step 3: Conduct data collection

Interview each otherLunch

Step 4: Analyze the data

Work with data for

meaningful resultsBreak

Step 5: Act on the results

Put it all together!

Why survey your customers?

Part of the Ontario government’s plan to… Achieve mandate and strategic priorities Improve organizational performance

Obtain timely and reliable information to… Make informed decisions Eliminate a bottleneck Understand customers’ problems Document resource requirements Determine which changes will produce the biggest

payoff

Make changes that customers will notice and value

Improve products and servicesCustomer Surveys: Context

Break Time!

Research Process

ConstructChoose Data

Collection Method

Questionnaire - CMT

Pre-test

Sampling

ConductMonitor response

rates

Monitor timelines

Monitor data quality

AnalyzeAnalysis

Techniques

Adequacy of Findings

Interpretation

Presentation

ActReview implications

Communicate

Presentations

Action Plan

Future research

PlanPurpose & Objectives

Scale & Scope

Co-ordinate & Communicate

Initiate Project

Step 1

Five Steps to Systematically Improve Your Customers’ Satisfaction

Step 1:

Plan the Customer Feedback Project

Your Role…

Determine Purpose & Objectives of Research

Define Scale & Scope

Co-ordinate & Communicate management, staff, consultants, other departments, groups etc.

Initiate the Research Project

Step 1:


Establish the Research Direction

What do I need to know and who can tell me?

Programs & services that will benefit from the research?

Decisions that could be made using the results?

Information gaps?

Issues?

Step 1:


high employee turnoverhigh employee turnover

Research the potential

cause …

market changes?market changes?

new policies?new policies? poor facilities?poor facilities?

Iceberg PrincipleBe careful not to rely on just theBe careful not to rely on just the symptoms…symptoms… complaintscomplaints

low uselow use

staff performance?staff performance?

ContinuousContinuous

ImprovementImprovement

Cumulative Gains

Cumulative Gains

PlanningPlanningMonitoringMonitoring

ImplementationImplementation

AssessmenAssessmentt

Licensed for use by OPS, Copyright David Hutton Associates Inc.

Step 1:


Quality Planning Improvement Cycle

Step 1:


Define Research Objectives Process of determining issue and gaps

leads to research objectives Topic areas not survey questions!! No more than 6 objectives

Researchable Use action verbs

Each should be unique List most important first, profile info last

Step 1:


Well-defined Objectives To identify levels of

customer satisfaction

To measure use patterns of our customers

To suggest service and facility improvements

To profile our customers

Poorly-defined Objectives To find out what our customers

think

To see if customers want longer hours

To evaluate the performance of all employees over ten years at all of our offices

Writing Objectives

Step 1:


Refine Research Purpose Capture the essence of all objectives in one sentence Review, critique, edit & strengthen

Determine Scale & Scope Services/programs to be evaluated Customer group(s) to be surveyed

Customers - use the service directly Citizens - may or may not use the service

Baseline/benchmark/ longitudinal Current research – build on? improve? Project time frame Internal vs outsource

Step 1:


Co-ordinate & Communicate Approval of purpose, objectives, scope and

schedule by senior management Support, co-operation from related program

areas Freedom of Information

Initiate Project Consider alternative appropriate

methodologies Quantitative Qualitative

Develop RFP Contract research professionals

Step 1:


Establish the Research Direction Ask the questions

– Why do we need feedback?– Who could use the information?

Develop the purpose and objectives Write them out

Determine the scale and scope When? With whom?

Co-ordinate and communicate Not applicable

Initiate the project Who will do what?

Let’s Plan a Customer Feedback Project

Scenario

Plan a survey to evaluate this 2-day

workshop

Scenario

Plan a survey to evaluate this 2-day

workshop

Step 1:


Objectives: To determine the participants’

satisfaction with the workshop Content, delivery, location,

learning environment, facilitators

To explore the benefits of the workshop to participants

To identify workshop improvements

To profile the participants’ research backgrounds and roles

Scale and scope Who?: participants When?: lunch on day 2 Responsibility?

Tool?: Method?: Processing?: Analysis?: Act?:

Workshop Evaluation ObjectivesPurpose:

to evaluate the quality and usefulness of the Workshop

Rob & Peggy Wong:

This is left blank before the workshop and then you can record their plans from the previous small group exercise.

Rob & Peggy Wong:

This is left blank before the workshop and then you can record their plans from the previous small group exercise.

Step 1:


Involve the your staff in the process Close to the customer

Survey for the right reasons: Understanding vs scorekeeping

Collect only what you need not you would like Useful and meaningful information

Use your research to take action Link the results to improvements

Manage customer satisfaction More than just measuring

Keys to Success

Lunch break!

Research Process


Scale & Scope


Initiate Project


rates

Monitor timelines


AnalyzeAnalysis

Techniques


Interpretation

Presentation


Communicate

Presentations

Action Plan

Future research


Collection Method

Questionnaire - CMT

Sampling

Step 2


Step 2:

Construct Data Collection Procedure

Your Role…

Understand appropriate Methodologies & Data Collection Methods

Learn about CMT & Questionnaire Design

Discover the Sampling Mystery

Step 2:

Construct Data Collection Procedures

Data Collection Methodologies Overview

Two broad types: quantitative and qualitative Consider type of information needed, sampling, time and

costs Several possible options but one better to reach customers Ideally, triangulate with more that one method Technology can save time and money in long term

Step 2:


Quantitative (Survey) Large number of customers Need the number Conclusive factual information Know the possible options

customers might want but need to know their preference

Dig into the data for patterns and comparisons of sub groups of customers

Examples of surveys: mail self administered, Web based, personal interview, telephone interviews

Qualitative Small number of customers Need in-depth information about

an issue Exploratory Unsure of the options Need to understand perceptions Explore the range of ideas not the

preferred

Examples: observations, focus group discussions, mystery shoppers, in-depth interviews

Step 2:


Self-administered

questionnaire

Interview

Mail

Survey Research Methods

Web

Face-to-Face

PDA

Telephone

(CATI)

Step 2:


Survey Method Cost Customer

Convenience

Response Rate

Contact Info

Turn-around

Bias Data Entry Data Cleanup

Self-administer

Mail (scannable)

Moderate High

Low to moderate Yes Moderate

to slowSome Easy Some

Web Low High Moderate to high

No Fast Moderate to high

Easy None

Interview

Telephone Moderate Moderate

to low

High Yes High Low Easy Low

Face-to-Face High Moderate High No High Low Moderate Some

PDA High Moderate High No High Low Easy None

Comparison of Survey Research Methods

Updated from How to Conduct Customer Surveys: Manual

Step 2:


What is the Common Measurements Tool?

Set of commonly used survey questions and response measurements

Developed by Citizen-Centred Service Network Enables organizations to compare results Facilitates sharing of information gained and lessons

learned Can be used to set benchmarks

Step 2:


How does it work?

Central core of standard questions Standard 5-point measurement scales Customer satisfaction measurements based on

5 elements of service experience: expectations, perceptions, satisfaction, importance and priorities for improvement

5 service dimensions: responsiveness, reliability, access & facilities, communications, costs

Additional questions customized to meet needs

Step 2:


service/product delivery

location hours

access & facilities com m unication

reasonable

cost

frequency likes & dislikes

general use patterns

Com m on M easurem ents Tool

CMT: Let’s take a peek…

Let’s Take a Break!

Step 2:


Sandwich theory Top: Customer Behaviour – easy to answer, descriptive

Inside: Measurement – attitudes, needs, desires

Bottom: Customer Information – profile the respondents

Pretest questionnaire Wording, flow, instruction clarity, appropriate response

categories, validity

Three stages Operationalize – start with objectives, brainstorm & sort

Wording – select your questions, ensure relevance

Design – design for impact, readability, confidentiality and thanks

Step 2:


Let’s Try it Ourselves!

First critique questionnaires… Your Turn…Build your Own Feedback

Survey Sandwich Question Format:

introduction use 6 sections from CMT select questions, customize for workshop design closed-ended questions choose one open-ended question

Use the sandwich theory Pretest Print final version, we will review surveys

tomorrow

Scenario

Design your own survey to evaluate

this workshop

Finished for the day…

…see you bright and early tomorrow

Welcome back!

Step 2:


Sampling Mysteries…

Why? Saves time & money Enables repetition Reduces errors

Goal? Census vs sample Represent a larger POPULATION Reliable results

What is your role? Offer access to sample frames with ensured ethical practices Define the study population Understand sample size appropriate for your study Review and monitor the sampling procedure

Step 2:


Define population Sample element Sample unit Extent Time

Identify sampling frame Registration lists Customer lists In office intercepts

Issues: Confidentiality/anonymity

www.gov.on.ca/MBS/english/fip

Use by a third party Storage and destroying of lists

Example: Ontario Parks Visitors- camping visitors to Silent Lake provincial park in past three years

- winter visitors (day and overnight) to provincial parks this season

Step 2:


What size sample do you need? Census vs sample Dependent on qualitative vs quantitative measures

What do you really need to know? Need a sample between 350-400 minimum More for subgroup analysis

Based on knowing (handouts) Population size Acceptable margin of error/sampling error/confidence interval Estimate of population proportion Confidence level Accurate within+- 5.0 (ME), 19 times out of 20 (95% CL)

Step 2:


Comparing Population Size to Sample Size

0100200300400500

population of customers

sam

ple

size

*

* with ME=5, EPP=70 & CL=95

E-sample size calculatorSource: http://www.surveysystem.com/sscalc.htm#ssneeded

Step 2:


Comparing Margin of Error to Sample Size

0

500

1000

1500

2000

2500

2 3 4 5 6 8 10 15margin or sampling error/confidence interval

sam

ple

size

* with POP=75000, EPP=70 & CL=95

Industry target


Step 2:


Comparing Population Proportion to Sample Size

0

100

200

300

400

50 55 60 65 70 75 80 85estimate of population proportion

sam

ple

size

* With POP=75000, ME=5, & CL=95

worst case scenario


Step 2:


Comparing Confidence Level with Sample Size

0100200300400500600

90 95 97 99confidence level

sam

ple

siz

e

* With POP=75000, EPP=70 & ME=5

Industry target

Step 2:


Select for a representative sample Non-probability (no population size and no estimate of

error) Convenience Judgmental Quota

Probability (known and can calculate sampling error) Simple random Systematic Stratified Clustered Multi-stage

Sampling Procedures

Break…so soon!

Step 2:


Show time! Let’s Build Our Survey

Overnight the survey grew How did we get there? Started with our objectives

What sections were needed? What could we use from CMT? Examine, compare the questions, check for….

Logical order Response categories mutually exclusive and collectively exhaustive

Design Does the introduction have all of the information needed? Are the instructions clear? Is the questionnaire easy to follow?

Pretest it again Finalize

Rob & Peggy Wong:

Here you want to bring back their questionnaire design homework from the day before.

You can use the Palm computer or paper and pen to collect the feedback on the workshop.

Rob & Peggy Wong:

Here you want to bring back their questionnaire design homework from the day before.

You can use the Palm computer or paper and pen to collect the feedback on the workshop.

Step 2:


KISS Select method based on quality of information

Understand technological opportunities

Questionnaire design Use CMT to your advantage Fewer and focused questions

Sampling Use current lists Get advice on FOI Manage response rate

Seek advice from research professionals

Keys to Success


Scale & Scope


Initiate Project

Research Process


Collection Method

Questionnaire - CMT

Sampling

AnalyzeAnalysis

Techniques


Interpretation

Presentation


Communicate

Presentations

Action Plan

Future research


rates

Monitor timelines


Step 3


Step 3:

Conduct Data Collection

Your Role…

Monitor the data collection

Data quality

Response rates and sample size

Timelines

Step 3:


Method Advantages/Disadvantages Keys to Success

MailSelf-

administered

Wide geog distribution

Compete with junk mail

Longer response time

Wait for response rate

Low response rate (10-20%)

Ensure sample frame is up-to-date Avoid holidays and weekends

Design professionally & personally

Use scanning technology

Track response rates, use reminders, 2nd mailing

WebSelf-

administered

Easy to launch

Inexpensive, less labour

Sample bias potential

Too easy for non-research professionals

Security and privacy of data

Design by research professional

Ensure secure access & data storage and ownership

Monitor response rates

Implementing Data Collection

Step 3:


Method Advantages/Disadvantages Keys to Success

TelephoneInterview

Reach target respondents

Immediate response

Manage sample size

Compete with telemarketing & fundraising activities

Interfering Technology (answering machines, call display)

Low participation rate (10:1)

Well trained interviewers

Monitor calls for quality control

Monitor response rates and sample size

Consider computer assisted telephone interviewing (CATI)

Possibly sample non-respondents

PDAInterview

Reach target respondents

Available anywhere on-site

Immediate access to results

Dazzle factor results in higher response rates

Interviewer costs higher

Technology challenges

Design by research professional

Well-trained interviewers

Monitor quality control

Monitor response rates and sample size

Download data daily

Implementing Data Collection

Step 3:


Collecting and Managing the Data If paper-based, need to manually input in database or spreadsheet

software (SPSS, SAS, Excel, Access, Mintab, Lotus, Quatro Pro and more)

Set up a standardized coding Verify data input Then need to clean data (logic errors, input errors)

Step 3:


Monitoring data collection Work with research professionals Monitor response rates continually to ensure sample Monitor quality control interviewer training, quality control checks

Small cost compared to investment in all other steps

Keys to Success

Note: Under achieving sample size, undermines achieving meaningful information for decisions

Time for recharging -

Lunch

But while you wait…

Complete our Workshop Feedback Survey!


Scale & Scope


Initiate Project

Research Process


Collection Method

Questionnaire - CMT

Pre-test

Sampling

ConductMonitor data quality

Monitor response rates & sample size

Monitor timelines


Communicate

Presentations

Action Plan

Future research

AnalyzeAnalysis

Techniques


Interpretation

Presentation

Step 4


Step 4:

Analyze the Data

Your role…

Understand the research purpose & objectives

Know the decisions that have to be made

Work with the research professional… pose the questions… let them bring you the information you need… work together to draw the conclusions

Step 4:

Analyze the Data

Build Knowledge1. Review Big Picture

Descriptive statistics: frequency tables and descriptives Review the patterns for all of the data

2. Examine Subgroup Patterns Example: male vs female, high vs low users, first time vs repeat

visitors

3. Dig deeper for relationships Cross-tabulations: satisfaction by array of independent variables Explore for segments with factor or cluster analysis Explore for statistical relationships: differences among groups,

influencing factors, cause and effect relationships

Note: Your research professionals will be able to provide advice about meaningful information

Step 4:

Analyze the Data

Last Use of Service (n=359)

<week8%

2wks12%

month43%

2-5 mons23%

year+14%

Effective Graphs

Variables affecting opinions and satisfaction

Compare to other use information for representativeness of sample

Connect with other service providers for comparable information Are they similar? or different?

First vs Repeat Customers (n=440)

first40%

repeat60%

Step 4:

Analyze the Data

Satisfaction

LOW

LOW

HIGH

HIGH

Importance

Important to customers

Poor performance

Customer expectations not met

Important to customersImportant to customers

Good performanceGood performance

Customer expectations Customer expectations metmet

Less important to customers

Poor performance

Low customer expectations matched to poor performance

If importance increases then..?


Good performance

Clearly exceeding customer expectations

Efforts maybe unrecognized, priorities misplaced

Driver Analysis

Step 4:

Analyze the Data


Better performance

Clearly exceeding customer expectations

Efforts maybe unrecognized, priorities misplaced


Better performance

Customer expectations met

Satisfaction

LOW

LOW

HIGH

HIGH

Importance


Poor performance

Customer expectations not met


Poor performance

Low customer expectations to poor performance

If importance increases then..?

courteouscourteous

helpfulhelpful

number of peoplenumber of people

flexibleflexible

good listenersgood listeners

wait timewait time

up to date infoup to date info

Driver Analysis - Example

Step 4:

Analyze the Data

Examine the graphs Sample size Benchmark for comparison Units of measure

Conclusions Which sample 2000 or 2002 is more

representative of Ontario population?

What customers are more highly represented?

How has the customer base changed between 2000 and 2002?

What are the demographic characteristics that might affect use patterns, opinions, and satisfaction?

Age

05

1015

2025

3035

40

18-24 25-34 35-44 45-54 55-64 65+

% o

f re

spo

nde

nts

Ont pop 1996

2000 n=495

2002 n=448

Education

01020

30405060

708090

high school university post grad

% o

f re

spo

nde

nts

Ont pop

2000 n=432

2002 n=486

How representative are the customer samples?

Step 4:

Analyze the Data

Presenting the Results

Reporting Technical Report PowerPoint presentation

Spreading the Word Communicating with management & staff Relevant departments/agencies Customers

Step 4:

Analyze the Data

Let’s examine your feedback about the workshops:

Rob & Peggy Wong:

Incorporate the results from the workshop as tables using whatever software you want. It is also useful to present a graph and also a few of participant’s comments to demonstrate analysis, reporting and the value of both qualitative and quantitative information.

Rob & Peggy Wong:

Incorporate the results from the workshop as tables using whatever software you want. It is also useful to present a graph and also a few of participant’s comments to demonstrate analysis, reporting and the value of both qualitative and quantitative information.

Step 4:

Analyze the Data

Keys to Success

Understand the decisions to be made Work closely with research professionals

Ask questions Let the researchers find the meaningful information

Understand the limitations of the sample Know how to interpret information

Use quantitative information to conclude patterns, relationships, differences, customer segment

Use qualitative information to characterize the bread of issues through the use of their words and their examples

Think of audience when deciding on how to format results Use graphs and tables to summarize effectively

Break!


Scale & Scope


Initiate Project

Research Process


Collection Method

Questionnaire - CMT

Pre-test

Sampling

ConductMonitor Response

Rates

Monitor Timelines

Monitor Data Quality

AnalyzeAnalysis

Techniques


Interpretation

Presentation

ActReview Implications

Communicate

Presentations

Action Plan

Future Research

Step 5


Step 5:

Act on the Results

Your Role…

Review the implications Communicate the research results to management Engage management and staff to develop actions Develop action plans

measurable outcomes, responsibilities and timelines

Consider future customer feedback research

Step 5:

Act on the Results

Recover Quick response

ReportUseable, record for future

BriefShare findings with decision makers

PrioritizeSmaller action steps

CommunicateShare with your customers and colleagues

ImproveRespond to results with actions

EnhanceGive added value

RewardRecognize contributions by all

PlanImprove the feedback process

Source: Adapted from Customer Service Guidelines: How to Engage Management and Staff

Keys to Success

How to Conduct How to Conduct Customer Feedback?Customer Feedback?

Involve the your staff in process Closest to the customer

Survey for the right reasons: Understanding vs scorekeeping

Collect only what you need not you would like Useful and meaningful information

Use your research to take action Link the results to improvements

Manage customer satisfaction More than just measuring

Last Words…

Build Build Customer Satisfaction Customer Satisfaction

throughthrough Feedback Feedback

It’s in your hands!It’s in your hands!

Ontario Public Service

Restructuring Secretariat

Documents

How to Conduct Customer Feedback? Ontario Public Service Management Board Secretariat Training Workshop Rob & Peggy Wong: Add you date and location Rob