Test it in Service

Test it in Servicea.k.a. Voice of the Customer

The 17th Annual International Deming Research Seminar

The 17th Annual International Deming Research Seminar 2

What We Plan to Cover (AIM)

23 March 2011

Why Test in Service?

Why Good Sample Design?

Accuracy versus Precision

Some Statistical issues

Some Methodological Issues

Some Data Issues

Some Questionnaire Issues


Deming Lecture Japan 1950

23 March 2011

4

4. Test in Service

3. Sell

5. Redesign

(a. Old Way)

(b. Original Shewhart 1939) (c. Deming 1950 Version)

Evolution of Shewhart CycleSpecification

Design

Production

Make

Inspection

Sell

1. Design

2. Make


Production Viewed as a System

A

B

C

D

Suppliers ofMaterials andEquipment

Receipt and test of materials

Tests of Processes, machines, methods, costs

Production, assembly, inspectionConsumerConsumer

ConsumerConsumer

ConsumerConsumer

Consumer

ConsumerResearch

DesignandRedesign

23 March 2011


Process

6

Input Conversion Output

Voice of the Process

Voice of the Customer

GAP

23 March 2011


How do we get the Voice of the Customer?

7

Input Conversion Output

Voice of the Customer

Test it in Service

Consumer Research

23 March 2011


Test it in Service

Thought

Data

Knowledge23 March 2011


Survey Impact on Management

Wrong Design

Wrong Information

Wrong Decision

23 March 2011


What is the aim of the survey?

Who determines the aim? The Subject Matter Expert

Why is the aim important? Without an aim there is no system

23 March 2011


Who is involved in Surveys

Subject Matter Expert

Statistician

Respondents

23 March 2011


The Subject Matter Expert

■ Commissions the survey■ Determines the aim■ Can act on the results■ Has budget authority

23 March 2011


Aspects of Sample Design

■ Statistical Sample size Variance Accuracy

■ Methodological Sample selection Computation

23 March 2011


What is a sample

■ Complete coverage gives answer■ A subset of the interest group

Draw Sample

23 March 2011


The Frame

■ The Frame is a list of sample units■ Frames are not always complete

The Gap

23 March 2011


Data vs. Information

The sample yields data

Results Data

Information

AnalysisThe Subject Matter Expert needs Information

23 March 2011


Information contains

■ Result

■ Measure of uncertainty

23 March 2011

18

Which is more important

Sample size?

or

Sample method?

Bias

Sampling Error

Total Error

23 March 2011 The 17th Annual International Deming Research Seminar

Sample size?or

Sample method?


Accuracy and Precision

Not Accurate Accurate

Not Precise

Precise

Sample Size

Sample Method

23 March 2011


Sample Size (n)

Expected result percentage

Desired precisionWhere is the actual result

t the number of standard deviations under the Normal Curve

Then

p

/D p p p p

2

2

(1 )t p pn

D

tArea for Mean ±ts

1 68%

2 95%

3 99.73%23 March 2011


Sample Size Example

Q. Was the Conference useful?

Please check one: Yes □ or No □

If one wants 95% reliability that the actual responses fall within 10% of the response of 50% then .0.5, 0.10, 2p D t

2 2

2 2

(1 ) 2 0.5 (1 0.5)100

0.1

t p pn

D

23 March 2011

The 17th Annual International Deming Research Seminar 2223 March 2011

2

2

(1 )t p pn

D

Why = 50% estimate?p

2 2

2 2

2'

t tf p p

D D

2

2

2

2

1' 0.5

2 2

tDf p p

tD

2

2

2"

tf p

D 0.5 is maximum

2 2

2

t p pf p

D


23 March 2011

Because it is the maximum sample size

Why = 50% estimate?p


Sample Method

23 March 2011

■ Clear Aim■ Random Sampling■ Unbiased Questions


Clear Aim

23 March 2011

What does the owner wish to accomplish with the data?

How will the data be used in practice?

Is the data measureableCause

Effect

Is the data useable?

e.g. measure engagement with an Association

e.g. measure how often contacted by Association


Random sample

23 March 2011

■ Equal probability of selection Random numbers Random Start nth Sample Replicated design with random start per zone Deming's Replicated Design:

Total sample cut into zoneslike sliced bread

Total sample size

One zone


Reduce Bias in Questionnaire

23 March 2011

If questionnaire…

…is for interviewing then train interviewer toAvoid loaded questions

Avoid verbal bias

Avoid body language bias

…is for respondent's completionAvoid loaded questions

Avoid question overload

Open Ended Questions

28

An open-ended question is designed to encourage a full, meaningful answer using the subject's own knowledge and/or feelings.

Opposite of closed-ended question, which encourages a short or single-word answer.

Open-ended questions also tend to be more objective and less leading than closed-ended questions.

Courtesy Myriam Ochart, ASQ CSSBB, CMQ/OE

Open-Ended Question Examples

29

Closed-Ended Questions Open-Ended Questions

Do you get on well with your boss?

Tell me about your relationship with your boss.

Who will you vote for this election?

What do you think about the two candidates in this election?

Is that a photograph of your children?

Tell me about the children in this photograph.


Prevent Response Bias

30

Avoid Ambiguity: Questions not focused enough to obtain the needed information.

Poor question: How often do you buy fast food?

Better question:

The last 10 times you’ve eaten lunch out, how often have you purchased each of the following types of food?



31

Avoid questions that are too general.

Poor question: Do you like orange juice?

(Do you mean taste? texture? price?)

Better question - more specificity:

Do you like the taste of fresh-squeezed orange juice?



32

Avoid loaded questions: use of language likely to bias response:

Do you actually support Proposition X?



33

Avoid leading questions: Questions that lead respondents toward a particular answer.

Isn’t it true that women are more likely to talk on the phone while driving?



What We Covered

23 March 2011

Why Test in Service?

Why Good Sample Design?

Accuracy versus Precision

Some Statistical issues

Some Methodological Issues

Some Data Issues

Some Questionnaire Issues


Questions?

23 March 2011

Documents

Test it in Service