Some New Ideas for Reliability Studies

Stefan Steiner, Jock MacKay and Richard Cook

University of Waterloo

shsteiner@uwaterloo.ca

Quality and Productivity Research Conference

Minneapolis, Minnesota

May 19, 2005 2

Outline

Background and Motivation

Idea 1 – adding extra observations

Idea 2 – selecting units

Tentative conclusions

May 19, 2005 3

Measurement reliability studies

Medical examples reliability of the ultrasonographers for the measurement of

stenosis in the carotid artery clinical assessment of patients with psoriatic arthritis

Industrial examples Piston head diameters Parking brake tightness

May 19, 2005 4

Standard Measurement Reliability (Gage R&R) Study

Goal is to estimateor

y y

y y

y y

k

k

n nk

11 1

21 2

1

,...,

,...,

....

,...,

Y T Mij i j

( )total

( )parts

(measurement)

2 2( ) / ( )parts measurement

2 2 2parts parts measurement

May 19, 2005 5

General Context

Two sources of variation measurement model

general two component model: X vs. Rest

2 2, Y T MY T M

Y

R

Var E Y X E Var Y X

Var E Y X

[ ( | )] [ ( | )]

[ ( | )] 2

(within)(between)

May 19, 2005 6

Ideas

Suppose we have complete or partial knowledge of the distribution of Y:

What is the gain if we know (i.e. ) or use supplementary data from the baseline investigation?

Motivation: Gage R&R study on piston head diameters

What units should we select?

Motivation: Disassembly/Reassembly investigation for door closing effort

Y total

May 19, 2005 7

Idea 1: Add Baseline Data

Add Baseline data:

Standard reliability study

y y

y y

y y

k

k

n nk

11 1

21 2

1

,...,

,...,

....

,...,

Y T Mij i j

( )total

( )parts

(measurement)

y yn n m 1, ...

May 19, 2005 8

Estimating

Sum of squares df EMS Within W nk( )1 m2 Between B n1 m pk

2 2

Between B* m1 m p2 2

( ) / , ( )* 1 21 1

B

Wk

B

W

( ) a a1 21

Look at as a function of n, k, m andstdev( ) /

May 19, 2005 9

n=10, k=3, m=1,2,…., 0 33 1 0 3 0. , . , .

. 3 3

1 0.

3 0.

. 3 3

1 0.

3 0.

May 19, 2005 10

n=4, k=6, m=1,2,….,

.33

1 0.

3 0.

0 33 1 0 3 0. , . , .

May 19, 2005 11

n=2, k=11, m=1,2,….,

.33

1 0.

3 0.

0 33 1 0 3 0. , . , .

May 19, 2005 12

Conclusions for Idea 1

Effects of adding extra observations are greatest when is large, i.e. most variation comes from the parts when n is small

There are some design implications

May 19, 2005 13

Idea 2: Select Specify Units

Suppose the baseline distribution of Y is known, e.g. N(0,1).

Motivation: Consider a problem where we wish to determine whether variation in y is due to assembly or components?

In an investigation we repeatedly disassembly and reassembly two car doors, measuring the closing force each time.

Standard practice is to select two units with extreme and opposite values of y

May 19, 2005 14

Assessing Idea 2

Select one unit with

Disassemble & reassemble, measure

Model

Estimate the intra-class correlation

Y z0

y y yk1 2, , ...,

Y Y Y Y z N z Ikt

1 2 0 1 1 11, ,..., | ~ ( ,( ) ( ) )

2 2, Y C AY C A

May 19, 2005 15

Why Use Extreme Units?

Use two extremes Use two randomly chosen units

lowhigh

1.4

1.3

1.2

1.1

velocity

clos

ing

effo

rt

lowhigh

1.4

1.3

1.2

1.1

velocity

clos

ing

effo

rt

What happens when all the variation is due to Assembly (measurement)?Components (process/parts)?

May 19, 2005 16

When all Variation is Due to

Assembly (or Measurement)

We get a lot of information from the initial (extreme) values observed in the baseline

lowhigh

1.4

1.3

1.2

1.1

velocity

clos

ing

effo

rt

lowhigh

1.4

1.3

1.2

1.1

velocity

clos

ing

effo

rt

Use two extremes Use two randomly chosen units

May 19, 2005 17

Assessing Idea 2

Simulation study (use 5000 trials) Measure one unit k times Compare results for different true intraclass corrrelations

Compare the approaches Using the extreme unit (initial value z) – determine the

maximum likelihood estimate for Using a random unit – estimate the measurement variation

with the sample stdev, then

2p

21 m

May 19, 2005 18

= 0.2, k = 20, z = 2

May 19, 2005 19

= 0.2, k = 20, z = 4

May 19, 2005 20

= 0.2, k = 5, z = 2

May 19, 2005 21

= 0.8, k = 20, z = 2

May 19, 2005 22

Conclusions for Idea 2

Selection provides large advantages in some cases Better than random selection when is small. Good precision even with very small samples Advantage greater when

number of repeated measurements is small we use a more extreme unit

Selection can be used to advantage in reliability studies if we use order statistic to select units for repeated measurement.

May 19, 2005 23

Tentative Overall Conclusions

For problems with two variance components, we get relatively small benefit adding extra observations on the total large benefit by selecting units extreme in the total

Remaining Problems more than two components, e.g. different labs, operators

etc. selection based on order statistics