Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Copyright Autoliv Inc., All Rights ReservedSaving More Lives
Four Innovative Methods to Evaluate Attribute Measurement Systems
Thomas Rust – Reliability Engineer/Trainer
Sept. 2016
Minitab Insights 2016 – Four Methods for Attribute MSA - 2 Copyright Autoliv Inc., All Rights Reserved
31%
35%
17%
7%
10%
Autoliv
RoA
Japan
Europe
Americas
ChinaSales
2015
Global Footprint
Sales and technology leader
Global Company
Sales to all major vehicle manufacturers
Minitab Insights 2016 – Four Methods for Attribute MSA - 3 Copyright Autoliv Inc., All Rights Reserved
Autoliv Products
Inflatable curtain airbag
Passenger airbag
Steering wheel
Driver airbag
Side airbag
Seatbelt systems
Vision system
Electronic control unit
Night driving assist
Driver assist radar
Blind spot radar
Minitab Insights 2016 – Four Methods for Attribute MSA - 4 Copyright Autoliv Inc., All Rights Reserved
Our Guiding Principles
Minitab Insights 2016 – Four Methods for Attribute MSA - 5 Copyright Autoliv Inc., All Rights Reserved
Our Strategy to Stay AheadRelentless focus on Operational Excellence
Zero Defects by flawless
execution
One Product One Process to improve
cost effectiveness and robustness
Innovation to lead industry
in Real Life Safety
Minitab Insights 2016 – Four Methods for Attribute MSA - 6 Copyright Autoliv Inc., All Rights Reserved
Measurements
Minitab Insights 2016 – Four Methods for Attribute MSA - 7 Copyright Autoliv Inc., All Rights Reserved
The Gray Areas
Area I
Parts are measured bad
Area II
Parts are sometimes
measured good and
sometimes bad
Due to random variation in the
measuring system
Area III
Parts are measured good
I II II IIII
LSL USL
Target
Minitab Insights 2016 – Four Methods for Attribute MSA - 8 Copyright Autoliv Inc., All Rights Reserved
AIAG Attribute MSA Methods
Cross-Tab Method
Kappa
Confidence Intervals on
Percent Agreement
Signal Detection
Approach
Analytic Method
Minitab Insights 2016 – Four Methods for Attribute MSA - 9 Copyright Autoliv Inc., All Rights Reserved
The Attribute Paradigm
All we can do is count
the number of
agreements /
disagreements
All attribute data are the
same
It always takes a lot of
samples to evaluate and
attribute measurement
system
Minitab Insights 2016 – Four Methods for Attribute MSA - 10 Copyright Autoliv Inc., All Rights Reserved
Breaking The Attribute Paradigm
There are four types of Attribute Measurement systems
1. Underlying variable measurement
2. Attribute measurement of a variable product
3. Variable measurement of an attribute product
4. Attribute measurement of a attribute product
Think
Minitab Insights 2016 – Four Methods for Attribute MSA - 11 Copyright Autoliv Inc., All Rights Reserved
1- Underlying Variable Measurement
The characteristic is measured with variable data (or could be), but only the binary result is reported
ExamplesMeasuring height and
classifying as good or bad
Measuring force but only recording if above or below a limit
Operator could classify ‘how good/bad’ but only classifies as pass/fail
92
Minitab Insights 2016 – Four Methods for Attribute MSA - 12 Copyright Autoliv Inc., All Rights Reserved
Underlying Variable MeasurementMethod
MethodRecord the variable
measurement
Treat the same as other variable measurements Gage R&R
Have operators rate the ‘how’ good or bad each part is even if they don’t normally
A rating system (1-10) can be established1 = Best
4 = Barely passes
5 = Barely fails
10 = Worst
Minitab Insights 2016 – Four Methods for Attribute MSA - 13 Copyright Autoliv Inc., All Rights Reserved
Sprocket Inspection
A sprocket is supplied to a company to be included in an assembly
The end customer will interface with the sprocket so the appearance is important as well as the dimensions
The acceptance criteria is based on: Color
Surface Roughness
Paint Quality (no chipping)
Parts are inspected as pass or fail at the supplier, a sorting company, and the assembly company
All examples and data in
this presentation are
plausible fabrications
Minitab Insights 2016 – Four Methods for Attribute MSA - 14 Copyright Autoliv Inc., All Rights Reserved
Sprocket InspectionSet-Up and Method
43 Parts were selected
Both good and bad parts are found for different criteria Acceptable Color (AC) – 9
Bad Color (BC) – 13
High Roughness (RO) – 7
Chipped Paint (CP) – 9
Good Parts (GP) – 5
Three judges were selected as the experts to identify the standard rating of each partOne from the assembly
company (Customer)
One from the sorting company
One from the supplier
Each Part was inspected three times by each judge Parts were presented in a
random order
Part identity was hidden from each judge
Minitab Insights 2016 – Four Methods for Attribute MSA - 15 Copyright Autoliv Inc., All Rights Reserved
Initial Study - Roughness
Study Var %Study Var
Source StdDev (SD) (6 × SD) (%SV)
Total Gage R&R 2.11386 12.6831 95.30
Repeatability 1.71194 10.2716 77.18
Reproducibility 1.24003 7.4402 55.90
Judge 1.24003 7.4402 55.90
Part-To-Part 0.67238 4.0343 30.31
Total Variation 2.21822 13.3093 100.00
Minitab Insights 2016 – Four Methods for Attribute MSA - 16 Copyright Autoliv Inc., All Rights Reserved
Second Study - Roughness
Study Var %Study Var
Source StdDev (SD) (6 × SD) (%SV)
Total Gage R&R 1.69965 10.1979 84.39
Repeatability 1.37447 8.2468 68.24
Reproducibility 0.99981 5.9989 49.64
Judge 0.99981 5.9989 49.64
Part-To-Part 1.08074 6.4845 53.66
Total Variation 2.01415 12.0849 100.00
Minitab Insights 2016 – Four Methods for Attribute MSA - 17 Copyright Autoliv Inc., All Rights Reserved
Initial Study - Color
Study Var %Study Var
Source StdDev (SD) (6 × SD) (%SV)
Total Gage R&R 2.24318 13.4591 76.98
Repeatability 1.98639 11.9183 68.17
Reproducibility 1.04218 6.2531 35.77
Judge 0.79825 4.7895 27.39
Judge*Part 0.67003 4.0202 22.99
Part-To-Part 1.85980 11.1588 63.83
Total Variation 2.91389 17.4833 100.00
Minitab Insights 2016 – Four Methods for Attribute MSA - 18 Copyright Autoliv Inc., All Rights Reserved
Second Study - Color
Study Var %Study Var
Source StdDev (SD) (6 × SD) (%SV)
Total Gage R&R 2.22694 13.3616 83.22
Repeatability 0.86923 5.2154 32.48
Reproducibility 2.05029 12.3018 76.62
Judge 1.57253 9.4352 58.77
Judge*Part 1.31562 7.8937 49.17
Part-To-Part 1.48366 8.9019 55.44
Total Variation 2.67591 16.0555 100.00
Minitab Insights 2016 – Four Methods for Attribute MSA - 19 Copyright Autoliv Inc., All Rights Reserved
Initial Study – Chipped Paint
Study Var %Study Var
Source StdDev (SD) (6 × SD) (%SV)
Total Gage R&R 2.64770 15.8862 90.70
Repeatability 1.77333 10.6400 60.74
Reproducibility 1.96612 11.7967 67.35
Judge 1.32016 7.9209 45.22
Judge*Part 1.45699 8.7419 49.91
Part-To-Part 1.22964 7.3778 42.12
Total Variation 2.91931 17.5158 100.00
Minitab Insights 2016 – Four Methods for Attribute MSA - 20 Copyright Autoliv Inc., All Rights Reserved
Second Study – Chipped Paint
Study Var %Study Var
Source StdDev (SD) (6 × SD) (%SV)
Total Gage R&R 1.48823 8.9294 44.20
Repeatability 0.69921 4.1952 20.76
Reproducibility 1.31375 7.8825 39.02
Judge 0.65389 3.9233 19.42
Judge*Part 1.13945 6.8367 33.84
Part-To-Part 3.02051 18.1230 89.70
Total Variation 3.36724 20.2034 100.00
Minitab Insights 2016 – Four Methods for Attribute MSA - 21 Copyright Autoliv Inc., All Rights Reserved
2- Attribute measurement of a variable product
The characteristic could
be measured with
variable data but the
measurement system
being used can only
classify the results in
binary
Examples
Height Pass/Fail Gage
Diameter Go/No Go Gage
AIAG’s Signal Detection Approach and
Analytic Method apply to this type of system
Minitab Insights 2016 – Four Methods for Attribute MSA - 22 Copyright Autoliv Inc., All Rights Reserved
Logistic Regressionwith Normit Link Function
Parts across the range
from consistently good
to consistently bad
Measure multiple times
Quantify percent pass
Model and S-curve
Normit Link Function52.552.051.551.050.550.049.549.0
1 .0
0.8
0.6
0.4
0.2
0.0
Diameter
Pro
ba
bil
i ty
of
Pa
ss
Minitab Insights 2016 – Four Methods for Attribute MSA - 23 Copyright Autoliv Inc., All Rights Reserved
Go / No-Go Example
A Go / No-Go gage is used to verify that a hole is not too small
Parts are created that span the range from bad to good with some very close to the limit
Each part was measured carefully with a CMM
The parts where evaluated with the Go / No-Go gage Using actual operators
Each part was measured 10 times
Parts were evaluated in a random order
Minitab Insights 2016 – Four Methods for Attribute MSA - 24 Copyright Autoliv Inc., All Rights Reserved
Go / No-Go ExampleResults
Normit Link Function
P = Φ( α + βx )
μ = -α / β
σ = | 1 / β |
Example Calculations
μ = -(-136.7) / 2.674 = 51.1
σ = | 1 / (-136.7)| = 0.374
Bias = 51.1 – 50 = 1.1mm
%GRRTol = 6σ/Tol =
(6*0.374)/18 = 12.47%Specification Limits => 50-68mm
Binary Logistic Regression with Normit Link
Function
Minitab Insights 2016 – Four Methods for Attribute MSA - 25 Copyright Autoliv Inc., All Rights Reserved
Residual Analysis
Minitab Insights 2016 – Four Methods for Attribute MSA - 26 Copyright Autoliv Inc., All Rights Reserved
Go / No-Go ResultsMinitab Macro
Minitab Insights 2016 – Four Methods for Attribute MSA - 27 Copyright Autoliv Inc., All Rights Reserved
AIAG Example
Minitab Insights 2016 – Four Methods for Attribute MSA - 28 Copyright Autoliv Inc., All Rights Reserved
Logit ErrorCal ErrorSDA Error
1 0
0
-1 0
-20
-30
Data
Interval Plot of SDA Error, Cal Error, Logit Error95% CI for the Mean
The pooled standard deviation is used to calculate the intervals.
Gage R&R and Attribute MSA Simulation
Minitab Insights 2016 – Four Methods for Attribute MSA - 29 Copyright Autoliv Inc., All Rights Reserved
3 - Variable Measurement of an Attribute Product
Count of Paper Sheets by weight
Proximity Sensor verifies a
component is present
Camera detects a correct
configuration
Optical Sensor
Minitab Insights 2016 – Four Methods for Attribute MSA - 30 Copyright Autoliv Inc., All Rights Reserved
Attribute MSAVariable measurement of an attribute product
MethodRecord sensor output for
‘good’ and ‘bad’ conditions
Voltage, output, etc.
Create distribution curves
Estimate Alpha and Beta Risks
Include Lower Bound of Confidence Interval
Measurement ‘System’ includes the gage variation and the product variation
900.0
50.0
01.0
51.0
02.0
04 05 06 07 08 0
48.83 3.833 45
85.88 2.1 78 1 00
Mean StDev N
D
87
ytisn
eD
ata
M
elbairaV
traP dooG
traP gnissi
H lamroN
traP dooG ,traP gnissiM fo margotsi
Minitab Insights 2016 – Four Methods for Attribute MSA - 31 Copyright Autoliv Inc., All Rights Reserved
Part Presence Sensor Example
A camera system is used to detect if a washer is present
The camera is ‘taught’ what a good condition is
When connected to a computer, the camera reports the percent pixels that match the good condition
A threshold (limit) is set to define good from bad at 78%
Multiple good and bad conditions are measured and the percentages of pixel match are recorded
Good
Bad
Minitab Insights 2016 – Four Methods for Attribute MSA - 32 Copyright Autoliv Inc., All Rights Reserved
Histograms with Limit
900.0
50.0
01.0
51.0
02.0
04 05 06 07 08 0
48.83 3.833 45
85.88 2.1 78 1 00
Mean StDev N
D
87
ytisn
eD
ata
M
elbairaV
traP dooG
traP gnissi
H lamroN
traP dooG ,traP gnissiM fo margotsi
Minitab Insights 2016 – Four Methods for Attribute MSA - 33 Copyright Autoliv Inc., All Rights Reserved
Good ConditionsCapability Analysis
A capability analysis in Minitab can estimate the z-score of the threshold (limit)
A lower bound of the z-score can also be reported The lower bound takes into
account the sample size and the uncertainty in the estimate
Non-normal distributions can be used when applicable
Minitab Insights 2016 – Four Methods for Attribute MSA - 34 Copyright Autoliv Inc., All Rights Reserved
Alpha RiskProbability of Failing a Good Part
Using the Z-Score and a
normal distribution plot,
the alpha risk can be
calculated
The upper bound of the
alpha risk can be
calculated from the
lower bound Z-score
0.005
0.004
0.003
0.002
0.001
0.000
Percent Pixel Match
Den
sity
-3.62
0.0001473
Normal, Mean=0, StDev=1
Distribution Plot - Alpha Risk
0.005
0.004
0.003
0.002
0.001
0.000
Percent Pixel Match
Den
sity
-3.16
0.0007888
Normal, Mean=0, StDev=1
Distribution Plot - Alpha Risk Lower Bound
Minitab Insights 2016 – Four Methods for Attribute MSA - 35 Copyright Autoliv Inc., All Rights Reserved
Missing Part AnalysisBeta Risk
1 .0000E-1 2
8.0000E-1 3
6.0000E-1 3
4.0000E-1 3
2.0000E-1 3
0.0000E+00
Percent Pixel Match
Den
sity
7.61
1.3656E-14
Normal, Mean=0, StDev=1
Distribution Plot - Beta Risk
1 .0000E-09
8.0000E-1 0
6.0000E-1 0
4.0000E-1 0
2.0000E-1 0
0.0000E+00
Percent Pixel Match
Den
sity
6.25
2.0523E-10
Normal, Mean=0, StDev=1
Distribution Plot - Beta Risk Lower Bound
Minitab Insights 2016 – Four Methods for Attribute MSA - 36 Copyright Autoliv Inc., All Rights Reserved
Reporting Methods
Risk Alpha Risk = 0.0147%
Alpha LB = 0.0789%
Beta Risk = 1.37E-12%
Beta LB = 2.05E-8%
PPM Alpha Risk = 147 PPM
Alpha LB = 789 PPM
Beta Risk = 1.37E-8 PPM
Beta LB = 2.05E-4 PPM
Measurement Capability PpkGood = 1.21
LB for PpkGood = 1.05
PpkBad = 2.54
LB for PpkBad = 2.08
Reliability Pass a Good Part
99.985%
3 Nines
99.922% (LB)
3 Nines (LB)
Fail a Bad Part ~100%
13 Nines
99.99999998% (LB)
9 Nines (LB)
Minitab Insights 2016 – Four Methods for Attribute MSA - 37 Copyright Autoliv Inc., All Rights Reserved
Part Presence SensorMinitab Macro
800.0
50.0
01.0
51.0
02.0
04 84 65 46 27 08 8
85.88 2.1 78 1 00
48.83 3.833 45
Mean StDev N
P
87
ytisn
eD
hctaM lexiP tnecre
P
elbairaV
liaF
ssa
A
)seniN 9(08999999999.0
daB a liaF
)seniN 3(22999.0
dooG a ssaPdnuoB rewoL %59
527548.0-daB a liaF
73726.4dooG a ssaP
rotcaF ytefaS elitnauQ
00000000000.1
daB a liaF
58999.0dooG a ssaP
detamitsE
ytilibaborPnoitacifissalC
6102 naJ 61eoD nhoJ
rosneS ecneserP traP
:yduts fo etaD:yb detropeR
:eman egaG
detubirtsid yllamron eb ot demussa era ataD
traP gnissiM .sv traP dooG - yrammuS ASM .tt
Minitab Insights 2016 – Four Methods for Attribute MSA - 38 Copyright Autoliv Inc., All Rights Reserved
Part Presence SensorMinitab Macro - Residuals
Minitab Insights 2016 – Four Methods for Attribute MSA - 39 Copyright Autoliv Inc., All Rights Reserved
Other Examples
900.0
50.0
01.0
51.0
02.0
4.26 2.76 0.27 8.67 6.18 4.68 2.1
86.33 2.376 1 00
69.42 3.471 45
Mean StDev N
P
08
ytisn
eD
hctaM tnecreP lexi
P
elbairaV
liaF
ssa
A
)seniN 2(0399.0
daB a liaF
)seniN 1(099.0
dooG a ssaPdnuoB rewoL %59
621817.0-daB a liaF
00475.3dooG a ssaP
rotcaF ytefaS elitnauQ
8899.0
daB a liaF
699.0dooG a ssaP
detamitsE
ytilibaborPnoitacifissalC
6102 naJ 61eoD nhoJ
rosneS ecneserP traP
:yduts fo etaD:yb detropeR
:eman egaG
detubirtsid yllamron eb ot demussa era ataD
traP gnissiM .sv traP dooG - yrammuS ASM .tt
900.0
20.0
40.0
60.0
80.0
01.0
21.0
41.0
61.0
81.0
41 82 24 65 07 48 8
92.27 2.583 1 00
21 .1 6 3.881 45
Mean StDev N
S
56
ytisn
eD
egatloV rosne
P
elbairaV
liaF
ssa
A
)seniN 02>(000000000000.1
daB a liaF
)seniN 02>(000000000000.1
dooG a ssaPdnuoB rewoL %59
997256.0-daB a liaF
9204.71dooG a ssaP
rotcaF ytefaS elitnauQ
000000000000.1
daB a liaF
000000000000.1dooG a ssaP
detamitsE
ytilibaborPnoitacifissalC
6102 naJ 61eoD nhoJ
rosneS ecneserP traP
:yduts fo etaD:yb detropeR
:eman egaG
detubirtsid yllamron eb ot demussa era ataD
traP gnissiM .sv traP dooG - yrammuS ASM .tt
900.0
50.0
01.0
51.0
02.0
04 05 06 07 08 0
85.88 2.1 78 1 00
48.83 3.833 45
Mean StDev N
P
47
ytisn
eD
hctaM tnecreP lexi
P
elbairaV
liaF
ssa
A
)seniN 7(469999999.0
daB a liaF
)seniN 6(81999999.0
dooG a ssaPdnuoB rewoL %59
999927.0-daB a liaF
97219.6dooG a ssaP
rotcaF ytefaS elitnauQ
000000000.1
daB a liaF
89999999.0dooG a ssaP
detamitsE
ytilibaborPnoitacifissalC
6102 naJ 61eoD nhoJ
rosneS ecneserP traP
:yduts fo etaD:yb detropeR
:eman egaG
detubirtsid yllamron eb ot demussa era ataD
traP gnissiM .sv traP dooG - yrammuS ASM .tt
900.0
20.0
40.0
60.0
80.0
01.0
21.0
41.0
0 41 82 24 65 07 48 8
89.64 3.432 1 00
42.27 20.42 45
Mean StDev N
S
87
ytisn
eD
egatloV rosne
P
elbairaV
liaF
ssa
A
)seniN 1(319.0
daB a liaF
)seniN 2(5899.0
dooG a ssaPdnuoB rewoL %59
613487.0-daB a liaF
22815.7dooG a ssaP
rotcaF ytefaS elitnauQ
069.0
daB a liaF
7999.0dooG a ssaP
detamitsE
ytilibaborPnoitacifissalC
6102 naJ 61eoD nhoJ
rosneS ecneserP traP
:yduts fo etaD:yb detropeR
:eman egaG
detubirtsid yllamron eb ot demussa era ataD
traP gnissiM .sv traP dooG - yrammuS ASM .tt
Minitab Insights 2016 – Four Methods for Attribute MSA - 40 Copyright Autoliv Inc., All Rights Reserved
Evaluation over Time
454137332925211 71 3951
60
40
20
Observation
Ind
ivid
ua
l Va
lue
_X=42.27
UCL=53.44
LCL=31.10
454137332925211 71 3951
30
20
1 0
0
Observation
Mo
vin
g R
an
ge
__MR=4.20
UCL=13.73
LCL=0
111
1
11
11
11
1111
1
1
1
1
11111
11
111
1
111
1
11
1
11
111
1111
1
Worksheet: Worksheet 2
I-MR Chart of Good Parts
Minitab Insights 2016 – Four Methods for Attribute MSA - 41 Copyright Autoliv Inc., All Rights Reserved
Risks and Concerns
Non – normal
Multiple failure modes
‘Like catching Terrorists’
No variation
Difficult to collect data
Changes over time
Variation in the process
vs. measurement
system
Minitab Insights 2016 – Four Methods for Attribute MSA - 42 Copyright Autoliv Inc., All Rights Reserved
4 - Attribute Measurement of an Attribute Product
Judgement of a Coin
toss
Mechanical detection of
a part present
Binary response of
correct set-up
Minitab Insights 2016 – Four Methods for Attribute MSA - 43 Copyright Autoliv Inc., All Rights Reserved
Attribute Measurement of an Attribute Product
Method
Attribute Agreement
Analysis with Lower-Bound
Confidence Interval
Caution needs to be taken
on selection of good and
bad parts
A minimum of 50 total parts
with at least 150
observations should be used
for a reliable analysis
Selecting parts
At least 25%
of parts will
be in all
categories?
Randomly select parts
from the entire process
A significant
sample of
parts from
each
category is
available?
Cautiously create at
least 10 parts for each
category that will
represent the process
Randomly select at least
10 parts from each
category
no
yes
yes
no
Minitab Insights 2016 – Four Methods for Attribute MSA - 44 Copyright Autoliv Inc., All Rights Reserved
Examples of Attribute Agreement Analysis
CBA
95
90
85
80
75
70
65
Appraiser
Perc
en
t
95.0% CI
Percent
CBA
95
90
85
80
75
70
65
Appraiser
Perc
en
t
95.0% CI
Percent
Date of study:
Reported by: John Doe
Name of product: Sprocket
Misc: Chipped Paint
Assessment Agreement
Worksheet: All Data Attribute Example
Within Appraisers Appraiser vs Standard
Date of study:
Reported by: John Doe
Name of product: Sprocket
Misc: Roughness
SupplierSorting CoCustomer
1 00
80
60
40
20
0
AppraiserP
erc
en
t
95.0% CI
Percent
Assessment Agreement
Within Appraisers
Minitab Insights 2016 – Four Methods for Attribute MSA - 45 Copyright Autoliv Inc., All Rights Reserved
Attribute Agreement AnalysisImpact of Parts Range
Minitab Insights 2016 – Four Methods for Attribute MSA - 46 Copyright Autoliv Inc., All Rights Reserved
Best Practices
Mission
Possible
The New Approach
To Attribute MSA
Historical
Approach
to Attribute
MSA
Minitab Insights 2016 – Four Methods for Attribute MSA - 47 Copyright Autoliv Inc., All Rights Reserved
Every year, Autoliv’s products
save over 30,000 lives
and prevent ten times
as many severe injuries
Thank You!
Minitab Insights 2016 – Four Methods for Attribute MSA - 48 Copyright Autoliv Inc., All Rights Reserved
References
MEASUREMENT SYSTEMS ANALYSIS Reference Manual
Fourth Edition
AIAG
Copyright 2010, Chrysler Group LLC, Ford Motor Company, General
Motors Corporation
EMP III, Evaluating the Measurement Process & Using
Imperfect Data
Donald J. Wheeler
Copyright 2006 SPC Press