Sampling Plans Part 2 - aami.confedge.comaami.confedge.com/asset/confEdge/aami/_warehouse/files/Sampling... · Sampling Plans –Part 2 of 3: ... Acceptance Sampling Plans can re

Sampling Plans – Part 2 of 3: Acceptance Sampling Plans for Inspection by Variables

Sampling PlansPart 2 of 3

Acceptance Sampling Plans for Inspection by Variables

Peter Knepell, PhDPeak Quality Services

[email protected]

© 2014 Association for the Advancement of Medical Instrumentation www.aami.org6/24/14 1

© 2014 Association for the Advancement of Medical Instrumentation www.aami.org 2

Introducing the Presenter

• Peter Knepell, President of Peak Quality Services

• PhD, Cornell University, Operations Research

• Certified Quality Engineer (CQE) and SoftwareQuality Engineer (CSQE) by the American Societyfor Quality

• Started assisting medical device and pharmaceutical manufacturers in 1994. Presented over 10 popular, AAMI-sponsored webinars.

• Since 1998, specialized in Lean Six Sigma & Design for Six Sigma implementation for a variety of industries & organizations

• AAMI faculty for: Statistics, Design of Experiments, Risk Management, and Process Validation workshops

6/24/14 2


Sampling Plans Webinar SeriesOverall Objectives

36/24/14 © 2014 Association for the Advancement of Medical Instrumentation www.aami.org

At the end of this series, you will be able to:

Create a risk-based sampling plan for the validation of a new production process

Create acceptance sampling plans for inspection by variable

• Create acceptance sampling plans for inspection by attribute

Explain the risks involved with decisions

Balance cost of inspection with risks of making a wrong decision

3

© 2014 Association for the Advancement of Medical Instrumentation www.aami.org

Sampling Plans - Part 2 Objectives

46/24/14

At the end of this webinar, you will be able to:

• Define the fundamental concepts for Acceptance Sampling Plans

• Create Acceptance Sampling Plans for inspection by variable

• Establish an appropriate sample size for inspection by variable

• Explain the risks involved with decisions

4



Webinar Outline

6/24/14 5

• Quick Review of Key Topics Q&A and Email Review from Part 1

FDA References

Definitions

Scenario from Part 1

• Process Capability Measures for Variable Data

• Acceptance Sampling Plans for Variables


Q&A and Email Review from Part 1

6/24/14 6

• Small Production Lots For certain manufacturing methods, first article or first-and-last

article inspections are adequate. (eg, CNC manufacturing)

For other manufacturing methods, 100% inspection may be appropriate. (eg, manual assembly)

Acceptance Sampling Plans can reflect these strategies but will not depend on statistical techniques.

Process monitoring should be considered.

• Use/Misuse of AQL vs. LTPD These terms will be completely covered in Part 3

If you MUST have an immediate answer, drop me an email request for some definitions ([email protected])



Q&A and Email Review from Part 1

6/24/14 7

• Censored Data for Reliability Scenario: Want a device to last at least 50,000 cycles. Test a

sample to 100,000 cycles and collect the times to failure.

Attribute Data Example: One out of twenty tested failed in less than 50,000 cycles. Estimated Failure Rate = 1/20 = 5% + X% . This will be covered in a future webinar: How Much Is Enough?

Variable Data Example: You want to establish a mean time to failure (MTTF) for the device. Beyond scope for these webinars. Consult a reliability expert.


Confidence in Statistical Conclusions Depend on Sampling

86/24/14

“Sampling plans, when used, shall be written and based on a valid statistical rationale. Each manufacturer shall establish and maintain procedures to ensure that sampling methods are adequate for their intended use and to ensure that when changes occur the sampling plans are reviewed. These activities shall be documented.”

21 CFR 820.250 (b)

8



Acceptance Activities

96/24/14

Establish and maintain procedures for acceptance activities .

Include inspections, tests or other verification activities.

• Receiving acceptance

• In-process

• Final acceptance21 CFR 820.80 (a) – (d)

9


Definition of an Acceptance Sampling Plan

106/24/14

• Acceptance Sampling PlanA pathway for deciding on the disposition of a product based on the inspection of one or more samples.

• GoalTo minimize the cost of inspection while understanding the risks of making a wrong decision.

10



High Level Acceptance Sampling Plan

116/24/14

Receive Lot

(N)

Inspect Sample

(n)Accept Lot

Meets Criteria?

Disposition Decision

Some Alternatives:

• Inspect another sample

• Inspect 100%

• Fine your supplier

• Reject lot

Yes

No

11


Specific Example for Variable Data

126/24/14

Receive Lot

N = 1,000

Inspect Sample

n = 15*

Cpk > 1.5 ?

Inspect New Sample

n = 24*

For all 39*, Cpk > 1.25 ?

Accept the lot with 95% confidence the Cpk > 1.0

Inspect 100%

Yes

No

Yes

No

Cpk > 1.25 ?

Yes

No

12

* Based on Cpk Sample Size Table, Slide 40.



Noteworthy Observations

136/24/14

• Acceptance plans can be innovative • Previous example is called a Double Sampling Plan

• Decisions• Expressed in terms of confidence in a quality level

• To perform 100% inspection of the lot after failure to accept is an expensive disposition

• Need to be based on the cost of inspection versus the benefit to the customer

13


Key Assumptions for Acceptance Sampling Plans

6/24/14 14

• Results of inspections are valid (ie, measurement system is good and samples are randomly selected)

• Cost of 100% inspection exceeds the benefits

• Underlying probability distributions are appropriate

• Setting a performance goal is not permission to produce defectives – it is guidance for disposition decisions




Scenario: A manufacturer is going to set up two parallel production lines for a new reagent. During Operational Qualification (OQ) they found that a key quality characteristic for the reagent is very sensitive to:

• pH of a raw material

• purity of water used in the process

In January they will set up both production lines that will operate over two shifts a day, five days a week. Each day, each line will produce a batch of reagent. At the end of the second shift, the equipment will be cleaned.

How should they evaluate PQ results for pH?

15

Analysis of ResultsCpk for Each Batch’s pH


Cpk Line 2Batch 1 Batch 2 Batch 3 Batch 4

1.39 1.94 2.07 1.71


1.28 1.74 1.76 1.56

Goal: Cpk > 1.33Each Sample Size = 16

16

Analysis is incomplete. No level of risk is assigned these performance capability measures. We’ll revisit this table later.



Webinar Outline

6/24/14 17

• Quick Review of Key Topics Q&A and Email Review from Part 1

FDA & GHTF References

Definitions


• Process Capability Measures for Variable Data FDA & GHTF References

Definitions of Cpk, Ppk, level , & DPM Technical Details

• Acceptance Sampling Plans for Variables

Process Capability Requirement

“Where appropriate, each manufacturer shall establish and maintain procedures for identifying valid statistical techniques required for establishing, controlling, and verifying the acceptability of process capability and product characteristics.” [emphasis added]

21 CFR 820.250 (a)

6/24/14 18© 2014 Association for the Advancement of Medical Instrumentation www.aami.org



Capability Studies

• Capability studies are performed to evaluate the ability of a process to consistently meet a specification.

• The most commonly used capability indices are Cp and Cpk.

• Capability studies are frequently used towards the end of the validation to demonstrate that the outputs consistently meet the specifications.

Global Harmonization Task Force (GHTF),

Process Validation Guidance, page 20.

6/24/14 19

Statistical Measures of Process Capability

Name Symbol Estimate of:

Process Capability Index

Cpk Defect-free work

Sigma Level level Defect-free work

Defects per Million DPM or PPM Defect-free work

Defects per Unit dpu Defect-free work

First Pass Yield FPY Defect-free work

Do you notice a trend here?




Cpk – Process CapabilityIf Metric is Variable AND

Normally Distributed

Process Capability Index

Cpk = minimum { USL - , - LSL } 3

• Common measure

• Normally distributed data that do not have to be centered on target

• Red Area represents proportion out of specification (aka, proportion defective)

• DPM = Red Area x 1,000,000

LSL USL

6/24/14 21


Six Sigma Quality SystemsMetric is Variable ANDNormally Distributed

Sigma Level

minimum { USL - , - LSL }

• Exactly 3 times Cpk

• Represents number of standard deviations between current mean of process & closest specification limit

6/24/14 22

level =



Process Capability Example

The packaging team established Seal Strength specs of 10 and 30 lbs. They took a random sample and found:

• an average of 25 lbs

• a standard deviation of 2.5 lbs.

Cpk = Min {30 - 25 , 25 – 10 } / (3 x 2.5) = 5 / 7.5 = 0.67

level = min {30 – 25 , 25 - 10} / 2.5 = 5 / 2.5 = 2.0

DPM = 0.02275 x 1,000,000 = 22,750 (from statistics software)

Compute Cpk, Cp and level .

6/24/14 23


Equivalence of Measures

* For non-centered processes (ie., Cp ≠ Cpk)

• DPM drops as Cpk increases

• Double Cpk by halving standard deviation

• Increasing Cpk dramatically decreases DPM

Cpk level DPM*.33 1 158,655

.67 2 22,750

1 3 1,350

1.33 4 32

1.5 4.5 3.4

Cpk level DPM*.33 1 158,655

.67 2 22,750

1 3 1,350

1.33 4 32

1.5 4.5 3.4

6/24/14 24



Technical NotesCpk vs. Ppk

6/24/14 25

Short-Term Capability

Long-Term Capability

Standard Deviation ST = R / d2 LT =

Process Capability Measure

Cpk =Min { USL - , - LSL }

3 ST

Ppk =Min { USL - , - LSL }

3 LT

n

2

ii 1

y y

n 1√

NOTE: In theory ST < LT , thus Cpk > Ppk (theoretically)

Ppk is often the preferred measure of capability


Technical NotesNon-Normal Data

6/24/14 26

0

5

10

15

20

25

30

24.0 to<=

30.0

30.0 to<=

36.0

36.0 to<=

42.0

42.0 to<=

48.0

48.0 to<=

54.0

54.0 to<=

60.0

60.0 to<=

66.0

USL

0

5

10

15

20

25

3.178 to <=3.328

3.328 to <=3.478

3.478 to <=3.628

3.628 to <=3.778

3.778 to <=3.928

3.928 to <=4.078

4.078 to <=4.228

Class

# O

bse

rvat

ion

s

Normal Distribution Mean = 3.629Std Dev = 0.2282KS Test p-value = .3398

USL

Original Time-to-Beep Data Transformed Data

log(USL) = log(60) = 1.778

time log(time)37 1.56829 1.46234 1.53147 1.67254 1.73252 1.71650 1.69938 1.58033 1.51950 1.69934 1.53144 1.64330 1.477

Most common approach is transform data to make it look more Normal.



Webinar Outline

6/24/14 27

• Quick Review of Key Topics

• Process Capability Measures for Variable Data FDA & GHTF References

Definitions of Cpk, Ppk, level , & DPM Technical Details

• Acceptance Sampling Plans for Variables Key Assumptions

Motivational Example (Scenario 2)

Deriving Sample Size

Acceptance Sampling Plan Pathway

Final Examples (Scenario 1)


Key Assumptions for Acceptance Sampling Plans

(repeated)

6/24/14 28

• Results of inspections are valid (ie, measurement system is good and samples are randomly selected)

• Cost of 100% inspection exceeds the benefits

• Underlying probability distributions are appropriate

• Setting a performance goal is not permission to produce defectives – it is guidance for disposition decisions


Scenario 2 for Variable Data

• You sampled 100 temperature sensors from a lot of 1,000 and created a histogram. The upper and lower spec limits (USL & LSL) are shown.

• Is there an simple way to express how well or poorly your supplier of temperature sensors is performing? 0

5

10

15

20

25

30

35

98.2 98.3 98.4 98.5 98.6 98.7 98.8 98.9

LSL USL



Process Capability Measures for the Sample of 100 Sensors

0

5

10

15

20

25

30

35

98.2 98.3 98.4 98.5 98.6 98.7 98.8 98.9

LSL USL

We would like a Cpk > 1.5. Should we be happy with our supplier of sensors?

98.3 98.4 98.5 98.5 98.6 98.7 98.7 98.8 98.9

USLLSL

N = 100Mean = 98.59Std Dev = 0.056USL = 98.9LSL = 98.3Cpk = 1.73Sigma Lvl = 5.19DPM = 0.1

6/24/14 30


Key Points to Consider About a Result from a Sample

• Sample mean and standard deviation are estimates of the true parameters of the lot. Using them to calculate Cpk also gives an

estimate of the true value for Cpk.

Could get over- or under-estimate of the true Cpk and DPM

• Intuition says that large sample sizes help reduce the risk of over or under estimating. That is correct but how much is enough?

• Need a way to quantify the risk in deciding that the lot is acceptable


Confidence Intervals for Cpk

1. Take a sample (n)

2. Compute estimate for Cpk

3. Compute lower bound for confidence interval based on:

• Sample size (n)

• Estimated Cpk (Est Cpk)

• Confidence Level (1-)%

Area =

Lower Bound

Estimated Cpk

Lower Bound* = Est Cpk –229n

1

2

Z

n

estCpk

*Ref: Crossley, Mark, The Desk Reference of Statistical Quality Methods (2nd Ed), ASQ Quality Press, 2007, pg. 371.

Probability distribution for Cpk



Computing Confidence Interval for Cpk Using SPC XL

• Based on our sample of 100 sensors we are 95% confident that the lot has a Cpk of at least 1.52.

• Since our goal was Cpk > 1.5, we will accept the lot of 1,000 temperature sensors from our supplier.

Sample Size 100Sample Mean 98.59Sample Standard Deviation 0.056Upper Spec Limit (USL) 98.9Lower Spec Limit (LSL) 98.3Confidence Level 95.00%

Cp 1.79Lower Bound for Cp 1.58

Cpk 1.73Lower Bound for Cpk 1.52

Cp and Cpk Confidence Interval (lower bound)

Statistics and Confidence Intervals

User Defined Parameters


Computing Confidence Interval for Cpk Using Minitab




Discussion of Results

If Lower Bound Cpk > Desired Cpk, then we have met our goals for Desired Cpk with a certain level of confidence

Notice for this scenario, our Estimated Cpk was 1.73, which is 0.23 higher than the Desired Cpk of 1.50.

Sample Size 100Sample Mean 98.59Sample Standard Deviation 0.056Upper Spec Limit (USL) 98.9Lower Spec Limit (LSL) 98.3Confidence Level 95.00%

Cp 1.79Lower Bound for Cp 1.58

Cpk 1.73Lower Bound for Cpk 1.52

Cp and Cpk Confidence Interval (lower bound)

Statistics and Confidence Intervals

User Defined Parameters

For this scenario we can say: “We are 95% confident that the lot has a Cpk of at least 1.52.”

6/24/14 35


Illustration of Decision Logic for Sample Size

1) Decide on a minimum acceptable Cpk (“Desired Cpk” or “Spec Cpk”)

2) Take a sample & compute a Cpk for the sample (“Estimated Cpk”)

3) Compute a confidence interval to get a “Lower Bound Cpk”

4) If Lower Bound Cpk < Desired Cpk, then the risk is too great that the lot’s Cpk is less than the Desired Cpk.

5) If we sample more, the lower bound should move up. So we can sample more to see if we can get Lower Bound Cpk > Desired Cpk.

Desired Cpk

Estimated Cpk

Lower Bound Cpk

Lower Bound Cpk

Estimated Cpk

Ideal Situation After Sampling More

Desired Cpk

6/24/14 36


Introduction to Sample Size for Cpk

An Estimated Cpk must be greater than the Desired Cpk. But how much greater?

Answer: The Lower Bound Cpk must be at or above the Desired Cpk

Equivalent Statement:

CI Width < Est Cpk – Desired Cpk

Area =

Lower Bound = Desired Cpk

Est Cpk

CI Width

229n

1

2

Z

n

estCpkConfidence Interval (CI) =Width

Limiting Case



Determining Lower Bound Cpk

229n

1

2

Z

n


Area =

Lower Bound Cpk

Est Cpk

CI Width

Lower Bound Cpk = Est Cpk - CI Width

6/24/14 38



95% Lower Bound Cpk TableLower Bound Cpk = Est Cpk - CI Width

0.60 0.80 1.00 1.20 1.40 1.60 1.80 2.00

10 0.31 0.44 0.58 0.70 0.83 0.96 1.08 1.21

15 0.37 0.51 0.66 0.80 0.94 1.08 1.22 1.36

20 0.40 0.55 0.71 0.86 1.01 1.16 1.30 1.45

25 0.42 0.58 0.74 0.89 1.05 1.20 1.36 1.51

30 0.44 0.60 0.76 0.92 1.08 1.24 1.40 1.56

35 0.45 0.62 0.78 0.94 1.11 1.27 1.43 1.59

40 0.46 0.63 0.79 0.96 1.13 1.29 1.45 1.62

45 0.47 0.64 0.81 0.97 1.14 1.31 1.47 1.64

50 0.47 0.65 0.82 0.99 1.15 1.32 1.49 1.66

55 0.48 0.65 0.83 1.00 1.17 1.34 1.51 1.67

60 0.48 0.66 0.83 1.00 1.18 1.35 1.52 1.69

Estimated CpkSample

Size

6/24/14 39


Determining Sample Size for Variable Data Sampling Plan

Confidence Level (1-)%

max CI Width =Est Cpk – Desired Cpk

Sample Size (n)Estimated Cpk

229n

1

2

Z

n


6/24/14 40



Cpk Sample Size Table

Est Cpk -Desired Cpk 0.67 1.00 1.33 1.50 1.67 2.00

0.05 332 594 952 1175 1424 19800.10 101 177 280 343 413 5700.15 52 88 137 167 200 2740.20 33 55 84 102 121 1640.25 24 39 58 70 83 1110.30 18 29 43 52 61 820.35 15 23 34 41 48 630.40 13 19 28 33 39 510.45 11 17 24 28 32 420.50 10 15 20 24 28 36

Desired Cpk

Sample Size Needed for 95% Confidence That At Least the Desired Cpk is Achieved

6/24/14 41

Acceptance Sampling Plan for Inspection by Variables

1. Decide on a Desired Cpk and a Confidence Level, ()%.

2. Determine an Estimated Cpk based on a small sample, say 10 items.

3. Is Est Cpk < Desired Cpk?• YES: Then sampling more will probably not help. Decide on lot

disposition. DONE• NO: Compute the Lower Bound Cpk using Table.

4. Is Lower Bound Cpk > Desired Cpk?• YES: Then accept the lot with at least ()% confidence. DONE• NO: Go to Cpk Sample Size Table (previous slide) and decide on how

many additional samples to take.

5. Take a additional sample and compute Cpk based on all samples.6. Compute the Lower Bound Cpk using Table.7. Is Lower Bound Cpk > Desired Cpk?

• YES: Then accept the lot with at least ()% confidence. DONE• NO: Decide on lot disposition. DONE



43

Decide on Desired Cpk andConf Lvl (1- )

Take a small sample (n>10)

Est Cpk >Desired Cpk ?

Accept the Lot

Cpk based on small sample?

Yes

No

No

Yes

Yes

No

Decide on Lot

Disposition

A

A

B

Get Lower Bound Cpk from table

Lower Bnd Cpk >Desired Cpk ?

Get Cpk sample size from table

Take additional sample &

compute Est CpkA

B

Acceptance Sampling Plan Pathway

Analysis of ResultsCpk for Each Batch’s pH



1.39 1.94 2.07 1.71


1.28 1.74 1.76 1.56

Goal: Cpk > 1.33Each Sample Size = 16

44

Analysis is incomplete. Need to assign a level of risk to these performance capability measures.

Case 1

Case 3Case 4

Case 2


Case 1: Est Cpk = 2.07


Basic Information:Estimated Cpk = 2.07 for n=16Want 95% Confidence Level that Cpk > 1.33

Pathway Result

• Est Cpk > Desired Cpk

• Lower Bound Cpk ≈ 1.36 (from Table)

• Lower Bound Cpk > Desired Cpk of 1.33

• Accept the lot with 95% confidence that Cpk > 1.33

45




Pathway Result


• Lower Bound Cpk ≈ 1.16 < Desired Cpk of 1.33

• Sample Size from Table = 30 (using interpolation)

• Must sample additional: 30 – 16 = 14

• Compute Estimated Cpk based on all 30 samples

• Compute Lower Bound Cpk based on 30 samples

• If Lower Bound Cpk > 1.33, then accept lototherwise decide on lot disposition

46





Pathway Result


• Lower Bound Cpk ≈ 1.06 < Desired Cpk of 1.33

• Sample Size from Table = 68 (using interpolation)

• Must sample additional: 68 – 16 = 52

• Must decide if sampling this much is worth the investment. NOTE: For a Cpk at the lower bound (1.06) we will have fewer than 1,350 DPM or a defect rate lower than 0.135%.

47



Basic Information:This is the first batch producedEstimated Cpk = 1.28 for n=16Want 95% Confidence Level that Cpk > 1.33

Pathway Result

• Est Cpk < Desired Cpk so go directly to deciding on lot disposition

• Potential disposition decisions: Sample more to bring up lower bound Cpk. Then decide

on disposition based on potential DPM or defect rate.

“Blend” with Batch 3 that has Cpk = 2.07

Scrap this lot and caulk it up to learning curve cost

48


© 2014 Association for the Advancement of Medical Instrumentation www.aami.org6/24/14 49

Key Takeaways

• Computing Cpk or Ppk alone is not enough

• Must establish level of risk with your decision

• Must match assumptions

• Can review results in terms of risk to the customer and DPM

Webinar Series on Statistical Methods & Tools

Scheduled for 2014:• Sampling Plans – Part 3

• July 16 – Acceptance Sampling Plans for Attribute Data

Future Webinars Based on Your Feedback:• Sampling - How Much is Enough?

• Keeping Score - Cpk, FPY, and other process capability metrics

• Visualizing Your Data - Quick & Cheap Tricks

• Control Charts - Visually Monitoring Your Processes

• Design of Experiments – A Three-Part Series

506/24/14 © 2014 Association for the Advancement of Medical Instrumentation www.aami.org 50


On-Site Workshops Available

Statistical Methods & Tools for a Quality System• 3-day Workshop

• Hands on with user-friendly, Excel-based statistical software

• Can integrate your challenges or data into workshop

• No prerequisite knowledge of statistics necessary

Design of Experiments for a Quality System• 3-day Workshop

• Hands on with user-friendly, Excel-based design of experiments software

• Can integrate your challenges or data into workshop

• No prerequisite knowledge of statistics necessary



Questions?

Or press *1 on your telephone keypad

Type your question in the Q&A box on the left side of your screen and press Enter

526/24/14 52


Closing Reminders

• Be sure to fill out the evaluation form at: http://aami.confedge.com/ap/survey/s.cfm?s=Sampling2

• Optional exam for this webinar is available through the eLearning Portal at: http://univerity.aami.org

• AAMI is planning the following webinars that may be of interest to you:

• July 16: Sampling Plans – Part 3 of 3: Attribute Acceptance Sampling Plans


Closing Reminders

• Announcing AAMI University - a better way to manage your professional development

• Online and live comprehensive education resources for medical technology professionals

• Access to AAMI’s industry-leading curriculum and instructors

• Please visit AAMI U at http://university.aami.org/


Learn. Think. Implement.

Documents

Sampling Plans Part 2 - aami.confedge.comaami.confedge.com/asset/confEdge/aami/_warehouse/files/Sampling... · Sampling Plans –Part 2 of 3: ... Acceptance Sampling Plans can re