Reliability for NDT1.pdf · data analysis. Empirical & Model Assisted methods for NDE Reliability measurement-Fore runners: Review of MSE history for NDE processes-Fitting NDT MSE

Reliability for NDT

Tutorials: POD Advanced

June

200

9

4th European-American Workshop

on Reliability of NDE

23 June, 2009

L. Schaefer

ww

w.ndt.net/index.php?id=

8312

Introduction Round:

- How are you involved with NDE reliability in your present role?

- What do you hope to accomplish in the next 6 months with the tools learned here?

Goals:

� Understand history of NDE MSE� Know how to vet requests for POD information� Improve skills for selection and application of POD

tools� Know resources for POD modeling� Become familiar with current state of the art in PO D

data analysis

� Empirical & Model Assisted methods for NDE Reliabil ity measurement

- Fore runners: Review of MSE history for NDE process es

- Fitting NDT MSE tasking to customer protocols – “Cus tomer focus”

- Modular approach for NDT Reliability

- Deploying transfer functions in reproducibility & r epeatability

- POD generation & Confidence bound estimation tools - 1823

- Probability models for noise

Advanced Roadmap –

- A-hat vs a models & applicability

- Incorporation of S+N relationships into POD vs A

- Binomial analysis and asymptotic PODs

- Multiparameter PODs

- Signal & Image based PODs

- Selection & application of models for substantiatio n of capability

Objective: Resource & SME awareness, adopting and developing best practices

Review of MSE history for NDE processes

� Capability and reliability assessments for NDE in u se since the early 1970s

� Motivation: large gaps of “hoped for”, vs validated and experienced on failed hardware

� Various tools/programs/protocols emerged to answer calls for NDE MSE solutions:

� 29/29, GLM, Pass/Fail, PEM, a-hat vs. a, DOE-POD, R OC, Re, S&N� ASM NDT Hnbk, Mil STD 1823� ASM NDT Hnbk, Mil STD 1823� Capabilities handbooks� NDE Reliability working groups: E-A (this week), M APOD, Agency,…

� Continued dueling concerns must be addressed: � Cost of knowledge development high (samples Mfg, la rge canvas of

inspector and inspection facilities)� Desire for high quality reliability knowledge to s upport lean and robust

designs and cost of ownership

Solution: Exploit what we know, drive common practices, advance Modeling

Fitting NDT MSE tasking to customer protocols –“Customer focus”

How^ n for thinking about measuring NDE effectiveness� How can we use our present knowledge to characteriz e an in

use system (instead of re-measuring)?� For an NDE process you assume capable to design

requirements; what size flaw is it capable of detec ting, what proportion of those defects are detected…. And how (confidence) do you know that?(confidence) do you know that?

� Comparing NDE processes, which one will find the sm allest defects? Which one will find the most defects? How do you know this?

� For any durable infrastructure… or product: Would a better inspection improve product life? When should you s chedule an in-service inspection? How do you know that?

� Thanks to Derek Sturges & Chuck Annis

Fitting NDT MSE tasking to customer protocols –“Customer focus”

� What is our task/deliverable?� Validate smallest crack we can find?� Identify the largest crack we can miss?� Validate a detection capability over a range of fla ws to support

probabilistic lifing & maintenance planning?� Who are our stakeholders? What are their higher tie r Reqts?� Resource stewardship:

� Apply knowledge management to understand what we ha ve � Apply knowledge management to understand what we ha ve learned/validated to date

� Conduct gap analysis against deliverables� Design and execute effective NDE MSE workscope

� Understanding the component/infrastructure and flaw genealogy

� Drive toward zoning and focused inspections� Understand the implications of flaw type, size, loca tion and

interaction (with other flaws and matrix)

Modular Approach to NDE Reliability� Initiated by BAM & promoted in the European America n

NDE Reliability working group� Initially defined NDE reliability as:

� Degree that an NDT system is capable of achieving i ts purpose regarding detection, characterization and false call s

� Evolved through the proposals of BAM from� R = f(IC) - g(AP) - h(HF)� Further work led to conclusions that there is no � Further work led to conclusions that there is no

absolute truth regards how to determine NDI reliabi lity, and especially how to quantify the human factors.

� Rather every NDI system must be qualified and valid ated on a case by case basis, leveraging accumulated knowledg e over time

Present vision of Modular Concept

� Continue to work to understand the influences and their interaction for the method and interaction physics, the specific the specific application parameters and influence of human factors

Debated again this week in the workshop

Modular Model concept - MAPOD

� MAPOD is “Model Assisted Probability of Detection” and there exists a working group led by ISU-CNDE to adv ance partial/full model based generation of POD

� Present concept:

Model assisted PODUnderstanding and deploying transfer functions

� When substantiated by examination of data and devel oped understanding of the inspection model, efficiencies can exist where a previously demonstrated POD capability can be tra nsferred to a closely related inspection without a repeat of the experiment

� These partial model assisted PODs are called transf er functions� Examples:

� Flat to curved� Crack to notch

Alloy to alloy� Alloy to alloy� Moderate, but well characterized process changes

� Key Enabler: Appropriate understanding of the under lying physical model

� Effectively these are realized as a debit or credit in the predicted NDE system response to the target flaw or range of flaws

� Its central advantage is to broaden the understandi ng of exam elements and facilitate their application without ( significant) additional cost/effort

Lets: Walk a typical process - MAPOD


- Next after 15 minute break:





- Binomial analysis






POD generation and confidence bound estimation tools

� There exists a significant number of legacy POD/CL generation and analysis tools

� Base:� DOS – PF and Ahat vs. A (WPAFB) use restricted� Excel (w/ WPAFB add on executable)� Minitab� S-Plus� Mathematica…

DOEPOD (NASA)� DOEPOD (NASA)

� The new Mil 1823 (still in draft form) has an avail able and uniquely advantaged software created in open source R

� http://www.r-project.org/� Software and professional training available from

- www.statisticalengineering.com Charles (Chuck) Annis- Short courses - http://www.statisticalengineering.com/mh1823/mh1823-POD-

Course.htm

POD generation and confidence bound estimation tools

How to understand your need:� Literature search and vetting of existing

demonstrations:� NTIAC Capabilities handbook overview

- Available from NTIAC www.ntiac.com

� Steps to incorporate into your assessment







- Binomial analysis






Signal/noise Discrimination

SignalNoiseGood

Poor Poor

Poor process/setupPoor DiscriminationPoor reliability

Good procedure, equipment, … Inspector dependent reliability

SignalNoise

v

Ind

icat

ed

crac

k le

ngth

(a)

True crack length (a)

a vs. a Analysis

� Consider a lognormal scatter in indicated crack length for various cracks lengths

� POD is the probability of indicated

Threshold

� POD is the probability of indicated crack length exceeding the threshold of detection

� Requires quantification of signal leading to detect call


POD







- Binomial analysis






Working with noise

� Calibration “noise” vs. component

� Often weakly assessed

� Noise impacts the POD/ROC Globally

vInd

icat

ion/

com

pone

ntResp

onse

(a)

Decision Threshold

Noise Threshold� Noise impacts the POD/ROC Globally

� Noise needs to be locally assessed� Sampled to the same confidence level as (a)

� Lessons Learned: ETC CBS - UT

� Improved component/method POD/ROC possible when noise/decision are implemented locally

Ind

icat

ion/

com

pone

ntResp

onse

(a)


Noise Threshold

1823: Ensure a thorough understanding of S:N relationships for structure & flaw

� Ensure global and local noise events are characteri zed and accounted for:

1823 Notes on noise–

� See Appendix G







- Binomial analysis






Considerations for Binomial analysis

� Data collection� 1s and 0s (hit/miss) are straightforward

� Implications based on unknown S:N relationships

� How do we know how close to the “edge”?� Implications for Sample library requirements

� Understanding Asymptotic PODs which result

� Challenges in fitting confidence bounds to 1/0 data� Challenges in fitting confidence bounds to 1/0 data

� When might it better than a-hat vs a?� Cost? Risk?

� DOEPOD application – NASA LaRC� Binomialize test data� Improve small data set risk awareness� Method/software in formalization

- SME [email protected]







- Binomial analysis






Multiparameter PODsSignal & Image based PODs

� Approaches recognize the complex nature of POD, and the need to decompose it to understand all sources of v ariance

� Examples- Volume detection probability - Ultrasonic- Signal (amplitude) & Spatial decomposition in image based POD

� Recommended attendance (4 th E-A Reliability workshop)� Multi-Parameter Influence on the Response of the Fl aw to the Phased

Array Ultrasonic NDT System. The Volume PODArray Ultrasonic NDT System. The Volume POD







- Binomial analysis






How to start, Which way to go?

� How do we determine which analysis model is appropriate/best?

� How do we optimize the size/scope of our demonstrat ion?� How do we know we are right?

What question regards POD is being asked?

� Corollary from Understanding Variation





� Hat tip, Bill Bellows

Step 1 – A Do loop of understanding,….

� Stakeholders fully understand the question and consequences of potential solutions

� Different from hearing/reading comprehension, a pro found system of knowledge is required to make effective use of: Ta gucci Methods, Six Sigma, Lean… and POD using NDE MSE methods

� Helpful guidance to motivate further thought www.in2in.org

� Literature search and vetting prior to program deve lopment –� Literature search and vetting prior to program deve lopment –obvious but done less often…

� Models vetted

Then, � Definition of optimal approach to meet objective� Execution of the MSE work scope in accordance with

understanding of requirements &objectives


� Fore runners: Review of MSE history for NDE processes

� Fitting NDT MSE tasking to customer protocols – “Customer focus”

� Modular approach for NDT Reliability

� Deploying transfer functions in reproducibility & repeatability

� POD generation & Confidence bound estimation tools - 1823

� Probability models for noise

Summary & Closure –

� Probability models for noise

� A-hat vs a models & applicability

� Incorporation of S+N relationships into POD vs A

� Binomial analysis and asymptotic PODs

� Multiparameter PODs

� Signal & Image based PODs

� Selection & application of models for substantiation of capability

Feedback:

Did you achieve goals?

� Understand scope and application of majority of POD assessment tools?

� Confident to select and apply?� Motivated to apply to your challenges?

Confident? How do you know that?

References/ResourcesStat Modeling & Hnbk 1823 help� http://www.measuringusability.com/wald.htm� http://www.causascientia.org/math_stat/ProportionCI .html� http://stattrek.com/Help/Glossary� http://www.statisticalengineering.com/Modeling SMEs & Resources� http://www.cnde.iastate.edu/MAPOD/Handbooks/Software� ASM NDT handbook� Mil Std 1823� Mil Std 1823� ASQ Reliability Engineering Bible� Minitab� R – Open source statistical softwarePartner Information Exchange� www.9095.net New release August 2009� How well does your NDT Work

Documents

Reliability for NDT1.pdf · data analysis. Empirical & Model Assisted methods for NDE Reliability measurement-Fore runners: Review of MSE history for NDE processes-Fitting NDT MSE