Design Verification White Paper

8/13/2019 Design Verification White Paper

1/12

DesignVerificationTheCaseforVerification,NotValidation

Page1of12

MEDIcept,Inc. 200HomerAvenue Ashland,MA01721

MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept 11/2010

Overview:The

FDA

requires

medical

device

companies

to

verify

that

all

the

design

outputs

meet

the

design

inputs.

TheFDAalsorequiresthatthefinalmedicaldevicemustbevalidatedtotheuserneeds.Whenthereare

somanymoredesigninputsandoutputsthanspecificuserneeds,whydocompaniesspendsolittle

timeverifyingthedeviceandsomuchtimeandmoneyonvalidation?Andwhatistheroleofrisk

managementindeterminingtheamountoftestingrequired?Thispresentationwilldemonstratethatif

developersconductmorecompleteverificationsofdesignoutputsandriskmitigations,validationscan

becompletedinashortertime,morereliably,andmoresuccessfully.

Introduction:InourexperienceworkingwithmedicaldevicemanufacturerstoimprovetheirQualityManagement

Systemsandtogainregulatoryclearanceofnewdevices,wehavefoundthatDesignVerificationisan

oftenunderutilizedtoolforensuringsuccessduringthelatterstagesofdevicedevelopmentefforts.In

manycases,limitedverificationeffortsrepresentalostopportunitytosignificantlyreducethescopeof

validationeffortsandtherebyreducetimetomarketanddevelopmentcosts.

ThispaperexplorestheuseandmisuseofDesignVerificationandhowdevicemanufacturescangetthe

mostoutoftheirverificationefforts. Butfirst,somebackground...

Background:Medicaldevicemanufacturershavebeenworkingtoaligntheirdesignanddevelopmentsystemswith

theFDAsdesigncontrolregulations(21CFR820.30)sincetheirreleasein1996.Theregulationsare

structuredtoensurethatdevicemanufacturersmaintaincontrolovertheirdesignsthroughoutthe

developmentprocessandthatthemarketeddeviceissafeandeffectiveforitsintendeduse. Attheir

mostbasiclevel,theregulationsrequirethatmanufacturers:

Clearlystatewhattheyintendtoproduce(PlanningandDesignInputs);

Developa

design

that

meets

those

needs

(Design

Outputs

and

Design

Review);

Confirmthatthedesignmeetstheoriginalintent(Verification);

Confirmthatmanufacturedproductcanbeproducedreliablyandachievethedesiredresult

(TransferandValidation);and


2/12


Page2of12



Maintainrecordsofkeydevelopmentactivitiesanddecisions(DesignChangesandDesign

HistoryFile)

Itwouldbehardtoarguethatanyofthesestepsshouldbeexcludedfromanydevelopmenteffort.

Whetheryouarebuildingabridge,acellphone,orasurgicaltool,thiscombinationofstepshasalogical

flowthatmostengineerswouldsimplyconsidertobegoodpractice.

Thechallengeariseswhenyoumovefromtheconceptualworld,whereproductdevelopmentcanbe

seenassinglestreamofwaterflowingdownahill:steadilymovingfromonepointtothenextwithout

interuption;totherealworld,fullofrapids,eddies,branchingstreams,and (occasionally)deeppoolsof

waterthatdontseemtobemovingatall.Inthisturbulentenvironment,itscanbedifficulttomaintain

adisciplineddesignapproachparticularlywhenbusinessschedulesdemandrapidprogressandany

delaysinmovingontothenextdevelopmentphaseriskaffectingprojectmilestonesanddevelopment

staffdeploymentplans.Thesechallengesaremultipliedwhenthedevicehasmultiplecomponentsand

integrationpointsthatmustbemanagedthroughoutthedevelopmentprocess.

Muchhasbeenwrittenabouttheimportanceofearlystageplanningandproblemsolvingtospeed

timetomarketandreducedevelopmentcostsandthevalueofclearDesignandDevelopmentPlans

andwelldefinedDesignInputscannotbeoverstated.Anequalamountofattentionhasbeenfocused

onDesignTransferandValidationactivities.Atthislatestageofadevelopmenteffort,theprojectscope

willhaveexpandedtoincludemoreactiveinvolvementofClinical,OperationsandMarketingstaff.In

addition,thecostofvalidationstudiesandthemakeorbreakaspectofthesestudiesrequireagreat

dealofattentionandcompanyresources.Problemsatthevalidationstagewillhaveseriousimplications

forthe

success

of

the

project

and

especially

for

small

companies

could

determine

the

fate

of

the

wholecompany.

Comparedtotheseearly andlatestageactivities,theimportanceofDesignVerificationtendstobe

overlooked.Thereareseveralreasonswhy:

Definitions:Manypractitioners(andestablishedQualitySystems)continuetohavetroubledefiningexactlywhatverificationis,andhowitfitsintothedesigncontrolprocess.Thetermis

oftenconsideredtobesynonymouswithvalidation andthedevelopmentofcombined

V&Vplansthatdonotestablishacleardistinctionbetweentheobjectivesofthetwoefforts

dont

help.

[Note:

when

discussing

verification

with

the

FDA,

its

best

not

to

talk

about

your

V&Vplanincreasingly,inspectorspreferencesaretoaddressthetwoactivitiesseparately.]

Timeframe:Verificationisoftenanongoingeffortconductedthroughoutthedevelopmentofoutputs.So,agreatdealoftimemaypassbetweenwhenthefirstandlastoutputisverified.


3/12


Page3of12



Withthisextendedtimeframe,verificationactivitiescangetlostwithinallofthechurning

associatedwithdevelopingfinaloutputs.

Approaches:Thereareavarietyofwaystocompletetheverificationforanoutput,soitcanbe

difficulttocommunicatethatalloftheseactivities,asagroup,representthedesignverification.

Theproblemwiththislackofattentionisthatitweakensacriticallinkinthedesigncontrolchain,

affectingthestrengthoftheoveralldesigncontroleffort.Poordesignverificationcanleadtoproblems

duringdesigntransferandvalidation,andcanreduceyourabilitytotrackdownandcorrectproblemsif

(andwhen)theyoccur.Justasimportantly,itmayrepresentalostopportunitytooptimizevalidation

activities,reducetimetomarket,andincreaseyouroverallconfidenceinthesafetyandefficacyofyour

products.

KeyConcepts:Tomakesurethatwerealignedontheproperuseofthetermverification,hereareafewkeypoints

fromtheFDAsDesignControlGuidanceForMedicalDeviceManufacturers: Definition:Verificationmeansconfirmationbyexaminationandprovisionofobjectiveevidencethatspecifiedrequirementshavebeenfulfilled[820.3(aa)].

TypesofVerificationActivities: Verificationactivitiesareconductedatallstagesandlevelsofdevicedesign.Thebasisofverificationisathreeprongedapproachinvolving: tests,inspections,

andanalyses.

Anyapproachwhichestablishesconformancewithadesigninputrequirementisanacceptable

meansofverifyingthedesignwithrespecttothatrequirement.Inmanycases,avarietyof

approachesarepossible...themanufacturershouldselectandapplyappropriateverification

techniquesbasedonthegenerallyacceptedpracticesforthetechnologiesemployedintheir

products.Table

1

provides

some

examples

of

the

types

of

verification

activities

that

device

manufacturers

often

employ.


4/12


Page4of12



Table 1: Types of Verification Activities

Tests Inspections Analyses

o Material performance /fatigue tests

o Package integrity tests*o Biocompatibility testing of

materials*o Bioburden testing of products

to be sterilized*

o First article inspectiono Comparison of a design to a

previous product having anestablished history ofsuccessful use*

o Worst case analysis of anassembly to verify thatcomponents are deratedproperly and not subject tooverstress during handlingand use*

o Thermal analysis of anassembly to assure thatinternal or surfacetemperatures do not exceed

specified limits*o Fault tree analysis of a

process or design*o Failure modes and effects

analysis*o Engineering analyses:

o Finite Element Analysis(FEA)

o Computational FluidDynamics (CFD)

o Tolerance stack-up

*Source:DesignControlGuidanceForMedicalDeviceManufacturers(1997)

Akeydistinctionbetweendesignverificationanddesignvalidationactivitiesisthatverificationonly

requiresthatasingleunitbeassessed.Whatconstitutesthatsingleunitwillvarydependingonthe

intentoftheverification.Itmightbeonebatchofrawmaterial(formaterialperformancetests),one

machinedpart(forfirstarticleinspection),ononepackagesample(forintegritytests).Theintentof

verificationistoconfirmthatthedesignoutputs(i.e.,thematerialsorcomponentsspecifiedindesign

documents)meetthedesigninputrequirements.

Validations,on

the

other

hand,

require

that

studies

be

conducted

to

ensure

that

the

device

can

be

manufacturedtomeetdesignspecificationsonaconsistentbasis(i.e.,processvalidation),andtoensure

thatthefinisheddeviceissafeandeffectiveforitsintendedpurpose(i.e.,designvalidation).Forthe

processvalidation,multipledevicesmustbemanufacturedandevaluatedtoconfirmthatthe

productionprocessiscapableofproducingdeviceswithinspecificationsonaconsistentbasis.Forthe

designvalidation,multipledevicesmustbeusedtotreatmultiplepatientstoconfirmthatthetreatment

issafeandeffective.


5/12


Page5of12



Thenumberofdevicesthatneedtobeproducedandthenumberofpatientsthatneedtobetreatedis

afunctionoftheriskassociatedwiththeparticularaspectofthedeviceandtheneedtoestablish

confidencein

the

study

results.

As

we

will

see,

the

effective

use

of

risk

analysis

tools

and

solid

verificationresultscanhelptoreducethesizeofthevalidationstudieswithoutnegativelyaffectingthe

confidenceinthestudyresults.

WhatGoesWrong?Asdescribedabove,toooftendesignverificationdoesnotreceivetheamountofattentionneededto

ensuresuccessinthelatterstagesofadevicedevelopmentprogram.Whilenotaperfectrepresentation

ofwhatgoesoninalldevicecompanies,itisinterestingtolookattheFDAsdesigncontrolaudit

findingstoseewhatproblemstheyhavefoundwithcompaniesdesignverificationprograms.Figure1,

showsthenumberofwarninglettersintheFDAsdatabasethatincludefindingsrelatedtospecific

sectionsoftheDesignControlregulation(21CFR820.30).

Figure1:DesignControlWarningLetters


6/12


Page6of12



Interestingly,themostobservationswerefoundwithregardtotheDesignChange,DesignValidation,

andtheGeneraldesigncontrolsubheadings(theGeneralfindingssuggestanoverallfailureofthe

companysdesign

control

process).

We

argue

that

failures

in

these

three

areas

are

largely

the

result

of

poorperformanceintheearlierstagesoftheprogram(e.g.,validationfailuresduetopoorlystructured

inputrequirements,outputsandincompleteverification;anddesignchangefailuresduetoan

ineffectivesystemtoupdateverificationsandvalidationswhenchangesaremade).

Oftheremainingcategories,mostwarningletterscite21CFR820.30(f)DesignVerification.Inthese64warningletters,theFDAidentifiedthatthemanufacturerdidnotcompleteallrequiredverificationsin

nearly45%ofthecases.Alackof(orsignificantgapsin)verificationprocedureswereidentifiedinabout

25%ofthewarningletters.Inaddition,outofspecificationresults,alackofrecordsintheDHF,anda

lackofacceptancecriteriawerefoundin16%,14%,and14%ofthesecompanies,respectively.Figure2

providesa

breakdown

of

all

of

the

types

of

violations

identified

in

these

warning

letters.

Figure2:TypesofDesignVerificationViolations


7/12


Page7of12



Note:92separateviolationswereidentifiedinthe64warningletters.

Why

are

these

failures

occurring?

One

possibility

is

that

by

viewing

verification

as

a

regulation,

device

developersarelosingsightofthefactthatconductingverificationsissimplygoodengineering.For

example,itmakesnosensetomoveforwardwiththedevelopmentofacomponentbeforeyouaresure

thatthematerialpropertiesmeettheperformanceandsafetyrequirementsestablishedinthedesign

inputs.

Thismeetingtheregulationsmindsetcanleadengineerstoviewdesignverificationsimplyasatask

thatmustbecheckedoffbyQAinsteadofvaluabletoolforbuildingknowledgeabouttheirdevice.

Failuresoccurbecausesometimesthecheckoffsgetmissed.

Evenwhenalltheboxesarechecked,theregulationmindsetcandriveengineerstofocusonlythe

minimumnumber

of

verification

activities

needed

to

satisfy

the

design

control

requirements.

While

this

approachmayhelpsatisfyshorttermbudgetorscheduleconstraints,thedevelopmentteamwilllose

theopportunitytolearnpotentiallyimportantinformationabouttheperformanceandbehaviorofits

deviceanditscomponents.Ifthatlearningisputofftothevalidationstage,thecostoffailurecangrow

dramatically.

WhatToDo?Thereareavarietyoftoolsthatdeveloperscanusetoimprovetheeffectivenessoftheirdesign

verificationprocess.Inthispaperwediscussthreetoolsthatcanhelpensurethatverificationactivities

areappropriateandcompleteand,ifwelldocumented,canprovideregulatorswithsufficient

confidenceinthedesignthatexcessivevalidationcanbeavoided.Thethreetoolsare

TraceabiltyMatrix

FailureModesandEffectsAnalysis(FMEA)

FaultTreeofDesignInputs

TraceabilityMatrix:Thetraceability(trace)matrixisabasicdesigncontroltoolthatalldevicedevelopers

should

use

throughout

the

design

control

process

to

establish

clear

linkages

between

Design

Inputs,Outputs,Verifications,ValidationsandRiskAnalyses.ToooftenthetracematrixisleftforQAto

completejustintimeforthefinaldesignreview.However,ifitisbegunassoonasDesignInputsare

approvedandmanagedthroughoutthedevelopmenteffort,thetracematrixprovidedanexcellent

roadmapguidingdevelopersthroughkeystepsofthedesigncontrolprocessandensuringthat

requireddocumentationiscreatedandcontrolled.


8/12


Page8of12



Toillustratehowthetracematrixcanbeused,Table2showswhatonerowinatracematrixmightlook

likewhenthematrixisfirstdevelopedandapproved.Atthisstageofthedevelopmenteffort,thematrix

issolely

a

list

of

the

design

inputs

(the

reference

number

from

the

Product

Requirements

Document,

andadescription).Atthisstagethereisnoinformationabouthowthatdesigninputwillbesatisfied,but

itprovidesaroadmapforwhatquestionsneedtobeaskedandwhatdocumentsneedtobedeveloped.

Astheprojectproceedsthroughthedevelopstages,subsequentcellswillbefilledin,identifyinghow

thedesigninputisbeingmet.

Table2:TraceabilityMatrixatDesignInputsStageTraceabilityMatrixDesignReq.No.

DesignInput(Requirement) DesignOutput

(Specification)

RiskAnalyses

Verification ProcessValidation DesignValidation Comments

1.1.1 CanbeETO

sterilized

Table3illustrateshowarowofatracematrixmightlookwhenbeingreviewedattheFinalDesign

Review. Atthispointthedesignoutputshavebeenapprovedandtherequirementthatthematerialin

questionisappropriateforETOsterilizationisestablishedinDrawingNo.1.2.Thematrixalsoidentifies

thatidentifiedrisksassociatedwiththedevicehavebeenmitigated(inpart)throughtheuseofthis

material.The

references

to

design

FMEA

1.3

and

process

FMEA

5.4

identify

two

places

where

this

materialisaddressed.Verificationwasachievedbyconfirmingthatthespecifiedmaterialisincludedin

thepurchasingbillofmaterial(BOM2.3).TheProcessValidationcolumnidentifiesthatproductionof

thematerialwasassessedinanoperationqualification(OQStudy3.5)andaperformancequalification

(PQStudy4.3).Finally,itshowsthatadesignvalidation(SterilizationStudy2.1)wasconductedto

confirmthatthefinalmanufactureddevicecouldinfactbeeffectivelysterilizedusingETO.

Table3:TraceabilityMatrixatFinalDesignReviewTraceabilityMatrixDesignReq.No.

DesignInput(Requirement) DesignOutput

(Specification)

RiskAnalyses

Verification ProcessValidation DesignValidation Comments

1.1.1 CanbeETO

sterilized

MaterialA

(Dwg1.2)

dFMEA

1.3

pFMEA

5.4

BOM2.3 OQStudy

3.5

PQStudy

4.3

Sterilization

Study2.1

None


9/12


Page9of12



Thetracematrixnowservesasaclearrecordofthechainofanalysesandstudiesusedtoensurethat

thespecificdesigninputismet.Allofthedocumentsreferencedinthetracematrixareincludedinyour

designhistory

file

(DHF),

so

if

a

question

arises

regarding

any

stage

in

the

process,

the

trace

matrix

pointstothekeydocumentationdevelopedatthatstage.

Whilethetracematrixisanecessaryandvaluabletool,itlargelyservesjustarecordkeepingrole.Itdoes

nothingtoinformtheteamabouthowtoprioritizeverificationandvalidationefforts.Forthatinput,risk

analysesareneeded.

FMEA:Likethetracematrix,riskanalysesneedtobestartedearlyinthedesignanddevelopment

processandfilledinandenhancedasprojectproceeds.Whilethispaperwillnotdiscussthedetailsof

theapplicationofriskanalyses,andFMEAsinparticular,wewilladdressthelinkbetweenFMEAsand

verification.

Toooften,riskanalysesareconductedinisolationfromtheotherdesigncontrolactivities.Asdiscussed

earlier,theregulationmindsetcanleadtotheperspectivethatriskanalysesarejustarequiredtaskto

becompletedandnotastoolstosupportdecisionmakingwhichtheyare.

FMEAsareconductedtoprovidedeveloperswithasharedunderstandingoftherisksassociatedwith

theirdeviceusuallyfromthreeperspectives:design,use/application,andprocessing.Theresultofthis

tool,asillustratedinFigure1,istheclassificationofriskstopeople,property,andtheenvironmentinto

threecategories:BroadlyAcceptable,AsLowasReasonablyPracticle(ALARP),andIntolerable.Typically,

applicationofthistoolfocusesontheIntolerablerisksandthedevelopmentofmitigationstrategiesto

reducethe

probability

of

occurrence

for

those

risks,

bringing

them

into

the

ALARP

region.


10/12


Page10of12



Figure2:RiskClassification

Whileriskidentificationandmitigationisclearlytheprimaryobjectiveoftheriskmanagementactivity,

theseanalysescanalsobeusedtofocusverificationandvalidationefforts.Forexample,considera

devicethat

includes

two

components,

one

affects

how

the

device

is

held

by

the

user

and

the

other

comesintocontactwiththepatient.Bothcomponentshaveasimilarnumberofphysicaldimensionsto

verify,butrisksassociatedwiththeuserfacingcomponentareBroadlyAcceptable,whilethepatient

facingcomponentrisksareALARP.WhileaFirstArticleinspectionoftheuserfacingcomponentmay

beappropriate,amorethoroughanalysisofthepatientfacingcomponentmaybewarranted.This

analysiscouldincludeatoleranceanalysisortheinspectionofmultiplepiecestoprovideabetter

indicationofthepotentialcapabilitytoproducethecomponentwithinspecification.

Whiletheadditionaleffortsdescribedabovemaynotberequiredattheverificationstage(conductinga

FirstArticlewouldbesufficienttoclaimthattheverificationiscomplete),theALARPriskclassificationis

an

indication

that

validation

of

this

component

may

be

challenging.

The

more

you

learn

about

this

componentduringverification,thebetterpreparedyouwillbeonceyougettothevalidation.Iffact,if

theverificationisthorough,itmayeliminatetheneedforsomeaspectsofvalidationsavingtimeand

moneylaterinthedevelopmenteffort.

Thekeypointisthatincompleteorminimalverificationsmaynotprovideasufficientunderstandingof

thelikelyperformanceofadeviceduringvalidation,andexcessiveverificationcanbeawasteof

resources.Toeffectivefocusyourverificationefforts,riskanalyses,andFMEAsinparticular,canprovide


11/12


Page11of12



aclearrationaleforhowtofocusyourtimeandresourcespotentiallyreducingvalidation

requirements.

FaultTreeAnalysis:WhileFMEAsareawellacceptedriskanalysistool,oneweaknessoftheapproachisthateachriskisconsideredinisolation.Thereisnoabilitytoassesstheriskoftwoindependentfailures

(i.e.,dimensionAistooshortandtheuserappliestoomuchforce).AsillustratedinFigure3,Faulttree

analysis(FTA)allowsyoutoconsidertheriskoftwoindependentfailuresoccurringinparallel(the

AND),andtheriskofanyoneofaseriesoffailures(theOR).Forexample,inthisBadCoffee

example,addingOldCreamrequiresthecreamtobepastitsexpirationdateANDforthecoffee

drinkernottoreadthedate.Unsatisfactoryservingtemperaturecouldbeduetothecoffeebeingeither

toohotORtoocold.

Figure3:FaultTreeAnalysis

WhilemorechallengingtostructureandassessthananFMEA,theFTAprovidesabetterunderstanding

oftherisksassociatedwiththesystem.Sincetheperformanceofthesystemistheobjectiveof

validationactivities,FTAisanidealtooltoprepareforvalidations.Withthetopofthefaulttree

representingtheintendeduseofthedevice,thistoolallowsdevicedeveloperstofilloutthetree


12/12


Page12of12


identifyingthatrangeofdesignorusagefaultsthatcouldleadtoaproblem.JustastheFMEAhelpedto

focusattentiononindependentrisks,theFTAhelpstofocusonsystemrisks.

Oncethebranchesofatreeassociatedwithasignificantriskareidentified,additionalverification

resourcescanbefocusedonverifyingthedesignofcomponentslinkedtothatrisk.Again,theobjective

ofverificationandvalidationactivitiesistobuildconfidencewithinyourorganizationandwith

regulatorsthatyourdeviceisgoingtobesafeandeffective.AwellstructureFTAcombinedwith

thoroughverificationsofkeycomponentscanbeinstrumentalintheidentificationandresolutionof

problemsearlyinthedevelopmentprocess,allowingyourteamtoenterintothevalidationphaseofthe

processwithconfidenceintheperformanceofthedeviceandtofocusitsvalidationonthosesystem

elementsthatmostdirectlyaffectuserneeds.

WhataretheBenefits?Whileitcansometimesgetlostinthechurningofengineeringprocess,verificationisacriticalelement

ofthedesigncontrolprocess.Whilemeetingthethresholdrequirementofdocumentingaverification

foreachdesigninputmayhelptomovethedesignthroughthedevelopmentstages,suchanapproach

doesnotprovidethevaluethatastrongverificationeffortcouldprovide.Bydevelopingatracematrix

toensurethatalldesigninputsareproperlyaddressedandleveragingriskanalysistools,medicaldevice

developerswouldbebetterabletofocusscarceresourcesonthoseverificationactivitiesthatwill

providethegreatestbenefit.Wellstructuredverificationactivitiesprovidethefoundationfor

validations.Iftheverificationsaresound,validationscanbebetterfocused,helpingtoreducethescope

oftheseactivitiesreducingvalidationcostsandacceleratingtimetomarket.

Documents

Design Verification White Paper