View
4
Download
0
Category
Preview:
Citation preview
Risk-Based Thinking and
the Supplier Audit
Lance Coleman, Sr., ASQ CQE/SSGB/CQA/CBA, Exemplar Global QMS-PR
Kristen Wagner, B.S. Materials Science and Engineering
ASQ Section 0606 Meeting - September 13, 2018
❑ ASQ Senior Member, CQE, CSSGB, CQA, CBA
❑ 2018 ASQ Lean Enterprise Division Chair
❑ ASQ Instructor for Certified Quality Auditor Exam Preparatory Course
❑ Exemplar Global Principle QMS Auditor
❑ 2016-2017 Chair, US TAG 302 –Auditing Management Systems
❑ Voting Member US TAG 176 – Quality Assurance/Quality Management
❑ AAS EET, Southern Polytechnical University
❑ Author, Managing Organizational Risk Using the Supplier Audit Program
(Quality Press 2018)
❑ Author, Advanced Quality Auditing: An Auditor’s Review of Risk
Management, Lean Auditing and Data Analysis (Quality Press 2015)
Lance B. Coleman
(C) Lance Coleman2
❑ Supplier Quality Engineer II at Boston Scientific
❑ B.S. in Materials Science and Engineering from the
University of Minnesota – Twin Cities
❑ Social Media Chair for the Society of Women Engineers –
Minnesota Section
Kristen Wagner
3
Learning Objectives
1. How Risk Based Thinking Leads to Risk
Management
2. Supply Chain Management Overview
3. Supplier Auditing Fundamentals
4. Risk Based Supplier Auditing
5. Useful Tools
• 3-Keys to Asking Good Questions
• External Provider Audit Decision Trees
• Supplier Audit Preparation Checklist
Defining Risk
❑ISO 14971:2007– combination of the
probability of occurrence of harm and the
severity of harm
❑ISO 31000:2009 – the effect of uncertainty
on objectives
❑ISO 9001:2015 – the effect of uncertainty
Note: Hazards are things in an environment
(nouns) that have some risk attached to them
Classifying Risk
❑Consumer Risk – the risk of accepting a
bad part as good
oHarm or injury
oMalfunction
oNot meeting requirements
❑ Producer Risk – the risk of rejecting a good
part as bad
o Part costs
o Other failure costs
For want of a nail
ISO 9001:2015:6.1.1
When planning for the quality management
system, the organization shall consider the
issues...and determine the risks and
opportunities...
“Not all change is improvement but all
improvement is change”
Chuck Anger
V.P. Operations
Ultradent Products Inc.
ISO 9001:2015 Increased
Emphasis on Risk-Based Thinking
❑There is no requirement for formal methods
for risk management or a documented risk
management process in the standard
❑ Organizations can determine how to best
address and manage risk for their needs
❑ An Organizations’ risk management program
should be constantly improving
Risk Based Thinking
Risk-based thinking enables an organization to
determine the factors that could cause its
processes and its quality management system
to deviate from the planned results, to put in
place preventative controls to minimized
negative effects and to make maximum use of
opportunities as they arise
ISO 9001:2015 - 0.1 General
Risk Based Thinking
PROCESS INPUTS RISK MANAGEMENTRisk BasedThinking
- Inspection data
- Audit findings
- Management Review
- Test data
- Continuous Improvement
- Operator feedback
- Risk Model
- Risk Management Plan
- Reporting structure
- Feedback loops
What is an audit?
Results of the evaluation of the collected audit
evidence against audit criteria
ISO 19011:2018
Systematic, independent, and documented
process for obtaining audit evidence and
evaluating it objectively to determine the extent to
which audit criteria are fulfilled
ASQ CQA Handbook, Fourth Edition
And a supplier audit...
Onsite [supplier] verification activity, such as
inspection or examination, of a process or quality
system to ensure compliance to requirements…
The ASQ CSQP Handbook, First Edition
Key purpose is to confirm…
❑ Continued ability to meet product requirements
❑ Continued ability to meet production demands
❑ Maintenance of an effective quality
management system
❑ Maintenance of positive supplier relationship
Supplier vs. Internal Audit
❑Less process knowledge
❑Less visibility
❑Less control
❑Planning lead time
Supplier Audit Process
❑ Selection of Suppliers to Audit
o Supplier Assessment
o Supply Chain Management Feedback Loop
❑ Planning Audit
o Purpose, Scope, Resources
❑ Conducting Audit
❑ Report Write-Up
❑ Audit Follow-Up and Closure
Selection of Suppliers to Audit
❑ New or current supplier?
❑ Supplier assessments including surveys
❑ Supplier performance management
including scorecards
❑ Supply Chain Management feedback loop
Supplier Survey
❑ Should cover cross-functional areas
oQMS, Facilities, Engineering, R&D, Etc.
❑ Should include a section for key
personnel contact information
❑ Should be used to select areas of
focus for the on-site audit
Supply Chain Management
Supplier Performance Risk Based Decision
Feedback
Supplier
Management- Audits
- Inspection/testing
- Document Review
- Data Collection
- Reporting
(c) 2012 Lance Coleman
- Supplier risk classification
- Supplier Quality
- On-Time-Delivery
- Continue supplier
- Downgrade supplier
- Upgrade supplier
- Adjust rating
- Discontinue supplier
Supplier Performance Monitoring
Tangibles
❑ Quality(e.g. - Nonconformances and Supplier Corrective Action Requests issued)
❑ On-Time-Delivery
❑ In-Full-Delivery
❑ Audit results
Supplier Performance Monitoring
Intangibles
❑ Supplier partnership
o Participation in Continuous Improvement,
Six Sigma, and/or Lean Projects
o Excellent service
o Easy to work with
❑ Responsive to questions or concerns
❑ Open and forthcoming with information
Applying Risk Based Thinking to
Audit Planning
1) Understand financial, time, and personnel
resource allocation
2) Perform risk assessment of supplier pool
o Creation of justification during exercise
3) Utilize risk assessment learnings to support
supplier selection and justification for more
resources as needed
Risk Assessment Inputs
❑ New or current supplier
❑ Single or multi-sourced product
❑ Supplier audit schedule
❑ Major observations or findings from previous audit
❑ Results of change impact assessment
o Significant change of business focus for supplier
o Acquisition
o New management
o High employee turnover
❑ Supplier Performance Monitoring results
o SCAR and NC frequency
o Deviations from procedures and/or specifications
o Poor communication
❑ Internal, external, or third-party rejections
Audit Schedule Decision Tree
Possibiliy of serious injury
Significant Performance Degradation
Possibility of Injury
Significant Business Risk
Performance Degradation
ISO Certified
ISO Certified
Somewhat to very likely
Onsite Audit Somewhat to very likely
Somewhat to very likely
Onsite Audit Onsite Audit
Cosmetic Issues
Somewhat to very likely
ISO Certified
Audit
Audit
No AuditNew Supplier/
Product
Y Y
N
Y Y
YY
N
N N
N N NN
N
N
N
N
Y
Y
Y Y
Y
Y
Y
If Everything Is Important, Then Nothing Is
Apply impact values to risk assessment inputs
based on…
❑ Company strategy
❑ Industry requirements
❑ Business needs
…to acquire final risk score per supplier
Backstory
❑ There are 3 suppliers that are being
considered for audits this year
o Supplier A, B, and C
❑ Resource restrictions dictate that you can
only select 2 suppliers to audit
Who do you choose?
Supplier Risk Matrix
Supplier A
❑ Current Classification A supplier that is due for an
audit this year
❑ There were 2 marginal findings during the last audit;
however they are easily corrected given supplier
dedication
❑ A new inspector was hired; however, the supplier
provided training records
❑The supplier has had numerous issues with on-time-
delivery in the past year
❑ With the implementation of a safety stock, the Supply
Chain team has been able to handle the on-time
delivery issues with minimal customer upsets;
however, upper management set a strategic company
goal to reduce inventory cost which requires a
reduction in safety stock
Supplier B
❑ Current Classification B supplier for a raw material with
several non-active alternative qualified suppliers
❑ Supplier not due for audit this year
❑ There were minor nonconformances during the last audit;
however, they were corrected the day of the audit
❑ A new inspector was hired; however, the supplier provided
training records
❑ There have been several instances of nonconforming
product that Incoming has captured. There have been
sporadic issues on the production floor connected to this
material that is causing product failures.
❑ Some product failures were not captured under final
inspection and made it’s way to a customer. The customer
sent the product to a third party organization to confirm the
failure.
Supplier C
❑ Supplier C is a new Classification A supplier
that is connected to a company top priority
project.
o New suppliers’ risk assessment should have their
own unique risk factor inputs based on intended
use and should involve project team collaboration
❑ No previous performance information is
known about supplier
❑ Material provided by supplier is a new
material to your company
Supplier AImpact Value Mitigation Value Risk Ranking
Supplier Classification 4 3 12
Due for Scheduled Audit 4 3 12
Major Findings from Previous Audit 3 2 6
Change Impact Assessment Results 1 1 1
Supplier Performance Monitoring Results 4 3 12
Internal Rejections 3 3 9
External Rejections 2 2 4
Third-Party Rejections 0 0 0
Final Risk Score:56
Supplier BImpact Value Mitigation Value Risk Ranking
Supplier Classification 2 2 4
Due for Scheduled Audit 1 1 1
Major Findings from Previous Audit 1 1 1
Change Impact Assessment Results 1 1 1
Supplier Performance Monitoring Results 4 3 12
Internal Rejections 4 4 16
External Rejections 4 3 12
Third-Party Rejections 4 3 12
Final Risk Score:59
Which suppliers should be audited?
❑ Supplier C
o Most Risk
o Unknown performance
o New material to company
o Connection to a company strategic project
o BUDGET SAVINGS! Apply audit expense under
budget for project
❑ Supplier B
o Even though Supplier A is due for an audit and
there is potential risk to meeting customer orders,
Supplier B has more realized risk per the risk
assessment
What about Supplier A?
JUSTIFICATION
Use learnings from risk assessment and other
supplier management tools, such as
performance monitoring, to build a case for
additional resources
Important Do’s and Don’ts
Do : demonstrate risk based thinking throughout
your QMS
Don’t: just sprinkle the word risk throughout your
documents with nothing to support the concept
Do : ask and answer the hard questions
Don’t: state that you will do something in order to
meet the requirement and then don’t do it
Do : include an appropriate amount of specificity
in your procedures
Don’t: write yourself into a corner or overuse or
imply phrases such as whenever, always, every
time
Supplier Audit Needs Decision Tree
Conducting Risk Based Audits
❑ Directly audit the risk management (RM)
program itself
❑ Conduct RBQA of aspects of the QMS or
of the QMS as a whole
o Standalone risk management audit of
QMS elements
o Incorporate risk management into
existing audits
Robust Risk Management
For a truly robust risk management program,
the following as a minimum should occur:
1. The program should encompass all aspects
of a product life cycle from design to end-of-
life disposal.
2. Data from external as well as internal
sources should be captured and analyzed
and the risk model updated as necessary.
3.Teams, when formed, should be cross-
functional in nature in order to model the
broadest range of risks
Auditing the RM Program
1. First, confirm all three of the items from the
previous slide are occurring
2. Confirm that results from the risk management
program are reported as necessary to
appropriate levels of management
3. Confirm that existing risk management
procedures and work instructions are followed
4. Ensure that organizational training supports the
risk management program
5. Confirm that adequate resources are supplied
to meet the goals of the risk management
program
Where does risk lie in our process?
1. Complexity of the process
2. Complexity of the product
3. Criticality of the product
4. Location where most processing has occurred
5. Newness of the product
6. Newness of employees
7. History of the process
Risky Behavior
How do we know risky behavior or situations
when we see them?
• Variance from industry norms
• Employee concerns
• Established feedback channels
• Identified in risk management plan
Risky Behavior
You will often know it when you see it...
Scenario: An auditor visits a machine shop
and witnesses a welding operation where
sparks are flying in the immediate vicinity. The
operator is following their instructions and
wearing the appropriate PPE, but the auditor
notices a small puddle of oil on the floor
nearby.
Identify the hazards, risks, and mitigations
(controls) in this scenario.
Identifying Hazards & Risks
3-Keys to Good Questioning
1. Ask the right question
2. Ask the obvious question
3. Let one question lead you to the next
Risk Based Thinking & Audit Findings
Key Thought:
In thinking about how risk affects the
classification of audit findings, we can look at
the risk of individual nonconformities, the risk
found within aspects of the quality management
system, or the risk found within the overall
quality management system
Risk Level Definitions - 1
RA-Significant Risk:
• Potential for product contamination, complete
product failure, or serious supply chain
disruption
• Potential violation of customs/regulatory
requirements or blatant disregard of
Technical/Quality Agreement
• Multiple systemic or chronic deviations from the
requirements
• Conformity required within three months or
according to agreement
Risk Level Definitions - 2
B-Moderate Risk:
• Required procedures do not exist or exist, but not
implemented or followed
• Lack of awareness or attention to cGMP requirements
• Lack of IT and other technical resources available and
appropriate to the size of the business
• Deficient management systems to handle Customer
Service, Production Planning, and Inbound/ Outbound
Logistics
• Significant number of instances of partial fulfillment of
requirements
• System is not achieving defined objectives
• Compliance required within six months or according to
agreement
Risk Level Definitions - 3
C-Minor Risk:
• Regulatory, ISO, contractual, and internal requirements
met in principle but not in full
• Current system needs additional focus and other
improvements
• Compliance required within six months or according to
agreement
Note: In this case, it is not each individual audit finding
that is assessed for risk, but rather the aggregate effect
of all audit nonconformities related to a particular
aspect of the quality management system and the risk
of those nonconformities causing a failure in the
system
QMS
Improperly
completed forms
and records
(Information still
retrievable)
Violation of
internal procedure
or work
instruction;
Current practice
that meets
requirement is not
accurately
documented
Violation of customer
requirement or
internal requirement.
Systemic or chronic
failure of QMS
requirement. Multiple
related minor
violations. Cause great
harm to other
operations in the
company.
Noncompliance
that is itself a
hazard or may
lead to hazardous
condition. Direct
violation of ISO
standards or
cGMP. Absense of
required procedure
or record
Impact Negligible
(1)
Minor
(2)
Major
(3)
Critical
(4)
Applying Risk Based Thinking to Audit Findings
Failure Likelihood Estimation Chart
LikelihoodProbability
RankDefinition
Very Low1
Unlikely to happen, rare, remote
Low2
Can happen, but not frequently
High3
Likely to happen, often, frequent
Very High4
Very likely to happen, more often than not
Applying Risk Based Thinking to Audit Findings
A
RISK
MATRIX
IMPACT
L
i
k
e
l
i
h
o
o
d
Negligible
(1)
Moderate
(2)
Marginal
(3)
Critical
(4)
Very Unlikely
(1)
Unlikely
(2)
Likely
(3)
Very Likely
(4)
Low Risk Medium Risk High Risk
Applying Risk Based Thinking to Audit Findings
Risk Based Audit Findings
Risk Level Low Medium High
Risk Level Description
Nonconformities that do not
affect form, fit or function.
Documentation errors that can
be fixed.
Regulatory noncompliances
other than those that could
cause injury, harm or
malfunction.
Product nonconformities
that may partially inhibit
function.
Noncompliances which could
cause harm or injury to end
users, distributors, company
employees or the public at
large.
Noncompliances that could
cause significant or total
functional failures.
Finding Classification Minor Minor/Major Major/Critical
“the single biggest problem in communication
is the illusion that it has taken place”
George Bernard Shaw
Audit Report Content - 1
A typical individual supplier audit report will include:
❑ Audit purpose and scope
❑ Audit criteria
❑ Lead Auditor and audit team members
❑ Summary of results
❑ Result details including identified risks
❑ Audit finding definitions/explanations
❑ Review of findings (nonconformities, opportunities for
improvement, and positive practices)
❑ Review of corrective actions
❑ Identification of the need (or not) for a follow-up audit
❑ Opportunities to improve for your organization
Audit Report Content - 2
The risk-based individual supplier audit report
will also address:
❑ Identified supplier risks and their
assessments
❑ Risk mitigations
❑ Residual risks
❑ Any risks attached to how your organization
conducts business with the supplier
Report Distribution
❑ Send report to supplier and open the floor
for discussion
❑Establish dates for any action items and
follow-up as necessary
❑ Discuss report with supplier stakeholders
❑Summarize risks and report to upper
management as needed
Key Takeaways
1. Applying risk based thinking to audits
allows us to ask better questions and
provide more critical review of data
2. When interviewing, it is important to ask
the correct and most obvious questions
3. Risk should be defined by each
organization according to their business
model
4. Following a structured approach for
supplier auditing will provide better results
ReferencesManaging Organizational Risk Using the
Supplier Audit Program
ISBN: 978-0-87389-968-0
Quality Press 2018
The Certified Supplier Quality Professional
Handbook
ISBN: 978-0-87389-943-7
Quality Press 2017
Advanced Quality Auditing
ISBN: 978-0-87389-913-0
Quality Press 2015
Performance Metrics:
The Levers for Process Management
Quality Press 2013
ISBN-13: 978-0873898508
Next Steps
❑ Assess your risk management program to
ensure that all types of risk are addressed
❑ Review your procedures to ensure
appropriate level of specificity and no
overuse of whenever, always, every and so
forth.
❑ Take ownership of closing identified gaps
Questions ??? Lance B. Coleman ASQ CQE/CSSGB/CQA/CBAQA&R ManagerIDEX Health and Science LLCSeattle, WA 98108360-682-4242Lcoleman@idexcorp.comhttps://www.linkedin.com/in/lance-b-coleman-asq-cqe-cssgb-cqa-cba-rabqsa-cqms-pr-7418131b/
Kristen WagnerSupplier Quality Engineer IIBoston ScientificMaple Grove, MN 55311KMWagner.MSE@gmail.comhttps://www.linkedin.com/in/kristen-wagner
Recommended