How to Audit Risk Management

Company Confidential

Registration Management Committee (RMC)

1

How to Audit Risk Management

Atlanta, GAJuly 22 & 23, 2010

Kimberly MaggieRon Tarach

QUAL-TECH, INC.

Auditor WorkshopAtlanta, GA

July 22-23, 2010


Atlanta, GAJuly 22-23, 2010 2

Agenda• What is Risk?• Risk Management Process • Examples Risk Management Criteria• Auditor perceptions of Risk Management• Risk Management Tools

– Auditor knowledge of tools and actions



Agenda (continued)• Audit Planning

– Audit Planning Tools• Activity 1 - Brainstorming session using

Audit Planning Tool• Conducting the Audit of Risk Management

Process– Examples of areas to evaluate

• Activity 2 - Brainstorming session using Case Study and Failure Modes and Effects Analysis (FMEA)



Ice Breaker!



What is Risk?An undesirable situation or circumstance that

has both a likelihood of occurring and a potentially negative consequence.

AS9100:2009, clause 3.1



“Risk is inherent in all processes. Unfortunately, we don’t see the results of ineffective risk management methods

until later”.



Risk Management Process – Most organizations spend a great deal of time and

manpower trying to document “Risks” but many times this data is decentralized and not easily accessible to the functions that need this information.

– Process manufacturing can be so complex that “Risks” can be very subtle and if there is not a structured “Risk Management Process” that takes advantage of corporate knowledge, lessons learned an organization’s exposure to “Risk” can remain high.





Examples of Risk Management Criteria» Understanding the types of risk that could come

into a company. They could be related to• Employees• Process• Design• Manufacturing• Equipment• Environment• Project• Security



Examples of Risk Management Criteria» Understanding the types of risk that could come

into a company cont. • External• Contractor



Examples of Risk Management Criteria (continued)

– Employees – the organizations need to ensure the safety, training, and qualifications of employees.

– Process – managing process variation.– Design – building quality into the product

design from the start, including it’s affect on planning.

– Manufacturing – ensuring that manufacturing is more efficient with streamlined quality planning.



Criteria for Risk Management Process (continued)

– Equipment – ensuring that equipment can meet capabilities, current and future.

– Environment – ensuring that the operations are not compromising the environment (adequate lighting, temperature control, noise, cleanliness, etc).

– Security – managing the security needed by the facility.

– Project – ensuring project risks are evaluated before beginning.



Criteria for Risk Management Process (continued)

– External – developing plans to address the potential impact of weather, issues with transportation companies, city infrastructure (relating to construction, road closures).

– Contractor – ensuring impact is considered for contractors working on the building, equipment, or with employees.



Auditor Perceptions of Risk Management• That’s the way we identified and handled risk

when I worked at Aviation Anywhere, Inc.• When I audited a Original Equipment

Manufacturer (OEM) last month they were using FMEAs.

• This little company only uses tool XYZ – they can’t be managing risk properly.



Auditor Perceptions of Risk Management (continued)

“Remember, the design and implementation of an organization’s aerospace quality management system is influenced by varying needs, particular objectives, the products provided, the processes employed and the size and structure of the organization.”

AS9100:2009 General



Auditor Perceptions of Risk Management (continued)

• Organizational application of Risk can vary based on situation, customer, product line.

• Audit approach & interviewing will need to be appropriate to the organization.

• Remember, what is “Appropriate” to the organization.





Risk Management Tools– FMEAs e.g. dFMEA, pFMEA, etc.– Fault Tree Analysis (FTA)– Probabilistic Risk Assessment (PRA)– Event Tree Analysis (ETA)– Event Sequence Diagram (ESD)– Master Logic Diagrams (MLD)– Reliability Block Diagram (RBD)



Risk Management Tools (continued)– Risk Assessment Matrix– Likeliness/Consequence Table– SWOT (Strength Weakness Opportunity

Threat)– Business Continuity/Current Capability

Matrix– Risk Map and Control Scale



Risk Management Tools (continued)– Auditor knowledge of tools and actions

» No one auditor has experience with all the tools available in the industry and how they are used.

» Familiarize your self with the various Risk Management Tools (self study).



Risk controlled – or “Oh No”?



Risk Management Tools (FMEA)



Risk Management Tools (Influencer Analysis)



Risk Management Tools (Risk Consequence)



Risk Management Tools



Audit Planning– Selecting the right audit tool.– Identifying your audit criteria and any

reference documents.– Identifying your audit scope, including

identification of the organizational and functional units and processes to be audited.

– Identifying an appropriate audit scope.



Audit Planning Tools– Process (Turtle) Tool– Process Map Tool– Supplier Input Process Output Customer

(SIPOC) Form– Process Based Management (PBM) Process

Flow



Process (Turtle) Tool With What

(Materials, Equipment, Facilities)

Inputs (information and

material from other

processes)

How?

(Methods/Procedures/Techniques

With Who?

(Comp./Skills/Training)

Outputs (information

and Material to other

processes

How Effective/Efficient?

(Measurable Objective)

Process



Process Map



Supplier Input Process Output Customer (SIPOC) Form



Process Based Management (PBM) Process Flow



Activity 1 - Brainstorming session using Audit Planning Tool



Process (Turtle) Tool (Design) With What

Risk Management Software

Forms

Documents

Inputs

Customer, Internal Organization, Regulatory, Statutory

Special Requirements (e.g. product or process complexity)

Critical Items (functions, parts, software, characteristics, processes)

How?

AS9100, AS9110 and AS9120 Standards

Quality Manual

Standard Operating Procedure for Contracts

FMEA

Risk Assessment Matrix

With Who?

Sales

Engineering

Production

Quality

Outputs

Design

Planning

Production

Purchasing

Suppliers

Shipping


Customer complaints

In process/final rejection

Design verification/validation

Process

Contract Review

- Risk Management

Outputs

Drawing/Spec

Travelers

Routers

Work Orders

Inspection Reports



Process (Turtle) Tool (Design Excluded) With What

Risk Management Software

Forms

Documents

Inputs

Customer, Internal Organization, Regulatory, Statutory

Special Requirements (e.g. product or process complexity)

Critical Items (functions, parts, software, characteristics, processes)

How?

AS9100, AS9110 and AS9120 Standards

Quality Manual

Standard Operating Procedure for Contracts

FMEA

Risk Assessment Matrix

With Who?

Sales

Engineering

Production

Quality

Outputs

Planning

Production

Purchasing

Suppliers

Shipping


Customer complaints

In process rejection

Final rejection

Process

Contract Review

- Risk Management

Outputs

Travelers

Routers

Work Orders

Inspection Reports



Conducting the Audit of Risk Management Process

– Examples of areas to evaluate» Are all “Risk” identified during the RFQ and Contract

Review Process e.g. special requirements, critical requirements.

» Ensure Top management clearly understands what “Risks” they have and what they are doing to ensure they are mitigating those “Risk”.

» Evaluate the selected Risk Management Tool for effectiveness.

» How are “Risks” communicated and managed throughout the organization e.g. Design, Planning, Purchasing, Suppliers, Manufacturing, Inspection, Delivery and Post Delivery.

» Design inputs, Design FMEAs, Design Verification and Validation.




– Examples of areas to evaluate continued» Critical characteristics across the quality lifecycle,

ensuring the Process FMEAs and Control Plans are linked.

» Processes in place for capturing leading and lagging indicators related to Design Quality Performance.

» Evaluate whether the organization has closed loop Continual Improvement Processes that captures and sustains Product and Process Quality.

» Organization is using Lessons Learned and Best Practices.




– Examples of areas to evaluate continued» Ensure organization’s Change Management Process

involves the right people at the right time with the right process.

» Ensure integration of Change Management with assessments to ensure correct consideration of “Risk”.

» Ensure “Risk Assessment” tracked, recommended controls to completion and ensured that “Risk” were mitigated as prescribed.

» Ensure controls are in place for “Risk” that still remain after mitigation actions.



Activity 2 - Brainstorming session using Case Study and FMEA



Closing!



Questions!



References1.AS9100:20092.ISO 190113.FAA Risk Management Handbook 20094.NASA