Company Confidential Registration Management Committee (RMC) 1 AQMS Accreditation Programs ANAB Findings Atlanta, GA July 22, 2010 Steve Holladay ANAB,

Company Confidential

Registration Management Committee (RMC)

1

AQMS Accreditation Programs ANAB Findings

Atlanta, GAJuly 22, 2010

Steve HolladayANAB, Accreditation Assessor

Auditor WorkshopAtlanta, GA

July 22-23, 2010


Atlanta, GAJuly 22-23, 2010

Objectives

• Provide an overview of the NCR’s identified from the witness audits.

• Present in process approach.

• Discuss steps that could have been taken to prevent the NCR’s.

• Rules of engagement.

2



Goal

• Provide knowledge/information to the AS auditor pool with the goal to improve auditor/CB performance that will reduce repeat NCR’s which will, in turn, add value to the assessment process and the industry as a whole.

3


Atlanta, GAJuly 22-23, 2010 4

• 120 NCR’s in 54 Witnessed Audits (WA)– 2.22 Average per WA

• 50 NCR’s in approx 40 Office Assessments (OA)– 1.25 Average per OA

Overall



• 3 NCR

• DETAIL:– Assigning Auditors who either did not have the

proper NACE code or AS91XX qualifications.

– Client not made aware of the OP assessor.

• LEARNINGS: – Verify auditors are qualified for the full

dynamics of the audit.

– Communicate clearly with the client.

Pre-Audit Assignment



• 11 NCR’s

• DETAIL: 19011, 6.4.1.c– (4) The audit plan did not ID the organizations

functional units and processes to be audited.

• LEARNINGS:– The audit plan ensure the client will have the

proper resources available to the audit.

– Clearly demonstrates what processes are intended to be audited.

Pre-Audit Planning



• 11 NCR’s

• DETAIL: 17021. 9.3.2.1– (2) Audit Planning not effective in assuring the

AQMS is assessed to the min requirements for surveillance audits.

• LEARNINGS:– Ensure the auditors understand where the full

auditing requirements are for surv audits if not clearly ID in the audit plan.

Pre-Audit Planning



• 11 NCR’s

• DETAIL: MD 5 and AS9104– (5) Insufficient Auditor Days in the planning

without clear justification.

• LEARNINGS:– If deviation…..JUSTIFY.

– If the audit is being witnessed, provide to ANAB in the pre-audit information.

Pre-Audit Planning



• 5 NCR’s, several OFI’s.

• DETAIL:– Information missing or not correct in EQM.

– Client Profile form not complete or accurate. (scope, head count, regulatory, ITAR)

• LEARNINGS:– Ensure the person entering information is

knowledgeable about the information.

– Verify with Client and lead auditor before submitting.

Pre-Audit Preparation



• Total of 24 NCR’s

• DETAIL:– Sampling part of an element and making a

conclusion on the whole requirement.

• LEARNINGS:– Audit plan and report accuracy is critical.

On-site Audit Depth of auditing



• DETAIL: – Auditing to the scope of certification is not

adequate.

• LEARNINGS:– Ensuring the scope of certification, scope of

audit and audit plan is consistent.

– Validate the exclusions. Is scope consistent with the QM.

– Ensure the address and scope on the Certificate is correct.




• DETAIL: – Not following audit trails when objective

evidence suggest otherwise.

• LEARNINGS:– A plan is a plan. Follow the trail to its natural

conclusion if potential findings are evident.

– Keep head on a swivel. Don’t ignore clear findings.




• DETAIL: – Not fully verifying the effectiveness of the

actions taken on nonconformities identified during the previous audit.

• LEARNINGS:– Ensure the evidence is more than a record

review.




• 4 NCR’s

• DETAIL:– Design Applicability

• LEARNINGS– More of an issue with AS9110.

On-site Audit Interpretation



• DETAIL:– Not fully auditing Outsourcing when

applicable.

• LEARNINGS– Standard clearly requires “control of such

outsourced processes shall be identified” and therefore subject to audit. Not just a document verification.




• DETAIL:– Changes to the capability listing of the client.

• LEARNINGS– Similar to the scoping discussion. Verify any

changes to the capability listing as this could affect the scope of certification or introduce new processes and technology.




• 11 NCR’s

• DETAIL: – Calling the Clients CA/PA system effective with

ineffective/inadequate correction, root cause or corrective action.

• LEARNINGS:– Deep dive the CA/PA system. Validate the

information. If it doesn’t make since….make the call.

On-site Audit Decisions



• DETAIL:– Soft Grading 8 NCR’s

– Clearly stating findings during the audit and not raising the NCR or improper categorization.

– Accepting informal correction during the audit.

• LEARNINGS– Learn the definitions and follow them. Keep track

of the “verbal” findings identified during the audit. Accepting correction to verbal findings without reporting is consulting.

– As an auditor it is not your job to justify an NCR down.




• DETAIL:– Continuing the audit when the audit objectives

are clearly unattainable.

• LEARNINGS:– Only the client can make the decision to

continue an audit.

– There should be a clear conclusion that the objectives are unattainable and reported to the client with options.» At the time this is realized not at the end!!




• DETAIL: – Closing meeting does not address all of the

requirements.

• LEARNINGS– Clearly address the CA follow up and impacts

on the existing cert.

– Ensure the CA are presented to the correct requirements.

On-Site Audit Closing



• 22 NCR’s

• DETAIL:– Accuracy of the Audit Report and Checklist to

the observed audit.

• LEARNINGS– Anecdotal evidence should be documented as

such. Same for deductive evidence.

On-site/Off-site Audit Reporting



• DETAIL: – Report not in conformance with

AS9014/AS91XX

• LEARNINGS– Justify/explain all NA.

– Clear evidence to support the conclusion.

– Clear evidence to support the SCOPE.

– Include detailed notes

On Site/Off-Site Audit Reporting



• 3 NCR’s

• DETAIL:– Report did not contain mandatory items.

• LEARNINGS– Ensure the Checklist are complete or

references the location of the information.

– Explain and differences between the information left on site to the published report.

Post Audit Reporting



• 8 NCR’s

• DETAIL: Improper closing of NCR’s and Improper Certification Decision.

• LEARNINGS:– Clear evidence of Correction, RC and CA.

– Ensure persons closing NCR are AEA.

– Ensure the NCR’s are closed with appropriate evidence PRIOR to the cert decision.

Post Audit NCR Closure



Office Audit NCR’s

• OASIS data base admin/accuracy

• Verification of Client OASIS data admin

• Justification of auditor days.

• Improper qualification of AS cert decision maker.

• Improper decision made with certification information.

25



Questions

Documents

Company Confidential Registration Management Committee (RMC) 1 AQMS Accreditation Programs ANAB Findings Atlanta, GA July 22, 2010 Steve Holladay ANAB,