Embed Size (px)
Citation preview
2
Agenda
• AS9100 Importance to MDA
• MDA Assurance Provisions (MAP)
• MDA Audit Process
• AS9100 Findings
• Analysis
• Examples of AS9100 Audit Findings
• Common AS9100 Issues
• How You Can Help
• Conclusion
3
AS9100Important to MDA
• AS9100 is a key QMS document – MDA Core standard
• The MAP builds on AS9100 requirements/disciplines to create a very rigorous set of QMS implementation requirements
• MAP includes additional Quality, Safety, and Mission Assurance (QSMA) disciplines for developing mission and safety critical items such as:
– Systems Engineering
– Reliability, Maintainability, and Availability
– Parts and Materials Management
– Software
– Risk Management
– Configuration Management
– Safety and Environmental Management
– Interface Management
4
MDA Assurance Provisions (MAP)
• A standardized set of 14 QSMA provisions, requirements, and processes for mission and safety critical items:
– Authored by joint Gov/Industry Team
– Goal is to implement on all MDA contracts
– 160 pages, 1800+ shall statements
• Provide methods to measure, verify, and validate mission success through:
– Collection and analysis of metrics
– Risk Assessment, technical evaluations, independent assessment, reviews
– Conduct of qualification, ground, and flight tests
• Promote continual process improvement to ensure QSMA, reduce cost, and improve productivity
Proven Mission Assurance Requirements on Contracts
MDA Assurance Provisions (MAP)
•3.1 Management
•3.2 Design and Development
•3.3 Software and Firmware
•3.4 Technical and Mission Assurance Reviews
•3.5 Reliability, Maintainability, and Availability
•3.6 Parts and Materials Control Program
•3.7 Integrated Test and Evaluation Program
•3.8 Test, Meas., and Diagnostic Equip. & Stds.
•3.9 Interface Management
•3.10 Configuration Management
•3.11 Control of Nonconforming Items & Materials
•3.12 Fabrication and Quality
•3.13 Supplier Management
•3.14 Safety
5
MDA MAP Program Usage
• As part of the MDA Acquisition Strategy the MAP was developed for MDA Program Office implementation and flow down to prime contractors and suppliers
– Use existing processes, when adequate, to achieve and sustain mission success
– Tailored by the MDA Program Offices for each program
– Further tailored for subsystem components through supplier management
• Implementation of MAP (including AS9100) assessed via:
– Mission Assurance audits
– Supplier audits
– On-Site Mission Assurance Representatives (MARs)
– Focus Teams – (VAFB Flight Test Team, PMRF, NGST)
6
Why Audit?
QS Organizationfounded in Late 2002
Majority of Failures Were Lack of Process Discipline
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Aegis
NMD/GMD
THAAD
FM-5 6/03FTR-1 7/00
IFT-01 1/97 IFT-04 1/00
IFT-05 7/00
IFT-10 12/02 IFT-13c 12/04
IFT-14 2/05
FT-07 3/97
FT-08 5/98
FT-09 3/99
No Interceptor
FailuresBV-2 8/01
BV-3 12/01 BV-5 1/04
22 Flight Test / 10 Failures
(45% failure rate)
32 Flight Test / 4 Failures
(13% failure rate)
17 Flight Test / 0 Failures
(0% failure rate)
First MDA/QS Mission Assurance Audit performed
7
GMD Flight Test-03a28 Sept 07
• Engages
- Target launch from Kodiak
- Interceptor launched VAFB
- In midcourse phase of flight
- EKV discrimination
8
MDA QS Audit Program
• Purpose
– Conduct Mission Assurance audits of MDA prime contractors and suppliers
• MDA Audit Objectives
– Validate that the end item at a selected level conforms to engineering design requirements
– Validate that the end item at a selected level is qualified
– Compare fabrication methods and acceptance test levels against MDA and industry standards as appropriate for MDA use
– Validate production readiness
– Assess contractor/government Mission Assurance practices, procedures, and implementation
• MDA Audit Findings
– Deficiency - a non-compliance to a contractual requirement or internal procedure/process
• Includes AS9100 deficiencies (if AS9100 is on contract or if audit site is certified or compliant)
– Observation - when an MDA or Industry “best-practice” should be used
9
All MDA Audits CY05-CY08
Program Program Contractor Location Date ofType
of CommentsOffice Audit Audit
EKV GM Raytheon Tucson, AZ 28 Feb - 11 Mar 05 MASM3 AB Raytheon Tucson, AZ 28 Feb - 11 Mar 05 MATHAAD TH Lockheed Martin Troy, AL 25 - 29 Apr 05 MATHAAD TH Lockheed Martin Sunnyvale, CA 2 - 6 May 05 MAOBV GM Orbital Chandler, AZ 21 - 30 Jun 05 MASBX GM Kiewit Corpus Christi, TX 26 Jul - 4 Aug 05 MA Weld AuditSBX GM AmFELS Brownsville, TX 26 Jul - 4 Aug 05 MA Weld AuditCMCM TC MIT/LL Lexington, MA 30 Sep - 1 Oct 05 MA Lab AuditSTSS SS Northrop Grumman Redondo Beach, CA 25 Oct - 4 Nov 05 MASTSS SS Raytheon El Segundo, CA 25 Oct - 4 Nov 05 MASM3 AB Aerojet Camden, AR 6-8 Dec 05 Safety Safety AuditSBX GM Boeing Pearl Harbor, HI 30 Jan - 10 Feb 06 MA Weld AuditGBI/KEI/STARS GM/KI/DET ATK Magna, UT 31 Jan - 2 Feb 06 Safety Safety AuditGMD/KEI GM/KI ATK Elkton, MD 25 Apr - 4 May 06 MASM3 AB ATK Elkton, MD 25 Jul - 3 Aug 06 MAEKV GM Raytheon Tucson, AZ 22-31 Aug 06 MA EKV AssessmentOBV GM Boeing/Orbital VAFB, CA 7-16 Nov 06 MAFBX-T SN Raytheon VAFB, CA 7-16 Nov 06 MA Minimum HardwareNFIRE Booster/Targets SS Orbital VAFB, CA 7-16 Nov 06 MA Minimum HardwareNFIRE SS General Dynamics Gilbert, AZ 7-16 Nov 06 MAEKV IMU/THAAD TARS GM/TH Northrop Grumman Canton, MA 9-18 Jan 07 MAEKV Subsystems GM Raytheon Tucson, AZ 19-28 June 07 MA 5 EKV SubsystemsTHAAD TC Orbital Chandler, AZ 21-30 Aug 07 MA SRALT/MRT/TLV
GMD/KEI/TC GM/KI/TC ATK Magna, Clearfield, Promontory, UT 6-16 Nov 07 MAGMD/KEI/TC/SM3/THAAD GM/KI/TC/AB/TH Pacific Scientific Chandler, AZ & Hollister, CA 17-28 Mar 08 MASM3/GMD/KEI AB/GM/KI L-3 Comm TW San Diego, CA 13-22 May 08 MA
MDA QS Mission Assurance and Safety Audits
10
AS9100 Findings
MDA Audit Results
0
20
40
60
80
100
120
140
7.5 4.2 7.4 7.3 7.6 7.1 8.3 8.2 6.2 4.3 7.2 8.5 8.4 8.1
AS9100 Requirement
Nu
mb
er
of
No
nc
on
form
an
ce
s
11
Analysis
• Majority of MDA deficiencies can be tied to an AS9100 element
• AS9100 related deficiencies are primarily in the following areas:
– Production and Service Provision: Quality Records, Procedural Discipline, Foreign Object Debris/Damage (FOD) , and Electrostatic Discharge (ESD)
– Documentation Requirements: Control of Documents (Change and Approval), Quality Management System Records
– Purchasing: Supplier Requirements Flow Down, Supplier Rating System
– Design and Development: Inputs/Outputs, Traceability, Design Reviews, Design Verification/Validation
– Control of Monitoring and Measuring Devices: Design and Selection, Calibration, and Tool Control
– Planning of Product Realization: Establishment and Documentation of Processes
– Control of Nonconforming Product: Nonconformance Disposition and Nonconforming Material Identification and Segregation
12
Finding Examples
• Production and Service Provision, AS9100, paragraph 7.5
– Drawing parameters and/or pass/fail criteria were not accurately represented in the build paper.
– Foreign Object Debris/Damage (FOD) was found in the Building Powder Blend Area.
– Limited life materials were expired and available for use in the production of deliverable hardware.
– Electrostatic discharge equipment continuity and environmental limits are not adequately controlled in the Bonded Stores and Electrical Inspection Areas.
– Desiccant used to minimize moisture in motor manufacturing is not used per Manufacturing and Inspection Records (M&IR) or drawing requirements.
– Preventive Maintenance Work Orders are not verified and Repair Work Orders are not generated to correct deficiencies.
13
Finding Examples
• Documentation Requirements, AS9100, paragraph 4.2
– The audited program did not implement enterprise company practices for requirement traceability, and did not have an approved process for the unique implementation of their particular practices.
– X-Ray inspection procedures are not controlled.
– Changes to a Mix and Cast Building checklist were not approved before use.
– Receiving Inspection Records were missing required documentation (Inspection report, Government Source Inspection Stamp, Material Certification).
– The contractor could not provide supporting information that correlated or related to waivers and deviations.
– Uncontrolled and improperly marked documentation were posted in the Process Control Lab.
14
Finding Examples
• Purchasing, AS9100, paragraph 7.4
– Parts that did not comply with all supplier quality assurance requirements were accepted at Receiving Inspection.
– The contractor’s Approved Vendor List does not include some vendors and does not reflect the vendor’s quality rating.
– The prime contractor Statement of Work imposes an incorrect quality requirement.
– The contractor did not impose core quality requirements (corrective action system, inspection procedures, and quality records) on suppliers.
– The contractor accepted a supplier certificate of analysis that certified material for a storage temperature in conflict with the material specification.
15
Finding Examples
• Design and Development, AS9100, paragraph 7.3
– Contractor Automated Test Station configuration identification does not reflect the current approved configuration.
– The contractor does not have a formal process for documenting and tracking design review action items and responses.
• Control of Monitoring and Measuring Devices, AS9100, paragraph 7.6
– Calibrated equipment at a work station in the Destructive Lot Acceptance Test Area had broken integrity seals.
– Test and Measuring Equipment with expired calibration was in the receiving inspection area.
16
Finding Examples
• Planning of Product Realization, AS9100, paragraph 7.1
– The contractor’s software development practices were not documented and were dependent on the cognizant engineer.
– The mass test limit for the Inertial Measurement Unit was incorrectly noted on the test sheet in the Engineering Note Book .
• Control of Nonconforming Product, AS9100, paragraph 8.3
– The contractor processed “Use as Is” dispositions for hardware without proper review and approval, including the customer.
– Undocumented items (various mechanical, electrical and optical parts) were noted in the bonded Material Review Board lockers and cages which appear to be scrap.
17
Common AS9100 Issues
• Interpretation varies between contractors and government organizations
– AS9100 Paragraph 7.5.1.1, Production Documentation: Production operations shall be carried out in accordance with approved data
• MDA Interpretation: Mission and Safety Critical processes shall be carried out in accordance with approved detailed plans and procedures
– AS9100 Paragraph 8.3, Control of Nonconforming Product: The organization shall ensure that product that does not conform to product requirements is identified and controlled to prevent its unintended use or delivery
• MDA Interpretation: Nonconforming product is conspicuously identified (tagged), segregated, and retained in a hold status until dispositioned by formal review process
– AS9100 Paragraph 4.3 Configuration Management: The organization shall establish, document and maintain a configuration management process appropriate to the product. NOTE: Guidance on configuration management is given in ISO 10007
• MDA Interpretation: A formal Configuration Management Program includes: a CM Plan which addresses Configuration Planning, Configuration Identification, Configuration Change Management (Waivers/Deviations), Configuration Status Accounting, Configuration Verification / Audit, and CM of Digital Data
18
Common AS9100 Issues Continued
• Compliance and interpretation varies from contractor to contractor
– Some contractors manufacture mission and safety critical hardware per an approved drawing, while others use an approved drawing, traveler, and detailed procedures
– Contractors advertise AS9100 Certification, however, implementation varies based on programs and funding
• Certification achieved at division level, but not implemented across all programs
– MDA experience: Many contractors have stated that AS9100 requires them to have a documented system, but the documented system does not have to be effective or implemented for compliance.
19
AS9100How You Can Help
• Strengthen Implementation Requirements
– MDA experience: There is a gap between system assessment and implementation. AS9100 certification should include assessment of critical processes and the associated detailed planning and procedures.
• Audit/Assess MDA Programs as part of Certification
– Identify MDA Programs with MAP on Contract
– Assess effectiveness of Corrective Actions from MDA Audit Findings
– Provide feedback for high risk MDA Suppliers
20
Conclusion
• Majority of MDA deficiencies can be tied to an AS9100 Element
• Most audit findings are violations of internal processes
– Not following their own procedures or processes
• Our audits have indicated gaps in PDCA implementation
– Most sites do the “Plan” portion
– They do less well with the “Do” portion
– Even do less well with the “Check” portion
– Most “Act” actions are short term, focusing on immediate fixes rather than well thought out long term actions that address systemic issues and improve performance across the enterprise
Output of Requirements Must be Defined
Backup
22
MDA Audit FindingsMAP to AS9100 Correlation
0
20
40
60
80
100
120
140
7.5 4.2 7.4 7.3 7.6 7.1 8.3 8.2 6.2 4.3 7.2 8.5 8.4 8.1
AS9100 Requirement
0
100
200
300
400
500
600
3.12 3.10 3.14 3.1 3.2 3.13 3.5 3.7 3.3 3.6 3.8 3.11 3.4 3.9
MAP Requirement
No
nco
nfo
rman
ces
No
nco
nfo
rman
ces
