Improving Safety by Reducing Design

Assurance Overhead

Presented to

AEA / GAMA Rotorcraft Forum

April 8, 2015

Discussion Topics

• Design Assurance Considerations for Change

• Proportionate Design Assurance

• Certification Uncertainty Considerations for Change

• Conclusion and Recommendations

Design Assurance Considerations for Change

Do Assurance Processes Contribute to Improved Aviation

Safety Record?

Accident Rate Source: US DOT, Research and Innovative Technology Administration (RITA), Bureau of Transportation Statistics (BTS) National Transportation Statistics

Table 2-14 - US General Aviation Safety Data (downloaded 29-Jan-2015). 2011 flight hours not available. Excludes US registered civil aircraft operated under 14 CFR 121

or 14 CFR 135. Accidents on foreign soil and in foreign waters excluded.

• 1999-2012: Total and fatal accident rates largely unchanged

• Neither design assurance nor technology appear to have obvious effect

How to Determine if Design Assurance or Technology Has

Impact on Safety Record?

• Observations:

• CY2001 – CY2012, Loss of Control – Inflight (LOC-I) is largest category followed by

Controlled Flight into Terrain (CFIT) then System Component Failure – Powerplant

(SCF-PP)

• CY2012, LOC-I is still largest category but CFIT and SCF-PP have changed positions

• Is this change an anomaly or a persistent trend? See next slide

GAJSC GA Fatal Accidents CY2001 – CY2012 by

Top Ten CICTT Occurrence Category

GAJSC GA Fatal Accidents CY2012 by Top Ten

CICTT Occurrence Category

CFIT Fatal Accident Rate Reduction: Design Assurance or

Technology?

• Observations:• Portables appear to have had larger effect on CFIT accident rate reduction than certified avionics

• FAA credits certified MFD terrain display with contributing to Alaska CFIT accident rate reduction

• Portables have no design assurance; certified MFD incorporates Windows-based COTS operating

system*

• Conclusion: Consistent with GAJSC general consensus: CFIT fatal accident rate reduction is due to

technology not design assurance

• 2002 - 2003, certified products

had higher total due to MFD w/

terrain display

• 2004 - 2012, portables w/

terrain/obstacle alerting have

higher total and increasingly

higher sales rate than

combined certified MFDs w/

terrain display and certified

avionics w/ terrain/obstacle

alerting, whose sales started in

2005

• Certified glass cockpits have

minimal impact due to low

quantities

*via Notice 8110.92, Guidelines for Applying RTCA/DO-178B Level D to Previously Developed Software, now Order 8110.49, Software Approval Guidelines, Chapter 8

Experimental Amateur-Built (E-AB) Accidents: Due to Lack of

Design Assurance?

Source: NTSB, NTSB/SS-12/01, The Safety of Experimental Amateur-Built Aircraft, Aviation Safety Study, May 22, 2012, Figure 13, p. 24

• NTSB studied E-AB accidents from

2001-2011

• Observations:

• E-AB accident categories similar

to certified aircraft but SCF-PP is

largest category followed by

LOC-I; CFIT is 8th

• E-AB installed avionics have no

design assurance requirement

• 12 recommendations to FAA;

none related to design

assurance

• Conclusion: E-AB accident rate is

not due to lack of design assurance

Certified vs. E-AB Avionics Comparison

Characteristic Certified G1000 GDU E-AB G3X GDU

Functionality PFD; MFD; digital AHRS & air data; 3-axis autopilot

with envelope protection and electronic stability

protection; electronic engine gauges, GPS/SBAS

RNAV en route, terminal, approach navigation

including LPV, LNAV/VNAV and LNAV with advisory

vertical guidance; synthetic vision; terrain/obstacle

alerting; electronic geo-referenced departure,

arrival, approach charts and airport diagrams;

SafeTaxi®; weather radar; datalink weather; traffic;

transponder control; VHF com control; VOR / ILS

navigation; radar altimeter

Same capabilities except:

• weather radar, and radar altimeter not supported

TSOs / Non-TSO Functions 36 / 8 0 / 0

Design Assurance Levels A, B, C, D, E None

Software Source Lines of

Code1.6 million 2.4 million

Software Open Problem

Reports> 460 < 10

Software Release Schedule 1 per year As needed

System Price ~ 15% of G1000 for dual display configuration

with comparable functionality

How to Reduce LOC-I Accidents?

• What happened here? Likely reduction in largest LOC-I accident category but why?

• FAA and industry organization (AOPA, EAA, GAMA, etc.) coordinated safety message to GA

community focused on LOC-I accident causes may have had an effect:

• Medical issues coupled with prescription medicine effects

• Restoration of E-AB Phase 1 flight mentoring program

• VFR flight into IMC

• Preliminary 2014 data shows accident rate back to “normal” rates

• Continuing progress in preventing LOC-I accidents requires:

• Continuing improvement in pilot training reliability coupled with

• Readily available, cost-effective technological mitigations enabling fleet-wide adoption rate

approximating that which affected CFIT fatal accident rate reduction

• Comparatively, other accident rate improvements, including all forms of design

assurance, are immeasurably small

GAJSC GA Fatal Accidents CY2001 – CY2012 by Top Ten CICTT Occurrence Category

Proportionate Design Assurance

Applies Across

Transportation Modes

Safety Continuum – slide source

Zero Risk

No

Operations

No

Innovation

Societally

Accepted

Risk & Desire for

Low Cost

•Part 25 Transport Category Passenger Aircraft & UAS Risk Class 6

•Amateur Built

•sUAS Risk Class 1&2

•Models

•Large Part 25 Business Jets

•Part 23 Commuter Aircraft & UAS Risk Class 6

•Part 23 Business Jets

•Part 23 Light Jets, Twins, & UAS Risk Class 5

•Part 23 Single Engine & UAS Risk Class 4

•Light Sport Aircraft & UAS Risk Class 3

Society’s

Demand for

Safe Outcomes

Society’s Aviation Safety Expectations – slide source

Proportionate Design Assurance

• 1999 AC 23.1309-1C introduced guidance to proportionally apply design

assurance level (DAL) based on safety continuum concept

• Proportionate DAL model is appropriate since system capability, intended use, and

desired function, not concern for design assurance, drive safety-enhancing

avionics purchases

• Guidance was initially useful to target Garmin glass cockpit platform for part 23

Class I & II aircraft (catastrophic DAL = C) then expand into Class III aircraft

(catastrophic DAL = B)

• No longer useful to Garmin since sustainable glass cockpit platform target market

requires spanning part 23 Class I to Class IV aircraft (catastrophic DAL = A)

• Never useful for retrofit equipment that spans parts 23, 25, 27, & 29 aircraft

• Need significant expansion of proportionate design assurance model to

achieve significant safety improvement

Expand Design Assurance Proportionality Across All Aircraft

Parts

• Consistent with safety continuum concept, proportionate design assurance model should

be expanded within part 23 and applied to parts 25, 27, & 29

• Reducing certification burden will help retrofit all aircraft with NextGen technology

Ideas for Expanding Design Assurance Proportionality (1 of 2)

• Apply scalable certification requirements akin to AC 23.1309

• Part 23 rules currently being revised

• Other parts should be similarly revised

• Change failure classification methodology

• Currently focused on failures of installed equipment rather than risk associated with

not installing equipment

• Credit should be given to safety improvements realized by installation of equipment

(e.g., digital AHRS-based attitude display, H/TAWS, traffic, moving map, etc.)

• Other standards use a cost / risk / probability / value ratio; e.g., bridges / buildings

outside earthquake zones are not designed to worst-case earthquake standards

• Exposure interval is not accounted for in design assurance level

• Implement policy that reduces HIRF/lightning requirements

• Part 23 and 27 aircraft subject to same requirements as 100+ passenger part 25

aircraft (see AC 20-136B, “Lightning”, Table 1 and AC 20-158A, “HIRF”, Table 1)

Ideas for Expanding Design Assurance Proportionality (2 of 2)

• Implement policy that promotes installation of autopilots with envelope

protection and electronic stability protection (ESP) capability

• Autopilots with these capabilities are available and desired by market

• Modern digital AHRS are much more reliable than iron gyros but new autopilots

using digital AHRS are held to higher design assurance requirements than old

autopilots using iron gyros

• To improve installation rate provide relief by accepting use of single digital AHRS

• Implement policy that allows E-AB glass cockpit avionics installation in part

23 and 27 aircraft <= 6 passengers

• No DO-178 software or DO-254 AEH design assurance

• Instead, use ASTM F39 “Standard Specification for Verification of Avionics Systems”

for black-box system-level requirements verification that will also cover software and

AEH

• Aircraft level hazards must be appropriately addressed/minimized

• Mitigation could include E-AB PFD miscompare monitoring from dual E-AB AHRS

in combination with certified backup primary flight instruments (attitude, altitude,

airspeed)

Streamline Aircraft-Level Design Assurance Audits

• Equipment manufacturers subject to repeated audits by cert authorities and

OEMs

• Audits often on the same processes/standards, LRUs and/or “critical” functions

• Such audits do not result in improved function or safety

• Preparation, participation, and post-audit activities can reduce safety due to lost

opportunity cost to address bigger picture activities that do improve safety

• Implement “trusted” applicant design assurance process oversight

• Define criteria for trusted auditors (who, what, when/how often, etc.)

• Audit equipment manufacturer using trusted auditor criteria

• Upon successful audit completion, issue “letter of acceptance” (LOA) similar to

AC 20-153A database LOA

• LOA specifies demonstrated areas of competence such as design assurance

levels, tool qualification, etc.

• LOA specifies conditions for continued acceptance such as process error

reporting, process change submittal, and internal audits

Streamline Aircraft-Level Design Assurance Issue Papers

• Eliminate/reduce design assurance issue papers (IP) applied by cert

authorities

• Increase cost but no real evidence of improved safety

• Micro-oversight of design assurance processes & equipment designs; e.g.,

graphics co-processor monitor, multi-core processor and single event effects

• Inconsistently applied; e.g., to same avionics installations in different aircraft and

from one authority to another

• Cert authority changes to IPs without industry review require updates to

previously accepted response without added safety

• Assumption seems to be that if a technology did not exist at the time a standard or

guidance was written then additional requirements must be applied via IP

• DO-178B/C, DO-254, ARP 4754A are performance-based objective standards

that continue to work well as technology steps forward

• IPs should be reserved for situations where performance-based objectives truly do

not cover new technology

Multi-Core Processor (MCP) Issue Paper Example (1 of 2)

Evolution of Processor / Peripheral Integration

Multi-Core Processor (MCP) Issue Paper Example (2 of 2)

• Processor / peripheral integration has been occurring for > 30 years

• Past integrations similar to MCP have also raised concerns about complexity and

inability to completely test design, e.g., floating-point and graphics co-processors

• Design errors present since 1st generation CPU where instructions and interrupts did not

work

• Industry concerns with MCP CAST-32 and related cert authority issue papers

• Some CAST-32 “objectives” are prescriptive in nature rather than being performance-

based and scalable as technology changes like those in DO-178B/C and DO-254

• Other CAST-32 objectives require detailed design knowledge of the MCP itself and thus

are impractical for avionics manufacturer to accomplish

• Existing guidance is applicable, scalable, and sufficient for MCP designs

• DO-254/ED-90 section 11.2, DO-178B/ED-12B sections 2.3 and 6.3, & EASA CM-

SWCEH-001 chapter 9

Proportionate Application of ARP 4754A

• Cert authorities note concerns with

increasing system integration and

incorrect/incomplete classification of

system requirement errors as software

development/verification errors

• To address these issues, cert authorities

are applying ARP 4754A more broadly

• OEMs are pushing ARP 4754A system requirement development & validation down

to avionics manufacturers

• Avionics manufacturers have finite resources

• Need proportionate application of 4754A

• Eliminate/reduce DO-178() statement, decision, & modified condition/decision

coverage activities/objectives that have low probability of finding system errors

• Doing so will free resources for activities like:

• System test, which has high probability of finding system errors and

• Open problem report resolution to reduce errors in certified systems

Figure Source: FAA System Safety Handbook

Proportionate Application of Cyber Security

• DO-326A scope recently limited to part 25 with approved seating configuration of

more than 19 passenger seats

• Cert authority cyber security policy applying

DO-326A should be similarly proportionate

• Cyber security assurance should be based on

predictable and credible determination of

Risk = Threat x Vulnerability x Consequence

• Postulated aviation cyber security Threats

typically associated with taking over

flight-critical avionics

• Mitigating factors already in place; e.g.,

flight crew must approve data linked flight

plan changes and can disconnect autopilot

• Smaller aircraft pose smaller Threat based on safety continuum concept

• Consumer product cyber security Vulnerabilities typically due to open operating system and

open protocols, which are atypical in aviation

• Vulnerability “research” reported in press lacks credibility; e.g., does not use certified

avionics hardware and software and does not include entire avionics suite alleged to be

affected

• Even assuming worst-case Consequence, Risk is very low due to low Threat x low Vulnerability

Certification Uncertainty Considerations for Change

Eliminate Certification Uncertainty (1 of 2)

• System safety assessment

• New project 2x.1309 level of effort and deliverables ever increasing, even when new

installation is based on previously certified aircraft installation / functions

• Cert authority retains findings even when delegated organization has capability

• Late cert authority comments disruptive to project schedule

• Additional effort / deliverables not based on regulation, policy, guidance or

referenced industry standards

• Significant cost without corresponding safety benefit

Eliminate Certification Uncertainty (2 of 2)

• Flight test assessment

• Issues with assessment of new functions, e.g.:

• New minor airspace label function occasionally covers red or yellow obstacle on

situational awareness moving map

• Current regulations, policy, and guidance do not address the issue nor

preclude the design

• Pilots understand moving map use and accept it is not for obstacle alerting

• Pilots also understand significant benefit of workload reduction associated with

presenting airspace data directly on map

• Additionally, pilots have options to turn off airspace labels and/or view

obstacles on dedicated terrain/obstacle moving map

• Same software includes new major safety-enhancing powerline display/alerting

function

• ~16% of all rotorcraft accidents attributed to wire or obstacle strikes*

• Result: Certification uncertainty delays fielding safety-enhancing functionality,

ultimately decreasing overall safety

*As quoted in Aviation International News AINalerts: March 30, 2015 “Wire obstacles or strikes present a significant danger to helicopter operations, according to a recent

United States Helicopter Safety Team (USHST) analysis into the dangers of low-altitude flight. Approximately 16 percent of all helicopter accidents have been attributed to

wire or obstacle strikes, the report says.”

Streamline Aircraft-Level Avionics Certification Across Part 23,

25, 27, & 29

• Apply AC 20-180 AML STC across all aircraft parts consistent with FAA SAD

application to part 23

• Implement Part 23 ARC Report section 5.0 recommendations applicable to all

aircraft parts including:

• FAA Conformity, Minor Change Approval, Applicant Showing Only, Test Witnessing

via Remote Video/Video Recording

• Streamline ability to field software updates

• Technology (e.g., wireless) outstripping regulation

• Too burdensome for owner-operators to update software (even DAL E) without

becoming entangled in regulations spanning multiple areas of FAA oversight

(certification, operation, owner-maintenance, aircraft records, etc.)

• Consistent with E-AB methodology, part 23 aircraft with maximum seating

capacity of 6 or less should be able to update software in similar fashion to part

43.3(k) aeronautical database updates

• Consider how to streamline software updates for other aircraft

Conclusion and Recommendations

Before conscientious effort to

simplify and improve safety

… after simplification

Assurance Process Recommended Changes (1 of 2)

Recognize that perfection is the enemy of safety

• Joint cert authority / industry effort is required to improve safety

• The investment in design assurance has already exceeded

its ability to positively impact safety

• Further increasing avionics design assurance investment is not

rational based on the accident data since the safety return on

the present investment is small and not commensurate with the

safety returns experienced with safety-enhancing technology

• i.e., it is inconsistent with risk management principals to require design assurance to

protect for failures that occur 1 / 100,000 hours that block avionics installations that

reduce accidents that occur 1 / 1,000 hours

• Data driven risk management should stop/reduce cert authority “latest scary technology”

research that results in additional design assurance guidance/requirements with low

safety return

• Raising the “level of certitude” by requiring new design assurance processes &

documentation to address issues having low safety return actually reduces overall safety

by shrinking resources available to address items on the safety Pareto chart (e.g.,

LOC-I, CFIT) that have substantial safety return

Assurance Process Recommended Changes (2 of 2)

Recognize that perfection is the enemy of safety

• Eliminate/reduce authority-induced certification uncertainty

• Consider that:

• Accident analysis has not implicated lack of design assurance

or improper design assurance as a factor in accident causes

• “Design assurance” ≠ “good design”; i.e., it is possible to have

excellent design assurance on a poor design

• Move safety focus from “inside the avionics” to actions that encourage certifying and

installing cost-effective technology solutions like autopilots and HTAWS to reduce

accident causes like LOC-I and CFIT

• Design assurance requirements must be proportionate with the safety continuum

concept

Acronyms• AC – Advisory Circular

• AEA – Aircraft Electronics Association

• AEH – Airborne Electronic Hardware

• AHRS – Attitude/Heading Reference System

• AML – Approved Model List

• AOPA – Aircraft Owners and Pilots Association

• ARC – Aviation Rulemaking Committee

• ASTM – American Society for Testing and Materials

• BTS – Bureau of Transportation Statistics

• CAST – Certification Authorities Software Team

• CAST – Commercial Aviation Safety Team

• CICTT – CAST/ICAO Common Taxonomy Team

• CFIT – Controlled Flight Into Terrain

• CFR – Code of Federal Regulations

• COTS – Commercial-Off-The-Shelf

• CPU – Central Processing Unit

• CY – Calendar Year

• DAL – Design Assurance Level

• DMA – Direct Memory Access

• DOT – Department of Transportation

• E-AB – Experimental Amateur-Built

• EAA – Experimental Aircraft Association

• EASA – European Aviation Safety Agency

• EIR – Enroute Instrument Rating

• ESP – Electronic Stability Protection

• FAA – Federal Aviation Administration

• GA – General Aviation

• GAJSC – General Aviation Joint Steering Committee

• GAMA – General Aviation Manufacturers Association

• GPS – Global Positioning System

• HIRF – High-Intensity Radiated Fields

• HTAWS – Helicopter Terrain Awareness and Warning System

• ICAO – International Civil Aviation Organization

• IFR – Instrument Flight Rules

• ILS – Instrument Landing System

• IMC – Instrument Meteorological Conditions

• IP – Issue Paper

• LNAV – Lateral NAVigation

• LOA – Letter of Acceptance

• LOC-I – Loss of Control – Inflight

• LPV – Localizer Performance with Vertical guidance

• LRU – Line Replaceable Unit

• MCP – Multi-Core Processor

• MFD – Multi-Function Display

• NTSB – National Transportation Safety Board

• OEM – Original Equipment Manufacturer

• PFD – Primary Flight Display

• RITA – Research and Innovative Technology Administration

• RNAV – aRea NAVigation

• SAD – Small Airplane Directorate

• SBAS – Space-Based Augmentation System

• SCF-PP – System Component Failure – Powerplant

• STC – Supplemental Type Certificate

• TAWS – Terrain Awareness and Warning System

• TSO – Technical Standard Order

• UART – Universal Asynchronous Receiver/Transmitter

• UAS – Unmanned Aircraft System

• US – United States

• VFR – Visual Flight Rules

• VHF – Very High Frequency

• VNAV – Vertical NAVigation

• VOR – VHF Omni-directional Range